.. _brash_distributed_robo_sim:

Running BRASH in a multi-robot setup
######################################

The previous tutorials have shown how to run Brash using two setups:

* *Single-host:* Run everything in a single machine.
* *Multi-host:* Simulate running a multi-computer system using Docker in one machine.

In this tutorial, we'll run the same RoboSim example from tutorials :ref:`3<brash_robotsim_demo>` and :ref:`6<brash_docker_robo_sim>`, 
but now in a more realistic setup with multiple machines.

Pre-Requisites:
================ 
You'll need 3 machines within the same network:

* *fsw* :  Spacecraft machine that will run the cFS process.  
* *rosgsw* : Ground machine from which the user will send commands using ROS2 and will receive robot data to visualize in Rviz/RQT
* *rosfsw* : Flight robot machine that runs ROS2 (e.g. a rover, a robot arm).


You'll need to have installed either cFS or brash on the machines as follows:

* *fsw* : Install cFS (steps :ref:`here<install_cfs>`)
* *rosgsw* : Install Brash (steps: :ref:`here<install_brash>`)
* *rosfsw* : Install Brash (steps: :ref:`here<install_brash>`)


.. note::

   If your machines are in the same network, ROS2 by default will make information such as 
   topics visible among machines. To avoid this, set the  **ROS_DOMAIN_ID** environmental variable
   to different non-zero values in your **rosgsw** and **rosfsw** machines (e.g. 20 and 21), 
   such that by default they do not eavesdrop on each other.

Set up IP addresses:
====================

Start by writing down the IP addresses of your 3 machines. You 
can use the **ifconfig** command in your terminal to get this information.


fsw machine
------------

1. In your *fsw* machine, go to *sample_defs/tables*: 

   .. code-block:: bash

      $ cd ~/cFS 
      $ cd sample_defs/tables

2. And edit *cpu2_sbn_conf_tbl.c* . Modify the IPs for Self and rosfsw (processors
   2 and 3) to be the IPs for your **fsw** and **rosfsw** machines. respectively.
   Leave the ports (2235, and 2236) as they are.
   
   .. code-block:: c
      
      .Peers =
        {
           {/* [0] - Corresponds to cpu1 and not used in typical brash config */
             .ProcessorID  = 1, 
             .SpacecraftID = 0x42,
             .NetNum       = 0,
             .ProtocolName = "UDP",
             .Filters      = {"Remap"},
             .Address      = "10.5.0.3:2234",
             .TaskFlags    = SBN_TASK_RECV},
           {/* [1] - Self */
             .ProcessorID  = 2,
             .SpacecraftID = 0x42,
             .NetNum       = 0,
             .ProtocolName = "UDP",
             .Filters      = {"Remap"},
             .Address      = "XXX.XXX.XXX.XXX:2235", // <== WRITE fsw's IP HERE !!! 
             .TaskFlags    = SBN_TASK_POLL},
           {/* [2] - rosfsw */
             .ProcessorID  = 3,
             .SpacecraftID = 0x42,
             .NetNum       = 0,
             .ProtocolName = "UDP",
             .Filters      = {"Remap"},
             .Address      = "XXX.XXX.XXX.XXX:2236", // <== WRITE rosfsw's IP HERE!!!
             .TaskFlags    = SBN_TASK_POLL},
        },
      .PeerCnt = 3};

3. Build cFS, so your changes take effect:

   .. code-block:: bash
      
      $ cd ~/cFS
      $ make && make install

rosgsw machine
--------------

1. Open a terminal and go to **cfe_plugin** in your brash workspace:

   .. code-block:: bash
      
      cd ~/brash
      cd src/cfe_ros2_bridge_plugin/cfe_plugin/config
      
2. Open the **cfe_config_multihost.yaml** file and change the value of **udp_command_ip**
   to be the IP of your **fsw** machine:
   
   .. code-block:: yaml

      cfe_bridge:
        ros__parameters:
          plugin_name: cfe_plugin.cfe_plugin
          namespace: /groundsystem
          plugin_params:
            udp_telemetry_ip: "0.0.0.0"
            udp_command_ip: "XXX.XXX.XXX.XXX" # <== WRITE fsw's IP HERE !!!  
            udp_receive_port: 1235
            udp_send_port: 1235       # 1234 for cpu1 (singlehost setup), 1235 for cpu2 (multihost setup)      


rosfsw machine
---------------

1. Open a terminal and go to **cfe_sbn_plugin** in your brash workspace:

   .. code-block:: bash
      
      cd ~/brash
      cd src/cfe_sbn_bridge_plugin/cfe_sbn_plugin/config
      
2. Open the **cfe_sbn_config_multihost.yaml** file and change the value of **peer_ip**
   to be the IP of your **fsw** machine:
   
   .. code-block:: yaml

      cfe_sbn_bridge:
        ros__parameters:
          plugin_name: cfe_sbn_plugin.cfe_sbn_plugin
          namespace: /flightsystem
          plugin_params:
            msg_pkg: cfe_msgs
            udp_receive_port: 2236
            udp_receive_ip: '0.0.0.0'
            spacecraft_id: 66
            processor_id: 3
            epoch_delta: 315532800
            rosout_name_max_length: 32
            rosout_msg_max_length: 128
            rosout_file_max_length: 64
            rosout_function_max_length: 32
            # fsw information
            peer_ip: "XXX.XXX.XXX.XXX"         # <== WRITE fsw's IP HERE !!! 
            peer_port: 2235             # 2234 for cpu1, 2235 for cpu2
            peer_processor_id: 2        # Processor id (1 or 2, for cpu1 - singlehost setup, or cpu2 - multihost setup)


You are all set and ready to run!


Running the RobotSim demo in a distributed setup:
=================================================

1. **fsw machine**  : Run cFS. Make sure to use **cpu2** :

   .. code-block:: bash
    
      $ cd ~/cFS/build/exe/cpu2
      $ ./core-cpu2 2>&1 | tee out.txt


2. **rosgsw machine** : Run the cfe/ROS2 bridge:

   .. code-block:: bash
    
      $ cd ~/code/brash
      $ . install/local_setup.bash
      $ ros2 launch cfe_plugin cfe_bridge.launch.py cfe_config:=cfe_config_multihost.yaml

   Note that we are using the **cfe_config** argument to use the yaml file which we edited earlier
   to have our setup IPs. 

3. **rosgsw machine** : In another terminal, start the UI tools that will allow us to send commands and visualize data from the flight system:

   .. code-block:: bash

      $ cd ~/code/brash
      $ . install/local_setup.bash
      $ ros2 launch brash_demo robot_sim_tracarm.launch.py 

   In order to get telemetry data, we need to send a request. Go to the rqt window,
   in the Message Publisher plugin, edit the **dest_ip** parameter for the **/groundsystem/to_lab_enable_output_cmd** topic and 
   set the value to be the **rosgsw IP** .  

   .. figure:: _static/rqt_distributed_setup.png
      :width: 800px
      :alt: groundsystem_robosim_tracarm
      :align: center

   Once you right-click on top of the topic and click *Publish once*, you should see the robot shown straight up, in a zero pose. If you
   move the sliders in the joint publisher gui, you'll see that the robot starts to slowly move in Rviz. The slow motion is due to the 
   fact that the joint states are being calculated in the **fsw** machine and being returned through the bridge at 1Hz. 

4. **rosfsw machine** : For this demo, the rosfsw machine is not doing any actual calculation or interacting with an actual robot, 
   so here we'll just use it to show how ROS2 data can be shown in the flight side using the SBN bridge. Go to the terminal:
   
   .. code-block:: bash

      $ cd ~/code/brash
      $ . install/local_setup.bash
      $ ros2 launch cfe_sbn_plugin cfe_sbn_bridge.launch.py cfe_sbn_config:=cfe_sbn_config_multihost.yaml

   Note that we are using the yaml file that we edited earlier to have our setup IPs.   
   
   
  That is it! You should now have access to the **/flightsystem** topics, which are coming from the **fsw** machine. You could 
  use rqt to visualize this data.