lerobot/examples/8_use_stretch.md

This tutorial explains how to use Stretch 3 with LeRobot.

Setup

Familiarize yourself with Stretch by following its tutorials (recommended).

To use LeRobot on Stretch, 3 options are available:

• tethered setup
• untethered setup
• ssh directly into Stretch (you will first need to install and configure openssh-server on stretch using one of the two above setups)

Install LeRobot

On Stretch's CLI, follow these steps:

1. Install Miniconda:

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash

2. Comment out these lines in ~/.profile (this can mess up paths used by conda and ~/.local/bin should already be in your PATH)

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

3. Restart shell or source ~/.bashrc

4. Create and activate a fresh conda environment for lerobot

conda create -y -n lerobot python=3.10 && conda activate lerobot

5. Clone LeRobot:

git clone https://github.com/huggingface/lerobot.git ~/lerobot

6. Install LeRobot with stretch dependencies:

cd ~/lerobot && pip install --no-binary=av -e ".[stretch]"

Note: If you get this message, you can ignore it: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.

7. Run a system check to make sure your robot is ready:

stretch_system_check.py

Note: You may need to free the "robot process" after booting Stretch by running stretch_free_robot_process.py. For more info this Stretch's doc.

You should get something like this:

For use with S T R E T C H (R) from Hello Robot Inc.
---------------------------------------------------------------------

Model = Stretch 3
Tool = DexWrist 3 w/ Gripper
Serial Number = stretch-se3-3054

---- Checking Hardware ----
[Pass] Comms are ready
[Pass] Actuators are ready
[Warn] Sensors not ready (IMU AZ = -10.19 out of range -10.1 to -9.5)
[Pass] Battery voltage is 13.6 V

---- Checking Software ----
[Pass] Ubuntu 22.04 is ready
[Pass] All APT pkgs are setup correctly
[Pass] Firmware is up-to-date
[Pass] Python pkgs are up-to-date
[Pass] ROS2 Humble is ready

Teleoperate, record a dataset and run a policy

Calibrate (Optional) Before operating Stretch, you need to home it first. Be mindful about giving Stretch some space as this procedure will move the robot's arm and gripper. Now run this command:

python lerobot/scripts/control_robot.py \
    --robot.type=stretch \
    --control.type=calibrate

This is equivalent to running stretch_robot_home.py

Note: If you run any of the LeRobot scripts below and Stretch is not properly homed, it will automatically home/calibrate first.

Teleoperate Before trying teleoperation, you need activate the gamepad controller by pressing the middle button. For more info, see Stretch's doc.

Now try out teleoperation (see above documentation to learn about the gamepad controls):

python lerobot/scripts/control_robot.py \
    --robot.type=stretch \
    --control.type=teleoperate

This is essentially the same as running stretch_gamepad_teleop.py

Record a dataset Once you're familiar with the gamepad controls and after a bit of practice, you can try to record your first dataset with Stretch.

If you want to use the Hugging Face hub features for uploading your dataset and you haven't previously done it, make sure you've logged in using a write-access token, which can be generated from the Hugging Face settings:

huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential

Store your Hugging Face repository name in a variable to run these commands:

HF_USER=$(huggingface-cli whoami | head -n 1)
echo $HF_USER

Record one episode:

python lerobot/scripts/control_robot.py \
  --robot.type=stretch \
  --control.type=record \
  --control.fps=30 \
  --control.single_task="Grasp a lego block and put it in the bin." \
  --control.repo_id=${HF_USER}/stretch_test \
  --control.tags='["tutorial"]' \
  --control.warmup_time_s=5 \
  --control.episode_time_s=30 \
  --control.reset_time_s=30 \
  --control.num_episodes=2 \
  --control.push_to_hub=true

Note: If you're using ssh to connect to Stretch and run this script, you won't be able to visualize its cameras feed (though they will still be recording). To see the cameras stream, use tethered or untethered setup.

Replay an episode Now try to replay this episode (make sure the robot's initial position is the same):

python lerobot/scripts/control_robot.py \
  --robot.type=stretch \
  --control.type=replay \
  --control.fps=30 \
  --control.repo_id=${HF_USER}/stretch_test \
  --control.episode=0

Follow previous tutorial to train a policy on your data and run inference on your robot. You will need to adapt the code for Stretch.

TODO(rcadene, aliberts): Add already setup environment and policy yaml configuration files

If you need help, please reach out on Discord in the channel #stretch3-mobile-arm.