wangzixiang
/
lerobot
mirror of https://github.com/huggingface/lerobot.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Minor Updates and Cleanup (#13) 

* Pin Torchvision

* Readme Cleanup

* Concurrent Resetting of Arms

* Update Bimanual to Stationary
This commit is contained in:
shantanuparab-tr 2025-03-20 17:07:50 -05:00 committed by Luke Schmitt
parent f0908e15d1
commit 5ed5725c4e
No known key found for this signature in database
GPG Key ID: E9957A727DDEB555
7 changed files with 63 additions and 86 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
examples/12_use_trossen_ai.md
View File

@ -44,36 +44,12 @@ pip uninstall -y opencv-python
conda install -y -c conda-forge "opencv>=4.10.0"
```
## Troubleshooting
If you encounter the following error.
```bash
ImportError: /xxx/xxx/xxx/envs/lerobot/lib/python3.10/site-packages/cv2/python-3.10/../../../.././libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0
```
The below are the 2 known system specific solutions
### System 76 Serval Workstation (serw13) & Dell Precision 7670
```bash
conda install pytorch==2.5.1=cpu_openblas_py310ha613aac_2 -y
conda install torchvision==0.21.0 -y
```
### HP
```bash
pip install torch==2.5.1+cu121 torchvision==0.20.1+cu121 torchaudio==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu121
```
## Teleoperate
By running the following code, you can start your first **SAFE** teleoperation:
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai_bimanual \
--robot.type=trossen_ai_stationary \
--robot.max_relative_target=5 \
--control.type=teleoperate
```
@ -82,7 +58,7 @@ By adding `--robot.max_relative_target=5`, we override the default value for `ma
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai \
--robot.type=trossen_ai_stationary \
--robot.max_relative_target=null \
--control.type=teleoperate
```
@ -106,52 +82,36 @@ echo $HF_USER
Record 2 episodes and upload your dataset to the hub:
Note: We recommend using `--control.num_image_writer_threads_per_camera=8` for best results while recording episodes.
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai_bimanual \
--robot.type=trossen_ai_stationary \
--robot.max_relative_target=null \
--control.type=record \
--control.fps=30 \
--control.single_task="Grasp a lego block and put it in the bin." \
--control.repo_id=${HF_USER}/trossen_ai_bimanual_test \
--control.tags='["tutorial"]' \
--control.warmup_time_s=5 \
--control.episode_time_s=30 \
--control.reset_time_s=30 \
--control.num_episodes=2 \
--control.push_to_hub=true
```
Note: If the camera fps is unstable consider increasing the number of image writers per thread.
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai_bimanual \
--robot.max_relative_target=null \
--control.type=record \
--control.fps=30 \
--control.single_task="Grasp a lego block and put it in the bin." \
--control.repo_id=${HF_USER}/trossen_ai_bimanual_test \
--control.repo_id=${HF_USER}/trossen_ai_stationary_test \
--control.tags='["tutorial"]' \
--control.warmup_time_s=5 \
--control.episode_time_s=30 \
--control.reset_time_s=30 \
--control.num_episodes=2 \
--control.push_to_hub=true \
--control.num_image_writer_threads_per_camera = 8
--control.num_image_writer_threads_per_camera=8
```
## Visualize a dataset
If you uploaded your dataset to the hub with `--control.push_to_hub=true`, you can [visualize your dataset online](https://huggingface.co/spaces/lerobot/visualize_dataset) by copy pasting your repo id given by:
```bash
echo ${HF_USER}/trossen_ai_bimanual_test
echo ${HF_USER}/trossen_ai_stationary_test
```
If you didn't upload with `--control.push_to_hub=false`, you can also visualize it locally with:
```bash
python lerobot/scripts/visualize_dataset_html.py \
--repo-id ${HF_USER}/trossen_ai_bimanual_test
--repo-id ${HF_USER}/trossen_ai_stationary_test
```
## Replay an episode
@ -160,11 +120,11 @@ python lerobot/scripts/visualize_dataset_html.py \
Now try to replay the first episode on your robot:
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai_bimanual \
--robot.type=trossen_ai_stationary \
--robot.max_relative_target=null \
--control.type=replay \
--control.fps=30 \
--control.repo_id=${HF_USER}/trossen_ai_bimanual_test \
--control.repo_id=${HF_USER}/trossen_ai_stationary_test \
--control.episode=0
```
@ -173,47 +133,47 @@ python lerobot/scripts/control_robot.py \
To train a policy to control your robot, use the [`python lerobot/scripts/train.py`](../lerobot/scripts/train.py) script. A few arguments are required. Here is an example command:
```bash
python lerobot/scripts/train.py \
--dataset.repo_id=${HF_USER}/trossen_ai_bimanual_test \
--dataset.repo_id=${HF_USER}/trossen_ai_stationary_test \
--policy.type=act \
--output_dir=outputs/train/act_trossen_ai_bimanual_test \
--job_name=act_trossen_ai_bimanual_test \
--output_dir=outputs/train/act_trossen_ai_stationary_test \
--job_name=act_trossen_ai_stationary_test \
--device=cuda \
--wandb.enable=true
```
Let's explain it:
1. We provided the dataset as argument with `--dataset.repo_id=${HF_USER}/trossen_ai_bimanual_test`.
1. We provided the dataset as argument with `--dataset.repo_id=${HF_USER}/trossen_ai_stationary_test`.
2. We provided the policy with `policy.type=act`. This loads configurations from [`configuration_act.py`](../lerobot/common/policies/act/configuration_act.py). Importantly, this policy will automatically adapt to the number of motor sates, motor actions and cameras of your robot (e.g. `laptop` and `phone`) which have been saved in your dataset.
4. We provided `device=cuda` since we are training on a Nvidia GPU, but you could use `device=mps` to train on Apple silicon.
5. We provided `wandb.enable=true` to use [Weights and Biases](https://docs.wandb.ai/quickstart) for visualizing training plots. This is optional but if you use it, make sure you are logged in by running `wandb login`.
For more information on the `train` script see the previous tutorial: [`examples/4_train_policy_with_script.md`](../examples/4_train_policy_with_script.md)
Training should take several hours. You will find checkpoints in `outputs/train/act_trossen_ai_bimanual_test/checkpoints`.
Training should take several hours. You will find checkpoints in `outputs/train/act_trossen_ai_stationary_test/checkpoints`.
## Evaluate your policy
You can use the `record` function from [`lerobot/scripts/control_robot.py`](../lerobot/scripts/control_robot.py) but with a policy checkpoint as input. For instance, run this command to record 10 evaluation episodes:
```bash
python lerobot/scripts/control_robot.py \
--robot.type=trossen_ai_bimanual \
--robot.type=trossen_ai_stationary \
--control.type=record \
--control.fps=30 \
--control.single_task="Grasp a lego block and put it in the bin." \
--control.repo_id=${HF_USER}/eval_act_trossen_ai_bimanual_test \
--control.repo_id=${HF_USER}/eval_act_trossen_ai_stationary_test \
--control.tags='["tutorial"]' \
--control.warmup_time_s=5 \
--control.episode_time_s=30 \
--control.reset_time_s=30 \
--control.num_episodes=10 \
--control.push_to_hub=true \
--control.policy.path=outputs/train/act_trossen_ai_bimanual_test/checkpoints/last/pretrained_model \
--control.policy.path=outputs/train/act_trossen_ai_stationary_test/checkpoints/last/pretrained_model \
--control.num_image_writer_processes=1
```
As you can see, it's almost the same command as previously used to record your training dataset. Two things changed:
1. There is an additional `--control.policy.path` argument which indicates the path to your policy checkpoint with (e.g. `outputs/train/eval_act_trossen_ai_bimanual_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `${HF_USER}/act_trossen_ai_bimanual_test`).
2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `${HF_USER}/eval_act_trossen_ai_bimanual_test`).
1. There is an additional `--control.policy.path` argument which indicates the path to your policy checkpoint with (e.g. `outputs/train/eval_act_trossen_ai_stationary_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `${HF_USER}/act_trossen_ai_stationary_test`).
2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `${HF_USER}/eval_act_trossen_ai_stationary_test`).
3. We use `--control.num_image_writer_processes=1` instead of the default value (`0`). On our computer, using a dedicated process to write images from the 4 cameras on disk allows to reach constent 30 fps during inference. Feel free to explore different values for `--control.num_image_writer_processes`.
## More

2
lerobot/common/datasets/video_utils.py
View File

@ -247,7 +247,7 @@ def encode_video_frames(
imgs_dir: Path | str,
video_path: Path | str,
fps: int,
vcodec: str = "libx264",
vcodec: str = "libsvtav1",
pix_fmt: str = "yuv420p",
g: int | None = 2,
crf: int | None = 30,

17
lerobot/common/robot_devices/control_utils.py
View File

@ -288,13 +288,16 @@ def reset_environment(robot, events, reset_time_s, fps):
if has_method(robot, "teleop_safety_stop"):
robot.teleop_safety_stop()
control_loop(
robot=robot,
control_time_s=reset_time_s,
events=events,
fps=fps,
teleoperate=True,
)
if robot.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]:
time.sleep(reset_time_s)
else:
control_loop(
robot=robot,
control_time_s=reset_time_s,
events=events,
fps=fps,
teleoperate=True,
)
def stop_recording(robot, listener, display_cameras):

8
lerobot/common/robot_devices/motors/trossen_arm_driver.py
View File

@ -132,7 +132,7 @@ class TrossenArmDriver:
# Move the arms to the home pose
self.driver.set_all_modes(trossen.Mode.position)
self.driver.set_all_positions(self.home_pose, 2.0, True)
self.driver.set_all_positions(self.home_pose, 2.0, False)
# Allow to read and write
self.is_connected = True
@ -244,7 +244,7 @@ class TrossenArmDriver:
self.driver.set_all_external_efforts([0.0] * 7, 0.0, True)
elif data_name == "Reset":
self.driver.set_all_modes(trossen.Mode.position)
self.driver.set_all_positions(self.home_pose, 2.0, True)
self.driver.set_all_positions(self.home_pose, 2.0, False)
else:
print(f"Data name: {data_name} value: {values} is not supported for writing.")
@ -256,8 +256,8 @@ class TrossenArmDriver:
f"Trossen Arm Driver ({self.port}) is not connected. Try running `motors_bus.connect()` first."
)
self.driver.set_all_modes(trossen.Mode.position)
self.driver.set_all_positions(self.home_pose, 2.0, True)
self.driver.set_all_positions(self.sleep_pose, 2.0, True)
self.driver.set_all_positions(self.home_pose, 2.0, False)
self.driver.set_all_positions(self.sleep_pose, 2.0, False)
self.is_connected = False

12
lerobot/common/robot_devices/robots/configs.py
View File

@ -614,9 +614,9 @@ class LeKiwiRobotConfig(RobotConfig):
mock: bool = False
@RobotConfig.register_subclass("trossen_ai_bimanual")
@RobotConfig.register_subclass("trossen_ai_stationary")
@dataclass
class TrossenAIBimanualRobotConfig(ManipulatorRobotConfig):
class TrossenAIStationaryRobotConfig(ManipulatorRobotConfig):
# /!\ FOR SAFETY, READ THIS /!\
# `max_relative_target` limits the magnitude of the relative positional target vector for safety purposes.
@ -664,7 +664,13 @@ class TrossenAIBimanualRobotConfig(ManipulatorRobotConfig):
cameras: dict[str, CameraConfig] = field(
default_factory=lambda: {
"cam_high": IntelRealSenseCameraConfig(
serial_number=130322270184,
serial_number=218622270304,
fps=30,
width=640,
height=480,
),
"cam_low": IntelRealSenseCameraConfig(
serial_number=130322272628,
fps=30,
width=640,
height=480,

22
lerobot/common/robot_devices/robots/manipulator.py
View File

@ -160,7 +160,7 @@ class ManipulatorRobot:
):
self.config = config
self.robot_type = self.config.type
if not self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]:
if not self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]:
self.calibration_dir = Path(self.config.calibration_dir)
self.leader_arms = make_motors_buses_from_configs(self.config.leader_arms)
self.follower_arms = make_motors_buses_from_configs(self.config.follower_arms)
@ -224,13 +224,17 @@ class ManipulatorRobot:
return available_arms
def teleop_safety_stop(self):
if self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]:
for arms in self.follower_arms:
self.follower_arms[arms].write("Reset", 1)
self.follower_arms[arms].write("Torque_Enable", 1)
if self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]:
for arms in self.leader_arms:
self.leader_arms[arms].write("Reset", 1)
for arms in self.follower_arms:
self.follower_arms[arms].write("Reset", 1)
time.sleep(2)
for arms in self.leader_arms:
self.leader_arms[arms].write("Torque_Enable", 0)
for arms in self.follower_arms:
self.follower_arms[arms].write("Torque_Enable", 1)
def connect(self):
if self.is_connected:
@ -250,8 +254,10 @@ class ManipulatorRobot:
for name in self.leader_arms:
print(f"Connecting {name} leader arm.")
self.leader_arms[name].connect()
time.sleep(2)
if self.robot_type in ["koch", "koch_bimanual", "aloha", "trossen_ai_bimanual", "trossen_ai_solo"]:
if self.robot_type in ["koch", "koch_bimanual", "aloha", "trossen_ai_stationary", "trossen_ai_solo"]:
from lerobot.common.robot_devices.motors.dynamixel import TorqueMode
elif self.robot_type in ["so100", "moss", "lekiwi"]:
from lerobot.common.robot_devices.motors.feetech import TorqueMode
@ -263,7 +269,7 @@ class ManipulatorRobot:
for name in self.leader_arms:
self.leader_arms[name].write("Torque_Enable", TorqueMode.DISABLED.value)
if not self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]:
if not self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]:
print("Checking if calibration is needed.")
self.activate_calibration()
@ -628,6 +634,8 @@ class ManipulatorRobot:
for name in self.leader_arms:
self.leader_arms[name].disconnect()
time.sleep(2)
for name in self.cameras:
self.cameras[name].disconnect()

6
lerobot/common/robot_devices/robots/utils.py
View File

@ -24,7 +24,7 @@ from lerobot.common.robot_devices.robots.configs import (
RobotConfig,
So100RobotConfig,
StretchRobotConfig,
TrossenAIBimanualRobotConfig,
TrossenAIStationaryRobotConfig,
TrossenAISoloRobotConfig,
)
@ -64,8 +64,8 @@ def make_robot_config(robot_type: str, **kwargs) -> RobotConfig:
return StretchRobotConfig(**kwargs)
elif robot_type == "lekiwi":
return LeKiwiRobotConfig(**kwargs)
elif robot_type == "trossen_ai_bimanual":
return TrossenAIBimanualRobotConfig(**kwargs)
elif robot_type == "trossen_ai_stationary":
return TrossenAIStationaryRobotConfig(**kwargs)
elif robot_type == "trossen_ai_solo":
return TrossenAISoloRobotConfig(**kwargs)
else: