diff --git a/examples/12_use_trossen_ai.md b/examples/12_use_trossen_ai.md index bd1a8feb..1015a234 100644 --- a/examples/12_use_trossen_ai.md +++ b/examples/12_use_trossen_ai.md @@ -44,36 +44,12 @@ pip uninstall -y opencv-python conda install -y -c conda-forge "opencv>=4.10.0" ``` -## Troubleshooting - -If you encounter the following error. - -```bash -ImportError: /xxx/xxx/xxx/envs/lerobot/lib/python3.10/site-packages/cv2/python-3.10/../../../.././libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0 -``` - -The below are the 2 known system specific solutions - -### System 76 Serval Workstation (serw13) & Dell Precision 7670 - -```bash - conda install pytorch==2.5.1=cpu_openblas_py310ha613aac_2 -y - conda install torchvision==0.21.0 -y -``` - -### HP - -```bash - pip install torch==2.5.1+cu121 torchvision==0.20.1+cu121 torchaudio==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu121 -``` - - ## Teleoperate By running the following code, you can start your first **SAFE** teleoperation: ```bash python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai_bimanual \ + --robot.type=trossen_ai_stationary \ --robot.max_relative_target=5 \ --control.type=teleoperate ``` @@ -82,7 +58,7 @@ By adding `--robot.max_relative_target=5`, we override the default value for `ma ```bash python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai \ + --robot.type=trossen_ai_stationary \ --robot.max_relative_target=null \ --control.type=teleoperate ``` @@ -106,52 +82,36 @@ echo $HF_USER Record 2 episodes and upload your dataset to the hub: +Note: We recommend using `--control.num_image_writer_threads_per_camera=8` for best results while recording episodes. + ```bash python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai_bimanual \ + --robot.type=trossen_ai_stationary \ --robot.max_relative_target=null \ --control.type=record \ --control.fps=30 \ --control.single_task="Grasp a lego block and put it in the bin." \ - --control.repo_id=${HF_USER}/trossen_ai_bimanual_test \ - --control.tags='["tutorial"]' \ - --control.warmup_time_s=5 \ - --control.episode_time_s=30 \ - --control.reset_time_s=30 \ - --control.num_episodes=2 \ - --control.push_to_hub=true -``` - -Note: If the camera fps is unstable consider increasing the number of image writers per thread. - -```bash -python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai_bimanual \ - --robot.max_relative_target=null \ - --control.type=record \ - --control.fps=30 \ - --control.single_task="Grasp a lego block and put it in the bin." \ - --control.repo_id=${HF_USER}/trossen_ai_bimanual_test \ + --control.repo_id=${HF_USER}/trossen_ai_stationary_test \ --control.tags='["tutorial"]' \ --control.warmup_time_s=5 \ --control.episode_time_s=30 \ --control.reset_time_s=30 \ --control.num_episodes=2 \ --control.push_to_hub=true \ - --control.num_image_writer_threads_per_camera = 8 + --control.num_image_writer_threads_per_camera=8 ``` ## Visualize a dataset If you uploaded your dataset to the hub with `--control.push_to_hub=true`, you can [visualize your dataset online](https://huggingface.co/spaces/lerobot/visualize_dataset) by copy pasting your repo id given by: ```bash -echo ${HF_USER}/trossen_ai_bimanual_test +echo ${HF_USER}/trossen_ai_stationary_test ``` If you didn't upload with `--control.push_to_hub=false`, you can also visualize it locally with: ```bash python lerobot/scripts/visualize_dataset_html.py \ - --repo-id ${HF_USER}/trossen_ai_bimanual_test + --repo-id ${HF_USER}/trossen_ai_stationary_test ``` ## Replay an episode @@ -160,11 +120,11 @@ python lerobot/scripts/visualize_dataset_html.py \ Now try to replay the first episode on your robot: ```bash python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai_bimanual \ + --robot.type=trossen_ai_stationary \ --robot.max_relative_target=null \ --control.type=replay \ --control.fps=30 \ - --control.repo_id=${HF_USER}/trossen_ai_bimanual_test \ + --control.repo_id=${HF_USER}/trossen_ai_stationary_test \ --control.episode=0 ``` @@ -173,47 +133,47 @@ python lerobot/scripts/control_robot.py \ To train a policy to control your robot, use the [`python lerobot/scripts/train.py`](../lerobot/scripts/train.py) script. A few arguments are required. Here is an example command: ```bash python lerobot/scripts/train.py \ - --dataset.repo_id=${HF_USER}/trossen_ai_bimanual_test \ + --dataset.repo_id=${HF_USER}/trossen_ai_stationary_test \ --policy.type=act \ - --output_dir=outputs/train/act_trossen_ai_bimanual_test \ - --job_name=act_trossen_ai_bimanual_test \ + --output_dir=outputs/train/act_trossen_ai_stationary_test \ + --job_name=act_trossen_ai_stationary_test \ --device=cuda \ --wandb.enable=true ``` Let's explain it: -1. We provided the dataset as argument with `--dataset.repo_id=${HF_USER}/trossen_ai_bimanual_test`. +1. We provided the dataset as argument with `--dataset.repo_id=${HF_USER}/trossen_ai_stationary_test`. 2. We provided the policy with `policy.type=act`. This loads configurations from [`configuration_act.py`](../lerobot/common/policies/act/configuration_act.py). Importantly, this policy will automatically adapt to the number of motor sates, motor actions and cameras of your robot (e.g. `laptop` and `phone`) which have been saved in your dataset. 4. We provided `device=cuda` since we are training on a Nvidia GPU, but you could use `device=mps` to train on Apple silicon. 5. We provided `wandb.enable=true` to use [Weights and Biases](https://docs.wandb.ai/quickstart) for visualizing training plots. This is optional but if you use it, make sure you are logged in by running `wandb login`. For more information on the `train` script see the previous tutorial: [`examples/4_train_policy_with_script.md`](../examples/4_train_policy_with_script.md) -Training should take several hours. You will find checkpoints in `outputs/train/act_trossen_ai_bimanual_test/checkpoints`. +Training should take several hours. You will find checkpoints in `outputs/train/act_trossen_ai_stationary_test/checkpoints`. ## Evaluate your policy You can use the `record` function from [`lerobot/scripts/control_robot.py`](../lerobot/scripts/control_robot.py) but with a policy checkpoint as input. For instance, run this command to record 10 evaluation episodes: ```bash python lerobot/scripts/control_robot.py \ - --robot.type=trossen_ai_bimanual \ + --robot.type=trossen_ai_stationary \ --control.type=record \ --control.fps=30 \ --control.single_task="Grasp a lego block and put it in the bin." \ - --control.repo_id=${HF_USER}/eval_act_trossen_ai_bimanual_test \ + --control.repo_id=${HF_USER}/eval_act_trossen_ai_stationary_test \ --control.tags='["tutorial"]' \ --control.warmup_time_s=5 \ --control.episode_time_s=30 \ --control.reset_time_s=30 \ --control.num_episodes=10 \ --control.push_to_hub=true \ - --control.policy.path=outputs/train/act_trossen_ai_bimanual_test/checkpoints/last/pretrained_model \ + --control.policy.path=outputs/train/act_trossen_ai_stationary_test/checkpoints/last/pretrained_model \ --control.num_image_writer_processes=1 ``` As you can see, it's almost the same command as previously used to record your training dataset. Two things changed: -1. There is an additional `--control.policy.path` argument which indicates the path to your policy checkpoint with (e.g. `outputs/train/eval_act_trossen_ai_bimanual_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `${HF_USER}/act_trossen_ai_bimanual_test`). -2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `${HF_USER}/eval_act_trossen_ai_bimanual_test`). +1. There is an additional `--control.policy.path` argument which indicates the path to your policy checkpoint with (e.g. `outputs/train/eval_act_trossen_ai_stationary_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `${HF_USER}/act_trossen_ai_stationary_test`). +2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `${HF_USER}/eval_act_trossen_ai_stationary_test`). 3. We use `--control.num_image_writer_processes=1` instead of the default value (`0`). On our computer, using a dedicated process to write images from the 4 cameras on disk allows to reach constent 30 fps during inference. Feel free to explore different values for `--control.num_image_writer_processes`. ## More diff --git a/lerobot/common/datasets/video_utils.py b/lerobot/common/datasets/video_utils.py index 58a32112..c38d570d 100644 --- a/lerobot/common/datasets/video_utils.py +++ b/lerobot/common/datasets/video_utils.py @@ -247,7 +247,7 @@ def encode_video_frames( imgs_dir: Path | str, video_path: Path | str, fps: int, - vcodec: str = "libx264", + vcodec: str = "libsvtav1", pix_fmt: str = "yuv420p", g: int | None = 2, crf: int | None = 30, diff --git a/lerobot/common/robot_devices/control_utils.py b/lerobot/common/robot_devices/control_utils.py index 78a8c6a6..44432254 100644 --- a/lerobot/common/robot_devices/control_utils.py +++ b/lerobot/common/robot_devices/control_utils.py @@ -288,13 +288,16 @@ def reset_environment(robot, events, reset_time_s, fps): if has_method(robot, "teleop_safety_stop"): robot.teleop_safety_stop() - control_loop( - robot=robot, - control_time_s=reset_time_s, - events=events, - fps=fps, - teleoperate=True, - ) + if robot.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]: + time.sleep(reset_time_s) + else: + control_loop( + robot=robot, + control_time_s=reset_time_s, + events=events, + fps=fps, + teleoperate=True, + ) def stop_recording(robot, listener, display_cameras): diff --git a/lerobot/common/robot_devices/motors/trossen_arm_driver.py b/lerobot/common/robot_devices/motors/trossen_arm_driver.py index cec5267b..a5650e9a 100644 --- a/lerobot/common/robot_devices/motors/trossen_arm_driver.py +++ b/lerobot/common/robot_devices/motors/trossen_arm_driver.py @@ -132,7 +132,7 @@ class TrossenArmDriver: # Move the arms to the home pose self.driver.set_all_modes(trossen.Mode.position) - self.driver.set_all_positions(self.home_pose, 2.0, True) + self.driver.set_all_positions(self.home_pose, 2.0, False) # Allow to read and write self.is_connected = True @@ -244,7 +244,7 @@ class TrossenArmDriver: self.driver.set_all_external_efforts([0.0] * 7, 0.0, True) elif data_name == "Reset": self.driver.set_all_modes(trossen.Mode.position) - self.driver.set_all_positions(self.home_pose, 2.0, True) + self.driver.set_all_positions(self.home_pose, 2.0, False) else: print(f"Data name: {data_name} value: {values} is not supported for writing.") @@ -256,8 +256,8 @@ class TrossenArmDriver: f"Trossen Arm Driver ({self.port}) is not connected. Try running `motors_bus.connect()` first." ) self.driver.set_all_modes(trossen.Mode.position) - self.driver.set_all_positions(self.home_pose, 2.0, True) - self.driver.set_all_positions(self.sleep_pose, 2.0, True) + self.driver.set_all_positions(self.home_pose, 2.0, False) + self.driver.set_all_positions(self.sleep_pose, 2.0, False) self.is_connected = False diff --git a/lerobot/common/robot_devices/robots/configs.py b/lerobot/common/robot_devices/robots/configs.py index c055a9c4..b995b745 100644 --- a/lerobot/common/robot_devices/robots/configs.py +++ b/lerobot/common/robot_devices/robots/configs.py @@ -614,9 +614,9 @@ class LeKiwiRobotConfig(RobotConfig): mock: bool = False -@RobotConfig.register_subclass("trossen_ai_bimanual") +@RobotConfig.register_subclass("trossen_ai_stationary") @dataclass -class TrossenAIBimanualRobotConfig(ManipulatorRobotConfig): +class TrossenAIStationaryRobotConfig(ManipulatorRobotConfig): # /!\ FOR SAFETY, READ THIS /!\ # `max_relative_target` limits the magnitude of the relative positional target vector for safety purposes. @@ -664,7 +664,13 @@ class TrossenAIBimanualRobotConfig(ManipulatorRobotConfig): cameras: dict[str, CameraConfig] = field( default_factory=lambda: { "cam_high": IntelRealSenseCameraConfig( - serial_number=130322270184, + serial_number=218622270304, + fps=30, + width=640, + height=480, + ), + "cam_low": IntelRealSenseCameraConfig( + serial_number=130322272628, fps=30, width=640, height=480, diff --git a/lerobot/common/robot_devices/robots/manipulator.py b/lerobot/common/robot_devices/robots/manipulator.py index cf788de8..452abbb1 100644 --- a/lerobot/common/robot_devices/robots/manipulator.py +++ b/lerobot/common/robot_devices/robots/manipulator.py @@ -160,7 +160,7 @@ class ManipulatorRobot: ): self.config = config self.robot_type = self.config.type - if not self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]: + if not self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]: self.calibration_dir = Path(self.config.calibration_dir) self.leader_arms = make_motors_buses_from_configs(self.config.leader_arms) self.follower_arms = make_motors_buses_from_configs(self.config.follower_arms) @@ -224,13 +224,17 @@ class ManipulatorRobot: return available_arms def teleop_safety_stop(self): - if self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]: - for arms in self.follower_arms: - self.follower_arms[arms].write("Reset", 1) - self.follower_arms[arms].write("Torque_Enable", 1) + if self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]: for arms in self.leader_arms: self.leader_arms[arms].write("Reset", 1) + for arms in self.follower_arms: + self.follower_arms[arms].write("Reset", 1) + time.sleep(2) + for arms in self.leader_arms: self.leader_arms[arms].write("Torque_Enable", 0) + for arms in self.follower_arms: + self.follower_arms[arms].write("Torque_Enable", 1) + def connect(self): if self.is_connected: @@ -250,8 +254,10 @@ class ManipulatorRobot: for name in self.leader_arms: print(f"Connecting {name} leader arm.") self.leader_arms[name].connect() + + time.sleep(2) - if self.robot_type in ["koch", "koch_bimanual", "aloha", "trossen_ai_bimanual", "trossen_ai_solo"]: + if self.robot_type in ["koch", "koch_bimanual", "aloha", "trossen_ai_stationary", "trossen_ai_solo"]: from lerobot.common.robot_devices.motors.dynamixel import TorqueMode elif self.robot_type in ["so100", "moss", "lekiwi"]: from lerobot.common.robot_devices.motors.feetech import TorqueMode @@ -263,7 +269,7 @@ class ManipulatorRobot: for name in self.leader_arms: self.leader_arms[name].write("Torque_Enable", TorqueMode.DISABLED.value) - if not self.robot_type in ["trossen_ai_bimanual", "trossen_ai_solo"]: + if not self.robot_type in ["trossen_ai_stationary", "trossen_ai_solo"]: print("Checking if calibration is needed.") self.activate_calibration() @@ -628,6 +634,8 @@ class ManipulatorRobot: for name in self.leader_arms: self.leader_arms[name].disconnect() + + time.sleep(2) for name in self.cameras: self.cameras[name].disconnect() diff --git a/lerobot/common/robot_devices/robots/utils.py b/lerobot/common/robot_devices/robots/utils.py index f5e537e4..76b8b90b 100644 --- a/lerobot/common/robot_devices/robots/utils.py +++ b/lerobot/common/robot_devices/robots/utils.py @@ -24,7 +24,7 @@ from lerobot.common.robot_devices.robots.configs import ( RobotConfig, So100RobotConfig, StretchRobotConfig, - TrossenAIBimanualRobotConfig, + TrossenAIStationaryRobotConfig, TrossenAISoloRobotConfig, ) @@ -64,8 +64,8 @@ def make_robot_config(robot_type: str, **kwargs) -> RobotConfig: return StretchRobotConfig(**kwargs) elif robot_type == "lekiwi": return LeKiwiRobotConfig(**kwargs) - elif robot_type == "trossen_ai_bimanual": - return TrossenAIBimanualRobotConfig(**kwargs) + elif robot_type == "trossen_ai_stationary": + return TrossenAIStationaryRobotConfig(**kwargs) elif robot_type == "trossen_ai_solo": return TrossenAISoloRobotConfig(**kwargs) else: