diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..c0d45362 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,236 @@ + + +# Generating the documentation + +To generate the documentation, you first have to build it. Several packages are necessary to build the doc, +you can install them with the following command, at the root of the code repository: + +```bash +pip install -e ".[docs]" +``` + +Then you need to install our special tool that builds the documentation: + +```bash +pip install git+https://github.com/huggingface/doc-builder +``` + +--- +**NOTE** + +You only need to generate the documentation to inspect it locally (if you're planning changes and want to +check how they look before committing for instance). You don't have to `git commit` the built documentation. + +--- + +## Building the documentation + +Once you have setup the `doc-builder` and additional packages, you can generate the documentation by +typing the following command: + +```bash +doc-builder build lerobot docs/source/ --build_dir ~/tmp/test-build +``` + +You can adapt the `--build_dir` to set any temporary folder that you prefer. This command will create it and generate +the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite +Markdown editor. + +## Previewing the documentation + +To preview the docs, first install the `watchdog` module with: + +```bash +pip install watchdog +``` + +Then run the following command: + +```bash +doc-builder preview {package_name} {path_to_docs} +``` + +For example: + +```bash +doc-builder preview lerobot docs/source/ +``` + +The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives. + +--- +**NOTE** + +The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again). + +--- + +## Adding a new element to the navigation bar + +Accepted files are Markdown (.md). + +Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting +the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/lerobot/blob/main/docs/source/_toctree.yml) file. + +## Renaming section headers and moving sections + +It helps to keep the old links working when renaming the section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums, and Social media and it'd make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information. + +Therefore, we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor. + +So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file: + +``` +Sections that were moved: + +[ Section A ] +``` +and of course, if you moved it to another file, then: + +``` +Sections that were moved: + +[ Section A ] +``` + +Use the relative style to link to the new file so that the versioned docs continue to work. + +For an example of a rich moved sections set please see the very end of [the transformers Trainer doc](https://github.com/huggingface/transformers/blob/main/docs/source/en/main_classes/trainer.md). + +## Writing Documentation - Specification + +The `huggingface/lerobot` documentation follows the +[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings, +although we can write them directly in Markdown. + +### Adding a new tutorial + +Adding a new tutorial or section is done in two steps: + +- Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md). +- Link that file in `./source/_toctree.yml` on the correct toc-tree. + +Make sure to put your new file under the proper section. If you have a doubt, feel free to ask in a Github Issue or PR. + +### Writing source documentation + +Values that should be put in `code` should either be surrounded by backticks: \`like so\`. Note that argument names +and objects like True, None or any strings should usually be put in `code`. + +When mentioning a class, function or method, it is recommended to use our syntax for internal links so that our tool +adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`function\`\]. This requires the class or +function to be in the main package. + +If you want to create a link to some internal class or function, you need to +provide its path. For instance: \[\`table.InMemoryTable\`\]. This will be converted into a link with +`table.InMemoryTable` in the description. To get rid of the path and only keep the name of the object you are +linking to in the description, add a ~: \[\`~table.InMemoryTable\`\] will generate a link with `InMemoryTable` in the description. + +The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\]. + +#### Defining arguments in a method + +Arguments should be defined with the `Args:` (or `Arguments:` or `Parameters:`) prefix, followed by a line return and +an indentation. The argument should be followed by its type, with its shape if it is a tensor, a colon and its +description: + +``` + Args: + n_layers (`int`): The number of layers of the model. +``` + +If the description is too long to fit in one line, another indentation is necessary before writing the description +after the argument. + +Here's an example showcasing everything so far: + +``` + Args: + input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): + Indices of input sequence tokens in the vocabulary. + + Indices can be obtained using [`AlbertTokenizer`]. See [`~PreTrainedTokenizer.encode`] and + [`~PreTrainedTokenizer.__call__`] for details. + + [What are input IDs?](../glossary#input-ids) +``` + +For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the +following signature: + +``` +def my_function(x: str = None, a: float = 1): +``` + +then its documentation should look like this: + +``` + Args: + x (`str`, *optional*): + This argument controls ... + a (`float`, *optional*, defaults to 1): + This argument is used to ... +``` + +Note that we always omit the "defaults to \`None\`" when None is the default for any argument. Also note that even +if the first line describing your argument type and its default gets long, you can't break it into several lines. You can +however write as many lines as you want in the indented description (see the example above with `input_ids`). + +#### Writing a multi-line code block + +Multi-line code blocks can be useful for displaying examples. They are done between two lines of three backticks as usual in Markdown: + + +```` +``` +# first line of code +# second line +# etc +``` +```` + +#### Writing a return block + +The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation. +The first line should be the type of the return, followed by a line return. No need to indent further for the elements +building the return. + +Here's an example of a single value return: + +``` + Returns: + `List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token. +``` + +Here's an example of tuple return, comprising several objects: + +``` + Returns: + `tuple(torch.FloatTensor)` comprising various elements depending on the configuration ([`BertConfig`]) and inputs: + - ** loss** (*optional*, returned when `masked_lm_labels` is provided) `torch.FloatTensor` of shape `(1,)` -- + Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. + - **prediction_scores** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- + Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). +``` + +#### Adding an image + +Due to the rapidly growing repository, it is important to make sure that no files that would significantly weigh down the repository are added. This includes images, videos, and other non-text files. We prefer to leverage a hf.co hosted `dataset` like +the ones hosted on [`hf-internal-testing`](https://huggingface.co/hf-internal-testing) in which to place these files and reference +them by URL. We recommend putting them in the following dataset: [huggingface/documentation-images](https://huggingface.co/datasets/huggingface/documentation-images). +If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images +to this dataset. diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml new file mode 100644 index 00000000..35c8d0a9 --- /dev/null +++ b/docs/source/_toctree.yml @@ -0,0 +1,12 @@ +- sections: + - local: index + title: LeRobot + - local: installation + title: Installation + title: Get started +- sections: + - local: assemble_so101 + title: Assemble SO-101 + - local: use_so101 + title: Use SO-101 with LeRobot + title: "Tutorials" diff --git a/docs/source/assemble_so101.mdx b/docs/source/assemble_so101.mdx new file mode 100644 index 00000000..5f92da67 --- /dev/null +++ b/docs/source/assemble_so101.mdx @@ -0,0 +1,373 @@ +# Assemble SO101 + +TODO(pepijn): Create element interactive that shows steps + assembly video and people can click next step which loads next video. Progress is shown in top with progress bar? + +## Source the parts + +Follow this [README](https://github.com/TheRobotStudio/SO-ARM100). It contains the bill of materials, with a link to source the parts, as well as the instructions to 3D print the parts, +and advice if it's your first time printing or if you don't own a 3D printer. + +Before assembling, you will first need to configure your motors. To this end, we provide a nice script, so let's first install LeRobot. After configuration, we will also guide you through assembly. + +## Install LeRobot + +Follow the [Installation Guide](./installation) + +## Configure the motors + +Designate one bus servo adapter and 6 motors for your leader arm, and similarly the other bus servo adapter and 6 motors for the follower arm. It's convenient to label them and write on each motor if it's for the follower `F` or for the leader `L` and it's ID from 1 to 6 (F1...F6 and L1...L6). + +### Find the USB ports associated to each arm + +To find the port for each bus servo adapter, run the utility script: +```bash +python lerobot/scripts/find_motors_bus_port.py +``` +#### Example outputs + +Example output when identifying the leader arm's port (e.g., `/dev/tty.usbmodem575E0031751` on Mac, or possibly `/dev/ttyACM0` on Linux): +``` +Finding all available ports for the MotorBus. +['/dev/tty.usbmodem575E0032081', '/dev/tty.usbmodem575E0031751'] +Remove the usb cable from your MotorsBus and press Enter when done. + +[...Disconnect leader arm and press Enter...] + +The port of this MotorsBus is /dev/tty.usbmodem575E0031751 +Reconnect the usb cable. +``` +Example output when identifying the follower arm's port (e.g., `/dev/tty.usbmodem575E0032081`, or possibly `/dev/ttyACM1` on Linux): +``` +Finding all available ports for the MotorBus. +['/dev/tty.usbmodem575E0032081', '/dev/tty.usbmodem575E0031751'] +Remove the usb cable from your MotorsBus and press Enter when done. + +[...Disconnect follower arm and press Enter...] + +The port of this MotorsBus is /dev/tty.usbmodem575E0032081 +Reconnect the usb cable. +``` + +#### Troubleshooting +On Linux, you might need to give access to the USB ports by running: +```bash +sudo chmod 666 /dev/ttyACM0 +sudo chmod 666 /dev/ttyACM1 +``` + +#### Update config file + +IMPORTANTLY: Now that you have your ports, update the **port** default values of [`SO100RobotConfig`](../lerobot/common/robot_devices/robots/configs.py). You will find something like: +```python +@RobotConfig.register_subclass("so100") +@dataclass +class So100RobotConfig(ManipulatorRobotConfig): + calibration_dir: str = ".cache/calibration/so100" + # `max_relative_target` limits the magnitude of the relative positional target vector for safety purposes. + # Set this to a positive scalar to have the same value for all motors, or a list that is the same length as + # the number of motors in your follower arms. + max_relative_target: int | None = None + + leader_arms: dict[str, MotorsBusConfig] = field( + default_factory=lambda: { + "main": FeetechMotorsBusConfig( + port="/dev/tty.usbmodem58760431091", <-- UPDATE HERE + motors={ + # name: (index, model) + "shoulder_pan": [1, "sts3215"], + "shoulder_lift": [2, "sts3215"], + "elbow_flex": [3, "sts3215"], + "wrist_flex": [4, "sts3215"], + "wrist_roll": [5, "sts3215"], + "gripper": [6, "sts3215"], + }, + ), + } + ) + + follower_arms: dict[str, MotorsBusConfig] = field( + default_factory=lambda: { + "main": FeetechMotorsBusConfig( + port="/dev/tty.usbmodem585A0076891", <-- UPDATE HERE + motors={ + # name: (index, model) + "shoulder_pan": [1, "sts3215"], + "shoulder_lift": [2, "sts3215"], + "elbow_flex": [3, "sts3215"], + "wrist_flex": [4, "sts3215"], + "wrist_roll": [5, "sts3215"], + "gripper": [6, "sts3215"], + }, + ), + } + ) +``` + +### Set IDs for all 12 motors + +TODO(pepijn): add video instruction for setting motor ID + +Plug your first motor F1 and run this script to set its ID to 1. It will also set its present position to 2048, so expect your motor to rotate. Replace the text after --port to the corresponding follower control board port and run this command in cmd: +```bash +python lerobot/scripts/configure_motor.py \ + --port /dev/tty.usbmodem58760432961 \ + --brand feetech \ + --model sts3215 \ + --baudrate 1000000 \ + --ID 1 +``` + +Then unplug your motor and plug the second motor and set its ID to 2. +```bash +python lerobot/scripts/configure_motor.py \ + --port /dev/tty.usbmodem58760432961 \ + --brand feetech \ + --model sts3215 \ + --baudrate 1000000 \ + --ID 2 +``` + +Redo the process for all your motors until ID 6. Do the same for the 6 motors of the leader arm. + +## Step-by-Step Assembly Instructions + +### Clean Parts +Remove all support material from the 3D-printed parts. + +### First Motor + +**Step 2: Insert Wires** +- Insert two wires into the first motor. + + + +**Step 3: Install in Base** +- Place the first motor into the base. + + + +**Step 4: Secure Motor** +- Fasten the motor with 4 screws. Two from the bottom and two from top. + +**Step 5: Attach Motor Holder** +- Slide over the first motor holder and fasten it using two screws (one on each side). + + + +**Step 6: Attach Motor Horns** +- Install both motor horns, securing the top horn with a screw. Try not to move the motor position when attaching the motor horn, especially for the leader arms, where we removed the gears. + + +
+ Video adding motor horn + +
+ +**Step 7: Attach Shoulder Part** +- Route one wire to the back of the robot and the other to the left or in photo towards you (see photo). +- Attach the shoulder part. + + + +**Step 8: Secure Shoulder** +- Tighten the shoulder part with 4 screws on top and 4 on the bottom +*(access bottom holes by turning the shoulder).* + +--- + +### Second Motor Assembly + +**Step 9: Install Motor 2** +- Slide the second motor in from the top and link the wire from motor 1 to motor 2. + + + +**Step 10: Attach Shoulder Holder** +- Add the shoulder motor holder. +- Ensure the wire from motor 1 to motor 2 goes behind the holder while the other wire is routed upward (see photo). +- This part can be tight to assemble, you can use a workbench like the image or a similar setup to push the part around the motor. + +
+ + + +
+ +**Step 11: Secure Motor 2** +- Fasten the second motor with 4 screws. + +**Step 12: Attach Motor Horn** +- Attach both motor horns to motor 2, again use the horn screw. + +**Step 13: Attach Base** +- Install the base attachment using 2 screws. + + + +**Step 14: Attach Upper Arm** +- Attach the upper arm with 4 screws on each side. + + + +--- + +### Third Motor Assembly + +**Step 15: Install Motor 3** +- Route the motor cable from motor 2 through the cable holder to motor 3, then secure motor 3 with 4 screws. + +**Step 16: Attach Motor Horn** +- Attach both motor horns to motor 3 and secure one again with a horn screw. + + + +**Step 17: Attach Forearm** +- Connect the forearm to motor 3 using 4 screws on each side. + + + +--- + +### Fourth Motor Assembly + +**Step 18: Install Motor 4** +- Slide in motor 4, attach the cable from motor 3, and secure the cable in its holder with a screw. + +
+ + +
+ +**Step 19: Attach Motor Holder 4** +- Install the fourth motor holder (a tight fit). Ensure one wire is routed upward and the wire from motor 3 is routed downward (see photo). + + + +**Step 20: Secure Motor 4 & Attach Horn** +- Fasten motor 4 with 4 screws and attach its motor horns, use for one a horn screw. + + + +--- + +### Wrist Assembly + +**Step 21: Install Motor 5** +- Insert motor 5 into the wrist holder and secure it with 2 front screws. + + + +**Step 22: Attach Wrist** +- Connect the wire from motor 4 to motor 5. And already insert the other wire for the gripper. +- Secure the wrist to motor 4 using 4 screws on both sides. + + + +**Step 23: Attach Wrist Horn** +- Install only one motor horn on the wrist motor and secure it with a horn screw. + + + +--- + +### Follower Configuration + +**Step 24: Attach Gripper** +- Attach the gripper to motor 5. + + + +**Step 25: Install Gripper Motor** +- Insert the gripper motor, connect the motor wire from motor 5 to motor 6, and secure it with 3 screws on each side. + + + +**Step 26: Attach Gripper Horn & Claw** +- Attach the motor horns and again use a horn screw. +- Install the gripper claw and secure it with 4 screws on both sides. + + + +**Step 27: Mount Controller** +- Attach the motor controller on the back. + +
+ + +
+ +*Assembly complete – proceed to Leader arm assembly.* + +--- + +### Leader Configuration + +For the leader configuration, perform **Steps 1–23**. Make sure that you removed the motor gears from the motors. + +**Step 24: Attach Leader Holder** +- Mount the leader holder onto the wrist and secure it with a screw. + + + +**Step 25: Attach Handle** +- Attach the handle to motor 5 using 4 screws. + + + +**Step 26: Install Gripper Motor** +- Insert the gripper motor, secure it with 3 screws on each side, attach a motor horn using a horn screw, and connect the motor wire. + + + +**Step 27: Attach Trigger** +- Attach the follower trigger with 4 screws. + + + +**Step 28: Mount Controller** +- Attach the motor controller on the back. + +
+ + +
+ + +## Calibrate + +Next, you'll need to calibrate your SO-100 robot to ensure that the leader and follower arms have the same position values when they are in the same physical position. This calibration is essential because it allows a neural network trained on one SO-100 robot to work on another. + +TODO(pepijn): Update with new calibratio method + +#### a. Manual calibration of follower arm + +You will need to move the follower arm to these positions sequentially: + +| 1. Zero position | 2. Rotated position | 3. Rest position | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| SO-100 follower arm zero position | SO-100 follower arm rotated position | SO-100 follower arm rest position | + +Make sure both arms are connected and run this script to launch manual calibration: +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --robot.cameras='{}' \ + --control.type=calibrate \ + --control.arms='["main_follower"]' +``` + +#### b. Manual calibration of leader arm +Follow step 6 of the [assembly video](https://youtu.be/FioA2oeFZ5I?t=724) which illustrates the manual calibration. You will need to move the leader arm to these positions sequentially: + +| 1. Zero position | 2. Rotated position | 3. Rest position | +| ------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| SO-100 leader arm zero position | SO-100 leader arm rotated position | SO-100 leader arm rest position | + +Run this script to launch manual calibration: +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --robot.cameras='{}' \ + --control.type=calibrate \ + --control.arms='["main_leader"]' +``` diff --git a/docs/source/index.mdx b/docs/source/index.mdx new file mode 100644 index 00000000..ae47f1c9 --- /dev/null +++ b/docs/source/index.mdx @@ -0,0 +1,23 @@ +# LeRobot + +TODO(pepijn): Add license everywhere +TODO(pepijn): Make layout everywhere nice +TODO(pepijn): Check all text according to style guide + +
+ + HuggingFace Expert Acceleration Program + +
+ +**State-of-the-art machine learning for real-world robotics** + +🤗 LeRobot aims to provide models, datasets, and tools for real-world robotics in PyTorch. The goal is to lower the barrier for entry to robotics so that everyone can contribute and benefit from sharing datasets and pretrained models. + +🤗 LeRobot contains state-of-the-art approaches that have been shown to transfer to the real-world with a focus on imitation learning and reinforcement learning: github.com/huggingface/lerobot + +🤗 LeRobot already provides a set of pretrained models, datasets with human collected demonstrations, and simulated environments so that everyone can get started. In the coming weeks, the plan is to add more and more support for real-world robotics on the most affordable and capable robots out there. + +🤗 LeRobot hosts pretrained models and datasets on this HuggingFace community page: huggingface.co/lerobot + +Join the LeRobot community on [Discord](https://discord.gg/s3KuuzsPFb) diff --git a/docs/source/installation.mdx b/docs/source/installation.mdx new file mode 100644 index 00000000..920851d1 --- /dev/null +++ b/docs/source/installation.mdx @@ -0,0 +1,47 @@ +# Installation + +TODO(pepijn): Compare installation method description and other descriptions with transformers/datasets docs + +Download our source code: +```bash +git clone https://github.com/huggingface/lerobot.git +cd lerobot +``` + +Create a virtual environment with Python 3.10, e.g. with [`miniconda`](https://docs.anaconda.com/free/miniconda/index.html): +```bash +conda create -y -n lerobot python=3.10 +``` + +Then activate your conda environment, you have to do this each time you open a shell to use lerobot: +```bash +conda activate lerobot +``` + +When using `miniconda`, if you don't have `ffmpeg` in your environment: +```bash +conda install ffmpeg +``` + +Install 🤗 LeRobot: +```bash +pip install --no-binary=av -e . +``` + +> [!NOTE] If you encounter build errors, you may need to install additional dependencies (`cmake`, `build-essential`, and `ffmpeg libs`). On Linux, run: +> `sudo apt-get install cmake build-essential python-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libswresample-dev libavfilter-dev pkg-config`. For other systems, see: [Compiling PyAV](https://pyav.org/docs/develop/overview/installation.html#bring-your-own-ffmpeg) + +For simulations, 🤗 LeRobot comes with gymnasium environments that can be installed as extras: +- [aloha](https://github.com/huggingface/gym-aloha) +- [xarm](https://github.com/huggingface/gym-xarm) +- [pusht](https://github.com/huggingface/gym-pusht) + +For instance, to install 🤗 LeRobot with aloha and pusht, use: +```bash +pip install --no-binary=av -e ".[aloha, pusht]" +``` + +To use [Weights and Biases](https://docs.wandb.ai/quickstart) for experiment tracking, log in with +```bash +wandb login +``` diff --git a/docs/source/use_so101.mdx b/docs/source/use_so101.mdx new file mode 100644 index 00000000..4c878492 --- /dev/null +++ b/docs/source/use_so101.mdx @@ -0,0 +1,139 @@ +# Use SO-101 with LeRobot + +TODO(pepijn) + +## Teleoperate + +**Simple teleop** +Then you are ready to teleoperate your robot! Run this simple script (it won't connect and display the cameras): +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --robot.cameras='{}' \ + --control.type=teleoperate +``` + +#### a. Teleop with displaying cameras +Follow [this guide to setup your cameras](https://github.com/huggingface/lerobot/blob/main/examples/7_get_started_with_real_robot.md#c-add-your-cameras-with-opencvcamera). Then you will be able to display the cameras on your computer while you are teleoperating by running the following code. This is useful to prepare your setup before recording your first dataset. +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --control.type=teleoperate +``` + +## G. Record a dataset + +Once you're familiar with teleoperation, you can record your first dataset with SO-100. + +If you want to use the Hugging Face hub features for uploading your dataset and you haven't previously done it, make sure you've logged in using a write-access token, which can be generated from the [Hugging Face settings](https://huggingface.co/settings/tokens): +```bash +huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential +``` + +Store your Hugging Face repository name in a variable to run these commands: +```bash +HF_USER=$(huggingface-cli whoami | head -n 1) +echo $HF_USER +``` + +Record 2 episodes and upload your dataset to the hub: +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --control.type=record \ + --control.fps=30 \ + --control.single_task="Grasp a lego block and put it in the bin." \ + --control.repo_id=${HF_USER}/so100_test \ + --control.tags='["so100","tutorial"]' \ + --control.warmup_time_s=5 \ + --control.episode_time_s=30 \ + --control.reset_time_s=30 \ + --control.num_episodes=2 \ + --control.push_to_hub=true +``` + +Note: You can resume recording by adding `--control.resume=true`. + +## H. Visualize a dataset + +If you uploaded your dataset to the hub with `--control.push_to_hub=true`, you can [visualize your dataset online](https://huggingface.co/spaces/lerobot/visualize_dataset) by copy pasting your repo id given by: +```bash +echo ${HF_USER}/so100_test +``` + +If you didn't upload with `--control.push_to_hub=false`, you can also visualize it locally with (a window can be opened in the browser `http://127.0.0.1:9090` with the visualization tool): +```bash +python lerobot/scripts/visualize_dataset_html.py \ + --repo-id ${HF_USER}/so100_test \ + --local-files-only 1 +``` + +## I. Replay an episode + +Now try to replay the first episode on your robot: +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --control.type=replay \ + --control.fps=30 \ + --control.repo_id=${HF_USER}/so100_test \ + --control.episode=0 +``` + +## J. Train a policy + +To train a policy to control your robot, use the [`python lerobot/scripts/train.py`](../lerobot/scripts/train.py) script. A few arguments are required. Here is an example command: +```bash +python lerobot/scripts/train.py \ + --dataset.repo_id=${HF_USER}/so100_test \ + --policy.type=act \ + --output_dir=outputs/train/act_so100_test \ + --job_name=act_so100_test \ + --policy.device=cuda \ + --wandb.enable=true +``` + +Let's explain it: +1. We provided the dataset as argument with `--dataset.repo_id=${HF_USER}/so100_test`. +2. We provided the policy with `policy.type=act`. This loads configurations from [`configuration_act.py`](../lerobot/common/policies/act/configuration_act.py). Importantly, this policy will automatically adapt to the number of motor sates, motor actions and cameras of your robot (e.g. `laptop` and `phone`) which have been saved in your dataset. +4. We provided `policy.device=cuda` since we are training on a Nvidia GPU, but you could use `policy.device=mps` to train on Apple silicon. +5. We provided `wandb.enable=true` to use [Weights and Biases](https://docs.wandb.ai/quickstart) for visualizing training plots. This is optional but if you use it, make sure you are logged in by running `wandb login`. + +Training should take several hours. You will find checkpoints in `outputs/train/act_so100_test/checkpoints`. + +To resume training from a checkpoint, below is an example command to resume from `last` checkpoint of the `act_so100_test` policy: +```bash +python lerobot/scripts/train.py \ + --config_path=outputs/train/act_so100_test/checkpoints/last/pretrained_model/train_config.json \ + --resume=true +``` + +## K. Evaluate your policy + +You can use the `record` function from [`lerobot/scripts/control_robot.py`](../lerobot/scripts/control_robot.py) but with a policy checkpoint as input. For instance, run this command to record 10 evaluation episodes: +```bash +python lerobot/scripts/control_robot.py \ + --robot.type=so100 \ + --control.type=record \ + --control.fps=30 \ + --control.single_task="Grasp a lego block and put it in the bin." \ + --control.repo_id=${HF_USER}/eval_act_so100_test \ + --control.tags='["tutorial"]' \ + --control.warmup_time_s=5 \ + --control.episode_time_s=30 \ + --control.reset_time_s=30 \ + --control.num_episodes=10 \ + --control.push_to_hub=true \ + --control.policy.path=outputs/train/act_so100_test/checkpoints/last/pretrained_model +``` + +As you can see, it's almost the same command as previously used to record your training dataset. Two things changed: +1. There is an additional `--control.policy.path` argument which indicates the path to your policy checkpoint with (e.g. `outputs/train/eval_act_so100_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `${HF_USER}/act_so100_test`). +2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `${HF_USER}/eval_act_so100_test`). + +## L. More Information + +Follow this [previous tutorial](https://github.com/huggingface/lerobot/blob/main/examples/7_get_started_with_real_robot.md#4-train-a-policy-on-your-data) for a more in-depth tutorial on controlling real robots with LeRobot. + +> [!TIP] +> If you have any questions or need help, please reach out on [Discord](https://discord.com/invite/s3KuuzsPFb) in the channel [`#so100-arm`](https://discord.com/channels/1216765309076115607/1237741463832363039).