Commit Graph

4 Commits

Author SHA1 Message Date
Remi Cadene 1d9433b411 WIP
Add hf-transfer, num_workers 8

gc collect, rmtree save_to_disk False

Track .safetensors in git lfs

run save_dataset_to_safetensors on all datasets

Update datasets

Update download_raw
2024-05-17 02:11:28 +00:00
Simon Alibert df914aa76c
Update dev docker build (#148) 2024-05-08 17:21:58 +00:00
Simon Alibert fe2b9af64f
Add test-docker-build workflow (#109)
Adds a `test-docker-build.yml` workflow to ensure that docker images can correctly build when their Dockerfile has been modified on PRs.
2024-04-27 09:40:21 +02:00
Simon Alibert b980c5dd9e
CI nightlies cpu/gpu & cleanup (#75) 2024-04-25 14:58:39 +02:00