LerobotDataset pushable to HF from any folder

[Raziel90]

Path to card_template.md passed in relation to the current location of the lerobot package.

In the previous version the path was served statically and relative to the current folder: ./lerobot/common/datasets/card_template.md This creates problems when launching LerobotDataset.push_to_hub() outside of the package folder.

In the current version the path is provided in relation to the current path of the package: importlib.resources.path("lerobot.common.datasets", "card_template.md") This allows to execute the method create_lerobot_dataset_card from any folder. As long as Lerobot is installed.

On branch fix--dataset_push_to_hub
Changes to be committed:
modified: lerobot/common/datasets/utils.py

What this does

Explain what this PR does. Feel free to tag your PR with the appropriate label(s).

| Fixes #561 | (🐛 Bug) |

How it was tested

• Executed the dataset creation from both inside and outside the lerobot folder.
  It worked in both cases: https://huggingface.co/datasets/ccop/aloha_stationary_replay_test_v3.
  The script used for the creation of the dataset will be object of another PR once refined. It converts a single episode aloha_hd5 dataset into a Lerobot Dataset V2. A draft snippet will be made available below.
• tests executed with pytest without problems (Although, I notice that there is no current test for push_to_hub in the suite).

How to checkout & try? (for the reviewer)

Provide a simple way for the reviewer to try out your changes.

import h5py
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
from pathlib import Path
import cv2
import torch
from importlib.resources import path
data_path =  Path('/home/ccop/code/aloha_data')

def get_features(hdf5_file):
    topics = []
    features = {}
    hdf5_file.visititems(lambda name, obj : topics.append(name) if isinstance(obj, h5py.Dataset) else None)
    for topic in topics:
        # print(topic.replace('/', '.'))
        if 'images' in topic.split('/'):
            features[topic.replace('/', '.')] = {
                'dtype': "image",
                'shape': cv2.imdecode(hdf5_file[topic][0], 1).transpose(2, 0, 1).shape,
                'names': None
            }
        elif 'compress_len'  in topic.split('/'):
            continue
        else:
            features[topic.replace('/', '.')] = {
                'dtype': str(hdf5_file[topic][0].dtype),
                'shape': hdf5_file[topic][0].shape,
                'names': None
            }
            
    return features
if __name__ == '__main__':

    with h5py.File(data_path.absolute() / 'aloha_stationary_replay_test/episode_0.hdf5', 'r') as file:
        # List all groups
        print("Keys: %s" % file.keys())
        features = get_features(file)
        n_frames = file['observations/images/cam_high'][:].shape[0]
        print(n_frames)
        # print(cv2.imdecode(file['observations/images/cam_high'][0],1).shape)

    dataset = LeRobotDataset.create(
            repo_id='ccop/aloha_stationary_replay_test_v3',
            fps=50,
            robot_type="aloha-stationary",
            features=features,
            image_writer_threads=4,
        )
    with h5py.File(data_path.absolute() / 'aloha_stationary_replay_test/episode_0.hdf5', 'r') as file:
        # List all groups
        for frame_idx in range(n_frames):
            frame = {}
            for feature in features:
                if 'images' in feature.split('.'):
                    frame[feature] = torch.from_numpy(
                        cv2.imdecode(file[feature.replace('.', '/')][frame_idx], 1).transpose(2, 0, 1))
                else:    
                    frame[feature] = torch.from_numpy(file[feature.replace('.', '/')][frame_idx])
                # print(feature, frame[feature].shape)

            dataset.add_frame(frame)
    print('save episode!')
    dataset.save_episode(task='move_cube')
    dataset.consolidate()
    dataset.push_to_hub()

[aliberts]

Awesome, thank you @Raziel90!
We indeed didn't focus yet on packaging and releases of our code, this will come after refactoring but this is a welcome fix for people already using LeRobot as a dependency.

Side notes on your conversion script:

• The task argument for save_episode is supposed to be a prompt in natural language describing your task. I'll try to make this appear more clearly in the code/docs.

- dataset.save_episode(task='move_cube')
+ dataset.save_episode(task='Move the cube to this spot.')

• I would suggest using the "video" mode for storing images in your dataset as it would really benefit from it given their size (480x848)

if 'images' in topic.split('/'):
    features[topic.replace('/', '.')] = {
-       'dtype': "image",
+       'dtype': "video",
        'shape': cv2.imdecode(hdf5_file[topic][0], 1).transpose(2, 0, 1).shape,
        'names': None
    }