[FEA] Add support for huggingface datasets

[ayushdg]

Is your feature request related to a problem? Please describe.
NeMo curator supports document datasets as dataframes today and includes some helpers to read from json/parquet files.

Describe the solution you'd like
Support to read in/ work with hugging face datasets.

Describe alternatives you've considered
Dumping from huggingface datasets to json/parquet before reading with Curator

Additional context
N/A

[ayushdg]

Quick update: There exists a HfFileSystem that is fsspec compatible. https://huggingface.co/docs/huggingface_hub/main/en/guides/hf_file_system

It should be possible to use this to read in dataframes as dask dataframes with the hf:// protocol prefix.

[sarahyurick]

Would love to keep this issue open. Maybe we can find a way to fit it into our download_and_extract framework.

[sarahyurick]

This should work OOTB now. Here was my test:

from nemo_curator.core.client import RayClient
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import ParquetReader
from nemo_curator.stages.text.io.writer import ParquetWriter

client = RayClient(num_cpus=32)
client.start()

# 171 GB of Parquet files from https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle
hf_path = "hf://datasets/karpathy/fineweb-edu-100b-shuffle/**/*.parquet"

pipeline = Pipeline("parquet_hf_reader")

reader = ParquetReader(hf_path, files_per_partition=3)
pipeline.add_stage(reader)

writer = ParquetWriter("/path/to/results")
pipeline.add_stage(writer)

pipeline.run()

client.stop()