🎉 feature(storage) Add multiple storage backends

[casenave]

Checklist

• Typing enforced
• Documentation updated
• Changelog updated
• Tests and Example updates
• Coverage should be 100%

Implemented a facotrization of function needed for all backends and refactored the HF_datasets backend while added an additional zarr backend.
The webdataset backend is not provided yet, since there are some unexpected difficulties ocurring when trying to set one tar per feature per sample: iterating over the dataset will not enable easily to group files per sample - must find an efiicient trick, maybe by storing a dict with key the sample id and value the list of tar files corresponding to the sample ? Will this be efficient for iteration and plaid sample reconstruction ?

Implemented:

• advanced parallel and OOM writers for hf_datasets and zarr (based on generator iterating over samples)
• three reading functions common to hf_datasets and zarr backends (I downloaded a PLAID dataset. How do I read it from disk? #277 and [STORAGE] implement a download_dataset for hf_dataset storage mode #278)
  • load_datasetdict
  • download_datasetdict
  • init_streamed_datasetdict

Tests, examples and tutorials to come

🔗 Related issues

Closes #270, #277

[codecov]

Codecov Report

❌ Patch coverage is 38.63636% with 216 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/plaid/storage/reader.py	0.00%	144 Missing ⚠️
src/plaid/storage/writer.py	0.00%	64 Missing ⚠️
src/plaid/problem_definition.py	45.45%	6 Missing ⚠️
src/plaid/storage/__init__.py	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!