Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- (huggingface bridge) full parallel support in `from_generator`, with optimization of constant leaf detection (no large data communicated between processes).

### Fixes

- (samples/features) add string support to globals.
- (huggingface bridge) correct split_constant tree derivation, add heuristic for number of shards usage in push_to_dict, robustify infer_hf_features_from_value with respect to numpy arrays of strings, modernize update_dataset_card.

### Removed

## [0.1.10] - 2025-10-29
Expand Down
19 changes: 14 additions & 5 deletions examples/bridges/huggingface_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,16 +153,25 @@ def get_mem():
# Ganarators are used to handle large datasets that do not fit in memory:

# %%
gen_kwargs = {}
gen_kwargs["train"] = {"shards_ids": [[0, 1]]}
gen_kwargs["test"] = {"shards_ids": [[2]]}

generators = {}
for split_name, ids in main_splits.items():
def generator_(ids=ids):
for id in ids:
yield dataset[id]
for split_name in gen_kwargs.keys():

def generator_(shards_ids):
for ids in shards_ids:
if isinstance(ids, int):
ids = [ids]
for id in ids:
yield dataset[id]

generators[split_name] = generator_

hf_datasetdict, flat_cst, key_mappings = (
huggingface_bridge.plaid_generator_to_huggingface_datasetdict(
generators
generators, gen_kwargs
)
)
print(f"{hf_datasetdict = }")
Expand Down
Loading
Loading