gR V d Z ddlZddlmZmZ ed Zdee dedeee fdZy) z9Contains a utility to iterate by chunks over an iterator. N)IterableTypeVarTiterable
chunk_sizereturnc # K t |t r|dk r t d t | } t | }t
j |ft
j ||dz
=# t
$ r Y yw xY ww)a Iterates over an iterator chunk by chunk.
Taken from https://stackoverflow.com/a/8998040.
See also https://github.com/huggingface/huggingface_hub/pull/920#discussion_r938793088.
Args:
iterable (`Iterable`):
The iterable on which we want to iterate.
chunk_size (`int`):
Size of the chunks. Must be a strictly positive integer (e.g. >0).
Example:
```python
>>> from huggingface_hub.utils import chunk_iterable
>>> for items in chunk_iterable(range(17), chunk_size=8):
... print(items)
# [0, 1, 2, 3, 4, 5, 6, 7]
# [8, 9, 10, 11, 12, 13, 14, 15]
# [16] # smaller last chunk
```
Raises:
[`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
If `chunk_size` <= 0.
The last chunk can be smaller than `chunk_size`.
r z6`chunk_size` must be a strictly positive integer (>0).N )
isinstanceint
ValueErroriternext
StopIteration itertoolschainislice)r r iterator next_items W/var/www/openai/venv/lib/python3.12/site-packages/huggingface_hub/utils/_chunk_utils.pychunk_iterabler s~ @ j# &*/QRRH~H
XI ooylI,<,r sH @ $ CL)XXa[ )Xc )XhxPQ{>S )Xr