gQy|dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZd d lm Z d d l!m"Z"d d l#m$Z$d dl%m&Z&m'Z'm(Z(d dl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1d dl)m2Z3erd dl4m5Z5e.jle7Z8edZ9dZ:e GddZ;e GddZde>fdZ?ee=ed#e>d$ee>e>fd%ee>d&eBd'ee>fd(ZCd)eDfd*ZEe1 d6d!ee=d"e>d#e>d$ee>e>fd'e>d%ee>d+eFd,ee>ddfd-ZGe1 d7d.ee<d"e>d#e>d$ee>e>fd'e>d%ee>deee>ee>fed/eHfffd0ZI d8dee@d1eee>ee>fed/eHffd2e>d3ee>d4ee>deee>eff d5ZJy)9z< Type definitions and utilities for the `create_commit` API N) defaultdict)contextmanager) dataclassfield)groupby)Path PurePosixPath) TYPE_CHECKINGAnyBinaryIODictIterableIteratorListLiteralOptionalTupleUnion) thread_map) constants)EntryNotFoundError) hf_hub_url) UploadInfo lfs_uploadpost_lfs_batch_info)FORBIDDEN_FOLDERSchunk_iterable get_sessionhf_raise_for_statusloggingshatqdm_stream_filevalidate_hf_hub_args)tqdm)RepoFilelfsregularic@eZdZUdZeed<dZeee dfed<dZ y)CommitOperationDeletea Data structure holding necessary info to delete a file or a folder from a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` for a file or `"checkpoints/1fec34a/"` for a folder. is_folder (`bool` or `Literal["auto"]`, *optional*) Whether the Delete Operation applies to a folder or not. If "auto", the path type (file or folder) is guessed automatically by looking if path ends with a "/" (folder) or not (file). To explicitly set the path type, you can set `is_folder=True` or `is_folder=False`. path_in_repoauto is_folderct|j|_|jdk(r |jjd|_t |jt st d|jdy)Nr-/zNWrong value for `is_folder`. Must be one of [`True`, `False`, `'auto'`]. Got 'z'.)_validate_path_in_repor,r.endswith isinstancebool ValueErrorselfs P/var/www/openai/venv/lib/python3.12/site-packages/huggingface_hub/_commit_api.py __post_init__z#CommitOperationDelete.__post_init__Gsn243D3DE >>V #!..77r?r@rErr9rAr:r8rCrCRs)&"&L(3-&Fr:rCcVeZdZUdZeed<eeeee fed<e ddZ e ed<e dddZ eeed <e dddZeeed <e dddZeeed <e dddZeed <e dddZeed <ddZeddedee fdZdefdZedeefdZy)CommitOperationAdda Data structure holding necessary info to upload a file to a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` path_or_fileobj (`str`, `Path`, `bytes`, or `BinaryIO`): Either: - a path to a local file (as `str` or `pathlib.Path`) to upload - a buffer of bytes (`bytes`) holding the content of the file to upload - a "file object" (subclass of `io.BufferedIOBase`), typically obtained with `open(path, "rb")`. It must support `seek()` and `tell()` methods. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is not one of `str`, `Path`, `bytes` or `io.BufferedIOBase`. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `str` or `Path` but not a path to an existing file. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `io.BufferedIOBase` but it doesn't support both `seek()` and `tell()`. r,path_or_fileobjF)initrepr upload_infoN)rKrLdefault _upload_mode_should_ignore _remote_oid _is_uploaded _is_committedreturnct|j|_t|jtrt |j|_t|jt rtt jjt jj|j}t jj|sDtd|dt|jtjtfs tdt|jtjrE |jj|jj!dt j"t|jt r%t)j*|j|_yt|jtr%t)j.|j|_yt)j0|j|_y#t$t&f$r}td|d}~wwxYw)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)r1r,r3rJrr?ospathnormpath expanduserisfiler5ioBufferedIOBasebytestellseekSEEK_CUROSErrorAttributeErrorr from_pathrM from_bytes from_fileobj)r7rJexcs r8r9z CommitOperationAdd.__post_init__s243D3DE d**D 1#&t';';#>/2 #3O3DDl!mnnD0023D3De2LM#  d**B,=,= > $$))+$$))!R[[9 d**C 0)33D4H4HID  ,,e 4)44T5I5IJD )66t7K7KLD ^,  d s7AHH>- H99H> with_tqdmc#Kt|jtst|jtrI|r#t |j5}|dddyt |jd5}|dddyt|jt r"tj|jyt|jtjrS|jj}|j|jj|tjyy#1swYyxYw#1swYyxYww)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` Nrb) r3rJr?rr#openr]r[BytesIOr\r^r_SEEK_SET)r7rgfileprev_poss r8as_filezCommitOperationAdd.as_filesH d**C 0Jt?S?SUY4Z%d&:&:;tJ<;$..5J65 ,,e 4**T112 2 ,,b.?.? @++002H&& &  % %h <A<;65s7A E D4E 2E7B=E 4D=9E E E c|j5}tj|jcdddS#1swYyxYw)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)robase64 b64encodereadr7rms r8 b64contentzCommitOperationAdd.b64contents- \\^t##DIIK0^^s #>Ac|jy|jdk(r$|jjjS|j 5}t j |jcdddS#1swYyxYw)aReturn the OID of the local file. This OID is then compared to `self._remote_oid` to check if the file has changed compared to the remote one. If the file did not change, we won't upload it again to prevent empty commits. For LFS files, the OID corresponds to the SHA256 of the file content (used a LFS ref). For regular files, the OID corresponds to the SHA1 of the file content. Note: this is slightly different to git OID computation since the oid of an LFS file is usually the git-SHA1 of the pointer file content (not the actual file content). However, using the SHA256 is enough to detect changes and more convenient client-side. Nr()rOrMsha256hexror"git_hashrsrts r8 _local_oidzCommitOperationAdd._local_oidse    $   % '##**..0 04||DIIK0 s #A>>BrTN)F)r;r<r=r>r?r@rrr]r rrMrrOr UploadModerPr4rQrRrSr9rrrorupropertyrzrAr:r8rIrIps.3eX566#U;K; */Et)TL(:&T&+E4%PNHTNP"'Et!LK#LEuEL$E UFM4F!MF/=/=(82D/=/=b1E11HSM11r:rIr,rTc4|jdr|dd}|dk(s|dk(s|jdrtd|d|jdr|d d}tD]7tfd |j dDs'td d |d |S)Nr0r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./c3(K|] }|k( ywrGrA).0part forbiddens r8 z)_validate_path_in_repo..sE-DTty -DszHInvalid `path_in_repo` in CommitOperation: cannot update files under a 'z/' folder (path: 'z').) startswithr5ranysplit)r,rs @r8r1r1ss##AB' sld2l6M6Me6TG ~UVWXXt$#AB' & E\-?-?-DE EZ[dZef!N#' ' r: operationsctt}|D]}|j}t|tr^||dkDrt j d|d||xxdz cc<t|jD]}|t|xxdz cc<t|ts|tt|dkDs|jrt j d|dt j d|dy)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr,r3rIwarningswarnr parentsr?r+r.)rnb_additions_per_path operationr,parents r8_warn_on_overwriting_operationsr*s -8,<  -- i!3 4$\2Q6 %'"" ", /1 4 /' 5==&c&k2a72> i!6 7$S|)D%EFJ&&MM==INKII MM**6899/ r:)endpoint num_threadsrevision additions repo_typerepo_idheadersrrrc fg}t|dD]}t|D cgc]} | jc} |||d\} } | radj| D cgc]8} d| j dd| j d ij d :c} } t d | || z }|Dcic]'}|jj j|)c}g}|D]N}|j d *tjd |djd>|j|Pt|dk(rtjdydfd }tjr=tjdt|dt|dD] }|| yt|dk(r!tjd||dytjdt|d|dt!||dt|d|tycc} wcc} wcc}w)a2 Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. revision (`str`, *optional*): The git revision to upload to. Raises: [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) If an upload failed for any reason [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the server returns malformed responses [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the LFS batch endpoint returned an HTTP error. ) chunk_sizeN) upload_infosrrrrrtoken z$Encountered error for file with OID oidz: `errormessagez$LFS batch endpoint returned errors: actionszContent of file z/ is already present upstream - skipping upload.rzNo LFS files to upload.c |d}t||y#t$r}tdjd|d}~wwxYw)Nr)rlfs_batch_actionrrzError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr,) batch_actionrrfrr oid2addops r8_wrapped_lfs_uploadz._upload_lfs_files.._wrapped_lfs_uploadsZ i!,u"56I \SZem n i!89O9O8PP]^_eh h is A>Az Uploading z* LFS files to the Hub using `hf_transfer`.zhuggingface_hub.lfs_upload)namerzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workers tqdm_classr{)rrrMjoingetr5rwrxloggerdebugr,appendlenrHF_HUB_ENABLE_HF_TRANSFERhf_tqdmr)rrrrrrr batch_actionschunkopbatch_actions_chunkbatch_errors_chunkerrradd_opfiltered_actionsactionrrs `` @r8_upload_lfs_filesrXs\V!#M c:2E3895R"..593 // ii 21;3775>:J#cggV]_aNbNfNfgpNqMrs1G DWINO O,, +;,HQQyV##**..0&8yQI ::i ( LL"9VE]#;#H#H"IJ77   # #F +  ! ./i** z#&6"7!88bcd.5QRF  'S  ! # 67,Q/0 -.//QR]Q^^s t    3/01<#  q:RsH$ =H) 1,H.preupload_infoc,|jd}t|ts td|D]b}t|trFt|jdt r't|jdt r|ddvrYtd|S)Nfilesz&preupload_info is improperly formattedrW uploadModer'z'preupload_info is improperly formatted:)rr3listr5dictr?)rr file_infos r8_validate_preupload_infors   w 'E eT "ABB y$ '9==0#69==6<<(,>>FG G r: create_prgitignore_contentc ||ntj}i}i} i} t|dD]P} d| D cgc]\} | jt j | j jjd| j jd^c} i} ||| d<tj|d|d|d || ||rd d ind }t|t|j}|jdi|dDcic] }|d |d c}| jdi|dDcic] }|d |d c}| jdi|dDcic]}|d |j!dc}S|D]>}||j|_| |j|_| |j|_@|D]#}|j jdk(sd|_%ycc} wcc}wcc}wcc}w)a Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob or as git LFS blob. Input `additions` are mutated in-place with the upload mode. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. gitignore_content (`str`, *optional*): The content of the `.gitignore` file to know which files should be ignored. The order of priority is to first check if `gitignore_content` is passed, then check if the `.gitignore` file is present in the list of files to commit and finally default to the `.gitignore` file already hosted on the Hub (if any). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. Nrrascii)rWsamplesize gitIgnorez/api/zs/z /preupload/r1)jsonrparamsrWr shouldIgnorerrr)rA)rENDPOINTrr,rqrrrMrdecoderrpostr rrupdaterrOrPrQ)rrrrrrrr upload_modesshould_ignore_infooid_inforrpayloadresprrmadditions r8_fetch_upload_modesrs7N$/xY5G5GH+-L*,)+H 3/     B OO$..r~~/D/DELLWUNN//      (#4GK }!!ji[7);xj I)2K% "  D!1$))+> eN[bLcdLcDtF|T,-??Lcde!!!mTbcjTk$lTkDT&\43G%GTk$lm^^T[E\]E\T4<%8E\]^106 ,X-B-B C"4X5J5J"K'(=(=>    $ $ )$-H !A(e$l]sA!G'G,-G1G6copiesr&c ddlm}m}|||}i} t|dD]>\} } t | } | D cgc]} | j } } t dt| tD]}|j|| ||tz| xs||}|D]}t||r td|jr|| |j| f<8t|||| xs||j }tj!|| }t#||j$| |j| f<| D]2}|j | f| vst'd |j d | xs|d A| Scc} w)a; Fetch information about the files to copy. For LFS files, we only need their metadata (file size and sha256) while for regular files we need to download the raw content from the Hub. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], Union[RepoFile, bytes]]]` Key is the file path and revision of the file to copy. Value is the raw content as bytes (for regular files) or the file information as a RepoFile (for LFS files). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi RepoFolder)rrc|jSrG)rE)rs r8z&_fetch_files_to_copy..Vs2??r:)keyr)rpathsrrz$Copying a folder is not implemented.)rrrrfilename)rz Cannot copy z at revision z: file is missing on repo.)hf_apirrrrrDrangerFETCH_LFS_BATCH_SIZEget_paths_infor3NotImplementedErrorr(rWrrrr contentr)rrrrrrrrr files_to_copyrErrroffsetsrc_repo_files src_repo_fileurlresponsers r8_fetch_files_to_copyr+sN* Hg 6FOQM$+F8R$S$S j*% /9:z$$z:As5z+?@F#22FV.B%BC%1# 3N "0 mZ8-.TUU $$HUM=#5#5|"DE%!)"+ '!-!9!.!3!3 C +}00g0FH'1HPHXHXM=#5#5|"DE!"0A0$I**L9N("9#=#=">m#/x00JL$7%TB ?;sE'rcommit_messagecommit_description parent_commitc #K||nd}||d}|||d<d|dd}|D]}t|tr5|jr)tj d|j d |d z }It|trA|j d k(r2d |jj|j d ddt|tr]|j dk(rNd|j d|jjj|jjddt|tr#|jrdndd|j id:t|tr||j |j"f}t|t$r8d t'j(|j|j d dd|j*r+d|j d|j*jddt-dt-d|dt/|dd|dkDrtj1d|dyyw)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summary description parentCommitheader)rvaluerzSkipping file 'z(' in commit (ignored by gitignore file).rr)rmrq)rrWencodingr(lfsFilerw)rWalgorr deletedFolder deletedFilerW)rWrrz_Malformed files_to_copy (should be raw file content as bytes or RepoFile objects with LFS info.z(Unknown operation to commit. Operation: z. Upload mode: rOzSkipped z/ file(s) in commit (ignored by gitignore file).)r3rIrPrrr,rOrurrMrwrxrr+r.rCrDrEr]rqrrr(r5getattrinfo) rrrrr header_valuenb_ignored_filesr file_to_copys r8_prepare_commit_payloadr zsz"0B/M+SU .>PQL '4 ^$\ 22 i!3 49Q9Q LL?9+A+A*BBjk l  !   i!3 49O9OS\9\(335<<>%22 (  #5 69;Q;QUZ;Z %22$$0077;;=%1166   #8 9*3*=*== )"8"89   #6 7()*D*DiF\F\)]^L,.!#)#3#3L#A#H#H#J ) 6 6$,!!$ ) 6 6 (+//66!u :9+FI~t<=? { D! h/00_`asI I )NFNrG)NN)Kr>rqr[rVr collectionsr contextlibr dataclassesrr itertoolsrpathlibrr typingr r r r rrrrrrrtqdm.contrib.concurrentrrrerrorsr file_downloadrr(rrrutilsrrrr r!r"r#r$r%rrr& get_loggerr;rr|rr+rCrIr?r1CommitOperationrrrrrr4rr]rr rAr:r8rsl #%('pppp.&%<<   #    H %% &   < FF F: b1b1 b1J"*,?AVVW+_0E+$+\#"k &'k k  k #s(^ k sm k k smk k \ T #'+Q.*+Q.Q.Q.#s(^ Q.  Q. sm Q.Q. }Q. Q.Q.h# K ( )KKK#s(^ K  K sm K %Xc]" #U:u+<%= =>KKd)-#' _b)_bc8C=015U9J3KKL_b_b! _b C= _b d38n _br: