g A(dZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZdd lmZd d lmZmZmZmZmZm Z m!Z!m"Z"d d l#m$Z$d d l%m&Z&m'Z'd dl(m)Z)erd dl*m+Z+e jXe-Z.ej^dZ0dZ1dddZ2eGddZ3e" d:dee3dee4de4de4dee4dee4deee4e4fdeee5ee5ffdZ6Gdd eZ7Gd!d"eZ8 d:d#d$d%edee4deee4e4fdee4ddf d&Z9d'e5fd(Z:d)e5fd*Z;d+e5fd,Zd-e4ddf d1Z?d/ed2e3d0e>dee4fd3Z@d4eed5e4de8fd6ZAd#d$d7ee4d0e>deefd8ZBd#d$d7ee4d0e>deefd9ZCy);z.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize)Path) TYPE_CHECKINGBinaryIODictIterableListOptionalTuple TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_url get_sessionhf_raise_for_status http_backoffloggingtqdmvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)is_tqdm_disabled)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-TypecreZdZUdZeed<eed<eed<edefdZ edefdZ ed e fd Z y ) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathct|}tj|d5}|jddd}t |}ddd||S#1swYxYw)Nrbr"rr#)rioopenpeekr)clsr$r"filer#shas H/var/www/openai/venv/lib/python3.12/site-packages/huggingface_hub/lfs.py from_pathzUploadInfo.from_pathOsXt} WWT4 DYYs^DS)Fd#C!S88! s AAdatacbt|j}|t||dd|S)Nr')r"r#r)rdigestlen)r,r1r.s r/ from_byteszUploadInfo.from_bytesWs-Tl!!#D $t*SAAfileobjc|jd}|jdtjt |}|j }|jdtj||||S)Nr'rr()readseekr)SEEK_SETrtell)r,r7r#r.r"s r/ from_fileobjzUploadInfo.from_fileobj\sXc" Q $'"||~ Q $S88r6N) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr0r5rr=r6r/r!r!<sj  M I M9S99BeBB9899r6r! upload_infostoken repo_typerepo_idrevisionendpointheadersreturnc||ntj}d}|tjvrtj|}|d||d}dddg|D cgc])} | jj | j d+c} d d } |d t |i| d <itt| |xsi}tj||| } t| | j} | jdd} t| ts t!d| Dcgc]}d|vst#|c}| Dcgc]}d|vst%|c}fScc} wcc}wcc}w)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request Returns: `LfsBatchInfo`: 2-tuple: - First element is the list of upload instructions from the server - Second element is an list of errors, if any Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartoidr"r) operation transfersobjects hash_algonameref)rIrNjsonrZzMalformed response from servererror)rENDPOINTREPO_TYPES_URL_PREFIXESrhexr"r LFS_HEADERSrrpostrr_get isinstancelist ValueError_validate_batch_actions_validate_batch_error)rHrIrJrKrLrMrN url_prefix batch_urlrSpayloadresp batch_inforZobjs r/post_lfs_batch_inforrfsP$/xY5G5GHJI55566yA *Aj\'2MNI{+ '  '}}((*  '   G '("34   ' =bG =  iw  GDJnnY-G gt $9::29O#G3r?r@rDrCrFrGr6r/rtrts O Ir6rtc,eZdZUdZeed<eeed<y)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rWpartsN)r>r?r@rArFrCr rtrGr6r/rxrxsI H  r6rxrXrlfs_batch_actioncBt||jd}|$tjd|jdy|dd}t ||djd}| t ||jdi}|jd} t |d | } |  t| } t||| | n t|| |t |t |d |} tj| t|||jj j#|jj$d} t'| tj|jdy#ttf$rtd | d wxYw)aw Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrSverifyheader chunk_sizehref)rMzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rXr~r upload_url)rXr)rIrNrVr^z: Upload successful)rjrfloggerdebug path_in_repo_validate_lfs_actionrrDri TypeError_upload_multi_part_upload_single_partrrer upload_inforrcr"r) rXrzrIrNrMr| upload_action verify_actionr~rr verify_url verify_resps r/ lfs_uploadrs8,-""9-G ' (>(>'??mno%Y/9M'$Y/33H=M ]+  x ,FL)J' f(=QJ ZJ Yv*akliJG ]++M&,A8L !m(( $5'B"..5599;YEZEZE_E_`) K( LLI**++>?@%I& fgqfrrtu  s - E??F lfs_actionct|jdtr0|jd*t|jdts t d|S).validates response from the LFS batch endpointrr~z"lfs_action is improperly formatted)rgrfrFdictri)rs r/rrsK :>>&)3/ ^^H % -JNN8> r6lfs_batch_actionscPt|jdtrt|jdts t d|jdijd}|jdijd}| t || t ||S)rrWr"z)lfs_batch_actions is improperly formattedr|rSr})rgrfrFrDrir)rrrs r/rjrjs (,,U3S 9jIZI^I^_eIfhk>lDEE%)))R8<t|jdtrt|jdts t d|S)rrWr"z'lfs_batch_error is improperly formattedr`messagecode)rgrfrFrDrir)r error_infos r/rkrks **513 7JGZGZ[aGbdgz(_get_sorted_parts_urls..os!A$r6)keyz0Invalid server response to upload large LFS file)sorteditemsisdigitr4rDrr"ri)r~rrpart_numr_sorted_part_upload_urls num_partss r/rrfs$-3LLN ,:(Hj##%#h-!*;Xr"J/,:      MAz   +,ID))J677KLL ""  s 6B  B%rrWcg}t|D]H\}}|jd}||dk(rtd|d|dz|j|dz|dJ||dS)NrvrQzInvalid etag (`z`) returned for part r)rurv)rWry) enumeraterfriappend)rrWry part_numberr~rvs r/rrxs| "E()9: Vzz&! <42:tf4I+XY/IZ[\ \ )Ao  ; ''r6rc 8g}|jd5}t|D]T\}}t|||z|5}td||d}t ||j |j dddV ddd|S#1swYkxYw#1swY|SxYw)NTr) seek_from read_limitrrr)rrrrrrrN) rXrrrNr7part_idxpart_upload_url fileobj_slicepart_upload_ress r/rrsG   T  *g)23D)E %Ho$x/%".?Vj#$O4667*F + N + Ns"#B6B. BB BBc ^ ddlm}dt j |j v}|stjd|jj}|j}t|dkDrd|dd}td d |d|ttj!d 5} |d|j"||d ddd|rd|j$ini}|s|j%||cdddS#t$r tdwxYw#t&$r} t)d| d} ~ wwxYw#1swYyxYw)Nr)multipart_uploadzFast uploading using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.callbackzYou are using an outdated version of `hf_transfer`. Consider upgrading to latest version to enable progress bars using `pip install -U hf_transfer`.(u(…)iBTzhuggingface_hub.lfs_upload)unit unit_scaletotalinitialdescdisabler\) file_path parts_urlsr max_filesparallel_failures max_retriesz~An error occurred while uploading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.rG) hf_transferr ImportErrorriinspect signature parametersrrrr"rr4rrrgetEffectiveLevelrupdate Exception RuntimeError) rXrrrsupports_callbackrrprogressoutputes r/rrsc 0#g&7&78H&I&T&TT   c   ! ! & &E  ! !D 4y2~tCDzl#   !9!9!;< )   %#33,%"% 5FJ02F! OOE "5  #   Q   H *  '  s;C.%D#')DD#.D D  DD  D##D,)NNN)DrArr)rer dataclassesrmathros.pathrpathlibrtypingrrr r r r r r urllib.parserhuggingface_hubrutilsrrrrrrrr utils._lfsr utils.sharr utils.tqdmr _commit_apir get_loggerr>rcompile OID_REGEXLFS_MULTIPART_UPLOAD_COMMANDrdr!rFrrrrtrxrrrjrkrrDrrrrrrGr6r/rs5 !\\\ %   %*(/   H % BJJ( ) 5-2  &9&9 &9R #"(,K:&K C=KK K sm K sm Kd38n %K 4:tDz !"KK\9  (," CA#CACA C=CAd38n % CA sm CA  CALT t  4 &S&T&, ("6 ( (RU (cf (ko (F#4#j#c#VZ[^V_#$ (d4j (s (GY (#8