gj@$dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZd d lmZmZmZmZmZmZd d lmZd d lmZmZe rd d lmZeje Z!ej"dZ#dZ$dddZ%GddZ&e d8dee&de'dzde'de'de'dzde'dzde(e'e'fdzde)e'dzde*e)e(e)e(e'dzffdZ+Gdde Z,Gd d!e Z- d9d"d#d$e(de'dzde(e'e'fdzde'dzddf d%Z.d&e(fd'Z/d(e(fd)Z0d*e(fd+Z1d"d#d,e'ddfd-Z2d"d#d.e(d/e3d,e'ddf d0Z4d.e(d1e&d/e3de)e'fd2Z5d3e)e(d4e'de-fd5Z6d"d#d6e)e'd/e3de)e(fd7Z7dS):z.Git LFS related type definitions and utilitiesN)Iterable)ceil)getsize) TYPE_CHECKINGBinaryIO TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_urlhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec eZdZdZ ddedededzdedzfdZedefd Z e j d eddfd Z ede fd Z defd Z edefdZedefdZedefdZdS) UploadInfoa Data structure holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol. The SHA256 of the blob is computed lazily: creating an `UploadInfo` from a local path only reads the first 512 bytes of the file. The full file is read (and hashed) only if `sha256` is accessed before it has been set. When a file is uploaded through the Xet protocol, the SHA256 is computed during upload (single read pass) and set afterwards. Args: size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob sha256 (`bytes`, *optional*): SHA256 hash of the blob, if already known. Otherwise computed lazily from `source_path`. source_path (`str`, *optional*): Path to the local file the blob comes from. Required to lazily compute `sha256` if not provided. Nsizesampler source_pathcd||td||_||_||_||_dS)Nz2Either `sha256` or `source_path` must be provided.) ValueErrorrr_sha256 _source_path)selfrrrrs W/var/www/app.qaans.com/backend/venv/lib/python3.11/site-packages/huggingface_hub/lfs.py__init__zUploadInfo.__init__Js@ >k1QRR R   'returnc|jJ|jJt|jd5}t||_dddn #1swxYwY|jS)zZSHA256 of the blob. If not set yet, reads the whole file from `source_path` to compute it.Nrb)rropenr)r files r!rzUploadInfo.sha256Xs < $000d'.. 1$*400  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1|sAA A valuec||_dS)Nr)r r)s r!rzUploadInfo.sha256as  r#c|jduS)zVWhether the SHA256 is already known (accessing `sha256` will not trigger a file read).Nr+)r s r! is_hashedzUploadInfo.is_hashedes|4''r#cb|j|jnd}d|jd|dS)NzzUploadInfo(size=z , sha256=))rhexr)r shas r!__repr__zUploadInfo.__repr__js=$(L$>>>sA  AAdatact|}|t||dd|S)Nr5)rrr)rdigestlen)r7r9r1s r! from_byteszUploadInfo.from_bytesus>Tll!!##sD $tt*SAAAAr#fileobjc|d}|dtjt |}|}|dtj||||S)Nr5r)rrr)readseekioSEEK_SETrtell)r7r>rr1rs r! from_fileobjzUploadInfo.from_fileobjzspc"" Q $$$'""||~~ Q $$$sS8888r#)NN)__name__ __module__ __qualname____doc__intbytesstrr"propertyrsetterboolr-r2 classmethodr8r=rrEr#r!rr5s0 $"& ( ( ( ( ( 4Z ( ( ( (X ]Ed](4(((X(=#====?S???[? BeBBB[B98999[999r#r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersr$c|||n tj}d}|tjvrtj|}|d||d} d||nddgd|Dd d } |d t|i| d <itt | |pi}t d| || } t| | } | dd} t| tstd| d}t|tr|nd}d| Dd| D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` token (`str` or `None`): An authentication token (see https://huggingface.co/settings/token). Pass `None` to fall back to the local cached token (or no token if unauthenticated). repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. endpoint (`str`, *optional*): The Hub endpoint to send the request to. Defaults to the value of `HF_ENDPOINT`. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcPg|]#}|j|jd$S)oidr)rr0r).0r]s r! z'post_lfs_batch_info..sG   }((**      r#r) operationrYobjects hash_algonameref)rSPOSTrXjsonrfzMalformed response from servertransferc4g|]}d|vt|Serror)_validate_batch_actionsrcobjs r!rdz'post_lfs_batch_info..s+OOO#G3.s&III'S.. s # #...r#)r ENDPOINTREPO_TYPES_URL_PREFIXESr LFS_HEADERSr rrrlget isinstancelistrrL)rRrSrTrUrVrWrXrY url_prefix batch_urlpayloadresp batch_inforfchosen_transfers r!post_lfs_batch_inforsb$/xxY5GHJI5556yA NNjN'NNNI"+"7YYg{=S  '      G '("3"34   ' ' ' =bG  7 I I IDJnnY--G gt $ $;9::: nnZ00O)3OS)I)ISootO POOOOIIwIII r#c$eZdZUeed<eed<dS) PayloadPartT partNumberetagN)rFrGrHrJ__annotations__rLrQr#r!rrs"OOO IIIIIr#rc4eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rbpartsN)rFrGrHrIrLrr{rrQr#r!rrs2II HHH  r#rrerlfs_batch_actionct||d}|%td|jddS|dd}t ||dd}|t ||di}|d} t |d | } | M t| } n(#ttf$rtd | d wxYwt||| | nt|| |t |t |d |} td| t|||jj|jjd} t%| t|jddS)a Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. token (`str`, *optional*): An authentication token (see https://huggingface.co/settings/token). Used to call the optional LFS verify step at the end of the upload. If `None`, falls back to the local cached token. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. endpoint (`str`, *optional*): The Hub endpoint to send the request to. Defaults to the value of `HF_ENDPOINT`. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadr]verifyheader chunk_sizehref)rWzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rerr upload_url)rerrj)rSrXrarkz: Upload successful)rqryloggerdebug path_in_repo_validate_lfs_actionr rJr TypeError_upload_multi_part_upload_single_partrr upload_inforr0rr) rerrSrXrWr upload_action verify_actionrrr verify_url verify_resps r! lfs_uploadrsD,---""9--G n (>nnnooo%Y/9M'''$Y/33H==M ]+++  x , ,FL))J' f(=QQQJ ZJJI&   ugquuu   Yv*akllllliJGGGG ]++++M&,A8LL "  $5'BBB".599;;YEZE_``    K((( LLI*???@@@@@s C**%D lfs_actionct|dtr=|d7t|dtst d|S).validates response from the LFS batch endpointrrNz"lfs_action is improperly formatted)rzryrLdictr)rs r!rr9se :>>&))3//? ^^H % % -JNN8>> r#lfs_batch_actionsct|dtr(t|dtst d|did}|did}|t ||t ||S)rrbrz)lfs_batch_actions is improperly formattedrr]r)rzryrLrJrr)rrrs r!rqrqCs (,,U33S 9 9FjIZI^I^_eIfIfhk>l>lFDEEE%)))R88<responses r!rr_s   T  * *&gz@@@H%%%&&&&&&&&&&&&&&&&&&s"AA  A rrc t||j|}t|||}td|t ||jjt}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rrr)resorted_parts_urlsrrj)rlrXN) _get_sorted_parts_urlsr_upload_parts_iterativelyr_get_completion_payloadrr0rxr)rerrrrresponse_headerscompletion_ress r!rrss /f)J_lvwww1/@Z " $%5y7L7S7W7W7Y7Y Z Z N '''''r#rcdtd|DdD}t|}|t|j|z krt d|S)Ncg|]\}}|SrQrQ)rc_rs r!rdz*_get_sorted_parts_urls..s,    Az    r#cg|]>\}}|r%t|dk,t|d|f?S)r )isdigitr<rJ)rcpart_numrs r!rdz*_get_sorted_parts_urls..s\   (Hj##%% +.h--!*;*;Xr""J/*;*;*;r#c|dS)NrrQ)ts r!z(_get_sorted_parts_urls..s !A$r#)keyz0Invalid server response to upload large LFS file)sorteditemsr<rrr)rrrsorted_part_upload_urls num_partss r!rrs  #  ,2LLNN           +,,ID)J67777KLLL ""r#rrbcg}t|D]U\}}|d}||dkrtd|d|dz||dz|dV||dS)Nrr[zInvalid etag (`z`) returned for part r )rr)rbr) enumerateryrappend)rrbr part_numberrrs r!rrs "E()9::    Vzz&!! <42::[t[[+XY/[[\\ \ )Ao       ' ''r#rcbg}|d5}t|D]m\}}t|||z|5}td||}t |||jdddn #1swxYwYn dddn #1swxYwY|S)NTr) seek_from read_limitrr)rrrrrrrX) rerrrXr>part_idxpart_upload_url fileobj_slicepart_upload_ress r!rrsKG   T  * * 8g)23D)E)E 8 8 %Ho$x/% 8".uoM"Z"Z"Z#O4446777 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 Ns5*B$rs54 $$$$$$5555555555 %%%%%%%$$$$$********0//////  H % % BJ( ) ) 5-2 L9L9L9L9L9L9L9L9^  %)"&XX:&X :XX X Dj X Dj X#s(^d "XCy4X 4:tDz3: -.XXXXv9 %) JAJA#JAJA :JA#s(^d " JA Dj JA  JAJAJAJAZT t     4    &S&T&&&&(("6((RU(cf(ko((((.#4#j#c#VZ[^V_####$ (d4j (s (GY ( ( ( (#8