g!ddlZddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZmZmZmZdZdaGdd e Zed gd Gd d ej,ZGddeZy)N)suppress)cached_propertywraps)parse_qs)AbstractFileSystem)"get_package_version_without_importinfer_storage_options mirror_fromtokenizec.tfd}|S)Nc |i|S#t$rP}|js|j^}}t|trd|vrt t j ||d}~wwxYw)Nzdoes not exist)OSErrorargs isinstancestrFileNotFoundErrorerrnoENOENT)rkwargs exceptionmessagefuncs Q/var/www/openai/venv/lib/python3.12/site-packages/fsspec/implementations/arrow.pywrapperz wrap_exceptions..wrappersg (( ( >>&^^NGd'3',<,G' g>IM s A$A AA$)r)rrs` rwrap_exceptionsrs  4[   NceZdZdZdZfdZedZedZ e dZ ddZ dZ d Zd Zed Zed Zed ZeddZeddZeddZeddZedZedZdfd ZfdZxZS)ArrowFSWrapperzwFSSpec-compatible wrapper of pyarrow.fs.FileSystem. Parameters ---------- fs : pyarrow.fs.FileSystem /c Htda||_t|di|y)Npyarrow)rPYARROW_VERSIONfssuper__init__)selfr$r __class__s rr&zArrowFSWrapper.__init__3s$r?)r'r4rinfos rrEzArrowFSWrapper.infoWs;##D)&&v.%%rch|j|} |j|y#t$rYywxYw)NTF)r9rErr'r4s rexistszArrowFSWrapper.exists\s;##D)  IIdO!  s % 11cddlm}|j|jurd}n{|j|jurd}n`|j|j urFt tjtjtj|jd}|j|j||jdS)Nr)FileType directoryfileother)r<sizetypemtime)r=rJrO DirectoryFileNotFoundrrrosstrerrorr4rNrP)r'rErJkinds rr?zArrowFSWrapper._make_entryes' 99** *D YY(-- 'D YY(++ +#ELL"++ell2KTYYW WDIIIIZZ   rc V|j|jd}|j|jd}|j|d5}|dtjd} |j |d5}t j||ddd|jj|| dddy#1swY/xYw#t$rAtt5|jj|ddd#1swYxYwwxYw#1swYyxYw)Nrrbz.tmp.wb)r9rstrip_opensecrets token_hexopenshutil copyfileobjr$move BaseExceptionrr delete_file)r'path1path2rlstream tmp_fnamerstreams rcp_filezArrowFSWrapper.cp_filexs$$U+2237$$U+2237 ZZt $ 'w'8'8';&<=I YYy$/7&&w80 Y. % $0/! /0GG'' 211  % $sND/CC$CC CD*D DD DDD(c |j|jd}|j|jd}|jj||y)Nr)r9r[r$rb)r'rerfrs rmvzArrowFSWrapper.mvsJ$$U+2237$$U+2237  UE"rc\|j|}|jj|yr*)r9r$rdrGs rrm_filezArrowFSWrapper.rm_files$##D) D!rc|j|jd}|j|r)|r|jj |yt d|jj |y)Nrz0Can't delete directories without recursive=False)r9r[isdirr$ delete_dir ValueErrorrd)r'r4 recursivemaxdepths rrmzArrowFSWrapper.rms\##D)005 ::d ""4( !STT GG   %rc |dk(r0|r|jj}n]|jj}nF|dk(r|jj}n*|dk(r|jj}nt d|i}|dk7s|s)t tjdddk\rd|d<||fi|}t|||||fi|S) NrXrZabz'unsupported mode for Arrow filesystem: .r compression) r$open_input_fileopen_input_streamopen_output_streamopen_append_streamrrintr#split ArrowFile) r'r4mode block_sizeseekablermethod_kwargsstreams rr\zArrowFSWrapper._opens 4<0022 T\WW//F T\WW//FFthOP P 4rPrGs rmodifiedzArrowFSWrapper.modifieds-##D)ww$$T*000rc 8|dv|d<t||fddd|S)N)Nrr)startend)r%cat_file)r'r4rrrr(s rrzArrowFSWrapper.cat_files.")3zwEDdEfEErc 2d|d<t|||fi|y)NFr)r%get_file)r'rpathlpathrr(s rrzArrowFSWrapper.get_files!"z 00r)F)FN)rXNT)T)NN)__name__ __module__ __qualname____doc__ root_markerr&propertyr-rr2 classmethodr9rCrErHr?rrjrlrnrur\rrrrrr __classcell__r(s@rrr(s*K# !!>> 8&  &  ## ""&&II,6611!!11F11rrr) readseektellwritereadablewritablecloserNrc eZdZddZdZdZy)rNc f||_||_||_||_|x|_|_||_yr*)r4rr$r blocksizerr)r'r$rr4rrrs rr&zArrowFile.__init__s3   +55 rc|Sr*r"r,s r __enter__zArrowFile.__enter__s rc"|jSr*)r)r'rs r__exit__zArrowFile.__exit__szz|rr*)rrrr&rrr"rrrrsrrcDeZdZdZdZ dfd ZedZxZS)HadoopFileSystemz]A wrapper on top of the pyarrow.fs.HadoopFileSystem to connect it's interface with fsspechdfsc Pddlm}|||||||} t | dd| i|y)a; Parameters ---------- host: str Hostname, IP or "default" to try to read from Hadoop config port: int Port to connect on, or default from Hadoop config if 0 user: str or None If given, connect as this username kerb_ticket: str or None If given, use this ticket for authentication replication: int set replication factor of file for write operations. default value is 3. extra_conf: None or dict Passed on to HadoopFileSystem r)r)r0r1user kerb_ticket replication extra_confr$Nr")r=rr%r&) r'r0r1rrrrrrr$r(s rr&zHadoopFileSystem.__init__s;6 0 ##!   )B)&)rcHt|}i}|jddr|d|d<|jddr|d|d<|jddr|d|d<|jddr4t|d}|jddrt|dd|d<|S)Nr0usernamerr1 url_queryrr)r getrr)r4r8outqueriess r_get_kwargs_from_urlsz&HadoopFileSystem._get_kwargs_from_urls"s#D) 7764 f+CK 77:t $j/CK 7764 f+CK 77; %s;/0G{{=$/%()?)B%CM" r)defaultrNNN) rrrrr-r& staticmethodrrrs@rrrs=-H  %*N  rr)riorTr]r` contextlibr functoolsrr urllib.parser fsspec.specr fsspec.utilsrr r r rr#rIOBaserrr"rrrs~  ,!*$j1'j1Z      $;~;r