gޘdZddlZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddlmZmZmZmZmZmZmZddlmZd d lmZd d lmZd d lmZd d lmZm Z m!Z!m"Z"m#Z#d dl$m%Z%m&Z&m'Z'd dl(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0edZ1eZ2edd d ejfZ4edd d Z5dee6e'ffdZ7dZ8dZ9dddifdZ:dldZ;dddifdZdldZ?dddifdZ@dldZAdddifd ZBdld!ZCdddifd"ZDdld#ZEdddifd$ZFdld%ZGdddifd&ZHdld'ZIddifd(ZJdmd)ZKdifd*ZLd+ZMdifd,ZNd-ZOdifd.ZPd/ZQdifd0ZRd1ZSdifd2ZTd3ZUid4e:d5e<d6eHd7e>d8e@d9eBd:eDd;eFdeNd?ePd@eRdAeRdBeTdCeTdDeTZVid4e;d5e=d6eId7e?d8eAd9eCd:eEd;eGdeOd?eQd@eSdAeSdBeUdCeUdDeUZWdEZXdifdFZYdGZZdHZ[dIZ\dJZ]dKZ^dLZ_e\e]e^e_dMZ`dNZa ddOlbmcZcecjZeeae`dP<dSZi ddljZjeie`dT<dUZk ddllZmeke`dV<dWZndXZoGdYdZZpGd[d\ee1ZqGd]d^eqe&ZrGd_d`eqepZs dndaedbe%dcee%ddetdeetdfe6dgetdhetde&fdiZudjee6efdetfdkZvy#ef$r: ddlcZcecjZeedQeheae`dP<n#ef$re-dPdRe`dP<YnwxYwYwxYw#ef$re-dTdTe`dT<YwxYw#ef$re-dVdVe`dV<YwxYw)oz"Python code for reading AVRO filesN)datetimetimezone)Context)BytesIO)error)IOUnionOptionalGenericTypeVarIteratorDict)warn) BinaryDecoder)AvroJSONDecoder)LOGICAL_READERS)extract_record_typeis_single_record_unionis_single_name_unionextract_logical_type parse_schema)Schema AvroMessage NamedSchemas)SchemaResolutionErrorMAGIC SYNC_SIZE HEADER_SCHEMAmissing_codec_lib) NAMED_TYPES AVRO_TYPESTi)tzinforeturnc iidS)N)writerreaderr)F/var/www/openai/venv/lib/python3.12/site-packages/fastavro/_read_py.py_default_named_schemasr,-s B ''r*ct|tst|tryt|tst|tr t|||S||k(ry|dk(r|dvry|dk(r|dvry|dk(r|dk(ry|d k(r|d k(ry|d k(r|d k(ry|d j |}|d j |}|| t |||Sy#t$rYywxYw) NTFint)longfloatdoubler/)r0r1r0r1stringbytesr'r() isinstancelistdict match_schemasrget match_types) writer_type reader_type named_schemas writer_schema reader_schemas r+r9r91s+t$ ;(E+t$ ;(E  k=I Ik!  +1L"L  ;2E#E  K8$;  [G%;  K8$;!(+// =-GG '%  s C CCc d|d|}t|tr|St|tr#|D]}t|||s|cSt|t|tr|d}n|}t|tr|d}n|}||cxk(rdk(rnnt|d|d|r|S||cxk(rdk(rnnt|d|d|r|S|t vr|t vr}||cxk(rdk(r nn|d |d k7rtd|d |d |d j d d}|d j d d}|jdg} ||k(s |d | vs|| vr6|S|tvr|t vrt||d |r|d St|||r|St|)NzSchema mismatch: z is not typemapvaluesarrayitemsfixedsizez size is different than z sizename.aliases) r4r5r9rr6r!splitr8r") w_schemar_schemar< error_msgschemaw_typer_type w_unqual_name r_unqual_name r_aliasess r+r7r7Ms#H:XhZ@I(D! Hd #F8V]; ( 2 2 h %f%FF h %f%FF V $u $8H-x/A=Q v ( (8G,hw.?O { "v'<*7*x/?8FCS/S+'z1I(SXY%V,2237;M$V,2237;M Y3I.F#y0 I- : %&K*?68F#3]C''  7O#I..r*c"|jSN read_nulldecoderr=r<r>optionss r+rXrX    r*c$|jyrVrWrZr=r<s r+ skip_nullr_ r*c"|jSrV read_booleanrYs r+rcrcs    !!r*c$|jyrVrbr^s r+ skip_booleanres r*c"|jSrVread_intrYs r+rhrhs    r*c$|jyrVrgr^s r+skip_intrjs r*c"|jSrV read_longrYs r+rmrmr\r*c$|jyrVrlr^s r+ skip_longror`r*c"|jSrV read_floatrYs r+rrrr    r*c$|jyrVrqr^s r+ skip_floatru r*c"|jSrV read_doublerYs r+ryrys     r*c$|jyrVrxr^s r+ skip_doubler{s r*c"|jSrV read_bytesrYs r+r~r~rsr*c$|jyrVr}r^s r+ skip_bytesrrvr*cF|j|jddS)Nhandle_unicode_errorsstrict)r) read_utf8r8rYs r+rrs+   %kk*A8L  r*c$|jyrV)rr^s r+ skip_utf8rr`r*c.|d}|j|SNrF read_fixed)rZr=r<r>r[rFs r+rrs  D   d ##r*c0|d}|j|yrr)rZr=r<rFs r+ skip_fixedrs  D tr*c|d|j}|r9||dvr2|jd}|r|S|d}|d|dd|}t||S)Nsymbolsdefaultz! not found in reader symbol list rGz, known symbols: ) read_enumr8r) rZr=r<r>r[symbolrsymlistmsgs r+rrs}9 %g&7&7&9 :F}Y'??##I. N#I.GH=mF>S=TTefmenoC', , Mr*c$|jyrV)rr^s r+ skip_enumrr`r*c |rfd}nfd}g}|j|jD]}|j||||||j|S)Nc.t||d|d|SNrD read_datarZrLrMr[r<s r+ item_readerzread_array..item_reader%s)!!  r*c(t||dd|Srrrs r+rzread_array..item_reader0s$!  r*)read_array_start iter_arrayappendread_array_end)rZr=r<r>r[r read_itemsitems ` r+ read_arrayrsn  J ""$    %  r*c|j|jD]}t||d||jyr)rr skip_datarrZr=r<rs r+ skip_arrayrLs@ ""$'=1=A% r*c|rfd}nfd}i}|j|jD]}|j}||||||<!|j|S)Nc.t||d|dSNrBrrZrLrMr<r[s r+rzread_map..item_reader^s)""  r*c(t||ddSrrrs r+rzread_map..item_readeris$"  r*)read_map_startiter_mapr read_map_end) rZr=r<r>r[rrrkeys ` ` r+read_maprUsl  J   "!%g}mL 3#  r*c|j|jD]"}|jt||d|$|j yr)rrrrrrs r+skip_maprsL   "'=2MB# r*c@|j}||}d}|rrd|d|}t|ts(t|||rt |||||} nLt ||D]"} t|| |s| }t |||| |} nt |t |||d|} |j d} |j d} |j d} |j d}| r t|r| S|r!t|tvr|r|dn|d}|| fS|r-t|tvr|r |d|dn |d |d}|| fS| r t|r| S| rt|d k(r|r|dn|d}|| fS| r-t|tvr|r |d|dn |d |d}|| fS| S) Nzschema mismatch: z not found in return_record_name_overridereturn_record_namereturn_named_type_overridereturn_named_typerGr(r'record) read_indexr4r5r9rrr8rrr!r"r)rZr=r<r>r[index idx_schemaidx_reader_schemarresultrOrrrr schema_names r+ read_unionrs5    Eu%J!-}oN-.:}mD"!! ,C00'z6=A(.%&"% F(,C007J tWM")++.K"L %9:!(-I!J $78!&:=&I 2:>+M): f % 6@R V$$ 2:>jP! ( #$5 6v >x(4V<  V$$ $)? )N  3J ?8 K): f % 6@R V$$  3J ?z Q! ( #$5 6v >x(4V<  V$$ r*cD|j}t||||yrV)rr)rZr=r<rs r+ skip_unionrs!    E g}U+];r*cni}|$|dD]}t||d|d|||d<|Si}i}|dD]&} | || d<| jdgD]} | || < (|dD]W}|j|d|j|d} | rt||d|| d||| d<Ht||d|Yt|t|kDr_|dD cgc]} | d } } |j D]7\} }| | vs | |vsd|vr |d||d< d|dd|d}t ||Scc} w) aA record is encoded by encoding the values of its fields in the order that they are declared. In other words, a record is encoded as just the concatenation of the encodings of its fields. Field values are encoded per their schema. Schema Resolution: * the ordering of fields may be different: fields are matched by name. * schemas for fields with the same name in both records are resolved recursively. * if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored. * if the reader's record schema has a field that contains a default value, and writer's schema does not have a field with the same name, then the reader should use the default value from its field. * if the reader's record schema has a field with no default value, and writer's schema does not have a field with the same name, then the field's value is unset. Nfieldsr@rGrJrzNo default value for field z in )rr8rlenrDr)rZr=r<r>r[rfieldreaders_field_dictaliases_field_dictfalias readers_field writer_fieldsf_namers r+ read_recordrs2F"8,E$-f  %F5= !-V ME x(A,- qy )y"-,-"5).) #8,E.22f "&&uV}5M09&M!!&) 1}V,-'5=-@-" ! "S[ 00=h0GH0G1QvY0GMH!3!9!9!; .63G E)05i0@uV}- ;E&M?$}]cOdNef3C88 "< MIs D2c8|dD]}t||d|y)Nrr@)r)rZr=r<rs r+ skip_recordr)s!x('5=-8)r*nullbooleanr2r.r/r0r1r3rEenumrCrAunion error_unionrrrequestc|dk(r|dk(s|dk(r t|S|dk(r|dk(s|dk(r t|S|dk(r|dk(r|jS|dk(r|dk(r|jS|S)Nr.r0r1r/r2r3)r0encodedecode)datar:r;s r+ maybe_promoterWse ' ![H%<; f ' ![H%<; h;'#9{{}g+"9{{} Kr*ct|}|r t|||}tj|}|rW ||||||}d|vr,t|}tj|} | r | |||S|t||t|S|St||d|||dj||S#t$rt d|d|j wxYw)z/Read data from file object according to schema.z cannot read z from logicalTyper'r() rr7READERSr8 StructErrorEOFErrorforrrr) rZr=r<r>r[ record_type reader_fnr logical_typefns r+rrfs&m4K%     K(I K D M )/ >L $$\2B$ }==  $ {4G 4VW WK  ( #K 0  ( # ' ' 6     K\+fWZZLIJ J Ks B00%Cct|}tj|}|r ||||yt||d||y)Nr')rSKIPSr8r)rZr=r<rrs r+rrs?%m4K +&I'=-8'=2;?Or*cJ|jt|k7r tdy)z=Skip an expected sync marker, complaining if it doesn't matchzexpected sync marker not foundN)readr ValueError)r sync_markers r+ skip_syncrs$ wwy[(9::)r*c4t|jS)zRead block in "null" codec.)rr~)rZs r+null_read_blockrs 7%%' ((r*c||j}ttjdj |S)zRead block in "deflate" codec.i)r~rzlib decompressobj decompressrZrs r+deflate_read_blockrs3    D 4%%c*55d; <>$' ((r*cvt|}|j|}ttj|SrV)rmrrlzmarrZlengthrs r+ xz_read_blockrs0 w F   f %D 4??4( ))r*)rdeflatebzip2xzct|}|j|dz }|jdtt|S)N)rmrrsnappy_decompressrs r+snappy_read_blockrs? w F   fqj )D q $T* ++r*)snappyrzbSnappy compression will use `cramjam` in the future. Please make sure you have `cramjam` installedcramjamct|}|j|}ttjj j |SrV)rmrr zstandardZstdDecompressorrrrs r+zstandard_read_blockr sD w F   f %D 9--/==?JJ4P QQr*rct|}|j|}ttjj |SrV)rmrrlz4blockrrs r+lz4_read_blockrs6 w F   f %D 399''- ..r*r c#6K|d}tj|}|std|d} |j} ||} t | D]} t t| ||||t|j|Z#t$rYywxYww)z"Return iterator over avro records.syncUnrecognized codec: rN) BLOCK_READERSr8rrmrrangerrrr) rZheadercodecr=r<r>r[r read_block block_countblock_fois r+_iter_avro_recordsrs.K""5)J /w788K  !++-Kg&{#Ah'  $ '**k*#    s).BB A B BBBBc #vK|d}tj|}|std| |jj } |j } ||} t|j||jj | z } t| | ||||| | | |#t $rYywxYww)z!Return iterator over avro blocks.rrN) rr8rrtellrmrrBlock) rZrrr=r<r>r[rroffsetnum_block_records block_bytesrFs r+_iter_avro_blocksr!'s.K""5)J /w788 "  ' 1 1 3 !) '**k*zz 6)               s+AB9 B*AB9* B63B95B66B9c"eZdZdZdZdZdZy)raAn avro block. Will yield records when iterated over .. attribute:: num_records Number of records in the block .. attribute:: writer_schema The schema used when writing .. attribute:: reader_schema The schema used when reading (if provided) .. attribute:: offset Offset of the block from the beginning of the avro file .. attribute:: size Size of the block in bytes c ||_||_||_||_||_||_||_||_| |_yrV) bytes_ num_recordsrr>r=_named_schemasrrFr[) selfr$r%rr>r=r<rrFr[s r+__init__zBlock.__init__isG & **+   r*c#Kt|jD]N}tt|j|j |j |j|jPywrV) rr%rrr$r=r&r>r[)r'rs r+__iter__zBlock.__iter__sVt''(Adkk*""##""  )sA(A*cdt|jd|jdzd|jd|jd|j zS)Nz Avro block: z bytes, z records, zcodec: z , position +)rr$r%rrrFr's r+__str__z Block.__str__s[3t{{+,H 5!!"*- . |;t{{m1TYYKH I r*N)__name__ __module__ __qualname____doc__r(r*r.r)r*r+rrQs., r*rcLeZdZdifdZdZedZdeefdZ defdZ y) file_readerNct|tr||_nt||_t |_|r!t ||j dd|_nd|_||_d|_ y)Nr(F) _write_hint) r4rrZrr,r&rr>r[_elems)r' fo_or_decoderr>r[s r+r(zfile_reader.__init__sh m_ 5(DL)7DL46 !-t228<%"D  "&D   r*c@ t|jt|jd|j|_|j djDcic]\}}||jc}}|_ tj|jd|_ |jjdd|_|j d}nd}t#|j|jddd| |_y#t $r tdwxYwcc}}w) Nz(cannot read header - is it an avro file?metaz avro.schemaz avro.codecrTFr')r6_force_ignore_default_error)rrZrr&r[_headerrrrDrmetadatajsonloads_schemar8rr>rr=)r'kvignore_default_errors r+ _read_headerzfile_reader._read_headers I$ ## DL48<<3G3M3M3OP3O41aAHHJ3OP zz$-- ">? ]]&&|V<     )#' #( * LL    )"6  + IGH H IQs6DDDcNddl}|jdt|jS)Nrz@The 'schema' attribute is deprecated. Please use 'writer_schema')warningsrDeprecationWarningrA)r'rGs r+rOzfile_reader.schemas# N  ||r*r%c>|jst|jSrV)r7NotImplementedErrorr-s r+r*zfile_reader.__iter__s{{% %{{r*c,t|jSrV)nextr7r-s r+__next__zfile_reader.__next__sDKK  r*) r/r0r1r(rEpropertyrOr r#r*rMr)r*r+r4r4sE .$ L(1+ !!!r*r4c^eZdZdZ d deeefdeede de de de de ffd Z xZ S) r(a Iterator over records in an avro file. Parameters ---------- fo File-like object to read from reader_schema Reader schema return_record_name If true, when reading a union of records, the result will be a tuple where the first value is the name of the record and the second value is the record itself return_record_name_override If true, this will modify the behavior of return_record_name so that the record name is only returned for unions where there is more than one record. For unions that only have one record, this option will make it so that the record is returned by itself, not a tuple with the name. return_named_type If true, when reading a union of named types, the result will be a tuple where the first value is the name of the type and the second value is the record itself NOTE: Using this option will ignore return_record_name and return_record_name_override return_named_type_override If true, this will modify the behavior of return_named_type so that the named type is only returned for unions where there is more than one named type. For unions that only have one named type, this option will make it so that the named type is returned by itself, not a tuple with the name handle_unicode_errors Default `strict`. Should be set to a valid string that can be used in the errors argument of the string decode() function. Examples include `replace` and `ignore` Example:: from fastavro import reader with open('some-file.avro', 'rb') as fo: avro_reader = reader(fo) for record in avro_reader: process_record(record) The `fo` argument is a file-like object so another common example usage would use an `io.BytesIO` object like so:: from io import BytesIO from fastavro import writer, reader fo = BytesIO() writer(fo, schema, records) fo.seek(0) for record in reader(fo): process_record(record) .. attribute:: metadata Key-value pairs in the header metadata .. attribute:: codec The codec used when writing .. attribute:: writer_schema The schema used when writing .. attribute:: reader_schema The schema used when reading (if provided) rr>rrrrrc V|||||d}t |||tjtrjj j jdj _d_jdjd<ijd<fd} | _ yjtjjjjjj j_ y)Nrrrrrr(r'c3$Kjjsutjjjj j jjjjstyywrV)rZdonerr=r&r>r[drainr-sr+r7zreader.__init__.._elemsFsk,,++# **++** LL&&(,,++s B BB)superr(r4rZr configurer>r&r=r7rErr=rr[) r'rr>rrrrrr[r7 __class__s ` r+r(zreader.__init__+s#5+F%:!2*D   ]G4 dllO 4 LL " "4#5#5t7J7J87T U!%!3!3D !%D ,0,?,?,ID   ),.D   ) )!(DK    ,   ""##"" DKr*NFFrFF) r/r0r1r2r rrr rboolstrr( __classcell__rWs@r+r(r(s{FV+/#(,1%-"'+03 "o% &3 '3! 3 &* 3 # 3 3%)33r*r(cTeZdZdZ d dedeedededededeffd Z xZ S) block_readera?Iterator over :class:`.Block` in an avro file. Parameters ---------- fo Input stream reader_schema Reader schema return_record_name If true, when reading a union of records, the result will be a tuple where the first value is the name of the record and the second value is the record itself return_record_name_override If true, this will modify the behavior of return_record_name so that the record name is only returned for unions where there is more than one record. For unions that only have one record, this option will make it so that the record is returned by itself, not a tuple with the name. return_named_type If true, when reading a union of named types, the result will be a tuple where the first value is the name of the type and the second value is the record itself NOTE: Using this option will ignore return_record_name and return_record_name_override return_named_type_override If true, this will modify the behavior of return_named_type so that the named type is only returned for unions where there is more than one named type. For unions that only have one named type, this option will make it so that the named type is returned by itself, not a tuple with the name handle_unicode_errors Default `strict`. Should be set to a valid string that can be used in the errors argument of the string decode() function. Examples include `replace` and `ignore` Example:: from fastavro import block_reader with open('some-file.avro', 'rb') as fo: avro_reader = block_reader(fo) for block in avro_reader: process_block(block) .. attribute:: metadata Key-value pairs in the header metadata .. attribute:: codec The codec used when writing .. attribute:: writer_schema The schema used when writing .. attribute:: reader_schema The schema used when reading (if provided) rr>rrrrrc |||||d}t |||||jt|j|j |j |j|j|j|j|_ y)NrQ) rUr(rEr!rZr=rr=r&r>r[r7) r'rr>rrrrrr[rWs r+r(zblock_reader.__init__s}#5+F%:!2*D   ]G4 ' LL LL JJ          LL  r*rX) r/r0r1r2rr rrYrZr(r[r\s@r+r^r^asp:~+/#(,1%-"'+0   ' !  &*  #    %)  r*r^rr=r>rrrrrc||k(rd}t}t||d}|rt||d}t|} |||||d} t| |||| S)aReads a single record written using the :meth:`~fastavro._write_py.schemaless_writer` Parameters ---------- fo Input stream writer_schema Schema used when calling schemaless_writer reader_schema If the schema has changed since being written then the new schema can be given to allow for schema migration return_record_name If true, when reading a union of records, the result will be a tuple where the first value is the name of the record and the second value is the record itself return_record_name_override If true, this will modify the behavior of return_record_name so that the record name is only returned for unions where there is more than one record. For unions that only have one record, this option will make it so that the record is returned by itself, not a tuple with the name. return_named_type If true, when reading a union of named types, the result will be a tuple where the first value is the name of the type and the second value is the record itself NOTE: Using this option will ignore return_record_name and return_record_name_override return_named_type_override If true, this will modify the behavior of return_named_type so that the named type is only returned for unions where there is more than one named type. For unions that only have one named type, this option will make it so that the named type is returned by itself, not a tuple with the name handle_unicode_errors Default `strict`. Should be set to a valid string that can be used in the errors argument of the string decode() function. Examples include `replace` and `ignore` Example:: parsed_schema = fastavro.parse_schema(schema) with open('file', 'rb') as fp: record = fastavro.schemaless_reader(fp, parsed_schema) Note: The ``schemaless_reader`` can only read a single record. Nr'r(rQ)r,rrr) rr=r>rrrrrr<rZr[s r+schemaless_readerrasr % -C-EM  h0GHM$]M(4KL BG1'B!6.&@ G   r*path_or_bufferct|trt|d}d}n|}d} |jt t }|t k(|r|j SS#|r|j wwxYw)aReturn True if path (or buffer) points to an Avro file. This will only work for avro files that contain the normal avro schema header like those create from :func:`~fastavro._write_py.writer`. This function is not intended to be used with binary data created from :func:`~fastavro._write_py.schemaless_writer` since that does not include the avro header. Parameters ---------- path_or_buffer Path to file rbTF)r4rZopenrrrclose)rbfprfrs r+is_avrorhsj.#& .$ ' U$  HHJ 5 HHJ s &AA4)NNrVrX)wr2rr?rrrrdecimalriorstructrrtypingrr r r r r rrGrio.binary_decoderrio.json_decoderrlogical_readersrrOrrrrrtypesrrr _read_commonrrrrr constr!r"r#decimal_contextutcepoch epoch_naiverZr,r9r7rXr_rcrerhrjrmrorrruryr{r~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrdecompress_rawr ImportErrorrrHr rr lz4.blockr rr!rr4r(r^rYrarhr)r*r+rzs(  ''HHH,,,54+ CL)q!HLL1tQ" (S,%6 7(83/p     "             !         $  &  -`  'T  Nb<  FR9   I  |  i  8    I    Z   k  Z  Z  I  Z  8  Z :  k   [! "{# (  I  |  i  8    I    Z   k  Z  Z  I  Z  8  Z :  k   [! "{# * &  0 fP; ) =) * !    ,0--0M(R 6"6M+/ **M%"+J' T= = @N!'!*N!b|[ %|~Z ;u%Z @'+$(-!)#',S SSF#S S "& S  SS!%SSlE#r'Nt 4 4"-- p  #4 h I"3Hi"H hI 42M!2; !LM+M;,UE:M%;sZ-JKKK J+%K+J?<K>J??KKKKK32K3