g>dZddlZddlmZddlmZddlddlmZddl m Z ddl m Z Gd d eZ Gd d eZy) z Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) N) ElementTree) CorpusReader)*)SeekableUnicodeStreamReader)ElementWrapper)WordPunctTokenizerc(eZdZdZddZddZddZy)XMLCorpusReadera Corpus reader for corpora whose documents are xml files. Note that the ``XMLCorpusReader`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. c@||_tj|||yN) _wrap_etreer__init__)selfrootfileids wrap_etrees O/var/www/openai/venv/lib/python3.12/site-packages/nltk/corpus/reader/xmldocs.pyrzXMLCorpusReader.__init__!s%dD'2Ncj|'t|jdk(r|jd}t|ts t d|j |j 5}tj|j}ddd|jr t}S#1swY"xYw)Nrz(Expected a single file identifier string) len_fileids isinstancestr TypeErrorabspathopenrparsegetrootr r)rfileidfpelts rxmlzXMLCorpusReader.xml%s >c$--0A5]]1%F&#&FG G \\& ! & & (B##B'//1C)    %C ) (s $$B))B2cd|j|}|j|}t} |j}g}|D]T}|j }|t |tr|j|}|j|} |j| V|S#|j }YqxYw)aE Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. :return: the given file's text nodes as a list of words and punctuation symbols :rtype: list(str) ) r#encodingr getiteratoritertextrbytesdecodetokenizeextend) rr r"r%word_tokenizeriteratoroutnoder(tokss rwordszXMLCorpusReader.words4shhv==(+- "(HD99DdE*;;x0D%..t4 4   "xxzHs BB/)Fr )__name__ __module__ __qualname____doc__rr#r2rrr r s3 rr ceZdZdZdZdZd dZdZdZe jde je jzZ e jd Ze jd e je jzZd Zdd Zy) XMLCorpusViewam A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: ``XMLCorpusView`` is not used by ``XMLCorpusReader`` itself, but may be used by subclasses of ``XMLCorpusReader``.) Every XML corpus view has a "tag specification", indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - ``'foo'``: A top-level element whose tag is ``foo``. - ``'foo/bar'``: An element whose tag is ``bar`` and whose parent is a top-level element whose tag is ``foo``. - ``'.*/foo'``: An element whose tag is ``foo``, appearing anywhere in the xml tree. - ``'.*/(foo|bar)'``: An wlement whose tag is ``foo`` or ``bar``, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method ``handle_elt()``. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the ``elt_handler`` constructor parameter. FiNc|r||_tj|dz|_ ddi|_ |j |}t j|||y)aW Create a new corpus view based on a specified XML file. Note that the ``XMLCorpusView`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. :type tagspec: str :param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. :param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then ``self.handle_elt()`` is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value z\Zrr7)r%N) handle_eltrecompile_tagspec _tag_context_detect_encodingStreamBackedCorpusViewr)rr tagspec elt_handlerr%s rrzXMLCorpusView.__init__us]* )DO 7U?3 9G  ((0''fx'Hrc2t|tr2 |j}|j}|j n%t|d5}|j}dddj t jry|j t jry|j t jry|j t jry|j t jrytjd|}|r|jdjStjd |}|r|jdjSy#j wxYw#1swY.xYw) Nrbz utf-16-bez utf-16-lez utf-32-bez utf-32-lezutf-8s!\s*<\?xml\b.*\bencoding="([^"]+)"rs!\s*<\?xml\b.*\bencoding='([^']+)')r PathPointerrreadlineclose startswithcodecs BOM_UTF16_BE BOM_UTF16_LE BOM_UTF32_BE BOM_UTF32_LEBOM_UTF8r<matchgroupr*)rr infilesms rr@zXMLCorpusView._detect_encodings" fk * OO% fd#vOO%$ <<++ , <<++ , <<++ , <<++ , << ( HH:A > 771:$$& & HH:A > 771:$$& &+ ##s E7F 7F  Fc|S)a Convert an element into an appropriate value for inclusion in the view. Unless overridden by a subclass or by the ``elt_handler`` constructor argument, this method simply returns ``elt``. :return: The view value corresponding to ``elt``. :type elt: ElementTree :param elt: The element that should be converted. :type context: str :param context: A string composed of element tags separated by forward slashes, indicating the XML context of the given element. For example, the string ``'foo/bar/baz'`` indicates that the element is a ``baz`` element whose parent is a ``bar`` element and whose grandparent is a top-level ``foo`` element. r7)rr"contexts rr;zXMLCorpusView.handle_elts ( ra; [^<]* ( (() | # comment () | # doctype decl (<[^!>][^>]*>)) # tag or PI [^<]*)* \Zz<\s*(?:/\s*)?([^\s>]+)a6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P ]*(\[[^\]]*])?\s*>)| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )cd}t|tr|j} |j|j}||z }|j j |r|Stjd|jddk(rO|jt|tjd|jz z }td|z|s td|jd}|dkDru|j j |d|rWt|tr#|j|j|n|jt||z d |d|SN) a{ Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size ``self._BLOCK_SIZE``. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. z[<>]r>zUnexpected ">" near char %sz&Unexpected end of file: tag not closed!DEE !IJJ !)s 3  1$%%++H5G6G,HI!&*EF H-001BC c(m6G&G$H!L#$6%677?rc | |j}| |j}t|jj |j }|Jg}d}d}d}|gk(s|ft |tr|j } |j|} | s|n/td|jj| D]q} |jr;tdjdj|dd| j!| j!dr|j"j%| j!j!d} |j'| |t(j%|dj|s| j+}t-|}| j!d r|j"j%| j!j!d} |std | z| |d k7rtd |d d | d|L|t-|k(r>|| || j/z }|j'|dj|fdx}}d}|j1| j!ds|j"j%| j!j!d} |t(j%|dj|dz| zs<|j'| j!dj|dz| zft||gk(r || |dz }d}nw|jr tdt |tr#|j3 |j5|n|j3t-| |z d|d|dz }dx}}d}|gk(rb|f|j } | |jvrt7||j| k(sJt7||j| <|Dcgc]1\}}|t9j:|j=dd|3c}}Scc}}w)z Read from ``stream`` until we find at least one element that matches ``tagspec``, and return the result of applying ``elt_handler`` to each element found. NrXzUnexpected end of filez {:>25} {}/i START_TAGrEND_TAGzUnmatched tag zUnmatched tag ...r;listr?getr[rrrkra _XML_PIECEfinditer_DEBUGprintformatjoinrQ _XML_TAG_NAMErPappendr<startrr`poprcrdtupler fromstringencode)rrerBrCrVelts elt_start elt_depthelt_textrg xml_fragmentpiecenamerir"s r read_blockzXMLCorpusView.read_block"s ?mmG  //Kt((,,V[[];<"""  bjI1&"=>!;;=226:L $$%=>>11,?;;+,,SXXg->st-DekkmTU;;{+--33EKKMBHHKDNN4( (88GSXXg->?(- I(+G I[[+--33EKKMBHHKD"()>)EFFwr{*(?72;-vdVST)UVV ,c'l1J LUYY[$II Xsxx/@$AB044 I#%KKM[[1--33EKKMBHHKD (88GSXXg->-Dt-KL KK8IC8ORV8V(WXE@H$2: YZ 88H !I{{67!&*EF H-00; c,&7)&C$DaH%o A 6G,00I !HObjI1Tkkm $## #>T%6%6s%;; ;;%*7^D  c "#'  #'g &&szz';N'OP #'    s!6Qr )NN)r3r4r5r6rxr]rr@r;r<r=DOTALLVERBOSEr^r|rvrkrr7rrr9r9Qs<FK"IH:0JJ  BJJ MJJ89M  E BJJ J,8bk rr9)r6rJ xml.etreernltk.corpus.reader.apirnltk.corpus.reader.util nltk.datarnltk.internalsr nltk.tokenizerr rAr9r7rrrs= !/%1),6l6r| *| r