g6dZddlZddlZddlmZddlmZmZddlm Z dZ Gdde Z Gd d Z Gd d Z Gd deZy)z9 A reader for corpora whose documents are in MTE format. N)reduce)TaggedCorpusReaderconcat) XMLCorpusViewc&|j||SN)findall)rootpathnss K/var/www/openai/venv/lib/python3.12/site-packages/nltk/corpus/reader/mte.pyxpathr s <<b !!c eZdZdZddZddZy) MTECorpusViewz0 Class for lazy viewing the MTE Corpus. Nc4tj||||yr)r__init__)selffileidtagspec elt_handlers r rzMTECorpusView.__init__stVWkBrc Zttdtj||||S)Nc |duSrxs r z*MTECorpusView.read_block..!4-r)listfilterr read_block)rstreamrrs r r!zMTECorpusView.read_blocks. '((vw L   rr)NN)__name__ __module__ __qualname____doc__rr!rrr rrsC rrceZdZdZdddZdZdZdZdZd Z d Z e d Z e d Z e d Ze dZe dZe dZe dZe dZe dZdZdZdZdZdZdZdZdZdZy) MTEFileReaderz Class for loading the content of the multext-east corpus. It parses the xml files and does some tag-filtering depending on the given method parameters. zhttps://www.tei-c.org/ns/1.0z%https://www.w3.org/XML/1998/namespace)teixmlz{https://www.tei-c.org/ns/1.0}z'{https://www.w3.org/XML/1998/namespace}zTEI/text/body/div/div/p/s/(w|c)zTEI/text/body/div/div/p/szTEI/text/body/div/div/pc||_yr)_MTEFileReader__file_path)r file_paths r rzMTEFileReader.__init__3s $rc|jSr)textclseltcontexts r _word_eltzMTEFileReader._word_elt6s xxrcvt|d|jDcgc]}|j|dc}Scc}wN*)rr r4r1r2r3ws r _sent_eltzMTEFileReader._sent_elt:405c30GH0G1 a&0GHHH6cvt|d|jDcgc]}|j|dc}Scc}wr6)rr r:r1r2r3ss r _para_eltzMTEFileReader._para_elt>r;r<cd|jvr|jdfS|jdk(r*|jdk(r|j|jdfS|jdk(r=|jdk(r.|jtj |jdfSt jdt jdd|jzdz}|j|jdrX|jdk(r|j|jdfS|jtj |jdfSy) Nanamsd universal^-.z.*$) attribr/_MTEFileReader__tags_MTEFileReader__tagsetMTETagConvertermsd_to_universalrecompilesubmatch)r1r2r3tagss r _tagged_word_eltzMTEFileReader._tagged_word_eltBs  "HHb> ! ::   5HHcjj/0 0 ZZ2 #,,+"=HHo>>szz%?PQR R::cBFF3SZZ$@@5HIDzz#**U+,<<5(HHcjj&788'88E9JK rc ttdt|d|jDcgc]}|j |dc}Scc}w)Nc |duSrrrs r rz0MTEFileReader._tagged_sent_elt..\rrr7)rr rr rSr8s r _tagged_sent_eltzMTEFileReader._tagged_sent_eltXN '8=c38OP8O1%%a.8OP   QA c ttdt|d|jDcgc]}|j |dc}Scc}w)Nc |duSrrrs r rz0MTEFileReader._tagged_para_elt..errr7)rr rr rVr>s r _tagged_para_eltzMTEFileReader._tagged_para_eltarWrXcpd|jvr|jdfS|j|jdfS)NlemmarC)rIr/r0s r _lemma_word_eltzMTEFileReader._lemma_word_eltjs4 #** $HHb> !HHcjj12 2rcvt|d|jDcgc]}|j|dc}Scc}wr6)rr r^r8s r _lemma_sent_eltzMTEFileReader._lemma_sent_eltq66;Ccff6MN6M##At,6MNNNr<cvt|d|jDcgc]}|j|dc}Scc}wr6)rr r`r>s r _lemma_para_eltzMTEFileReader._lemma_para_elturar<cht|jtjtjSr)rr,r( word_pathr4rs r wordszMTEFileReader.wordsy(   m55}7N7N  rcht|jtjtjSr)rr,r( sent_pathr:rfs r sentszMTEFileReader.sents~rhrcht|jtjtjSr)rr,r( para_pathr@rfs r paraszMTEFileReader.parasrhrcht|jtjtjSr)rr,r(rer^rfs r lemma_wordszMTEFileReader.lemma_words(   m55}7T7T  rc|t_|t_t|jtj tj Sr)r(rKrJrr,rerSrtagsetrRs r tagged_wordszMTEFileReader.tagged_words8!' #    m55}7U7U  rcht|jtjtjSr)rr,r(rjr`rfs r lemma_sentszMTEFileReader.lemma_sentsrqrc|t_|t_t|jtj tj Sr)r(rKrJrr,rjrVrss r tagged_sentszMTEFileReader.tagged_sentsrvrcht|jtjtjSr)rr,r(rmrcrfs r lemma_paraszMTEFileReader.lemma_parasrqrc|t_|t_t|jtj tj Sr)r(rKrJrr,rmr[rss r tagged_paraszMTEFileReader.tagged_parasrvrN)r#r$r%r&r tag_nsxml_nsrerjrmr classmethodr4r:r@rSrVr[r^r`rcrgrkrnrprurxrzr|r~rrr r(r("s.6 B.F 6F1I+I)I%IIII*    33 OOOO         rr(c >eZdZdZdddddddd d d d d d ZedZy)rLzu Class for converting msd tags to universal tags, more conversion options are currently not implemented. ADJADPADVCONJDETNOUNNUMPRTPRONVERBrHX) ASRCDNMQPVrHrGct|ddk(s|dn|d}|tjvrd}tj|S)z This function converts the annotation from the Multex-East to the universal tagset as described in Chapter 5 of the NLTK-Book Unknown Tags will be mapped to X. Punctuation marks are not supported in MSD tags, so r#rG)rLmapping_msd_universal)tag indicators r rMz MTETagConverter.msd_to_universalsA#&a&C-CFSV OAAAI44Y??rN)r#r$r%r&r staticmethodrMrrr rLrLsK              @ @rrLcfeZdZdZddZdZddZddZddZddZ dd Z dd Z dd Z dd Z dd Zy)MTECorpusReaderz Reader for corpora following the TEI-p5 xml scheme, such as MULTEXT-East. MULTEXT-East contains part-of-speech-tagged words with a quite precise tagging scheme. These tags can be converted to the Universal tagset NcBtj||||d|_y)a. Construct a new MTECorpusreader for a set of documents located at the given root directory. Example usage: >>> root = '/...path to corpus.../' >>> reader = MTECorpusReader(root, 'oana-*.xml', 'utf8') # doctest: +SKIP :param root: The root directory for this corpus. (default points to location in multext config file) :param fileids: A list or regexp specifying the fileids in this corpus. (default is oana-en.xml) :param encoding: The encoding of the given files (default is utf8) z 00README.txtN)rr_readme)rr fileidsencodings r rzMTECorpusReader.__init__s ##D$B% rc| j}nt|tr|g}tfd|}td|}|s t d|S)Nc |jvSr)_fileids)rrs r rz+MTECorpusReader.__fileids..s1 #5rc |dvS)N)z oana-bg.xmlz oana-mk.xmlrrs r rz+MTECorpusReader.__fileids..s 1,J#Jrz$No valid multext-east file specified)r isinstancestrr print)rrs` r __fileidszMTECorpusReader.__fileidssM ?mmG  %iG5w?JGT 8 9rc t|j|Dcgc]C}ttjj |j |jEc}Scc}w)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) )r_MTECorpusReader__fileidsr(osr join_rootrgrrfs r rgzMTECorpusReader.wordss] 0 0Abggll4::q9:@@B0    AA(c t|j|Dcgc]C}ttjj |j |jEc}Scc}w)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings :rtype: list(list(str)) )rrr(rr rrrkrs r rkzMTECorpusReader.sents]0 0Abggll4::q9:@@B0    rc t|j|Dcgc]C}ttjj |j |jEc}Scc}w)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word string :rtype: list(list(list(str))) )rrr(rr rrrnrs r rnzMTECorpusReader.parasrrc t|j|Dcgc]C}ttjj |j |jEc}Scc}w)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words, the corresponding lemmas and punctuation symbols, encoded as tuples (word, lemma) :rtype: list(tuple(str,str)) )rrr(rr rrrprs r rpzMTECorpusReader.lemma_wordss]0 0Abggll4::q9:FFH0    rc  |dk(s|dk(rit|j|Dcgc]E}ttjj |j |j||Gc}Stdycc}w)a; :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of tagged words and punctuation symbols encoded as tuples (word, tag) :rtype: list(tuple(str, str)) rErDUnknown tagset specified.N) rrr(rr rrrurrrrtrRrs r ruzMTECorpusReader.tagged_words, [ FeO "^^G4 5""'',,tzz1"=>KK5   - .A Bc t|j|Dcgc]C}ttjj |j |jEc}Scc}w)aB :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(list(tuple(str, str))) )rrr(rr rrrxrs r rxzMTECorpusReader.lemma_sentsC]0 0Abggll4::q9:FFH0    rc  |dk(s|dk(rit|j|Dcgc]E}ttjj |j |j||Gc}Stdycc}w)aH :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of sentences or utterances, each each encoded as a list of (word,tag) tuples :rtype: list(list(tuple(str, str))) rErDrN) rrr(rr rrrzrrs r rzzMTECorpusReader.tagged_sentsRrrc t|j|Dcgc]C}ttjj |j |jEc}Scc}w)am :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(List(List(tuple(str, str)))) )rrr(rr rrr|rs r r|zMTECorpusReader.lemma_parasirrc  |dk(s|dk(rit|j|Dcgc]E}ttjj |j |j||Gc}Stdycc}w)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of (word,tag) tuples :rtype: list(list(list(tuple(str, str)))) rErDrN) rrr(rr rrr~rrs r r~zMTECorpusReader.tagged_parasxs [ FeO "^^G4 5""'',,tzz1"=>KK5   - .r)NNutf8r)NrDrC)r#r$r%r&rrrgrkrnrprurxrzr|r~rrr rrs> &         /.  /.  /rr)r&rrN functoolsrnltk.corpus.readerrrnltk.corpus.reader.xmldocsrrrr(rLrrrr rsU 94" M "H H V"@"@J|/(|/r