gddlZddlZddlmZddlmZGddZGddZGdd eZGd d eZ Gd d eZ GddeZ GddeZ GddeZ GddeZGddeZej dj#ddZdZGddeZGddeZd Zd!Zed"k(reyy)#N) ChunkParserITreeceZdZdZdZdezZdZdZdezZdezZ e jdezZ e jd Z e jd Zdd Zd Zd ZddZdZdZdZy) ChunkStringaL A string-based encoding of a particular chunking of a text. Internally, the ``ChunkString`` class uses a single string to encode the chunking of the input text. This string contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is:: {
}{
}<.>{
}<.> ``ChunkString`` are created from tagged texts (i.e., lists of ``tokens`` whose type is ``TaggedType``). Initially, nothing is chunked. The chunking of a ``ChunkString`` can be modified with the ``xform()`` method, which uses a regular expression to transform the string representation. These transformations should only add and remove braces; they should *not* modify the sequence of angle-bracket delimited tags. :type _str: str :ivar _str: The internal string representation of the text's encoding. This string representation contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is:: {
}{
}<.>{
}<.> :type _pieces: list(tagged tokens and chunks) :ivar _pieces: The tagged tokens and chunks encoded by this ``ChunkString``. :ivar _debug: The debug level. See the constructor docs. :cvar IN_CHUNK_PATTERN: A zero-width regexp pattern string that will only match positions that are in chunks. :cvar IN_STRIP_PATTERN: A zero-width regexp pattern string that will only match positions that are in strips. z [^\{\}<>]z(<%s+?>)z (?=[^\{]*\})z(?=[^\}]*(\{|$))z (\{%s+?\})+?z(%s+?)+?z^(\{?%s\}?)*?$z[^\{\}]+z(\{\})*$c|j|_|dd|_|jDcgc]}|j|}}ddj |zdz|_||_ycc}w)a} Construct a new ``ChunkString`` that encodes the chunking of the text ``tagged_tokens``. :type chunk_struct: Tree :param chunk_struct: The chunk structure to be further chunked. :type debug_level: int :param debug_level: The level of debugging which should be applied to transformations on the ``ChunkString``. The valid levels are: - 0: no checks - 1: full check on to_chunkstruct - 2: full check on to_chunkstruct and cursory check after each transformation. - 3: full check on to_chunkstruct and full check after each transformation. We recommend you use at least level 1. You should probably use level 3 if you use any non-standard subclasses of ``RegexpChunkRule``. N<>)label _root_label_pieces_tagjoin_str_debug)self chunk_struct debug_leveltoktagss F/var/www/openai/venv/lib/python3.12/site-packages/nltk/chunk/regexp.py__init__zChunkString.__init__Hse.(--/#A *.,,7,3 #,7$))D/)C/ ! 8sA,ct|tr|dSt|tr|jSt d)Nz4chunk structures must contain tagged tokens or trees) isinstancetuplerr ValueError)rrs rrzChunkString._tages7 c5 !q6M T "99; VW Wc tjj|std|ztjj d|}t dt|dzzD]=}||dz|dzdz}tjj|r1td|z|dkrytjd|dd}|jDcgc]}|j|}}||k7r td ycc}w) a Check to make sure that ``s`` still corresponds to some chunked version of ``_pieces``. :type verify_tags: bool :param verify_tags: Whether the individual tags should be checked. If this is false, ``_verify`` will check to make sure that ``_str`` encodes a chunked version of *some* list of tokens. If this is true, then ``_verify`` will check to make sure that the tags in ``_str`` match those in ``_pieces``. :raise ValueError: if the internal string representation of this ``ChunkString`` is invalid or not consistent with _pieces. z2Transformation generated invalid chunkstring: %srirNz [\{\}<>]+z9Transformation generated invalid chunkstring: tag changed) r_VALIDmatchr _BRACKETSsubrangelen_BALANCED_BRACKETSresplitr r) rs verify_tagsbracketsisubstrtags1piecetags2s r_verifyzChunkString._verifyms "!!''*H1L ((,,R3q3x=D001Aa$hTD9F1177? LqP2 !  ,*Ab1/3||<|e5!|< E>N  =sDcv|jdkDr|j|jdg}d}d}tjd|jD]P}|j d}|j |||z}|r|jt||n||z }||z }| }Rt|j|S)z Return the chunk structure encoded by this ``ChunkString``. :rtype: Tree :raise ValueError: If a transformation has generated an invalid chunkstring. rr[{}]r ) rr3rr)r*countr appendrr )r chunk_labelpiecesindexpiece_in_chunkr1length subsequences rto_chunkstructzChunkString.to_chunkstructs ;;? LLA &XXfdii0E[[%F,,uuv~>K d; <=+% VOE!//N1D$$f--rctj|||j}tjdd|}|jdkDr|j ||jdz ||_y)aH Apply the given transformation to the string encoding of this ``ChunkString``. In particular, find all occurrences that match ``regexp``, and replace them using ``repl`` (as done by ``re.sub``). This transformation should only add and remove braces; it should *not* modify the sequence of angle-bracket delimited tags. Furthermore, this transformation may not result in improper bracketing. Note, in particular, that bracketing may not be nested. :type regexp: str or regexp :param regexp: A regular expression matching the substring that should be replaced. This will typically include a named group, which can be used by ``repl``. :type repl: str :param repl: An expression specifying what should replace the matched substring. Typically, this will include a named replacement group, specified by ``regexp``. :rtype: None :raise ValueError: If this transformation generated an invalid chunkstring. z\{\}r rN)r)r%rrr3)rregexpreplr+s rxformzChunkString.xformsZ4 FF64 + FF7B " ;;? LLDKK!O , rc2dt|jzS)z Return a string representation of this ``ChunkString``. It has the form:: }{
}'> :rtype: str z)reprrrs r__repr__zChunkString.__repr__s#T$))_44rctjdd|j}tjdd|}|ddk(rd|z}|S)a) Return a formatted representation of this ``ChunkString``. This representation will include extra spaces to ensure that tags will line up with the representation of other ``ChunkStrings`` for the same text, regardless of the chunking. :rtype: str z>(?!\})z> z([^\{])rCrGrKrrrrs#J"Nn,I&*y (F 9 $F RZZ)I5 6F ;'I#K0":X)V.@&P 5rrc8eZdZdZdZdZdZdZedZ y)RegexpChunkRulea A rule specifying how to modify the chunking in a ``ChunkString``, using a transformational regular expression. The ``RegexpChunkRule`` class itself can be used to implement any transformational rule based on regular expressions. There are also a number of subclasses, which can be used to implement simpler types of rules, based on matching regular expressions. Each ``RegexpChunkRule`` has a regular expression and a replacement expression. When a ``RegexpChunkRule`` is "applied" to a ``ChunkString``, it searches the ``ChunkString`` for any substring that matches the regular expression, and replaces it using the replacement expression. This search/replace operation has the same semantics as ``re.sub``. Each ``RegexpChunkRule`` also has a description string, which gives a short (typically less than 75 characters) description of the purpose of the rule. This transformation defined by this ``RegexpChunkRule`` should only add and remove braces; it should *not* modify the sequence of angle-bracket delimited tags. Furthermore, this transformation may not result in nested or mismatched bracketing. cxt|trtj|}||_||_||_y)a Construct a new RegexpChunkRule. :type regexp: regexp or str :param regexp: The regular expression for this ``RegexpChunkRule``. When this rule is applied to a ``ChunkString``, any substring that matches ``regexp`` will be replaced using the replacement string ``repl``. Note that this must be a normal regular expression, not a tag pattern. :type repl: str :param repl: The replacement expression for this ``RegexpChunkRule``. When this rule is applied to a ``ChunkString``, any substring that matches ``regexp`` will be replaced using ``repl``. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. N)rrJr)rW_repl_descr_regexp)rrArBdescrs rrzRegexpChunkRule.__init__s1$ fc "ZZ'F   rcP|j|j|jy)a Apply this rule to the given ``ChunkString``. See the class reference documentation for a description of what it means to apply a rule. :type chunkstr: ChunkString :param chunkstr: The chunkstring to which this rule is applied. :rtype: None :raise ValueError: If this transformation generated an invalid chunkstring. N)rCr^r\)rchunkstrs rapplyzRegexpChunkRule.apply3s t||TZZ0rc|jS)zt Return a short description of the purpose and/or effect of this rule. :rtype: str )r]rFs rr_zRegexpChunkRule.descrBs{{rc~dt|jjzdzt|jzdzS)a@ Return a string representation of this rule. It has the form:: }'->''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str zr )rEr^patternr\rFs rrGzRegexpChunkRule.__repr__KsH !4<<''( ) 4::    rctjd|}|jdj}|jdxsdddj} |s t d|dd k(r|d d k(rt |dd |S|dd k(r|d d k(rt |dd |Sd |vr!|jd \}}t|||Sd |vr!|jd \}}t|||Stjd|r(tjd|\}}}t||||St d|z#ttjf$r}t d|z|d}~wwxYw)a^ Create a RegexpChunkRule from a string description. Currently, the following formats are supported:: {regexp} # chunk rule }regexp{ # strip rule regexp}{regexp # split rule regexp{}regexp # merge rule Where ``regexp`` is a regular expression for the rule. Any text following the comment marker (``#``) will be used as the rule's description: >>> from nltk.chunk.regexp import RegexpChunkRule >>> RegexpChunkRule.fromstring('{
?+}') ?+'> z&(?P(\\.|[^#])*)(?P#.*)?rulecommentr rNzEmpty chunk patternr{r!}z}{z{}z[^{}]*{[^{}]*}[^{}]*r5zIllegal chunk pattern: %s) r)r#groupstripr ChunkRule StripRuler* SplitRule MergeRuleChunkRuleWithContexterror)r+mrgrhleftrightchunkes r fromstringzRegexpChunkRule.fromstring_sy( HH> Bwwv$$&779%+QR0668 H !677Aw#~$r(c/ aW55aCDHO aW55"jj. e ug66"jj. e ug660$7%'XXfd%;"eU+D%HH ! D>($D> $D>2=D>0D>>E+E&&E+N) rMrNrOrPrrbr_rG staticmethodrxrXrrrZrZs120 1 (+H+HrrZceZdZdZdZdZy)rma) A rule specifying how to add chunks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any substring that matches this tag pattern and that is not already part of a chunk, and create a new chunk containing that substring. c||_tjdt|dtj }t j||d|y)a Construct a new ``ChunkRule``. :type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will chunk any substring that matches this tag pattern and that is not already part of a chunk. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. z (?P)z {\g}N)_patternr)rWtag_pattern2re_patternrrTrZrr tag_patternr_rAs rrzChunkRule.__init__sE$ %k2K4P4P R    v~uErc8dt|jzdzS)a0 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str z r|z }\g{N)r}r)rWr~rrSrZrrs rrzStripRule.__init__sE$ %k2K4P4P R    v~uErc8dt|jzdzS)a0 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str z %s)\}z \gN)r}r)rWr~rZrrs rrzUnChunkRule.__init__s9$ 03I+3VVW  v|UCrc8dt|jzdzS)a2 Return a string representation of this rule. It has the form:: '> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str zz)}{(?=r|z\gNr)rWr~_left_tag_pattern_right_tag_patternrZrrleft_tag_patternright_tag_patternr_rAs rrzMergeRule.__init__sw2 )*:;< )*;<=!1"3''78&'89     v{EBrcjdt|jzdzt|jzdzS)a< Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str z }{Nrrs rrzSplitRule.__init___sw0 )*:;< )*;<=!1"3''78&'89     v}eDrcjdt|jzdzt|jzdzS)a3 Return a string representation of this rule. It has the form:: ', '
'> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str z r|z{\g\gNrrs rrzExpandLeftRule.__init__x2 )*:;< )*;<=!1"3''78&'89     v/DeLrcjdt|jzdzt|jzdzS)aA Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str zr|z\g\g}Nrrs rrzExpandRightRule.__init__rrcjdt|jzdzt|jzdzS)aB Return a string representation of this rule. It has the form:: ', ''> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str zz )(?Pr|z\g{\g}\gN) r)rWr~_left_context_tag_pattern_chunk_tag_pattern_right_context_tag_patternrrTrZr)rleft_context_tag_patternchunk_tag_patternright_context_tag_patternr_rA replacements rrzChunkRuleWithContext.__init__6s8 )*BCD )*;<= )*CDE)A&"3*C'''?@&'89&'@A,,   6   v{EBrcddj|j|j|jS)aF Return a string representation of this rule. It has the form:: ', '', '
'> Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method. :rtype: str z))formatrrrrFs rrGzChunkRuleWithContext.__repr__es2;AA  * *  # #  + +  rNrrXrrrqrq&s -C^ rrqz^(({}|<{}>)*)$z"([^\{\}<>]|\{\d+,?\}|\{\d*,\d+\})+z [^\{\}<>]+cVtjdd|}tjdd|}tjdd|}tj|st d|zd}|t j }||}tjd ||}||}|S) a Convert a tag pattern to a regular expression pattern. A "tag pattern" is a modified version of a regular expression, designed for matching sequences of tags. The differences between regular expression patterns and tag patterns are: - In tag patterns, ``'<'`` and ``'>'`` act as parentheses; so ``'+'`` matches one or more repetitions of ``''``, not ``''``. - Whitespace in tag patterns is ignored. So ``'
| '`` is equivalent to ``'
|'`` - In tag patterns, ``'.'`` is equivalent to ``'[^{}<>]'``; so ``''`` matches any single tag starting with ``'NN'``. In particular, ``tag_pattern2re_pattern`` performs the following transformations on the given pattern: - Replace '.' with '[^<>{}]' - Remove any whitespace - Add extra parens around '<' and '>', to make '<' and '>' act like parentheses. E.g., so that in '+', the '+' has scope over the entire ''; and so that in '', the '|' has scope over 'NN' and 'IN', but not '<' or '>'. - Check to make sure the resulting pattern is valid. :type tag_pattern: str :param tag_pattern: The tag pattern to convert to a regular expression pattern. :raise ValueError: If ``tag_pattern`` is not a valid tag pattern. In particular, ``tag_pattern`` should not include braces; and it should not contain nested or mismatched angle-brackets. :rtype: str :return: A regular expression pattern corresponding to ``tag_pattern``. z\sr r z(<(r z)>)zBad tag pattern: %rcZt|}|jdj|S)Nr )listreverser)rJlsts r reverse_strz+tag_pattern2re_pattern..reverse_strs"3i wws|rz\.(?!\\(\\\\)*($|[^\\])))r)r%CHUNK_TAG_PATTERNr#rrrQ)rrtc_revreverseds rr~r~sJ&&K0K&&uk2K&&uk2K  " "; /.<== 33 4F;'Hvv168DHh'K rc>eZdZdZd dZdZdZd dZdZdZ d Z y) RegexpChunkParseran A regular expression based chunk parser. ``RegexpChunkParser`` uses a sequence of "rules" to find chunks of a single type within a text. The chunking of the text is encoded using a ``ChunkString``, and each rule acts by modifying the chunking in the ``ChunkString``. The rules are all implemented using regular expression matching and substitution. The ``RegexpChunkRule`` class and its subclasses (``ChunkRule``, ``StripRule``, ``UnChunkRule``, ``MergeRule``, and ``SplitRule``) define the rules that are used by ``RegexpChunkParser``. Each rule defines an ``apply()`` method, which modifies the chunking encoded by a given ``ChunkString``. :type _rules: list(RegexpChunkRule) :ivar _rules: The list of rules that should be applied to a text. :type _trace: int :ivar _trace: The default level of tracing. c<||_||_||_||_y)a Construct a new ``RegexpChunkParser``. :type rules: list(RegexpChunkRule) :param rules: The sequence of rules that should be used to generate the chunking for a tagged text. :type chunk_label: str :param chunk_label: The node value that should be used for chunk subtrees. This is typically a short string describing the type of information contained by the chunk, such as ``"NP"`` for base noun phrases. :type root_label: str :param root_label: The node value that should be used for the top node of the chunk structure. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. N)_rules_trace _chunk_labelr )rrulesr8 root_labeltraces rrzRegexpChunkParser.__init__s"*  '%rc"tdt||jD]j}|j||r-td|jdzt |zdzntd|jdzt|ly)a Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn. Generate trace output between each rule. If ``verbose`` is true, then generate verbose output. :type chunkstr: ChunkString :param chunkstr: The chunk string to which each rule should be applied. :type verbose: bool :param verbose: Whether output should be verbose. :rtype: None z# Input:#z (z)::N)printrrbr_rE)rraverbosergs r _trace_applyzRegexpChunkParser._trace_applyso j hKKD JJx c4::<$.d;dBCc4::<#-. (O  rcH|jD]}|j|y)z Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn. :param chunkstr: The chunk string to which each rule should be applied. :type chunkstr: ChunkString :rtype: None N)rrb)rrargs r_notrace_applyz RegexpChunkParser._notrace_applysKKD JJx  rNct|dk(r!tdt|jgS |j | |j }t|}|r|dkD}|j||n|j||j|jS#t $rt|j|}YwxYw)aP :type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :rtype: Tree :return: a chunk structure that encodes the chunks in a given tagged sentence. A chunk is a non-overlapping linguistic group, such as a noun phrase. The set of chunks identified in the chunk structure depends on the rules used to define this ``RegexpChunkParser``. rzWarning: parsing empty textr) r'rrr r AttributeErrorrrrrr>r)rrrrars rparsezRegexpChunkParser.parses$ |  ! / 0(("- - @    =KKE|, aiG   h 0    )&&t'8'899# @ 0 0,?L @sB CCc|jS)zu :return: the sequence of rules used by ``RegexpChunkParser``. :rtype: list(RegexpChunkRule) )rrFs rrzRegexpChunkParser.rulesHs {{rc2dt|jzS)zy :return: a concise string representation of this ``RegexpChunkParser``. :rtype: str z!)r'rrFs rrGzRegexpChunkParser.__repr__Os 3S5EEErcFdt|jz}d}|jD]%}t|t|j}'|dkrdt |dz zdz}nd}|jD]#}|||jt |fzz }%|dd S) zm :return: a verbose string representation of this ``RegexpChunkParser``. :rtype: str z!RegexpChunkParser with %d rules: r#z %zs%s z %s %s Nr!)r'rmaxr_rE)rr+marginrgrs rrKzRegexpChunkParser.__str__Ws 13t{{3C CKKDTZZ\!23F B;tfqjM22W} # chunk determiners and adjectives }<[\.VI].*>+{ # strip any tag beginning with V, I, or . <.*>}{
# split a chunk at a determiner {} # merge chunk ending with det/adj # with one starting with a noun The patterns of a clause are executed in order. An earlier pattern may introduce a chunk boundary that prevents a later pattern from executing. Sometimes an individual pattern will match on multiple, overlapping extents of the input. As with regular expression substitution more generally, the chunker will identify the first match possible, then continue looking for matches after this one has ended. The clauses of a grammar are also executed in order. A cascaded chunk parser is one having more than one clause. The maximum depth of a parse tree created by this chunk parser is the same as the number of clauses in the grammar. When tracing is turned on, the comment portion of a line is displayed each time the corresponding pattern is applied. :type _start: str :ivar _start: The start symbol of the grammar (the root node of resulting trees) :type _stages: int :ivar _stages: The list of parsing stages corresponding to the grammar c,||_g|_||_||_t |t r|j |||yd} t|}|D]}t |trt|||_y#t$r}t||d}~wwxYw)a Create a new chunk parser, from the given start state and set of chunk patterns. :param grammar: The grammar, or a list of RegexpChunkParser objects :type grammar: str or list(RegexpChunkParser) :param root_label: The top node of the tree being created :type root_label: str or Nonterminal :param loop: The number of times to run through the patterns :type loop: int :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. z>Expected string or list of RegexpChunkParsers for the grammar.N) r_stages_grammar_looprrJ _read_grammarr BaseException TypeErrorr)rgrammarrlooprtype_errrwelts rrzRegexpParser.__init__s"    gs #   w E :T  1w-!#'89#H--#DL ! 1)q0 1s A99 B BBcg}d}tjd}|jdD]}|j}|j |}|rT|j |||||j dj}g}|j dj}|dk(s|jdr|jtj||j ||||y)z[ Helper function for __init__: read the grammar if it is a string. Nz*(?P(\.|[^:])*)(:(?P.*)) nonterminalrgr r) regexrWr*rlr# _add_stagerk startswithr7rZrx) rrrrrlhsrelinerss rrzRegexpParser._read_grammars -- MNMM$'D::ggm,224wwv,,.rzT__S1 LL33D9 :%(* sJ6rc~|gk7r8|s tdt||||}|jj|yy)zN Helper function for __init__: add a new stage to the parser. zExpected stage marker (eg NP:))r8rrN)rrrr7)rrrrrparsers rrzRegexpParser._add_stagesF B; !ABB&3:UF LL   ' rNc| |j}t|jD]&}|jD]}|j ||}(|S)a Apply the chunk parser to this input. :type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked (this tree is modified, and is also returned) :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :return: the chunked output. :rtype: Tree r)rr&rrr)rrrr.rs rrzRegexpParser.parsesM" =KKEtzz"A,,%||L|F '#rc2dt|jzS)zn :return: a concise string representation of this ``chunk.RegexpParser``. :rtype: str z#r'rrFs rrGzRegexpParser.__repr__s 5s4<<7HHHrcrdt|jz}d}|jD] }|d|zz } |ddS)zt :return: a verbose string representation of this ``RegexpParser``. :rtype: str z#chunk.RegexpParser with %d stages: rz%s Nr!r)rr+rrs rrKzRegexpParser.__str__sC 3S5F FllF & A#"v r)rrrr) rMrNrOrPrrrrrGrKrXrrrrns('R$#L7> (0I rrc ddlm}ddlm}|j }|j dD]w}t ||j}|s!|j|}|j}|j|d|d}|j||t yt dt d |t d t d |jd zzd dt d|jd zzddt d|jd zz|jrut d|j} | ddD]+}t ddj!t#t$|-t'|jdkDr t d|j)rut d|j)} | ddD]+}t ddj!t#t$|-t'|j)dkDr t dt dt y)a Demonstration code for evaluating a chunk parser, using a ``ChunkScore``. This function assumes that ``text`` contains one sentence per line, and that each sentence has the form expected by ``tree.chunk``. It runs the given chunk parser on each sentence in the text, and scores the result. It prints the final score (precision, recall, and f-measure); and reports the set of chunks that were missed and the set of chunks that were incorrect. (At most 10 missing chunks and 10 incorrect chunks are reported). :param chunkparser: The chunkparser to be tested :type chunkparser: ChunkParserI :param text: The chunked tagged text that should be used for evaluation. :type text: str r)rvrrrrrzM/===========================================================================\ScoringzM-----------------------------------------------------------------------------zPrecision: %5.1f%%dz rI)endzRecall: %5.1f%%z zF-Measure: %5.1f%%zMissed:N z z ...z Incorrect:zM\===========================================================================/)nltkrv nltk.treer ChunkScorer*rrl tagstr2treeleavesrscore precisionrecall f_measuremissedrmaprJr' incorrect) chunkparsertextrvr chunkscoresentencegoldtokenstestrrs r demo_evalr s"!!#JJJt$ h>>#   *  c6!2! <t$ % !" )[! (O *"6"6"83"> ?cR z002S8 97L *"6"6"83"> ?@ i""$CR[E $S%1 2! z  " #b ( 'N l((* s^E $S%1 2$ z##% & + 'N !" Grc ^ddlm}m}d}tdtdt|tdtd}|j |}t ||d}|j |}t ||d}|j |}t ||d }|j |}t ||dd lm}ttd |j d }t|j||jd dttdd}|j |}t|j||jd ddttdd }|j |}t|jgdy)z A demonstration for the ``RegexpChunkParser`` class. A single text is parsed with four different chunk parsers, using a variety of rules and strategies. r)rrvz [ the/DT little/JJ cat/NN ] sat/VBD on/IN [ the/DT mat/NN ] ./. [ John/NNP ] saw/VBD [the/DT cats/NNS] [the/DT dog/NN] chased/VBD ./. [ John/NNP ] thinks/VBZ [ Mary/NN ] saw/VBD [ the/DT cat/NN ] sit/VB on/IN [ the/DT mat/NN ]./. zK***************************************************************************zEvaluation text:z NP: # NP stage {
?*} # chunk determiners, adjectives and nouns {+} # chunk proper nouns z NP: {<.*>} # start by chunking each tag }<[\.VI].*>+{ # unchunk any verbs, prepositions or periods {} # merge det/adj with nouns zy NP: {
?*} # chunk determiners, adjectives and nouns VP: {?} # VP = verb words a6 NP: {<.*>*} # start by chunking everything }<[\.VI].*>+{ # strip any verbs, prepositions or periods <.*>}{
# separate on determiners PP: {} # PP = preposition + noun phrase VP: {*} # VP = verb words + NPs and PPs ) conll2000zDemonstration of empty grammar:r ztest.txt)r) chunk_typesz6Demonstration of accuracy evaluation using CoNLL tags:Nz#Demonstration of tagged token input)theDT)littleJJ)catNN)satVBD)onINr)matr).r) rrrvrrr  nltk.corpusraccuracy chunked_sentsr)rrvrrcprs rdemor"Ys! D  (O  $K (O GG   G $B b$G   G $B b$G   G $B b$G   G $B b$& G +,   B B %..Y44ZW4U VW G BCG   G $B %..Y44Z@!D EF G /0G   G $B     r__main__)r)rnltk.chunk.apirrrrrZrmrnrrprorrrqrWrrr~rrr r"rMrXrrr%s  'ddXJHJHZ):):X*:*:Z%