g~dZddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZeGd d Zd ZGd d e eZdZdZeGddZGddeZGddeeZGddZGddeZeGddZGddZGddZGd d!eZ d"Z!d#Z"d$Z#d%Z$ejJd&ejLZ'ejJd'ejLZ(ejJd(ejLZ)ejJd)ejLZ*d9d*Z+d:d+Z,ejJd,ejLZ-d-Z.ejJd.ejLZ/ejJd/Z0d0Z1d1Z2d2Z3d3Z4d4Z5d5Z6d6Z7e8d7k(re7gd8Z9y);a Basic data classes for representing context free grammars. A "grammar" specifies which trees can represent the structure of a given text. Each of these trees is called a "parse tree" for the text (or simply a "parse"). In a "context free" grammar, the set of parse trees for any piece of a text can depend only on that piece, and not on the rest of the text (i.e., the piece's context). Context free grammars are often used to find possible syntactic structures for sentences. In this context, the leaves of a parse tree are word tokens; and the node values are phrasal categories, such as ``NP`` and ``VP``. The ``CFG`` class is used to encode context free grammars. Each ``CFG`` consists of a start symbol and a set of productions. The "start symbol" specifies the root node value for parse trees. For example, the start symbol for syntactic parsing is usually ``S``. Start symbols are encoded using the ``Nonterminal`` class, which is discussed below. A Grammar's "productions" specify what parent-child relationships a parse tree can contain. Each production specifies that a particular node can be the parent of a particular set of children. For example, the production `` -> `` specifies that an ``S`` node can be the parent of an ``NP`` node and a ``VP`` node. Grammar productions are implemented by the ``Production`` class. Each ``Production`` consists of a left hand side and a right hand side. The "left hand side" is a ``Nonterminal`` that specifies the node type for a potential parent; and the "right hand side" is a list that specifies allowable children for that parent. This lists consists of ``Nonterminals`` and text types: each ``Nonterminal`` indicates that the corresponding child may be a ``TreeToken`` with the specified node type; and each text type indicates that the corresponding child may be a ``Token`` with the with that type. The ``Nonterminal`` class is used to distinguish node values from leaf values. This prevents the grammar from accidentally using a leaf value (such as the English word "A") as the node of a subtree. Within a ``CFG``, all node values are wrapped in the ``Nonterminal`` class. Note, however, that the trees that are specified by the grammar do *not* include these ``Nonterminal`` wrappers. Grammars can also be given a more procedural interpretation. According to this interpretation, a Grammar specifies any tree structure *tree* that can be produced by the following procedure: | Set tree to the start symbol | Repeat until tree contains no more nonterminal leaves: | Choose a production prod with whose left hand side | lhs is a nonterminal leaf of tree. | Replace the nonterminal leaf with a subtree, whose node | value is the value wrapped by the nonterminal lhs, and | whose children are the right hand side of prod. The operation of replacing the left hand side (*lhs*) of a production with the right hand side (*rhs*) in a tree (*tree*) is known as "expanding" *lhs* to *rhs* in *tree*. N)deque)total_ordering)SLASHTYPEFeatDict FeatStructFeatStructReader)raise_unorderable_types)ImmutableProbabilisticMixIn) invert_graphtransitive_closurecLeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z y ) Nonterminala. A non-terminal symbol for a context free grammar. ``Nonterminal`` is a wrapper class for node values; it is used by ``Production`` objects to distinguish node values from leaf values. The node value that is wrapped by a ``Nonterminal`` is known as its "symbol". Symbols are typically strings representing phrasal categories (such as ``"NP"`` or ``"VP"``). However, more complex symbol types are sometimes used (e.g., for lexicalized grammars). Since symbols are node values, they must be immutable and hashable. Two ``Nonterminals`` are considered equal if their symbols are equal. :see: ``CFG``, ``Production`` :type _symbol: any :ivar _symbol: The node value corresponding to this ``Nonterminal``. This value must be immutable and hashable. c||_y)z Construct a new non-terminal from the given symbol. :type symbol: any :param symbol: The node value corresponding to this ``Nonterminal``. This value must be immutable and hashable. N_symbol)selfsymbols A/var/www/openai/venv/lib/python3.12/site-packages/nltk/grammar.py__init__zNonterminal.__init__is  c|jS)ze Return the node value corresponding to this ``Nonterminal``. :rtype: (any) rrs rrzNonterminal.symbolts ||rcft|t|k(xr|j|jk(S)z Return True if this non-terminal is equal to ``other``. In particular, return True if ``other`` is a ``Nonterminal`` and this non-terminal's symbol is equal to ``other`` 's symbol. :rtype: bool )typerrothers r__eq__zNonterminal.__eq__|s)DzT%[(JT\\U]]-JJrc||k( SNrs r__ne__zNonterminal.__ne__5=  rcnt|ts td|||j|jkSN<) isinstancerr rrs r__lt__zNonterminal.__lt__s+%- #Cu 5||emm++rc,t|jSr )hashrrs r__hash__zNonterminal.__hash__sDLL!!rct|jtrd|jzSdt|jzSz_ Return a string representation for this ``Nonterminal``. :rtype: str %sr'rstrreprrs r__repr__zNonterminal.__repr__5 dllC ($,,& &$t||,, ,rct|jtrd|jzSdt|jzSr-r/rs r__str__zNonterminal.__str__r3rcJt|jd|jS)aA Return a new nonterminal whose symbol is ``A/B``, where ``A`` is the symbol for this nonterminal, and ``B`` is the symbol for rhs. :param rhs: The nonterminal used to form the right hand side of the new nonterminal. :type rhs: Nonterminal :rtype: Nonterminal /)rrrrhss r__div__zNonterminal.__div__s"dll^1S[[M:;;rc$|j|S)a Return a new nonterminal whose symbol is ``A/B``, where ``A`` is the symbol for this nonterminal, and ``B`` is the symbol for rhs. This function allows use of the slash ``/`` operator with the future import of division. :param rhs: The nonterminal used to form the right hand side of the new nonterminal. :type rhs: Nonterminal :rtype: Nonterminal )r:r8s r __truediv__zNonterminal.__truediv__s||C  rN)__name__ __module__ __qualname____doc__rrrr"r(r+r2r5r:r<r!rrrrUs:$ K!, " - - < !rrcd|vr|jd}n|j}|Dcgc]}t|jc}Scc}w)a Given a string containing a list of symbol names, return a list of ``Nonterminals`` constructed from those symbols. :param symbols: The symbol name string. This string can be delimited by either spaces or commas. :type symbols: str :return: A list of ``Nonterminals`` constructed from the symbol names given in ``symbols``. The ``Nonterminals`` are sorted in the same order as the symbols names. :rtype: list(Nonterminal) ,)splitrstrip)symbols symbol_listss r nonterminalsrHsG g~mmC( mmo ,7 8KqK "K 88 8s AceZdZdZdZdZy)FeatStructNonterminalz|A feature structure that's also a nonterminal. It acts as its own symbol, and automatically freezes itself when hashed.cL|jtj|Sr )freezerr+rs rr+zFeatStructNonterminal.__hash__s ""4((rc|Sr r!rs rrzFeatStructNonterminal.symbols rN)r=r>r?r@r+rr!rrrJrJsA)rrJc"t|tS)zJ :return: True if the item is a ``Nonterminal``. :rtype: bool )r'ritems ris_nonterminalrQs dK ((rc@t|dxrt|t S)z Return True if the item is a terminal, which currently is if it is hashable and not a ``Nonterminal``. :rtype: bool r+)hasattrr'rrOs r is_terminalrTs  4 $ JZk-J)JJrcXeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd Zy) Productiona A grammar production. Each production maps a single symbol on the "left-hand side" to a sequence of symbols on the "right-hand side". (In the case of context-free productions, the left-hand side must be a ``Nonterminal``, and the right-hand side is a sequence of terminals and ``Nonterminals``.) "terminals" can be any immutable hashable object that is not a ``Nonterminal``. Typically, terminals are strings representing words, such as ``"dog"`` or ``"under"``. :see: ``CFG`` :see: ``DependencyGrammar`` :see: ``Nonterminal`` :type _lhs: Nonterminal :ivar _lhs: The left-hand side of the production. :type _rhs: tuple(Nonterminal, terminal) :ivar _rhs: The right-hand side of the production. cht|tr td||_t ||_y)a  Construct a new ``Production``. :param lhs: The left-hand side of the new ``Production``. :type lhs: Nonterminal :param rhs: The right-hand side of the new ``Production``. :type rhs: sequence(Nonterminal and terminal) z9production right hand side should be a list, not a stringN)r'r0 TypeError_lhstuple_rhs)rlhsr9s rrzProduction.__init__s2 c3 N  #J rc|jS)z` Return the left-hand side of this ``Production``. :rtype: Nonterminal )rYrs rr\zProduction.lhs# yyrc|jS)zx Return the right-hand side of this ``Production``. :rtype: sequence(Nonterminal and terminal) )r[rs rr9zProduction.rhs+r^rc,t|jS)zP Return the length of the right-hand side. :rtype: int )lenr[rs r__len__zProduction.__len__3s 499~rc:td|jDS)zi Return True if the right-hand side only contains ``Nonterminals`` :rtype: bool c32K|]}t|ywr )rQ).0ns r z+Production.is_nonlexical..As8i>!$i)allr[rs r is_nonlexicalzProduction.is_nonlexical;s 8dii888rc$|j S)zj Return True if the right-hand contain at least one terminal token. :rtype: bool )rjrs r is_lexicalzProduction.is_lexicalCs %%'''rcdt|jz}|djd|jDz }|S)zd Return a verbose string representation of the ``Production``. :rtype: str z%s ->  c32K|]}t|ywr )r1)reels rrgz%Production.__str__..Rs8i48irh)r1rYjoinr[)rresults rr5zProduction.__str__Ks8 DO+#((8dii888 rc d|zS)zd Return a concise string representation of the ``Production``. :rtype: str r.r!rs rr2zProduction.__repr__Us d{rct|t|k(xr4|j|jk(xr|j|jk(S)za Return True if this ``Production`` is equal to ``other``. :rtype: bool )rrYr[rs rrzProduction.__eq__]sC J$u+ % ( UZZ' ( UZZ' rc||k( Sr r!rs rr"zProduction.__ne__ir#rct|ts td|||j|jf|j|jfkSr%)r'rVr rYr[rs rr(zProduction.__lt__ls=%, #Cu 5 499%UZZ(@@@rcDt|j|jfS)zR Return a hash value for the ``Production``. :rtype: int )r*rYr[rs rr+zProduction.__hash__qs TYY *++rN)r=r>r?r@rr\r9rbrjrlr5r2rr"r(r+r!rrrVrVsD& 9(  !A ,rrVceZdZdZdZy)DependencyProductionz A dependency grammar production. Each production maps a single head word to an unordered list of one or more modifier words. cZd|jd}|jD] }|d|dz } |S)zn Return a verbose string representation of the ``DependencyProduction``. :rtype: str 'z' ->z ')rYr[)rrrelts rr5zDependencyProduction.__str__s; TYYKt$99C 3%qk !F rN)r=r>r?r@r5r!rrryryzs  rryc:eZdZdZdZfdZdZdZdZxZ S)ProbabilisticProductiona A probabilistic context free grammar production. A PCFG ``ProbabilisticProduction`` is essentially just a ``Production`` that has an associated probability, which represents how likely it is that this production will be used. In particular, the probability of a ``ProbabilisticProduction`` records the likelihood that its right-hand side is the correct instantiation for any given occurrence of its left-hand side. :see: ``Production`` c ^tj|fi|tj|||y)a Construct a new ``ProbabilisticProduction``. :param lhs: The left-hand side of the new ``ProbabilisticProduction``. :type lhs: Nonterminal :param rhs: The right-hand side of the new ``ProbabilisticProduction``. :type rhs: sequence(Nonterminal and terminal) :param prob: Probability parameters of the new ``ProbabilisticProduction``. N)r rrV)rr\r9probs rrz ProbabilisticProduction.__init__s) $,,T:T:D#s+rcvt||jdk(rdzSd|jzzS)N?z [1.0]z [%g])superr5r)r __class__s rr5zProbabilisticProduction.__str__s?w +H  29DIIK2G  rct|t|k(xrW|j|jk(xr<|j|jk(xr!|j|jk(Sr )rrYr[rrs rrzProbabilisticProduction.__eq__s\ J$u+ % , UZZ' , UZZ' , uzz|+  rc||k( Sr r!rs rr"zProbabilisticProduction.__ne__r#rcbt|j|j|jfSr )r*rYr[rrs rr+z ProbabilisticProduction.__hash__s"TYY 499;788r) r=r>r?r@rr5rr"r+ __classcell__)rs@rr~r~s!  ,  !9rr~ceZdZdZddZdZdZeddZdZ ddZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZddZedZed dZedZdZdZy)!CFGa# A context-free grammar. A grammar consists of a start state and a set of productions. The set of terminals and nonterminals is implicitly specified by the productions. If you need efficient key-based access to productions, you can use a subclass to implement it. c.t|s!tdt|jz||_||_|Dchc]}|j c}|_|j|j|r|jyycc}w)a Create a new context-free grammar, from the given start state and set of ``Production`` instances. :param start: The start symbol :type start: Nonterminal :param productions: The list of productions that defines the grammar :type productions: list(Production) :param calculate_leftcorners: False if we don't want to calculate the leftcorner relation. In that case, some optimized chart parsers won't work. :type calculate_leftcorners: bool z.start should be a Nonterminal object, not a %sN) rQrXrr=_start _productionsr\ _categories_calculate_indexes_calculate_grammar_forms_calculate_leftcorners)rstart productionscalculate_leftcornersprods rrz CFG.__init__se$"5k223   '3>?;4DHHJ;? ! %%'  ' ' ) !@sBcxi|_i|_i|_i|_|jD]}|j }||jvrg|j|<|j|j ||jrK|jd}||jvrg|j|<|j|j |n||j|j<|jD]A}t|s|jj|tj|CyNr) _lhs_index _rhs_index _empty_index_lexical_indexrrYappendr[r\rT setdefaultsetaddrrr\rhs0tokens rrzCFG._calculate_indexess %%D))C$//)')$ OOC ' ' -yyyy|t.,.DOOD)%,,T215!!$((*-u%''225#%@DDTJ#!&rc |jDcic]}||hc}|_|jDcic] }|tc}|_|j D]|}t |dkDs|j |jd}}t|r|j|j|_|j|j|~t|jd}||_ t||_ ttt |jj!}ttt |jj!}||cxkDr dkDr d|_yi|_|jD]d}|j|}tx}|j"|<|D]5}|j%|jj'|t7fycc}wcc}w)NrT) reflexivei')r _immediate_leftcorner_categoriesr_immediate_leftcorner_wordsrrar\r9rQrr _leftcornersr _leftcorner_parentssummapvalues_leftcorner_wordsupdateget)rcatrleftlcnr_leftcorner_categoriesnr_leftcorner_wordsleftss rrzCFG._calculate_leftcornerssGKGWGW0XGWseGW0X-BFBRBR+SBR3CJBR+S($$&D4y1} HHJ 1 T!$'99#>BB4H44S9==dC '  E EQU V#/#3 #& T::AAC D$  "#c4+K+K+R+R+T"UV !9 AE A&*D "  B "$$$C%%c*E/2u 4B'', $::>>tSUKL%31Y+Ss G8G=Nc>t|t|\}}|||S)z Return the grammar instance corresponding to the input string(s). :param input: a grammar, either in the form of a string or as a list of strings. encoding read_grammarstandard_nonterm_parserclsinputrrrs r fromstringzCFG.fromstrings** *X {5+&&rc|jS)zU Return the start symbol of the grammar :rtype: Nonterminal )rrs rrz CFG.start)s {{rc|r |r td|s*|s(|s |jS|jjS|r@|s>|s|jj |gS||jvr|j|gSgS|r|s|j j |gS|jj |gDcgc]"}||j j |gvr|$c}Scc}w)a Return the grammar productions, filtered by the left-hand side or the first item in the right-hand side. :param lhs: Only return productions with the given left-hand side. :param rhs: Only return productions with the given first item in the right-hand side. :param empty: Only return productions with an empty right-hand side. :return: A list of productions matching the given constraints. :rtype: list(Production) CYou cannot select empty and non-empty productions at the same time.) ValueErrorrrrrrrrr\r9emptyrs rrzCFG.productions3s 5X  3(((((//11**333)))))#.// ??&&sB/ / !OO//R88D4??..sB778 s<'C&c<|jj||hS)a Return the set of all nonterminals that the given nonterminal can start with, including itself. This is the reflexive, transitive closure of the immediate leftcorner relation: (A > B) iff (A -> B beta) :param cat: the parent of the leftcorners :type cat: Nonterminal :return: the set of all leftcorners :rtype: set(Nonterminal) )rrrrs r leftcornerszCFG.leftcorners`s  $$S3%00rctrj|vSjr&jj|t vSt fdj|DS)a- True if left is a leftcorner of cat, where left can be a terminal or a nonterminal. :param cat: the parent of the leftcorner :type cat: Nonterminal :param left: the suggested leftcorner :type left: Terminal or Nonterminal :rtype: bool c3jK|]*}jj|tv,ywr )rrr)reparentrrs rrgz$CFG.is_leftcorner..s33F88<.s:'Q1%'s z3Grammar does not cover some of the input words: %r.N)rrrqr)rtokenstokmissings rcheck_coveragezCFG.check_coveragesb#)M&30C0C0G0G0L3&M ii:'::GH7R  Ns AAc|j}td|D|_td|D|_t d|D|_t d|D|_td|D|_y)z@ Pre-calculate of which form(s) the grammar is. c3<K|]}|jywr )rlreps rrgz/CFG._calculate_grammar_forms..s=u!q||~usc3ZK|]#}t|dk7s|j%ywN)rarjrs rrgz/CFG._calculate_grammar_forms..s#!RUc!fPQk!//"3U++c32K|]}t|ywr rars rrgz/CFG._calculate_grammar_forms..2EqCFErhc32K|]}t|ywr rrs rrgz/CFG._calculate_grammar_forms..rrhc3ZK|]#}t|dk(s|j%ywr)rarlrs rrgz/CFG._calculate_grammar_forms..s")W%Q3q6UV;!,,.%rN) rri _is_lexical_is_nonlexicalmin_min_lenmax_max_len_all_unary_are_lexical)rprodss rrzCFG._calculate_grammar_formssl!!=u==!!RU!RR2E22 2E22 &))W%)W&W#rc|jS)zA Return True if all productions are lexicalised. )rrs rrlzCFG.is_lexicalsrc|jS)a Return True if all lexical rules are "preterminals", that is, unary rules which can be separated in a preprocessing step. This means that all productions are of the forms A -> B1 ... Bn (n>=0), or A -> "s". Note: is_lexical() and is_nonlexical() are not opposites. There are grammars which are neither, and grammars which are both. )rrs rrjzCFG.is_nonlexicals"""rc|jS)zW Return the right-hand side length of the shortest grammar production. rrs rmin_lenz CFG.min_len}}rc|jS)zV Return the right-hand side length of the longest grammar production. rrs rmax_lenz CFG.max_lenrrc |jdkDS)z@ Return True if there are no empty productions. rrrs r is_nonemptyzCFG.is_nonemptys}}q  rc |jdkS)z Return True if all productions are at most binary. Note that there can still be empty and unary productions. rrs r is_binarisedzCFG.is_binariseds }}!!rcj|jxr"|jxr|jS)zh Return True if all productions are of the forms A -> B C, A -> B, or A -> "s". )rrjrrs ris_flexible_chomsky_normal_formz#CFG.is_flexible_chomsky_normal_forms/ !Rd&8&8&:Rt?P?P?RRrc>|jxr |jS)z Return True if the grammar is of Chomsky Normal Form, i.e. all productions are of the form A -> B C, or A -> "s". )rrrs ris_chomsky_normal_formzCFG.is_chomsky_normal_forms 335U$:U:UUrc >|jr|S|jdr td|jD]\}|jst |j dkDs1td|j d|j tj|}tj||}|r|Stj|}t|jtt|j}|S)z Returns a new Grammar that is in chomsky normal :param: new_token_padding Customise new rule formation during binarisation T)rz )rrrrlrar9r\reliminate_startbinarizeremove_unitary_rulesrlistr)rnew_token_paddingflexiblerulestep1step2step3step4s rchomsky_normal_formzCFG.chomsky_normal_forms  & & (K   $  'Q  $$&D S_q%8 0 D M' ##D) U$56 L((/EKKM4E,=,=,?(@#AB rcLg}tg}|jD]C}t|dk(r"|jr|j |3|j |E|r|j }|j|j dD]k}t|j|j }t|dk7s|jr|j |[|j |m|rt|j|}|S)zU Remove nonlexical unitary rules and convert them to lexical rr)r\) rrrarjrpopleftr9rVr\rlrr)rgrammarrrunitaryrrPnew_rule n_grammars rrzCFG.remove_unitary_ruless )'')D4yA~$"4"4"6t$ d# * ??$D++ 1 +>%dhhj$((*=x=A%)<)<)>MM(+NN8, ? 0 rcXg}|jD]}t|jdkDr|j}t dt|jdz D]c}|j|}t |j |z|j z}t|||f} |}|j| et||jdd} |j| |j|t|j|} | S)z Convert all non-binary rules into binary by introducing new tokens. Example:: Original: A => B C D After Conversion: A => B A@$@B A@$@B => C D rrN) rrar9r\rangerrrVrrr) rr paddingrrr left_sidektsymnew_symnew_productionlast_prdrs rrz CFG.binarizes'')D488:" HHJ q#dhhj/A"56A88:a=D))*:*:* S0 S1 and S0 -> S1 S Then another rule S0_Sigma -> S is added NTS0_SIGMA)rrr9rrrVr)rr rrr need_to_addrrs rrzCFG.eliminate_start;s  '')D "" MM$ *  +E MM*UW]]_,=> ?E6*I rc2dt|jzS)Nzrarrs rr2z CFG.__repr__Ps.T5F5F1GGGrcdt|jz}|d|jzz }|jD] }|d|zz } |S)NzGrammar with %d productionsz (start state = %r)z %s)rarr)rrr productions rr5z CFG.__str__SsO.T5F5F1GG'$++55++J j:- -F, rTr NNF)@$@F)r#)r=r>r?r@rrr classmethodrrrrrrrrrlrjrrrrrrr rrrr2r5r!rrrrs*6K2MB ' '+Z 1* 8  X #  ! "SV:2@(HrrcJeZdZdZdZdZe d dZd dZdZ dZ d Z y) FeatureGrammara A feature-based grammar. This is equivalent to a ``CFG`` whose nonterminals are all ``FeatStructNonterminal``. A grammar consists of a start state and a set of productions. The set of terminals and nonterminals is implicitly specified by the productions. c2tj|||y)a@ Create a new feature-based grammar, from the given start state and set of ``Productions``. :param start: The start symbol :type start: FeatStructNonterminal :param productions: The list of productions that defines the grammar :type productions: list(Production) N)rr)rrrs rrzFeatureGrammar.__init__fs T5+.rc4i|_i|_i|_g|_i|_|j D]e}|j |j}||jvrg|j|<|j|j||jrZ|j |jd}||jvrg|j|<|j|j|nV||jvrg|j|<|j|j||jj||jD]A}t|s|jj|tj|Chyr)rrr_empty_productionsrr_get_type_if_possiblerYrr[rTrrrrs rrz!FeatureGrammar._calculate_indexesvsS"$ %%D,,TYY7C$//)')$ OOC ' ' -yy11$))A,?t.,.DOOD)%,,T2d///-/D%%c*!!#&--d3''..t4u%''225#%@DDTJ#'&rNc| ttf}|t|t|}n | t dt ||j |\}}|||S)a Return a feature structure based grammar. :param input: a grammar, either in the form of a string or else as a list of strings. :param features: a tuple of features (default: SLASH, TYPE) :param logic_parser: a parser for lambda-expressions, by default, ``LogicParser()`` :param fstruct_reader: a feature structure parser (only if features and logic_parser is None) ) logic_parserz8'logic_parser' and 'fstruct_reader' must not both be setr)rrr rJ Exceptionr read_partial)rrfeaturesr,fstruct_readerrrrs rrzFeatureGrammar.fromstringsp  t}H  !-/lN %M * >.. {5+&&rc H|r |r td|s|s|r |jS|jS|rZ|sX|r+|jj |j |gS|j j |j |gS|r-|s+|jj |j |gS|j j |j |gDcgc]1}||jj |j |gvr|3c}Scc}w)a Return the grammar productions, filtered by the left-hand side or the first item in the right-hand side. :param lhs: Only return productions with the given left-hand side. :param rhs: Only return productions with the given first item in the right-hand side. :param empty: Only return productions with an empty right-hand side. :rtype: list(Production) r)rr)rrrr*rrrs rrzFeatureGrammar.productionss 5X  3...(((((,,T-G-G-LbQQ**4+E+Ec+JBOO??&&t'A'A#'FK K !OO//0J0J30OQSTTD4??..t/I/I#/NPRSST s&6Dctd)z Return the set of all words that the given category can start with. Also called the "first set" in compiler construction. Not implemented yetNotImplementedErrorrs rrzFeatureGrammar.leftcorners ""788rctd)zi Return the set of all categories for which the given category is a left corner. r3r4rs rrz!FeatureGrammar.leftcorner_parentsr6rcZt|trt|vrt|tS|S)z Helper function which returns the ``TYPE`` feature of the ``item``, if it exists, otherwise it returns the ``item`` itself )r'dictrFeatureValueType)rrPs rr*z$FeatureGrammar._get_type_if_possibles( dD !ddl#DJ/ /Kr)NNNNr") r=r>r?r@rrr$rrrrr*r!rrr&r&[s> / K:TX''>(T99rr&c4eZdZdZdZdZdZdZdZdZ y) r:z A helper class for ``FeatureGrammars``, designed to be different from ordinary strings. This is to stop the ``FeatStruct`` ``FOO[]`` from being compare equal to the terminal "FOO". c||_yr _value)rvalues rrzFeatureValueType.__init__s  rc d|jzS)Nz<%s>r=rs rr2zFeatureValueType.__repr__s ##rcft|t|k(xr|j|jk(Sr )rr>rs rrzFeatureValueType.__eq__s'DzT%[(HT[[ELL-HHrc||k( Sr r!rs rr"zFeatureValueType.__ne__r#rcnt|ts td|||j|jkSr%)r'r:r r>rs rr(zFeatureValueType.__lt__ s,%!12 #Cu 5{{U\\))rc,t|jSr )r*r>rs rr+zFeatureValueType.__hash__sDKK  rN) r=r>r?r@rr2rr"r(r+r!rrr:r:s& $I!* !rr:c>eZdZdZdZedZdZdZdZ dZ y) DependencyGrammarz A dependency grammar. A DependencyGrammar consists of a set of productions. Each production specifies a head/modifier relationship between a pair of words. c||_y)z Create a new dependency grammar, from the set of ``Productions``. :param productions: The list of productions that defines the grammar :type productions: list(Production) N)r)rrs rrzDependencyGrammar.__init__s (rc>g}t|jdD];\}}|j}|jds|dk(r- |t |z }=t |dk(r t d||S#t $r}t d|d||d}~wwxYw)N #Unable to parse line : rNo productions found!) enumeraterCrD startswith_read_dependency_productionrra)rrrlinenumlinees rrzDependencyGrammar.fromstring#s &u{{4'89MGT::Must use a tuple of strings, e.g. `('price', 'of') in grammar`N)rr[)rhead_modrXrYrTs r __contains__zDependencyGrammar.__contains__CsH  ID# }}T3''  P  s 3 .3chdt|jz}|jD] }|d|zz } |S)zj Return a verbose string representation of the ``DependencyGrammar`` :rtype: str &Dependency grammar with %d productions %sr)rr0r s rr5zDependencyGrammar.__str__`s= 7T=N=N9OO++J 8j( (C, rc2dt|jzS)zU Return a concise string representation of the ``DependencyGrammar`` rarrs rr2zDependencyGrammar.__repr__ks8#d>O>O:PPPrN) r=r>r?r@rr$rr[r_r5r2r!rrrFrFs5 (    "(: QrrFc(eZdZdZdZdZdZdZy)ProbabilisticDependencyGrammarrnc.||_||_||_yr )r_events_tags)rreventstagss rrz'ProbabilisticDependencyGrammar.__init__us'  rcx|jD]+}|jD]}|j|k(s||k(sy-y)a( Return True if this ``DependencyGrammar`` contains a ``DependencyProduction`` mapping 'head' to 'mod'. :param head: A head word. :type head: str :param mod: A mod word, to test as a modifier of 'head'. :type mod: str :rtype: bool TFrVrWs rr[z'ProbabilisticDependencyGrammar.containszr\rc dt|jz}|jD] }|d|zz } |dz }|jD]}|d|j||fzz }|dz }|jD]}|d|d|j|dz }|S) zw Return a verbose string representation of the ``ProbabilisticDependencyGrammar`` :rtype: str z2Statistical dependency grammar with %d productionsrbz Events:z %d:%sz Tags:z z: ())rarrgrh)rr0r eventtag_words rr5z&ProbabilisticDependencyGrammar.__str__s CS   F  ++J 8j( (C, {\\E ;$,,u"5u!== =C" y H S $tzz(';& >C# rc2dt|jzS)zb Return a concise string representation of the ``ProbabilisticDependencyGrammar`` z2Statistical Dependency grammar with %d productionsrrs rr2z'ProbabilisticDependencyGrammar.__repr__s#Dc   G   rN)r=r>r?r@rr[r5r2r!rrrerers  "& rrec.eZdZdZdZddZeddZy)PCFGa A probabilistic context-free grammar. A PCFG consists of a start state and a set of productions with probabilities. The set of terminals and nonterminals is implicitly specified by the productions. PCFG productions use the ``ProbabilisticProduction`` class. ``PCFGs`` impose the constraint that the set of productions with any given left-hand-side must have probabilities that sum to 1 (allowing for a small margin of error). If you need efficient key-based access to productions, you can use a subclass to implement it. :type EPSILON: float :cvar EPSILON: The acceptable margin of error for checking that productions with a given left-hand side have probabilities that sum to 1. g{Gz?cttj||||i}|D]D}|j|jd|j z||j<F|j D]B\}}dt jz |cxkrdt jzkr4ntd|zy)a Create a new context-free grammar, from the given start state and set of ``ProbabilisticProductions``. :param start: The start symbol :type start: Nonterminal :param productions: The list of productions that defines the grammar :type productions: list(Production) :raise ValueError: if the set of productions with any left-hand-side do not have probabilities that sum to a value within EPSILON of 1. :param calculate_leftcorners: False if we don't want to calculate the leftcorner relation. In that case, some optimized chart parsers won't work. :type calculate_leftcorners: bool rrz"Productions for %r do not sum to 1N) rrrr\ritemsrrEPSILONr)rrrrprobsr r\rs rrz PCFG.__init__s T5+/DE%J&+ii 0@!&DzGX&XE*.." #&kkmFC%?a$,,.>? !E!KLL$rNc@t|td|\}}|||S)z Return a probabilistic context-free grammar corresponding to the input string(s). :param input: a grammar, either in the form of a string or else as a list of strings. T) probabilisticrrrs rrzPCFG.fromstrings,* *$ {5+&&rr!r )r=r>r?r@rurr$rr!rrrrrrs(&GM4 ' 'rrrc ni}i}|D]N}|j|jddz||j<|j|ddz||<P|Dcgc]C}t|j|j||||jz E}}t ||Scc}w)a Induce a PCFG grammar from a list of productions. The probability of a production A -> B C in a PCFG is: | count(A -> B C) | P(B, C | A) = --------------- where \* is any right hand side | count(A -> \*) :param start: The start symbol :type start: Nonterminal :param productions: The list of productions that defines the grammar :type productions: list(Production) rrr)rr\r~r9rr)rrpcountlcountrrrs r induce_pcfgr}s FF#ZZ A6:txxzzz$*Q.t   A vay6!%%'?7RS   u   sAB2c"t|tS)z8 Return a list of context-free ``Productions``. _read_productionrrs r_read_cfg_productionrs E#: ;;rc&t|tdS)z= Return a list of PCFG ``ProbabilisticProductions``. T)rxrrs r_read_pcfg_productionrs E#:$ OOrct||S)z9 Return a list of feature-based ``Productions``. )r)rr0s r_read_fcfg_productionr s E> 22rz \s* -> \s*z( \[ [\d\.]+ \] ) \s*z( "[^"]*" | \'[^\']*\' ) \s*z\| \s*c d}|||\}}tj||}|s td|j}dg}gg}|t |krFt j||}|rL|rJ|j}t |jddd|d<|ddkDrtd|dfz||dvrZtj||}|s td |dj|jddd|j}nq||d k(rItj||}|jd|jg|j}n |||\}}|dj||t |krF|r+t||D cgc]\} } t|| | c} } S|D cgc]} t|| c} Scc} } wcc} w) zX Parse a grammar rule, given as a string, and return a list of productions. rzExpected an arrowgrrz9Production probability %f, should not be greater than 1.0'"zUnterminated string|rz) _ARROW_REmatchrendra_PROBABILITY_REfloatgroup _TERMINAL_REr_DISJUNCTION_REzipr~rV) rSnonterm_parserrxposr\m probabilitiesrhsidesnontermr9 probabilitys rrr/s CdC(HC c"A ,-- %%'CEMdG D /  ! !$ , Q%%'C %aggaj2&6 7M" R 3& 58Eb8I7KL #Y% ""4-A !677 BK  qwwqz!B/ 0%%'C#Y# %%dC0A   % NN2 %%'C*$4LGS BK  w '= D /@'*'=&A &A"k $C; ?&A  188 3$88  9s .G$G*c ||j|}t|tr|jd}n|}d}g}d}t |D]\}} || j z} | j ds| dk(r0| jdr| ddjdz}Xd} | dd k(rM| d djdd \} } | d k(r%|| d\}} | t| k7r&td td |t| ||z }|s td|s|dj}||fS#t$r} td|d zd| d| | d} ~ wwxYw)a7 Return a pair consisting of a starting category and a list of ``Productions``. :param input: a grammar, either in the form of a string or else as a list of strings. :param nonterm_parser: a function for parsing nonterminals. It should take a ``(string, position)`` as argument and return a ``(nonterminal, position)`` as result. :param probabilistic: are the grammar rules probabilistic? :type probabilistic: bool :param encoding: the encoding of the grammar, if it is a binary string :type encoding: str NrIrKrJ\rrnr%rrzBad argument to start directivez Bad directiverLrMrN) decoder'r0rCrOrDrPendswithrstriprarrr\)rrrxrlinesrr continue_linerRrS directiveargsrrTs rrrps X&% D! EKM"5) tzz|+ ??3 42:  ==  "I,,.4M   XAw#~"&qr(..q"9 4'!/a!8JE3c$i'()JKK$_55/nmTT '*. 011 A""$ ;  X4Wq[MD6A3OPVW W Xs$A%D11 E:EEz( [\w/][\w/^<>-]* ) \s*ctj||}|std||dzt|j d|j fS)NzExpected a nonterminal, found: r)_STANDARD_NONTERM_RErrrrr)stringrrs rrrsL""63/A :VCD\IJJ  #QUUW --raH^\s* # leading whitespace ('[^']+')\s* # single-quoted lhs (?:[-=]+>)\s* # arrow (?:( # rhs: "[^"]+" # doubled-quoted terminal | '[^']+' # single-quoted terminal | \| # disjunction ) \s*) # trailing space *$z"('[^']'|[-=]+>|"[^"]+"|'[^']+'|\|)ctj|s tdtj |}t |Dcgc]\}}|dzdk(s|}}}|dj d}gg}|ddD]<}|dk(r|jg|dj|j d>|Dcgc]}t||c}Scc}}wcc}w)NzBad production stringrrrrrr) _READ_DG_RErr _SPLIT_DG_RErCrOrDrry)rGpiecesirlhsiderpiecerhsides rrQrQs   Q 011    "F%f- <-DAqQ!a-F < AY__U #FdG C< NN2  BK  u{{51 2  @G GwV  0w GG= HsCC;Cc *ddlm}m}m}|d\}}}}|d\}}} } ||z } t d||||||| | ||z g t dt |j t t |||g|jd} t dt | t d t | jt d d t t | jjd dt y)zG A demonstration showing how ``CFGs`` can be created and used. r)rrVrHz S, NP, VP, PPz N, V, P, DetzSome nonterminals:z S.symbol() =>z S -> NP VP PP -> P NP NP -> Det N | NP PP VP -> V NP | VP PP Det -> 'a' | 'the' N -> 'dog' | 'cat' V -> 'chased' | 'sat' P -> 'on' | 'in' z A Grammar: grammar.start() => grammar.productions() =>rnrrBz, N) nltkrrVrHprintr1rrrrreplace) rrVrHSNPVPPPNVPDet VP_slash_NPr s rcfg_demors 32!1MAr2r/LAq!Sr'K BB1ab2g FG tAHHJ/0 G *Q nn  G ,W & ($w}}*?@ (c2 $w""$ % - -c3C DE Grcddlm}m}ddlm}ddlm}tjd}tjd}|j}|d}tdt|td t|jtd t|jtd t|jt|}td t|td t|jtddtt|jj!ddttdg} |j"d} |j%| ddD]9} | j'd| j)d| | jz } ;t+d} || | }t|ttd|j-|} | j/d|j%| dj1}t|| j3|D] }t|y)zI A demonstration showing how a ``PCFG`` can be created and used. r)r}treetransforms)treebank)pcharta[ S -> NP VP [1.0] NP -> Det N [0.5] | NP PP [0.25] | 'John' [0.1] | 'I' [0.15] Det -> 'the' [0.8] | 'my' [0.2] N -> 'man' [0.5] | 'telescope' [0.5] VP -> VP PP [0.1] | V NP [0.7] | V [0.2] V -> 'ate' [0.35] | 'saw' [0.65] PP -> P NP [1.0] P -> 'with' [0.61] | 'under' [0.39] aD S -> NP VP [1.0] VP -> V NP [.59] VP -> V [.40] VP -> VP PP [.01] NP -> Det N [.41] NP -> Name [.28] NP -> NP PP [.31] PP -> P NP [1.0] V -> 'saw' [.21] V -> 'ate' [.51] V -> 'ran' [.28] N -> 'boy' [.11] N -> 'cookie' [.12] N -> 'table' [.13] N -> 'telescope' [.14] N -> 'hill' [.5] Name -> 'Jack' [.52] Name -> 'Bob' [.48] P -> 'with' [.61] P -> 'under' [.39] Det -> 'the' [.41] Det -> 'a' [.31] Det -> 'my' [.28] rzA PCFG production:z pcfg_prod.lhs() =>z pcfg_prod.rhs() =>z pcfg_prod.prob() =>zA PCFG grammar:rrrnrrBz, z'Induce PCFG grammar from treebank data:NF) collapsePOS) horzMarkovrz%Parse sentence using induced grammar:)rr}r nltk.corpusr nltk.parserrrrrrr1r\r9rrr_fileids parsed_sentscollapse_unaryr rInsideChartParsertraceleavesparse)r}rrr toy_pcfg1 toy_pcfg2 pcfg_prods pcfg_prodr rrPtreerparsersentrs r pcfg_demors 1$!  I I8&&(J1 I Y0 #T)--/%:; #T)--/%:; #T)..*:%;< GG T']+ ($w}}*?@ (c2 $w""$ % - -c3C DE G 34K   Q D%%d+BQ/ .   A .t'')) 0 CA![)G 'N G 12  % %g .F LLO   &q ) 0 0 2D $Kd# e $rclddl}|jjd}t|ty)Nrz!grammars/book_grammars/feat0.fcfg) nltk.datadataloadr)rgs r fcfg_demorcs$ :;A !H GrcDtjd}t|y)z] A demonstration showing the creation and inspection of a ``DependencyGrammar``. zP 'scratch' -> 'cats' | 'walls' 'walls' -> 'the' 'cats' -> 'the' N)rFrr)r s rdg_demorks"  ** G 'Nrcrddlm}|d}|j}t|j y)zg A demonstration of how to read a string representation of a CoNLL format dependency tree. r)DependencyGraphag 1 Ze ze Pron Pron per|3|evofmv|nom 2 su _ _ 2 had heb V V trans|ovt|1of2of3|ev 0 ROOT _ _ 3 met met Prep Prep voor 8 mod _ _ 4 haar haar Pron Pron bez|3|ev|neut|attr 5 det _ _ 5 moeder moeder N N soort|ev|neut 3 obj1 _ _ 6 kunnen kan V V hulp|ott|1of2of3|mv 2 vc _ _ 7 gaan ga V V hulp|inf 6 vc _ _ 8 winkelen winkel V V intrans|inf 11 cnj _ _ 9 , , Punc Punc komma 8 punct _ _ 10 zwemmen zwem V V intrans|inf 11 cnj _ _ 11 of of Conj Conj neven 7 vc _ _ 12 terrassen terras N N soort|mv|neut 11 cnj _ _ 13 . . Punc Punc punt 12 punct _ _ N)rrrrpprint)rdgrs rsdg_demorzs1 +   B" 779D $++-rchttttt yr )rrrrrr!rrdemors J K K I Jr__main__) rrHrrVrrr~rFryrer}r)F)FN):r@re collectionsr functoolsrnltk.featstructrrrrr nltk.internalsr nltk.probabilityr nltk.utilr r rrHrJrQrTrVryr~rr&r:rFrerrr}rrrcompileVERBOSErrrrrrrrrrrQrrrrrrr=__all__r!rrrs9t $OO286i!i!i!X9( Hk )Kx,x,x,v:$*9j*E*9dZZz XSXv!!!8\Q\Q~2 2 j<'3<'LJ<P3 BJJ}bjj 1 "**5rzzBrzz92::F "**Y 399B4 n"rzz"