gpddlZddlZddlmZejeZGddZGddeZGddeZ Gd d Z Gd d Z d Z dZ dZdZedk(re yy)N)DependencyGraphc"eZdZdZdZdZdZy)DependencyScorerIa A scorer for calculated the weights on the edges of a weighted dependency graph. This is used by a ``ProbabilisticNonprojectiveParser`` to initialize the edge weights of a ``DependencyGraph``. While typically this would be done by training a binary classifier, any class that can return a multidimensional list representation of the edge weights can implement this interface. As such, it has no necessary fields. c@|jtk(r tdy)Nz*DependencyScorerI is an abstract interface) __class__r TypeErrorselfs ]/var/www/openai/venv/lib/python3.12/site-packages/nltk/parse/nonprojectivedependencyparser.py__init__zDependencyScorerI.__init__"s >>. .HI I /ct)a& :type graphs: list(DependencyGraph) :param graphs: A list of dependency graphs to train the scorer. Typically the edges present in the graphs can be used as positive training examples, and the edges not present as negative examples. NotImplementedErrorr graphss r trainzDependencyScorerI.train&s "##r ct)ae :type graph: DependencyGraph :param graph: A dependency graph whose set of edges need to be scored. :rtype: A three-dimensional list of numbers. :return: The score is returned in a multidimensional(3) list, such that the outer-dimension refers to the head, and the inner-dimension refers to the dependencies. For instance, scores[0][1] would reference the list of scores corresponding to arcs from node 0 to node 1. The node's 'address' field can be used to determine its number identification. For further illustration, a score list corresponding to Fig.2 of Keith Hall's 'K-best Spanning Tree Parsing' paper:: scores = [[[], [5], [1], [1]], [[], [], [11], [4]], [[], [10], [], [5]], [[], [8], [8], []]] When used in conjunction with a MaxEntClassifier, each score would correspond to the confidence of a particular edge being classified with the positive training examples. rr graphs r scorezDependencyScorerI.score0s 2"##r N__name__ __module__ __qualname____doc__r rrr r rrs J$$r rc"eZdZdZdZdZdZy)NaiveBayesDependencyScorera A dependency scorer built around a MaxEnt classifier. In this particular class that classifier is a ``NaiveBayesClassifier``. It uses head-word, head-tag, child-word, and child-tag features for classification. >>> from nltk.parse.dependencygraph import DependencyGraph, conll_data2 >>> graphs = [DependencyGraph(entry) for entry in conll_data2.split('\n\n') if entry] >>> npp = ProbabilisticNonprojectiveParser() >>> npp.train(graphs, NaiveBayesDependencyScorer()) >>> parses = npp.parse(['Cathy', 'zag', 'hen', 'zwaaien', '.'], ['N', 'V', 'Pron', 'Adj', 'N', 'Punc']) >>> len(list(parses)) 1 cyNrr s r r z#NaiveBayesDependencyScorer.__init__cs r c @ddlm}g}|D]{}|jjD]\}|jj D]=\}}||dvrd}nd}|j t |d|d|d|d|f?^}|j||_y ) as Trains a ``NaiveBayesClassifier`` using the edges present in graphs list as positive examples, the edges not present as negative examples. Uses a feature vector of head-word, head-tag, child-word, and child-tag. :type graphs: list(DependencyGraph) :param graphs: A list of dependency graphs to train the scorer. r)NaiveBayesClassifierdepsTFwordtagabcdN) nltk.classifyr#nodesvaluesitemsappenddictr classifier) r rr#labeled_examplesr head_node child_index child_nodelabels r rz NaiveBayesDependencyScorer.trainfs 7E"[[//1 /4{{/@/@/B+K"i&77 # #$++ "+F"3"+E"2",V"4",U"3  "  0C2&/445EFr c dg}|jjD]K}|jjD],}|jt|d|d|d|d.Mg}g}d}|jj |D]}t jd|jd|jd|jtj|jddzg|d z }|t|jk(s|j|g}d}|S) a Converts the graph into a feature-based representation of each edge, and then assigns a score to each based on the confidence of the classifier in assigning it to the positive label. Scores are returned in a multidimensional list. :type graph: DependencyGraph :param graph: A dependency graph to score. :rtype: 3 dimensional list :return: Edge scores for the graph parameter. r'r(r)rz %.4f %.4fr%r&gdy=) r/r0r2r3r4prob_classify_manyloggerdebugprobmathloglen) r redgesr6r8 edge_scoresrowcountpdists r rz NaiveBayesDependencyScorer.scores++-I#kk002  #F+#E*$V,$U+ 3. __77>E LLejjouzz# G JJC=!@AB C QJEEKK((""3'?r Nrrr r rrQs" "GH&r rceZdZdZdZy) DemoScorerctdy)Nz Training...)printrs r rzDemoScorer.trains  mr c@gdgdgdggggdgdgggdggdgggdgdggggS)Nr;  rrs r rzDemoScorer.scoresP!qcA3  bTA3  "rA3  !qc2    r N)rrrrrrr r rIrIs  r rIcLeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z y ) ProbabilisticNonprojectiveParsera A probabilistic non-projective dependency parser. Nonprojective dependencies allows for "crossing branches" in the parse tree which is necessary for representing particular linguistic phenomena, or even typical parses in some languages. This parser follows the MST parsing algorithm, outlined in McDonald(2005), which likens the search for the best non-projective parse to finding the maximum spanning tree in a weighted directed graph. >>> class Scorer(DependencyScorerI): ... def train(self, graphs): ... pass ... ... def score(self, graph): ... return [ ... [[], [5], [1], [1]], ... [[], [], [11], [4]], ... [[], [10], [], [5]], ... [[], [8], [8], []], ... ] >>> npp = ProbabilisticNonprojectiveParser() >>> npp.train([], Scorer()) >>> parses = npp.parse(['v1', 'v2', 'v3'], [None, None, None]) >>> len(list(parses)) 1 Rule based example >>> from nltk.grammar import DependencyGrammar >>> grammar = DependencyGrammar.fromstring(''' ... 'taught' -> 'play' | 'man' ... 'man' -> 'the' | 'in' ... 'in' -> 'corner' ... 'corner' -> 'the' ... 'play' -> 'golf' | 'dachshund' | 'to' ... 'dachshund' -> 'his' ... ''') >>> ndp = NonprojectiveDependencyParser(grammar) >>> parses = ndp.parse(['the', 'man', 'in', 'the', 'corner', 'taught', 'his', 'dachshund', 'to', 'play', 'golf']) >>> len(list(parses)) 4 c.tjdy)z6 Creates a new non-projective parser. z#initializing prob. nonprojective...N)loggingr>r s r r z)ProbabilisticNonprojectiveParser.__init__s  ;remove_by_addressadd_node redirect_arcs)r new_node cycle_pathg_graphb_graphc_graphcycle_node_indexs r collapse_nodesz/ProbabilisticNonprojectiveParser.collapse_nodessN  *+ *   % %&6 7!+"j(9*=>r ctjd||j|}tjd|tjd|jt |jD]\}}t |j|D]\}}tj|j||||vs0||vs5|j||sH|j ||}tjd|j|||g}|j||D]} |j | |z ||j||<t |jD]A\}}t |j|D]!\}} ||vs ||vsg|j||<#Ctjd|jy)aE Updates the edge scores to reflect a collapse operation into new_node. :type new_node: A Node. :param new_node: The node which cycle nodes are collapsed into. :type cycle_path: A list of integers. :param cycle_path: A list of node addresses that belong to the cycle. zcycle %sz old cycle %szPrior to update: %sz%s - %szAfter update: %sN)r=r>compute_original_indexesrZ enumeratecompute_max_subtract_scorer2) r rarbirEjcolumn subtract_valnew_valscur_valcells r update_edge_scoresz3ProbabilisticNonprojectiveParser.update_edge_scores1s  Z,22:>  ^Z0 *DKK8 ,FAs&t{{1~6 6 T[[^A./ ?q ':t{{1~a?P#'#B#B1j#QLLLDKKN1,=|L!H#';;q>!#4 ,(>?$5)1DKKN1%7-  ,FAs$T[[^44 ?qJ(*DKKN1%5-  '5r cd}|r\g}d}|D]N}||jvr-|j|D]}||vs|j|d}>|j|P|}|r\|S)a As nodes are collapsed into others, they are replaced by the new node in the graph, but it's still necessary to keep track of what these original nodes were. This takes a list of node addresses and replaces any collapsed node addresses with their original addresses. :type new_indexes: A list of integers. :param new_indexes: A list of node addresses to check for subsumed nodes. TF) inner_nodesr2)r new_indexesswapped originals new_indexold_vals r riz9ProbabilisticNonprojectiveParser.compute_original_indexesWsIG(  0 00#'#3#3I#>")3%,,W5&*G$? $$Y/)$Kr cVd}|D]!}|j||D] }||kDs |} #|S)a When updating scores the score of the highest-weighted incoming arc is subtracted upon collapse. This returns the correct amount to subtract from that edge. :type column_index: integer. :param column_index: A index representing the column of incoming arcs to a particular node being updated :type cycle_indexes: A list of integers. :param cycle_indexes: Only arcs from cycle nodes are considered. This is a list of such nodes addresses. i`y)rZ)r column_index cycle_indexes max_score row_indexros r rkz;ProbabilisticNonprojectiveParser.compute_max_subtract_scorersA &I $ I 6| D )+ ,I!E'r c|j|g}tjd|d}d}tt |j D]p}tt |j |D]J}||vs||j |||kDs |j ||}|}tjd||Lrtj||j D]}|j |}||vs|cS|S)z Returns the source of the best incoming arc to the node with address: node_index :type node_index: integer. :param node_index: The address of the 'destination' node, the node that is arced to. z originals: %sNz%s, %s)rir=r>rangerBrZru) r node_indexrxmax_arcr~r col_indexkeyreplaced_nodess r best_incoming_arcz2ProbabilisticNonprojectiveParser.best_incoming_arcs11:,?  _i0 s4;;/0I"3t{{9'=#>?  )%Y)? )JY)V $ I 6y AI'GLL9i@ @1  Y##C!--c2N.( $ r c2|j|g}d}d}d}tt|jD][}tt|j|D]5}||vs||j|||kDs |j||}|}|}7]||gSr!)rirrBrZ)r rrxrr~max_origrrs r original_best_arcz2ProbabilisticNonprojectiveParser.original_best_arcs11:,?  s4;;/0I"3t{{9'=#>?  )%Y)? )JY)V $ I 6y AI'G(H @1""r c#Ki|_t}t|D]1\}}|j|dzj |||d|dzd3|j t}t|D]1\}}|j|dzj |||d|dzd3t}t}t|D]1\}}|j|dzj |||d|dzd3|j |tj|j|jjD cgc]} | d } } t|} i} | r| jd} tjd| |j| }tjd||j| }|j| | | <tjd|| | |fD]$}|j|j d d|d &|j!|| |j#}|rd d| dzd }|j%||j'|||j)||||||D]}|j!|d|||j|d<| j+d| dz| dz } |D]}|j-|tjd |tjd |tjd|tjd| tjd|j| rtjd|jtjdt/t|dz| dzD]}| || | |d<tjd| |jjD]}i|d< t/dt|dzD] }|j!| |d| |d"tjd|ycc} ww)a Parses a list of tokens in accordance to the MST parsing algorithm for non-projective dependency parses. Assumes that the tokens to be parsed have already been tagged and those tags are provided. Various scoring methods can be used by implementing the ``DependencyScorerI`` interface and passing it to the training algorithm. :type tokens: list(str) :param tokens: A list of words or punctuation to be parsed. :type tags: list(str) :param tags: A list of tags corresponding by index to the words in the tokens list. :return: An iterator of non-projective parses. :rtype: iter(DependencyGraph) r;NTOP)r'r(relr]r]rzcurrent_vertex: %szcurrent_node: %szbest in arc: %s --> %sTEMP)r'rr]NONEz g_graph: %sz b_graph: %sz c_graph: %sz Betas: %szreplaced nodes %szFinal scores: %szRecovering parse...r$zDone.N)rurrjr/update connect_graphr[r=r>rZr0rBpopget_by_addressrradd_arccontains_cycler_rsrginsertr^r)r tokenstagsrcindextokenoriginal_graphrdrevertexunvisited_vertices nr_verticesbetascurrent_vertex current_node best_in_edge new_vertexrbra cycle_indexcycle_node_addressrlnodes r parsez&ProbabilisticNonprojectiveParser.parses"#%f-LE5 MM%!) $ + +tE{6eVWiX . (*%f-LE5   + 2 2tE{6eVWiX . "#!#%f-LE5 MM%!) $ + +tE{6eVWiX . ##G, T[[!>Emm>R>R>TU>TFfY/>TU&k  /33A6N LL-~ >"11.AL LL+\ :11.AL$($:$:>$JE. ! LL1< P-|<  j)00#FzJ= OOL. 9!//1J$*6kTUoV  *''*=##Hj'7GT#-KOOHY$7E$.9C  )!45#))![1_=q +5&--.@A+5 LL 0 LL 0 LL 0 LLe , LL,d.>.> ?a!f  '5 *+s6{Q a8A!&qE%(1+ 9  [%("((//1D DL 2 q#f+/*A  " "58A;a <+  WQVsE Q QHQC7QN)rrrrr rr[rgrsrirkrrrrr r rSrSs=/b= # 0?($6L6(@ #tr rSceZdZdZdZdZy)NonprojectiveDependencyParserau A non-projective, rule-based, dependency parser. This parser will return the set of all possible non-projective parses based on the word-to-word relations defined in the parser's dependency grammar, and will allow the branches of the parse tree to cross in order to capture a variety of linguistic phenomena that a projective parser will not. c||_y)z Creates a new ``NonprojectiveDependencyParser``. :param dependency_grammar: a grammar of word-to-word relations. :type dependency_grammar: DependencyGrammar N)_grammar)r dependency_grammars r r z&NonprojectiveDependencyParser.__init__;s + r c#Kt|_t|D]#\}}|gd|d|jj|<%|jjj D]u}g}|jjj D]E}|j j |d|ds&|d|dk7s2|j|dG||d<wg}g}t|D]~\} } g} t|D]9\} } | | k7s |j j | | s)| j| ;t| dk(r|j| |j| t|dkrt|dk(r*tt|D]} |j| g}|D]%}g}tt|D cgc]} g}} 'd} d}| dk\rN|r_t|| d k(r || d| <nBt|| dk(rd | <n+|| j} | | <j| | g|sd }D] }|d| k(s d}t|| }|rX|dk(rStt|d z d d D]6} || }|d| k(s|| j|j| d 8n4|r2|dkDr-|| j} | | <|j| | gd}| d zt|k(r|jd d d }|r| d z } n| d z} | dk\rND]}|jd d kDrt}|j|jd d z|_ tt||d D]a\}\}}|d z}|j|}|j||d|dk(rd}nd}|j|d zd|j|c|y cc} ww)a Parses the input tokens with respect to the parser's grammar. Parsing is accomplished by representing the search-space of possible parses as a fully-connected directed graph. Arcs that would lead to ungrammatical parses are removed and a lattice is constructed of length n, where n is the number of input tokens, to represent all possible grammatical traversals. All possible paths through the lattice are then enumerated to produce the set of non-projective parses. param tokens: A list of tokens to parse. type tokens: list(str) return: An iterator of non-projective parses. rtype: iter(DependencyGraph) r)r'r$rr]r'r]r$rTr;FN)start)r'r]ROOT)r_graphrjr/r0rcontainsr2rBrrrFrrootzipr)r rrrr6r$dep_noderootspossible_headsrlr'headsrmheadanalyses_stackanalysisforwardindex_on_stack stack_item orig_lengthrr] head_index head_addressrrs r rz#NonprojectiveDependencyParser.parseDs. &' %f-LE5 (DKK  e $.**113ID KK--446MM**9V+NO!&)Xf-==KK 34 7 !%If 4 (GAtE$V,4F 6 6tT BLLO-5zQ Q  ! !% () u:>5zQs6{+ALLO,H(-c..A(BC(B1B(BCAGq&>!,-2&4Q&7&: ^A./14&( -a0446&*  aY/%*N&+ %a=A--1N',#&nQ&7"8K%+*:!&s5zA~r2!>A).qJ)!}1 .q 1 8 81a I"? (K!O-a0446&*  aY/"&q5C//OOHQK0#GFAFAGq&N!H~~b!A%#%EX^^B%7!%; 'play' | 'man' 'man' -> 'the' | 'in' 'in' -> 'corner' 'corner' -> 'the' 'play' -> 'golf' | 'dachshund' | 'to' 'dachshund' -> 'his' ) themaninrcornertaughthis dachshundtoplaygolfzGraphs:) nltk.grammarr fromstringrKrr)rgrammarndprrs r rrs[.**  G 'N ' 0C YY F ) e r __main__)rUr@rr getLoggerrr=rrrIrSrrrrrrr r rs 6   8 $3$3$v_!2_L  "  &__N NNl   F zFr