g,ddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z m Z mZmZddlmZddlmZmZmZmZmZmZmZddlddlmZmZmZedd Zd Zd3d Z d4d Z!d5d Z"d4dZ#Gdde Z$d6dZ%dZ&e'dfdZ(e'ddfdZ)d7dZ*e'ddfdZ+e'ddfdZ,e'ddddfdZ-e'ddddfdZ.dZ/e'fdZ0e'fdZ1dZ2d Z3d!Z4d8d"Z5d#Z6d$Z7d%Z8d&Z9 d9d'Z:d(Z;d)Zd,Z?d;d-Z@dN) defaultdictdeque)chain combinationsislicetee)pprint)HTTPPasswordMgrWithDefaultRealmProxyBasicAuthHandlerProxyDigestAuthHandler ProxyHandler build_opener getproxiesinstall_opener)*) deprecatedraise_unorderable_types slice_boundszUse help(obj) instead.c Bt|t|ts |j}t |j dt tj|jD] \}}|jdrt|ddr' ttj|}|jdj!dj#d}tj$||}t|t&t(frd |}n |r|d d k(rd |}|j+d t t-j.|ddj1|dd dt3|dzzy#t$r}dt|vrYd}~,d}~wwxYw)Nz# supports the following operations:___deprecated__Fbuiltin(), zcls.rselfzself.z -  )initial_indentsubsequent_indent)str isinstancetype __class__print__name__sortedpydoc allmethodsitems startswithgetattrinspect signature ValueErrorlstriprstripsplitgetattr_static classmethod staticmethodpoptextwrapfilljoinlen)objnamemethodsigeargsmeths >/var/www/openai/venv/lib/python3.12/site-packages/nltk/util.pyusagerC%siH c4 mm S\\N= >?u//4::<= f ??3   6+U 3  g''/0Czz#%%c*006%%c40 d[,7 8$=D d1g'4&>D HHQK MM&$))D/*!,%"%TQ"7  -> CF"  sE<< F FFFcNddl}|jjjdvS)a Return True if this function is run within idle. Tkinter programs that are run in idle should never call ``Tk.mainloop``; so this function should be used to gate all calls to ``Tk.mainloop``. :warning: This function works by checking ``sys.stdin``. If the user has modified ``sys.stdin``, then it may return incorrect results. :rtype: bool rN)PyShellRPCProxy)sysstdinr$r&)rGs rBin_idlerIQs# 99   ' '+B BBc Bttt|||y)z Pretty print a sequence of data items :param data: the data stream to print :type data: sequence or iter :param start: the start position :type start: int :param end: the end position :type end: int N)r listr)datastartends rBprrPfs 4tUC( )*rJcbtdjtj||y)z Pretty print a string, breaking lines on whitespace :param s: the string to print, consisting of words and spaces :type s: str :param width: the display width :type width: int  widthN)r%r9r7wrap)srTs rB print_stringrWts  $))HMM!51 23rJcldjtj|j||S)a# Pretty print a list of text tokens, breaking lines on whitespace :param tokens: the tokens to print :type tokens: list :param separator: the string to use to separate tokens :type separator: str :param width: the display width (default=70) :type width: int rRrS)r9r7rU)tokens separatorrTs rB tokenwrapr[s' 99X]]9>> G HHrJcd}d}d}t|}t|}||krM||krH|dkr||dz }||z}n ||}||z}|dz }tj|s|dz }||kr||krH|S)a Cut off and return a given width of a string Return the same as s[:width] if width >= 0 or s[-width:] if width < 0, as long as s has no unicode combining characters. If it has combining characters make sure the returned string's visible width matches the called-for width. :param s: the string to cut :type s: str :param width: the display_width :type width: int r)absr: unicodedata combining)rVrT chars_sofar width_sofarresult abs_width max_charschars rB cut_stringrhsKK FE IAI  !kI&= 19{Q'(DF]F[>Dd]Fq $$T* 1 K  !kI&= MrJceZdZdZy)Indexcttj|t|D]\}}||j|yN)r__init__rLappend)rpairskeyvalues rBrmzIndex.__init__s2T4(JC I  U # rJN)r& __module__ __qualname__rmrJrBrjrjs$rJrjcttj|tjj |dz|z|j y)a3 Return a string with markers surrounding the matched substrings. Search str for substrings matching ``regexp`` and wrap the matches with braces. This is convenient for learning about regular expressions. :param regexp: The regular expression. :type regexp: str :param string: The string being matched. :type string: str :param left: The left delimiter (printed before the matched substring) :type left: str :param right: The right delimiter (printed after the matched substring) :type right: str :rtype: str z\g<0>N)r%recompileMsubr1)regexpstringleftrights rBre_showr~s7  "**VRTT " & &th'>  PQrJct|dr|jSt|tr%t |5}|jcdddSt d#1swYyxYw)Nreadz2Must be called with a filename or file-like object)hasattrrr"r!openr/)finfiles rB filestringrsPq&vvx As  !W;;=WMNNWs AA&c#Kt|dfg}|rA|j\}||k7r! |jfd||D|r@yy#t$rYwxYww)zTraverse the nodes of a tree in breadth-first order. (No check for cycles.) The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. rc3,K|] }|dzf yw)r^Nrt).0cdepths rB z breadth_first..sD^a^^sN)rpopleftextend TypeError)treechildrenmaxdepthqueuenoders @rB breadth_firstrsk D!9+ E mmo e H   DXd^DD   s.-A% AA%A% A"A%!A""A%Fc#Kt}t}t|dfg}|r|j\}}|j|||k7rj ||D]]} | |vr|j | |dzfn"|r t j d| d|dzd|| f} | |vsI| |j| _ |ryy#t$rYwxYww)a :param tree: the tree root :param children: a function taking as argument a tree node :param maxdepth: to limit the search depth :param verbose: to print warnings when cycles are discarded Yield the edges of a graph in breadth-first order, discarding eventual cycles. The first argument should be the start node; children should be a function taking as argument a graph node and returning an iterator of the node's children. >>> from nltk.util import edge_closure >>> print(list(edge_closure('A', lambda node:{'A':['B','C'], 'B':'C', 'C':'B'}[node]))) [('A', 'B'), ('A', 'C'), ('B', 'C'), ('C', 'B')] rr^zDiscarded redundant search for z at depth  stacklevelN)setrraddrnwarningswarnr) rrrverbose traversededgesrrrchildedges rB edge_closurers"I EE D!9+ E mmo e d H  %d^EI- eUQY%78"$MM"A% SX[\S\R] ^+,!%=D5("  $, $  s7ACAB?"B?9C=C? C C C  Cc X|s t}|s t}d}|jD]}||dd|ddz }|D]_}|jD]8}tdD](}|dt||vs|d||d|dd z }*:|d|dd |dd z }a|d z }|S) a :param edges: the set (or list) of edges of a directed graph. :param shapes: dictionary of strings that trigger a specified shape. :param attr: dictionary with global graph attributes :return: a representation of 'edges' as a string in the DOT graph language. Returns dot_string: a representation of 'edges' as a string in the DOT graph language, which can be converted to an image by the 'dot' program from the Graphviz package, or nltk.parse.dependencygraph.dot2img(dot_string). >>> import nltk >>> from nltk.util import edges2dot >>> print(edges2dot([('A', 'B'), ('A', 'C'), ('B', 'C'), ('C', 'B')])) digraph G { "A" -> "B"; "A" -> "C"; "B" -> "C"; "C" -> "B"; } z digraph G { rz = r^z; r"z " [shape = z]; z" -> "z"; z} )dictr*rangerepr)rshapesattr dot_stringpairrshapers rB edges2dotr-s,  v J a T!WIS11 \\^Ea8tDJ//Ad4j\U1XJd"KKJ!$ $q'&a 66 %J rJc<ttfd||S)a' :param tree: the tree root :param children: a function taking as argument a tree node :param shapes: dictionary of strings that trigger a specified shape. :param attr: dictionary with global graph attributes Build a Minimum Spanning Tree (MST) of an unweighted graph, by traversing the nodes of a tree in breadth-first order, discarding eventual cycles. Return a representation of this MST as a string in the DOT graph language, which can be converted to an image by the 'dot' program from the Graphviz package, or nltk.parse.dependencygraph.dot2img(dot_string). The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. >>> import nltk >>> wn=nltk.corpus.wordnet >>> from nltk.util import unweighted_minimum_spanning_digraph as umsd >>> print(umsd(wn.synset('bound.a.01'), lambda s:sorted(s.also_sees()))) digraph G { "Synset('bound.a.01')" -> "Synset('unfree.a.02')"; "Synset('unfree.a.02')" -> "Synset('confined.a.02')"; "Synset('unfree.a.02')" -> "Synset('dependent.a.01')"; "Synset('unfree.a.02')" -> "Synset('restricted.a.01')"; "Synset('restricted.a.01')" -> "Synset('classified.a.02')"; } c"t|Srl) unweighted_minimum_spanning_dict)rrrs rBz5unweighted_minimum_spanning_digraph..zs?hOPTUrJ)rr)rrrrs`` rB#unweighted_minimum_spanning_digraphrXs)@  U    rJc#xKt}t|dfg}|r|j\}}||vr||j|||k7rW ||D]J}||vr|j ||dzf|s!t j dj||dzdL |ryy#t$rYwxYww)a@ :param tree: the tree root :param children: a function taking as argument a tree node :param maxdepth: to limit the search depth :param verbose: to print warnings when cycles are discarded :return: the tree in breadth-first order Adapted from breadth_first() above, to discard cycles. Traverse the nodes of a tree in breadth-first order, discarding eventual cycles. The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. rr^-Discarded redundant search for {} at depth {}rrN) rrrrrnrrformatr) rrrrrrrrrs rBacyclic_breadth_firstrs I D!9+ E mmo e 9    d H  %d^EI- eUQY%78  KRR %uqy()  , $  s6A B:(B+9,B+%B:)B:+ B74B:6B77B:c N||h}|g}|dk7r ||D]p}||vr(|j||t|||dz ||gz }/|r*tjdj ||dz d|s^|d|d|dz d|dgz }r |S|r||gz }|S#t $rY|SwxYw) a+ :param tree: the tree root :param children: a function taking as argument a tree node :param depth: the maximum depth of the search :param cut_mark: the mark to add when cycles are truncated :param traversed: the set of traversed nodes :param verbose: to print warnings when cycles are discarded :return: the tree in depth-first order Traverse the nodes of a tree in depth-first order, discarding eventual cycles within any branch, adding cut_mark (when specified) if cycles were truncated. The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. Catches all cycles: >>> import nltk >>> from nltk.util import acyclic_depth_first as acyclic_tree >>> wn=nltk.corpus.wordnet >>> from pprint import pprint >>> pprint(acyclic_tree(wn.synset('dog.n.01'), lambda s:sorted(s.hypernyms()),cut_mark='...')) [Synset('dog.n.01'), [Synset('canine.n.02'), [Synset('carnivore.n.01'), [Synset('placental.n.01'), [Synset('mammal.n.01'), [Synset('vertebrate.n.01'), [Synset('chordate.n.01'), [Synset('animal.n.01'), [Synset('organism.n.01'), [Synset('living_thing.n.01'), [Synset('whole.n.02'), [Synset('object.n.01'), [Synset('physical_entity.n.01'), [Synset('entity.n.01')]]]]]]]]]]]]], [Synset('domestic_animal.n.01'), "Cycle(Synset('animal.n.01'),-3,...)"]] rr^rrCycle(,r)racyclic_depth_firstrrrrrrrcut_markrrout_treers rBrrsTF vH z !$ )MM%(+!8UQY)!H  KRR %uqy()    veWAeai[(1%M$NN%(. O XJ O    O  sA%B5B B$#B$c L||h}|g}|dk7r~ ||D]o}||vr'|t|||dz ||j|hgz }.|r*tjdj ||dz d|s]|d|d|dz d|dgz }q |S|r||gz }|S#t $rY|SwxYw) a :param tree: the tree root :param children: a function taking as argument a tree node :param depth: the maximum depth of the search :param cut_mark: the mark to add when cycles are truncated :param traversed: the set of traversed nodes :param verbose: to print warnings when cycles are discarded :return: the tree in depth-first order Adapted from acyclic_depth_first() above, to traverse the nodes of a tree in depth-first order, discarding eventual cycles within the same branch, but keep duplicate paths in different branches. Add cut_mark (when defined) if cycles were truncated. The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. Catches only only cycles within the same branch, but keeping cycles from different branches: >>> import nltk >>> from nltk.util import acyclic_branches_depth_first as tree >>> wn=nltk.corpus.wordnet >>> from pprint import pprint >>> pprint(tree(wn.synset('certified.a.01'), lambda s:sorted(s.also_sees()), cut_mark='...', depth=4)) [Synset('certified.a.01'), [Synset('authorized.a.01'), [Synset('lawful.a.01'), [Synset('legal.a.01'), "Cycle(Synset('lawful.a.01'),0,...)", [Synset('legitimate.a.01'), '...']], [Synset('straight.a.06'), [Synset('honest.a.01'), '...'], "Cycle(Synset('lawful.a.01'),0,...)"]], [Synset('legitimate.a.01'), "Cycle(Synset('authorized.a.01'),1,...)", [Synset('legal.a.01'), [Synset('lawful.a.01'), '...'], "Cycle(Synset('legitimate.a.01'),0,...)"], [Synset('valid.a.01'), "Cycle(Synset('legitimate.a.01'),0,...)", [Synset('reasonable.a.01'), '...']]], [Synset('official.a.01'), "Cycle(Synset('authorized.a.01'),1,...)"]], [Synset('documented.a.01')]] rr^rrrrrr)acyclic_branches_depth_firstunionrrrrrs rBrrsdF vH z !$ )4!$!AI$%OOUG4 !H KRR %uqy()    veWAeai[(1%M$NN+(4 O XJ O    O  sA$B4B B#"B#cN|g||Dcgc]}t||c}zScc}w)a :param node: the root node :param dic: the dictionary of children Convert acyclic dictionary 'dic', where the keys are nodes, and the values are lists of children, to output tree suitable for pprint(), starting at root 'node', with subtrees as nested lists.)acyclic_dic2tree)rdicrs rBrrKs/ 6s4yIye%eS1yI IIIs"c*t}t|g}|h}i}|ru|j}g||<||vrY|j|||D]=}||vs||j ||j ||j|?|ru|S)aH :param tree: the tree root :param children: a function taking as argument a tree node Output a dictionary representing a Minimum Spanning Tree (MST) of an unweighted graph, by traversing the nodes of a tree in breadth-first order, discarding eventual cycles. The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. >>> import nltk >>> from nltk.corpus import wordnet as wn >>> from nltk.util import unweighted_minimum_spanning_dict as umsd >>> from pprint import pprint >>> pprint(umsd(wn.synset('bound.a.01'), lambda s:sorted(s.also_sees()))) {Synset('bound.a.01'): [Synset('unfree.a.02')], Synset('classified.a.02'): [], Synset('confined.a.02'): [], Synset('dependent.a.01'): [], Synset('restricted.a.01'): [Synset('classified.a.02')], Synset('unfree.a.02'): [Synset('confined.a.02'), Synset('dependent.a.01'), Synset('restricted.a.01')]} )rrrrrn)rrrragendamstdicrrs rBrrVs8I 4&MEVF F }}t y MM$ !$&4L''.LL'JJu% (  MrJc.t|t||S)aS :param tree: the tree root :param children: a function taking as argument a tree node Output a Minimum Spanning Tree (MST) of an unweighted graph, by traversing the nodes of a tree in breadth-first order, discarding eventual cycles. The first argument should be the tree root; children should be a function taking as argument a tree node and returning an iterator of the node's children. >>> import nltk >>> from nltk.util import unweighted_minimum_spanning_tree as mst >>> wn=nltk.corpus.wordnet >>> from pprint import pprint >>> pprint(mst(wn.synset('bound.a.01'), lambda s:sorted(s.also_sees()))) [Synset('bound.a.01'), [Synset('unfree.a.02'), [Synset('confined.a.02')], [Synset('dependent.a.01')], [Synset('restricted.a.01'), [Synset('classified.a.02')]]]] )rr)rrs rB unweighted_minimum_spanning_treers0 D"B4"R SSrJc d}dg} |jtjtj |jtj d |jtjd|jd|D]}|s t||}|}n|s7tddj|Dcgc]}|st|c}z|fS#t$rYwxYw#tt f$rYwxYw#tt f$rYwxYw#ttf$rYwxYwcc}w)at Given a byte string, attempt to decode it. Tries the standard 'UTF8' and 'latin-1' encodings, Plus several gathered from locale information. The calling program *must* first call:: locale.setlocale(locale.LC_ALL, '') If successful it returns ``(decoded_unicode, successful_encoding)``. If unsuccessful it raises a ``UnicodeError``. Nzutf-8r^zlatin-1z?Unable to decode input data. Tried the following encodings: %s.r) rnlocale nl_langinfoCODESETAttributeError getlocale IndexErrorgetdefaultlocaler! UnicodeError LookupErrorr9r)rMsuccessful_encoding encodingsencdecodeds rBguess_encodingrs` I ++FNN;< ))+A./ 002156 Y  $nG"%    1iii?is3ci?@ A  ,--C     J '    J '   k*   @sR2C/&C>!&D!D( D= D=/ C;:C;>DDD%$D%(D:9D:cnt}|Dcgc]}||vs|j|r|c}Scc}wrl)rr)xsseenxs rB unique_listrs1 5D ?r!Qd]488A;Ar ?? ?s 222ctt}|D]8}t||dr||D]}||j|1||||<:|S)N__iter__)rrLrrn)d inverted_dictrpterms rB invert_dictrs]%M 1S6: &#d#**3/%(M!C& !  rJc|rd}nd}|Dcic]}|||j}}|Dcic] }||| }}|D]n}||}||}|s|j} |j| ||j| || z}||j | || z}||z}|r_p|Scc}wcc}w)a Calculate the transitive closure of a directed graph, optionally the reflexive transitive closure. The algorithm is a slight modification of the "Marking Algorithm" of Ioannidis & Ramakrishnan (1998) "Efficient Transitive Closure Algorithms". :param graph: the initial graph, represented as a dictionary of sets :type graph: dict(set) :param reflexive: if set, also make the closure reflexive :type reflexive: bool :rtype: dict(set) c|hSrlrtks rBrz$transitive_closure..saSrJctSrl)rrs rBrz$transitive_closure..sSUrJ)copyr6r setdefaultget) graph reflexivebase_setr agenda_graph closure_graphirclosurejs rBtransitive_closurers "0561AuQx}}&L6-23UQ ^UM3 a" A KKN }//8A;? ?G l&&q(1+6 6F g F  73s B6B;c~i}|D]5}||D]+}|j|tj|-7|S)z Inverts a directed graph. :param graph: the graph, represented as a dictionary of sets :type graph: dict(set) :return: the inverted graph :rtype: dict(set) )rrr)rinvertedrprqs rB invert_graphr!sEH3ZE   su - 1 1# 6  OrJctdNz>To remove HTML markup, use BeautifulSoup's get_text() functionNotImplementedError)htmls rB clean_htmlr6 H rJctdrr)urls rB clean_urlr<rrJcg}|D]d}t|ttfs|g}|D]D}t|ttfr|jt |4|j |Ff|S)z Flatten a list. >>> from nltk.util import flatten >>> flatten(1, 2, ['b', 'a' , ['c', 'd']], 3) [1, 2, 'b', 'a', 'c', 'd', 3] :param args: items and lists to be combined into a single list :rtype: list )r"rLtuplerflattenrn)r@rlitems rBrrGsb A !dE]+AD$u .'  HrJcpt|}|rt|f|dz z|}|rt||f|dz z}|S)a Returns a padded sequence of items before ngram extraction. >>> list(pad_sequence([1,2,3,4,5], 2, pad_left=True, pad_right=True, left_pad_symbol='', right_pad_symbol='')) ['', 1, 2, 3, 4, 5, ''] >>> list(pad_sequence([1,2,3,4,5], 2, pad_left=True, left_pad_symbol='')) ['', 1, 2, 3, 4, 5] >>> list(pad_sequence([1,2,3,4,5], 2, pad_right=True, right_pad_symbol='')) [1, 2, 3, 4, 5, ''] :param sequence: the source data to be padded :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param pad_left: whether the ngrams should be left-padded :type pad_left: bool :param pad_right: whether the ngrams should be right-padded :type pad_right: bool :param left_pad_symbol: the symbol to use for left padding (default is None) :type left_pad_symbol: any :param right_pad_symbol: the symbol to use for right padding (default is None) :type right_pad_symbol: any :rtype: sequence or iter r^)iterr)sequencenpad_left pad_rightleft_pad_symbolright_pad_symbols rB pad_sequencerdsJ@H~H/+q1u5x@$4#6!a%#@A OrJc+Kt||fi|}t|}tt|||}t ||k(r t ||D] }|j |t |"yw)a Return the ngrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import ngrams >>> list(ngrams([1,2,3,4,5], 3)) [(1, 2, 3), (2, 3, 4), (3, 4, 5)] Wrap with list for a list version of this function. Set pad_left or pad_right to true in order to get additional ngrams: >>> list(ngrams([1,2,3,4,5], 2, pad_right=True)) [(1, 2), (2, 3), (3, 4), (4, 5), (5, None)] >>> list(ngrams([1,2,3,4,5], 2, pad_right=True, right_pad_symbol='')) [(1, 2), (2, 3), (3, 4), (4, 5), (5, '')] >>> list(ngrams([1,2,3,4,5], 2, pad_left=True, left_pad_symbol='')) [('', 1), (1, 2), (2, 3), (3, 4), (4, 5)] >>> list(ngrams([1,2,3,4,5], 2, pad_left=True, pad_right=True, left_pad_symbol='', right_pad_symbol='')) [('', 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, '')] :param sequence: the source data to be converted into ngrams :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param pad_left: whether the ngrams should be left-padded :type pad_left: bool :param pad_right: whether the ngrams should be right-padded :type pad_right: bool :param left_pad_symbol: the symbol to use for left padding (default is None) :type left_pad_symbol: any :param right_pad_symbol: the symbol to use for right padding (default is None) :type right_pad_symbol: any :rtype: sequence or iter )maxlenN)rr rrr:rrn)r r kwargsitwindowrs rBngramsrsmHHa262H hB 6"a= +F 6{aFm  aFmsA1A3c+:Kt|dfi|Ed{y7w)a Return the bigrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import bigrams >>> list(bigrams([1,2,3,4,5])) [(1, 2), (2, 3), (3, 4), (4, 5)] Use bigrams for a list version of this function. :param sequence: the source data to be converted into bigrams :type sequence: sequence or iter :rtype: iter(tuple) rNrr rs rBbigramsr h,V,,, c+:Kt|dfi|Ed{y7w)a Return the trigrams generated from a sequence of items, as an iterator. For example: >>> from nltk.util import trigrams >>> list(trigrams([1,2,3,4,5])) [(1, 2, 3), (2, 3, 4), (3, 4, 5)] Use trigrams for a list version of this function. :param sequence: the source data to be converted into trigrams :type sequence: sequence or iter :rtype: iter(tuple) rNrrs rBtrigramsrrrc+K|dk(r t|}t||||fi|}tt ||}|rOt |t|dzD]}t |d| |jt||d=|rNyy#t$rt|}t|}YwxYw#t$rY8wxYww)av Returns all possible ngrams generated from a sequence of items, as an iterator. >>> sent = 'a b c'.split() New version outputs for everygrams. >>> list(everygrams(sent)) [('a',), ('a', 'b'), ('a', 'b', 'c'), ('b',), ('b', 'c'), ('c',)] Old version outputs for everygrams. >>> sorted(everygrams(sent), key=len) [('a',), ('b',), ('c',), ('a', 'b'), ('b', 'c'), ('a', 'b', 'c')] >>> list(everygrams(sent, max_len=2)) [('a',), ('a', 'b'), ('b',), ('b', 'c'), ('c',)] :param sequence: the source data to be converted into ngrams. If max_len is not provided, this sequence will be loaded into memory :type sequence: sequence or iter :param min_len: minimum length of the ngrams, aka. n-gram order/degree of ngram :type min_len: int :param max_len: maximum length of the ngrams (set to length of sequence by default) :type max_len: int :param pad_left: whether the ngrams should be left-padded :type pad_left: bool :param pad_right: whether the ngrams should be right-padded :type pad_right: bool :rtype: iter(tuple) rr^Nr) r:rrLrrrrrnnext StopIteration)r min_lenmax_lenr r rhistory ngram_lens rB everygramsr&sD"} $(mG HgxMfMH6(G,-G wG q(89I +, ,:  NN4> * AJ  $H~H(mG $$   sRB> B AB>(B/B>B> B,)B>+B,,B>/ B;8B>:B;;B>c+Kd|vsd|vr t||fi|}t}t|||zd|D]/}|dd}|dd}t||dz D]}|d|ur ||z1yw)a Returns all possible skipgrams generated from a sequence of items, as an iterator. Skipgrams are ngrams that allows tokens to be skipped. Refer to http://homepages.inf.ed.ac.uk/ballison/pdf/lrec_skipgrams.pdf >>> sent = "Insurgents killed in ongoing fighting".split() >>> list(skipgrams(sent, 2, 2)) [('Insurgents', 'killed'), ('Insurgents', 'in'), ('Insurgents', 'ongoing'), ('killed', 'in'), ('killed', 'ongoing'), ('killed', 'fighting'), ('in', 'ongoing'), ('in', 'fighting'), ('ongoing', 'fighting')] >>> list(skipgrams(sent, 3, 2)) [('Insurgents', 'killed', 'in'), ('Insurgents', 'killed', 'ongoing'), ('Insurgents', 'killed', 'fighting'), ('Insurgents', 'in', 'ongoing'), ('Insurgents', 'in', 'fighting'), ('Insurgents', 'ongoing', 'fighting'), ('killed', 'in', 'ongoing'), ('killed', 'in', 'fighting'), ('killed', 'ongoing', 'fighting'), ('in', 'ongoing', 'fighting')] :param sequence: the source data to be converted into trigrams :type sequence: sequence or iter :param n: the degree of the ngrams :type n: int :param k: the skip distance :type k: int :rtype: iter(tuple) r r T)r rNr^r)robjectrr) r r rrSENTINELngramheadtail skip_tails rB skipgramsr.#s,V{f4!6v6 xH!a%4(SRayQRy%dAE2I}(" "3TsA%A'c2|dz}t|}d}d}t|dr-tj|jj dz }n6|j dd|jdz }|j d|i}||kr ||f}||zdz} |j| r || \} } nzd} |j td| dz | dkDr|j|j} |j} | dk7rn|| zdz} | |dz k(ryk||kr| | f|| <| |kDr|| dz k7sJd| dz }n:| d||k(r| S| |kDr|| dz k7sJd| dz }n| |kr| t| zdz }|dz }||f} || k(ry||kr y) a Return the line from the file with first word key. Searches through a sorted file using the binary search algorithm. :type file: file :param file: the file to be searched through. :type key: str :param key: the identifier we are searching for. rrr<r^rNr]z infinite loop) r:rosstatr<st_sizeseektellrmax discard_linereadline) filerpcache cacheDepthkeylenrN currentDepthrO lastStatemiddleoffsetline thisStates rBbinary_search_filerBOs )C XF ELtVggdii ((1, !QiikAo !  } #+3J #+!# 99V  =LFDD #a!,-A:%%'}}2:&.Q.S1W$j(!'f C<&1*$ 5o 5$1*C '6]c !K CZ&1*$ 5o 5$1*C CZSY&*E 3J  !S #+V rJcP| td}t||d}t |}|St }|j d||||jt||jt|t|y#t$r}td|d}~wwxYw)a Set the HTTP proxy for Python to download through. If ``proxy`` is None then tries to set proxy from environment or system settings. :param proxy: The HTTP proxy server to use. For example: 'http://proxy.example.com:3128/' :param user: The username to authenticate with. Use None to disable authentication. :param password: The password to authenticate with. Nhttpz'Could not detect default proxy settings)httpsrD)realmuriuserpasswd) rKeyErrorr/r rr add_password add_handlerr r r)proxyrHpasswordr? proxy_handleropenerpassword_managers rB set_proxyrRs } OL(E !5%!@AM - (F :<%%De$x%X01ABC12BCD6 OFGQ N Os B B% B  B%cvd|dzz}t|ru|jr|jjs |dz|_|D]}t||dz|jr|jjs||_yy|r/|jr|jjs||_yyy)a Recursive function to indent an ElementTree._ElementInterface used for pretty printing. Run indent on elem and then output in the normal way. :param elem: element to be indented. will be modified. :type elem: ElementTree._ElementInterface :param level: level of indentation for this element :type level: nonnegative integer :rtype: ElementTree._ElementInterface :return: Contents of elem indented to reflect its structure rRz r^N)r:textstripelementtree_indentr,)elemlevelrs rBrVrVs ut|A 4yyy  1DDID tUQY /yy  1DI!2 $))499??+<DI,=5rJcd|cxkr|kr=nyd\}}tdt|||z dzD]}||z}||z}|dz}||zSy)a9 This function is a fast way to calculate binomial coefficients, commonly known as nCk, i.e. the number of combinations of n things taken k at a time. (https://en.wikipedia.org/wiki/Binomial_coefficient). This is the *scipy.special.comb()* with long integer computation but this approximation is faster, see https://github.com/nltk/nltk/issues/1181 >>> choose(4, 2) 6 >>> choose(6, 2) 15 :param n: The number of things. :type n: int :param r: The number of times a thing is taken. :type r: int r)r^r^r^)rmin)r rntokktokts rBchooser^si& A{{ dq#aQ-!+,A AID AID FA-t|rJcNt|\}}t|dt||S)z$s -> (s0,s1), (s1,s2), (s2, s3), ...N)rr zip)iterableabs rBpairwiserds$ x=DAqDM q!9rJcddlm}mddlm}|r||n|}|dkr t |S||fd|DS)Nr)Paralleldelayed)tqdmr^)n_jobsc3:K|]}|ywrlrt)rr@rgfuncs rBrz)parallelize_preprocess..s%OhdmgdmD&9hs)joblibrfrgrhmap)rkiterator processes progress_barrfrhrgs` @rBparallelize_preprocessrqsB(!-tH~8HA~4"" %89 %%Oh%O OOrJ)rN)F)rrr){})NN)F)FFNN)r^rFF)Nr)Nr])r)Fr-rr0r(rvr7r`r collectionsrr itertoolsrrrrr urllib.requestr r r r rrrnltk.collectionsnltk.internalsrrrrCrIrPrWr[rhrjr~rr rrrrrrrrrrrrrrrrrrrrrrr&r.rBrRrVr^rdrqrtrJrBrzs *66LL $%# &# V C* + 4 I P$K$R2O"&2!%r5'T(V8