gyRdZddlZddlZddlmZddlmZddlmZGddeZ y)a The Sonority Sequencing Principle (SSP) is a language agnostic algorithm proposed by Otto Jesperson in 1904. The sonorous quality of a phoneme is judged by the openness of the lips. Syllable breaks occur before troughs in sonority. For more on the SSP see Selkirk (1984). The default implementation uses the English alphabet, but the `sonority_hiearchy` can be modified to IPA or any other alphabet for the use-case. The SSP is a universal syllabification algorithm, but that does not mean it performs equally across languages. Bartlett et al. (2009) is a good benchmark for English accuracy if utilizing IPA (pg. 311). Importantly, if a custom hierarchy is supplied and vowels span across more than one level, they should be given separately to the `vowels` class attribute. References: - Otto Jespersen. 1904. Lehrbuch der Phonetik. Leipzig, Teubner. Chapter 13, Silbe, pp. 185-203. - Elisabeth Selkirk. 1984. On the major class features and syllable theory. In Aronoff & Oehrle (eds.) Language Sound Structure: Studies in Phonology. Cambridge, MIT Press. pp. 107-136. - Susan Bartlett, et al. 2009. On the Syllabification of Phonemes. In HLT-NAACL. pp. 308-316. N) punctuation) TokenizerI)ngramsc*eZdZdZddZdZdZdZy)SyllableTokenizera Syllabifies words based on the Sonority Sequencing Principle (SSP). >>> from nltk.tokenize import SyllableTokenizer >>> from nltk import word_tokenize >>> SSP = SyllableTokenizer() >>> SSP.tokenize('justification') ['jus', 'ti', 'fi', 'ca', 'tion'] >>> text = "This is a foobar-like sentence." >>> [SSP.tokenize(token) for token in word_tokenize(text)] [['This'], ['is'], ['a'], ['foo', 'bar', '-', 'li', 'ke'], ['sen', 'ten', 'ce'], ['.']] c|s |dk(rgd}|d|_i|_t|D]F\}}|D]<}t||z }||j|<||j|j <>Hy)a :param lang: Language parameter, default is English, 'en' :type lang: str :param sonority_hierarchy: Sonority hierarchy according to the Sonority Sequencing Principle. :type sonority_hierarchy: list(str) en)aeiouylmnrwzvsf bcdgtkpqxhjrN)vowels phoneme_map enumeratelenupper)selflangsonority_hierarchyilevelcsonority_levels V/var/www/openai/venv/lib/python3.12/site-packages/nltk/tokenize/sonority_sequencing.py__init__zSyllableTokenizer.__init__9s"ddl" )+ !"45HAu!$%7!81!<&4  #.<  +6c g}|D]#} |j||j|f%|S#t$r|dvr|tvr|t j dj ||j|t|jjf||jvr)|xj|z c_ n|j|dfYwxYw)af Assigns each phoneme its value from the sonority hierarchy. Note: Sentence/text has to be tokenized first. :param token: Single word or token :type token: str :return: List of tuples, first element is character/phoneme and second is the soronity value. :rtype: list(tuple(str, int)) 0123456789zECharacter not defined in sonority_hierarchy, assigning as vowel: '{}') appendrKeyErrorrwarningswarnformatmaxvaluesr)rtokensyllables_valuesrs r assign_valueszSyllableTokenizer.assign_valuesUsA 5 ''D,<,'O'1,#**58+;<#**844"rc&|j}tfd|jDdkrgSg}|dd}t|dD]}t |\}}|\}} } |d} | dk(r%|j ||j | d}C|| cxk\r| k(rnn|| z }|j |d}j|| cxkDr| krnn|j |d}|| z }|| z }||ddz }|j ||j |S)a" Apply the SSP to return a list of syllables. Note: Sentence/text has to be tokenized first. :param token: Single word or token :type token: str :return syllable_list: Single word or token broken up into syllables. :rtype: list(str) c3@K|]}j|yw)N)count).0xr's r z-SyllableTokenizer.tokenize..s3{!u{{1~{sr)nrr+)r)sumrrzipr r6) rr'r(r1r5trigramphonemesr& prev_value focal_value next_value focal_phonemes ` rtokenizezSyllableTokenizer.tokenizes@ --e4 3t{{3 3q 87N #A&q).!4G"G} Hf28 /J Z$QKMb $$X.$$]3{8j8M)$$X.k6J6$$X.M)M)154 $R(++X&&&}55rN)r F)__name__ __module__ __qualname____doc__rr)r6rHrrrr+s =8 8B06rr) rLr-r"stringrnltk.tokenize.apir nltk.utilrrrMrrrQs)4 (W6 W6r