gFdZddlZddlZddlmZddlmZddZddZdZ y) zNIST score implementation.N)Counter)ngramsc t|g|g|S)a Calculate NIST score from George Doddington. 2002. "Automatic evaluation of machine translation quality using n-gram co-occurrence statistics." Proceedings of HLT. Morgan Kaufmann Publishers Inc. https://dl.acm.org/citation.cfm?id=1289189.1289273 DARPA commissioned NIST to develop an MT evaluation facility based on the BLEU score. The official script used by NIST to compute BLEU and NIST score is mteval-14.pl. The main differences are: - BLEU uses geometric mean of the ngram overlaps, NIST uses arithmetic mean. - NIST has a different brevity penalty - NIST score from mteval-14.pl has a self-contained tokenizer Note: The mteval-14.pl includes a smoothing function for BLEU score that is NOT used in the NIST score computation. >>> hypothesis1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which', ... 'ensures', 'that', 'the', 'military', 'always', ... 'obeys', 'the', 'commands', 'of', 'the', 'party'] >>> hypothesis2 = ['It', 'is', 'to', 'insure', 'the', 'troops', ... 'forever', 'hearing', 'the', 'activity', 'guidebook', ... 'that', 'party', 'direct'] >>> reference1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'that', ... 'ensures', 'that', 'the', 'military', 'will', 'forever', ... 'heed', 'Party', 'commands'] >>> reference2 = ['It', 'is', 'the', 'guiding', 'principle', 'which', ... 'guarantees', 'the', 'military', 'forces', 'always', ... 'being', 'under', 'the', 'command', 'of', 'the', ... 'Party'] >>> reference3 = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the', ... 'army', 'always', 'to', 'heed', 'the', 'directions', ... 'of', 'the', 'party'] >>> sentence_nist([reference1, reference2, reference3], hypothesis1) # doctest: +ELLIPSIS 3.3709... >>> sentence_nist([reference1, reference2, reference3], hypothesis2) # doctest: +ELLIPSIS 1.4619... :param references: reference sentences :type references: list(list(str)) :param hypothesis: a hypothesis sentence :type hypothesis: list(str) :param n: highest n-gram order :type n: int ) corpus_nist) references hypothesisns N/var/www/openai/venv/lib/python3.12/site-packages/nltk/translate/nist_score.py sentence_nistr sh  |j\1 55c Zt|t|k(sJdt}d}|D]F}|D]?}td|dzD]}|jt |||t|z }AHi|D]4}|dd} | r | |vr|| } n|} t j | ||z d|<6t} t} d\} }td|dzD]'}t||D]\}}t|}g}|D]}t|}t||k\rtt ||n t}t||k\rtt ||n t}||z}tfd|jD}t|j}|dk(rdn||z }|j||||ft|\}} }}| |xx| z cc<| |xx|z cc<| |z } ||z }*d}| D]}| || |z }||z }|t| |zS) a Calculate a single corpus-level NIST score (aka. system-level BLEU) for all the hypotheses and their respective references. :param references: a corpus of lists of reference sentences, w.r.t. hypotheses :type references: list(list(list(str))) :param hypotheses: a list of hypothesis sentences :type hypotheses: list(list(str)) :param n: highest n-gram order :type n: int zBThe number of hypotheses and their reference(s) should be the samerN)rrc34K|]\}}||zyw)N).0_ngramcountinformation_weightss r zcorpus_nist..s'!)? (/%7)?s)lenrrangeupdatermathlogzipsumitemsvaluesappendmaxnist_length_penalty)list_of_references hypothesesr ngram_freqtotal_reference_wordsr referenceir_mgram numerator"nist_precision_numerator_per_ngram$nist_precision_denominator_per_ngraml_refl_sysrhyp_lennist_score_per_ref_ref_len hyp_ngrams ref_ngramsngram_overlaps _numerator _denominator _precision precision denominatorref_lennist_precisionrs @r rrIs ! "c' LKL  J  #I1a!e_!!&A"67% !S^ 3 ! $  f *"6*I-I&*hhy:f;M/Mq&QF#*1&+29(LE5 1a!e_&)*:!+F:q12  69^q5HGF9a01gi",j!8 !)7)=)=)?!  #:#4#4#67 ".!"3Ql9R "))\8D'(.:==O9P 6Iy+w .q 1Y > 1 0 3{ B 3 W E W EA'JJN / .q 1215 6  )# 0 /u= ==r c||z }d|cxkrdkrdnnad\}}tj|tj|dzz }tj|tj|dzzStt |ddS)a" Calculates the NIST length penalty, from Eq. 3 in Doddington (2002) penalty = exp( beta * log( min( len(hyp)/len(ref) , 1.0 ))) where, `beta` is chosen to make the brevity penalty factor = 0.5 when the no. of words in the system output (hyp) is 2/3 of the average no. of words in the reference translation (ref) The NIST penalty is different from BLEU's such that it minimize the impact of the score of small variations in the length of a translation. See Fig. 4 in Doddington (2002) rr)g?g?rg?g)rrexpr"min)r;r0ratioratio_xscore_xbetas r r#r#sx g E5}1}#xx 488G#4#99xxtxx!33443uc?C((r )) __doc__ fractionsr collectionsr nltk.utilrr rr#rr r rIs(! 46na>H)r