g x d dl mZ d dlmZ d dlmZmZ ej Z eedddd Z de _
d Z d
Z y ) )partial)chain)
everygrams pad_sequenceTzz)pad_leftleft_pad_symbol pad_rightright_pad_symbolzPads both ends of a sentence to length specified by ngram order.
Following convention pads the start of sentence pads its end.
c D t t t || | S )zpHelper with some useful defaults.
Applies pad_both_ends to sentence and follows it up with everygrams.
nmax_len)r list
pad_both_ends)ordersentences J/var/www/openai/venv/lib/python3.12/site-packages/nltk/lm/preprocessing.pypadded_everygramsr s
d=U;z,padded_everygram_pipeline..1 s& LttDD)*E : :ts &))r r flattenmap)r textr s ` @r padded_everygram_pipeliner " s2 %0JLtLJ%& r N)
functoolsr itertoolsr nltk.utilr r
from_iterabler r __doc__r r r r r' sI .
Mr