gddlmZmZddlZddlmZddlZddlmZddl GddeZ Gd d e Z Gd d e Z Gd de Z Gdde ZGdde ZGdde ZGdde Zy))ABCabstractmethodN)Counterload_nltk_punkt)*c*eZdZdZededefdZy)ChunkingStrategyz6 Abstract base class for chunking strategies. textreturncy)z Abstract method to chunk the given text. Args: text (str): The text to chunk. Returns: list: A list of chunks. Nselfr s O/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/chunking_strategy.pychunkzChunkingStrategy.chunks N)__name__ __module__ __qualname____doc__rstrlistrrrrr r s)  #  $    rr c eZdZdZdedefdZy)IdentityChunkingzJ Chunking strategy that returns the input text as a single chunk. r r c|gSNrrs rrzIdentityChunking.chunk s v rN)rrrrrrrrrrrrs#$rrc(eZdZdZddZdedefdZy) RegexChunkingzR Chunking strategy that splits text based on regular expression patterns. Nc |dg}||_y)z Initialize the RegexChunking object. Args: patterns (list): A list of regular expression patterns to split text. Nz\n\n)patterns)rr"kwargss r__init__zRegexChunking.__init__(s  yH  rr r c|g}|jD]2}g}|D]'}|jtj||)|}4|Sr)r"extendresplit)rr paragraphspatternnew_paragraphs paragraphs rrzRegexChunking.chunk3sLV }}GN' %%bhhw &BC('J % rrrrrrr$rrrrrrr r $s !#$rr c&eZdZdZdZdedefdZy)NlpSentenceChunkingz\ Chunking strategy that splits text into sentences using NLTK's sentence tokenizer. c ty)z< Initialize the NlpSentenceChunking object. Nr)rr#s rr$zNlpSentenceChunking.__init__As  rr r cddlm}||}|Dcgc]}|j}}tt |Scc}w)Nr) sent_tokenize) nltk.tokenizer2striprset)rr r2 sentencessentsenss rrzNlpSentenceChunking.chunkHs> 0!$' *34)$)4CI5sANr-rrrr/r/=s # $ rr/cHeZdZdZd dZdedefdZdedefdZdedefdZ y) TopicSegmentationChunkingz Chunking strategy that segments text into topics using NLTK's TextTilingTokenizer. How it works: 1. Segment the text into topics using TextTilingTokenizer 2. Extract keywords for each topic segment c Xddl}|jj|_||_y)z Initialize the TopicSegmentationChunking object. Args: num_keywords (int): The number of keywords to extract for each topic segment. rN)nltktokenizeTextTilingTokenizer tokenizer num_keywords)rr@r#nls rr$z"TopicSegmentationChunking.__init___s# 88:(rr r c<|jj|}|Sr)r?r=)rr segmented_topicss rrzTopicSegmentationChunking.chunkjs>>2248rczddl}|jj|}|Dcgc]M}||jjj dvs+|t jvs>|jO}}t|}|j|jDcgc]\}}| }}}|Scc}wcc}}w)Nrenglish) r<toknize word_tokenizecorpus stopwordswordsstring punctuationlowerr most_commonr@) rr rAtokenstoken freq_distwordfreqkeywordss rextract_keywordsz*TopicSegmentationChunking.extract_keywordsos))$/-3NVEuBIIDWDWD]D]^gDh7hmr{A{M{MnM%++-VNFO +4+@+@ARAR+ST+SZT4D+ST NUs,B2B2$B2! B7cr|j|}|Dcgc]}||j|f}}|Scc}wr)rrU)rr segmentssegmentsegments_with_topicss rchunk_with_topicsz+TopicSegmentationChunking.chunk_with_topicszsC::d#YabYag$*?*?*H IYab## cs4N)) rrrrr$rrrrUrZrrrr:r:VsE ) # $ S T $c$d$rr:c(eZdZdZddZdedefdZy)FixedLengthWordChunkingz Chunking strategy that splits text into fixed-length word chunks. How it works: 1. Split the text into words 2. Create chunks of fixed length 3. Return the list of chunks c ||_y)z Initialize the fixed-length word chunking strategy with the given chunk size. Args: chunk_size (int): The size of each chunk in words. N) chunk_size)rr_r#s rr$z FixedLengthWordChunking.__init__s %rr r c |j}tdt||jDcgc]#}dj ||||jz%c}Scc}wNr )r(rangelenr_join)rr rJis rrzFixedLengthWordChunking.chunksT @EaUUYUdUd@ef@e1qT__!456@efffs(AN)dr-rrrr]r]s"%g#g$grr]c(eZdZdZddZdedefdZy)SlidingWindowChunkingz Chunking strategy that splits text into overlapping word chunks. How it works: 1. Split the text into words 2. Create chunks of fixed length 3. Return the list of chunks c ||_||_y)a Initialize the sliding window chunking strategy with the given window size and step size. Args: window_size (int): The size of the sliding window in words. step (int): The step size for sliding the window in words. N) window_sizestep)rrkrlr#s rr$zSlidingWindowChunking.__init__s' rr r c|j}g}t||jkr|gStdt||jz dz|jD]4}dj ||||jz}|j |6|jzt|kr.|j dj ||j d|S)Nrrrb)r(rdrkrcrlreappend)rr rJchunksrfrs rrzSlidingWindowChunking.chunks  u:)) )6Mq#e*t'7'77!;TYYGAHHU1Q)9)9%9:;E MM% H t #e* , MM#((5$*:*:):);#<= > rN)rg2r-rrrriris #$rric(eZdZdZddZdedefdZy)OverlappingWindowChunkinga Chunking strategy that splits text into overlapping word chunks. How it works: 1. Split the text into words using whitespace 2. Create chunks of fixed length equal to the window size 3. Slide the window by the overlap size 4. Return the list of chunks c ||_||_y)a) Initialize the overlapping window chunking strategy with the given window size and overlap size. Args: window_size (int): The size of the window in words. overlap (int): The size of the overlap between consecutive chunks in words. N)rkoverlap)rrkrtr#s rr$z"OverlappingWindowChunking.__init__s' rr r cF|j}g}t||jkr|gSd}|t|krc||jz}dj|||}|j ||t|k\r |S||j z }|t|krc|Sra)r(rdrkrernrt)rr rJrostartendrs rrzOverlappingWindowChunking.chunks  u:)) )6Mc%j $***CHHU5-.E MM% c%j  $,,&Ec%j  rN)irgr-rrrrrrrs #$rrr)abcrrr' collectionsrrK model_loaderrutilsr rr r/r:r]rirrrrrr|s#  ) s &'$2*2)$ 0)$Xg.g.$,$L( 0(r