gPddlZddlmZddlZddlmZmZmZmZm Z ddl Z ddl m Z ddl Z ddl Z ddlZddlmZddlmZddlmZddlmZdd lmZmZd d lmZddlZddlZddlZdd lmZddl Z d e_!dede"fdZ#GddZ$y)N)Path)DictListTupleOptionalAny)tqdm) BM25Okapi) word_tokenize) stopwords)WordNetLemmatizer) completionbatch_completion) AsyncLogger)fnmatchF file_pathreturnctj}|jd5tfddD]}|j | ddd|j S#1swY|j SxYw)z/Compute MD5 hash for the file's entire content.rbc&jdS)Ni)read)fsD/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/llmtxt.pyz$_compute_file_hash..s !&&,rN)hashlibmd5openiterupdate hexdigest)rhash_md5chunkrs @r_compute_file_hashr%sg{{}H  .4E OOE "5          s &A&&A>c 2eZdZ d"dedeedededdf dZdeeddfd Z d e de e ee ffd Z d edefd Zd ededdfdZde dee fdZd#de fdZd#d$dZd%de de ddfdZd&dee de de fdZd'de dede fdZdeededee defd Zd$d!Zy)(AsyncLLMTextManagerNdocs_dirloggermax_concurrent_calls batch_sizerc||_||_||_||_d|_i|_g|_|jdz |_y)Nzbm25_index.pkl)r(r)r*r+ bm25_index document_maptokenized_factsbm25_index_file)selfr(r)r*r+s r__init__zAsyncLLMTextManager.__init__!sJ!  $8!$,.*,#}}/??r doc_batchc Kg}|D]9} t|dd5}|j|jddd;d}|Dcgc]}|r d |d |d g}} td |d } t| |D]\} } tjd| jdjjtj} | s|jjd|otj dd| j#dj%} | rX|j'd} t| dd5}|j)| ddd|jj+d| n|jjd|y#1swYbxYw#t$rF}|jj d|dt ||jdYd}~d}~wwxYwcc}w#1swYxYw#t$r5}|jj d|dt |Yd}~d}~wwxYw#t$r1}|jj dt |Yd}~yd}~wwxYww)z(Process a batch of documents in parallelrutf-8encodingNError reading : aGiven a documentation file, generate a list of atomic facts where each fact: 1. Represents a single piece of knowledge 2. Contains variations in terminology for the same concept 3. References relevant code patterns if they exist 4. Is written in a way that would match natural language queries Each fact should follow this format: : | | Example Facts: browser_config: Configure headless mode and browser type for AsyncWebCrawler | headless, browser_type, chromium, firefox | BrowserConfig(browser_type="chromium", headless=True) redis_connection: Redis client connection requires host and port configuration | redis setup, redis client, connection params | Redis(host='localhost', port=6379, db=0) pandas_filtering: Filter DataFrame rows using boolean conditions | dataframe filter, query, boolean indexing | df[df['column'] > 5] Wrap your response in ... tags. userz* Generate index for this documentation: )rolecontentz"anthropic/claude-3-5-sonnet-latest)modelmessages logger_fnz(.*?)rz(No ... content found for z\n\s*\n r.q.mdwzCreated index file: z'No index content found in response for zError processing response for zError in batch completion: )rappendr Exceptionr)errorstrrzipresearchchoicesmessager>DOTALLwarningsubgroupstrip with_suffixwriteinfo)r1r3contentsrrepromptr> messages_list responsesresponseindex_content_match index_content index_files r_process_document_batchz+AsyncLLMTextManager._process_document_batch1sx"I $)S7;qOOAFFH-<#,$  $w vh6deldm,no $  " F(:&I(+9i'@#)^*,))/ ((+33;; +' / ++.VW`Va,bc $&FF"D*=*C*CA*F%eg"%%.%:%:7%C !*cGDGGM2E ((+? |)LM ++.UV_U`,ab)(AO<; $ !!N9+RAx"HI## $, >ED !^KK%%(FykQSTWXYTZS[&\]]^ F KK   ;CF8D E E FsJF& FF&J G8J !I A&H (I )AH >G=AH I JF# F&& G5/;G0*J0G55J=H H  I*I<I II J'I?:J?JJlinecd|vry|jdDcgc]}|j}}t|dk7rddt|fS|d}d|vryy cc}w) N|)FzMissing separator '|'FzExpected 3 parts, got r:)Fz!Missing ':' in concept definition)TN)splitrRlen)r1r`pparts concept_parts r_validate_fact_linez'AsyncLLMTextManager._validate_fact_lineysl d?1$(JJsO4OqO4 u:?23u:,?? ?Qx l "=5sA fact_filec "|jd}t|}|jr` t|d5}t j |}dddj d|k(r|S|jjd|di|d S#1swYCxYw#tj$r"|jjd|dYEt$r4}|jjd |d t|Yd}~|d}~wwxYw) z Load token cache from .q.tokens if present and matching file hash. Otherwise return a new structure with updated file-hash. .q.tokensr5N content_hashzHash changed for z, reindex needed.zCorrupt token cache for z , rebuilding.zError reading cache for r:factsrn) rSr%existsrjsonloadgetr)rUJSONDecodeErrorrOrFrH)r1rk cache_file current_hashrcacherWs r_load_or_create_token_cachez/AsyncLLMTextManager._load_or_create_token_caches **;7 ))4     V*c*a IIaLE+99^, < L   #4YK?P!QR\::+*'' Y ##&>yk$WX V ##&>ykCPQF8$TUU Vs: BBB.BBB2DD*D  Drxc|jd}t||d<t|d5}tj||dddy#1swYyxYw)NrmrnrD)rSr%rrrdump)r1rkrxrvrs r_save_token_cachez%AsyncLLMTextManager._save_token_cachesG**;7 29 =n *c "a IIeQ # " "s A  Atextc d|vr-|jdDcgc]}|jc}n|g}tjdd|d|d<t }t t jdhdz }g}|D]|}d|vr+d|vr'tjd |}|j|t|j} |j| D cgc]} | |vr|j| c} ~|Scc}wcc} w) Nrbz^(.*?):z\1renglish>howwhywhatwhenwherewhich()z.[\w_]+(?=\()|[\w_]+(?==[\'"]{1}[\w_]+[\'"]{1})) rerRrJrPr setr wordsfindallextendr lower lemmatize) r1r}xrh lemmatizer stop_wordstokenspart code_tokensrtokens rpreprocess_textz#AsyncLLMTextManager.preprocess_texts 8;t DJJsO4OqO4$66*eU1X6a&( 348  Dd{sd{ jjEt  k*!$**,/E MM"'!&J.((/!&   55&s DD c:|stjj|jrd|jj dt |jd5}tj|}dddd|_ |d|_ yy#1swYxYw)zW Load existing BM25 index from disk, if present and clear_cache=False. z&Loading existing BM25 index from disk.rNr/r-TF) ospathrqr0r)rUrpicklersr/r-)r1 clear_cacherdatas rmaybe_load_bm25_indexz)AsyncLLMTextManager.maybe_load_bm25_indexs|rww~~d.B.BC KK  E Fd**D1Q{{1~2#'(9#:D "<0DO 21s BBc X |rO|jjd|jjr|jj t j }|jjdtj|jDcgc]#}|jds|j|z %}}g}g}g}g}|D]} | jd} |s| js|j| 8|j| } t| ddk(s| jdt!| k7r|j| | dj#D]9\} } |j| |j| d| |j$| <;|s|s|j'd r|jjd y |jjd |r|jjdt|dt)||_||_t/|jd5}t1j2|j*|j,d|d d d y |jj5dy |jjt|dg}g}t7t|d5}|D]}it!|d} t/|dd5}|j9j;}|j=dDcgc]#}|j;s|j;%}}d d d D]} |j?| \}}|s|j|| |f.|jA| }|tCjBd|d| <|j| |j|||j$| <|jE|||jGjHdz dz }|jjKd|jLd|dd |jUd" d d d |rZ|jj5d#t|d$|D]-\}} }|jj5|d|d%| d d&d'/||z}||z}|jjdt|d(t)||_||_t/|jd5}t1j2|j*|j,d|d d d |jGjHdz dz }|jjd)|dd y cc}w#1swYy xYwcc}w#1swY_xYw#tN$r5}|jjQd!|dtS|Yd }~d }~wwxYw#1swYxYw#1swYxYw)*a Checks for new or modified .q.md files by comparing file-hash. If none need reindexing and clear_cache is False, loads existing index if available. Otherwise, reindexes only changed/new files and merges or creates a new index. z0Clearing cache and rebuilding full search index.z/Checking which .q.md files need (re)indexing...rCrmrprrnrFrzllinesis_validrGr mem_usagerW all_facts all_tokens final_mems rbuild_search_indexz&AsyncLLMTextManager.build_search_indexs  KK  O P##**,$$++-.." JK/1jj.G_.G1::V]K^4==1$.G_%'+- B!~~k: "2"9"9";r"44R8E5>"a'599^+DHZ[]H^+^r").g(<(<(>$D*"))$/#**:h+?@.0D%%d+)?({))e)<   !_`   !\]!KK$$'@^AT@UUc%de&/&@DO+9D(d22D9Q *.///3/C/C%:KK''(UV CL>)QRS  G +C D (*W 2W>W W &W(C8W X8.X 4V=WW W X*X X XXX X)force_generate_factsclear_bm25_cachec tK|jjdtj|jDcgc]6j dr#t fddDs|jz 8}}|sF|Dcgc];}|j|jjddz js|=}}|s|jjdntdt||jD]w}||||jz}|jjd||jzd zd t||jzd z|j|d {y|jjd |j| y cc}wcc}w7>w)a  Generate index files for all documents in parallel batches Args: force_generate_facts (bool): If True, regenerate indexes even if they exist clear_bm25_cache (bool): If True, clear existing BM25 index cache z2Starting index generation for documentation files..mdc3@K|]}j|ywN)r).0rrs r z;AsyncLLMTextManager.generate_index_files..es,XDWqQZZ]DWs)rC.xs.mdrCz4All index files exist. Use force=True to regenerate.rzProcessing batch r/Nz:Index generation complete, building/updating search index.r)r)rUrrr(ranyrreplacerqrangerfr+r_r)r1rrrmd_filesibatchs ` rgenerate_index_filesz(AsyncLLMTextManager.generate_index_filesYs MN(*zz$--'@ '@!zz% ,XDW,X)X MMA '@  $##a ug(FFNNP8   KK  S T1c(mT__= 1t#67   #4Q5G!5K4LAsS[}^b^m^mOmqrNrMs!tu225999> UV ,<=- :s+>F8;F,<F8AF1B3F87F68?F8sectionsmodec ntjt|jdz tjt|jdz z}|Dchc]M}t|jj ds't|jj ddO}}|r$|Dchc]tfd|Dr}}g}t|dD]|d k(rQ|jd z }|jd z } |jt|jr|n| Y|jt|jd z g} |D]Y} t| d d 5}t| j} | jdd| ddd|jddd[| rdj!| SdScc}wcc}w#1swY*xYw#t$r4} |jjd| dt| Yd} ~ d} ~ wwxYw)Nz [0-9]*.mdz [0-9]*.xs.mdrC.rc3^K|]$}|jjv&ywr)r)rsectiondocs rrz/AsyncLLMTextManager.generate..s#VX'w}}#))+=Xs*-c|jddjrt|jddSdS)N_ri?B)reisdigitintrs rrz.AsyncLLMTextManager.generate..s7177SVrfnamerWs ` rgeneratezAsyncLLMTextManager.generate{s IIc$--+"=>?IIc$--."@ABC :C?A#Aw||44W=!W\\'',Q/ ? (1X VXVV IX))pqC{"--SE.8#}}#c{:  SGNN,<,OP SC5!<=> rD E$g6! JOOENNfXT%6($qvvxj#QR7/6}!!'*=2=9? X"76 E !!N4&3q6("CDD Es>AG!8G&4G7AG+G7+G4 0G77 H4*H//H4querytop_kc|jsy|j|}|jj|}tj|}tj |}|d|zz}|j |||}t|jddd|} g} | D]\} } t| jdd} tjj|j| z sNt|j| z d d 5}| j!d d }dd|dd|j#g}| j%dj'|ddddj'| S#1swYxYw)Nz;No search index available. Call build_search_index() first.g?) doc_scoresscore_threshold query_tokenscB|dddz|dddzz|ddzS)Nrcode_match_scoreg@ match_countg? total_scorers rrz,AsyncLLMTextManager.search..s;!'(3.A$}%+,A$}%&rT)rreverserCrr5r6r7rrz# r;rBr)r-r get_scoresnpmeanstd_aggregate_search_scoresrrrHrrrrqr(rrerrEr)r1rrrr mean_score std_scorer file_data ranked_filesresultsrrmain_docronly_file_namer>s rrKzAsyncLLMTextManager.searchstP++E2 __// = WWZ( FF:& $y(8911!+%2  OO    5 #GD!4y((%8Hww~~dmmh67$--(2C'Ja%-^^C%8% rrrc$i}t|D]\}}||kr |j|}|j|}||vr dddgd||<d|vr|jdn|g} d} t | dk(rP| dj } |j | } t t|t| zt |z } ||dxx|z cc<||dxxdz cc<t||d | ||d <||d j||S) Nr)rrr matched_factsrbrcrrrrr ) enumerater/r.rerfrRrrmaxrE) r1rrrridxscorefactr componentsrcode_refrs rrz,AsyncLLMTextManager._aggregate_search_scoressK #J/JC''',D))$/I )#$#$()%' ( )$-04KCdVJ :!#%a=..0"228< #&s<'83{;K'K#LsS_O`#` i  /5 8 / i  /1 4 /7:)$%78:J8Ii !3 4 i  1 8 8 >90<rc(|jdy)z&Convenience method for a full rebuild.TrN)r)r1s r refresh_indexz!AsyncLLMTextManager.refresh_indexs D1r)Nrc)F)rN)FF)extended)r)__name__ __module__ __qualname__rrrrr2rr_rHrboolrjrryr|rrrrrrKfloatrrrrrr'r' so)-$% @@%@" @  @  @ FFtDzFFdFFP  dHSM6I0J ;T;d;2 4   CDI: $ FYP >t >_c >pt >D">c">#">s">H*+C*+*+C*+X#u+#8=#MQRUY# #J2rr')%rpathlibrrJtypingrrrrrrrr rrnumpyr rank_bm25r nltk.tokenizer nltk.corpusr nltk.stemr litellmrr async_loggerrrrrr set_verboserHr%r'rrrr(sk  33  '!'0%   $ 3 R2R2r