gThddlmZmZddlmZmZmZmZmZddl m Z m Z ddl Z ddl Z ddlddlddlddlddlmZddlddlZddlZddlZddlmZdd lmZmZdd lmZGd d eZ Gd de Z!Gdde Z"Gdde Z#Gdde Z$Gdde$Z%Gdde$Z&y))ABCabstractmethod)AnyListDictOptionalUnion)ThreadPoolExecutor as_completedN)*)partial BeautifulSoup)htmletree) dataclassc veZdZdZd defdZedededeeee ffdZ dedeedeeee ffd Z y ) ExtractionStrategyz< Abstract base class for all extraction strategies. input_formatc ||_d|_|jj|_|j dd|_y)a Initialize the extraction strategy. Args: input_format: Content format to use for extraction. Options: "markdown" (default), "html", "fit_markdown" **kwargs: Additional keyword arguments z<|DEL|>verboseFN)rDEL __class____name__namegetr)selfrkwargss Q/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/extraction_strategy.py__init__zExtractionStrategy.__init__s6)NN++ zz)U3 urlrreturncy)z Extract meaningful blocks or chunks from the given HTML. :param url: The URL of the webpage. :param html: The HTML content of the webpage. :return: A list of extracted blocks or chunks. Nrr#rqrs r extractzExtractionStrategy.extract&s r"sectionsc  g}t5}|Dcgc]!}|j|j||fi|#}}t|D]!} |j | j # ddd|Scc}w#1swY|SxYw)z Process sections of text in parallel by default. :param url: The URL of the webpage. :param sections: List of sections (strings) to process. :return: A list of processed JSON blocks. N)r submitr)r extendresult) rr#r*r(rextracted_contentexecutorsectionfuturesfutures r runzExtractionStrategy.run1s  !X\de\dQXxt||S'LVL\dGe&w/!((90"! f"! sA9&A41A94A99BN)markdown) r __module__ __qualname____doc__strr!rrrrr)r4r&r"r rrsu 4S 4 3 c Dc3h) enumerate)rr#r*r(rir1s r r4zNoExtractionStrategy.runJs-OXYaObcObG!RG/, e.g., "ollama/llama3.3". api_token: The API token for the provider. instruction: The instruction to use for the LLM model. schema: Pydantic model schema for structured data. extraction_type: "block" or "schema". chunk_token_threshold: Maximum tokens per chunk. overlap_rate: Overlap between chunks. word_token_rate: Word to token conversion rate. apply_chunking: Whether to apply chunking. base_url: The base URL for the API request. api_base: The base URL for the API request. extra_args: Additional arguments for the API request, such as temprature, max_tokens, etc. verbose: Whether to print verbose output. usages: List of individual token usages. total_usage: Accumulated token usage. Nblockprovider api_token instructionschemac t|di|||_|xs-tj |dxst j d|_||_||_ ||_ |rd|_ |j dt|_ |j dt|_|j dt|_|j dd|_|j d d |_|j d |j d d |_|j d i|_|j"sd |_ |j dd|_g|_t/|_|js t3dy )a* Initialize the strategy with clustering parameters. Args: provider: The provider to use for extraction. It follows the format /, e.g., "ollama/llama3.3". api_token: The API token for the provider. instruction: The instruction to use for the LLM model. schema: Pydantic model schema for structured data. extraction_type: "block" or "schema". chunk_token_threshold: Maximum tokens per chunk. overlap_rate: Overlap between chunks. word_token_rate: Word to token conversion rate. apply_chunking: Whether to apply chunking. base_url: The base URL for the API request. api_base: The base URL for the API request. extra_args: Additional arguments for the API request, such as temprature, max_tokens, etc. verbose: Whether to print verbose output. usages: List of individual token usages. total_usage: Accumulated token usage. zno-tokenOPENAI_API_KEYrJchunk_token_threshold overlap_rateword_token_rateapply_chunkingTbase_urlNapi_base extra_argsgeArFzvAPI token must be provided for LLMExtractionStrategy. Update the config.py or set OPENAI_API_KEY environment variable.r&)superr!rGPROVIDER_MODELSrosgetenvrHrI extract_typerJCHUNK_TOKEN_THRESHOLDrM OVERLAP_RATErNWORD_TOKEN_RATErOrPrQrRrSrusages TokenUsage total_usage ValueError)rrGrHrIrJextraction_typerrs r r!zLLMExtractionStrategy.__init__fsO0 "6"  "no&9&9(J&OnSUS\S\]mSn&+  (D %+ZZ0GI^%_""JJ~|D%zz*;_M$jj)94@ :t4  :vzz*d/KL  **\26""),D &zz)U3  %<~~VW Wr"r#ixrr$cd|jrtd|d||tt|d}t}|j r|j |d<t }|jdk(r6|jr*tj|jd|d<t}|D]}|jd |zd z||}t|j||j|j xs |j"|j$ }t'|j(j*|j(j,|j(j.|j(j0r |j(j0j2ni|j(j4r |j(j4j2ni }|j6j9||j:xj*|j*z c_|j:xj,|j,z c_|j:xj.|j.z c_ t=d g|j>dj@jBd } tjD| } | D]} d| d<  |jrtdtK| d|d|| S#tF$rT} tI|j>dj@jB\} } | } | r| j9dddg| dYd} ~ d} ~ wwxYw)a Extract meaningful blocks or chunks from the given HTML using an LLM. How it works: 1. Construct a prompt with variables. 2. Make a request to the LLM using the prompt. 3. Parse the response and extract blocks or chunks. Args: url: The URL of the webpage. ix: Index of the block. html: The HTML content of the webpage. Returns: A list of extracted blocks or chunks. z[LOG] Call LLM for z - block index: )URLHTMLREQUESTrJ)indentSCHEMA{})rQrS)completion_tokens prompt_tokens total_tokenscompletion_tokens_detailsprompt_tokens_detailsblocksrFerrorTr=rqrAr>Nz[LOG] Extractedzblocks from URL:z block index:)&rprintescape_json_string sanitize_htmlPROMPT_EXTRACT_BLOCKSrI&PROMPT_EXTRACT_BLOCKS_WITH_INSTRUCTIONrXrJjsondumps&PROMPT_EXTRACT_SCHEMA_WITH_INSTRUCTIONreplaceperform_completion_with_backoffrGrHrRrQrSr]usagerkrlrmrn__dict__ror\appendr^extract_xml_datachoicesmessager>loads Exceptionsplit_and_parse_json_objectslen)rr#rarvariable_valuesprompt_with_variablesvariableresponser}rprFeparsedunparseds r r)zLLMExtractionStrategy.extracts" << 'u,>"..66!44[c[i[i\D\Dhnn&N&N&W&WJLS[SaSaSwSw(.."F"F"O"O}   5! **e.E.EE* &&%*=*==& %%););;% %xj(2B2B12E2M2M2U2UVW_`FZZ'F!&g  << #S[2Dc>[] ^  ;H  !H^^C(Ff+(<(<12z------------------------------ Completionz>12,PromptTotalz === Usage History ===z Request #z<10z0------------------------------------------------r N)rsr^rkrlrmrBr\)rrCr}s r show_usagez LLMExtractionStrategy.show_usagePs( -.  Agc]+, h c"!D$4$4$F$Ft#LMN #a 0 0 > >tDEF  Qt//<Fsm0W 0W040WfP3P3PcPd4S>6JPd/d0!s0!d3i0!Dc3h4H0!f rr"rEc eZdZdZdfd ZddeedededeefdZddeefd Z ddeefd Z d e eeefde eeeffd Z d ededee ee ffdZd edeedee ee ffdZxZS)CosineStrategya Extract meaningful blocks or chunks from the given HTML using cosine similarity. How it works: 1. Pre-filter documents using embeddings and semantic_filter. 2. Perform clustering using cosine similarity. 3. Organize texts by their cluster labels, retaining order. 4. Filter clusters by word count. 5. Extract meaningful blocks or chunks from the filtered clusters. Attributes: semantic_filter (str): A keyword filter for document filtering. word_count_threshold (int): Minimum number of words per cluster. max_dist (float): The maximum cophenetic distance on the dendrogram to form clusters. linkage_method (str): The linkage method for hierarchical clustering. top_k (int): Number of top categories to extract. model_name (str): The name of the sentence-transformers model. sim_threshold (float): The similarity threshold for clustering. c t |d i|ddl} ||_||_||_||_||_||_tj|_ |jdd|_ | jg|_d|_t!|_t%|j"|_|jr#t)d|j"j*dt-|\|_|_|j0j3|j"|j0j5d|_| jg|_|jr#t)d |j"j*dt7\|_} |jr>t)d |d t;tj|jz zd zyy)a Initialize the strategy with clustering parameters. Args: semantic_filter (str): A keyword filter for document filtering. word_count_threshold (int): Minimum number of words per cluster. max_dist (float): The maximum cophenetic distance on the dendrogram to form clusters. linkage_method (str): The linkage method for hierarchical clustering. top_k (int): Number of top categories to extract. rNrFdirectz#[LOG] Loading Extraction Model for z device.batchz([LOG] Loading Multilabel Classifier for z[LOG] Model loaded z, models/reuters, took  secondsr&)rTr!numpysemantic_filterword_count_thresholdmax_distlinkage_methodtop_k sim_thresholdrtimerrrarraybuffer_embeddingsget_embedding_method get_devicedevicecalculate_batch_sizedefault_batch_sizerstypeload_HF_embedding_model tokenizermodeltoevalload_text_multilabel_classifiernlpr9) rrrrrr model_namerrnp_rs r r!zCosineStrategy.__init__ws "6".$8!  , *YY[ zz)U3 !)"$,! l #7t{{"C << 7 8H8H7IR S&=Z%H"  dkk" $+!!)" << z.sadr"T)keyreverseN) rsklearn.metrics.pairwiserget_embeddingsflattenziprsortr-) rrrrrquery_embeddingdocument_embeddings similaritiesdocsim filtered_docsremaining_docsrs r filter_documents_embeddingsz*CosineStrategy.filter_documents_embeddingssV  y>J &Y1,J>--.?@C#11)<)/):P1P!Q R,99=a= 9[j))os :s4D  D 2D& D& D, sentencesc |jjdvr"ddl}| |j}g}t dt ||D]}||||z}|j |ddd}|jD cic]!\} } | | j|j#}} } |j5|jd i|} ddd jjdjj} |j| t!j"||_|j$S|jjd k(ro| |j}g}t dt ||D],}||||z}|j|} |j| .t!j"||_|j$Scc} } w#1swYxYw) z Get BERT embeddings for a list of sentences. Args: sentences (List[str]): A list of text chunks (sentences). Returns: NumPy array of embeddings. )cpugpucudampsrNTpt)padding truncationreturn_tensorsr )dimrr&)rrtorchrrangerritemsrno_gradrlast_hidden_statemeanrrrrvstackr) rr batch_size bypass_bufferrall_embeddingsrCbatch_sentences encoded_inputrtensor model_output embeddingss r rzCosineStrategy.get_embeddingss ;;  = = !!44 N1c)nj9"+Aa*n"= $Y]nr s P]PcPcPe fPeffii & #>L%*;;@@Q@GKKMSSU %%j1:&(YY~%>D "%%%[[   &!!44 N1c)nj9"+Aa*n"=!ZZ8 %%j1: &(YY~%>D "%%%/!g%_s7&G0GG cddlm}m}ddlm}t j |_|j|d}||d}|||j}|||jd }|S) z Perform hierarchical clustering on sentences and return cluster labels. Args: sentences (List[str]): A list of text chunks (sentences). Returns: NumPy array of cluster labels. r)linkagefcluster)pdistT)rcosine)methoddistance) criterion) scipy.cluster.hierarchyrrscipy.spatial.distancerrrrrr) rrrrrrdistance_matrixlinkedlabelss r hierarchical_clusteringz&CosineStrategy.hierarchical_clusteringsc >0YY[ (($(G  H51D1DE&$--:F r"clustersci}|jD]D\}}dj|}t|j}||jk\s@|||<F|S)a Filter clusters to remove those with a word count below the threshold. Args: clusters (Dict[int, List[str]]): Dictionary of clusters. Returns: Dict[int, List[str]]: Filtered dictionary of clusters. r)rrrrr)rrfiltered_clusters cluster_idtexts full_text word_counts r filter_clusters_by_word_countz,CosineStrategy.filter_clusters_by_word_count4sa!)!1 JIY__./JT66605!*-"2! r"r#rc Vtj}|j|j}|j||j}|sgS|j |}tj}i}t |D])\} } |j| gj|| +|j|} t| D cgc]#} t| gdj| | d%} } |jrtd|j|jj dvr;|j#| Dcgc]}|d c}}t%| |D] \}} | |d< |jr%tdtj|z dd | Scc} wcc}w) a Extract clusters from HTML content using hierarchical clustering. Args: url (str): The URL of the webpage. html (str): The HTML content of the webpage. Returns: List[Dict[str, Any]]: A list of processed JSON blocks. rr@u[LOG] 🚀 Assign tags using )rrrrr>rAu"[LOG] 🚀 Categorization done in z.2fr)rrrrrrrB setdefaultrrsortedrrrrsrrrr)rr#rr(rt text_chunksrrr=labelridx cluster_listclusters r r)zCosineStrategy.extractKs IIK djj* 66{DDXDXY I--k: IIK%f-LE5   r * 1 1+e2D E.!>>xHrxyJrKLrKjm#c(RCHHM^_bMcDderK L << 1$++? @ ;;  < <XXNgwy1NOF"%lF";"'#<& << 6tyy{Qs6K8T U=L Os (F! F&r*c\|j||jj|fi|S)z Process sections using hierarchical clustering. Args: url (str): The URL of the webpage. sections (List[str]): List of sections (strings) to process. Returns: )r)rr)rr#r*r(rs r r4zCosineStrategy.runs)t||Cx!8CFCCr")N g?wardz&sentence-transformers/all-MiniLM-L6-v2g333333?))NFN)rr6r7r8r!rr9rrrrrrrr)r4rrs@r rrcs&GzR**T#Y**QT**be**ostwox**X.&S .&`c0!d3S >6J!tTWY]^aYbTbOc!.A3AcADc3hO"P7F!3DN\t)).%/JdbddV}&--guZ7HIOWXx//E(ODxXXV} ---guZ7HIJRS(B**2uX?(SS--gu= = YT (||/f bQIJ99Y' ' (sRC!C"C>CC C D*&9D%D*%D*cd|vr.|j||d}|s|jdS|d}n|}d}|ddk(r|j|}n|ddk(r|j||d}na|ddk(r|j |}nG|dd k(r?|j|}t j |d |}|r|jd nd}d |vr|j||d }||S|jdS) a Extract a single field based on its type. How it works: 1. Selects the target element using the field's selector. 2. Extracts the field value based on its type (e.g., text, attribute, regex). 3. Applies transformations if defined in the schema. Args: element: The base element to extract the field from. field (Dict[str, Any]): The field definition in the schema. Returns: Any: The extracted field value. rBrMrNrtext attributerregexpatternr transform) rGr_get_element_text_get_element_attribute_get_element_htmlresearchgroup_apply_transform)rr:r<selectedr=rUmatchs r r4z3JsonElementExtractionStrategy._extract_single_fields"  ))'53DEHyy++{HH =F "**84E 6]k )//% :LME 6]f $**84E 6]g %))(3DIIeI.5E&+EKKNE % ))%{1CDE)uCuyy/CCr"cRi}|D]}|j||}||||d<!|S)Nr)r4rr:r1r;r<r=s r rNz0JsonElementExtractionStrategy._extract_list_itemEs>E..w>E &+U6]# r"ci}|D]:}|ddk(r|j||}n|j||}|3|||d<<|S)a Extracts fields from a given element. How it works: 1. Iterates through the fields defined in the schema. 2. Handles computed, single, and nested field types. 3. Updates the item dictionary with extracted field values. Args: element: The base element to extract fields from. fields (List[Dict[str, Any]]): The list of fields to extract. Returns: Dict[str, Any]: A dictionary representing the extracted item. rcomputedr)_compute_fieldrSrds r r5z+JsonElementExtractionStrategy._extract_itemMs^"EV} *++D%8++GU; &+U6]#  r"c|dk(r|jS|dk(r|jS|dk(r|jS|S)a Apply a transformation to a value. How it works: 1. Checks the transformation type (e.g., `lowercase`, `strip`). 2. Applies the transformation to the value. 3. Returns the transformed value. Args: value (str): The value to transform. transform (str): The type of transformation to apply. Returns: str: The transformed value. lowercase uppercasestrip)lowerupperrk)rr=rYs r r`z.JsonElementExtractionStrategy._apply_transformhsE"  #;;= + %;;= ' !;;=  r"c  d|vrt|di|Sd|vr |d|Sy#t$rD}|jrtd|ddt ||j dcYd}~Sd}~wwxYw)N expressionfunctionzError computing field rrLrM)rrrrsr9r)rr;r<rs r rgz,JsonElementExtractionStrategy._compute_fields (u$E,/T::u$(uZ(..% (||.uV}oRAxHI99Y' ' (s&& A39A.(A3.A3r*c`|jj|}|j||fi|S)a4 Run the extraction strategy on a combined HTML content. How it works: 1. Combines multiple HTML sections using the `DEL` delimiter. 2. Calls the `extract` method with the combined HTML. Args: url (str): The URL of the page being processed. sections (List[str]): A list of HTML sections. *q: Additional positional arguments. **kwargs: Additional keyword arguments for custom extraction. Returns: List[Dict[str, Any]]: A list of extracted items. )rrr))rr#r*r(r combined_htmls r r4z!JsonElementExtractionStrategy.runs.$ h/ t||C9&99r"cy)zGet text content from elementNr&rr:s r rZz/JsonElementExtractionStrategy._get_element_textrAr"cy)zGet HTML content from elementNr&rts r r\z/JsonElementExtractionStrategy._get_element_htmlrAr"rVcy)z Get attribute value from elementNr&rr:rVs r r[z4JsonElementExtractionStrategy._get_element_attributerAr")rr6r7r8rrr9rr!rr)rr2r3rGrSr4rNr5r`rgr4rZr\r[rrs@r r)r)s;@ C 4tCH~ 4'3'c'DcSVhDX'R       s  (*(DT62 (:s:d3i:$tCQTH~BV:* C   C     r"r)czeZdZdZdeeefffd ZdefdZdefdZ defdZ d efd Z d efd Z d efd Z xZS)JsonCssExtractionStrategya Concrete implementation of `JsonElementExtractionStrategy` using CSS selectors. How it works: 1. Parses HTML content with BeautifulSoup. 2. Selects elements using CSS selectors defined in the schema. 3. Extracts field data and applies transformations as defined. Attributes: schema (Dict[str, Any]): The schema defining the extraction rules. verbose (bool): Enables verbose logging for debugging purposes. Methods: _parse_html(html_content): Parses HTML content into a BeautifulSoup object. _get_base_elements(parsed_html, selector): Selects base elements using a CSS selector. _get_elements(element, selector): Selects child elements using a CSS selector. _get_element_text(element): Extracts text content from a BeautifulSoup element. _get_element_html(element): Extracts the raw HTML content of a BeautifulSoup element. _get_element_attribute(element, attribute): Retrieves an attribute value from a BeautifulSoup element. rJc 0d|d<t||fi|yNrrrTr!r,s r r!z"JsonCssExtractionStrategy.__init__!'~ *6*r"r-ct|dS)Nz html.parserrr@s r r2z%JsonCssExtractionStrategy._parse_htmls\=99r"rBc$|j|Sr'selectrDs r r3z,JsonCssExtractionStrategy._get_base_elementss!!(++r"c$|j|Sr'rrFs r rGz'JsonCssExtractionStrategy._get_elementss~~h''r"r$c&|jdS)NT)rk)get_textrts r rZz+JsonCssExtractionStrategy._get_element_textsd++r"ct|Sr')r9rts r r\z+JsonCssExtractionStrategy._get_element_htmls 7|r"rVc$|j|Sr'rrws r r[z0JsonCssExtractionStrategy._get_element_attribute{{9%%r")rr6r7r8rr9rr!r2r3rGrZr\r[rrs@r ryrysd*+tCH~+::,,(s( ,C,C&&r"ryceZdZdZdeeefffd ZdefdZdefdZ ded efd Z ded efd Z defd Z d efd Z d efdZdefdZxZS)JsonXPathExtractionStrategya Concrete implementation of `JsonElementExtractionStrategy` using XPath selectors. How it works: 1. Parses HTML content into an lxml tree. 2. Selects elements using XPath expressions. 3. Converts CSS selectors to XPath when needed. Attributes: schema (Dict[str, Any]): The schema defining the extraction rules. verbose (bool): Enables verbose logging for debugging purposes. Methods: _parse_html(html_content): Parses HTML content into an lxml tree. _get_base_elements(parsed_html, selector): Selects base elements using an XPath selector. _css_to_xpath(css_selector): Converts a CSS selector to an XPath expression. _get_elements(element, selector): Selects child elements using an XPath selector. _get_element_text(element): Extracts text content from an lxml element. _get_element_html(element): Extracts the raw HTML content of an lxml element. _get_element_attribute(element, attribute): Retrieves an attribute value from an lxml element. rJc 0d|d<t||fi|yr{r|r,s r r!z$JsonXPathExtractionStrategy.__init__r}r"r-c,tj|Sr')r fromstringr@s r r2z'JsonXPathExtractionStrategy._parse_htmls|,,r"rBc$|j|Sr')xpathrDs r r3z.JsonXPathExtractionStrategy._get_base_elementss  **r" css_selectorr$c0d|vr|S|j|S)z'Convert CSS selector to XPath if needed/)_basic_css_to_xpath)rrs r _css_to_xpathz)JsonXPathExtractionStrategy._css_to_xpaths! ,  '' 55r"cd|vr%|jd}ddj|zSd|vr%|jd}ddj|zSd|zS)z.Basic CSS to XPath conversion for common casesz > z//rr)rr)rrpartss r rz/JsonXPathExtractionStrategy._basic_css_to_xpathsd L  &&u-E#((5/) ) ,  &&s+E$))E** *l""r"cr|j|}|jdsd|z}|j|S)N.)rrr)rr:rBrs r rGz)JsonXPathExtractionStrategy._get_elementss8""8,$%KE}}U##r"c^dj|jdjS)Nz .//text())rrrkrts r rZz-JsonXPathExtractionStrategy._get_element_texts#www}}[1288::r"c0tj|dS)Nunicode)encoding)rtostringrts r r\z-JsonXPathExtractionStrategy._get_element_htmls~~g ::r"rVc$|j|Sr'rrws r r[z2JsonXPathExtractionStrategy._get_element_attributerr")rr6r7r8rr9rr!r2r3rrrGrZr\r[rrs@r rrs,+tCH~+--++6#6#6 ###$s$ ;C;;C;&&r"r)'abcrrtypingrrrrr concurrent.futuresr r rxrpromptsconfigutilsmodels functoolsr model_loaderrrrr]bs4rlxmlrr dataclassesrrr;rErr)ryrr&r"r rs#33?  !+!+!Z d- d Mr.MrfwD'wDz N $6N `,& =,&\>&"?>&r"