g+xddlZddlZddlmZmZddlmZmZmZddl m Z ddl m Z ddl Z ddlZddlZddlZddlddl mZmZmZdd l mZmZdd lmZdd lmZdd lmZmZdd lmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ejNdZ(ejNdZ)ejNdZ*dZ+dZ,GddeZ-Gdde-Z.y)N)ABCabstractmethod)DictAnyOptional) BeautifulSoup)ThreadPoolExecutor)*)elementNavigableStringComment) PageElementTag)urljoin) InvalidSchema)RelevantContentFilterBM25ContentFilter)MarkdownGenerationStrategyDefaultMarkdownGenerator)MarkdownGenerationResult)extract_metadata normalize_urlis_external_urlget_base_domainz^og:z ^twitter:z (\d+)(\D*)c|rJtj|}|r3t|jd}|jdxsd}||fSy)Nr px)NN)DIMENSION_REGEXmatchintgroup) dimensionr numberunits W/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/content_scraping_strategy.pyparse_dimensionr'sI%%i0 Q(F;;q>)TD4<  ct||jd} tj|}|jdk(r|j jddyt d| y#t$r }Yd}~yd}~wwxYw#YyxYw)NsrczContent-Lengthz!Failed to retrieve file size for )rgetrequestshead status_codeheadersprintr)imgbase_urlimg_urlresponsees r&fetch_image_file_sizer7*shswwu~.G ==)   3 &##''(8>  5gY? @   s)?A.A.. B7B<BBBc \eZdZedededeeeffdZedededeeeffdZy)ContentScrapingStrategyurlhtmlreturnc yNselfr:r;kwargss r&scrapzContentScrapingStrategy.scrap:s r(c Kywr>r?r@s r&ascrapzContentScrapingStrategy.ascrap>s  sN) __name__ __module__ __qualname__rstrrrrCrEr?r(r&r9r99sb  C d38n    3 T#s(^  r(r9c eZdZdZddZddZdededeeeffdZ dededeeeffd Z d Z d Z dd Z d ZdedeeeffdZdedeeefdeeefdeeefdef dZedfdededededeeeff dZy)WebScrapingStrategya Class for web content scraping. Perhaps the most important class. How it works: 1. Extract content from HTML using BeautifulSoup. 2. Clean the extracted content using a content cleaning strategy. 3. Filter the cleaned content using a content filtering strategy. 4. Generate markdown content from the filtered content. 5. Return the markdown content. Nc||_yr>)logger)rArMs r&__init__zWebScrapingStrategy.__init__Ns  r(c `|jr"t|j|}|d||d|yy)z#Helper method to safely use logger.messagetagNr?)rMgetattr)rAlevelrQrRrB log_methods r&_logzWebScrapingStrategy._logQs0 ;; e4J  :wC :6 : r(r:r;r<c .|j||fddi|S)a Main entry point for content scraping. Args: url (str): The URL of the page to scrape. html (str): The HTML content of the page. **kwargs: Additional keyword arguments. Returns: Dict[str, Any]: A dictionary containing the scraped content. This dictionary contains the following keys: - 'markdown': The generated markdown content, type is str, however soon will become MarkdownGenerationResult via 'markdown.raw_markdown'. - 'fit_markdown': The generated markdown content with relevant content filtered, this will be removed soon and available in 'markdown.fit_markdown'. - 'fit_html': The HTML content with relevant content filtered, this will be removed soon and available in 'markdown.fit_html'. - 'markdown_v2': The generated markdown content with relevant content filtered, this is temporary and will be removed soon and replaced with 'markdown' is_asyncF)_scrapr@s r&rCzWebScrapingStrategy.scrapWs!"t{{3?u???r(cbKtj|j||fi|d{S7w)a Main entry point for asynchronous content scraping. Args: url (str): The URL of the page to scrape. html (str): The HTML content of the page. **kwargs: Additional keyword arguments. Returns: Dict[str, Any]: A dictionary containing the scraped content. This dictionary contains the following keys: - 'markdown': The generated markdown content, type is str, however soon will become MarkdownGenerationResult via 'markdown.raw_markdown'. - 'fit_markdown': The generated markdown content with relevant content filtered, this will be removed soon and available in 'markdown.fit_markdown'. - 'fit_html': The HTML content with relevant content filtered, this will be removed soon and available in 'markdown.fit_html'. - 'markdown_v2': The generated markdown content with relevant content filtered, this is temporary and will be removed soon and replaced with 'markdown' N)asyncio to_threadrYr@s r&rEzWebScrapingStrategy.ascrapjs,"&&t{{CHHHHHs &/-/ct|tr|St|jdk(rat|jdtrD|jdj |j k(r|j |jdS|jDcgc]}|j |c}|_|Scc}w)z Flatten nested elements in a HTML tree. Args: node (Tag): The root node of the HTML tree. Returns: Tag: The flattened HTML tree. r r) isinstancer lencontentsrnameflatten_nested_elements)rAnodechilds r&rbz+WebScrapingStrategy.flatten_nested_elements}s dO ,K t}}  "z$--2BC'HT]][\M]MbMbfjfofoMo// a0@A AJN--X-55e<-X  YsB<c |jdt}|}|rB|j}|r1|jdd}t |j |k\r|S|rBy)a Find the closest parent with useful text. Args: tag (Tag): The starting tag to search from. **kwargs: Additional keyword arguments. Returns: Tag: The closest parent with useful text, or None if not found. $image_description_min_word_threshold T) separatorstripN)r,$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDparentget_textr_split)rArRrBrf current_tag text_contents r&$find_closest_parent_with_useful_textz8WebScrapingStrategy.find_closest_parent_with_useful_textsp06zz:`cG0H, %,,K*33c3M |))+,0TT''r(cg}|jD]>}||vs|r$|jdr|j|.|j|@|D]}||=y)aS Remove unwanted attributes from an HTML element. Args: element (Tag): The HTML element to remove attributes from. important_attrs (list): List of important attributes to keep. keep_data_attributes (bool): Whether to keep data attributes. Returns: None data-N)attrs startswithappend)rAr important_attrskeep_data_attributesattrs_to_removeattrs r&remove_unwanted_attributesz.WebScrapingStrategy.remove_unwanted_attributess]MMD?*'??73'..t4#**40 "$D $r(c  !"#$d}tgdtddg}tgd!|jdd}|jdd|jd d#|jd d} |jd d} |jd d} |jd } |jd} |j}|jdg}d|vsJ|j|vsz3WebScrapingStrategy.process_image..s67ggfo!KoD*o!K"M!KA+,'')//*;A*>!8JKIZ[]I^IeIefiIj)-#/!K"M!K"MsB B AB)buttoniconlogorinput)jpgjpegpngwebpavifgifstylealtr*data-srcsrcset data-srcsetrheightclassz display:nonec34K|]}D]}||v ywr>r?).0cclsclasses_to_checks r& z4WebScrapingStrategy.process_image..s M.Qr?)rrr*s r&rz4WebScrapingStrategy.process_image..3"2QS"2c3&K|]}|v ywr>r?)rrrs r&rz4WebScrapingStrategy.process_image..rrNrr g?c.tfdDS)Nc3BK|]}|jvywr>)lower)rfmtr:s r&rzNWebScrapingStrategy.process_image..has_image_format..sC]csciik)]s)any)r: image_formatss`r&has_image_formatz;WebScrapingStrategy.process_image..has_image_formatsC]CC Cr(c3.K|] }|ywr>r?)rr:rs r&rz4WebScrapingStrategy.process_image.. sU0T$0Tspictureimage_score_thresholdrfimage)rdescscoretypegroup_idformatc|rA|jds/|vr*j|ji||dyyyy)Nzdata:)r*r)rtaddru)r*r base_infoimage_variants unique_urlss r& add_variantz6WebScrapingStrategy.process_image..add_variant0sJ3>>'2s+7M$%%&O&O3&OP8N2sr()rrr:sourcerrr}r>) frozensetr,rkrarisdigitr! find_parentrIMAGE_SCORE_THRESHOLDsetrjrpfind_allrsitemsrt)%rAr2r:index total_imagesrB parse_srcset tags_to_checkrdata_srcr data_srcsetrrrkparent_classesr width_val height_valdetected_formatrformat_matchesrrfrryvaluerrrrrrrrr*rs% @@@@@@@@r& process_imagez!WebScrapingStrategy.process_images*M %%?@!8W"56 !"OP $ggeR ggeR 77:r*2&ggmR0  "GR0 e # KK= ( M.M M 3"23 3 3"23 3 U]]_E I )c/Qq 0E fnn&VJ *s*Q 1E  QJE |#c)) D Uh 0TU U QJE [ QJE ??9 % QJE6;7C1>!U##BT#!U!&4Q&7O 8 FJJ68MN Ne 06zz:`cG0H,=D==cLVL %   Q CH.D %u%*51Fu vg?2. ooi0 07 0!**84#ZZ1161+F3#CJG = 45 99??,KD%w'Ud]h$>NTZ^cTcE"-"0~9T9m"Vs 'M ?M r c Pgggd}i}i}|j|||||fi||||dS)aD Process an HTML element. How it works: 1. Check if the element is an image, video, or audio. 2. Extract the element's attributes and content. 3. Process the element based on its type. 4. Return the processed element information. Args: url (str): The URL of the page containing the element. element (Tag): The HTML element to process. **kwargs: Additional keyword arguments. Returns: dict: A dictionary containing the processed element information. imagesvideosaudios)mediainternal_links_dictexternal_links_dict)_process_element)rAr:r rBrrrs r&process_elementz#WebScrapingStrategy.process_elementMsZ$r:           #6#6  r(rrrc f t|tr"t|tr|jy|j dt |}|j dvr|jyd}|j dg} |j dk(r |jdr|jddj} | sy|jdd } t| |} | |jj|jd dj|d }t| |}d }|rLt | }||d<|j ddr|jy| r|| vr|jy|r | |vr||| <n | |vr||| < |j dk(rgd}|jdd}|s)|r'|j|j!dd}|s|r'|s|jyd|j"vr3|j"djddjdd}t||sy t |}|j ddr|jy| r|| vr|jyy |j ddr!|j dk(r|jy|j dvr||j dj%|jd|jd|j |j&|fi|d|j(d}|D]b}||j dj%|j d|jd|j |j&|fi|ddy |j t*vr3|j d dr!|j,|j |j/|t0|j d!dt5|j6D]\}t|tr0t|ts t9|jdkDs@d }C|j:|||||fi|s[d }^|j d&t<}|s/t9|jd 'j}||k\}|s|j|S#t$r } Yd } ~ yd } ~ wwxYw#t$r} tdt| d } ~ wwxYw#t$r} dd } ~ wwxYw#t$r+} |j3d"d#d$d"t| i%Yd } ~ Pd } ~ wwxYw#t$r*} |j3d"d(d$d"t| i%Yd } ~ yd } ~ wwxYw))z2 Process an HTML element. F base_domain)scriptrlinkmetanoscriptexclude_domainsahrefr/rNtitle)rtextrrTexclude_external_linkszError processing links: r2)r*rzsrcsetdata-lazy-srcz data-originalr*rr,rgexclude_external_imageszError processing images remove_formsform)videoaudiorr)r*rr descriptionr only_textrwerrorz+Error removing unwanted attributes: {error}SCRAPErQrRparamsword_count_threshold)riz!Error processing element: {error})r^r rextractr,rra decomposerirmr ValueErrorrlr ExceptionrIpoprsrurprONLY_TEXT_ELIGIBLE_TAGS replace_withrzIMPORTANT_ATTRSrVlistchildrenr_rMIN_WORD_THRESHOLD)rAr:r rrrrBr keep_elementrrurl_basenormalized_hrefr6 link_data is_externallink_base_domainpotential_sourcesr*image_src_base_domain source_tags source_tagrdr word_counts r&rz$WebScrapingStrategy._process_elementpsE '?3gw/#GOO% !**]OC4HIK||NN!!!# L$jj):B?O 9 E<<3&;7;;v+>&7;;vr288:D$"yy~a0H%*7c*B !0 0 0 0 2 8 8 :!,Wb!9!?!?!A'2 !I#2/;"OK#'L#+:?+K(3C -0!::&>F-G--/#(-/?B 1 1 1 3', #*2EECL/@*2EECL/@ 3 0<<5((f%%'++eR0C!&7)gkk*;*?*?*BBG"&7)))+$ 7==0%mmH5;;C@CII#NqQ+3 <#,;C,@)zz";UC)))+$$'0OC-G--/#(  a)lzz.%0W\\V5K!!!#||11a()00&7;;u-&7;;u-#LL#L4#L#LW#_X^#_ 2 /g..x8 "-JW\\N!,-44%>>%0&7;;u-#LL#L4#L#LW#_X^#_ 6#.||66::k51(G(()9)9)9);< //&**UkmrJstg../e_5jPW>X5;;=)A-'+ ,t,,S%@SUhslrs'+ 0$*::.DFX#Y  !1!1!1!=!C!C!EF )-AA !!!# y&%$%Z E":3q6( CDD Ej 0// 0@  'I #SV, 6  IIg;Q(    s1U=:U=/U=AT T  S5+BT 2T T AT31T3A T3.T3T33U=C(U=8AU=>'U%AU=:U=AU=5 T>T TT T0T++T00U=3 U<T>>UU= U: U5/U=5U::U== V0 V++V0r css_selectorc d}|sy|jdd}t||}|j} t} t d|} t|jd gxsgr*| jfd D]} | j|jd d}|rld |vxrd|v}|r9| j|x} rM| j| j|x} r%n'| j|D]} | j|rP| j|}|sdddgggdggdid|dS|jd} |D]}| j|t|jdgtz|d<t|jdg|d<|jddr|dj!|d|d<j"| f|| d|}ggd}|d}|d}|d}t%|j'|d<t%|j'|d<| jdfdt)DDcgc] }| |D]}|c}}|d <j+| } t-j.d!}D];}|jd"d}|j1|s'|j3d||d"<=d} | j5j7d#}|j=d+d,j=d-d}||||| d.S#t $r-} j ddddt| i i} Yd} ~ d} ~ wwxYwcc}}w#t $r} d}t|d$} | jdd%&}d'|_| jj|| j5j7d#}t;d(j dd)d*Yd} ~ d} ~ wwxYw)/a Extract content from HTML using BeautifulSoup. Args: url (str): The URL of the page to scrape. html (str): The HTML content of the page to scrape. word_count_threshold (int): The minimum word count threshold for content extraction. css_selector (str): The CSS selector to use for content extraction. **kwargs: Additional keyword arguments. Returns: dict: A dictionary containing the extracted content. TNparserlxmlrrz"Error extracting metadata: {error}rr excluded_tagsc |jvSr>)ra)rRr s r&rz,WebScrapingStrategy._scrap.._sSXX5Nr(excluded_selectorrrgr)internalexternalz$No elements found for CSS selector: )markdown cleaned_htmlsuccessrlinksmetadatarQdivexclude_social_media_domainsrexclude_social_media_linksF)rrrrrrrr2c 3^K|]$\}}j||t&ywr>)rr_)rir2imgsrAr:s r&rz-WebScrapingStrategy._scrap..s2;*93 $11#sAs4yI*9s*-rzdata:image/[^;]+;base64,([^"]+)r*zutf-8z html.parsercrawl4ai_error_message)ida8 Crawl4AI Error: This page is not fully supported. Possible reasons: 1. The page may have restrictions that prevent crawling. 2. The page might not be fully loaded. Suggestions: - Try calling the crawl function with these parameters: magic=True, - Set headless=False to visualize what's happening on the page. If the issue persists, please check the page's structure and any potential anti-crawling measures. u[LOG] 😧 Error: After processing the crawled HTML and removing irrelevant tags, nothing was left in the page. Check the markdown for further details.zAfter processing the crawled HTML and removing irrelevant tags, nothing was left in the page. Check the markdown for further details.rPz  z )rrrrr)r,rbodyrrrrVrIrrr select_oneselectnew_tagruSOCIAL_MEDIA_DOMAINSunionrrvalues enumeraterbrecompiler subencode_contentsdecodestringr1replace) rAr:r;rrrBr parser_typesouprrrr6r ris_single_selectorselected_elementsel result_objrrrrresultr2base64_patternr*str_body error_divrr rs `` @@r&rYzWebScrapingStrategy._scrap;sjj62 T;/yy%c*  #B-DFJJ;ArB ==)NO!P#JJ':B? !$,=!=!^#M^B^ !!%1B!CCgC#GOO%"&1B!CCgC ${{+<=G#GOO% >  $ L 9 $ "$&#(*bBG*,"= "!El^T<<&D' B(25VZZ@^`b5cfz5z1{-.$' 3Db(I$J ! ::2E :(./@(A(G(GOmHn(oF$ %)T))   $8#     R07#()>?()>?!!4!;!;!=>j !4!;!;!=>j}}U#;*3D/; ;F!  ; h++D1$FGC''%$C##C(+//C8E   ++-44W=HB ''5==dCH )   q  IIg<Q(   D  F $ G }5D U/G HI I  II  Y '++-44W=H l n IIg`   7 s7 L0%M)'M/0 M&9"M!!M&/ P8BO>>Pr>)r)F)rFrGrH__doc__rNrVrIrrrCrErbrprzrrrboolrrr!rYr?r(r&rKrKBsT ; @@C@d38n@&II3IT#s(^I&".0N:`! K! d3PS8n! FI[I$sCx.IgkloqtltguIMQRUWZRZM[IjnIVGYnrX #X SX X hkX BFGJLOGOBPX r(rK)/r'timeabcrrtypingrrrbs4rconcurrent.futuresr r[r-osconfigr r rrr urllib.parserrequests.exceptionsrcontent_filter_strategyrrmarkdown_generation_strategyrrmodelsrutilsrrrrr(OG_REGEX TWITTER_REGEXrr'r7r9rKr?r(r&rIs #&&1 11 -M^, 2::g  <( "**]+  c Q  1Q  r(