g ddlZddlmZddlmZmZddlmZmZm Z m Z m Z ddl Z ddl Z ddlZddlZddlZddlmZddlddlmZdd lmZmZdd lmZddlZdd lmZdd lmZmZmZmZddlZdd l m!Z!m"Z"m#Z#ddl$Z$ddl%Z%ddl&Z&ddl'm(Z(ddl)Z)Gdde*Z+dEde,de,de-de.de.de,f dZ/dZ0dZ1dZ2dZ3dZ4dZ5de,de,fdZ6d Z7dFd!Z8e9dfd"Z:e9dfd#e,d$e,d%e-d&e,dee,eff d'Z;dGd(Z dHd+Z?e@ddfd,ZAdId-ZBd.ZCdGd#e,d/eDd0e,d1e,deDf d2ZEd3ZFd4ZGd5ZHd6ZId7ZJd#e,de,fd8ZKd#e,d9e,de.fd:ZLd;eDe,deDe,fd<ZMd=ZNd>e,de,fd?ZOd@e,dee,e,ffdAZPdBZQdJdCe-fdDZRy)KN)urlparse)ThreadPoolExecutor as_completed) BeautifulSoupCommentelementTagNavigableString)PROMPT_EXTRACT_BLOCKS)*)Path)DictAny)urljoin) InvalidSchema)OptionalTuplerr)ForeStyleinitwrapsc eZdZy)InvalidCSSSelectorErrorN)__name__ __module__ __qualname__C/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/utils.pyrrsr rmessagetypewidth add_newlines double_linereturnc 6ttjtjdftjtj dftj tjdftjtjdfd}|j|j|d\}}}ddd } |rd nd } | | \} } } }}}g}|jd }|r|d |dj}tj||dz }|j!|jd |ddD]o}|jrLtjd|j|dz }|j!|jd _|j#dq| |dz z}|| ||g|Dcgc]}|| |d |d|dz || c}||||t$j&}d j)|}|rd |d }|Scc}w)a Create a styled message box with colored borders and formatted text. How it works: 1. Determines box style and colors based on the message type (e.g., info, warning). 2. Wraps text to fit within the specified width. 3. Constructs a box using characters (single or double lines) with appropriate formatting. 4. Adds optional newlines before and after the box. Args: message (str): The message to display inside the box. type (str): Type of the message (e.g., "info", "warning", "error", "success"). Defaults to "info". width (int): Width of the box. Defaults to 120. add_newlines (bool): Whether to add newlines before and after the box. Defaults to True. double_line (bool): Whether to use double lines for the box border. Defaults to False. Returns: str: A formatted string containing the styled message box. u⚠uℹu✓×)warninginfosuccesserrorr+)u─u│u┌u┐u└u┘)u═u║u╔u╗u╚u╝)singledoubler/r.  r)r$r N <)rrYELLOWLIGHTYELLOW_EXBLUE LIGHTBLUE_EXGREEN LIGHTGREEN_EXRED LIGHTRED_EXgetlowersplitstriptextwrapfillextendappendr RESET_ALLjoin)r"r#r$r%r&styles border_color text_colorprefix box_chars line_styleh_linev_linetltrblbrformatted_lines raw_lines first_line wrapped_firstlinewrappedhorizontal_lineboxresults r!create_box_messager^s5* FKK!4!4e<D--u5JJ 2 2E:((D,,d3 F(.zz$**,v'O$L*f=<I)hJ%.z%:"FFBBO d#Ixq1!3!3!5 67  ja@ }22489abMDzz|"--"TZZ\N(;57K&&w}}T':;&&r* " *O ._-bT2 ds tds\`\N6(:,aQuQwiK/@vh Wds t  ._-bT%//1BC C YYs^FfXR M usHctj}tdz }td|dz}t |dz }t ||S)a Calculate the optimal semaphore count based on system resources. How it works: 1. Determines the number of CPU cores and total system memory. 2. Sets a base count as half of the available CPU cores. 3. Limits the count based on memory, assuming 2GB per semaphore instance. 4. Returns the minimum value between CPU and memory-based limits. Returns: int: The calculated semaphore count. i@r r6)os cpu_countget_system_memorymaxintmin)ra memory_gb base_countmemory_based_caps r!calculate_semaphore_countriesI I!#y1IQ Q'J9q=) z+ ,,r c tj}|dk(rYtdd5}|D]=}|jdst |j ddzccdddS dddy|dk(r?d dl}|jgd jd }t |jS|d k(rd dl jj} jG fd d j}|} j||_|j# j%||j&St)d#1swYyxYw)af Get the total system memory in bytes. How it works: 1. Detects the operating system. 2. Reads memory information from system-specific commands or files. 3. Converts the memory to bytes for uniformity. Returns: int: The total system memory in bytes. Raises: OSError: If the operating system is unsupported. Linuxz /proc/meminforz MemTotal:r iNDarwinr)sysctlz-nz hw.memsizeutf-8Windowsc eZdZdWjfdWjfdWfdWfdWfdWfdWfdWfd Wfg Zy ) )get_system_memory..MEMORYSTATUSEXdwLength dwMemoryLoad ullTotalPhys ullAvailPhysullTotalPageFileullAvailPageFileullTotalVirtualullAvailVirtualullAvailExtendedVirtualN)rrrc_ulong_fields_) c_ulonglongctypessr!MEMORYSTATUSEXrrsZV^^,0--#[1#[1"K0"K0*K8 Hr rzUnsupported operating system)platformsystemopen startswithrdrA subprocess check_outputdecoderBrwindllkernel32r~ StructuresizeofrsGlobalMemoryStatusExbyrefruOSError) rmemrYroutputrr memoryStatusr~rs @@r!rbrbys( __ F  /3 '3??;/tzz|A/$66( '( ' 8 (()GHOOPWX6<<>"" 9 ==))((  V-- &' & n = %%fll<&@A(((455;( 'sE E*EEc Ptjjtjdtjdt j d}tj |dtj |ddtj |dd|S)at Get or create the home folder for Crawl4AI configuration and cache. How it works: 1. Uses environment variables or defaults to the user's home directory. 2. Creates `.crawl4ai` and its subdirectories (`cache`, `models`) if they don't exist. 3. Returns the path to the home folder. Returns: str: The path to the Crawl4AI home folder. CRAWL4_AI_BASE_DIRECTORYz .crawl4aiTexist_okz/cachez/models)r`pathrHgetenvrhomemakedirs) home_folders r!get_home_folderrs'',,ryy)CRYYOikoktktkvEwx{FGKKK d+KK;-v&6KK;-w'$7 r chtj|}t|d}|j}|S)z Beautifies an escaped HTML string. Parameters: escaped_html (str): A string containing escaped HTML. Returns: str: A beautifully formatted HTML string. html.parser)htmlunescaperprettify) escaped_htmlunescaped_htmlsoup pretty_htmls r! beautify_htmlrs1]]<0N  7D--/K r c|jdr$|jdr|ddj}g}d}d}t|D]?\}}|dk(r |dk(r|}|dz }|dk(s|dz}|dk(s)|j |||dzAg}g}|D])} t j |} |j | +||fS#t j$r|j |YVwxYw)a Splits a JSON string which is a list of objects and tries to parse each object. Parameters: json_string (str): A string representation of a list of JSON objects, e.g., '[{...}, {...}, ...]'. Returns: tuple: A tuple containing two lists: - First list contains all successfully parsed JSON objects. - Second list contains the string representations of all segments that couldn't be parsed. []r r{})rendswithrB enumeraterFjsonloadsJSONDecodeError) json_stringsegmentsdepth start_indexicharparsed_objectsunparsed_segmentssegmentobjs r!split_and_parse_json_objectsrs c"{';';C'@!!B'--/ H EK[)4 3;z QJE S[ QJEz K! <=*N .**W%C  ! !# & , ,,## .  $ $W - .s&B??$C&%C&cN|}|jddjdd}|S)a< Sanitize an HTML string by escaping quotes. How it works: 1. Replaces all unwanted and special characters with an empty string. 2. Escapes double and single quotes for safe usage. Args: html (str): The HTML string to sanitize. Returns: str: The sanitized HTML string. "\"'z\')replace)rsanitized_htmls r! sanitize_htmlrs0 N$++C7??UKN r textc. |sy|jddjdS#t$r:}td||jddjdcYd}~Sd}~wwxYw#t$r}t dt ||d}~wwxYw) z3Sanitize input to handle potential encoding issues.r4roignore)errorszFWarning: Encoding issue detected. Some characters may be lost. Error: asciiNzError sanitizing input: )encoderUnicodeEncodeErrorprint Exception ValueErrorstr)res r!sanitize_input_encoders E I;;wx;8??H H! I Z[\Z]^ _;;wx;8??H H I E3CF8<=1DEs<(!( A+/A& A+!A.&A++A.. B7BBc2|jdd}|jdd}|jdd}|jdd}|jd d }|jd d }|jd d}tjdd|}|S)z Escapes characters in a string to be JSON safe. Parameters: s (str): The input string to be escaped. Returns: str: The escaped string, safe for JSON encoding. \z\\rrz\b z\fr0z\n z\r z\tz[\x00-\x1f\x7f-\x9f]cRdjt|jS)Nz\u{:04x})formatordgroup)xs r!z$escape_json_string..<s+2D2DS^2Tr )rresub)ss r!escape_json_stringr#s $A #uA $A $A $A $A $A &(TVWXA Hr ciddddddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'}|Dcgc]}||j|d(f}}|D]B\}}|j|D])}|r |jn||}|j|+D|Scc}w))aN Replace inline HTML tags with Markdown-style equivalents. How it works: 1. Maps specific tags (e.g., , ) to Markdown syntax. 2. Finds and replaces all occurrences of these tags in the provided BeautifulSoup object. 3. Optionally replaces tags with their text content only. Args: soup (BeautifulSoup): Parsed HTML content. tags (List[str]): List of tags to replace. only_text (bool): Whether to replace tags with plain text. Defaults to False. Returns: BeautifulSoup: Updated BeautifulSoup object with replaced tags. bc"d|jdSNz**rtags r!rz%replace_inline_tags..S2chhZr*r rc"d|jdSNr rrs r!rz%replace_inline_tags..T1SXXJar uc"d|jdS)N__rrs r!rz%replace_inline_tags..Urr spanc|jSNrrs r!rz%replace_inline_tags..V sxxjMr delc"d|jdSNz~~rrs r!rz%replace_inline_tags..WRz,r insc"d|jdS)Nz++rrs r!rz%replace_inline_tags..Xrr rc"d|jdS)N~rrs r!rz%replace_inline_tags..YQsxxj?r supc"d|jdS)Nz^^rrs r!rz%replace_inline_tags..Zrr strongc"d|jdSrrrs r!rz%replace_inline_tags..[s388*B/r emc"d|jdSrrrs r!rz%replace_inline_tags..\sAchhZq/r codec"d|jdSN`rrs r!rz%replace_inline_tags..]azOr kbdc"d|jdSrrrs r!rz%replace_inline_tags..^rr varc"d|jdSN_rrs r!rz%replace_inline_tags.._rr rc"d|jdSrrrs r!rz%replace_inline_tags..`rr qc"d|jdS)Nrrrs r!rz%replace_inline_tags..arr abbrcF|jd|jdddS)Nz (titler4))rr?rs r!rz%replace_inline_tags..bs!sxxj3777B+?*@Br citec"d|jdSr rrs r!rz%replace_inline_tags..crr c"d|jdSr rrs r!rz%replace_inline_tags..drr c|jSrrrs r!rz%replace_inline_tags..err c"d|jdS)Nzzrrs r!rz%replace_inline_tags..fswsxxj9r c"d|jdS)Nz==rrs r!rz%replace_inline_tags..gsb "-r )dfntimesmallmarkc|jSrr)ts r!rz%replace_inline_tags..js!&&r )r?find_allr replace_with) rtags only_texttag_replacementsrreplacement_datatag_namereplacement_funcreplacement_texts r!replace_inline_tagsr(@sx$ * ( * )  ,  ,  * , / ) + * * * ( B!" +#$+)9-+0W[[VZs.2238HIJVZ[&6""==*C+4sxx:J3:O    - .+'7 K\sB=c ~ |syt|d}|j}|rL|j|}|std||j d}|D]} |j | |}ggd} |j ddD]{} | d } |jd d } | jd r*| | vr&| d j | | jdW| dj | | jd}|j gdD]}|j|j D]}|jdk7si|_ gggd}|j dD]8}|dj |jd|jddd:|j dD]8}|dj |jd|jddd:|j dD]8}|dj |jd|jddd:|j dD]F}|jd}|r!|j|j|7|jHd}||}t!|gd|jdd }fd!||}t"fd"t$d#t&fd$}|||}d%t$ffd& d"t$ffd' }||}fd(|}|j d)*D]}|j)t+|j-d+d,j-d-d.}t/|}t0j3}t5}d|_|j9|}|j-d/d0} t;||}||d|| |d2S#t<$r!}t?d1t+|i}Yd}~.d}~wwxYw#t<$r)}t?d3t+|td4||d}~wwxYw)5a4 Extract structured content, media, and links from website HTML. How it works: 1. Parses the HTML content using BeautifulSoup. 2. Extracts internal/external links and media (images, videos, audios). 3. Cleans the content by removing unwanted tags and attributes. 4. Converts cleaned HTML to Markdown. 5. Collects metadata and returns the extracted information. Args: url (str): The website URL. html (str): The HTML content of the website. word_count_threshold (int): Minimum word count for content inclusion. Defaults to MIN_WORD_THRESHOLD. css_selector (Optional[str]): CSS selector to extract specific content. Defaults to None. Returns: Dict[str, Any]: Extracted content including Markdown, cleaned HTML, media, links, and metadata. Nrz;Invalid CSS selector , No elements found for CSS selector: divinternalexternalaT)hrefr//r6httpr-r/rr,scriptstylelinkmetanoscriptimgimagesvideosaudiosr;srcaltimage)r>r?r#videor<audior=c\|jdD]}|j|_|S)Npre)rget_textstring)nodechilds r!replace_pre_tags_with_textz:get_content_of_website..replace_pre_tags_with_texts*u-$~~/ .Kr rrrrrrrrrrrrrrr rrrrrrr"F)r"cD|jD]}t|tjs||t |j dj }t |jdk(r|j dr||ks|j|S)NTrBr)contents isinstancerr lenrErA decompose)rGword_count_thresholdrH word_count(remove_empty_and_low_word_count_elementss r!rSzHget_content_of_website..remove_empty_and_low_word_count_elementsseW[[1.remove_small_text_tagssN}}T*::#**"2"2"4!$SZZ%5%5%7%=%=%?!@J!$88&--c2+& &Kr rct|tr|j S|jsyt fd|jDS)NTc3.K|] }|ywrr).0rHis_empty_or_whitespaces r! zIget_content_of_website..is_empty_or_whitespace..%sO,-e4,s)rNr rBrMall)rr[s r!r[z6get_content_of_website..is_empty_or_whitespaces:#/99;&<<O#,,OO Or cd}|rEd}|jdDcgc] }|s |}}|D]}|jd}|rE|Scc}w)NTF)rrP)rTchangesr empty_tagsr[s r!remove_empty_tagsz1get_content_of_website..remove_empty_tags'sbG-1]]4-@`-@cDZ[^D_c-@ `%CMMO"G& K as A A c0|jD]}t|tjs|t |jdk(s?|jdj |j k(sf|jd}|j ||SNr r)rMrNrr rOnamer )rGrH child_contentflatten_nested_elementss r!rfz7get_content_of_website..flatten_nested_elements:sueW[[1+E25>>*a/ENN14E4J4Jejj4X(-q(9 **=9 'Kr c"t|tSr)rNrrs r!rz(get_content_of_website..Js D'9Rr )rF r0r3r1 ``````Error extracting metadata:markdown cleaned_htmlr,medialinksmetadatazError processing HTML content:zInvalid CSS selector: ) rrTselectrnew_tagrFrrArrErPrdattrsr?r  new_stringr(MIN_WORD_THRESHOLDr rdextractrrr html2text HTML2TextCustomHTML2Text ignore_linkshandleextract_metadatarr) urlrrQ css_selectorkwargsrrTselected_elementsdiv_tagelrpr.r/url_baserror9rArBalt_textrIrWracommentrnhrmr7rrfr[rSs @@@r!get_content_of_websiter}sO*\VT=1yy  $ L 9 $-0klxky.z{{ll5)G'r"(D  s.AV9Dyy~a(Hv&84+?j!(( JJL* j!(( $ ! /"==!PQC MMOR==?Cxx5  #   =='C (O " "wwu~wwu~$ (]]7+E (O " "yy'yy'$ ,]]7+E (O " "yy'yy'$ ,=='Cwwu~H  !:;  ( *$/#  `jje4  8>RSJ\  C ,&d,@A P P C  !& 't, }},R}SG OO T4y((6>>tSI %\2     !  88L)##Iu5 #D$/D!(     .A 7D  V .A7%(>|n&MNTUUVsNP EP I8P  OP  P&P=P PP P<$P77P<r~rrQrc |syt|d}|j}jdtjdgxsgD](}|j |D]}|j *|rJ|j |} | st d||jd}| D]}|j|ggdgggdfdfd d tjd tffd |jd fdtDD cgc]} | | c} d<|fd|}tjd} D];} | jdd} | j!| r| j#d| | d<=t%|j'ddj'dd}t)|}t+}d|_|j/|}|j'dd} t1||}||d|dScc} w#YxYw#t2$r!}t5dt%|i}Yd}~:d}~wwxYw)Nr$image_description_min_word_threshold excluded_tagsz:Invalid CSS selector, No elements found for CSS selector: r*r+r:c|}|rB|j}|r1|jdd}t|jk\r|S|rBy)Nr1T) separatorrB)parentrErOrA)r current_tag text_contentrs r!$find_closest_parent_with_useful_textzNget_content_of_website_optimized..find_closest_parent_with_useful_textsUK)00 #.#7#7#D#7#QL<--/04XX++r c0d}d}|||j|jjdgsy|||||}|tkry|jddjddj |jdd||d d S) Nc |jdd}|jdd}gdddg}td|v|tfd||jd dg|D |j|vgS) Nr5r4r>)buttoniconlogorinputz display:nonec34K|]}D]}||v ywrr)rZrrclasses_to_checks r!r\zbget_content_of_website_optimized..process_image..is_valid_image..s#q,VS`p[\S`p,Vsr?)r?r]anyrd)r9rparent_classesr5r> tags_to_checkrs @r!is_valid_imagezOget_content_of_website_optimized..process_image..is_valid_imagesGGGR(E''%$C9 %w/Me+qS#''%2D,V~,Vqqq =0  r c<d}d}|jd}||\}}|jd} || \} } d} tjj|jdddj j d d} |r|d k(r |d kDr| dz } |d vr |d kDr| dz } | r| d k(r | d kDr| dz } | d vr | d kDr| dz } | dkDr| dz } |jddk7r| dz } t fddDr| dz } ||z dkr| dz } | S)Nc|rKtjd|}|r3t|jd}|jdxsd}||fSy)Nz (\d+)(\D*)r r6px)NN)rmatchrdr) dimensionrnumberunits r!parse_dimensionztget_content_of_website_optimized..process_image..score_image_for_usefulness..parse_dimensionsIHH]I>E!$U[[^!4${{1~5%t|+!r ct||jd} tj|}|jdk(r|j jddyt d| y#t$r }Yd}~yd}~wwxYw#YyxYw)Nr>zContent-Lengthz!Failed to retrieve file size for )rr?requestshead status_codeheadersrr)r9base_urlimg_urlresponsers r!fetch_image_file_sizezzget_content_of_website_optimized..process_image..score_image_for_usefulness..fetch_image_file_sizes!(3775>: '}}W5H++s2'//334DTJ  A'KL#%  s)?A.A.. B7B<BBBheightr$rr>r4r .r)%vhvminvmaxi'r?c3(K|] }|k( ywrr)rZr image_formats r!r\znget_content_of_website_optimized..process_image..score_image_for_usefulness..sK6JF<'6Js)jpgpngwebp?)r?r`rsplitextr@rBr)r9rindex images_countrr image_height height_value height_unit image_width width_value width_unit image_sizescorers @r!score_image_for_usefulnessz[get_content_of_website_optimized..process_image..score_image_for_usefulnesssT " 778,L(7 (E %L+777+K&5k&B #KJ77++CGGE",=>qAGGIL'--c2LE$&<#+=QJE"::|R?OQJE%+*;QJE!99k2oQJEE! wwu~#qK6JKKq\!#%qLr classr>r4rrr?r@)r>r?descrr#)rr?IMAGE_SCORE_THRESHOLDrrB)r9r~r total_imagesrrrrs r! process_imagez7get_content_of_website_optimized..process_images 6 pc3::szz~~gr/JK*3ULI ) )775"%--eS9??A775"%8=   r rr'c 0 t|tr"t|tr|jy|jdvr|j yd}|jdk(r|j drn|d}jdd}||jd}|jdr||vr d j|n d j|d }n|jd k(ry |jd vrʼn |jdj|j d|j d|j |d|jd}|D]W} |jdj|j d|j d|j |dYy |jdk7rj|jdvrF j ddr"|j|jn(|jn|jd k7ri|_t|j D]M}t|tr0t|ts t#|j%dkDs@d }C|sLd }O|s/t#|jd j}|k\}|s|j |S#t&$r} t)dt+| Yd} ~ yd} ~ wwxYw)NFr3r.r/r0r6r2r1r-r,Tr9)rArBrr>r?)r>r?r# descriptionsourcerDrJr"rrLzError processing element:)rNr rrwrdrPr?rArErrFrr unwraprtlistchildrenrOrBrrr)r keep_elementr/r link_data source_tags source_tagrHrRrrrrproprocess_elementr~rQs r!rz9get_content_of_website_optimized..process_elementsH '?3gw/#GOO%||NN!!!# L||s"{w{{6':v99S>!,%)3C73C3C3EF ??6*xt/C*%,,Y7*%,,Y7# &!33a()00&7;;u-&7;;u-#LL#G#P 2 /g..x8 "-JW\\N!,-44%>>%0&7;;u-#LL#G#P 6#.||u$<<$wwzz+u5,,,-=W-=-=-?@&(\\U*$&GMg../e_5jPW>X5;;=)A-'+ &u-'+ 0  !1!1!1!=!C!C!EF )-AA !!!#   -s1v 6 s>1K-K-B K-6CK- C K- K-#A K-- L6LLr9c 3LK|]\}}||tywr)rO)rZrr9imgsrr~s r!r\z3get_content_of_website_optimized..8s%M_61csCCI ._s!$r;ctt|tr|St|jdk(rbt|jdtj r;|jdj |j k(r|jdS|jDcgc] }| c}|_|Scc}wrc)rNr rOrMrr rd)rGrHrfs r!rfzAget_content_of_website_optimized..flatten_nested_elements>s dO ,K t}}  "z$--2BGKK'PUYUbUbcdUeUjUjnrnwnwUw*4==+;< r4rhr0r3r1Trirjrkrl)rrTr?$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDrrrPrrsrFr PageElementboolrrrcompilerrrrrrzr{r|r}rr)r~rrQrrrrTrrrr]base64_patternr9r>rnrrmr7rrrfrrrprorrs` ` ` @@@@@@@@r! get_content_of_website_optimizedrps~  } -D 99D+1::6\_C,D(zz/2.4"4++c"B LLN#5 KK 5 ),fgsft*uv v||E"#B KKO$ ,ER2 6E R hI!4!4IIIX == D NYt_MM   ME(O D #4 (DZZ BCN ''%$C##C(+//C8E t9$$VT2::4EL .LAANxx %H 51Hd+ $  Q.    *CF3s** H%+8H* H1*H.1 I:IIci}|s|siS|s t|d}|j}|s|S|jd}|r&|jr|jj nd|d<|jdddi}|r |j dd j nd|d<|jddd i}|r |j dd j nd|d <|jddd i}|r |j dd j nd|d <|j dd tjd i}|D]M} | j d d j } | j dd j } | sF| sI| || <O|j ddtjdi} | D]M} | j dd j } | j dd j } | sF| sI| || <O|S)aR Extract optimized content, media, and links from website HTML. How it works: 1. Similar to `get_content_of_website`, but optimized for performance. 2. Filters and scores images for usefulness. 3. Extracts contextual descriptions for media files. 4. Handles excluded tags and CSS selectors. 5. Cleans HTML and converts it to Markdown. Args: url (str): The website URL. html (str): The HTML content of the website. word_count_threshold (int): Minimum word count for content inclusion. Defaults to MIN_WORD_THRESHOLD. css_selector (Optional[str]): CSS selector to extract specific content. Defaults to None. **kwargs: Additional options for customization. Returns: Dict[str, Any]: Extracted content including Markdown, cleaned HTML, media, links, and metadata. lxmlrNr7rdr)rtcontentr4keywordsauthorpropertyz^og:z ^twitter:) rrfindrFrBr?rrr) rrrqr title_tagdescription_tag keywords_tag author_tagog_tagsr property_namer twitter_tagss r!r}r}gs,H  T6* 99D  '"I4=)BRBR ((..0X\HWiiv}.EiFOL[o11)R@FFHaeH]99VFJ+?9@LFR<++Ir:@@BX\HZ6&();RVHXmmF:rzz'7J*KmLG B/557 '')R(..0 W&-H] # == <8P/Q=RL+113 '')R(..0 W&-H] #  Or cVtjd|}tt|S)z Extracts XML tags from a string. Args: string (str): The input string containing XML tags. Returns: List[str]: A list of XML tags extracted from the input string. z<(\w+)>)rfindallrset)rFr!s r!extract_xml_tagsrs" ::j& )D D ?r ci}|D]Z}d|d|d}tj||tj}|r#|jdj ||<Vd||<\|S)a Extract data for specified XML tags from a string. How it works: 1. Searches the string for each tag using regex. 2. Extracts the content within the tags. 3. Returns a dictionary of tag-content pairs. Args: tags (List[str]): The list of XML tags to extract. string (str): The input string containing XML data. Returns: Dict[str, str]: A dictionary with tag names as keys and extracted content as values. r5z>(.*?)r r4)rsearchDOTALLrrB)r!rFdatarpatternrs r!extract_xml_datarsj" DcU(3%q) '62995  A,,.DIDI  Kr c ddlm}ddlm}d}d} d||d} |rdd i| d <|j d r| j |d t |D]} |d|d |d gd| } | cSy#|$rd} tdt| | |dz kr-| d| zz}td|dtj|nddgdgdgcYd} ~ cSYd} ~ }d} ~ wwxYw)a Perform an API completion request with exponential backoff. How it works: 1. Sends a completion request to the API. 2. Retries on rate-limit errors with exponential delays. 3. Returns the API response or an error after all retries. Args: provider (str): The name of the API provider. prompt_with_variables (str): The input prompt for the completion request. api_token (str): The API token for authentication. json_response (bool): Whether to request a JSON response. Defaults to False. base_url (Optional[str]): The base URL for the API. Defaults to None. **kwargs: Additional arguments for the API request. Returns: dict: The API response or an error message after all retries. r) completion)RateLimitErrorr6{Gz?) temperatureapi_keyrr# json_objectresponse_format extra_argsuserroler)modelmessageszRate limit error:r z Waiting for z seconds before retrying...r-z)Rate limit error. Please try again later.)rr!rNr) litellmrlitellm.exceptionsrr?updaterangerrrsleep)providerprompt_with_variables api_token json_responserrrr max_attempts base_delayrattemptrrdelays r!perform_completion_with_backoffrs8#1LJJ *0-(A $% zz,&./&  #0EF  HO'  %s1v .))"a7l3 UG+FGH 5!$I KL" sA))C.AC C Cc@|stj|dn|}|tt|d}t}|D]}|j d|zdz||}t ||||} tdg|jdjjd} tj| } | D]} d| d <  | S#t$rU} t|jdjj\} } | } | r| jdd d g| d Yd} ~ | Sd} ~ wwxYw) a Extract content blocks from website HTML using an AI provider. How it works: 1. Prepares a prompt by sanitizing and escaping HTML. 2. Sends the prompt to an AI provider with optional retries. 3. Parses the response to extract structured blocks or errors. Args: url (str): The website URL. html (str): The HTML content of the website. provider (str): The AI provider for content extraction. Defaults to DEFAULT_PROVIDER. api_token (Optional[str]): The API token for authentication. Defaults to None. base_url (Optional[str]): The base URL for the API. Defaults to None. Returns: List[dict]: A list of extracted content blocks. NURLHTMLrrrblocksrFr-T)rr-r!r)PROVIDER_MODELSr?rrr rrrchoicesr"rrrrrrF)r~rrrrvariable_valuesrvariablerrblockrparsedunparseds r!extract_blocksr&s>*.>q.A.I.I.Q.QRS[\F#E"E'N M  78H8H8K8S8S8[8[\  MM #   M s*AB?? DA DDc|stjddn|}ddlm}g}|D]G\}}||d}t}|D]} |j d| zdz|| }|j d|d gI|||d } g} | D]Z} td g| jdjjd } tj| } | j | \t| gS#t$r}dd gdgdgdg} Yd}~8d}~wwxYw)at Extract content blocks from a batch of website HTMLs. How it works: 1. Prepares prompts for each URL and HTML pair. 2. Sends the prompts to the AI provider in a batch request. 3. Parses the responses to extract structured blocks or errors. Args: batch_data (List[Tuple[str, str]]): A list of (URL, HTML) pairs. provider (str): The AI provider for content extraction. Defaults to "groq/llama3-70b-8192". api_token (Optional[str]): The API token for authentication. Defaults to None. Returns: List[dict]: A list of extracted content blocks from all batch items. GROQ_API_KEYNr)batch_completionrrrrrr)r r rrr-zZError extracting blocks from the HTML content. Choose another provider/model or try again.z4What went wrong during the block extraction process?)rr!r questions)r`rr r)r rrFrr r"rrrrsum) batch_datarrr)r r~rr!rr" responses all_blocksrrrs r!extract_blocks_batchr/Ts>$8A .$/iI(H T  !6'H$9$A$Ah$oh&?% !( &5JKLM !I J %xj(2B2B12E2M2M2U2UVW_`FZZ'F &! z2   xyTU F sAC'' D0 DDc(g}g}d}|D]d}t|jdz}||z|kr|j|||z }>|r |jdj||g}|}f|r |jdj||S)a Merges small chunks into larger ones based on the total token threshold. :param chunks: List of text chunks to be merged based on token count. :param token_threshold: Max number of tokens for each merged chunk. :return: List of merged text chunks. rg?rh)rOrArFrH)chunkstoken_thresholdmerged_sections current_chunktotal_token_so_farchunkchunk_token_counts r!%merge_chunks_based_on_token_thresholdr8sOM .4  1 1O C   ' "3 3 &&v{{='AB"GM!2 v{{=9: r sectionsrrc g}|jdr=|D]6}|jt|||||tjd8|St 5}|Dcgc]}|j t|||||}}t|D]!} |j| j# ddd|Scc}w#1swY|SxYw)a Process sections of HTML content sequentially or in parallel. How it works: 1. Sequentially processes sections with delays for "groq/" providers. 2. Uses ThreadPoolExecutor for parallel processing with other providers. 3. Extracts content blocks for each section. Args: url (str): The website URL. sections (List[str]): The list of HTML sections to process. provider (str): The AI provider for content extraction. api_token (str): The API token for authentication. base_url (Optional[str]): The base URL for the API. Defaults to None. Returns: List[dict]: The list of extracted content blocks from all sections. zgroq/rrN) rrEr&rrrsubmitrr]) r~r9rrrextracted_contentsectionexecutorfuturesfutures r!process_sectionsrAs(7#G  $ $^C(I`h%i j JJsO    !X}EF}Eqxx~sGXyckl}EGF&w/!((90"  F" sC "B>1C>CC c0g}|j}|rrd}|rZ|jd||dz|d|kr:||jddzz }|r!|jd||dz|d|kr:|j||rrdj |S)a3 Wrap text to fit within a specified width for rendering. How it works: 1. Splits the text into words. 2. Constructs lines that fit within the maximum width using the provided font. 3. Returns the wrapped text as a single string. Args: draw (ImageDraw.Draw): The drawing context for measuring text size. text (str): The text to wrap. font (ImageFont.FreeTypeFont): The font to use for measuring text size. max_width (int): The maximum width for each line. Returns: str: The wrapped text. r4)rrr)fontr6r1r0)rAtextbboxpoprFrH)drawrrC max_widthlineswordsrYs r! wrap_textrJs( E JJLE  fdU1XoD I!LPYY UYYq\C' (D fdU1XoD I!LPYY T  99U r c:t|d}|jS)aO Prettify an HTML string using BeautifulSoup. How it works: 1. Parses the HTML string with BeautifulSoup. 2. Formats the HTML with proper indentation. 3. Returns the prettified HTML string. Args: html_string (str): The HTML string to format. Returns: str: The prettified HTML string. z lxml.parser)rr) html_stringrs r! format_htmlrMs m 4D ==?r cFd}d}g}d}|jddjddjd}|D]}|js|jd r|d z}|j ||z|zB|jdr)|j d r|j ||z|z||jdr|j ||z|z|d z }|j}|s|j ||z|zdj |S) z A fast HTML formatter that uses string operations instead of parsing. Args: html_string (str): The HTML string to format Returns: str: The formatted HTML string rr3Frz> r5z )rrArBrrFrrH)rLindent indent_str formatted in_contentpartspartrs r!fast_format_htmlrUs"FJIJ   U + 3 3C ? E Ed KEzz|  ??4 aKF   Z&047 8__S !dmmD&9   Z&047 8__S !   Z&047 8 aKFjjlG  f!4w!>?-0 99Y r cddlm}m}||}|jr |jst d||||j }|S)*Normalize URLs to ensure consistent formatr)rrInvalid base URL format: ) urllib.parserrschemenetlocrrB)r/rrr parsed_base normalizeds r! normalize_urlr^-sO.8$K   [%7%74XJ?@@4::<0J r c |jd}|d}|d}hd}tfd|Drj Sj dr|Sj dr|Sj dr |d|Sj d sj d |d|dSj S#t$rtd|wxYw) rWr0rr6rXftp:tel:data:file:mailto: javascript:c3\K|]#}jj|%ywrr@r)rZprotor/s r!r\z$normalize_url_tmp..Fs% I7He4::< " "5 )7H),#z//)zhttp://zhttps://z./)rA IndexErrorrrrBrlstrip)r/r base_partsprotocoldomainspecial_protocolss` r!normalize_url_tmprr:sA^^C( a=A U I7H IIzz| sD6"" tD6"" s2fXdV,, ??2 3{{4 2fXQtf-- ::<5 A4XJ?@@As C C"c` t|jj}|sy|jdd}t j dd|}|jd}t |dkDr|ddvrdj|d d Sdj|dd S#t$rYywxYw)  Extract the base domain from a given URL, handling common edge cases. How it works: 1. Parses the URL to extract the domain. 2. Removes the port number and 'www' prefix. 3. Handles special domains (e.g., 'co.uk') to extract the correct base. Args: url (str): The URL to extract the base domain from. Returns: str: The extracted base domain or an empty string if parsing fails. r4:rz^www\.rr6> acadaeafagcocomedugovrdmilnetorgN) rr[r@rArrrOrHr)r~rprSs r!get_base_domainr]s#%%++-c"1% 2v. S! u:>eBi,  88E"#J' 'xxbc ## s%B!A$B! B!! B-,B- base_domainc>hd}tfd|Dry t}|jsy|jjj dd}|jj dd}|j | S#t $rYywxYw)rtr`c3\K|]#}jj|%ywrrh)rZpr~s r!r\z"is_external_url..s# 6g399; ! !! $grjTFzwww.r4)rrr[r@rrr)r~rspecialr$ url_domainbases` r!is_external_urlrsKG 6g 66 #}}]]((*2262>   "**626&&t,,, sBAB BBtokenschd}hd}|Dcgc]M}t|dkDr=||vr9||vr5|jds$|jds|jds|Oc}Scc}w)u Clean a list of tokens by removing noise, stop words, and short tokens. How it works: 1. Defines a set of noise words and stop words. 2. Filters tokens based on length and exclusion criteria. 3. Excludes tokens starting with certain symbols (e.g., "↑", "▲"). Args: tokens (list[str]): The list of tokens to clean. Returns: list[str]: The cleaned list of tokens. >⬆️r.anatbyinofontoupccpthe↑▲>n'tcan'twon'tmustn'tcouldn'twouldn't shouldn'tr.ramrasrberdoherisitmemynorrorsorruswer]andrarebutcandidfewforhadhasherhimhishowitsmaynornotoffouroutsherwaswhowhyyetyoubeenbothdoesdowneachfromhavehersintominemoremostmustnearnoneoursoverpastsomesuchthatthemtheythisuponwerewhatwhenwhomwillwithyouraboutaboveafteralongamongbeingbelowcoulddoingmightothershallsincetheirthesethoseunderuntilwherewhichwhosewouldyoursacrossaroundbeforebehindbesidebeyondcannotduringexcepthavinginsideitselfmyselfshouldtheirstowardunlesswithinagainstbecausebeneathbetweenherselfhimselfoutsidethroughalthoughyourself ourselves themselves underneathr6rru⬆)rOr)rnoise STOP_WORDStokens r! clean_tokensr$s|" hE(JV & -ve5zA~U"Z'$$U+$$U+$$U+ v -- -sAA"c.tfd}|S)a Decorator to profile a function's execution time and performance. How it works: 1. Records the start time before executing the function. 2. Profiles the function's execution using `cProfile`. 3. Prints the elapsed time and profiling statistics. Args: func (Callable): The function to decorate. Returns: Callable: The decorated function with profiling and timing enabled. cltj}tj}|j |g|i|}|j tj|z }t d|ddtj|}|jd|jd|S)Nz![PROFILER] Scraping completed in z.2fz seconds cumulative) r perf_countercProfileProfileenabledisablerpstatsStats sort_stats print_stats) selfargsr start_timeprofilerr] elapsed_timestatsfuncs r!wrapperz!profile_and_time..wrappers&&( ##%d,T,V, ((*Z7  1,s1C8LM X& & " r r)r8r9s` r!profile_and_timer:s"  4[4 Nr rcdtj|jjS)z"Generate a unique hash for content)xxhashxxh64r hexdigest)rs r!generate_content_hashr?s! <<( ) 3 3 55r base_pathcddddddd}i}|jD]A\}}tjj||}tj|d|||<C|S) z.Create content directories if they don't exist html_contentrnmarkdown_contentr< screenshots)rcleanedrm extractedrD screenshotTr)itemsr`rrHr)r@dirs content_pathskeydirnamers r!ensure_content_dirsrMsl!&($#  DM Www||Iw/ D4(! c% r ctjdk(r(tjtjyy)a, Configure the Windows event loop to use ProactorEventLoop. This resolves the NotImplementedError that occurs on Windows when using asyncio subprocesses. This function should only be called on Windows systems and before any async operations. On non-Windows systems, this function does nothing. Example: ```python from crawl4ai.async_configs import configure_windows_event_loop # Call this before any async operations if you're on Windows configure_windows_event_loop() ``` rpN)rrasyncioset_event_loop_policyWindowsProactorEventLoopPolicyrr r!configure_windows_event_looprR0s/ I%%%g&L&L&NO&r context_linescddl}ddl}ddl}|j|d}|d}|j}|j }|j } td||z } ||zdz} g}t| | D]J} |j|| } | s| j} | |k(rdnd}|j| dd|d| Ld j|} |jj|}||| |d S#t$r|}YwxYw) a Extract error context with more reliable line number tracking. Args: exc_info: The exception info from sys.exc_info() context_lines: Number of lines to show before and after the error Returns: dict: Error context information rNr6rr u→r14dr0)filenameline_nofunction code_context) traceback linecacher` extract_tbrVlinenordrcrgetlinerstriprFrHrrelpathr)exc_inforSrZr[r`tb last_framerVrW func_name context_start context_endrrYpointerrYrel_paths r!get_error_contextriCs*    hqk *BBJ""HGI7]23MM)A-KM =+ .  1- ;;=D G|eG  Ab67)1TF!; < /99]+L77??8, $   sC00 C>=C>)r+xTF)Fr)FN)zgroq/llama3-70b-8192N))SrrYrconcurrent.futuresrrbs4rrrr r rrrr`rpromptsr configpathlibrtypingrrrrrequests.exceptionsrrrr<coloramarrrrCr*r. functoolsrrOrrrrdrr^rirbrrrrrrr(rvrrr}rrrDEFAULT_PROVIDERr&r/r8rrArJrMrUr^rrrrr$r:r?rMrRrirr r!rvsg !?EE * --- &&  i EE3EEY]EswEEHEN-(/6b&&*-X0 E E E :1z>P`dqVfWiCu#uSuPSux{uRVWZ\_W_R`unBH B  FP*:tX\5n:x:!#!!!!`d!F:&+ Z !F%%%N34BD-cD-tCyD-L+Z63636 34S>&P&6s6r