g$ddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z mZddlZddlmZej$d Zd ed ed efd ZGddeZGddeZy))ABCabstractmethod)OptionalDictAnyTuple)MarkdownGenerationResult)CustomHTML2Text)RelevantContentFilterBM25ContentFilterN)urljoinz+!?\[([^\]]+)\]\(([^)]+?)(?:\s+"([^"]*)")?\)baseurlreturnc|jdr|S|jdr|jdr|dd|zS||zSt||S)z"Fast URL joining for common cases.)http://https://mailto:z///N) startswithendswithr)rrs Z/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/markdown_generation_strategy.py fast_urljoinr sS ~~>?  ~~c == 9s? "cz 4 ceZdZdZd deedeeeeffdZ e d dededeeeefdeed e d e f d Z y)MarkdownGenerationStrategyz7Abstract base class for markdown generation strategies.Ncontent_filteroptionsc(||_|xsi|_yN)rr )selfrr s r__init__z#MarkdownGenerationStrategy.__init__s,}" r cleaned_htmlbase_urlhtml2text_options citationsrc y)z$Generate markdown from cleaned HTML.N)r#r%r&r'rr(kwargss rgenerate_markdownz,MarkdownGenerationStrategy.generate_markdowns rNN)NNT)__name__ __module__ __qualname____doc__rr rstrrr$rboolr r,r*rrrrsA%x0E'F%X`aefiknfnaoXp%*,GKKO+/  '* #& -5T#s(^,D *22G)H  %)  '?   rrceZdZdZd deedeeeefffd Z ddedede eeffdZ dd eded eeeefdeeeefdeed e de fd ZxZS)DefaultMarkdownGeneratoraw Default implementation of markdown generation strategy. How it works: 1. Generate raw markdown from cleaned HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. options (Optional[Dict[str, Any]]): Additional options for markdown generation. Defaults to None. Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. rr c&t|||yr")superr$)r#rr __class__s rr$z!DefaultMarkdownGenerator.__init__9s 1rmarkdownr&rc@i}i}g}d}d}tj|D]}|j|||j|j \} } } |r)| j ds| |vrt || || <|| } | |vrOg} | r| j| | r| | k7r| j| || rddj| zndf|| <|dz }|| d} |j|jdj ds| d| d nd | d| d |j}|j||d dj|}d g}|jdt|jdD|dj|fS)a" Convert links in markdown to citations. How it works: 1. Find all links in the markdown. 2. Convert links to citations. 3. Return converted markdown and references markdown. Note: This function uses a regex pattern to find links in markdown. Args: markdown (str): Markdown text. base_url (str): Base URL for URL joins. Returns: Tuple[str, str]: Converted markdown and references markdown. rr )rrrz: z - r.!⟨u⟩z![u⟩]Nz ## References c3>K|]\}\}}d|d||dyw)r=u⟩  Nr*).0rnumdescs r zFDefaultMarkdownGenerator.convert_links_to_citations..os4 $S [c4#d3%vR ($Sc|ddS)Nr rr*)xs rzEDefaultMarkdownGenerator.convert_links_to_citations..qs 1Q4PQ7r)key) LINK_PATTERNfinditerappendstartgroupsrrjoingroupendextendsorteditems)r#r:r&link_map url_cachepartslast_endcountermatchtextrtitlerBrAconverted_text referencess rconvert_links_to_citationsz3DefaultMarkdownGenerator.convert_links_to_citations<s& !**84E LL(5;;=9 :$||~ D#u/Q Ri'%1(C%@IcNn("$++e,DEM4;;t+>+;AR$S  rwwz222rr%r'r(c  t|}ddddddddd} |r| j|n;|r| j|n'|jr| j|j|jdi| |sd}nt |t s t |} |j |} | jd d } | } d} |r |j| |\} } d}d}|s |jrK |xs |j}|j|}d jd |D}|j |}t| xsd| xsd| xsd|xsd|xsdS#t$r} dt | } Yd} ~ d} ~ wwxYw#t$r} | } d t | } Yd} ~ d} ~ wwxYw#t$r} dt | }d}Yd} ~ d} ~ wwxYw#t$r(} dt | }t||dddcYd} ~ Sd} ~ wwxYw)a Generate markdown with citations from cleaned HTML. How it works: 1. Generate raw markdown from cleaned HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: cleaned_html (str): Cleaned HTML content. base_url (str): Base URL for URL joins. html2text_options (Optional[Dict[str, Any]]): HTML2Text options. options (Optional[Dict[str, Any]]): Additional options for markdown generation. content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. citations (bool): Whether to generate citations. Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. )baseurlrFT) body_widthignore_emphasis ignore_links ignore_images protect_linkssingle_line_break mark_code escape_snobr.z#Error converting HTML to markdown: Nz ```z```zError generating citations: r?c3>K|]}dj|yw)z
{}
N)format)r@ss rrCz=DefaultMarkdownGenerator.generate_markdown..s-_Q^Ao.D.DQ.GQ^rDzError generating fit markdown: ) raw_markdownmarkdown_with_citationsreferences_markdown fit_markdownfit_htmlzError in markdown generation: r*)r updater update_params isinstancer3handle Exceptionreplacer^rfilter_contentrNr )r#r%r&r'r rr(r+hdefault_optionsrlermrnro filtered_html error_msgs rr,z*DefaultMarkdownGenerator.generate_markdownvsA8O 1A#( %!&!%%)!$ O!&&'89&&w/&&t||4 AOO .o . !  c2"<0  N xx 5 (// 5AL,8 #') RCGCbCb$hD@+-@+-L+-M!4!4'%3%Jt7J7JN$2$A$A,$OM$(II-_Q^-_$_M#$88M#:L ,)/R(?(E2$7$=2)/R&,"  = N!DSVHM  N!R.:+,HQ*Q'R!'%DSVH#ML$&M' 8QAI+&(1$&   sBG E,G E?G -A F%7#G  E<$E72G 7E<<G ? F"FG F""G % G.G>G GG G<G71G<7G<r-)r.)r.NNNT)r/r0r1r2rr rr3rr$rr^r4r r, __classcell__)r9s@rr6r6(s 2x0E'F2X`aefiknfnaoXp283383#83uUXZ]U]83x*,GK=AKO+/ k'*k#&k-5T#s(^,Dk#+4S>": k *22G)H k %) k'?krr6)abcrrtypingrrrrmodelsr html2textr content_filter_strategyr r re urllib.parsercompilerIr3rrr6r*rrrse#--,&M rzzHI  s     "y9yr