gzddlmZmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZmZGdd ZGd d Zy ) )MIN_WORD_THRESHOLD$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDSCREENSHOT_HEIGHT_TRESHOLD PAGE_TIMEOUTIMAGE_SCORE_THRESHOLDSOCIAL_MEDIA_DOMAINS)UserAgentGenerator)ExtractionStrategy)ChunkingStrategy)MarkdownGenerationStrategy)UnionListc5eZdZdZ d!dedededededed ed ed ed ed edededededededededededededededef4dZ e deddfd Z y)" BrowserConfiga  Configuration class for setting up a browser instance and its context in AsyncPlaywrightCrawlerStrategy. This class centralizes all parameters that affect browser and context creation. Instead of passing scattered keyword arguments, users can instantiate and modify this configuration object. The crawler code will then reference these settings to initialize the browser in a consistent, documented manner. Attributes: browser_type (str): The type of browser to launch. Supported values: "chromium", "firefox", "webkit". Default: "chromium". headless (bool): Whether to run the browser in headless mode (no visible GUI). Default: True. use_managed_browser (bool): Launch the browser using a managed approach (e.g., via CDP), allowing advanced manipulation. Default: False. debugging_port (int): Port for the browser debugging protocol. Default: 9222. use_persistent_context (bool): Use a persistent browser context (like a persistent profile). Automatically sets use_managed_browser=True. Default: False. user_data_dir (str or None): Path to a user data directory for persistent sessions. If None, a temporary directory may be used. Default: None. chrome_channel (str): The Chrome channel to launch (e.g., "chrome", "msedge"). Only applies if browser_type is "chromium". Default: "chromium". channel (str): The channel to launch (e.g., "chromium", "chrome", "msedge"). Only applies if browser_type is "chromium". Default: "chromium". proxy (str or None): Proxy server URL (e.g., "http://username:password@proxy:port"). If None, no proxy is used. Default: None. proxy_config (dict or None): Detailed proxy configuration, e.g. {"server": "...", "username": "..."}. If None, no additional proxy config. Default: None. viewport_width (int): Default viewport width for pages. Default: 1080. viewport_height (int): Default viewport height for pages. Default: 600. verbose (bool): Enable verbose logging. Default: True. accept_downloads (bool): Whether to allow file downloads. If True, requires a downloads_path. Default: False. downloads_path (str or None): Directory to store downloaded files. If None and accept_downloads is True, a default path will be created. Default: None. storage_state (str or dict or None): Path or object describing storage state (cookies, localStorage). Default: None. ignore_https_errors (bool): Ignore HTTPS certificate errors. Default: True. java_script_enabled (bool): Enable JavaScript execution in pages. Default: True. cookies (list): List of cookies to add to the browser context. Each cookie is a dict with fields like {"name": "...", "value": "...", "url": "..."}. Default: []. headers (dict): Extra HTTP headers to apply to all requests in this context. Default: {}. user_agent (str): Custom User-Agent string to use. Default: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36". user_agent_mode (str or None): Mode for generating the user agent (e.g., "random"). If None, use the provided user_agent as-is. Default: None. user_agent_generator_config (dict or None): Configuration for user agent generation if user_agent_mode is set. Default: None. text_mode (bool): If True, disables images and other rich content for potentially faster load times. Default: False. light_mode (bool): Disables certain background features for performance gains. Default: False. extra_args (list): Additional command-line arguments passed to the browser. Default: []. N browser_typeheadlessuse_managed_browseruse_persistent_context user_data_dirchrome_channelchannelproxy proxy_configviewport_widthviewport_heightaccept_downloadsdownloads_pathignore_https_errorsjava_script_enabledsleep_on_closeverbosecookiesheaders user_agentuser_agent_modeuser_agent_generator_config text_mode light_mode extra_argsdebugging_portct||_||_||_||_||_|xs|jxsd|_|xs|jxsd|_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||ng|_||ni|_||_||_||_||_||_||ng|_||_||_||_t7}|j&dk7r2|j(r&|j8di|j(xsi|_n&|j&dk(r|j9|_n |j;|j$|_|j"j?d|j<|jrd|_yy)Nchromiumrandomz sec-ch-uaT) rrrrrrrrrrrrr storage_staterr r#r$r%r&r'r(r)r*r!r"r+r generategenerate_client_hints browser_hint setdefault)selfrrrrrrrrrrrrrr0rr r!r"r#r$r%r&r'r(r)r*r+user_agenr_generators K/var/www/openai/venv/lib/python3.12/site-packages/crawl4ai/async_configs.py__init__zBrowserConfig.__init__KsB)  #6 &<#*,O0A0AOZA$"3"3Az  (,. 0,*#6 #6 ")"5w2 ")"5w2 $.+F("$(2(>*B, ,13   8 +0P0P;2;;339rDO ! !X -2;;=DO 0FFtW  T->->?  & &'+D $ 'kwargsreturnc tdid|jddd|jddd|jddd|jddd|jdd |jd dd |jd dd |jd d |jd d |jd dd|jddd|jddd|jdd|jdd|jddd|jddd|jdgd|jdid|jddd|jdd|jdd|jddd|jddd|jdgS) Nrr-rTrFrrrrrrr8rXrrr0rr r#r$r%zuMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36r&r'r(r)r*r/)rgetr:s r7 from_kwargszBrowserConfig.from_kwargss NJ? ZZ D1 !' +@% H $*::.F#N  !**_5  "::&6 C  JJy*5 **W%  N3 "::&6= #JJ'8#> $ZZ(:EB "::&67 !**_5 !' +@$ G !' +@$ G! "JJy"-# $JJy"-% &zzX' 0#JJ'891 2)/ 3P(Q3 4jje45 6zz,67 8zz,39  r9)r-TFFNr-r-NNr=r>FNNTTFTNNzMozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.5845.187 Safari/604.1 Edg/117.0.2045.47NNFFNi$) __name__ __module__ __qualname____doc__strbooldictintlistr8 staticmethodrAr/r9r7rrs7v'$)',!(!!""!&"$($($ W $,0 #?L,L,L," L, !% L,  L,L,L,L,L,L,L,L,L, "!L,""#L,$%L,&'L,()L,*+L,,-L,45L,6&*7L,89L,:;L,<=L,>?L,\ D _  r9rcgeZdZdZeddddddddddddddddddddeddddddddd dd ddddddddedeedddddd ddf5d e d e d e de de dededede de de dede dede de de de dede ded e d!ed"ed#ed$e d%eeeefd&e d'e d(e d)ed*e d+e d,e d-e d.e d/e d0e d1ed2e d3e d4e d5e d6e d7ed8e d9e d:ed;e dZed?ed@dfdAZdBZy)CCrawlerRunConfiga Configuration class for controlling how the crawler runs each crawl operation. This includes parameters for content extraction, page manipulation, waiting conditions, caching, and other runtime behaviors. This centralizes parameters that were previously scattered as kwargs to `arun()` and related methods. By using this class, you have a single place to understand and adjust the crawling options. Attributes: # Content Processing Parameters word_count_threshold (int): Minimum word count threshold before processing content. Default: MIN_WORD_THRESHOLD (typically 200). extraction_strategy (ExtractionStrategy or None): Strategy to extract structured data from crawled pages. Default: None (NoExtractionStrategy is used if None). chunking_strategy (ChunkingStrategy): Strategy to chunk content before extraction. Default: RegexChunking(). markdown_generator (MarkdownGenerationStrategy): Strategy for generating markdown. Default: None. content_filter (RelevantContentFilter or None): Optional filter to prune irrelevant content. Default: None. only_text (bool): If True, attempt to extract text-only content where applicable. Default: False. css_selector (str or None): CSS selector to extract a specific portion of the page. Default: None. excluded_tags (list of str or None): List of HTML tags to exclude from processing. Default: None. excluded_selector (str or None): CSS selector to exclude from processing. Default: None. keep_data_attributes (bool): If True, retain `data-*` attributes while removing unwanted attributes. Default: False. remove_forms (bool): If True, remove all `
` elements from the HTML. Default: False. prettiify (bool): If True, apply `fast_format_html` to produce prettified HTML output. Default: False. parser_type (str): Type of parser to use for HTML parsing. Default: "lxml". # Caching Parameters cache_mode (CacheMode or None): Defines how caching is handled. If None, defaults to CacheMode.ENABLED internally. Default: None. session_id (str or None): Optional session ID to persist the browser context and the created page instance. If the ID already exists, the crawler does not create a new page and uses the current page to preserve the state. bypass_cache (bool): Legacy parameter, if True acts like CacheMode.BYPASS. Default: False. disable_cache (bool): Legacy parameter, if True acts like CacheMode.DISABLED. Default: False. no_cache_read (bool): Legacy parameter, if True acts like CacheMode.WRITE_ONLY. Default: False. no_cache_write (bool): Legacy parameter, if True acts like CacheMode.READ_ONLY. Default: False. # Page Navigation and Timing Parameters wait_until (str): The condition to wait for when navigating, e.g. "domcontentloaded". Default: "domcontentloaded". page_timeout (int): Timeout in ms for page operations like navigation. Default: 60000 (60 seconds). wait_for (str or None): A CSS selector or JS condition to wait for before extracting content. Default: None. wait_for_images (bool): If True, wait for images to load before extracting content. Default: False. delay_before_return_html (float): Delay in seconds before retrieving final HTML. Default: 0.1. mean_delay (float): Mean base delay between requests when calling arun_many. Default: 0.1. max_range (float): Max random additional delay range for requests in arun_many. Default: 0.3. semaphore_count (int): Number of concurrent operations allowed. Default: 5. # Page Interaction Parameters js_code (str or list of str or None): JavaScript code/snippets to run on the page. Default: None. js_only (bool): If True, indicates subsequent calls are JS-driven updates, not full page loads. Default: False. ignore_body_visibility (bool): If True, ignore whether the body is visible before proceeding. Default: True. scan_full_page (bool): If True, scroll through the entire page to load all content. Default: False. scroll_delay (float): Delay in seconds between scroll steps if scan_full_page is True. Default: 0.2. process_iframes (bool): If True, attempts to process and inline iframe content. Default: False. remove_overlay_elements (bool): If True, remove overlays/popups before extracting HTML. Default: False. simulate_user (bool): If True, simulate user interactions (mouse moves, clicks) for anti-bot measures. Default: False. override_navigator (bool): If True, overrides navigator properties for more human-like behavior. Default: False. magic (bool): If True, attempts automatic handling of overlays/popups. Default: False. adjust_viewport_to_content (bool): If True, adjust viewport according to the page content dimensions. Default: False. # Media Handling Parameters screenshot (bool): Whether to take a screenshot after crawling. Default: False. screenshot_wait_for (float or None): Additional wait time before taking a screenshot. Default: None. screenshot_height_threshold (int): Threshold for page height to decide screenshot strategy. Default: SCREENSHOT_HEIGHT_TRESHOLD (from config, e.g. 20000). pdf (bool): Whether to generate a PDF of the page. Default: False. image_description_min_word_threshold (int): Minimum words for image description extraction. Default: IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD (e.g., 50). image_score_threshold (int): Minimum score threshold for processing an image. Default: IMAGE_SCORE_THRESHOLD (e.g., 3). exclude_external_images (bool): If True, exclude all external images from processing. Default: False. # Link and Domain Handling Parameters exclude_social_media_domains (list of str): List of domains to exclude for social media links. Default: SOCIAL_MEDIA_DOMAINS (from config). exclude_external_links (bool): If True, exclude all external links from the results. Default: False. exclude_social_media_links (bool): If True, exclude links pointing to social media domains. Default: False. exclude_domains (list of str): List of specific domains to exclude from results. Default: []. # Debugging and Logging Parameters verbose (bool): Enable verbose logging. Default: True. log_console (bool): If True, log console messages from the page. Default: False. NFlxmldomcontentloaded皙?333333?T皙?word_count_thresholdextraction_strategychunking_strategymarkdown_generator only_text css_selector excluded_tagsexcluded_selectorkeep_data_attributes remove_forms prettiify parser_typefetch_ssl_certificate session_id bypass_cache disable_cache no_cache_readno_cache_write wait_until page_timeoutwait_forwait_for_imagesdelay_before_return_html mean_delay max_rangesemaphore_countjs_codejs_onlyignore_body_visibilityscan_full_page scroll_delayprocess_iframesremove_overlay_elements simulate_useroverride_navigatormagicadjust_viewport_to_content screenshotscreenshot_wait_forscreenshot_height_thresholdpdf$image_description_min_word_thresholdimage_score_thresholdexclude_external_imagesexclude_social_media_domainsexclude_external_linksexclude_social_media_linksexclude_domainsr" log_consoleurlc6|5|_||_||_||_||_||_||_||_|xsg|_| xsd|_ | |_ | |_ | |_ | |_ ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_ |!|_!|"|_"|#|_#|$|_$|%|_%|&|_&|'|_'|(|_(|)|_)|*|_*|+|_+|,|_,|-|_-|.|_.|/xst^|_0|0|_1|1|_2|2xsg|_3|3|_4|4|_5|j%tm|jtns tqd|j%tm|jtrs tqd|jddlm:}6|6|_yy)Nz=extraction_strategy must be an instance of ExtractionStrategyz9chunking_strategy must be an instance of ChunkingStrategyr) RegexChunking);rrTrUrVrWcontent_filterrXrYrZr[r\r]r^r_r` cache_moderarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrr"r isinstancer ValueErrorr r)7r5rTrUrVrWrrXrYrZr[r\r]r^r_r`rrarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrr"rrrs7 r7r8zCrawlerRunConfig.__init__<s@P%9!#6 !2"4,"(*0b!2!8b$8!("&&;"%$(**,%(  .(@%$".  &<#,(.'>$*"4 *D'%#6 +F(4X1%:"'>$-I,`L`)&<#*D'.4" &  # # /  $ $&89 \] ]  ! ! -j  " "$47 XY Y  ! ! ) 8%2_D " *r9r:r;c tdAid|jddd|jdd|jdd|jdd|jdd|jddd |jd d |jd gd |jd d d |jd dd|jddd|jddd|jddd|jddd|jdd|jdd|jddd|jddd|jddd|jddd|jddd|jddd|jdd|jddd|jdd d!|jd!d d"|jd"d#d$|jd$d%d&|jd&d'|jd'dd(|jd(d)d*|jd*dd+|jd+d,d-|jd-dd.|jd.dd/|jd/dd0|jd0dd1|jd1dd2|jd2dd3|jd3dd4|jd4d5|jd5td6|jd6dd7|jd7td8|jd8td9|jd9dd:|jd:t d;|jd;dd<|jd|jd>d)d?|jd?dd@|jd@S)BNrTrUrVrWrrXFrYrZr[rr\r]r^r_rNr`rrarbrcrdrerfrOrgi`rhrirjrPrkrlrQrmrRrnrorpTrqrrrSrsrtrurvrwrxryrzr{r|r}r~rrrrrr"rrr/)rMr?rrrrr@s r7rAzCrawlerRunConfig.from_kwargssF !',BC!HF !' +@ AF %jj)<= F &zz*>? F "::&67 F jje4F  N3F !**_b9F %jj)3F 8zz,0BC9F : NE:;F <ZZ +=F >#JJ'8%@?F @&,ZZ0JC%PAF Bzz,4CF Djjc2EF F#JJ'8!SF T NC8UF V#JJ'8%@WF X%+JJ/H%$PYF Z!**_e<[F \ &zz*>F]F ^**We,_F `(.zz2NPU'VaF fzz,6gF h!' +@ AiF j)/ 3PRl(mkF l 5%(mF n28d!|j@d"|jBid#|jDd$|jFd%|jHd&|jJd'|jLd(|jNd)|jPd*|jRd+|jTd,|jVd-|jXd.|jZd/|j\d0|j^d1|j`d2|jbd3|jd|jf|jhd4S)5NrTrUrVrWrrXrYrZr[r\r]r^r_r`rrarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrr")rr)5rTrUrVrWrrXrYrZr[r\r]r^r_r`rrarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrr"rr)r5s r7to_dictzCrawlerRunConfig.to_dict$s6 "D$=$=6 !4#;#;6  !7!76  !$"9"9 6 d11 6  6  D--6  T//6  !7!76  #D$=$=6  D--6  6  4++6  $T%?%?6  $//6 $//!6 " D--#6 $ T//%6 & T//'6 ( d11)6 * $//+6 , D---6 .  /6 0 t3316 2 '(E(E36 4 $//56 6 76 8 t3396 : t||;6 < t||=6 > %d&A&A?6 @ d11A6 B D--C6 D t33E6 F &t'C'CG6 H T//I6 J !$"9"9K6 L TZZM6 N )$*I*IO6 P $//Q6 R "4#;#;S6 T *4+K+KU6 V 488W6 X 3D4]4]Y6 Z $T%?%?[6 \ &t'C'C]6 ^ +D,M,M_6 ` %d&A&Aa6 b )$*I*Ic6 d t33e6 f t||g6 h ++88k6 6 r9)rBrCrDrErrrrrrIr r r rGrFrJfloatrrr8rKrHrArr/r9r7rMrMsg~F%726.29= "!%%*"!',"##$-( %*- *.'+$! %(-##(+0!%)+E4X%:(-.2',+0 $!M[5"[50 [5 , [5 7 [5[5[5[5[5#[5[5[5[5$ $%[5,-[5./[501[523[545[5:;[5<=[5>?[5@A[5B#(C[5DE[5FG[5HI[5NsDI~&O[5PQ[5R!%S[5TU[5VW[5XY[5Z"&[[5\][5^!_[5`a[5b%)c[5hi[5j#k[5l&)m[5no[5p/2q[5r #s[5t"&u[5z'+{[5|!%}[5~%)[5@A[5FG[5HI[5LM[5zG DG %7G G T7 r9rMN)configrrrrrruser_agent_generatorr rUr rVr markdown_generation_strategyr typingrrrrMr/r9r7rs953/Dg g T` ` r9