Home / unclecode / crawl4ai
AI Slop Code RIsk Monitor

unclecode/crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

33.5 High pattern signal View on GitHub

Maintenance and Governance Review

This report summarizes scanner evidence for unclecode/crawl4ai, a Python project with 85,146 GitHub stars. SynthScan examined 318,498 lines of code across 833 source files, recording 6050 pattern matches distributed across 25 syntactic categories. The legacy adjusted pattern signal is 33.5 (High pattern signal); it is not an authorship estimate.

Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.

33.5
Adjusted Signal
33.5
Raw Pattern Score
100%
Time Factor
2026-10-05
Last Push
85.1K
Stars
Python
Language
318.5K
Lines of Code
833
Files
6.0K
Pattern Hits
2026-10-12
Scan Date
0.64
HC Hit Rate

Risk Review · v3

Use each vector to guide manual verification of the underlying findings.

Cognitive Ownership Risk 41.4/100 · MEDIUM

Can maintainers explain intent, assumptions, and trade-offs? 2909 finding(s).

Hallucinated Dependency & Supply Chain Risk 15.8/100 · LOW

Are imports declared and package/API names verified? 765 finding(s).

Agent Over-Engineering Ratio 71.5/100 · HIGH

Are wrappers, guards, and repeated structures necessary? 4246 finding(s).

PR Review Velocity Impact 82.7/100 · CRITICAL

Which changes need focused human review? 6050 finding(s).

What These Metrics Mean

Adjusted Signal
Legacy pattern score normalized per 1,000 scanned lines and multiplied by the repository-level time factor. It is not an estimate of AI authorship.
Raw Pattern Score
Sum of severity- and context-weighted pattern findings before applying the time factor.
Time Factor
A coarse multiplier based on repository last-push metadata. It does not identify when individual code was written.
Pattern Hits
Count of configured pattern findings. Review each location and surrounding code before acting.
HC Hit Rate
High- and critical-severity findings per scanned file. Severity is a configured detector weight, not proof of a vulnerability or authorship.
Lines / Files
Scan coverage denominators used by the legacy normalized score.

Pattern-Signal History

This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.

Severity Breakdown

Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.

CRITICAL 1HIGH 536MEDIUM 1219LOW 4294

Directory Score Breakdown

This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.

Pattern Findings

The scanner identified 6050 distinct pattern matches across 25 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.

Decorative Section Separators879 hits · 2966 pts
SeverityFileLineSnippetContext
MEDIUMcrawl4ai/antibot_detector.py22# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py25# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py69# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py73# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py100# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py103# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py114# ---------------------------------------------------------------------------COMMENT
MEDIUMcrawl4ai/antibot_detector.py116# ---------------------------------------------------------------------------COMMENT
MEDIUM⚡crawl4ai/domain_mapper.py502 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUM⚡crawl4ai/domain_mapper.py504 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUM⚡crawl4ai/domain_mapper.py945 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUM⚡crawl4ai/domain_mapper.py947 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py59# ──────────────────────────────────────────────────────────────── constantsCOMMENT
MEDIUMcrawl4ai/domain_mapper.py109# ──────────────────────────────────────────────────────────────── dataclassCOMMENT
MEDIUMcrawl4ai/domain_mapper.py120# ──────────────────────────────────────────────────────────────── classCOMMENT
MEDIUMcrawl4ai/domain_mapper.py167 # ──────────────────────── lifecycleCOMMENT
MEDIUMcrawl4ai/domain_mapper.py183 # ──────────────────────── loggingCOMMENT
MEDIUMcrawl4ai/domain_mapper.py191 # ──────────────────────── seeder compositionCOMMENT
MEDIUMcrawl4ai/domain_mapper.py202 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py204 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py249 # ── Phase 1: Host Discovery ──────────────────────────────────────COMMENT
MEDIUMcrawl4ai/domain_mapper.py259 # ── Phase 2: Per-Host Scanning ───────────────────────────────────COMMENT
MEDIUMcrawl4ai/domain_mapper.py276 # ── Phase 3: Post-Processing ─────────────────────────────────────COMMENT
MEDIUMcrawl4ai/domain_mapper.py301 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py303 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py620 # ──────────────────────── Soft-404 DetectionCOMMENT
MEDIUMcrawl4ai/domain_mapper.py669 # ──────────────────────── robots.txtCOMMENT
MEDIUMcrawl4ai/domain_mapper.py711 # ──────────────────────── SitemapsCOMMENT
MEDIUMcrawl4ai/domain_mapper.py753 # ──────────────────────── Path ProbingCOMMENT
MEDIUMcrawl4ai/domain_mapper.py816 # ──────────────────────── Feed DiscoveryCOMMENT
MEDIUMcrawl4ai/domain_mapper.py900 # ──────────────────────── Homepage Link ExtractionCOMMENT
MEDIUMcrawl4ai/domain_mapper.py1087 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/domain_mapper.py1089 # ════════════════════════════════════════════════════════════════════════COMMENT
MEDIUMcrawl4ai/utils.py3209 # ── build signature ───────────────────────────────────────────COMMENT
MEDIUMcrawl4ai/utils.py3215 # ── first seen? keep – else drop ─────────────COMMENT
MEDIUM⚡crawl4ai/browser_profiler.py554 # 1. ── Start the browser ─────────────────────────────────────────COMMENT
MEDIUM⚡crawl4ai/browser_profiler.py557 # 2. ── Attach Playwright to that running Chrome ──────────────────COMMENT
MEDIUM⚡crawl4ai/browser_profiler.py586 # 3. ── Persist storage state *before* we kill Chrome ─────────────COMMENT
MEDIUM⚡crawl4ai/browser_profiler.py594 # 4. ── Close everything cleanly ──────────────────────────────────COMMENT
MEDIUMcrawl4ai/async_crawler_strategy.py780 # ──────────────────────────────────────────────────────────────STRING
MEDIUMcrawl4ai/async_crawler_strategy.py785 # ──────────────────────────────────────────────────────────────STRING
MEDIUMcrawl4ai/async_configs.py159# ───────────────────────── untrusted-input trust boundary ─────────────────────────COMMENT
MEDIUM⚡crawl4ai/async_url_seeder.py367 # ─────────────────────────────── discovery entryCOMMENT
MEDIUM⚡crawl4ai/async_url_seeder.py833 # ─────────────────────────────── CCCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py64# ────────────────────────────────────────────────────────────────────────── constsCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py79# ────────────────────────────────────────────────────────────────────────── helpersCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py259# ────────────────────────────────────────────────────────────────────────── classCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py327 # ───────── cache dirs ─────────COMMENT
MEDIUMcrawl4ai/async_url_seeder.py343 # ───────── cache helpers ─────────COMMENT
MEDIUMcrawl4ai/async_url_seeder.py889 # ─────────────────────────────── SitemapsCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py1285 # ─────────────────────────────── validate helpersCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py1470 # ─────────────────────────────── BM25 scoring helpersCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py1754 # ─────────────────────────────── cleanup methodsCOMMENT
MEDIUMcrawl4ai/async_url_seeder.py1770 # ─────────────────────────────── index helperCOMMENT
MEDIUMcrawl4ai/browser_manager.py549 # ── 1. cookies ────────────────────────────────────────────────────────────COMMENT
MEDIUMcrawl4ai/browser_manager.py554 # ── 2. localStorage / sessionStorage ──────────────────────────────────────COMMENT
MEDIUMcrawl4ai/browser_manager.py567 # ── 3. runtime-mutable extras from configs ────────────────────────────────COMMENT
MEDIUMcrawl4ai/browser_manager.py840 # ── Persistent context via Playwright's native API ──────────────COMMENT
MEDIUMcrawl4ai/legacy/llmtxt.py308 # -----------------------------------------------------STRING
MEDIUM⚡deploy/docker/server.py80# ── internal imports (after sys.path append) ─────────────────COMMENT
819 more matches not shown…
Hyper-Verbose Identifiers1840 hits · 1801 pts
SeverityFileLineSnippetContext
LOWPROGRESSIVE_CRAWLING.md174def generate_synthetic_dataset(domain_url):CODE
LOW⚡test_webhook_implementation.py43def test_webhook_service_init():CODE
LOWtest_webhook_implementation.py93def test_webhook_config_model():CODE
LOWtest_webhook_implementation.py145def test_payload_construction():CODE
LOWtest_llm_webhook_feature.py15def test_llm_job_payload_model():CODE
LOWtest_llm_webhook_feature.py65def test_handle_llm_request_signature():CODE
LOWtest_llm_webhook_feature.py101def test_process_llm_extraction_signature():CODE
LOWtest_llm_webhook_feature.py136def test_webhook_integration_in_api():CODE
LOWtest_llm_webhook_feature.py187def test_job_endpoint_integration():CODE
LOWtest_llm_webhook_feature.py239def test_create_new_task_integration():CODE
LOWcrawl4ai/antibot_detector.py138def _structural_integrity_check(html: str) -> Tuple[bool, str]:CODE
LOWcrawl4ai/adaptive_crawler.py259 def _embedding_llm_config_dict(self) -> Optional[Dict]:CODE
LOWcrawl4ai/adaptive_crawler.py632 def _get_embedding_llm_config_dict(self) -> Optional[Dict]:CODE
LOWcrawl4ai/adaptive_crawler.py642 def _get_query_llm_config_dict(self) -> Optional[Dict]:CODE
LOWcrawl4ai/adaptive_crawler.py708 def _get_cached_distance_matrix(self, query_embeddings: Any, kb_embeddings: Any) -> Any:CODE
LOWcrawl4ai/adaptive_crawler.py871 async def select_links_for_expansion(CODE
LOWcrawl4ai/adaptive_crawler.py1808 def _crawl_result_to_export_dict(self, result) -> Dict[str, Any]:CODE
LOWcrawl4ai/adaptive_crawler.py1880 def _import_dict_to_crawl_result(self, data: Dict[str, Any]):CODE
LOW⚡crawl4ai/extraction_strategy.py2188 def _make_context_sensitive_xpath(self, xpath, element):CODE
LOW⚡crawl4ai/extraction_strategy.py2240 def _fallback_class_id_search(self, element, selector_str):CODE
LOWcrawl4ai/extraction_strategy.py279 def filter_documents_embeddings(CODE
LOWcrawl4ai/extraction_strategy.py415 def filter_clusters_by_word_count(CODE
LOWcrawl4ai/extraction_strategy.py2109 def _create_selector_function(self, selector_str):CODE
LOWcrawl4ai/extraction_strategy.py2211 def _handle_nth_child_selector(self, element, selector_str):CODE
LOWcrawl4ai/browser_adapter.py43 async def retrieve_console_messages(self, page: Page) -> List[Dict]:CODE
LOWcrawl4ai/browser_adapter.py133 async def retrieve_console_messages(self, page: Page) -> List[Dict]:CODE
LOWcrawl4ai/browser_adapter.py159 def _check_stealth_availability(self) -> bool:CODE
LOWcrawl4ai/browser_adapter.py253 async def retrieve_console_messages(self, page: Page) -> List[Dict]:CODE
LOWcrawl4ai/browser_adapter.py372 async def retrieve_console_messages(self, page: UndetectedPage) -> List[Dict]:STRING
LOWcrawl4ai/__init__.py213# def is_sync_version_installed():COMMENT
LOWcrawl4ai/markdown_generation_strategy.py82 def convert_links_to_citations(CODE
LOWcrawl4ai/content_scraping_strategy.py380 def find_closest_parent_with_useful_text(CODE
LOWcrawl4ai/content_scraping_strategy.py517 def remove_empty_elements_fast(self, root, word_count_threshold=5):CODE
LOWcrawl4ai/content_scraping_strategy.py577 def remove_unwanted_attributes_fast(CODE
LOWcrawl4ai/user_agent_generator.py344 def generate_with_client_hints(self, **kwargs) -> Tuple[str, str]:CODE
LOW⚡crawl4ai/cli.py468def delete_profile_interactive(profiler: BrowserProfiler):CODE
LOWcrawl4ai/cli.py445async def create_profile_interactive(profiler: BrowserProfiler):CODE
LOW⚡crawl4ai/utils.py1186def get_content_of_website_optimized(CODE
LOW⚡crawl4ai/utils.py1238 def find_closest_parent_with_useful_text(tag):CODE
LOW⚡crawl4ai/utils.py3272def start_colab_display_server():CODE
LOWcrawl4ai/utils.py577def calculate_semaphore_count():CODE
LOWcrawl4ai/utils.py750def split_and_parse_json_objects(json_string):CODE
LOWcrawl4ai/utils.py1024 def replace_pre_tags_with_text(node):CODE
LOWcrawl4ai/utils.py1063 def remove_empty_and_low_word_count_elements(node, word_count_threshold):CODE
LOWcrawl4ai/utils.py1285 def score_image_for_usefulness(img, base_url, index, images_count):CODE
LOWcrawl4ai/utils.py1540def extract_metadata_using_lxml(html, doc=None):CODE
LOWcrawl4ai/utils.py1785def perform_completion_with_backoff(CODE
LOWcrawl4ai/utils.py1877async def aperform_completion_with_backoff(CODE
LOWcrawl4ai/utils.py2083def merge_chunks_based_on_token_threshold(chunks, token_threshold):CODE
LOWcrawl4ai/utils.py2360def normalize_url_for_deep_crawl(href, base_url, preserve_https=False, original_scheme=None):CODE
LOWcrawl4ai/utils.py2419def efficient_normalize_url_for_deep_crawl(href, base_url, preserve_https=False, original_scheme=None):CODE
LOWcrawl4ai/utils.py2971def configure_windows_event_loop():CODE
LOWcrawl4ai/utils.py3127def preprocess_html_for_schema(html_content, text_threshold=100, attr_value_threshold=200, max_size=100000):CODE
LOWcrawl4ai/utils.py3387def calculate_link_intrinsic_score(STRING
LOWcrawl4ai/utils.py3663def get_true_available_memory_gb() -> float:STRING
LOWcrawl4ai/utils.py3705def get_true_memory_usage_percent() -> float:STRING
LOWcrawl4ai/browser_profiler.py960 async def launch_standalone_browser(self,CODE
LOWcrawl4ai/browser_profiler.py1355 async def get_builtin_browser_status(self) -> Dict[str, Any]:CODE
LOW⚡crawl4ai/async_crawler_strategy.py679 async def handle_request_failed_capture(request):STRING
LOW⚡crawl4ai/async_crawler_strategy.py1696 async def _capture_console_messages(STRING
1780 more matches not shown…
Cross-File Repetition231 hits · 1155 pts
SeverityFileLineSnippetContext
HIGHCHANGELOG.md0const downloadlink = document.queryselector('a[href$=".exe"]'); if (downloadlink) { downloadlink.click(); }STRING
HIGHdeploy/docker/c4ai-doc-context.md0const downloadlink = document.queryselector('a[href$=".exe"]'); if (downloadlink) { downloadlink.click(); }STRING
HIGHdocs/md_v2/advanced/file-downloading.md0const downloadlink = document.queryselector('a[href$=".exe"]'); if (downloadlink) { downloadlink.click(); }STRING
HIGHcrawl4ai/proxy_strategy.py0configuration class for a single proxy. args: server: proxy server url (e.g., "http://127.0.0.1:8080") username: optionaSTRING
HIGHcrawl4ai/async_configs.py0configuration class for a single proxy. args: server: proxy server url (e.g., "http://127.0.0.1:8080") username: optionaSTRING
HIGHdeploy/docker/c4ai-code-context.md0configuration class for a single proxy. args: server: proxy server url (e.g., "http://127.0.0.1:8080") username: optionaSTRING
HIGHcrawl4ai/proxy_strategy.py0load proxies from environment variable. args: env_var: name of environment variable containing comma-separated proxy strSTRING
HIGHcrawl4ai/async_configs.py0load proxies from environment variable. args: env_var: name of environment variable containing comma-separated proxy strSTRING
HIGHdeploy/docker/c4ai-code-context.md0load proxies from environment variable. args: env_var: name of environment variable containing comma-separated proxy strSTRING
HIGHcrawl4ai/proxy_strategy.py0create a copy of this configuration with updated values. args: **kwargs: key-value pairs of configuration options to updSTRING
HIGHcrawl4ai/async_configs.py0create a copy of this configuration with updated values. args: **kwargs: key-value pairs of configuration options to updSTRING
HIGHdeploy/docker/c4ai-code-context.md0create a copy of this configuration with updated values. args: **kwargs: key-value pairs of configuration options to updSTRING
HIGHcrawl4ai/async_crawler_strategy.py0const _origattachshadow = element.prototype.attachshadow; element.prototype.attachshadow = function(init) { return _origSTRING
HIGHcrawl4ai/browser_manager.py0const _origattachshadow = element.prototype.attachshadow; element.prototype.attachshadow = function(init) { return _origSTRING
HIGHtests/browser/test_init_script_dedup.py0const _origattachshadow = element.prototype.attachshadow; element.prototype.attachshadow = function(init) { return _origSTRING
HIGHcrawl4ai/async_configs.py0recursively convert an object to a serializable dictionary using {type, params} structure for complex objects.STRING
HIGHdeploy/docker/c4ai-code-context.md0recursively convert an object to a serializable dictionary using {type, params} structure for complex objects.STRING
HIGHtests/docker/test_serialization.py0recursively convert an object to a serializable dictionary using {type, params} structure for complex objects.STRING
HIGHcrawl4ai/deep_crawling/bfs_strategy.py0batch (non-streaming) mode: processes one bfs level at a time, then yields all the results.STRING
HIGHcrawl4ai/deep_crawling/base_strategy.py0batch (non-streaming) mode: processes one bfs level at a time, then yields all the results.STRING
HIGHdeploy/docker/c4ai-code-context.md0batch (non-streaming) mode: processes one bfs level at a time, then yields all the results.STRING
HIGHcrawl4ai/deep_crawling/bfs_strategy.py0streaming mode: processes one bfs level at a time and yields results immediately as they arrive.STRING
HIGHcrawl4ai/deep_crawling/base_strategy.py0streaming mode: processes one bfs level at a time and yields results immediately as they arrive.STRING
HIGHdeploy/docker/c4ai-code-context.md0streaming mode: processes one bfs level at a time and yields results immediately as they arrive.STRING
HIGHdeploy/docker/c4ai-code-context.md0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdeploy/docker/c4ai-doc-context.md0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdocs/md_v2/complete-sdk-reference.md0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdocs/md_v2/core/quickstart.md0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdocs/examples/quickstart_examples_set_2.py0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdocs/examples/llm_extraction_openai_pricing.py0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdocs/examples/quickstart.py0from the crawled content, extract all mentioned model names along with their fees for input and output tokens. do not miSTRING
HIGHdeploy/docker/c4ai-code-context.md0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdeploy/docker/c4ai-doc-context.md0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdocs/md_v2/complete-sdk-reference.md0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdocs/md_v2/core/quickstart.md0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdocs/examples/quickstart_examples_set_2.py0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdocs/examples/quickstart.py0(async () => { const tabs = document.queryselectorall("section.charge-methodology .tabs-menu-3 > div"); for(let tab of tSTRING
HIGHdeploy/docker/c4ai-code-context.md0const button = document.queryselector('a[data-testid="pagination-next-button"]'); if (button) button.click();STRING
HIGHtests/async/test_edge_cases.py0const button = document.queryselector('a[data-testid="pagination-next-button"]'); if (button) button.click();STRING
HIGHdocs/examples/quickstart_examples_set_2.py0const button = document.queryselector('a[data-testid="pagination-next-button"]'); if (button) button.click();STRING
HIGHdocs/examples/quickstart.py0const button = document.queryselector('a[data-testid="pagination-next-button"]'); if (button) button.click();STRING
HIGHdeploy/docker/c4ai-code-context.md0(async () => { const getcurrentcommit = () => { const commits = document.queryselectorall('li.box-sc-g0xbh4-0 h4'); retuSTRING
HIGHdocs/examples/quickstart_examples_set_2.py0(async () => { const getcurrentcommit = () => { const commits = document.queryselectorall('li.box-sc-g0xbh4-0 h4'); retuSTRING
HIGHdocs/examples/quickstart.py0(async () => { const getcurrentcommit = () => { const commits = document.queryselectorall('li.box-sc-g0xbh4-0 h4'); retuSTRING
HIGHdeploy/docker/c4ai-code-context.md0part 6: wrap-up and key takeaways summarize the key concepts learned in this tutorial.STRING
HIGHdocs/examples/deepcrawl_example.py0part 6: wrap-up and key takeaways summarize the key concepts learned in this tutorial.STRING
HIGHdocs/examples/docker_config_obj.py0part 6: wrap-up and key takeaways summarize the key concepts learned in this tutorial.STRING
HIGHdeploy/docker/README.md0test the /crawl/stream endpoint with multiple urls.STRING
HIGHdeploy/docker/c4ai-doc-context.md0test the /crawl/stream endpoint with multiple urls.STRING
HIGHtests/docker/test_server_token.py0test the /crawl/stream endpoint with multiple urls.STRING
HIGHdocs/md_v2/core/self-hosting.md0test the /crawl/stream endpoint with multiple urls.STRING
HIGHdocs/examples/docker_python_rest_api.py0test the /crawl/stream endpoint with multiple urls.STRING
HIGHdeploy/docker/c4ai-doc-context.md0focus on extracting the core educational content. include: - key concepts and explanations - important code examples - eSTRING
HIGHdocs/md_v2/complete-sdk-reference.md0focus on extracting the core educational content. include: - key concepts and explanations - important code examples - eSTRING
HIGHdocs/md_v2/core/browser-crawler-config.md0focus on extracting the core educational content. include: - key concepts and explanations - important code examples - eSTRING
HIGHdocs/md_v2/core/markdown-generation.md0focus on extracting the core educational content. include: - key concepts and explanations - important code examples - eSTRING
HIGHdeploy/docker/c4ai-doc-context.md0extract the main educational content while preserving its original wording and substance completely. 1. maintain the exaSTRING
HIGHdocs/md_v2/complete-sdk-reference.md0extract the main educational content while preserving its original wording and substance completely. 1. maintain the exaSTRING
HIGHdocs/md_v2/core/markdown-generation.md0extract the main educational content while preserving its original wording and substance completely. 1. maintain the exaSTRING
HIGHdeploy/docker/c4ai-doc-context.md0focus on extracting specific types of content: - technical documentation - code examples - api references reformat the cSTRING
171 more matches not shown…
Excessive Try-Catch Wrapping1045 hits · 1043 pts
SeverityFileLineSnippetContext
LOWsetup.py40except Exception:CODE
LOW⚡test_webhook_implementation.py29 except Exception as e:CODE
LOW⚡test_webhook_implementation.py37 except Exception as e:CODE
LOWtest_webhook_implementation.py87 except Exception as e:CODE
LOWtest_webhook_implementation.py139 except Exception as e:CODE
LOWtest_webhook_implementation.py200 except Exception as e:CODE
LOWtest_webhook_implementation.py229 except Exception as e:CODE
LOWtest_webhook_implementation.py264 except Exception as e:CODE
LOWtest_llm_webhook_feature.py59 except Exception as e:CODE
LOWtest_llm_webhook_feature.py95 except Exception as e:CODE
LOWtest_llm_webhook_feature.py130 except Exception as e:CODE
LOWtest_llm_webhook_feature.py181 except Exception as e:CODE
LOWtest_llm_webhook_feature.py233 except Exception as e:CODE
LOWtest_llm_webhook_feature.py279 except Exception as e:CODE
LOWtest_llm_webhook_feature.py346 except Exception as e:CODE
LOWcrawl4ai/async_database.py82 except Exception as e:CODE
LOWcrawl4ai/async_database.py110 except Exception as e:CODE
LOWcrawl4ai/async_database.py164 except Exception as e:CODE
LOWcrawl4ai/async_database.py185 except Exception as e:CODE
LOWcrawl4ai/async_database.py217 except Exception as e:CODE
LOWcrawl4ai/async_database.py383 except Exception as e:CODE
LOWcrawl4ai/async_database.py425 except Exception as e:CODE
LOWcrawl4ai/async_database.py470 except Exception as e:CODE
LOWcrawl4ai/async_database.py511 except Exception as e:CODE
LOWcrawl4ai/async_database.py582 except Exception as e:CODE
LOWcrawl4ai/async_database.py600 except Exception as e:CODE
LOWcrawl4ai/async_database.py617 except Exception as e:CODE
LOWcrawl4ai/async_database.py633 except Exception as e:CODE
LOWcrawl4ai/ssl_certificate.py124 except Exception as e:CODE
MEDIUMcrawl4ai/ssl_certificate.py125 print(f"Error fetching/processing certificate for {url}: {e}")CODE
LOWcrawl4ai/ssl_certificate.py183 except Exception as e:CODE
MEDIUMcrawl4ai/ssl_certificate.py184 print(f"Error converting to PEM: {e}")CODE
LOWcrawl4ai/ssl_certificate.py196 except Exception as e:CODE
MEDIUMcrawl4ai/ssl_certificate.py197 print(f"Error converting to DER: {e}")CODE
LOWcrawl4ai/proxy_strategy.py45 except Exception:CODE
LOWcrawl4ai/proxy_strategy.py96 except Exception as e:CODE
MEDIUMcrawl4ai/proxy_strategy.py97 print(f"Error loading proxies from environment: {e}")CODE
MEDIUMcrawl4ai/adaptive_crawler.py1503 print(f"Error crawling {url}: {e}")CODE
MEDIUMcrawl4ai/adaptive_crawler.py1525 print(f"Error in batch crawl: {result}")CODE
LOWcrawl4ai/adaptive_crawler.py825 except Exception:CODE
LOWcrawl4ai/adaptive_crawler.py1502 except Exception as e:CODE
LOWcrawl4ai/cloud_notice.py59 except Exception:CODE
LOWcrawl4ai/cloud_notice.py69 except Exception:CODE
LOWcrawl4ai/cloud_notice.py111 except Exception:CODE
MEDIUMcrawl4ai/cloud_notice.py54def _home() -> Path:CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2077 print(f"Error parsing HTML, falling back to alternative method: {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2175 print(f"Error applying selector '{selector_str}': {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2183 print(f"Error compiling selector '{selector_str}': {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2236 print(f"Error handling nth-child selector: {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2305 print(f"Error serializing HTML: {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2314 print(f"Error getting attribute '{attribute}': {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2400 print(f"Error applying selector '{selector_str}': {e}")CODE
MEDIUM⚡crawl4ai/extraction_strategy.py2406 print(f"Error compiling selector '{selector_str}': {e}")CODE
MEDIUMcrawl4ai/extraction_strategy.py830 print(f"Error in thread execution: {e}")CODE
MEDIUMcrawl4ai/extraction_strategy.py1006 print(f"Error in async extraction: {result}")CODE
MEDIUMcrawl4ai/extraction_strategy.py1175 print(f"Error extracting field {field['name']}: {str(e)}")CODE
MEDIUMcrawl4ai/extraction_strategy.py1307 print(f"Error computing field {field['name']}: {str(e)}")CODE
MEDIUMcrawl4ai/extraction_strategy.py2263 print(f"Error in fallback class/id search: {e}")CODE
MEDIUMcrawl4ai/extraction_strategy.py2292 print(f"Error extracting text: {e}")CODE
MEDIUMcrawl4ai/extraction_strategy.py1148def _extract_field(self, element, field):CODE
985 more matches not shown…
Empty Error Handlers197 hits · 982 pts
SeverityFileLineSnippetContext
HIGHsetup.py40except Exception:CODE
HIGHcrawl4ai/cloud_notice.py111except Exception:CODE
HIGHcrawl4ai/extraction_strategy.py1420except Exception:CODE
HIGHcrawl4ai/extraction_strategy.py1891except (json.JSONDecodeError, TypeError):CODE
HIGHcrawl4ai/browser_adapter.py174except Exception:CODE
HIGHcrawl4ai/browser_adapter.py396except Exception:CODE
HIGHcrawl4ai/browser_adapter.py74except:CODE
HIGHcrawl4ai/browser_adapter.py80except:CODE
HIGHcrawl4ai/browser_adapter.py108except:CODE
HIGHcrawl4ai/browser_adapter.py114except:CODE
HIGHcrawl4ai/browser_adapter.py194except:CODE
HIGHcrawl4ai/browser_adapter.py200except:CODE
HIGHcrawl4ai/browser_adapter.py228except:CODE
HIGHcrawl4ai/browser_adapter.py234except:CODE
HIGHcrawl4ai/domain_mapper.py896except Exception:CODE
HIGHcrawl4ai/domain_mapper.py1131except Exception:CODE
HIGHcrawl4ai/domain_mapper.py810except Exception:CODE
HIGHcrawl4ai/domain_mapper.py587except Exception:CODE
HIGHcrawl4ai/content_scraping_strategy.py375except Exception:CODE
HIGHcrawl4ai/content_scraping_strategy.py789except Exception:CODE
HIGHcrawl4ai/cli.py1656except Exception:CODE
HIGHcrawl4ai/utils.py2492except Exception:CODE
HIGHcrawl4ai/utils.py3380except Exception:CODE
HIGHcrawl4ai/utils.py1513except Exception as _ex:CODE
HIGHcrawl4ai/browser_profiler.py69except (PermissionError, OSError):CODE
HIGHcrawl4ai/browser_profiler.py368except queue.Empty:CODE
HIGHcrawl4ai/browser_profiler.py583except asyncio.CancelledError:CODE
HIGHcrawl4ai/browser_profiler.py1154except asyncio.CancelledError:CODE
HIGHcrawl4ai/browser_profiler.py1098except Exception:CODE
HIGHcrawl4ai/browser_profiler.py1239except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py574except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py1213except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py1221except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py1230except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py1804except Exception:CODE
HIGHcrawl4ai/async_crawler_strategy.py1060except PlaywrightTimeoutError:CODE
HIGHcrawl4ai/table_extraction.py952except:CODE
HIGHcrawl4ai/install.py48except Exception:CODE
HIGHcrawl4ai/install.py80except Exception:CODE
HIGHcrawl4ai/content_filter_strategy.py138except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py92except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py175except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py251except json.JSONDecodeError:CODE
HIGHcrawl4ai/async_url_seeder.py364except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py916except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py226except json.JSONDecodeError:CODE
HIGHcrawl4ai/async_url_seeder.py822except Exception:CODE
HIGHcrawl4ai/async_url_seeder.py937except Exception:CODE
HIGHcrawl4ai/browser_manager.py296except subprocess.TimeoutExpired:CODE
HIGHcrawl4ai/browser_manager.py621except Exception:CODE
HIGHcrawl4ai/browser_manager.py653except Exception:CODE
HIGHcrawl4ai/browser_manager.py657except Exception:CODE
HIGHcrawl4ai/browser_manager.py671except Exception:CODE
HIGHcrawl4ai/browser_manager.py675except Exception:CODE
HIGHcrawl4ai/browser_manager.py807except BaseException:CODE
HIGHcrawl4ai/browser_manager.py1722except Exception:CODE
HIGHcrawl4ai/browser_manager.py1932except Exception:CODE
HIGHcrawl4ai/browser_manager.py1967except Exception:CODE
HIGHcrawl4ai/browser_manager.py2010except Exception:CODE
HIGHcrawl4ai/browser_manager.py2073except Exception:CODE
137 more matches not shown…
Unused Imports613 hits · 492 pts
SeverityFileLineSnippetContext
LOWtest_webhook_implementation.py34CODE
LOWtest_llm_webhook_feature.py23CODE
LOWtest_llm_webhook_feature.py24CODE
LOWcrawl4ai/async_database.py15CODE
LOWcrawl4ai/adaptive_crawler.py13CODE
LOWcrawl4ai/adaptive_crawler.py14CODE
LOWcrawl4ai/adaptive_crawler.py17CODE
LOWcrawl4ai/adaptive_crawler.py878CODE
LOWcrawl4ai/cloud_notice.py21CODE
LOWcrawl4ai/link_preview.py8CODE
LOWcrawl4ai/extraction_strategy.py18CODE
LOWcrawl4ai/extraction_strategy.py29CODE
LOWcrawl4ai/extraction_strategy.py33CODE
LOWcrawl4ai/browser_adapter.py10CODE
LOWcrawl4ai/domain_mapper.py16CODE
LOWcrawl4ai/domain_mapper.py22CODE
LOWcrawl4ai/domain_mapper.py24CODE
LOWcrawl4ai/domain_mapper.py26CODE
LOWcrawl4ai/domain_mapper.py27CODE
LOWcrawl4ai/domain_mapper.py30CODE
LOWcrawl4ai/domain_mapper.py48CODE
LOWcrawl4ai/domain_mapper.py41CODE
LOWcrawl4ai/domain_mapper.py57CODE
LOWcrawl4ai/__init__.py4CODE
LOWcrawl4ai/__init__.py4CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py6CODE
LOWcrawl4ai/__init__.py8CODE
LOWcrawl4ai/__init__.py8CODE
LOWcrawl4ai/__init__.py8CODE
LOWcrawl4ai/__init__.py13CODE
LOWcrawl4ai/__init__.py14CODE
LOWcrawl4ai/__init__.py14CODE
LOWcrawl4ai/__init__.py18CODE
LOWcrawl4ai/__init__.py18CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py22CODE
LOWcrawl4ai/__init__.py31CODE
LOWcrawl4ai/__init__.py31CODE
LOWcrawl4ai/__init__.py32CODE
LOWcrawl4ai/__init__.py33CODE
LOWcrawl4ai/__init__.py33CODE
LOWcrawl4ai/__init__.py33CODE
LOWcrawl4ai/__init__.py33CODE
LOWcrawl4ai/__init__.py39CODE
LOWcrawl4ai/__init__.py39CODE
553 more matches not shown…
Self-Referential Comments152 hits · 457 pts
SeverityFileLineSnippetContext
MEDIUMcrawl4ai/async_database.py677# Create a singleton instanceCOMMENT
MEDIUMcrawl4ai/ssl_certificate.py90 # Create the dictionary directlyCOMMENT
MEDIUMcrawl4ai/adaptive_crawler.py133 # Create a mock object that has the minimal interface we needCOMMENT
MEDIUMcrawl4ai/link_preview.py212 # Create a wrapper to track progressCOMMENT
MEDIUMcrawl4ai/link_preview.py241 # Create a custom progress tracking versionCOMMENT
MEDIUMcrawl4ai/extraction_strategy.py2125 # Create the wrapper function that implements the selection strategyCOMMENT
MEDIUMcrawl4ai/extraction_strategy.py2131 # Create a cache key based on element and selectorCOMMENT
MEDIUMcrawl4ai/extraction_strategy.py2356 # Create a function that will apply this selector appropriatelyCOMMENT
MEDIUMcrawl4ai/content_scraping_strategy.py919 # Create a config object for LinkPreviewCOMMENT
MEDIUM⚡crawl4ai/cli.py457 # Create the profileCOMMENT
MEDIUMcrawl4ai/cli.py362 # Create a profile and use it for crawlingCOMMENT
MEDIUMcrawl4ai/utils.py559 # Create the box with colored borders and lighter textCOMMENT
MEDIUMcrawl4ai/utils.py1023 # Create a function that replace content of all"pre" tag with its inner textCOMMENT
MEDIUMcrawl4ai/utils.py3229 # # Create a signature based on tag and classesCOMMENT
MEDIUMcrawl4ai/browser_profiler.py158 # Create a logger if not providedCOMMENT
MEDIUMcrawl4ai/browser_profiler.py995 # Create a temporary profile directoryCOMMENT
MEDIUMcrawl4ai/browser_profiler.py1200 # Create a user data directory for the builtin browserCOMMENT
MEDIUMcrawl4ai/browser_profiler.py1382 # Create a new profileCOMMENT
MEDIUMcrawl4ai/browser_profiler.py408 # Create a profile interactivelySTRING
MEDIUMcrawl4ai/browser_profiler.py829 # Define a custom crawl functionSTRING
MEDIUMcrawl4ai/async_crawler_strategy.py1673 # Create a new CDP sessionSTRING
MEDIUMcrawl4ai/model_loader.py226 # Create the models directory if it doesn't existCOMMENT
MEDIUMcrawl4ai/async_configs.py1129 # Create a funciton returns dict of the objectCOMMENT
MEDIUMcrawl4ai/async_configs.py2202 # Create a funciton returns dict of the objectCOMMENT
MEDIUMcrawl4ai/async_configs.py2331 # Create a new config with streaming enabledSTRING
MEDIUMcrawl4ai/async_configs.py2334 # Create a new config with multiple updatesSTRING
MEDIUMcrawl4ai/async_url_seeder.py1245 # Create a bounded queue for results to prevent RAM issuesCOMMENT
MEDIUM⚡crawl4ai/browser_manager.py1725 # Create a new page from the chosen contextCOMMENT
MEDIUMcrawl4ai/browser_manager.py484 # Create a BrowserProfiler instance and delegate to itCOMMENT
MEDIUMcrawl4ai/browser_manager.py507 # Create a BrowserProfiler instance and delegate to itCOMMENT
MEDIUMcrawl4ai/browser_manager.py530 # Create a BrowserProfiler instance and delegate to itCOMMENT
MEDIUMcrawl4ai/chunking_strategy.py7# Define the abstract base class for chunking strategiesCOMMENT
MEDIUMcrawl4ai/chunking_strategy.py27# Create an identity chunking strategy f(x) = [x]COMMENT
MEDIUMcrawl4ai/async_webcrawler.py272 # Initialize processing variablesCOMMENT
MEDIUMcrawl4ai/async_webcrawler.py843 # Define the source selection logic using dict dispatchCOMMENT
MEDIUMcrawl4ai/deep_crawling/bfs_strategy.py51 # Create a new logger if logger is None, dict, or any other non-Logger typeCOMMENT
MEDIUMcrawl4ai/deep_crawling/bff_strategy.py62 # Create a new logger if logger is None, dict, or any other non-Logger typeCOMMENT
MEDIUMcrawl4ai/js_snippet/__init__.py4# Create a function get name of a js script, then load from the CURRENT folder of this script and return its content as COMMENT
MEDIUMcrawl4ai/components/crawler_monitor.py167 # Create the status textCOMMENT
MEDIUMcrawl4ai/components/crawler_monitor.py180 # Create a table for status countsCOMMENT
MEDIUMcrawl4ai/components/crawler_monitor.py261 # Create a table for task detailsCOMMENT
MEDIUMcrawl4ai/components/crawler_monitor.py374 # Create a more visible footer panelCOMMENT
MEDIUM⚡tests/test_pyopenssl_security_fix.py75 # Create a basic SSL context to verify functionalityCOMMENT
MEDIUMtests/test_raw_html_redirected_url.py16 # Create a dummy decoratorCOMMENT
MEDIUMtests/test_raw_html_redirected_url.py51 # Create a large HTML (100KB+)COMMENT
MEDIUM⚡tests/test_raw_html_edge_cases.py259 # Create a temp fileCOMMENT
MEDIUMtests/docker/test_config_object.py54 # Create the configCOMMENT
MEDIUMtests/memory/test_dispatcher_stress.py36# Create a memory restrictor to simulate limited memory environmentCOMMENT
MEDIUMtests/memory/test_dispatcher_stress.py330 # Create a nightmare scenario - multiple overlapping spikesCOMMENT
MEDIUMtests/memory/benchmark_report.py230 # Create the plotCOMMENT
MEDIUMtests/memory/benchmark_report.py307 # Create the plotCOMMENT
MEDIUMtests/memory/benchmark_report.py871 # Create the benchmark reporterSTRING
MEDIUMtests/proxy/test_proxy_config.py477 # Create a large list of proxy stringsCOMMENT
MEDIUM⚡tests/general/test_mhtml.py27 # Create a fresh browser config and crawler instance for this testCOMMENT
MEDIUM⚡tests/general/test_mhtml.py32 # Create a fresh crawler instanceCOMMENT
MEDIUM⚡tests/general/test_mhtml.py89 # Create a fresh browser config and crawler instance for this testCOMMENT
MEDIUM⚡tests/general/test_mhtml.py94 # Create a fresh crawler instanceCOMMENT
MEDIUM⚡tests/general/test_mhtml.py129 # Create a fresh browser config and crawler instance for this testCOMMENT
MEDIUM⚡tests/general/test_mhtml.py134 # Create a fresh crawler instanceCOMMENT
MEDIUM⚡tests/general/test_mhtml.py167 # Create a fresh browser config and crawler instance for this testCOMMENT
92 more matches not shown…
Structural Annotation Overuse249 hits · 393 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/cache_validator.py112 # Step 1: Try HEAD request with conditional headersCOMMENT
LOWcrawl4ai/cache_validator.py156 # Step 2: No conditional headers available, try fingerprint onlyCOMMENT
LOWcrawl4ai/cache_validator.py180 # Step 3: No validation data availableCOMMENT
LOWcrawl4ai/utils.py2318 # IMPORTANT: Don't use quote(unquote()) as it mangles + signs in URLsCOMMENT
LOW⚡crawl4ai/async_url_seeder.py919 # Step 1: Find sitemap URL and get lastmod (needed for validation)COMMENT
LOW⚡crawl4ai/async_url_seeder.py943 # Step 2: Check cache validity (skip if force=True)COMMENT
LOW⚡crawl4ai/async_url_seeder.py998 # Step 4: Write to cache (FALLBACK: if write fails, URLs still yielded above)COMMENT
LOWcrawl4ai/async_url_seeder.py957 # Step 3: Fetch fresh URLsCOMMENT
LOW⚡crawl4ai/cloud/cli.py253 # Step 1: Shrink (unless --no-shrink)COMMENT
LOW⚡crawl4ai/cloud/cli.py266 # Step 2: Package as tar.gzCOMMENT
LOWcrawl4ai/cloud/cli.py281 # Step 3: UploadCOMMENT
LOW⚡deploy/docker/c4ai-doc-context.md2985 # Step 1: Create a pruning filterCOMMENT
LOW⚡deploy/docker/c4ai-doc-context.md2995 # Step 2: Insert it into a Markdown GeneratorCOMMENT
LOW⚡deploy/docker/c4ai-doc-context.md2998 # Step 3: Pass it to CrawlerRunConfigCOMMENT
LOWdeploy/docker/c4ai-doc-context.md3857 # Step 1: Crawl the Web URLCOMMENT
LOWdeploy/docker/c4ai-doc-context.md3871 # Step 2: Crawl from the Local HTML FileCOMMENT
LOWdeploy/docker/c4ai-doc-context.md3885 # Step 3: Crawl Using Raw HTML ContentCOMMENT
LOWdeploy/docker/c4ai-doc-context.md4554 # Step 1: Load initial Hacker News pageCOMMENT
LOWdeploy/docker/c4ai-doc-context.md4565 # Step 2: Let's scroll and click the "More" linkCOMMENT
LOWdeploy/docker/c4ai-doc-context.md4659 # Step 1: Load initial commitsCOMMENT
LOWdeploy/docker/c4ai-doc-context.md4674 # Step 2: For subsequent pages, we run JS to click 'Next Page' if it existsCOMMENT
LOW⚡tests/test_webhook_feature.sh104# Step 1: Save current branch and fetch PRCOMMENT
LOW⚡tests/test_webhook_feature.sh112# Step 2: Switch to new branchCOMMENT
LOW⚡tests/test_webhook_feature.sh117# Step 3: Activate virtual environmentCOMMENT
LOW⚡tests/test_webhook_feature.sh128# Step 4: Install server dependenciesCOMMENT
LOWtests/test_webhook_feature.sh147# Step 5: Start Redis in backgroundCOMMENT
LOWtests/test_webhook_feature.sh183# Step 6: Create and run webhook testCOMMENT
LOWtests/test_webhook_feature.sh292# Step 7: Verify resultsCOMMENT
LOWtests/test_webhook_feature.sh303# Step 8: Cleanup happens automatically via trapCOMMENT
LOW⚡tests/test_pyopenssl_update.py141 # Step 1: Check versionsCOMMENT
LOW⚡tests/test_pyopenssl_update.py147 # Step 2: Test basic crawlingCOMMENT
LOW⚡tests/test_pyopenssl_update.py153 # Step 3: Test stealth modeCOMMENT
LOW⚡tests/WEBHOOK_TEST_README.md45#### Step 1: Branch ManagementCOMMENT
LOW⚡tests/WEBHOOK_TEST_README.md50#### Step 2: Environment SetupCOMMENT
LOW⚡tests/WEBHOOK_TEST_README.md55#### Step 3: Service StartupCOMMENT
LOW⚡tests/WEBHOOK_TEST_README.md60#### Step 4: Webhook TestCOMMENT
LOW⚡tests/WEBHOOK_TEST_README.md66#### Step 5: CleanupCOMMENT
LOW⚡tests/proxy/test_proxy_verify.py79 # Step 1: Verify IPsCOMMENT
LOW⚡tests/proxy/test_proxy_verify.py86 # Step 2: Get NST proxiesCOMMENT
LOWtests/proxy/test_proxy_verify.py97 # Step 3: Test Chanel with all available proxiesCOMMENT
LOW⚡tests/general/test_async_url_seeder_bm25.py558 # Step 1: Discover and score URLsCOMMENT
LOW⚡tests/general/test_async_url_seeder_bm25.py587 # Step 3: Verify these URLs would be good for actual crawlingCOMMENT
LOWtests/general/test_async_url_seeder_bm25.py573 # Step 2: Analyze top resultsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py608 # Step 1: open all sessionsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py615 # Step 2: navigate each session to a second pageCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py620 # Step 3: kill sessions one by one, verify others unaffectedCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py936 # Step 1: open sessionCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py943 # Step 2: concurrent non-session crawlsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py952 # Step 3: kill sessionCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py955 # Step 4: trigger recycleCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py962 # Step 5: new session on fresh browserCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py970 # Step 6: verify it worksCOMMENT
LOW⚡tests/async/test_browser_memory.py782 # Step 1: login — sets cookieCOMMENT
LOW⚡tests/async/test_browser_memory.py787 # Step 2: dashboard — cookie should carry over via sessionCOMMENT
LOW⚡tests/browser/test_builtin_browser.py52 # Step 1: Create a BrowserManager with builtin modeCOMMENT
LOW⚡tests/browser/test_builtin_browser.py57 # Step 2: Check if we have a BuiltinBrowserStrategyCOMMENT
LOW⚡tests/browser/test_builtin_browser.py69 # Step 3: Start the manager to launch or connect to builtin browserCOMMENT
LOW⚡tests/browser/test_builtin_browser.py78 # Step 4: Get browser info from the strategyCOMMENT
LOW⚡tests/browser/test_builtin_browser.py122 # Step 2: Get multiple pagesCOMMENT
LOW⚡tests/browser/test_builtin_browser.py149 # Step 1: Get browser statusCOMMENT
189 more matches not shown…
Cross-Language Confusion (JS/TS)42 hits · 255 pts
SeverityFileLineSnippetContext
HIGHdocs/md_v2/ask_ai/ask-ai.js72 print(result.markdown[:300]) # Print first 300 charsCODE
HIGHdocs/md_v2/marketplace/frontend/app-detail.js155 print(result.markdown)`;CODE
HIGH⚡docs/md_v2/marketplace/frontend/app-detail.js172 print(result.status_code)`;CODE
HIGH⚡docs/md_v2/marketplace/frontend/app-detail.js191 print(result.extracted_content)`;CODE
HIGHdocs/md_v2/marketplace/frontend/app-detail.js240 print(f"Found {len(products)} products")CODE
HIGHdocs/md_v2/marketplace/frontend/app-detail.js243 print(f"- {product['title']}: {product['price']}")CODE
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2418 print("✅ Automation completed successfully!")CODE
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2419 print(f"Final URL: {result.url}")CODE
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2423 print("❌ Automation failed:", result.error_message)CODE
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2452print(f"💾 C4A Script saved to: {script_path}")STRING
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2453print("\\n📜 Generated C4A Script:")STRING
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2454print(C4A_SCRIPT)STRING
HIGH⚡…md_v2/apps/crawl4ai-assistant/content/scriptBuilder.js2464print("\\n💡 To execute this C4A script, compile it to JavaScript first!")STRING
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1737 print(f"\\n✅ Successfully extracted {len(data)} items!")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1744 print("\\n📊 Sample results (first 2 items):")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1746 print(f"\\nItem {i}:")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1748 print(f" {key}: {value}")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1752 print("❌ Extraction failed:", result.error_message)CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1753 return NoneCODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1759 print("\\n🎯 Next steps:")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1760 print("1. Install Crawl4AI: pip install crawl4ai")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1761 print("2. Modify the URL or add multiple URLs")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1762 print("3. Customize crawler options as needed")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1763 print("4. Check 'extracted_data.json' for full results")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1817 print("✅ Schema generated successfully!")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1818 print(f"📄 Schema saved to: {schema_path}")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1819 print("\\nGenerated schema:")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1820 print(json.dumps(schema, indent=2))CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1825 print(f"❌ Error generating schema: {e}")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1826 return NoneCODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1830 print("\\n🧪 Testing extraction on live webpage...")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1837 print("❌ Schema file not found. Run generate_schema() first.")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1859 print(f"\\n✅ Successfully extracted {len(data)} items!")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1866 print("\\n📊 Sample results (first 2 items):")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1868 print(f"\\nItem {i}:")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1870 print(f" {key}: {value}")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1872 print("❌ Extraction failed:", result.error_message)CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1882 print("\\n🎯 Next steps:")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1883 print("1. Review the generated schema in 'generated_schema.json'")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1884 print("2. Uncomment the test_extraction() line to test on the live site")CODE
HIGH⚡…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1885 print("3. Use the schema in your Crawl4AI projects!")CODE
HIGH…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js1803 print("🔧 Generating extraction schema...")CODE
Deep Nesting268 hits · 214 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/async_database.py102CODE
LOWcrawl4ai/async_database.py478CODE
LOWcrawl4ai/ssl_certificate.py62CODE
LOWcrawl4ai/adaptive_crawler.py871CODE
LOWcrawl4ai/adaptive_crawler.py1330CODE
LOWcrawl4ai/adaptive_crawler.py1570CODE
LOWcrawl4ai/adaptive_crawler.py1845CODE
LOWcrawl4ai/extraction_strategy.py641CODE
LOWcrawl4ai/extraction_strategy.py786CODE
LOWcrawl4ai/extraction_strategy.py843CODE
LOWcrawl4ai/extraction_strategy.py1088CODE
LOWcrawl4ai/extraction_strategy.py1178CODE
LOWcrawl4ai/extraction_strategy.py1764CODE
LOWcrawl4ai/extraction_strategy.py2109CODE
LOWcrawl4ai/extraction_strategy.py2348CODE
LOWcrawl4ai/extraction_strategy.py2126CODE
LOWcrawl4ai/extraction_strategy.py2357CODE
LOWcrawl4ai/domain_mapper.py305CODE
LOWcrawl4ai/domain_mapper.py361CODE
LOWcrawl4ai/domain_mapper.py471CODE
LOWcrawl4ai/domain_mapper.py506CODE
LOWcrawl4ai/domain_mapper.py671CODE
LOWcrawl4ai/domain_mapper.py713CODE
LOWcrawl4ai/domain_mapper.py853CODE
LOWcrawl4ai/domain_mapper.py902CODE
LOWcrawl4ai/domain_mapper.py477CODE
LOWcrawl4ai/cache_validator.py83CODE
LOWcrawl4ai/markdown_generation_strategy.py148CODE
LOWcrawl4ai/hub.py41CODE
LOWcrawl4ai/content_scraping_strategy.py231CODE
LOWcrawl4ai/content_scraping_strategy.py410CODE
LOWcrawl4ai/content_scraping_strategy.py517CODE
LOWcrawl4ai/content_scraping_strategy.py577CODE
LOWcrawl4ai/content_scraping_strategy.py615CODE
LOWcrawl4ai/user_agent_generator.py261CODE
LOWcrawl4ai/user_agent_generator.py299CODE
LOWcrawl4ai/async_dispatcher.py175CODE
LOWcrawl4ai/async_dispatcher.py228CODE
LOWcrawl4ai/async_dispatcher.py374CODE
LOWcrawl4ai/async_dispatcher.py471CODE
LOWcrawl4ai/async_dispatcher.py530CODE
LOWcrawl4ai/async_dispatcher.py651CODE
LOWcrawl4ai/cli.py111CODE
LOWcrawl4ai/cli.py502CODE
LOWcrawl4ai/cli.py581CODE
LOWcrawl4ai/cli.py1033CODE
LOWcrawl4ai/utils.py84CODE
LOWcrawl4ai/utils.py462CODE
LOWcrawl4ai/utils.py598CODE
LOWcrawl4ai/utils.py750CODE
LOWcrawl4ai/utils.py932CODE
LOWcrawl4ai/utils.py1186CODE
LOWcrawl4ai/utils.py2212CODE
LOWcrawl4ai/utils.py3127CODE
LOWcrawl4ai/utils.py3387CODE
LOWcrawl4ai/utils.py3663CODE
LOWcrawl4ai/utils.py1378CODE
LOWcrawl4ai/browser_profiler.py83CODE
LOWcrawl4ai/browser_profiler.py196CODE
LOWcrawl4ai/browser_profiler.py252CODE
208 more matches not shown…
Cross-Language Confusion41 hits · 210 pts
SeverityFileLineSnippetContext
HIGHcrawl4ai/browser_adapter.py306 window.__capturedConsole.push({CODE
HIGHcrawl4ai/browser_adapter.py340 window.__capturedErrors.push({CODE
HIGHcrawl4ai/browser_adapter.py357 window.__capturedErrors.push({CODE
HIGHcrawl4ai/browser_adapter.py360 stack: event.reason && event.reason.stack ? event.reason.stack : '',CODE
HIGHcrawl4ai/prompts.py1357if (card && card.shadowRoot) {CODE
HIGHcrawl4ai/prompts.py1366 if (card && card.shadowRoot) {CODE
HIGHcrawl4ai/async_crawler_strategy.py2093 # return {{ success: false, error: err.toString(), stack: err.stack }};COMMENT
HIGHcrawl4ai/async_crawler_strategy.py1393 htmlChunks.push(previousHTML);CODE
HIGHcrawl4ai/async_crawler_strategy.py1405 htmlChunks.push(currentHTML);CODE
HIGHcrawl4ai/async_crawler_strategy.py1434 uniqueElements.push(element.outerHTML);CODE
HIGHcrawl4ai/async_crawler_strategy.py1560 error: error.toString(),CODE
HIGHcrawl4ai/async_crawler_strategy.py1604 error: error.toString(),CODE
HIGHcrawl4ai/async_crawler_strategy.py1900 if (rect.width > 0 && rect.height > 0) {CODE
HIGHcrawl4ai/async_crawler_strategy.py2111 return {{ success: false, error: err.toString(), stack: err.stack }};CODE
HIGHcrawl4ai/async_crawler_strategy.py2242 error: error.toString(),CODE
HIGHcrawl4ai/async_crawler_strategy.py2253 error: error.toString(),CODE
HIGHcrawl4ai/async_crawler_strategy.py2374 error: e.toString()CODE
HIGHdeploy/docker/work_queue.py10limit set to 0 (or null) means "unbounded" - i.e. the previous behavior is fullySTRING
HIGHtests/test_cloud_bugs_batch.py96httpbin_anything = '<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{"args": {}, "daCODE
HIGHtests/test_main.py73 "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContenCODE
HIGHtests/test_virtual_scroll.py44 allData.push({CODE
HIGHtests/test_virtual_scroll.py57 items.push(`<div class="item" data-index="${item.id}">${item.text}</div>`);CODE
HIGHtests/test_docker.py91 "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.inCODE
HIGHtests/releases/test_release_0.6.4.py89 function gtag(){dataLayer.push(arguments);}CODE
HIGHtests/releases/test_release_0.6.4.py125 function gtag(){dataLayer.push(arguments);}CODE
HIGHtests/docker/test_hooks_comprehensive.py465 return element ? element.getAttribute('content') : null;CODE
HIGHtests/docker/test_server_requests.py145 # It might be null, missing, or populated depending on the server's default behaviorCOMMENT
HIGHtests/general/test_async_crawler_strategy.py283# results.push(e.name);COMMENT
HIGHtests/general/test_async_crawler_strategy.py288# results.push(e.name);COMMENT
HIGHtests/async/test_parameters_and_options.py52 "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.inCODE
HIGHdocs/examples/quickstart_examples_set_2.py94 js_code="const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContenCODE
HIGHdocs/examples/quickstart_examples_set_2.py361 return commits.length > 0 ? commits[0].textContent.trim() : null;CODE
HIGHdocs/examples/quickstart_examples_set_2.py371 if (newCommit && newCommit !== initialCommit) {CODE
HIGHdocs/examples/stealth_mode_example.py112 console.log('DETECTION_RESULTS:', JSON.stringify(detectionResults, null, 2));CODE
HIGHdocs/examples/rest_call.py32 loadMoreButton && loadMoreButton.click();STRING
HIGHdocs/examples/crawlai_vs_firecrawl.py53 "const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContenCODE
HIGHdocs/examples/docker_client_hooks_example.py234 return el ? el.getAttribute('content') : null;CODE
HIGHdocs/examples/quickstart.py94 js_code="const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContenCODE
HIGHdocs/examples/quickstart.py361 return commits.length > 0 ? commits[0].textContent.trim() : null;CODE
HIGHdocs/examples/quickstart.py371 if (newCommit && newCommit !== initialCommit) {CODE
HIGH…solver/capsolver_api_integration/solve_recaptcha_v3.py47 args[0] = url.toString();CODE
Redundant / Tautological Comments151 hits · 190 pts
SeverityFileLineSnippetContext
LOWtest_webhook_implementation.py240 # Check if api.py can import webhook moduleCOMMENT
LOWcrawl4ai/async_database.py49 # Check if version update is neededCOMMENT
LOWcrawl4ai/ssl_certificate.py74 # Set check_hostname to False and verify_mode to CERT_NONE temporarilyCOMMENT
LOWcrawl4ai/proxy_strategy.py244 # Check if session exists and hasn't expiredCOMMENT
LOWcrawl4ai/adaptive_crawler.py538 # Check if we have any links leftCOMMENT
LOWcrawl4ai/adaptive_crawler.py715 # Check if KB has changedCOMMENT
LOWcrawl4ai/adaptive_crawler.py1159 # Check if confidence is below minimum threshold (completely irrelevant)COMMENT
LOWcrawl4ai/extraction_strategy.py2222 # Check if there's content after the nth-child partCOMMENT
LOWcrawl4ai/content_scraping_strategy.py934 # Check if we're already in an async contextCOMMENT
LOWcrawl4ai/async_dispatcher.py288 # Check if we're in critical memory stateCOMMENT
LOWcrawl4ai/cli.py1133 # Set output to JSON if not explicitly specifiedCOMMENT
LOWcrawl4ai/cli.py1142 # Check if type does not exist show proper messageCOMMENT
LOWcrawl4ai/cli.py1401 # Check if the value should be one of the allowed optionsCOMMENT
LOWcrawl4ai/cli.py1551 # Display resultsCOMMENT
LOW⚡crawl4ai/utils.py1245 # Check if the text content has at least word_count_thresholdCOMMENT
LOW⚡crawl4ai/utils.py1251 # Check if an image has valid display and inside undesired html elementsCOMMENT
LOW⚡crawl4ai/utils.py3277 # Check if running in Google ColabCOMMENT
LOWcrawl4ai/utils.py320 # Check if cache is still fresh based on TTLCOMMENT
LOWcrawl4ai/utils.py327 # Check if content actually changedCOMMENT
LOWcrawl4ai/utils.py699 # Check if a path has already been saved for this browser typeCOMMENT
LOWcrawl4ai/utils.py1086 # Check if the tag contains text and if it's not just whitespaceCOMMENT
LOWcrawl4ai/utils.py1106 # Check if the tag itself is empty or all its children are empty/whitespaceCOMMENT
LOWcrawl4ai/utils.py1847 # Check if we have exhausted our max attemptsCOMMENT
LOWcrawl4ai/utils.py1940 # Check if we have exhausted our max attemptsCOMMENT
LOWcrawl4ai/utils.py2602 # Check if URL domain ends with base domainCOMMENT
LOWcrawl4ai/utils.py3369 # Check if this is a documentation/reference siteSTRING
LOW⚡crawl4ai/browser_profiler.py563 # Check if browser started successfullyCOMMENT
LOW⚡crawl4ai/browser_profiler.py1287 # Check if the browser is still runningCOMMENT
LOW⚡crawl4ai/browser_profiler.py1303 # Check if the process existsCOMMENT
LOWcrawl4ai/browser_profiler.py114 # Check if item matches any keep patternCOMMENT
LOWcrawl4ai/browser_profiler.py239 # Check if browser process endedCOMMENT
LOWcrawl4ai/browser_profiler.py297 # Check if browser process endedCOMMENT
LOWcrawl4ai/browser_profiler.py371 # Check if browser process endedCOMMENT
LOWcrawl4ai/browser_profiler.py658 # Check if this looks like a valid browser profileCOMMENT
LOWcrawl4ai/browser_profiler.py710 # Check if path exists and is a valid profileCOMMENT
LOWcrawl4ai/browser_profiler.py759 # Check if path exists and is a valid profileCOMMENT
LOWcrawl4ai/browser_profiler.py1115 # Check if browser started successfullyCOMMENT
LOWcrawl4ai/browser_profiler.py1194 # Check if there's an existing browser still runningCOMMENT
LOWcrawl4ai/browser_profiler.py1217 # Check if browser started successfullyCOMMENT
LOWcrawl4ai/docker_client.py94 # Check if hooks are already strings or need conversionCOMMENT
LOWcrawl4ai/async_crawler_strategy.py463 # Check if browser processing is required for file:// or raw: URLsSTRING
LOWcrawl4ai/async_crawler_strategy.py727 # Check if this is a file:// or raw: URL that needs set_content() instead of goto()STRING
LOWcrawl4ai/async_crawler_strategy.py1818 # Check if viewport-only screenshot is forcedSTRING
LOWcrawl4ai/model_loader.py195 # Check if the model directory already existsCOMMENT
LOWcrawl4ai/table_extraction.py110 # Check if this is a data table (not a layout table)COMMENT
LOWcrawl4ai/table_extraction.py807 # Check if there are any tables in the contentCOMMENT
LOWcrawl4ai/table_extraction.py816 # Check if chunking is neededCOMMENT
LOWcrawl4ai/table_extraction.py899 # Check if we got valid tablesCOMMENT
LOWcrawl4ai/table_extraction.py1071 # Check if adding this row would exceed thresholdCOMMENT
LOWcrawl4ai/async_configs.py2397 # Check if given provider starts with any of key in PROVIDER_MODELS_PREFIXESCOMMENT
LOWcrawl4ai/content_filter_strategy.py461 # Check if body is presentCOMMENT
LOWcrawl4ai/browser_manager.py1744 # Check if browser recycle threshold is hit — bump version for next requestsCOMMENT
LOWcrawl4ai/browser_manager.py1806 # Check if this signature belongs to an old browser waiting to be cleaned upCOMMENT
LOWcrawl4ai/browser_manager.py1970 # Check if any signatures from this old version remainCOMMENT
LOWcrawl4ai/browser_manager.py1368 # Check if there is value for crawlerRunConfig.proxy_config set add that to contextSTRING
LOWcrawl4ai/async_webcrawler.py512 # Check if blocked (skip for raw: URLs —COMMENT
LOWcrawl4ai/deep_crawling/bfs_strategy.py239 # Check if we've already reached max_pages before starting a new levelCOMMENT
LOWcrawl4ai/deep_crawling/bfs_strategy.py361 # Check if we've reached the limit during batch processingCOMMENT
LOWcrawl4ai/deep_crawling/filters.py170 # Check if it's a regex patternCOMMENT
LOWcrawl4ai/deep_crawling/filters.py491 # Check if domain matches any allowed domain (including subdomains)COMMENT
91 more matches not shown…
Verbosity Indicators107 hits · 170 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/cache_validator.py112 # Step 1: Try HEAD request with conditional headersCOMMENT
LOWcrawl4ai/cache_validator.py156 # Step 2: No conditional headers available, try fingerprint onlyCOMMENT
LOWcrawl4ai/cache_validator.py180 # Step 3: No validation data availableCOMMENT
LOW⚡crawl4ai/async_url_seeder.py919 # Step 1: Find sitemap URL and get lastmod (needed for validation)COMMENT
LOW⚡crawl4ai/async_url_seeder.py943 # Step 2: Check cache validity (skip if force=True)COMMENT
LOW⚡crawl4ai/async_url_seeder.py998 # Step 4: Write to cache (FALLBACK: if write fails, URLs still yielded above)COMMENT
LOWcrawl4ai/async_url_seeder.py957 # Step 3: Fetch fresh URLsCOMMENT
LOW⚡crawl4ai/cloud/cli.py253 # Step 1: Shrink (unless --no-shrink)COMMENT
LOW⚡crawl4ai/cloud/cli.py266 # Step 2: Package as tar.gzCOMMENT
LOWcrawl4ai/cloud/cli.py281 # Step 3: UploadCOMMENT
LOW⚡tests/test_webhook_feature.sh104# Step 1: Save current branch and fetch PRCOMMENT
LOW⚡tests/test_webhook_feature.sh112# Step 2: Switch to new branchCOMMENT
LOW⚡tests/test_webhook_feature.sh117# Step 3: Activate virtual environmentCOMMENT
LOW⚡tests/test_webhook_feature.sh128# Step 4: Install server dependenciesCOMMENT
LOWtests/test_webhook_feature.sh147# Step 5: Start Redis in backgroundCOMMENT
LOWtests/test_webhook_feature.sh183# Step 6: Create and run webhook testCOMMENT
LOWtests/test_webhook_feature.sh292# Step 7: Verify resultsCOMMENT
LOWtests/test_webhook_feature.sh303# Step 8: Cleanup happens automatically via trapCOMMENT
LOW⚡tests/test_pyopenssl_update.py141 # Step 1: Check versionsCOMMENT
LOW⚡tests/test_pyopenssl_update.py147 # Step 2: Test basic crawlingCOMMENT
LOW⚡tests/test_pyopenssl_update.py153 # Step 3: Test stealth modeCOMMENT
LOW⚡tests/proxy/test_proxy_verify.py79 # Step 1: Verify IPsCOMMENT
LOW⚡tests/proxy/test_proxy_verify.py86 # Step 2: Get NST proxiesCOMMENT
LOWtests/proxy/test_proxy_verify.py97 # Step 3: Test Chanel with all available proxiesCOMMENT
LOW⚡tests/general/test_async_url_seeder_bm25.py558 # Step 1: Discover and score URLsCOMMENT
LOW⚡tests/general/test_async_url_seeder_bm25.py587 # Step 3: Verify these URLs would be good for actual crawlingCOMMENT
LOWtests/general/test_async_url_seeder_bm25.py573 # Step 2: Analyze top resultsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py608 # Step 1: open all sessionsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py615 # Step 2: navigate each session to a second pageCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py620 # Step 3: kill sessions one by one, verify others unaffectedCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py936 # Step 1: open sessionCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py943 # Step 2: concurrent non-session crawlsCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py952 # Step 3: kill sessionCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py955 # Step 4: trigger recycleCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py962 # Step 5: new session on fresh browserCOMMENT
LOW⚡tests/async/test_browser_lifecycle.py970 # Step 6: verify it worksCOMMENT
LOW⚡tests/async/test_browser_memory.py782 # Step 1: login — sets cookieCOMMENT
LOW⚡tests/async/test_browser_memory.py787 # Step 2: dashboard — cookie should carry over via sessionCOMMENT
LOW⚡tests/browser/test_builtin_browser.py52 # Step 1: Create a BrowserManager with builtin modeCOMMENT
LOW⚡tests/browser/test_builtin_browser.py57 # Step 2: Check if we have a BuiltinBrowserStrategyCOMMENT
LOW⚡tests/browser/test_builtin_browser.py69 # Step 3: Start the manager to launch or connect to builtin browserCOMMENT
LOW⚡tests/browser/test_builtin_browser.py78 # Step 4: Get browser info from the strategyCOMMENT
LOW⚡tests/browser/test_builtin_browser.py122 # Step 2: Get multiple pagesCOMMENT
LOW⚡tests/browser/test_builtin_browser.py149 # Step 1: Get browser statusCOMMENT
LOW⚡tests/browser/test_builtin_browser.py160 # Step 2: Test killing the browserCOMMENT
LOW⚡tests/browser/test_builtin_browser.py172 # Step 3: Check status after killCOMMENT
LOW⚡tests/browser/test_builtin_browser.py184 # Step 4: Launch a new browserCOMMENT
LOW⚡tests/browser/test_builtin_browser.py206 # Step 1: Create first managerCOMMENT
LOW⚡tests/browser/test_builtin_browser.py211 # Step 2: Create second managerCOMMENT
LOW⚡tests/browser/test_builtin_browser.py216 # Step 3: Start both managers (should connect to the same builtin browser)COMMENT
LOW⚡tests/browser/test_builtin_browser.py241 # Step 4: Test using both managersCOMMENT
LOW⚡tests/browser/test_builtin_browser.py263 # Step 5: Close both managersCOMMENT
LOW⚡tests/browser/test_builtin_browser.py309 # Step 2: Test killing the browser while manager is activeCOMMENT
LOWtests/browser/test_builtin_browser.py103 # Step 1: Get a single pageCOMMENT
LOWtests/browser/test_builtin_browser.py282 # Step 1: Test multiple starts with the same managerCOMMENT
LOWtests/browser/test_builtin_browser.py472 # Step 1: Create and start multiple browser managers in parallelCOMMENT
LOWtests/browser/test_builtin_browser.py666 # Step 1: Create and start multiple browser managers in parallelCOMMENT
LOW⚡tests/async_assistant/test_extract_pipeline.py62 # Step 1: StartingCOMMENT
LOW⚡tests/async_assistant/test_extract_pipeline.py65 # Step 2: Quick crawl for analysisCOMMENT
LOWtests/async_assistant/test_extract_pipeline.py79 # Step 3: HTML Skimming using lxmlCOMMENT
47 more matches not shown…
Magic Placeholder Names15 hits · 80 pts
SeverityFileLineSnippetContext
HIGHdeploy/docker/c4ai-doc-context.md1105 llm_config = LLMConfig(provider="openai/gpt-4",api_token="sk-YOUR_API_KEY")STRING
HIGHdocs/md_v2/complete-sdk-reference.md2936 llm_config = LLMConfig(provider="openai/gpt-4",api_token="sk-YOUR_API_KEY")STRING
HIGH⚡docs/md_v2/core/adaptive-crawling.md119 api_token='your-api-key'CODE
HIGH⚡docs/md_v2/core/adaptive-crawling.md124 api_token='your-api-key'CODE
HIGH⚡docs/md_v2/core/adaptive-crawling.md133 'api_token': 'your-api-key'CODE
HIGH⚡docs/md_v2/core/adaptive-crawling.md137 'api_token': 'your-api-key'CODE
HIGHdocs/md_v2/core/table_extraction.md141 api_token="your_api_key",CODE
HIGHdocs/md_v2/core/content-selection.md305 llm_config = LLMConfig(provider="openai/gpt-4",api_token="sk-YOUR_API_KEY")CODE
HIGH⚡docs/md_v2/marketplace/frontend/app-detail.js181 api_token="your-api-key",CODE
HIGHdocs/md_v2/marketplace/frontend/app-detail.html167 api_token="your-api-key",CODE
HIGHdocs/md_v2/blog/releases/0.5.0.md409 llm_config = LLMConfig(provider="openai/gpt-4o", api_token="YOUR_API_KEY")CODE
HIGH…cs/examples/url_seeder/bbc_sport_research_assistant.py21- export GEMINI_API_KEY="your-api-key"STRING
HIGHdocs/examples/website-to-api/README.md68 "api_token": "your-api-key-here"CODE
HIGHdocs/examples/website-to-api/README.md160 "api_token": "your-api-key-here"CODE
HIGHdocs/examples/website-to-api/README.md204 api_token="your-api-key"CODE
Over-Commented Block66 hits · 56 pts
SeverityFileLineSnippetContext
LOWdocker-compose.yml1# Shared configuration for all environmentsCOMMENT
LOWcrawl4ai/adaptive_crawler.py201 embedding_top_k_weight: float = 0.3 # Weight for top-k average in hybrid scoringCOMMENT
LOWcrawl4ai/adaptive_crawler.py561 # if hasattr(result, 'extracted_content') and result.extracted_content:COMMENT
LOWcrawl4ai/adaptive_crawler.py761 COMMENT
LOWcrawl4ai/adaptive_crawler.py1021 COMMENT
LOWcrawl4ai/adaptive_crawler.py1041 COMMENT
LOWcrawl4ai/adaptive_crawler.py1061 # # Top-k average (top 3)COMMENT
LOWcrawl4ai/models.py161 # Anti-bot retry/proxy usage statsCOMMENT
LOWcrawl4ai/extraction_strategy.py241 # self.tokenizer = self.model.tokenizerCOMMENT
LOWcrawl4ai/__init__.py221COMMENT
LOWcrawl4ai/utils.py1861 ]COMMENT
LOWcrawl4ai/utils.py3221 # # 4. Find repeated patterns and keep only a few examplesCOMMENT
LOWcrawl4ai/utils.py3241 COMMENT
LOWcrawl4ai/async_crawler_strategy.py881 # () => {COMMENT
LOWcrawl4ai/async_crawler_strategy.py901 # hasContent: body.innerHTML.length,COMMENT
LOWcrawl4ai/async_crawler_strategy.py2081 # If the user code triggers navigation, we catch the "context destroyed" errorCOMMENT
LOWcrawl4ai/async_crawler_strategy.py2161 message="DOM content load timeout: {error}",COMMENT
LOWcrawl4ai/async_configs.py521 if isinstance(data, (str, int, float, bool)):COMMENT
LOWcrawl4ai/async_url_seeder.py1641 COMMENT
LOWcrawl4ai/async_url_seeder.py1661 # # 7. Media files (often not useful for text content)COMMENT
LOWcrawl4ai/html2text/__init__.py1201 # self.inside_pre = TrueCOMMENT
LOWcrawl4ai/legacy/crawler_strategy.py101 self.options.add_argument("--headless")COMMENT
LOWcrawl4ai/processors/pdf/__init__.py21# The library deliberately has no egress policy of its own: as a plain libraryCOMMENT
LOWdeploy/docker/server.py141COMMENT
LOWdeploy/docker/c4ai-code-context.md2001COMMENT
LOWdeploy/docker/c4ai-code-context.md5361 dispatch_result: Optional[DispatchResult] = NoneCOMMENT
LOWdeploy/docker/config.yml101 # SECURITY: --no-sandbox disables the Chromium renderer sandbox (a rendererCOMMENT
LOWtests/test_webhook_feature.sh1#!/bin/bashCOMMENT
LOWtests/test_llm_simple_url.py101 # result_default = await crawler.arun(COMMENT
LOWtests/test_llm_simple_url.py121 # print(f" Default headers: {len(default_first['headers'])} columns")COMMENT
LOWtests/test_cli_docs.py21COMMENT
LOWtests/docker/test_hooks_utility.py181 # print("✓ All tests completed successfully!")COMMENT
LOWtests/docker/simple_api_test.py141 # result = self.test_get_endpoint("/schema")COMMENT
LOWtests/docker/test_serialization.py121# WebScrapingStrategy, LXMLWebScrapingStrategyCOMMENT
LOWtests/docker/test_serialization.py141# print("\nSerialized Config:")COMMENT
LOWtests/docker/test_serialization.py161# "language": "english"COMMENT
LOWtests/general/test_async_crawler_strategy.py241# async def test_js_return_values(crawler_strategy):COMMENT
LOWtests/general/test_async_crawler_strategy.py281# nonExistentFunction();COMMENT
LOWtests/async/test_error_handling.py1# import osCOMMENT
LOWtests/async/test_error_handling.py21# async def cleanup(self):COMMENT
LOWtests/async/test_error_handling.py41# # # Simulating a timeout by using a very short timeout valueCOMMENT
LOWtests/async/test_error_handling.py61# # @pytest.mark.asyncioCOMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py281 # "exclude_social_media_links": {COMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py301 # "combo_mode": {COMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py321 # "css_selector": "section#promo-section"COMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py341 # "remove_forms": TrueCOMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py561COMMENT
LOW…est_evaluation_scraping_methods_performance.configs.py581 # if link_diff:COMMENT
LOWtests/async/test_chunking_and_extraction_strategies.py21 result = await crawler.arun(COMMENT
LOWtests/async/test_chunking_and_extraction_strategies.py61 assert len(extracted_data) > 0COMMENT
LOWtests/async/test_edge_cases.py21COMMENT
LOWtests/async/test_edge_cases.py41# url = "https://news.ycombinator.com/" # Hacker News has infinite scrollCOMMENT
LOWtests/browser/manager/demo_browser_manager.py461 start_time = time.time()COMMENT
LOWtests/profiler/test_create_profile.py21 COMMENT
LOWdocs/md_v2/complete-sdk-reference.md3661```pythonCOMMENT
LOWdocs/md_v2/advanced/hooks-auth.md81 # Example 2: (Optional) Simulate a login scenarioCOMMENT
LOWdocs/md_v2/core/self-hosting.md101cat > .llm.env << EOLCOMMENT
LOWdocs/md_v2/core/link-media.md281# ✅ Clean URL structure (docs.python.org/api/reference)COMMENT
LOWdocs/md_v2/ask_ai/ask-ai.js601 // NOTE: Virtual scrolling is complex. For now, we do direct rendering.COMMENT
LOWdocs/md_v2/assets/llm.txt/txt/cli.txt21### Profile Management CommandsCOMMENT
6 more matches not shown…
Fake / Example Data53 hits · 52 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/docker_client.py212 await client.authenticate("user@example.com")CODE
LOWcrawl4ai/js_snippet/update_image_dimensions.js15 if (img.src.includes("placeholder") || img.src.includes("icon")) return false;CODE
LOWdeploy/docker/WEBHOOK_EXAMPLES.md206 "author": "John Doe",CODE
LOWdeploy/docker/README.md484 # await client.authenticate("user@example.com") # See Server Configuration sectionCOMMENT
LOWdeploy/docker/README.md834 "author": "John Doe",CODE
LOWdeploy/docker/c4ai-doc-context.md2784 "email": "user@example.com"CODE
LOWdeploy/docker/c4ai-doc-context.md2791 "email": "user@example.com",CODE
LOWdeploy/docker/c4ai-doc-context.md2809 await client.authenticate("user@example.com")CODE
LOWdeploy/docker/c4ai-doc-context.md3549 ["John Doe", "34", "New York"],CODE
LOWtests/memory/test_stress_sdk.py57 self.lorem_words = " ".join("lorem ipsum dolor sit amet " * 100).split()CODE
LOWtests/memory/test_stress_sdk.py57 self.lorem_words = " ".join("lorem ipsum dolor sit amet " * 100).split()CODE
LOW⚡tests/general/test_generate_schema_usage.py87 api_token: str = "fake-token"STRING
LOW⚡tests/async/test_browser_memory.py37<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.CODE
LOW⚡tests/async/test_browser_memory.py37<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.CODE
LOWtests/regression/test_reg_extraction.py363 assert any("sales@example.org" in v for v in values), (CODE
LOWtests/adaptive/test_embedding_strategy.py141 'api_token': os.getenv('OPENAI_API_KEY', 'dummy-key')CODE
LOWtests/adaptive/test_embedding_performance.py125 'api_token': 'dummy-key'CODE
LOWdocs/md_v2/core/self-hosting.md821 "author": "John Doe",CODE
LOWdocs/md_v2/core/self-hosting.md1195 # await client.authenticate("user@example.com") # See Server Configuration sectionCOMMENT
LOWdocs/md_v2/core/c4a-script.md174| `SET` | Set input value directly | `SET \`#email\` "user@example.com"` |CODE
LOWdocs/md_v2/core/c4a-script.md204TYPE "user@example.com"CODE
LOWdocs/md_v2/core/c4a-script.md250SET `#name` "John Doe"CODE
LOWdocs/md_v2/core/c4a-script.md347TYPE "user@example.com"STRING
LOWdocs/md_v2/marketplace/README.md17python dummy_data.pyCODE
LOWdocs/md_v2/marketplace/README.md59Coming soon - for now, edit the database directly or modify `dummy_data.py`CODE
LOWdocs/md_v2/marketplace/backend/dummy_data.py154 "Review", "John Doe", ["Playwright Cloud", "Puppeteer Extra"],STRING
LOWdocs/md_v2/marketplace/backend/dummy_data.py209This is a comprehensive article about {title.lower()}. Lorem ipsum dolor sit amet, consectetur adipiscing elit.CODE
LOWdocs/md_v2/marketplace/backend/dummy_data.py209This is a comprehensive article about {title.lower()}. Lorem ipsum dolor sit amet, consectetur adipiscing elit.CODE
LOWdocs/md_v2/blog/releases/0.7.6.md105 "author": "John Doe",CODE
LOWdocs/md_v2/api/c4a-script-reference.md373TYPE "user@example.com"CODE
LOWdocs/md_v2/api/c4a-script-reference.md397SETVAR email = "user@example.com"CODE
LOWdocs/md_v2/api/c4a-script-reference.md527SET `#email` "user@example.com"CODE
LOW⚡docs/md_v2/api/c4a-script-reference.md766TYPE "user@example.com"CODE
LOWdocs/md_v2/api/c4a-script-reference.md894SETVAR email = "user@example.com"CODE
LOWdocs/md_v2/api/c4a-script-reference.md951SET `#name` "John Doe"CODE
LOWdocs/md_v2/assets/highlight.min.js317 }),re=["a","abbr","address","article","aside","audio","b","blockquote","body","button","canvas","caption","cite","coCODE
LOWdocs/md_v2/apps/crawl4ai-assistant/index.html616 <input type="text" id="userName" name="name" placeholder="John Doe" required>STRING
LOW⚡docs/md_v2/apps/c4a-script/server.py264TYPE "John Doe"CODE
LOWdocs/md_v2/apps/c4a-script/README.md149TYPE "user@example.com"CODE
LOWdocs/md_v2/apps/c4a-script/playground/index.html276 <p class="text-preview">Lorem ipsum dolor sit amet, consectetur adipiscing elit...</p>CODE
LOWdocs/md_v2/apps/c4a-script/playground/index.html276 <p class="text-preview">Lorem ipsum dolor sit amet, consectetur adipiscing elit...</p>CODE
LOW⚡docs/md_v2/apps/c4a-script/assets/app.js596 script: `# Multi-step form with validation\nCLICK \`a[href="#forms"]\`\nWAIT \`#survey-form\` 2\n\n# SteCODE
LOWdocs/releases_review/demo_v0.9.1.py104 has_author = "John Doe" in fitSTRING
LOWdocs/blog/release-v0.7.6.md105 "author": "John Doe",CODE
LOWdocs/examples/docker_config_obj.py124 await client.authenticate("user@example.com")CODE
LOWdocs/examples/docker_config_obj.py193 json={"email": "user@example.com"}CODE
LOWdocs/examples/c4a_script/generate_script_hello_world.py28 goal = "Fill in email 'user@example.com', password 'secret123', and submit the form"STRING
LOW⚡docs/examples/c4a_script/tutorial/server.py264TYPE "John Doe"CODE
LOWdocs/examples/c4a_script/tutorial/README.md149TYPE "user@example.com"CODE
LOWdocs/examples/c4a_script/tutorial/playground/index.html276 <p class="text-preview">Lorem ipsum dolor sit amet, consectetur adipiscing elit...</p>CODE
LOWdocs/examples/c4a_script/tutorial/playground/index.html276 <p class="text-preview">Lorem ipsum dolor sit amet, consectetur adipiscing elit...</p>CODE
LOW⚡docs/examples/c4a_script/tutorial/assets/app.js596 script: `# Multi-step form with validation\nCLICK \`a[href="#forms"]\`\nWAIT \`#survey-form\` 2\n\n# SteCODE
LOWdocs/examples/website-to-api/static/index.html89 "author": "John Doe",CODE
Docstring Block Structure10 hits · 50 pts
SeverityFileLineSnippetContext
HIGHcrawl4ai/extraction_strategy.py1706 Generate extraction schema from HTML content or URL(s) (sync version). Args: html (str, opSTRING
HIGHcrawl4ai/extraction_strategy.py1778 Generate extraction schema from HTML content or URL(s) (async version). Use this method when calling fSTRING
HIGHcrawl4ai/utils.py1193 Extracts and cleans content from website HTML, optimizing for useful media and contextual information. ParSTRING
HIGHcrawl4ai/utils.py3740 Convert hook function objects to string representations for Docker API. This utility simplifies the process ofSTRING
HIGHcrawl4ai/async_crawler_strategy.py301 Wait for a condition in a CSP-compliant way. Args: page: Playwright page object STRING
HIGHcrawl4ai/async_crawler_strategy.py439 Crawls a given URL or processes raw HTML/local file content based on the URL prefix. Args: STRING
HIGHcrawl4ai/async_webcrawler.py1006 Runs the crawler for multiple URLs concurrently using a configurable dispatcher strategy. Args: STRING
HIGHcrawl4ai/async_webcrawler.py1148 Discovers, filters, and optionally validates URLs for a given domain(s) using sitemaps and Common CrawlSTRING
HIGHcrawl4ai/script/c4ai_script.py624Compile C4A-Script from string or list of strings to JavaScript. Args: script: C4A-Script as a string oSTRING
HIGHdeploy/docker/c4ai-code-context.md2021 Runs the crawler for multiple URLs concurrently using a configurable dispatcher strategy. Args: STRING
AI Slop Vocabulary22 hits · 33 pts
SeverityFileLineSnippetContext
MEDIUMcrawl4ai/adaptive_crawler.py1571 """Print comprehensive statistics about the knowledge baseSTRING
MEDIUMcrawl4ai/prompts.py1174GENERATE_SCRIPT_PROMPT = r"""You are a world-class browser automation specialist. Your sole purpose is to convert a natuSTRING
LOW⚡crawl4ai/async_crawler_strategy.py1687 # Log the error but don't raise it - we'll just return None for the MHTMLSTRING
LOWcrawl4ai/async_crawler_strategy.py341 # For timeout or other cases, just return FalseSTRING
MEDIUMcrawl4ai/async_url_seeder.py1144 # Use lxml for XML parsing if available, as it's generally more robustCOMMENT
LOWcrawl4ai/browser_manager.py185 # If CDP URL provided, just return itCOMMENT
LOWdeploy/docker/server.py1140 # if no query, just return raw contextsCOMMENT
LOWtests/test_source_sibling_selector.py309 # This is actually fine — let's just use "source" with flat fields instead.COMMENT
MEDIUMtests/docker/test_hooks_comprehensive.py521 """Run comprehensive hook tests"""STRING
MEDIUMtests/memory/test_dispatcher_stress.py269 # First, elevate memory usage to create pressureCOMMENT
MEDIUMtests/memory/benchmark_report.py374 """Generate a comprehensive comparison report of multiple test runs.STRING
MEDIUMtests/general/test_mhtml.py5import re # For more robust MHTML checksCODE
MEDIUMtests/general/test_mhtml.py54 # 3. Check for MHTML structure indicators (more robust than simple string contains)COMMENT
MEDIUM⚡tests/general/test_async_url_seeder_bm25.py597 """Generate a comprehensive report of BM25 scoring effectiveness."""STRING
LOW…est_evaluation_scraping_methods_performance.configs.py69 # No <body> found; just return the <html> rootCOMMENT
MEDIUM…est_evaluation_scraping_methods_performance.configs.py110 # If you prefer ignoring newlines or multiple whitespace, do a more robust cleanupCOMMENT
MEDIUM…s/md_v2/apps/crawl4ai-assistant/content/click2crawl.js780 // Try to generate a robust selectorCOMMENT
LOWdocs/releases_review/v0.7.5_docker_hooks_demo.py367 # Use our reusable hook library - just pass the function objects!STRING
MEDIUMdocs/releases_review/demo_v0.7.7.py477 """Print comprehensive demo summary"""STRING
MEDIUMdocs/releases_review/demo_v0.7.7.py612 # Print comprehensive summaryCOMMENT
MEDIUMdocs/examples/stealth_mode_example.py510 # Show best practicesCOMMENT
LOWdocs/examples/docker_hooks_examples.py359 # Use our reusable hook library - just pass the function objects!STRING
AI Structural Patterns30 hits · 22 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/extraction_strategy.py556CODE
LOWcrawl4ai/extraction_strategy.py1692CODE
LOWcrawl4ai/extraction_strategy.py1764CODE
LOWcrawl4ai/async_dispatcher.py149CODE
LOWcrawl4ai/table_extraction.py737CODE
LOWcrawl4ai/async_configs.py896CODE
LOWcrawl4ai/async_configs.py1114CODE
LOWcrawl4ai/async_configs.py1275CODE
LOWcrawl4ai/async_configs.py1705CODE
LOWcrawl4ai/async_configs.py2347CODE
LOWcrawl4ai/async_configs.py2473CODE
LOWcrawl4ai/async_configs.py2572CODE
LOWcrawl4ai/content_filter_strategy.py849CODE
LOWcrawl4ai/browser_manager.py137CODE
LOWcrawl4ai/deep_crawling/bfs_strategy.py25CODE
LOWcrawl4ai/deep_crawling/bff_strategy.py36CODE
LOWcrawl4ai/legacy/web_crawler.py56CODE
LOWcrawl4ai/legacy/web_crawler.py82CODE
LOWcrawl4ai/legacy/web_crawler.py121CODE
LOWcrawl4ai/processors/pdf/__init__.py104CODE
LOWcrawl4ai/components/crawler_monitor.py512CODE
LOWdeploy/docker/api.py421CODE
LOWtests/memory/test_stress_sdk.py181CODE
LOWdocs/examples/docker/demo_docker_api.py290CODE
LOWdocs/examples/docker/demo_docker_api.py300CODE
LOWdocs/examples/docker/demo_docker_api.py322CODE
LOWdocs/examples/docker/demo_docker_api.py350CODE
LOWdocs/examples/docker/demo_docker_api.py377CODE
LOWdocs/examples/docker/demo_docker_api.py403CODE
LOWdocs/examples/c4a_script/api_usage_examples.py192CODE
Modern Structural Boilerplate20 hits · 19 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/adaptive_crawler.py296 async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:CODE
LOWcrawl4ai/adaptive_crawler.py548 async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:CODE
LOWcrawl4ai/adaptive_crawler.py1233 async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:CODE
LOWcrawl4ai/egress_policy.py27def set_egress_proxy(url: Optional[str]) -> None:CODE
LOWcrawl4ai/__init__.py115__all__ = [CODE
LOWcrawl4ai/hub.py9logger = logging.getLogger(__name__)CODE
LOWcrawl4ai/async_crawler_strategy.py2561 def set_hook(self, hook_type: str, hook_func: Callable) -> None:CODE
LOWcrawl4ai/deep_crawling/__init__.py26__all__ = [CODE
LOWcrawl4ai/script/__init__.py16__all__ = [CODE
LOWcrawl4ai/processors/pdf/__init__.py360__all__ = ["PDFCrawlerStrategy", "PDFContentScrapingStrategy"]STRING
LOWcrawl4ai/processors/pdf/processor.py22logger = logging.getLogger(__name__)CODE
LOWcrawl4ai/cloud/__init__.py12__all__ = [CODE
LOWdeploy/docker/monitor_routes.py11logger = logging.getLogger(__name__)CODE
LOW⚡deploy/docker/server.py543logger = logging.getLogger(__name__)CODE
LOWdeploy/docker/monitor.py14logger = logging.getLogger(__name__)CODE
LOWdeploy/docker/webhook.py16logger = logging.getLogger(__name__)CODE
LOWdeploy/docker/api.py95logger = logging.getLogger(__name__)CODE
LOWdeploy/docker/egress_broker.py163def set_egress_proxy(url: Optional[str]) -> None:CODE
LOWdeploy/docker/crawler_pool.py9logger = logging.getLogger(__name__)CODE
LOWdeploy/docker/work_queue.py104def set_job_queue(q: Optional[WorkQueue]) -> None:CODE
Hallucination Indicators1 hit · 10 pts
SeverityFileLineSnippetContext
CRITICALdocs/md_v2/apps/crawl4ai-assistant/libs/marked.min.js47`+s.text,this.inlineQueue.pop(),this.inlineQueue.at(-1).src=r.text):t.push(s);continue}if(e){let r="Infinite loop on bytCODE
Modern AI Meta-Vocabulary4 hits · 8 pts
SeverityFileLineSnippetContext
MEDIUMcrawl4ai/utils.py3578 # Get embedding model from config or use defaultSTRING
MEDIUMtests/adaptive/test_llm_embedding.py85 # "event-driven architecture patterns"COMMENT
MEDIUMdocs/md_v2/core/adaptive-crawling.md169 embedding_llm_config=None, # Use for API-based embeddings (embedding model)CODE
MEDIUMdocs/examples/adaptive_crawling/llm_config_example.py85 # "event-driven architecture patterns"COMMENT
Example Usage Blocks6 hits · 8 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/user_agent_generator.py417# Example usage:COMMENT
LOWcrawl4ai/user_agent_generator.py420 # Usage example:COMMENT
LOWcrawl4ai/browser_profiler.py1379 # Example usageCOMMENT
LOWcrawl4ai/docker_client.py209# Example usageCOMMENT
LOWcrawl4ai/processors/pdf/processor.py476# Usage exampleCOMMENT
LOWtests/profiler/test_create_profile.py6 # Example usageCOMMENT
Dead Code4 hits · 8 pts
SeverityFileLineSnippetContext
MEDIUMcrawl4ai/deep_crawling/crazy.py96CODE
MEDIUMcrawl4ai/legacy/web_crawler.py80CODE
MEDIUMdeploy/docker/webhook.py183CODE
MEDIUMdeploy/docker/webhook.py186CODE
Slop Phrases4 hits · 6 pts
SeverityFileLineSnippetContext
LOWcrawl4ai/models.py327# When removing this code in the future, make sure to:COMMENT
MEDIUMcrawl4ai/table_extraction.py1315 This is a basic implementation - for complex CSS selectors,STRING
MEDIUMcrawl4ai/processors/pdf/__init__.py197 # For simple cases, you can use the sync versionSTRING
MEDIUMdocs/examples/undetected_simple_demo.py88 # Test URLs - you can change theseCOMMENT