Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
This report summarizes scanner evidence for apify/crawlee-python, a Python project with 9,582 GitHub stars. SynthScan examined 116,275 lines of code across 1010 source files, recording 2430 pattern matches distributed across 20 syntactic categories. The legacy adjusted pattern signal is 32.5 (High pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 1724 finding(s).
Are imports declared and package/API names verified? 467 finding(s).
Are wrappers, guards, and repeated structures necessary? 1448 finding(s).
Which changes need focused human review? 2430 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 2430 distinct pattern matches across 20 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ides/code_examples/http_crawlers/selectolax_context.py | 25 | def from_parsed_http_crawling_context( | CODE |
| LOW | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 13 | def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None: | CODE |
| LOW | …/examples/code_examples/adaptive_playwright_crawler.py | 22 | async def request_handler_for_label( | CODE |
| LOW | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 13 | def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None: | CODE |
| LOW | …/examples/code_examples/adaptive_playwright_crawler.py | 21 | async def request_handler_for_label( | CODE |
| LOW | website/src/theme/Navbar/MobileSidebar/Layout/index.js | 5 | export default function NavbarMobileSidebarLayout({ header, primaryMenu, secondaryMenu }) { | CODE |
| LOW | website/src/theme/Navbar/MobileSidebar/Header/index.js | 29 | export default function NavbarMobileSidebarHeader() { | CODE |
| LOW | …/src/theme/NavbarItem/DocsVersionDropdownNavbarItem.js | 93 | export default function DocsVersionDropdownNavbarItem({ | CODE |
| LOW | tests/unit/conftest.py | 86 | def _isolate_test_environment(prepare_test_env: Callable[[], None]) -> None: | CODE |
| LOW | tests/unit/conftest.py | 99 | def _restore_crawlers_binding(monkeypatch: pytest.MonkeyPatch) -> None: | CODE |
| LOW | tests/unit/server.py | 350 | async def generic_response_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW | tests/unit/server.py | 358 | async def problematic_links_endpoint(scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW | tests/unit/server.py | 558 | async def resource_loading_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW | tests/unit/server.py | 576 | async def start_enqueue_non_href_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 23 | def test_configuration_rejects_out_of_range_max_used_cpu_ratio(invalid_value: float) -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 29 | def test_configuration_rejects_out_of_range_max_used_memory_ratio(invalid_value: float) -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 35 | def test_configuration_accepts_in_range_max_used_cpu_ratio(valid_value: float) -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 40 | def test_configuration_accepts_in_range_max_used_memory_ratio(valid_value: float) -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 44 | def test_global_configuration_works() -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 53 | def test_global_configuration_works_reversed() -> None: | CODE |
| LOW⚡ | tests/unit/test_configuration.py | 62 | async def test_storage_not_persisted_when_non_persistable_storage_used(tmp_path: Path, server_url: URL) -> None: | CODE |
| LOW | tests/unit/test_configuration.py | 78 | async def test_storage_persisted_with_explicit_statistics_with_persistable_storage( | CODE |
| LOW | tests/unit/test_configuration.py | 103 | async def test_storage_persisted_when_enabled(tmp_path: Path, server_url: URL) -> None: | CODE |
| LOW | tests/unit/test_request.py | 6 | def test_from_url_does_not_mutate_caller_user_data() -> None: | CODE |
| LOW⚡ | tests/unit/test_request.py | 17 | def test_from_url_can_reuse_same_user_data_dict() -> None: | CODE |
| LOW⚡ | tests/unit/test_request.py | 26 | def test_from_url_omits_crawlee_data_when_empty() -> None: | CODE |
| LOW⚡ | tests/unit/test_request.py | 33 | def test_from_url_keeps_crawlee_data_when_supplied() -> None: | CODE |
| LOW | tests/unit/test_router.py | 37 | async def test_router_no_default_handler() -> None: | CODE |
| LOW | tests/unit/test_router.py | 51 | async def test_router_default_handler_invoked() -> None: | CODE |
| LOW | tests/unit/test_router.py | 70 | async def test_router_specific_handler_invoked() -> None: | CODE |
| LOW | tests/unit/test_router.py | 95 | async def test_router_handler_not_nullified() -> None: | CODE |
| LOW | tests/unit/test_router.py | 105 | async def test_router_multi_labelled_handler() -> None: | CODE |
| LOW | tests/unit/test_router.py | 121 | async def test_router_use_middleware() -> None: | CODE |
| LOW | tests/unit/test_router.py | 149 | async def test_router_use_middleware_with_label() -> None: | CODE |
| LOW | tests/unit/test_router.py | 184 | async def test_router_middleware_order_execution() -> None: | CODE |
| LOW | tests/unit/test_router.py | 208 | async def test_router_middleware_exception_interrupts_chain() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 12 | def test_default_configuration() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 18 | def test_custom_configuration() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 25 | def test_configuration_overwrite_not_possible() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 34 | def test_configuration_conflict() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 42 | def test_default_event_manager() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 47 | def test_custom_event_manager() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 54 | def test_event_manager_overwrite_not_possible() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 63 | def test_event_manager_conflict() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 71 | def test_default_storage_client() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 76 | def test_custom_storage_client() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 83 | def test_storage_client_overwrite_not_possible() -> None: | CODE |
| LOW⚡ | tests/unit/test_service_locator.py | 92 | def test_storage_client_conflict() -> None: | CODE |
| LOW | tests/unit/test_log_config.py | 40 | def test_formatting_with_exception() -> None: | CODE |
| LOW | tests/unit/test_log_config.py | 54 | def test_formatter_without_name() -> None: | CODE |
| LOW | tests/unit/utils.py | 31 | def make_status_stream_client( | CODE |
| LOW | tests/unit/utils.py | 69 | async def open_throttler_held_until_reclaim( | CODE |
| LOW | tests/unit/test_cli.py | 57 | def test_create_interactive_non_default_template(mock_cookiecutter: Mock, monkeypatch: pytest.MonkeyPatch) -> None: | CODE |
| LOW | tests/unit/test_cli.py | 91 | def test_create_non_interactive(mock_cookiecutter: Mock) -> None: | CODE |
| LOW | tests/unit/test_cli.py | 125 | def test_create_existing_folder( | CODE |
| LOW | tests/unit/test_cli.py | 174 | def test_create_existing_folder_interactive( | CODE |
| LOW | tests/unit/test_cli.py | 214 | def test_create_existing_folder_interactive_multiple_attempts( | CODE |
| LOW | tests/unit/test_cli.py | 266 | def test_import_error_handled(optional_module_name: str) -> None: | CODE |
| LOW⚡ | tests/unit/test_throttling_request_manager.py | 296 | async def test_record_domain_delay_returns_false_for_unconfigured_domain( | CODE |
| LOW⚡ | tests/unit/test_throttling_request_manager.py | 304 | async def test_different_domains_independent(manager: ThrottlingRequestManager[RequestQueue]) -> None: | CODE |
| 1119 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …_docs/version-1.10/introduction/code_examples/02_bs.py | 17 | # Define a request handler and attach it to the crawler using the decorator. | COMMENT |
| MEDIUM | …ides/code_examples/request_router/router_middleware.py | 10 | # Create a custom router instance | COMMENT |
| MEDIUM | …_examples/request_router/custom_router_default_only.py | 8 | # Create a custom router instance | COMMENT |
| MEDIUM | …code_examples/request_router/failed_request_handler.py | 7 | # Create a crawler instance with retry settings | COMMENT |
| MEDIUM | …0/guides/code_examples/request_router/error_handler.py | 11 | # Create a crawler instance | COMMENT |
| MEDIUM | …code_examples/request_router/simple_default_handler.py | 7 | # Create a crawler instance | COMMENT |
| MEDIUM | …code_examples/request_router/basic_request_handlers.py | 9 | # Create a custom router instance | COMMENT |
| MEDIUM | …code_examples/request_router/basic_request_handlers.py | 12 | # Define the default handler (fallback for requests without specific labels) | COMMENT |
| MEDIUM | …code_examples/request_router/basic_request_handlers.py | 31 | # Define a handler for category pages | COMMENT |
| MEDIUM | …code_examples/request_router/basic_request_handlers.py | 53 | # Define a handler for product detail pages | COMMENT |
| MEDIUM | …ples/storages/dataset_with_crawler_explicit_example.py | 12 | # Create a new crawler (it can be any subclass of BasicCrawler). | COMMENT |
| MEDIUM | …ples/storages/dataset_with_crawler_explicit_example.py | 15 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …des/code_examples/storages/kvs_with_crawler_example.py | 7 | # Create a new Playwright crawler. | COMMENT |
| MEDIUM | …des/code_examples/storages/kvs_with_crawler_example.py | 10 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …code_examples/storages/dataset_with_crawler_example.py | 7 | # Create a new crawler (it can be any subclass of BasicCrawler). | COMMENT |
| MEDIUM | …code_examples/storages/dataset_with_crawler_example.py | 10 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …examples/storages/kvs_with_crawler_explicit_example.py | 12 | # Create a new Playwright crawler. | COMMENT |
| MEDIUM | …examples/storages/kvs_with_crawler_explicit_example.py | 15 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …ides/code_examples/storages/rq_with_crawler_example.py | 7 | # Create a new crawler (it can be any subclass of BasicCrawler). Request queue is | COMMENT |
| MEDIUM | …ides/code_examples/storages/rq_with_crawler_example.py | 11 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …_examples/storages/rq_with_crawler_explicit_example.py | 15 | # Create a new crawler (it can be any subclass of BasicCrawler) and pass the request | COMMENT |
| MEDIUM | …_examples/storages/rq_with_crawler_explicit_example.py | 19 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …s/code_examples/http_crawlers/beautifulsoup_example.py | 7 | # Create a BeautifulSoupCrawler instance | COMMENT |
| MEDIUM | …s/code_examples/http_crawlers/beautifulsoup_example.py | 13 | # Define the default request handler | COMMENT |
| MEDIUM | ….10/guides/code_examples/http_crawlers/http_example.py | 8 | # Create an HttpCrawler instance - no automatic parsing | COMMENT |
| MEDIUM | ….10/guides/code_examples/http_crawlers/http_example.py | 14 | # Define the default request handler | COMMENT |
| MEDIUM | …0/guides/code_examples/http_crawlers/parsel_example.py | 7 | # Create a ParselCrawler instance | COMMENT |
| MEDIUM | …0/guides/code_examples/http_crawlers/parsel_example.py | 13 | # Define the default request handler | COMMENT |
| MEDIUM⚡ | …reating_web_archive/manual_archiving_parsel_crawler.py | 37 | # Create a WARC archive file a prepare the writer. | COMMENT |
| MEDIUM⚡ | …reating_web_archive/manual_archiving_parsel_crawler.py | 42 | # Create a WARC info record to store metadata about the archive. | COMMENT |
| MEDIUM⚡ | …reating_web_archive/manual_archiving_parsel_crawler.py | 50 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …ing_web_archive/manual_archiving_playwright_crawler.py | 52 | # Create a WARC archive file a prepare the writer. | COMMENT |
| MEDIUM | …ing_web_archive/manual_archiving_playwright_crawler.py | 57 | # Create a WARC info record to store metadata about the archive. | COMMENT |
| MEDIUM | …-1.10/guides/code_examples/login_crawler/http_login.py | 68 | # Create a POST request to the authentication API endpoint | COMMENT |
| MEDIUM | …des/code_examples/http_clients/parsel_impit_example.py | 21 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …des/code_examples/http_clients/parsel_httpx_example.py | 20 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …amples/http_clients/parsel_curl_impersonate_example.py | 20 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …des/code_examples/cookie_management/persist_cookies.py | 9 | # Define the pool outside the crawler so its state can be read before the run. | COMMENT |
| MEDIUM | …des/code_examples/error_handling/handle_proxy_error.py | 27 | # Create a new `Request` with a modified key to avoid deduplication | COMMENT |
| MEDIUM | …0/guides/code_examples/error_handling/disable_retry.py | 10 | # Create a parsing error for demonstration | COMMENT |
| MEDIUM | …void_blocking/playwright_with_fingerprint_generator.py | 10 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …xamples/avoid_blocking/playwright_with_cloakbrowser.py | 56 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …examples/playwright_crawler/multiple_launch_example.py | 8 | # Create a plugin for each required browser. | COMMENT |
| MEDIUM | …/storage_clients/redis_storage_client_basic_example.py | 4 | # Create a new instance of storage client using connection string. | COMMENT |
| MEDIUM | …ge_clients/file_system_storage_client_basic_example.py | 4 | # Create a new instance of storage client. | COMMENT |
| MEDIUM | …ts/file_system_storage_client_configuration_example.py | 5 | # Create a new instance of storage client. | COMMENT |
| MEDIUM | …ts/file_system_storage_client_configuration_example.py | 8 | # Create a configuration with custom settings. | COMMENT |
| MEDIUM | …_clients/redis_storage_client_configuration_example.py | 7 | # Create a new instance of storage client using a Redis client with custom settings. | COMMENT |
| MEDIUM | …_clients/redis_storage_client_configuration_example.py | 20 | # Create a configuration with custom settings. | COMMENT |
| MEDIUM | …es/storage_clients/sql_storage_client_basic_example.py | 6 | # Create a new instance of storage client. | COMMENT |
| MEDIUM | …ge_clients/sql_storage_client_configuration_example.py | 9 | # Create a new instance of storage client. | COMMENT |
| MEDIUM | …ge_clients/sql_storage_client_configuration_example.py | 24 | # Create a configuration with custom settings. | COMMENT |
| MEDIUM | …storage_clients/memory_storage_client_basic_example.py | 4 | # Create a new instance of storage client. | COMMENT |
| MEDIUM | …des/code_examples/proxy_management/tiers_bs_example.py | 8 | # Create a ProxyConfiguration object and pass it to the crawler. | COMMENT |
| MEDIUM | …des/code_examples/proxy_management/tiers_bs_example.py | 28 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …des/code_examples/proxy_management/tiers_pw_example.py | 8 | # Create a ProxyConfiguration object and pass it to the crawler. | COMMENT |
| MEDIUM | …des/code_examples/proxy_management/tiers_pw_example.py | 28 | # Define the default request handler, which will be called for every request. | COMMENT |
| MEDIUM | …s/code_examples/proxy_management/session_bs_example.py | 8 | # Create a ProxyConfiguration object and pass it to the crawler. | COMMENT |
| MEDIUM | …s/code_examples/proxy_management/session_pw_example.py | 8 | # Create a ProxyConfiguration object and pass it to the crawler. | COMMENT |
| MEDIUM | …de_examples/proxy_management/integration_pw_example.py | 8 | # Create a ProxyConfiguration object and pass it to the crawler. | COMMENT |
| 304 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …reating_web_archive/manual_archiving_parsel_crawler.py | 0 | helper function for archiving response in warc format. | STRING |
| HIGH | …ing_web_archive/manual_archiving_playwright_crawler.py | 0 | helper function for archiving response in warc format. | STRING |
| HIGH | …reating_web_archive/manual_archiving_parsel_crawler.py | 0 | helper function for archiving response in warc format. | STRING |
| HIGH | …ing_web_archive/manual_archiving_playwright_crawler.py | 0 | helper function for archiving response in warc format. | STRING |
| HIGH | …_examples/playwright_crawler_adaptive/pre_nav_hooks.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …_examples/playwright_crawler_adaptive/pre_nav_hooks.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …_examples/playwright_crawler_adaptive/pre_nav_hooks.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would | STRING |
| HIGH | …0/guides/code_examples/running_in_web_server/server.py | 0 | <!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter | STRING |
| HIGH | …6/guides/code_examples/running_in_web_server/server.py | 0 | <!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter | STRING |
| HIGH | …s/guides/code_examples/running_in_web_server/server.py | 0 | <!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter | STRING |
| HIGH | …ples/pydantic_ai_crawler/selector_extractor_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …amples/pydantic_ai_crawler/custom_distiller_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …code_examples/pydantic_ai_crawler/debugging_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …des/code_examples/pydantic_ai_crawler/basic_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …de_examples/pydantic_ai_crawler/usage_limit_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …ples/pydantic_ai_crawler/selector_extractor_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …amples/pydantic_ai_crawler/custom_distiller_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …code_examples/pydantic_ai_crawler/debugging_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …des/code_examples/pydantic_ai_crawler/basic_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …de_examples/pydantic_ai_crawler/usage_limit_example.py | 0 | model representing the extracted data for an article. | STRING |
| HIGH | …0/deployment/code_examples/google/cloud_run_example.py | 0 | the crawler entry point that will be called when the http endpoint is accessed. | STRING |
| HIGH | …6/deployment/code_examples/google/cloud_run_example.py | 0 | the crawler entry point that will be called when the http endpoint is accessed. | STRING |
| HIGH | …s/deployment/code_examples/google/cloud_run_example.py | 0 | the crawler entry point that will be called when the http endpoint is accessed. | STRING |
| HIGH | …0/deployment/code_examples/google/cloud_run_example.py | 0 | default request handler that processes each page during crawling. | STRING |
| HIGH | …6/deployment/code_examples/google/cloud_run_example.py | 0 | default request handler that processes each page during crawling. | STRING |
| HIGH | …s/deployment/code_examples/google/cloud_run_example.py | 0 | default request handler that processes each page during crawling. | STRING |
| HIGH | …ples/code_examples/playwright_crawler_with_camoufox.py | 0 | example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin. | STRING |
| HIGH | …ples/code_examples/playwright_crawler_with_camoufox.py | 0 | example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin. | STRING |
| HIGH | …ples/code_examples/playwright_crawler_with_camoufox.py | 0 | example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin. | STRING |
| HIGH | …project_template/templates/main_playwright_camoufox.py | 0 | example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | stop crawler once specific url is visited. example of guard condition to stop the crawler. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | stop crawler once specific url is visited. example of guard condition to stop the crawler. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | stop crawler once specific url is visited. example of guard condition to stop the crawler. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | add requests to the queue after some time. can be done by external code. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | add requests to the queue after some time. can be done by external code. | STRING |
| HIGH | …ples/code_examples/beautifulsoup_crawler_keep_alive.py | 0 | add requests to the queue after some time. can be done by external code. | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed only in playwright sub crawler. it is safe to access `page` object. | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed only in playwright sub crawler. it is safe to access `page` object. | STRING |
| HIGH | …/examples/code_examples/adaptive_playwright_crawler.py | 0 | hook executed only in playwright sub crawler. it is safe to access `page` object. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test that open() raises an error when both id and name are provided. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test that open() raises an error when both id and name are provided. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test that open() raises an error when both id and name are provided. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test that providing both alias and id raises error. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test that providing both alias and id raises error. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test that providing both alias and id raises error. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test that providing both alias and name raises error. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test that providing both alias and name raises error. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test that providing both alias and name raises error. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test that conflicts between named and alias storages are detected. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test that conflicts between named and alias storages are detected. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test that conflicts between named and alias storages are detected. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged. | STRING |
| HIGH | tests/unit/storages/test_request_queue.py | 0 | test purge behavior when purge_on_start=false: all storages retain data regardless of type. | STRING |
| HIGH | tests/unit/storages/test_dataset.py | 0 | test purge behavior when purge_on_start=false: all storages retain data regardless of type. | STRING |
| HIGH | tests/unit/storages/test_key_value_store.py | 0 | test purge behavior when purge_on_start=false: all storages retain data regardless of type. | STRING |
| 51 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | website/generate_module_shortcuts.py | 3 | CODE | |
| LOW | ….10/introduction/code_examples/03_transform_request.py | 1 | CODE | |
| LOW | …uides/code_examples/http_crawlers/selectolax_parser.py | 1 | CODE | |
| LOW | …ides/code_examples/http_crawlers/selectolax_crawler.py | 1 | CODE | |
| LOW | …de_examples/cookie_management/retry_restore_cookies.py | 7 | CODE | |
| LOW | …/playwright_crawler/browser_pool_page_hooks_example.py | 1 | CODE | |
| LOW | …laywright_crawler/browser_pool_launch_hooks_example.py | 1 | CODE | |
| LOW | …mples/storage_clients/custom_storage_client_example.py | 1 | CODE | |
| LOW | …0/guides/code_examples/running_in_web_server/server.py | 1 | CODE | |
| LOW | …-1.10/examples/code_examples/configure_json_logging.py | 1 | CODE | |
| LOW | …0.6/introduction/code_examples/03_transform_request.py | 1 | CODE | |
| LOW | …6/guides/code_examples/running_in_web_server/server.py | 1 | CODE | |
| LOW | …n-0.6/examples/code_examples/configure_json_logging.py | 1 | CODE | |
| LOW | tests/unit/conftest.py | 1 | CODE | |
| LOW | tests/unit/server.py | 1 | CODE | |
| LOW | tests/unit/test_configuration.py | 1 | CODE | |
| LOW | tests/unit/test_request.py | 1 | CODE | |
| LOW | tests/unit/test_router.py | 1 | CODE | |
| LOW | tests/unit/test_service_locator.py | 1 | CODE | |
| LOW | tests/unit/test_log_config.py | 1 | CODE | |
| LOW | tests/unit/utils.py | 1 | CODE | |
| LOW | tests/unit/test_cli.py | 1 | CODE | |
| LOW | tests/unit/test_throttling_request_manager.py | 3 | CODE | |
| LOW | tests/unit/fingerprint_suite/test_header_generator.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_measure_time.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_retry.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_crypto.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_console.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_time.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_recurring_task.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_html.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_robots.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_log.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_sitemap.py | 34 | CODE | |
| LOW | tests/unit/_utils/test_sitemap.py | 34 | CODE | |
| LOW | tests/unit/_utils/test_try_import.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_http.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_system.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_system.py | 22 | CODE | |
| LOW | tests/unit/_utils/test_system.py | 22 | CODE | |
| LOW | tests/unit/_utils/test_system.py | 22 | CODE | |
| LOW | tests/unit/_utils/test_requests.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_file.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_globs.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_byte_size.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_timedelta_ms.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_urls.py | 1 | CODE | |
| LOW | tests/unit/_utils/test_html_to_text.py | 1 | CODE | |
| LOW | tests/unit/_statistics/test_persistence.py | 1 | CODE | |
| LOW | tests/unit/_statistics/test_request_max_duration.py | 1 | CODE | |
| LOW | tests/unit/_statistics/test_periodic_logging.py | 1 | CODE | |
| LOW | tests/unit/storages/test_request_manager_tandem.py | 1 | CODE | |
| LOW | tests/unit/storages/conftest.py | 1 | CODE | |
| LOW | tests/unit/storages/test_request_queue.py | 1 | CODE | |
| LOW | tests/unit/storages/test_dataset.py | 1 | CODE | |
| LOW | tests/unit/storages/test_key_value_store.py | 1 | CODE | |
| LOW | tests/unit/crawlers/_basic/test_context_pipeline.py | 1 | CODE | |
| LOW | tests/unit/crawlers/_basic/test_basic_crawler.py | 2 | CODE | |
| LOW | …/crawlers/_beautifulsoup/test_beautifulsoup_crawler.py | 1 | CODE | |
| LOW | tests/unit/crawlers/_parsel/test_parsel_crawler.py | 1 | CODE | |
| 396 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | src/crawlee/_utils/file.py | 67 | Resolve a storage subdirectory inside a base directory. Joins `subdirectory` onto `base_dir` and verifies that the | STRING |
| HIGH | src/crawlee/_utils/context.py | 12 | Ensure the (async) context manager is initialized before executing the method. This decorator checks if the calling | STRING |
| HIGH | src/crawlee/storages/_storage_instance_manager.py | 93 | Open a storage instance with caching support. Args: cls: The storage class to instantiate. | STRING |
| HIGH | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 353 | Execute an HTTP request utilizing the `BrowserPool` and the `Playwright` library. Args: context: Th | STRING |
| HIGH | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 501 | Record rate limiting and validate the HTTP status code, raising appropriate exceptions if needed. Args: | STRING |
| HIGH | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 531 | Try to detect if the request is blocked based on the response content. Args: context: The current c | STRING |
| HIGH | …wlee/crawlers/_abstract_http/_abstract_http_crawler.py | 298 | Record rate limiting and validate the HTTP status code, raising appropriate exceptions if needed. Args: | STRING |
| HIGH | …wlee/crawlers/_abstract_http/_abstract_http_crawler.py | 327 | Try to detect if the request is blocked based on the parsed response content. Args: context: The cu | STRING |
| HIGH | src/crawlee/http_clients/_base.py | 109 | Perform the crawling for a given request. This method is called from `crawler.run()`. Args: | STRING |
| HIGH | src/crawlee/http_clients/_base.py | 139 | Send an HTTP request via the client. This method is called from `context.send_request()` helper. Args: | STRING |
| HIGH | src/crawlee/http_clients/_base.py | 171 | Stream an HTTP request via the client. This method should be used for downloading potentially large data where | STRING |
| HIGH | src/crawlee/http_clients/_curl_impersonate.py | 335 | Convert from Crawlee HTTP method to curl-cffi HTTP method. Args: method: Crawlee HTTP method. | STRING |
| HIGH | src/crawlee/http_clients/_impit.py | 284 | Perform a request, following redirects one hop at a time. Redirects are resolved here instead of by `impit`, so | STRING |
| HIGH | src/crawlee/browsers/_browser_controller.py | 78 | Create a new page with the given context options. Args: browser_new_context_options: Keyword argume | STRING |
| HIGH | src/crawlee/browsers/_playwright_browser_controller.py | 163 | Create a new page with the given context options. Args: browser_new_context_options: Keyword argume | STRING |
| HIGH | src/crawlee/browsers/_stagehand_types.py | 119 | Extract structured data from the page using natural language. Argument `page` is automatically set. Args: | STRING |
| HIGH | src/crawlee/browsers/_stagehand_types.py | 131 | Run an autonomous multi-step AI agent on the page. Argument `page` is automatically set. Args: **kw | STRING |
| HIGH | …/crawlee/storage_clients/_sql/_request_queue_client.py | 133 | Open an existing request queue or create a new one. This method first tries to find an existing queue by ID or | STRING |
| HIGH | …rawlee/storage_clients/_sql/_key_value_store_client.py | 105 | Open or create a SQL key-value store client. This method attempts to open an existing key-value store from the | STRING |
| HIGH | src/crawlee/storage_clients/_sql/_dataset_client.py | 94 | Open an existing dataset or create a new one. Args: id: The ID of the dataset to open. If provided, | STRING |
| HIGH | …/storage_clients/_file_system/_request_queue_client.py | 161 | Open or create a file system request queue client. This method attempts to open an existing request queue from | STRING |
| HIGH | …torage_clients/_file_system/_key_value_store_client.py | 98 | Open or create a file system key-value store client. This method attempts to open an existing key-value store f | STRING |
| HIGH | …rawlee/storage_clients/_file_system/_dataset_client.py | 100 | Open or create a file system dataset client. This method attempts to open an existing dataset from the file sys | STRING |
| HIGH | …awlee/storage_clients/_memory/_request_queue_client.py | 72 | Open or create a new memory request queue client. This method creates a new in-memory request queue instance. U | STRING |
| HIGH | …lee/storage_clients/_memory/_key_value_store_client.py | 57 | Open or create a new memory key-value store client. This method creates a new in-memory key-value store instanc | STRING |
| HIGH | src/crawlee/storage_clients/_memory/_dataset_client.py | 62 | Open or create a new memory dataset client. This method creates a new in-memory dataset instance. Unlike persis | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …guides/code_examples/login_crawler/playwright_login.py | 47 | # Check if the session is available | COMMENT |
| LOW | …0/guides/code_examples/running_in_web_server/server.py | 57 | # Return the result | STRING |
| LOW⚡ | …code_examples/pydantic_ai_crawler/debugging_example.py | 40 | # Set concurrency to 1, which ensures only one request is processed at a time. | COMMENT |
| LOW⚡ | …code_examples/pydantic_ai_crawler/debugging_example.py | 44 | # Set abort_on_error to True to stop the crawl if an error occurs during | COMMENT |
| LOW | …les/code_examples/respect_robots_on_skipped_request.py | 22 | # Check if the request was skipped due to robots.txt rules | COMMENT |
| LOW | …guides/code_examples/login_crawler/playwright_login.py | 47 | # Check if the session is available | COMMENT |
| LOW | …6/guides/code_examples/running_in_web_server/server.py | 57 | # Return the result | STRING |
| LOW | …les/code_examples/respect_robots_on_skipped_request.py | 22 | # Check if the request was skipped due to robots.txt rules | COMMENT |
| LOW | tests/unit/crawlers/_basic/test_basic_crawler.py | 1292 | # Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately | COMMENT |
| LOW | tests/unit/crawlers/_basic/test_basic_crawler.py | 1316 | # Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately | COMMENT |
| LOW | tests/unit/crawlers/_basic/test_basic_crawler.py | 1566 | # Set max_concurrency to 1 to ensure testing urls are visited one by one in order. | COMMENT |
| LOW | tests/unit/crawlers/_basic/test_basic_crawler.py | 1594 | # Set concurrency to 2 to ensure two urls are being visited in parallel. | COMMENT |
| LOW | …/crawlers/_beautifulsoup/test_beautifulsoup_crawler.py | 119 | # Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately | COMMENT |
| LOW | tests/unit/crawlers/_parsel/test_parsel_crawler.py | 141 | # Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately | COMMENT |
| LOW⚡ | tests/unit/sessions/test_session_pool.py | 136 | # Check if the state is correctly persisted | COMMENT |
| LOW⚡ | tests/unit/sessions/test_session_pool.py | 141 | # Check if all the sessions are correctly persisted | COMMENT |
| LOW | …s/unit/storage_clients/_redis/test_redis_kvs_client.py | 59 | # Check if the records were created | COMMENT |
| LOW⚡ | …s/unit/storage_clients/_redis/test_redis_kvs_client.py | 120 | # Check if record content is valid JSON | COMMENT |
| LOW | …nit/storage_clients/_file_system/test_fs_kvs_client.py | 85 | # Check if the files were created | COMMENT |
| LOW⚡ | …nit/storage_clients/_file_system/test_fs_kvs_client.py | 130 | # Check if file content is valid JSON | COMMENT |
| LOW | …guides/code_examples/login_crawler/playwright_login.py | 47 | # Check if the session is available | COMMENT |
| LOW | …s/guides/code_examples/running_in_web_server/server.py | 57 | # Return the result | STRING |
| LOW⚡ | …code_examples/pydantic_ai_crawler/debugging_example.py | 40 | # Set concurrency to 1, which ensures only one request is processed at a time. | COMMENT |
| LOW⚡ | …code_examples/pydantic_ai_crawler/debugging_example.py | 44 | # Set abort_on_error to True to stop the crawl if an error occurs during | COMMENT |
| LOW | …les/code_examples/respect_robots_on_skipped_request.py | 22 | # Check if the request was skipped due to robots.txt rules | COMMENT |
| LOW | src/crawlee/_utils/sitemap.py | 428 | # Check if the first chunk is a valid gzip header | COMMENT |
| LOW | src/crawlee/_utils/file.py | 204 | # Set lineterminator to '\n' if not explicitly provided. This prevents double line endings on Windows. The | COMMENT |
| LOW | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 547 | # Check if the session is blocked based on the response content | COMMENT |
| LOW | src/crawlee/http_clients/_curl_impersonate.py | 307 | # Check if a session for the given proxy URL has already been created. | COMMENT |
| LOW⚡ | src/crawlee/project_template/hooks/pre_gen_project.py | 17 | # Check if package manager is available in PATH | COMMENT |
| LOW⚡ | src/crawlee/project_template/hooks/pre_gen_project.py | 22 | # Check if the package manager is executable | COMMENT |
| LOW⚡ | src/crawlee/project_template/hooks/pre_gen_project.py | 29 | # Check if the version matches the required regex | COMMENT |
| LOW | …/crawlee/storage_clients/_sql/_request_queue_client.py | 604 | # Check if there are any unhandled requests that are not blocked. | COMMENT |
| LOW | …/crawlee/storage_clients/_sql/_request_queue_client.py | 643 | # Check if there are any buffered updates that might change the pending count | COMMENT |
| LOW | …rawlee/storage_clients/_sql/_key_value_store_client.py | 399 | # Check if record exists | COMMENT |
| LOW | …rawlee/storage_clients/_redis/_request_queue_client.py | 508 | # Check if there are any requests in the queue. | COMMENT |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 342 | # Check if the request has already been handled. | COMMENT |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 351 | # Check if the request is already in progress. | COMMENT |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 500 | # Check if the request is in progress. | COMMENT |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 552 | # Check if the request is in progress. | COMMENT |
| LOW | …torage_clients/_file_system/_key_value_store_client.py | 372 | # Check if the KVS directory exists | COMMENT |
| LOW | …awlee/storage_clients/_memory/_request_queue_client.py | 234 | # Check if the request is in progress. | COMMENT |
| LOW | …awlee/storage_clients/_memory/_request_queue_client.py | 268 | # Check if the request is in progress. | COMMENT |
| LOW | src/crawlee/_autoscaling/snapshotter.py | 440 | # Check if the warning has been logged recently to avoid spamming | COMMENT |
| LOW⚡ | src/crawlee/request_loaders/_sitemap_request_loader.py | 416 | # Check if URL should be included | COMMENT |
| LOW⚡ | src/crawlee/request_loaders/_sitemap_request_loader.py | 423 | # Check if we have capacity in the queue | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …/playwright_crawler/browser_pool_page_hooks_example.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | …laywright_crawler/browser_pool_launch_hooks_example.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | …des/code_examples/http_headers/browser_page_headers.py | 15 | async def set_page_headers(context: PlaywrightPreNavCrawlingContext) -> None: | CODE |
| LOW | …mples/request_loaders/rl_basic_example_with_persist.py | 8 | logger = logging.getLogger(__name__) | CODE |
| LOW | …amples/request_loaders/sitemap_example_with_persist.py | 9 | logger = logging.getLogger(__name__) | CODE |
| LOW | tests/unit/conftest.py | 113 | def _set_crawler_log_level(pytestconfig: pytest.Config, monkeypatch: pytest.MonkeyPatch) -> None: | CODE |
| LOW | tests/unit/server.py | 183 | async def set_cookies(scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW | tests/unit/server.py | 435 | async def set_complex_cookies(_scope: dict[str, Any], _receive: Receive, send: Send) -> None: | CODE |
| LOW | …/playwright_crawler/browser_pool_page_hooks_example.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | …laywright_crawler/browser_pool_launch_hooks_example.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | …des/code_examples/http_headers/browser_page_headers.py | 15 | async def set_page_headers(context: PlaywrightPreNavCrawlingContext) -> None: | CODE |
| LOW | …mples/request_loaders/rl_basic_example_with_persist.py | 8 | logger = logging.getLogger(__name__) | CODE |
| LOW | …amples/request_loaders/sitemap_example_with_persist.py | 9 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/crawlee/configuration.py | 16 | __all__ = ['Configuration'] | CODE |
| LOW | src/crawlee/__init__.py | 10 | __all__ = [ | CODE |
| LOW | src/crawlee/proxy_configuration.py | 21 | __all__ = ['ProxyConfiguration', 'ProxyInfo'] | CODE |
| LOW | src/crawlee/errors.py | 10 | __all__ = [ | CODE |
| LOW | src/crawlee/router.py | 11 | __all__ = ['Router'] | CODE |
| LOW | src/crawlee/_service_locator.py | 46 | def set_configuration(self, configuration: Configuration) -> None: | CODE |
| LOW | src/crawlee/_service_locator.py | 76 | def set_event_manager(self, event_manager: EventManager) -> None: | CODE |
| LOW | src/crawlee/_service_locator.py | 106 | def set_storage_client(self, storage_client: StorageClient) -> None: | CODE |
| LOW | src/crawlee/statistics/__init__.py | 4 | __all__ = ['FinalStatistics', 'Statistics', 'StatisticsState'] | CODE |
| LOW | src/crawlee/fingerprint_suite/__init__.py | 6 | __all__ = [ | CODE |
| LOW | src/crawlee/storages/__init__.py | 5 | __all__ = [ | CODE |
| LOW | src/crawlee/storages/_dataset.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/crawlee/crawlers/__init__.py | 110 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_basic/__init__.py | 5 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_beautifulsoup/__init__.py | 15 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_parsel/__init__.py | 13 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_pydantic_ai/_types.py | 95 | def set_ai_usage(self, value: PydanticAiUsageStats) -> None: | CODE |
| LOW | src/crawlee/crawlers/_pydantic_ai/__init__.py | 31 | __all__ = [ | CODE |
| LOW | …c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py | 342 | def set_ai_usage(self, value: PydanticAiUsageStats) -> None: | CODE |
| LOW | src/crawlee/crawlers/_pydantic_ai/_base_extractor.py | 65 | def set_ai_usage(self, value: PydanticAiUsageStats) -> None: | CODE |
| LOW | src/crawlee/crawlers/_playwright/__init__.py | 17 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 593 | async def _update_cookies(self, page: Page, cookies: list[PlaywrightCookieParam]) -> None: | CODE |
| LOW | src/crawlee/crawlers/_stagehand/__init__.py | 24 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_http/__init__.py | 6 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_file_download/__init__.py | 4 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_adaptive_playwright/__init__.py | 24 | __all__ = [ | CODE |
| LOW | src/crawlee/crawlers/_abstract_http/__init__.py | 5 | __all__ = [ | CODE |
| LOW | src/crawlee/otel/__init__.py | 3 | __all__ = [ | CODE |
| LOW⚡ | src/crawlee/http_clients/_curl_impersonate.py | 55 | def update_cookies_from_curl(self, morsels: list[CurlMorsel]) -> None: | CODE |
| LOW⚡ | src/crawlee/http_clients/_curl_impersonate.py | 59 | def update_cookies_from_curl_changes(self, changes: list[bytes]) -> None: | CODE |
| LOW | src/crawlee/http_clients/__init__.py | 19 | __all__ = [ | CODE |
| LOW | src/crawlee/browsers/__init__.py | 28 | __all__ = [ | CODE |
| LOW | src/crawlee/sessions/_cookies.py | 230 | def set_cookies(self, cookie_dicts: list[CookieParam]) -> None: | CODE |
| LOW⚡ | src/crawlee/sessions/_cookies.py | 274 | def set_cookies_from_playwright_format(self, pw_cookies: list[PlaywrightCookieParam]) -> None: | CODE |
| LOW | src/crawlee/sessions/__init__.py | 5 | __all__ = ['CookieParam', 'Session', 'SessionCookies', 'SessionPool'] | CODE |
| LOW | src/crawlee/storage_clients/__init__.py | 19 | __all__ = [ | CODE |
| LOW | …rawlee/storage_clients/_sql/_key_value_store_client.py | 159 | async def set_value(self, *, key: str, value: Any, content_type: str | None = None) -> None: | CODE |
| LOW | src/crawlee/storage_clients/_sql/__init__.py | 6 | __all__ = ['SqlDatasetClient', 'SqlKeyValueStoreClient', 'SqlRequestQueueClient', 'SqlStorageClient'] | CODE |
| LOW | …wlee/storage_clients/_redis/_key_value_store_client.py | 138 | async def set_value(self, *, key: str, value: Any, content_type: str | None = None) -> None: | CODE |
| LOW | src/crawlee/storage_clients/_redis/__init__.py | 6 | __all__ = ['RedisDatasetClient', 'RedisKeyValueStoreClient', 'RedisRequestQueueClient', 'RedisStorageClient'] | CODE |
| LOW | …torage_clients/_file_system/_key_value_store_client.py | 302 | async def set_value(self, *, key: str, value: Any, content_type: str | None = None) -> None: | CODE |
| LOW | src/crawlee/storage_clients/_file_system/__init__.py | 6 | __all__ = [ | CODE |
| LOW | …lee/storage_clients/_memory/_key_value_store_client.py | 112 | async def set_value(self, *, key: str, value: Any, content_type: str | None = None) -> None: | CODE |
| LOW | src/crawlee/storage_clients/_memory/__init__.py | 6 | __all__ = [ | CODE |
| LOW | …awlee/storage_clients/_base/_key_value_store_client.py | 53 | async def set_value(self, *, key: str, value: Any, content_type: str | None = None) -> None: | CODE |
| LOW | src/crawlee/storage_clients/_base/__init__.py | 6 | __all__ = [ | CODE |
| LOW | src/crawlee/events/__init__.py | 15 | __all__ = [ | CODE |
| 3 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ing_web_archive/manual_archiving_playwright_crawler.py | 28 | except Exception as e: | CODE |
| LOW | tests/unit/_statistics/test_error_tracker.py | 34 | except Exception as e: # noqa:PERF203 | CODE |
| LOW | tests/unit/_statistics/test_error_tracker.py | 39 | except Exception as e: | CODE |
| LOW | tests/unit/_statistics/test_error_tracker.py | 69 | except Exception as e: # noqa:PERF203 | CODE |
| LOW⚡ | tests/unit/_statistics/test_error_tracker.py | 98 | except Exception as e: | CODE |
| LOW⚡ | tests/unit/_statistics/test_error_tracker.py | 110 | except Exception as e: | CODE |
| LOW⚡ | tests/unit/_statistics/test_error_tracker.py | 113 | except Exception as e: | CODE |
| LOW | tests/unit/storages/test_request_queue.py | 707 | except Exception: | CODE |
| LOW | tests/unit/crawlers/_parsel/test_parsel_crawler.py | 110 | except Exception as e: | CODE |
| LOW⚡ | …s/unit/crawlers/_playwright/test_playwright_crawler.py | 154 | except Exception as e: | CODE |
| LOW | …ing_web_archive/manual_archiving_playwright_crawler.py | 28 | except Exception as e: | CODE |
| LOW | src/crawlee/_cli.py | 258 | except Exception as exc: | CODE |
| LOW | src/crawlee/proxy_configuration.py | 186 | except Exception as e: | CODE |
| LOW | src/crawlee/statistics/_error_tracker.py | 104 | except Exception: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 224 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 235 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 374 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 503 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 658 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 677 | except Exception: | CODE |
| LOW | src/crawlee/_utils/sitemap.py | 749 | except Exception: | CODE |
| LOW | src/crawlee/_utils/wait.py | 40 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/wait.py | 85 | except Exception as e: | CODE |
| LOW | src/crawlee/_utils/recurring_task.py | 56 | except Exception: | CODE |
| LOW | src/crawlee/_utils/file.py | 61 | except Exception: | CODE |
| LOW | src/crawlee/_utils/requests.py | 82 | except Exception as exc: | CODE |
| LOW | src/crawlee/_utils/robots.py | 71 | except Exception as e: | CODE |
| LOW | src/crawlee/storages/_request_queue.py | 70 | except Exception: | STRING |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1215 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1280 | except Exception as secondary_error: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1302 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1317 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1601 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1630 | except Exception as internal_error: | CODE |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1642 | except Exception: # noqa: PERF203 | CODE |
| LOW | src/crawlee/crawlers/_basic/_context_pipeline.py | 50 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_context_pipeline.py | 107 | except Exception as e: | CODE |
| LOW | src/crawlee/crawlers/_basic/_context_pipeline.py | 118 | except Exception as e: | CODE |
| LOW | …rs/_playwright/_playwright_pre_nav_crawling_context.py | 39 | except Exception: | CODE |
| LOW | …rs/_playwright/_playwright_pre_nav_crawling_context.py | 44 | except Exception: | CODE |
| LOW | …s/_adaptive_playwright/_adaptive_playwright_crawler.py | 341 | except Exception as e: | CODE |
| LOW | src/crawlee/browsers/_browser_pool.py | 374 | except Exception: | CODE |
| LOW | src/crawlee/browsers/_stagehand_browser_controller.py | 187 | except Exception: | CODE |
| LOW | …rawlee/storage_clients/_redis/_request_queue_client.py | 437 | except Exception: | CODE |
| LOW | src/crawlee/events/_event_manager.py | 275 | except Exception: | CODE |
| LOW | src/crawlee/_autoscaling/autoscaled_pool.py | 248 | except Exception as exc: | CODE |
| LOW⚡ | src/crawlee/request_loaders/_sitemap_request_loader.py | 433 | except Exception: | CODE |
| LOW | src/crawlee/request_loaders/_sitemap_request_loader.py | 450 | except Exception: | CODE |
| LOW | src/crawlee/request_loaders/_request_manager_tandem.py | 87 | except Exception: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …sion-1.10/examples/code_examples/playwright_crawler.py | 56 | # browser page among other things. In this example, we log the URL being | COMMENT |
| HIGH | …1.10/examples/code_examples/respect_robots_txt_file.py | 19 | # In this example, 'https://news.ycombinator.com/login' will be skipped | COMMENT |
| HIGH | …rsion-0.6/examples/code_examples/playwright_crawler.py | 56 | # browser page among other things. In this example, we log the URL being | COMMENT |
| HIGH | …-0.6/examples/code_examples/respect_robots_txt_file.py | 19 | # In this example, 'https://news.ycombinator.com/login' will be skipped | COMMENT |
| HIGH | docs/examples/code_examples/playwright_crawler.py | 56 | # browser page among other things. In this example, we log the URL being | COMMENT |
| HIGH | docs/examples/code_examples/respect_robots_txt_file.py | 19 | # In this example, 'https://news.ycombinator.com/login' will be skipped | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/e2e/project_template/utils.py | 57 | CODE | |
| LOW | src/crawlee/_cli.py | 133 | CODE | |
| LOW | src/crawlee/fingerprint_suite/_browserforge_adapter.py | 56 | CODE | |
| LOW | src/crawlee/_utils/system.py | 40 | CODE | |
| LOW | src/crawlee/_utils/globs.py | 19 | CODE | |
| LOW | src/crawlee/_utils/globs.py | 76 | CODE | |
| LOW | src/crawlee/_utils/html.py | 513 | CODE | |
| LOW | src/crawlee/_utils/sitemap.py | 271 | CODE | |
| LOW | src/crawlee/_utils/sitemap.py | 380 | CODE | |
| LOW | src/crawlee/_utils/sitemap.py | 576 | CODE | |
| LOW | src/crawlee/_utils/sitemap.py | 698 | CODE | |
| LOW | src/crawlee/_utils/sitemap.py | 142 | CODE | |
| LOW | src/crawlee/_utils/wait.py | 49 | CODE | |
| LOW | src/crawlee/_utils/urls.py | 42 | CODE | |
| LOW | src/crawlee/storages/_storage_instance_manager.py | 40 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 693 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1109 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1482 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_context_pipeline.py | 82 | CODE | |
| LOW | src/crawlee/crawlers/_beautifulsoup/_utils.py | 20 | CODE | |
| LOW | src/crawlee/crawlers/_beautifulsoup/_utils.py | 38 | CODE | |
| LOW | src/crawlee/crawlers/_parsel/_utils.py | 16 | CODE | |
| LOW | src/crawlee/crawlers/_parsel/_utils.py | 34 | CODE | |
| LOW | …crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py | 176 | CODE | |
| LOW | …c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py | 656 | CODE | |
| LOW | src/crawlee/sessions/_cookies.py | 77 | CODE | |
| LOW | src/crawlee/sessions/_session.py | 94 | CODE | |
| LOW | src/crawlee/storage_clients/_sql/_storage_client.py | 112 | CODE | |
| LOW | …/crawlee/storage_clients/_sql/_request_queue_client.py | 207 | CODE | |
| LOW | …rawlee/storage_clients/_sql/_key_value_store_client.py | 159 | CODE | |
| LOW | …rawlee/storage_clients/_sql/_key_value_store_client.py | 223 | CODE | |
| LOW | …wlee/storage_clients/_redis/_key_value_store_client.py | 138 | CODE | |
| LOW | …wlee/storage_clients/_redis/_key_value_store_client.py | 199 | CODE | |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 153 | CODE | |
| LOW | …/storage_clients/_file_system/_request_queue_client.py | 321 | CODE | |
| LOW | …torage_clients/_file_system/_key_value_store_client.py | 90 | CODE | |
| LOW | …torage_clients/_file_system/_key_value_store_client.py | 223 | CODE | |
| LOW | …torage_clients/_file_system/_key_value_store_client.py | 302 | CODE | |
| LOW | …rawlee/storage_clients/_file_system/_dataset_client.py | 92 | CODE | |
| LOW | src/crawlee/_autoscaling/autoscaled_pool.py | 106 | CODE | |
| LOW | src/crawlee/_autoscaling/autoscaled_pool.py | 214 | CODE | |
| LOW | src/crawlee/request_loaders/_sitemap_request_loader.py | 353 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | website/generate_module_shortcuts.py | 54 | except ModuleNotFoundError: # noqa: PERF203 | CODE |
| HIGH | src/crawlee/_utils/system.py | 61 | except psutil.NoSuchProcess: | CODE |
| HIGH | src/crawlee/_utils/wait.py | 83 | except asyncio.CancelledError: | CODE |
| HIGH | src/crawlee/_utils/http.py | 43 | except ValueError: | CODE |
| HIGH | src/crawlee/_utils/http.py | 65 | except (ValueError, TypeError): | CODE |
| HIGH | src/crawlee/crawlers/_basic/_basic_crawler.py | 784 | except CancelledError: | CODE |
| HIGH | src/crawlee/crawlers/_basic/_context_pipeline.py | 46 | except StopAsyncIteration: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | src/crawlee/_types.py | 376 | CODE | |
| LOW | src/crawlee/_types.py | 392 | CODE | |
| LOW | src/crawlee/_types.py | 407 | CODE | |
| LOW | src/crawlee/_request.py | 244 | CODE | |
| LOW | src/crawlee/statistics/_statistics.py | 75 | CODE | |
| LOW | src/crawlee/statistics/_statistics.py | 131 | CODE | |
| LOW | src/crawlee/fingerprint_suite/_browserforge_adapter.py | 56 | CODE | |
| LOW | src/crawlee/storages/_dataset.py | 150 | CODE | |
| LOW | src/crawlee/storages/_dataset.py | 201 | CODE | |
| LOW | src/crawlee/storages/_dataset.py | 264 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 280 | CODE | |
| LOW | src/crawlee/crawlers/_basic/_basic_crawler.py | 1064 | CODE | |
| LOW | …crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py | 71 | CODE | |
| LOW | …c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py | 290 | CODE | |
| LOW | …c/crawlee/crawlers/_pydantic_ai/_skeleton_distiller.py | 64 | CODE | |
| LOW | src/crawlee/crawlers/_playwright/_playwright_crawler.py | 113 | CODE | |
| LOW | src/crawlee/crawlers/_stagehand/_stagehand_crawler.py | 94 | CODE | |
| LOW | src/crawlee/sessions/_session.py | 36 | CODE | |
| LOW | …/crawlee/storage_clients/_sql/_request_queue_client.py | 682 | CODE | |
| LOW | src/crawlee/storage_clients/_sql/_dataset_client.py | 163 | CODE | |
| LOW | src/crawlee/storage_clients/_sql/_dataset_client.py | 210 | CODE | |
| LOW | src/crawlee/storage_clients/_sql/_dataset_client.py | 243 | CODE | |
| LOW | src/crawlee/storage_clients/_redis/_dataset_client.py | 149 | CODE | |
| LOW | src/crawlee/storage_clients/_redis/_dataset_client.py | 235 | CODE | |
| LOW | …rawlee/storage_clients/_file_system/_dataset_client.py | 240 | CODE | |
| LOW | …rawlee/storage_clients/_file_system/_dataset_client.py | 339 | CODE | |
| LOW | src/crawlee/storage_clients/_memory/_dataset_client.py | 135 | CODE | |
| LOW | src/crawlee/storage_clients/_memory/_dataset_client.py | 194 | CODE | |
| LOW | src/crawlee/storage_clients/_base/_dataset_client.py | 53 | CODE | |
| LOW | src/crawlee/storage_clients/_base/_dataset_client.py | 74 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …rsion-1.10/introduction/code_examples/07_final_code.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | …version-1.10/introduction/code_examples/06_scraping.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | …-1.10/introduction/code_examples/05_crawling_detail.py | 13 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_context.py | 10 | # to facilitate working with the parsed document. | COMMENT |
| LOW | …ersion-0.6/introduction/code_examples/07_final_code.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | …/version-0.6/introduction/code_examples/06_scraping.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | …n-0.6/introduction/code_examples/05_crawling_detail.py | 13 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| MEDIUM | …daptive_playwright/test_adaptive_playwright_crawler.py | 52 | # Generous wait for the JS-injected element. Resolves as soon as it appears, so it stays robust on | COMMENT |
| LOW | docs/introduction/code_examples/07_final_code.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | docs/introduction/code_examples/06_scraping.py | 16 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| LOW | docs/introduction/code_examples/05_crawling_detail.py | 13 | # We're not processing detail pages yet, so we just pass. | COMMENT |
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_context.py | 10 | # to facilitate working with the parsed document. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | tests/unit/test_throttling_request_manager.py | 287 | # ── Core Throttling Tests ───────────────────────────────── | COMMENT |
| MEDIUM⚡ | tests/unit/test_throttling_request_manager.py | 487 | # ── Crawl-Delay Integration Tests ───────────────────────── | COMMENT |
| MEDIUM⚡ | tests/unit/test_throttling_request_manager.py | 1451 | # ── Utility Tests ────────────────────────────────────── | COMMENT |
| MEDIUM | tests/unit/test_throttling_request_manager.py | 117 | # ── Request Routing Tests ───────────────────────────────── | COMMENT |
| MEDIUM | tests/unit/test_throttling_request_manager.py | 542 | # ── Fetch Scheduling Tests ──────────────────────────── | COMMENT |
| MEDIUM | tests/unit/test_throttling_request_manager.py | 659 | # ── Delegation Tests ──────────────────────────────────── | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_context.py | 9 | # Custom context for Selectolax parser, you can add your own methods here | COMMENT |
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_crawler.py | 20 | # Custom crawler using custom context, It is optional and you can use | COMMENT |
| LOW | …e2e/project_template/test_static_crawlers_templates.py | 15 | # To run these tests locally, make sure you have apify-cli installed and available in the path. | COMMENT |
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_context.py | 9 | # Custom context for Selectolax parser, you can add your own methods here | COMMENT |
| MEDIUM | …ides/code_examples/http_crawlers/selectolax_crawler.py | 20 | # Custom crawler using custom context, It is optional and you can use | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | website/docusaurus.config.js | 121 | }, | COMMENT |
| LOW | website/docusaurus.config.js | 141 | // to: '/docs/introduction', | COMMENT |
| LOW | website/sidebars.js | 81 | type: 'autogenerated', | COMMENT |
| LOW | website/src/components/Highlights.jsx | 61 | // or plug an adapter for your DB. Your URLs are <a href="https://crawlee.dev/docs/guides/request-storag | COMMENT |
| LOW | …emplate/{{cookiecutter.project_name}}/requirements.txt | 1 | # % if cookiecutter.crawler_type == 'playwright-camoufox' | COMMENT |
| LOW | …_template/{{cookiecutter.project_name}}/pyproject.toml | 1 | # % if cookiecutter.crawler_type.startswith('adaptive-') | COMMENT |
| LOW | …oject_name}}/{{cookiecutter.__package_name}}/routes.py | 1 | # % if cookiecutter.crawler_type.startswith('playwright') | COMMENT |
| LOW | src/crawlee/project_template/hooks/post_gen_project.py | 41 | COMMENT | |
| LOW | src/crawlee/project_template/templates/main.py | 1 | # % if cookiecutter.enable_apify_integration | COMMENT |
| LOW | src/crawlee/project_template/templates/main.py | 21 | http_client=HttpxHttpClient(), | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ples/code_examples/fill_and_submit_web_form_request.py | 15 | 'custname': 'John Doe', | CODE |
| LOW | …ples/code_examples/fill_and_submit_web_form_crawler.py | 25 | 'custname': 'John Doe', | CODE |
| LOW | …ples/code_examples/fill_and_submit_web_form_request.py | 15 | 'custname': 'John Doe', | CODE |
| LOW | …ples/code_examples/fill_and_submit_web_form_crawler.py | 25 | 'custname': 'John Doe', | CODE |
| LOW | tests/unit/_utils/test_html.py | 317 | <select name="placeholder"><option value="" disabled selected>Pick</option><option value="p">P</option></select> | CODE |
| LOW | tests/unit/crawlers/_http/test_http_crawler.py | 33 | 'custname': 'John Doe', | CODE |
| LOW | …ples/code_examples/fill_and_submit_web_form_request.py | 15 | 'custname': 'John Doe', | CODE |
| LOW | …ples/code_examples/fill_and_submit_web_form_crawler.py | 25 | 'custname': 'John Doe', | CODE |
| LOW | …es/code_examples/fill_and_submit_web_form_automated.py | 15 | 'custname': 'John Doe', | CODE |
| LOW | …crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py | 35 | 'placeholder', | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/unit/crawlers/_basic/test_context_pipeline.py | 150 | CODE | |
| MEDIUM | …awlee/storage_clients/_base/_key_value_store_client.py | 79 | CODE | |
| MEDIUM | src/crawlee/storage_clients/_base/_dataset_client.py | 93 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/unit/server_endpoints.py | 127 | if (loading || page >= 3) return; | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …/code_examples/http_crawlers/selectolax_crawler_run.py | 12 | async def handle_request(context: SelectolaxLexborContext) -> None: | CODE |
| LOW | …code_examples/http_crawlers/selectolax_adaptive_run.py | 19 | async def handle_request(context: AdaptivePlaywrightCrawlingContext) -> None: | CODE |
| LOW | …/code_examples/http_crawlers/selectolax_crawler_run.py | 12 | async def handle_request(context: SelectolaxLexborContext) -> None: | CODE |
| LOW | …code_examples/http_crawlers/selectolax_adaptive_run.py | 19 | async def handle_request(context: AdaptivePlaywrightCrawlingContext) -> None: | CODE |