apify/crawlee-python

32.9

Adjusted Score

32.9

Raw Score

100%

Time Factor

2026-07-14

Last Push

9.3K

Stars

Python

Language

105.1K

Lines of Code

976

Files

2.1K

Pattern Hits

2026-07-14

Scan Date

0.15

HC Hit Rate

What These Metrics Mean

Adjusted Score: Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score: The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor: The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits: Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate: High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files: Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 143MEDIUM 373LOW 1571

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 2087 distinct pattern matches across 19 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Self-Referential Comments358 hits · 1081 pts

Severity	File	Line	Snippet	Context
MEDIUM	…d_docs/version-1.8/introduction/code_examples/02_bs.py	17	# Define a request handler and attach it to the crawler using the decorator.	COMMENT
MEDIUM	…ides/code_examples/request_router/router_middleware.py	10	# Create a custom router instance	COMMENT
MEDIUM	…_examples/request_router/custom_router_default_only.py	8	# Create a custom router instance	COMMENT
MEDIUM	…code_examples/request_router/failed_request_handler.py	7	# Create a crawler instance with retry settings	COMMENT
MEDIUM	…8/guides/code_examples/request_router/error_handler.py	11	# Create a crawler instance	COMMENT
MEDIUM	…code_examples/request_router/simple_default_handler.py	7	# Create a crawler instance	COMMENT
MEDIUM	…code_examples/request_router/basic_request_handlers.py	9	# Create a custom router instance	COMMENT
MEDIUM	…code_examples/request_router/basic_request_handlers.py	12	# Define the default handler (fallback for requests without specific labels)	COMMENT
MEDIUM	…code_examples/request_router/basic_request_handlers.py	31	# Define a handler for category pages	COMMENT
MEDIUM	…code_examples/request_router/basic_request_handlers.py	53	# Define a handler for product detail pages	COMMENT
MEDIUM	…ples/storages/dataset_with_crawler_explicit_example.py	12	# Create a new crawler (it can be any subclass of BasicCrawler).	COMMENT
MEDIUM	…ples/storages/dataset_with_crawler_explicit_example.py	15	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…des/code_examples/storages/kvs_with_crawler_example.py	7	# Create a new Playwright crawler.	COMMENT
MEDIUM	…des/code_examples/storages/kvs_with_crawler_example.py	10	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…code_examples/storages/dataset_with_crawler_example.py	7	# Create a new crawler (it can be any subclass of BasicCrawler).	COMMENT
MEDIUM	…code_examples/storages/dataset_with_crawler_example.py	10	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…examples/storages/kvs_with_crawler_explicit_example.py	12	# Create a new Playwright crawler.	COMMENT
MEDIUM	…examples/storages/kvs_with_crawler_explicit_example.py	15	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…ides/code_examples/storages/rq_with_crawler_example.py	7	# Create a new crawler (it can be any subclass of BasicCrawler). Request queue is	COMMENT
MEDIUM	…ides/code_examples/storages/rq_with_crawler_example.py	11	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…_examples/storages/rq_with_crawler_explicit_example.py	15	# Create a new crawler (it can be any subclass of BasicCrawler) and pass the request	COMMENT
MEDIUM	…_examples/storages/rq_with_crawler_explicit_example.py	19	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…s/code_examples/http_crawlers/beautifulsoup_example.py	7	# Create a BeautifulSoupCrawler instance	COMMENT
MEDIUM	…s/code_examples/http_crawlers/beautifulsoup_example.py	13	# Define the default request handler	COMMENT
MEDIUM	…1.8/guides/code_examples/http_crawlers/http_example.py	8	# Create an HttpCrawler instance - no automatic parsing	COMMENT
MEDIUM	…1.8/guides/code_examples/http_crawlers/http_example.py	14	# Define the default request handler	COMMENT
MEDIUM	…8/guides/code_examples/http_crawlers/parsel_example.py	7	# Create a ParselCrawler instance	COMMENT
MEDIUM	…8/guides/code_examples/http_crawlers/parsel_example.py	13	# Define the default request handler	COMMENT
MEDIUM⚡	…reating_web_archive/manual_archiving_parsel_crawler.py	37	# Create a WARC archive file a prepare the writer.	COMMENT
MEDIUM⚡	…reating_web_archive/manual_archiving_parsel_crawler.py	42	# Create a WARC info record to store metadata about the archive.	COMMENT
MEDIUM⚡	…reating_web_archive/manual_archiving_parsel_crawler.py	50	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…ing_web_archive/manual_archiving_playwright_crawler.py	52	# Create a WARC archive file a prepare the writer.	COMMENT
MEDIUM	…ing_web_archive/manual_archiving_playwright_crawler.py	57	# Create a WARC info record to store metadata about the archive.	COMMENT
MEDIUM	…n-1.8/guides/code_examples/login_crawler/http_login.py	68	# Create a POST request to the authentication API endpoint	COMMENT
MEDIUM	…des/code_examples/http_clients/parsel_impit_example.py	21	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…des/code_examples/http_clients/parsel_httpx_example.py	20	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…amples/http_clients/parsel_curl_impersonate_example.py	20	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…des/code_examples/error_handling/handle_proxy_error.py	27	# Create a new `Request` with a modified key to avoid deduplication	COMMENT
MEDIUM	…8/guides/code_examples/error_handling/disable_retry.py	10	# Create a parsing error for demonstration	COMMENT
MEDIUM	…void_blocking/playwright_with_fingerprint_generator.py	10	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…xamples/avoid_blocking/playwright_with_cloakbrowser.py	56	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…examples/playwright_crawler/multiple_launch_example.py	8	# Create a plugin for each required browser.	COMMENT
MEDIUM	…/storage_clients/redis_storage_client_basic_example.py	4	# Create a new instance of storage client using connection string.	COMMENT
MEDIUM	…ge_clients/file_system_storage_client_basic_example.py	4	# Create a new instance of storage client.	COMMENT
MEDIUM	…ts/file_system_storage_client_configuration_example.py	5	# Create a new instance of storage client.	COMMENT
MEDIUM	…ts/file_system_storage_client_configuration_example.py	8	# Create a configuration with custom settings.	COMMENT
MEDIUM	…_clients/redis_storage_client_configuration_example.py	7	# Create a new instance of storage client using a Redis client with custom settings.	COMMENT
MEDIUM	…_clients/redis_storage_client_configuration_example.py	20	# Create a configuration with custom settings.	COMMENT
MEDIUM	…es/storage_clients/sql_storage_client_basic_example.py	6	# Create a new instance of storage client.	COMMENT
MEDIUM	…ge_clients/sql_storage_client_configuration_example.py	9	# Create a new instance of storage client.	COMMENT
MEDIUM	…ge_clients/sql_storage_client_configuration_example.py	24	# Create a configuration with custom settings.	COMMENT
MEDIUM	…storage_clients/memory_storage_client_basic_example.py	4	# Create a new instance of storage client.	COMMENT
MEDIUM	…des/code_examples/proxy_management/tiers_bs_example.py	8	# Create a ProxyConfiguration object and pass it to the crawler.	COMMENT
MEDIUM	…des/code_examples/proxy_management/tiers_bs_example.py	28	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…des/code_examples/proxy_management/tiers_pw_example.py	8	# Create a ProxyConfiguration object and pass it to the crawler.	COMMENT
MEDIUM	…des/code_examples/proxy_management/tiers_pw_example.py	28	# Define the default request handler, which will be called for every request.	COMMENT
MEDIUM	…s/code_examples/proxy_management/session_bs_example.py	8	# Create a ProxyConfiguration object and pass it to the crawler.	COMMENT
MEDIUM	…s/code_examples/proxy_management/session_pw_example.py	8	# Create a ProxyConfiguration object and pass it to the crawler.	COMMENT
MEDIUM	…de_examples/proxy_management/integration_pw_example.py	8	# Create a ProxyConfiguration object and pass it to the crawler.	COMMENT
MEDIUM	…de_examples/proxy_management/integration_pw_example.py	17	# Define the default request handler, which will be called for every request.	COMMENT
298 more matches not shown…

Hyper-Verbose Identifiers887 hits · 900 pts

Severity	File	Line	Snippet	Context
LOW	…ides/code_examples/http_crawlers/selectolax_context.py	25	def from_parsed_http_crawling_context(	CODE
LOW	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	13	def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None:	CODE
LOW	…/examples/code_examples/adaptive_playwright_crawler.py	22	async def request_handler_for_label(	CODE
LOW	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	13	def stop_crawler_if_url_visited(context: BasicCrawlingContext) -> None:	CODE
LOW	…/examples/code_examples/adaptive_playwright_crawler.py	21	async def request_handler_for_label(	CODE
LOW	website/src/theme/Navbar/MobileSidebar/Layout/index.js	5	export default function NavbarMobileSidebarLayout({ header, primaryMenu, secondaryMenu }) {	CODE
LOW	website/src/theme/Navbar/MobileSidebar/Header/index.js	29	export default function NavbarMobileSidebarHeader() {	CODE
LOW	…/src/theme/NavbarItem/DocsVersionDropdownNavbarItem.js	93	export default function DocsVersionDropdownNavbarItem({	CODE
LOW	tests/unit/conftest.py	84	def _isolate_test_environment(prepare_test_env: Callable[[], None]) -> None:	CODE
LOW	tests/unit/server.py	297	async def generic_response_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW	tests/unit/server.py	305	async def problematic_links_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW	tests/unit/server.py	496	async def resource_loading_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW	tests/unit/server.py	514	async def start_enqueue_non_href_endpoint(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	23	def test_configuration_rejects_out_of_range_max_used_cpu_ratio(invalid_value: float) -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	29	def test_configuration_rejects_out_of_range_max_used_memory_ratio(invalid_value: float) -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	35	def test_configuration_accepts_in_range_max_used_cpu_ratio(valid_value: float) -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	40	def test_configuration_accepts_in_range_max_used_memory_ratio(valid_value: float) -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	44	def test_global_configuration_works() -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	53	def test_global_configuration_works_reversed() -> None:	CODE
LOW⚡	tests/unit/test_configuration.py	62	async def test_storage_not_persisted_when_non_persistable_storage_used(tmp_path: Path, server_url: URL) -> None:	CODE
LOW	tests/unit/test_configuration.py	78	async def test_storage_persisted_with_explicit_statistics_with_persistable_storage(	CODE
LOW	tests/unit/test_configuration.py	103	async def test_storage_persisted_when_enabled(tmp_path: Path, server_url: URL) -> None:	CODE
LOW	tests/unit/test_router.py	37	async def test_router_no_default_handler() -> None:	CODE
LOW	tests/unit/test_router.py	51	async def test_router_default_handler_invoked() -> None:	CODE
LOW	tests/unit/test_router.py	70	async def test_router_specific_handler_invoked() -> None:	CODE
LOW	tests/unit/test_router.py	95	async def test_router_handler_not_nullified() -> None:	CODE
LOW	tests/unit/test_router.py	105	async def test_router_multi_labelled_handler() -> None:	CODE
LOW	tests/unit/test_router.py	121	async def test_router_use_middleware() -> None:	CODE
LOW	tests/unit/test_router.py	149	async def test_router_use_middleware_with_label() -> None:	CODE
LOW	tests/unit/test_router.py	184	async def test_router_middleware_order_execution() -> None:	CODE
LOW	tests/unit/test_router.py	208	async def test_router_middleware_exception_interrupts_chain() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	12	def test_default_configuration() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	18	def test_custom_configuration() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	25	def test_configuration_overwrite_not_possible() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	34	def test_configuration_conflict() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	42	def test_default_event_manager() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	47	def test_custom_event_manager() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	54	def test_event_manager_overwrite_not_possible() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	63	def test_event_manager_conflict() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	71	def test_default_storage_client() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	76	def test_custom_storage_client() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	83	def test_storage_client_overwrite_not_possible() -> None:	CODE
LOW⚡	tests/unit/test_service_locator.py	92	def test_storage_client_conflict() -> None:	CODE
LOW	tests/unit/test_log_config.py	40	def test_formatting_with_exception() -> None:	CODE
LOW	tests/unit/test_log_config.py	54	def test_formatter_without_name() -> None:	CODE
LOW	tests/unit/test_cli.py	57	def test_create_interactive_non_default_template(mock_cookiecutter: Mock, monkeypatch: pytest.MonkeyPatch) -> None:	CODE
LOW	tests/unit/test_cli.py	91	def test_create_non_interactive(mock_cookiecutter: Mock) -> None:	CODE
LOW	tests/unit/test_cli.py	125	def test_create_existing_folder(	CODE
LOW	tests/unit/test_cli.py	174	def test_create_existing_folder_interactive(	CODE
LOW	tests/unit/test_cli.py	214	def test_create_existing_folder_interactive_multiple_attempts(	CODE
LOW	tests/unit/test_cli.py	266	def test_import_error_handled(optional_module_name: str) -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	150	async def test_record_domain_delay_returns_false_for_unconfigured_domain(	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	158	async def test_different_domains_independent(manager: ThrottlingRequestManager[RequestQueue]) -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	243	async def test_crawl_delay_integration(manager: ThrottlingRequestManager[RequestQueue]) -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	252	async def test_crawl_delay_throttles_after_dispatch(manager: ThrottlingRequestManager[RequestQueue]) -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	265	async def test_fetch_from_unthrottled_sub_manager(	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	533	def test_parse_retry_after_none_value() -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	537	def test_parse_retry_after_empty_string() -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	541	def test_parse_retry_after_integer_seconds() -> None:	CODE
LOW⚡	tests/unit/test_throttling_request_manager.py	546	def test_parse_retry_after_zero_seconds() -> None:	CODE
827 more matches not shown…

Cross-File Repetition111 hits · 555 pts

Severity	File	Line	Snippet	Context
HIGH	…reating_web_archive/manual_archiving_parsel_crawler.py	0	helper function for archiving response in warc format.	STRING
HIGH	…ing_web_archive/manual_archiving_playwright_crawler.py	0	helper function for archiving response in warc format.	STRING
HIGH	…reating_web_archive/manual_archiving_parsel_crawler.py	0	helper function for archiving response in warc format.	STRING
HIGH	…ing_web_archive/manual_archiving_playwright_crawler.py	0	helper function for archiving response in warc format.	STRING
HIGH	…_examples/playwright_crawler_adaptive/pre_nav_hooks.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…_examples/playwright_crawler_adaptive/pre_nav_hooks.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…_examples/playwright_crawler_adaptive/pre_nav_hooks.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed both in static sub crawler and playwright sub crawler. trying to access `context.page` in this hook would	STRING
HIGH	…8/guides/code_examples/running_in_web_server/server.py	0	<!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter	STRING
HIGH	…6/guides/code_examples/running_in_web_server/server.py	0	<!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter	STRING
HIGH	…s/guides/code_examples/running_in_web_server/server.py	0	<!doctype html> <html> <body> <h1>scraper server</h1> <p>to scrape some page, visit "scrape" endpoint with url parameter	STRING
HIGH	…ples/pydantic_ai_crawler/selector_extractor_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…amples/pydantic_ai_crawler/custom_distiller_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…code_examples/pydantic_ai_crawler/debugging_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…des/code_examples/pydantic_ai_crawler/basic_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…de_examples/pydantic_ai_crawler/usage_limit_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…ples/pydantic_ai_crawler/selector_extractor_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…amples/pydantic_ai_crawler/custom_distiller_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…code_examples/pydantic_ai_crawler/debugging_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…des/code_examples/pydantic_ai_crawler/basic_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…de_examples/pydantic_ai_crawler/usage_limit_example.py	0	model representing the extracted data for an article.	STRING
HIGH	…8/deployment/code_examples/google/cloud_run_example.py	0	the crawler entry point that will be called when the http endpoint is accessed.	STRING
HIGH	…6/deployment/code_examples/google/cloud_run_example.py	0	the crawler entry point that will be called when the http endpoint is accessed.	STRING
HIGH	…s/deployment/code_examples/google/cloud_run_example.py	0	the crawler entry point that will be called when the http endpoint is accessed.	STRING
HIGH	…8/deployment/code_examples/google/cloud_run_example.py	0	default request handler that processes each page during crawling.	STRING
HIGH	…6/deployment/code_examples/google/cloud_run_example.py	0	default request handler that processes each page during crawling.	STRING
HIGH	…s/deployment/code_examples/google/cloud_run_example.py	0	default request handler that processes each page during crawling.	STRING
HIGH	…ples/code_examples/playwright_crawler_with_camoufox.py	0	example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin.	STRING
HIGH	…ples/code_examples/playwright_crawler_with_camoufox.py	0	example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin.	STRING
HIGH	…ples/code_examples/playwright_crawler_with_camoufox.py	0	example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin.	STRING
HIGH	…project_template/templates/main_playwright_camoufox.py	0	example browser plugin that uses camoufox browser, but otherwise keeps the functionality of playwrightbrowserplugin.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	stop crawler once specific url is visited. example of guard condition to stop the crawler.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	stop crawler once specific url is visited. example of guard condition to stop the crawler.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	stop crawler once specific url is visited. example of guard condition to stop the crawler.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	add requests to the queue after some time. can be done by external code.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	add requests to the queue after some time. can be done by external code.	STRING
HIGH	…ples/code_examples/beautifulsoup_crawler_keep_alive.py	0	add requests to the queue after some time. can be done by external code.	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed only in playwright sub crawler. it is safe to access `page` object.	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed only in playwright sub crawler. it is safe to access `page` object.	STRING
HIGH	…/examples/code_examples/adaptive_playwright_crawler.py	0	hook executed only in playwright sub crawler. it is safe to access `page` object.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test that open() raises an error when both id and name are provided.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test that open() raises an error when both id and name are provided.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test that open() raises an error when both id and name are provided.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test that providing both alias and id raises error.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test that providing both alias and id raises error.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test that providing both alias and id raises error.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test that providing both alias and name raises error.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test that providing both alias and name raises error.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test that providing both alias and name raises error.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test that conflicts between named and alias storages are detected.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test that conflicts between named and alias storages are detected.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test that conflicts between named and alias storages are detected.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test purge behavior when purge_on_start=true: named storages retain data, unnamed storages are purged.	STRING
HIGH	tests/unit/storages/test_request_queue.py	0	test purge behavior when purge_on_start=false: all storages retain data regardless of type.	STRING
HIGH	tests/unit/storages/test_dataset.py	0	test purge behavior when purge_on_start=false: all storages retain data regardless of type.	STRING
HIGH	tests/unit/storages/test_key_value_store.py	0	test purge behavior when purge_on_start=false: all storages retain data regardless of type.	STRING
51 more matches not shown…

Unused Imports429 hits · 410 pts

Severity	File	Line	Context
LOW	website/generate_module_shortcuts.py	3	CODE
LOW	…1.8/introduction/code_examples/03_transform_request.py	1	CODE
LOW	…uides/code_examples/http_crawlers/selectolax_parser.py	1	CODE
LOW	…ides/code_examples/http_crawlers/selectolax_crawler.py	1	CODE
LOW	…/playwright_crawler/browser_pool_page_hooks_example.py	1	CODE
LOW	…laywright_crawler/browser_pool_launch_hooks_example.py	1	CODE
LOW	…mples/storage_clients/custom_storage_client_example.py	1	CODE
LOW	…8/guides/code_examples/running_in_web_server/server.py	1	CODE
LOW	…n-1.8/examples/code_examples/configure_json_logging.py	1	CODE
LOW	…0.6/introduction/code_examples/03_transform_request.py	1	CODE
LOW	…6/guides/code_examples/running_in_web_server/server.py	1	CODE
LOW	…n-0.6/examples/code_examples/configure_json_logging.py	1	CODE
LOW	tests/unit/conftest.py	1	CODE
LOW	tests/unit/server.py	1	CODE
LOW	tests/unit/test_configuration.py	1	CODE
LOW	tests/unit/test_router.py	1	CODE
LOW	tests/unit/test_service_locator.py	1	CODE
LOW	tests/unit/test_log_config.py	1	CODE
LOW	tests/unit/utils.py	1	CODE
LOW	tests/unit/test_cli.py	1	CODE
LOW	tests/unit/test_throttling_request_manager.py	3	CODE
LOW	tests/unit/fingerprint_suite/test_header_generator.py	1	CODE
LOW	tests/unit/_utils/test_measure_time.py	1	CODE
LOW	tests/unit/_utils/test_retry.py	1	CODE
LOW	tests/unit/_utils/test_crypto.py	1	CODE
LOW	tests/unit/_utils/test_console.py	1	CODE
LOW	tests/unit/_utils/test_time.py	1	CODE
LOW	tests/unit/_utils/test_recurring_task.py	1	CODE
LOW	tests/unit/_utils/test_robots.py	1	CODE
LOW	tests/unit/_utils/test_log.py	1	CODE
LOW	tests/unit/_utils/test_sitemap.py	25	CODE
LOW	tests/unit/_utils/test_system.py	1	CODE
LOW	tests/unit/_utils/test_requests.py	1	CODE
LOW	tests/unit/_utils/test_file.py	1	CODE
LOW	tests/unit/_utils/test_globs.py	1	CODE
LOW	tests/unit/_utils/test_byte_size.py	1	CODE
LOW	tests/unit/_utils/test_timedelta_ms.py	1	CODE
LOW	tests/unit/_utils/test_urls.py	1	CODE
LOW	tests/unit/_utils/test_html_to_text.py	1	CODE
LOW	tests/unit/_statistics/test_persistence.py	1	CODE
LOW	tests/unit/_statistics/test_request_max_duration.py	1	CODE
LOW	tests/unit/_statistics/test_periodic_logging.py	1	CODE
LOW	tests/unit/storages/test_request_manager_tandem.py	1	CODE
LOW	tests/unit/storages/conftest.py	1	CODE
LOW	tests/unit/storages/test_request_queue.py	1	CODE
LOW	tests/unit/storages/test_dataset.py	1	CODE
LOW	tests/unit/storages/test_key_value_store.py	1	CODE
LOW	tests/unit/crawlers/_basic/test_context_pipeline.py	1	CODE
LOW	tests/unit/crawlers/_basic/test_basic_crawler.py	2	CODE
LOW	…/crawlers/_beautifulsoup/test_beautifulsoup_crawler.py	1	CODE
LOW	tests/unit/crawlers/_parsel/test_parsel_crawler.py	1	CODE
LOW	…ts/unit/crawlers/_pydantic_ai/test_direct_extractor.py	1	CODE
LOW	…nit/crawlers/_pydantic_ai/test_clean_html_distiller.py	1	CODE
LOW	…/unit/crawlers/_pydantic_ai/test_skeleton_distiller.py	1	CODE
LOW	…/unit/crawlers/_pydantic_ai/test_selector_extractor.py	1	CODE
LOW	…unit/crawlers/_pydantic_ai/test_pydantic_ai_crawler.py	1	CODE
LOW	…s/unit/crawlers/_playwright/test_playwright_crawler.py	1	CODE
LOW	…sts/unit/crawlers/_stagehand/test_stagehand_crawler.py	1	CODE
LOW	tests/unit/crawlers/_http/test_http_crawler.py	1	CODE
LOW	…daptive_playwright/test_adaptive_playwright_crawler.py	1	CODE
369 more matches not shown…

Docstring Block Structure25 hits · 125 pts

Severity	File	Line	Snippet	Context
HIGH	src/crawlee/_utils/file.py	64	Resolve a storage subdirectory inside a base directory. Joins `subdirectory` onto `base_dir` and verifies that the	STRING
HIGH	src/crawlee/_utils/context.py	12	Ensure the (async) context manager is initialized before executing the method. This decorator checks if the calling	STRING
HIGH	src/crawlee/storages/_storage_instance_manager.py	93	Open a storage instance with caching support. Args: cls: The storage class to instantiate.	STRING
HIGH	src/crawlee/crawlers/_playwright/_playwright_crawler.py	378	Execute an HTTP request utilizing the `BrowserPool` and the `Playwright` library. Args: context: Th	STRING
HIGH	src/crawlee/crawlers/_playwright/_playwright_crawler.py	523	Validate the HTTP status code and raise appropriate exceptions if needed. Args: context: The curren	STRING
HIGH	src/crawlee/crawlers/_playwright/_playwright_crawler.py	551	Try to detect if the request is blocked based on the response content. Args: context: The current c	STRING
HIGH	…wlee/crawlers/_abstract_http/_abstract_http_crawler.py	292	Validate the HTTP status code and raise appropriate exceptions if needed. Args: context: The curren	STRING
HIGH	…wlee/crawlers/_abstract_http/_abstract_http_crawler.py	319	Try to detect if the request is blocked based on the parsed response content. Args: context: The cu	STRING
HIGH	src/crawlee/http_clients/_base.py	109	Perform the crawling for a given request. This method is called from `crawler.run()`. Args:	STRING
HIGH	src/crawlee/http_clients/_base.py	139	Send an HTTP request via the client. This method is called from `context.send_request()` helper. Args:	STRING
HIGH	src/crawlee/http_clients/_base.py	171	Stream an HTTP request via the client. This method should be used for downloading potentially large data where	STRING
HIGH	src/crawlee/http_clients/_curl_impersonate.py	310	Convert from Crawlee HTTP method to curl-cffi HTTP method. Args: method: Crawlee HTTP method.	STRING
HIGH	src/crawlee/browsers/_browser_controller.py	73	Create a new page with the given context options. Args: browser_new_context_options: Keyword argume	STRING
HIGH	src/crawlee/browsers/_playwright_browser_controller.py	158	Create a new page with the given context options. Args: browser_new_context_options: Keyword argume	STRING
HIGH	src/crawlee/browsers/_stagehand_types.py	119	Extract structured data from the page using natural language. Argument `page` is automatically set. Args:	STRING
HIGH	src/crawlee/browsers/_stagehand_types.py	131	Run an autonomous multi-step AI agent on the page. Argument `page` is automatically set. Args: **kw	STRING
HIGH	…/crawlee/storage_clients/_sql/_request_queue_client.py	133	Open an existing request queue or create a new one. This method first tries to find an existing queue by ID or	STRING
HIGH	…rawlee/storage_clients/_sql/_key_value_store_client.py	95	Open or create a SQL key-value store client. This method attempts to open an existing key-value store from the	STRING
HIGH	src/crawlee/storage_clients/_sql/_dataset_client.py	93	Open an existing dataset or create a new one. Args: id: The ID of the dataset to open. If provided,	STRING
HIGH	…/storage_clients/_file_system/_request_queue_client.py	162	Open or create a file system request queue client. This method attempts to open an existing request queue from	STRING
HIGH	…torage_clients/_file_system/_key_value_store_client.py	98	Open or create a file system key-value store client. This method attempts to open an existing key-value store f	STRING
HIGH	…rawlee/storage_clients/_file_system/_dataset_client.py	100	Open or create a file system dataset client. This method attempts to open an existing dataset from the file sys	STRING
HIGH	…awlee/storage_clients/_memory/_request_queue_client.py	69	Open or create a new memory request queue client. This method creates a new in-memory request queue instance. U	STRING
HIGH	…lee/storage_clients/_memory/_key_value_store_client.py	57	Open or create a new memory key-value store client. This method creates a new in-memory key-value store instanc	STRING
HIGH	src/crawlee/storage_clients/_memory/_dataset_client.py	62	Open or create a new memory dataset client. This method creates a new in-memory dataset instance. Unlike persis	STRING

Redundant / Tautological Comments50 hits · 80 pts

Severity	File	Line	Snippet	Context
LOW	…guides/code_examples/login_crawler/playwright_login.py	47	# Check if the session is available	COMMENT
LOW	…8/guides/code_examples/running_in_web_server/server.py	57	# Return the result	STRING
LOW⚡	…code_examples/pydantic_ai_crawler/debugging_example.py	40	# Set concurrency to 1, which ensures only one request is processed at a time.	COMMENT
LOW⚡	…code_examples/pydantic_ai_crawler/debugging_example.py	44	# Set abort_on_error to True to stop the crawl if an error occurs during	COMMENT
LOW	…les/code_examples/respect_robots_on_skipped_request.py	22	# Check if the request was skipped due to robots.txt rules	COMMENT
LOW	…guides/code_examples/login_crawler/playwright_login.py	47	# Check if the session is available	COMMENT
LOW	…6/guides/code_examples/running_in_web_server/server.py	57	# Return the result	STRING
LOW	…les/code_examples/respect_robots_on_skipped_request.py	22	# Check if the request was skipped due to robots.txt rules	COMMENT
LOW	tests/unit/crawlers/_basic/test_basic_crawler.py	963	# Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately	COMMENT
LOW	tests/unit/crawlers/_basic/test_basic_crawler.py	987	# Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately	COMMENT
LOW	tests/unit/crawlers/_basic/test_basic_crawler.py	1237	# Set max_concurrency to 1 to ensure testing urls are visited one by one in order.	COMMENT
LOW	tests/unit/crawlers/_basic/test_basic_crawler.py	1265	# Set concurrency to 2 to ensure two urls are being visited in parallel.	COMMENT
LOW	…/crawlers/_beautifulsoup/test_beautifulsoup_crawler.py	113	# Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately	COMMENT
LOW	tests/unit/crawlers/_parsel/test_parsel_crawler.py	132	# Set max_concurrency to 1 to ensure testing max_requests_per_crawl accurately	COMMENT
LOW⚡	tests/unit/sessions/test_session_pool.py	136	# Check if the state is correctly persisted	COMMENT
LOW⚡	tests/unit/sessions/test_session_pool.py	141	# Check if all the sessions are correctly persisted	COMMENT
LOW	…s/unit/storage_clients/_redis/test_redis_kvs_client.py	60	# Check if the records were created	COMMENT
LOW⚡	…s/unit/storage_clients/_redis/test_redis_kvs_client.py	121	# Check if record content is valid JSON	COMMENT
LOW	…nit/storage_clients/_file_system/test_fs_kvs_client.py	85	# Check if the files were created	COMMENT
LOW⚡	…nit/storage_clients/_file_system/test_fs_kvs_client.py	130	# Check if file content is valid JSON	COMMENT
LOW	…guides/code_examples/login_crawler/playwright_login.py	47	# Check if the session is available	COMMENT
LOW	…s/guides/code_examples/running_in_web_server/server.py	57	# Return the result	STRING
LOW⚡	…code_examples/pydantic_ai_crawler/debugging_example.py	40	# Set concurrency to 1, which ensures only one request is processed at a time.	COMMENT
LOW⚡	…code_examples/pydantic_ai_crawler/debugging_example.py	44	# Set abort_on_error to True to stop the crawl if an error occurs during	COMMENT
LOW	…les/code_examples/respect_robots_on_skipped_request.py	22	# Check if the request was skipped due to robots.txt rules	COMMENT
LOW	src/crawlee/_utils/sitemap.py	375	# Check if the first chunk is a valid gzip header	COMMENT
LOW	src/crawlee/_utils/file.py	199	# Set lineterminator to '\n' if not explicitly provided. This prevents double line endings on Windows.	COMMENT
LOW	src/crawlee/_utils/file.py	209	# Iterate over the dataset and write to CSV.	COMMENT
LOW	src/crawlee/crawlers/_playwright/_playwright_crawler.py	567	# Check if the session is blocked based on the response content	COMMENT
LOW	src/crawlee/http_clients/_curl_impersonate.py	292	# Check if a session for the given proxy URL has already been created.	COMMENT
LOW⚡	src/crawlee/project_template/hooks/pre_gen_project.py	17	# Check if package manager is available in PATH	COMMENT
LOW⚡	src/crawlee/project_template/hooks/pre_gen_project.py	22	# Check if the package manager is executable	COMMENT
LOW⚡	src/crawlee/project_template/hooks/pre_gen_project.py	29	# Check if the version matches the required regex	COMMENT
LOW	…/crawlee/storage_clients/_sql/_request_queue_client.py	604	# Check if there are any unhandled requests that are not blocked.	COMMENT
LOW	…/crawlee/storage_clients/_sql/_request_queue_client.py	643	# Check if there are any buffered updates that might change the pending count	COMMENT
LOW	…rawlee/storage_clients/_sql/_key_value_store_client.py	292	# Check if record exists	COMMENT
LOW	…rawlee/storage_clients/_redis/_request_queue_client.py	508	# Check if there are any requests in the queue.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	342	# Check if the request has already been handled.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	351	# Check if the request is already in progress.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	490	# Check if the request is in progress.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	538	# Check if the request is in progress.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	797	# Read the file content and parse it as JSON.	COMMENT
LOW	…/storage_clients/_file_system/_request_queue_client.py	839	# Check if request was already handled	COMMENT
LOW	…torage_clients/_file_system/_key_value_store_client.py	372	# Check if the KVS directory exists	COMMENT
LOW	…awlee/storage_clients/_memory/_request_queue_client.py	149	# Check if the request is already in the queue by unique_key.	COMMENT
LOW	…awlee/storage_clients/_memory/_request_queue_client.py	249	# Check if the request is in progress.	COMMENT
LOW	…awlee/storage_clients/_memory/_request_queue_client.py	286	# Check if the request is in progress.	COMMENT
LOW	src/crawlee/_autoscaling/snapshotter.py	423	# Check if the warning has been logged recently to avoid spamming	COMMENT
LOW	src/crawlee/request_loaders/_sitemap_request_loader.py	392	# Check if URL should be included	COMMENT
LOW	src/crawlee/request_loaders/_sitemap_request_loader.py	399	# Check if we have capacity in the queue	COMMENT

Modern Structural Boilerplate59 hits · 60 pts

Severity	File	Line	Snippet	Context
LOW	…/playwright_crawler/browser_pool_page_hooks_example.py	16	logger = logging.getLogger(__name__)	CODE
LOW	…laywright_crawler/browser_pool_launch_hooks_example.py	14	logger = logging.getLogger(__name__)	CODE
LOW	…mples/request_loaders/rl_basic_example_with_persist.py	8	logger = logging.getLogger(__name__)	CODE
LOW	…amples/request_loaders/sitemap_example_with_persist.py	9	logger = logging.getLogger(__name__)	CODE
LOW	tests/unit/conftest.py	97	def _set_crawler_log_level(pytestconfig: pytest.Config, monkeypatch: pytest.MonkeyPatch) -> None:	CODE
LOW	tests/unit/server.py	154	async def set_cookies(scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW	tests/unit/server.py	373	async def set_complex_cookies(_scope: dict[str, Any], _receive: Receive, send: Send) -> None:	CODE
LOW	…/playwright_crawler/browser_pool_page_hooks_example.py	16	logger = logging.getLogger(__name__)	CODE
LOW	…laywright_crawler/browser_pool_launch_hooks_example.py	14	logger = logging.getLogger(__name__)	CODE
LOW	…mples/request_loaders/rl_basic_example_with_persist.py	8	logger = logging.getLogger(__name__)	CODE
LOW	…amples/request_loaders/sitemap_example_with_persist.py	9	logger = logging.getLogger(__name__)	CODE
LOW	src/crawlee/configuration.py	16	__all__ = ['Configuration']	CODE
LOW	src/crawlee/__init__.py	10	__all__ = [	CODE
LOW	src/crawlee/proxy_configuration.py	21	__all__ = ['ProxyConfiguration', 'ProxyInfo']	CODE
LOW	src/crawlee/errors.py	10	__all__ = [	CODE
LOW	src/crawlee/router.py	11	__all__ = ['Router']	CODE
LOW	src/crawlee/_service_locator.py	46	def set_configuration(self, configuration: Configuration) -> None:	CODE
LOW	src/crawlee/_service_locator.py	76	def set_event_manager(self, event_manager: EventManager) -> None:	CODE
LOW	src/crawlee/_service_locator.py	106	def set_storage_client(self, storage_client: StorageClient) -> None:	CODE
LOW	src/crawlee/statistics/__init__.py	4	__all__ = ['FinalStatistics', 'Statistics', 'StatisticsState']	CODE
LOW	src/crawlee/fingerprint_suite/__init__.py	6	__all__ = [	CODE
LOW	src/crawlee/storages/__init__.py	5	__all__ = [	CODE
LOW	src/crawlee/storages/_dataset.py	29	logger = logging.getLogger(__name__)	CODE
LOW	src/crawlee/crawlers/__init__.py	99	__all__ = [	CODE
LOW	src/crawlee/crawlers/_basic/__init__.py	5	__all__ = [	CODE
LOW	src/crawlee/crawlers/_beautifulsoup/__init__.py	15	__all__ = [	CODE
LOW	src/crawlee/crawlers/_parsel/__init__.py	13	__all__ = [	CODE
LOW	src/crawlee/crawlers/_pydantic_ai/_types.py	95	def set_ai_usage(self, value: PydanticAiUsageStats) -> None:	CODE
LOW	src/crawlee/crawlers/_pydantic_ai/__init__.py	29	__all__ = [	CODE
LOW	…c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py	342	def set_ai_usage(self, value: PydanticAiUsageStats) -> None:	CODE
LOW	src/crawlee/crawlers/_pydantic_ai/_base_extractor.py	65	def set_ai_usage(self, value: PydanticAiUsageStats) -> None:	CODE
LOW	src/crawlee/crawlers/_playwright/__init__.py	17	__all__ = [	CODE
LOW	src/crawlee/crawlers/_playwright/_playwright_crawler.py	613	async def _update_cookies(self, page: Page, cookies: list[PlaywrightCookieParam]) -> None:	CODE
LOW	src/crawlee/crawlers/_stagehand/__init__.py	20	__all__ = [	CODE
LOW	src/crawlee/crawlers/_http/__init__.py	6	__all__ = [	CODE
LOW	src/crawlee/crawlers/_adaptive_playwright/__init__.py	22	__all__ = [	CODE
LOW	src/crawlee/crawlers/_abstract_http/__init__.py	5	__all__ = [	CODE
LOW	src/crawlee/otel/__init__.py	3	__all__ = [	CODE
LOW	src/crawlee/http_clients/_curl_impersonate.py	48	def update_cookies_from_curl(self, morsels: list[CurlMorsel]) -> None:	CODE
LOW	src/crawlee/http_clients/__init__.py	19	__all__ = [	CODE
LOW	src/crawlee/browsers/__init__.py	28	__all__ = [	CODE
LOW⚡	src/crawlee/sessions/_cookies.py	210	def set_cookies(self, cookie_dicts: list[CookieParam]) -> None:	CODE
LOW⚡	src/crawlee/sessions/_cookies.py	224	def set_cookies_from_playwright_format(self, pw_cookies: list[PlaywrightCookieParam]) -> None:	CODE
LOW	src/crawlee/sessions/__init__.py	5	__all__ = ['CookieParam', 'Session', 'SessionCookies', 'SessionPool']	CODE
LOW	src/crawlee/storage_clients/__init__.py	19	__all__ = [	CODE
LOW	…rawlee/storage_clients/_sql/_key_value_store_client.py	149	async def set_value(self, *, key: str, value: Any, content_type: str \| None = None) -> None:	CODE
LOW	src/crawlee/storage_clients/_sql/__init__.py	6	__all__ = ['SqlDatasetClient', 'SqlKeyValueStoreClient', 'SqlRequestQueueClient', 'SqlStorageClient']	CODE
LOW	…wlee/storage_clients/_redis/_key_value_store_client.py	125	async def set_value(self, *, key: str, value: Any, content_type: str \| None = None) -> None:	CODE
LOW	src/crawlee/storage_clients/_redis/__init__.py	6	__all__ = ['RedisDatasetClient', 'RedisKeyValueStoreClient', 'RedisRequestQueueClient', 'RedisStorageClient']	CODE
LOW	…torage_clients/_file_system/_key_value_store_client.py	302	async def set_value(self, *, key: str, value: Any, content_type: str \| None = None) -> None:	CODE
LOW	src/crawlee/storage_clients/_file_system/__init__.py	6	__all__ = [	CODE
LOW	…lee/storage_clients/_memory/_key_value_store_client.py	112	async def set_value(self, *, key: str, value: Any, content_type: str \| None = None) -> None:	CODE
LOW	src/crawlee/storage_clients/_memory/__init__.py	6	__all__ = [	CODE
LOW	…awlee/storage_clients/_base/_key_value_store_client.py	53	async def set_value(self, *, key: str, value: Any, content_type: str \| None = None) -> None:	CODE
LOW	src/crawlee/storage_clients/_base/__init__.py	6	__all__ = [	CODE
LOW	src/crawlee/events/__init__.py	15	__all__ = [	CODE
LOW	src/crawlee/_autoscaling/__init__.py	5	__all__ = ['AutoscaledPool', 'Snapshotter', 'SystemStatus']	CODE
LOW	src/crawlee/request_loaders/__init__.py	8	__all__ = [	CODE
LOW	…crawlee/request_loaders/_throttling_request_manager.py	339	def set_crawl_delay(self, url: str, delay_seconds: int) -> None:	CODE

Excessive Try-Catch Wrapping45 hits · 46 pts

Severity	File	Line	Snippet	Context
LOW	…ing_web_archive/manual_archiving_playwright_crawler.py	28	except Exception as e:	CODE
LOW	tests/unit/_statistics/test_error_tracker.py	34	except Exception as e: # noqa:PERF203	CODE
LOW	tests/unit/_statistics/test_error_tracker.py	39	except Exception as e:	CODE
LOW	tests/unit/_statistics/test_error_tracker.py	69	except Exception as e: # noqa:PERF203	CODE
LOW⚡	tests/unit/_statistics/test_error_tracker.py	98	except Exception as e:	CODE
LOW⚡	tests/unit/_statistics/test_error_tracker.py	110	except Exception as e:	CODE
LOW⚡	tests/unit/_statistics/test_error_tracker.py	113	except Exception as e:	CODE
LOW	tests/unit/storages/test_request_queue.py	707	except Exception:	CODE
LOW⚡	tests/unit/crawlers/_parsel/test_parsel_crawler.py	101	except Exception as e:	CODE
LOW⚡	…s/unit/crawlers/_playwright/test_playwright_crawler.py	151	except Exception as e:	CODE
LOW	…ing_web_archive/manual_archiving_playwright_crawler.py	28	except Exception as e:	CODE
LOW	src/crawlee/_cli.py	240	except Exception as exc:	CODE
LOW	src/crawlee/proxy_configuration.py	186	except Exception as e:	CODE
LOW	src/crawlee/statistics/_error_tracker.py	104	except Exception:	CODE
LOW	src/crawlee/_utils/sitemap.py	189	except Exception as e:	CODE
LOW	src/crawlee/_utils/sitemap.py	202	except Exception as e:	CODE
LOW	src/crawlee/_utils/sitemap.py	332	except Exception as e:	CODE
LOW	src/crawlee/_utils/sitemap.py	412	except Exception as e:	CODE
LOW	src/crawlee/_utils/sitemap.py	547	except Exception:	CODE
LOW	src/crawlee/_utils/sitemap.py	619	except Exception:	CODE
LOW	src/crawlee/_utils/wait.py	40	except Exception as e:	CODE
LOW	src/crawlee/_utils/wait.py	85	except Exception as e:	CODE
LOW	src/crawlee/_utils/recurring_task.py	56	except Exception:	CODE
LOW	src/crawlee/_utils/file.py	58	except Exception:	CODE
LOW	src/crawlee/_utils/requests.py	82	except Exception as exc:	CODE
LOW	src/crawlee/_utils/robots.py	67	except Exception as e:	CODE
LOW	src/crawlee/storages/_request_queue.py	70	except Exception:	STRING
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1171	except Exception as e:	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1200	except Exception as secondary_error:	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1222	except Exception as e:	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1237	except Exception as e:	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1509	except Exception as internal_error:	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1521	except Exception: # noqa: PERF203	CODE
LOW	src/crawlee/crawlers/_basic/_context_pipeline.py	50	except Exception as e:	CODE
LOW	src/crawlee/crawlers/_basic/_context_pipeline.py	107	except Exception as e:	CODE
LOW	src/crawlee/crawlers/_basic/_context_pipeline.py	118	except Exception as e:	CODE
LOW	…rs/_playwright/_playwright_pre_nav_crawling_context.py	39	except Exception:	CODE
LOW	…rs/_playwright/_playwright_pre_nav_crawling_context.py	44	except Exception:	CODE
LOW	…s/_adaptive_playwright/_adaptive_playwright_crawler.py	339	except Exception as e:	CODE
LOW	src/crawlee/browsers/_browser_pool.py	374	except Exception:	CODE
LOW	src/crawlee/browsers/_stagehand_browser_controller.py	182	except Exception:	CODE
LOW	…rawlee/storage_clients/_redis/_request_queue_client.py	437	except Exception:	CODE
LOW	src/crawlee/events/_event_manager.py	192	except Exception:	CODE
LOW	src/crawlee/request_loaders/_sitemap_request_loader.py	421	except Exception:	CODE
LOW	src/crawlee/request_loaders/_request_manager_tandem.py	87	except Exception:	CODE

AI Response Leakage6 hits · 45 pts

Severity	File	Line	Snippet	Context
HIGH	…rsion-1.8/examples/code_examples/playwright_crawler.py	56	# browser page among other things. In this example, we log the URL being	COMMENT
HIGH	…-1.8/examples/code_examples/respect_robots_txt_file.py	19	# In this example, 'https://news.ycombinator.com/login' will be skipped	COMMENT
HIGH	…rsion-0.6/examples/code_examples/playwright_crawler.py	56	# browser page among other things. In this example, we log the URL being	COMMENT
HIGH	…-0.6/examples/code_examples/respect_robots_txt_file.py	19	# In this example, 'https://news.ycombinator.com/login' will be skipped	COMMENT
HIGH	docs/examples/code_examples/playwright_crawler.py	56	# browser page among other things. In this example, we log the URL being	COMMENT
HIGH	docs/examples/code_examples/respect_robots_txt_file.py	19	# In this example, 'https://news.ycombinator.com/login' will be skipped	COMMENT

Deep Nesting39 hits · 38 pts

Severity	File	Line	Context
LOW	tests/e2e/project_template/utils.py	58	CODE
LOW	src/crawlee/_cli.py	126	CODE
LOW	src/crawlee/_utils/globs.py	19	CODE
LOW	src/crawlee/_utils/globs.py	76	CODE
LOW	src/crawlee/_utils/sitemap.py	229	CODE
LOW	src/crawlee/_utils/sitemap.py	338	CODE
LOW	src/crawlee/_utils/sitemap.py	465	CODE
LOW	src/crawlee/_utils/sitemap.py	568	CODE
LOW	src/crawlee/_utils/sitemap.py	107	CODE
LOW	src/crawlee/_utils/wait.py	49	CODE
LOW	src/crawlee/_utils/urls.py	42	CODE
LOW	src/crawlee/storages/_storage_instance_manager.py	40	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1071	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1147	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1559	CODE
LOW	src/crawlee/crawlers/_basic/_context_pipeline.py	82	CODE
LOW	src/crawlee/crawlers/_beautifulsoup/_utils.py	20	CODE
LOW	src/crawlee/crawlers/_beautifulsoup/_utils.py	38	CODE
LOW	src/crawlee/crawlers/_parsel/_utils.py	16	CODE
LOW	src/crawlee/crawlers/_parsel/_utils.py	34	CODE
LOW	…crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py	176	CODE
LOW	…c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py	656	CODE
LOW	src/crawlee/sessions/_cookies.py	63	CODE
LOW	src/crawlee/sessions/_session.py	94	CODE
LOW	src/crawlee/storage_clients/_sql/_storage_client.py	112	CODE
LOW	…/crawlee/storage_clients/_sql/_request_queue_client.py	207	CODE
LOW	…rawlee/storage_clients/_sql/_key_value_store_client.py	149	CODE
LOW	…rawlee/storage_clients/_sql/_key_value_store_client.py	191	CODE
LOW	…wlee/storage_clients/_redis/_key_value_store_client.py	125	CODE
LOW	…/storage_clients/_file_system/_request_queue_client.py	154	CODE
LOW	…/storage_clients/_file_system/_request_queue_client.py	322	CODE
LOW	…torage_clients/_file_system/_key_value_store_client.py	90	CODE
LOW	…torage_clients/_file_system/_key_value_store_client.py	223	CODE
LOW	…torage_clients/_file_system/_key_value_store_client.py	302	CODE
LOW	…rawlee/storage_clients/_file_system/_dataset_client.py	92	CODE
LOW	…awlee/storage_clients/_memory/_request_queue_client.py	141	CODE
LOW	src/crawlee/_autoscaling/autoscaled_pool.py	105	CODE
LOW	src/crawlee/_autoscaling/autoscaled_pool.py	213	CODE
LOW	src/crawlee/request_loaders/_sitemap_request_loader.py	341	CODE

AI Structural Patterns29 hits · 28 pts

Severity	File	Line	Context
LOW	src/crawlee/_types.py	387	CODE
LOW	src/crawlee/_types.py	412	CODE
LOW	src/crawlee/_request.py	244	CODE
LOW	src/crawlee/statistics/_statistics.py	71	CODE
LOW	src/crawlee/statistics/_statistics.py	127	CODE
LOW	src/crawlee/fingerprint_suite/_browserforge_adapter.py	56	CODE
LOW	src/crawlee/storages/_dataset.py	150	CODE
LOW	src/crawlee/storages/_dataset.py	201	CODE
LOW	src/crawlee/storages/_dataset.py	249	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	279	CODE
LOW	src/crawlee/crawlers/_basic/_basic_crawler.py	1026	CODE
LOW	…crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py	71	CODE
LOW	…c/crawlee/crawlers/_pydantic_ai/_selector_extractor.py	290	CODE
LOW	…c/crawlee/crawlers/_pydantic_ai/_skeleton_distiller.py	64	CODE
LOW	src/crawlee/crawlers/_playwright/_playwright_crawler.py	117	CODE
LOW	src/crawlee/crawlers/_stagehand/_stagehand_crawler.py	94	CODE
LOW	src/crawlee/sessions/_session.py	36	CODE
LOW	…/crawlee/storage_clients/_sql/_request_queue_client.py	682	CODE
LOW	src/crawlee/storage_clients/_sql/_dataset_client.py	163	CODE
LOW	src/crawlee/storage_clients/_sql/_dataset_client.py	210	CODE
LOW	src/crawlee/storage_clients/_sql/_dataset_client.py	243	CODE
LOW	src/crawlee/storage_clients/_redis/_dataset_client.py	146	CODE
LOW	src/crawlee/storage_clients/_redis/_dataset_client.py	232	CODE
LOW	…rawlee/storage_clients/_file_system/_dataset_client.py	243	CODE
LOW	…rawlee/storage_clients/_file_system/_dataset_client.py	342	CODE
LOW	src/crawlee/storage_clients/_memory/_dataset_client.py	138	CODE
LOW	src/crawlee/storage_clients/_memory/_dataset_client.py	197	CODE
LOW	src/crawlee/storage_clients/_base/_dataset_client.py	55	CODE
LOW	src/crawlee/storage_clients/_base/_dataset_client.py	76	CODE

Decorative Section Separators6 hits · 24 pts

Severity	File	Line	Snippet	Context
MEDIUM⚡	tests/unit/test_throttling_request_manager.py	141	# ── Core Throttling Tests ─────────────────────────────────	COMMENT
MEDIUM⚡	tests/unit/test_throttling_request_manager.py	240	# ── Crawl-Delay Integration Tests ─────────────────────────	COMMENT
MEDIUM⚡	tests/unit/test_throttling_request_manager.py	262	# ── Fetch Scheduling Tests ────────────────────────────	COMMENT
MEDIUM⚡	tests/unit/test_throttling_request_manager.py	530	# ── Utility Tests ──────────────────────────────────────	COMMENT
MEDIUM	tests/unit/test_throttling_request_manager.py	58	# ── Request Routing Tests ─────────────────────────────────	COMMENT
MEDIUM	tests/unit/test_throttling_request_manager.py	372	# ── Delegation Tests ────────────────────────────────────	COMMENT

AI Slop Vocabulary12 hits · 21 pts

Severity	File	Line	Snippet	Context
LOW	…ersion-1.8/introduction/code_examples/07_final_code.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	…/version-1.8/introduction/code_examples/06_scraping.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	…n-1.8/introduction/code_examples/05_crawling_detail.py	13	# We're not processing detail pages yet, so we just pass.	COMMENT
MEDIUM	…ides/code_examples/http_crawlers/selectolax_context.py	10	# to facilitate working with the parsed document.	COMMENT
LOW	…ersion-0.6/introduction/code_examples/07_final_code.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	…/version-0.6/introduction/code_examples/06_scraping.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	…n-0.6/introduction/code_examples/05_crawling_detail.py	13	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	docs/introduction/code_examples/07_final_code.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	docs/introduction/code_examples/06_scraping.py	16	# We're not processing detail pages yet, so we just pass.	COMMENT
LOW	docs/introduction/code_examples/05_crawling_detail.py	13	# We're not processing detail pages yet, so we just pass.	COMMENT
MEDIUM	…ides/code_examples/http_crawlers/selectolax_context.py	10	# to facilitate working with the parsed document.	COMMENT
LOW	src/crawlee/events/_event_manager.py	176	# If the listener is a coroutine function, just call it, otherwise, run it in a separate thread	COMMENT

Slop Phrases5 hits · 14 pts

Severity	File	Line	Snippet	Context
MEDIUM	…ides/code_examples/http_crawlers/selectolax_context.py	9	# Custom context for Selectolax parser, you can add your own methods here	COMMENT
MEDIUM	…ides/code_examples/http_crawlers/selectolax_crawler.py	20	# Custom crawler using custom context, It is optional and you can use	COMMENT
LOW	…e2e/project_template/test_static_crawlers_templates.py	15	# To run these tests locally, make sure you have apify-cli installed and available in the path.	COMMENT
MEDIUM	…ides/code_examples/http_crawlers/selectolax_context.py	9	# Custom context for Selectolax parser, you can add your own methods here	COMMENT
MEDIUM	…ides/code_examples/http_crawlers/selectolax_crawler.py	20	# Custom crawler using custom context, It is optional and you can use	COMMENT

Over-Commented Block10 hits · 10 pts

Severity	File	Line	Snippet	Context
LOW	website/docusaurus.config.js	121	},	COMMENT
LOW	website/docusaurus.config.js	141	// to: '/docs/introduction',	COMMENT
LOW	website/sidebars.js	81	type: 'autogenerated',	COMMENT
LOW	website/src/components/Highlights.jsx	61	// or plug an adapter for your DB. Your URLs are <a href="https://crawlee.dev/docs/guides/request-storag	COMMENT
LOW	…emplate/{{cookiecutter.project_name}}/requirements.txt	1	# % if cookiecutter.crawler_type == 'playwright-camoufox'	COMMENT
LOW	…_template/{{cookiecutter.project_name}}/pyproject.toml	1	# % if cookiecutter.crawler_type.startswith('adaptive-')	COMMENT
LOW	…oject_name}}/{{cookiecutter.__package_name}}/routes.py	1	# % if cookiecutter.crawler_type.startswith('playwright')	COMMENT
LOW	src/crawlee/project_template/hooks/post_gen_project.py	41		COMMENT
LOW	src/crawlee/project_template/templates/main.py	1	# % if cookiecutter.enable_apify_integration	COMMENT
LOW	src/crawlee/project_template/templates/main.py	21	http_client=HttpxHttpClient(),	COMMENT

Fake / Example Data8 hits · 8 pts

Severity	File	Line	Snippet	Context
LOW	…ples/code_examples/fill_and_submit_web_form_request.py	15	'custname': 'John Doe',	CODE
LOW	…ples/code_examples/fill_and_submit_web_form_crawler.py	25	'custname': 'John Doe',	CODE
LOW	…ples/code_examples/fill_and_submit_web_form_request.py	15	'custname': 'John Doe',	CODE
LOW	…ples/code_examples/fill_and_submit_web_form_crawler.py	25	'custname': 'John Doe',	CODE
LOW	tests/unit/crawlers/_http/test_http_crawler.py	28	'custname': 'John Doe',	CODE
LOW	…ples/code_examples/fill_and_submit_web_form_request.py	15	'custname': 'John Doe',	CODE
LOW	…ples/code_examples/fill_and_submit_web_form_crawler.py	25	'custname': 'John Doe',	CODE
LOW	…crawlee/crawlers/_pydantic_ai/_clean_html_distiller.py	35	'placeholder',	CODE

Dead Code3 hits · 6 pts

Severity	File	Line	Context
MEDIUM	tests/unit/crawlers/_basic/test_context_pipeline.py	150	CODE
MEDIUM	…awlee/storage_clients/_base/_key_value_store_client.py	79	CODE
MEDIUM	src/crawlee/storage_clients/_base/_dataset_client.py	95	CODE

Cross-Language Confusion1 hit · 5 pts

Severity	File	Line	Snippet	Context
HIGH	tests/unit/server_endpoints.py	127	if (loading \|\| page >= 3) return;	CODE

Overly Generic Function Names4 hits · 4 pts

Severity	File	Line	Snippet	Context
LOW	…/code_examples/http_crawlers/selectolax_crawler_run.py	12	async def handle_request(context: SelectolaxLexborContext) -> None:	CODE
LOW	…code_examples/http_crawlers/selectolax_adaptive_run.py	19	async def handle_request(context: AdaptivePlaywrightCrawlingContext) -> None:	CODE
LOW	…/code_examples/http_crawlers/selectolax_crawler_run.py	12	async def handle_request(context: SelectolaxLexborContext) -> None:	CODE
LOW	…code_examples/http_crawlers/selectolax_adaptive_run.py	19	async def handle_request(context: AdaptivePlaywrightCrawlingContext) -> None:	CODE

Analysis Overview

What These Metrics Mean

Score History

Severity Breakdown

Directory Score Breakdown

Pattern Findings