Repository Analysis

scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

26.9 Moderate AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of scrapy/scrapy, a Python project with 64,081 GitHub stars. SynthScan v2.0 examined 92,959 lines of code across 526 source files, recording 2136 pattern matches distributed across 17 syntactic categories. The overall adjusted score of 26.9 places this repository in the Moderate AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

26.9
Adjusted Score
26.9
Raw Score
100%
Time Factor
2026-08-28
Last Push
64.1K
Stars
Python
Language
93.0K
Lines of Code
526
Files
2.1K
Pattern Hits
2026-08-29
Scan Date
0.11
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 5HIGH 54MEDIUM 87LOW 1990

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 2136 distinct pattern matches across 17 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers1394 hits · 1350 pts
SeverityFileLineSnippetContext
LOWconftest.py181def pytest_collection_modifyitems(items):CODE
LOWtests/test_request_dict.py91def test_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py100def test_reference_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py112def test_private_reference_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py124def test_private_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py133def test_mixin_private_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py142def test_delegated_callback_serialization(spider: MethodsSpider) -> None:CODE
LOWtests/test_request_dict.py181def test_callback_not_available() -> None:CODE
LOWtests/test_utils_signal.py57 async def test_send_catch_log_dont_log(CODE
LOWtests/test_utils_signal.py162 def test_error_logged_if_deferred_not_supported(CODE
LOWtests/test_http_response_text.py161 def test_declared_encoding_invalid(self):CODE
LOWtests/test_http_response_text.py193 def test_bom_is_removed_from_body(self):CODE
LOWtests/test_http_response_text.py220 def test_replace_wrong_encoding(self):CODE
LOWtests/test_http_response_text.py271 def test_selector_shortcuts_kwargs(self):CODE
LOWtests/test_http_response_text.py292 def test_urljoin_with_base_url(self):CODE
LOWtests/test_http_response_text.py351 def test_follow_selector_list(self):CODE
LOWtests/test_http_response_text.py356 def test_follow_selector_invalid(self):CODE
LOWtests/test_http_response_text.py361 def test_follow_selector_attribute(self):CODE
LOWtests/test_http_response_text.py366 def test_follow_selector_no_href(self):CODE
LOWtests/test_http_response_text.py374 def test_follow_whitespace_selector(self):CODE
LOWtests/test_http_response_text.py424 def test_follow_all_css_skip_invalid(self):CODE
LOWtests/test_http_response_text.py445 def test_follow_all_xpath_skip_invalid(self):CODE
LOWtests/test_http_response_text.py462 def test_follow_all_too_many_arguments(self):CODE
LOWtests/test_http_response_text.py492 def test_json_response_wrong_charset(self):CODE
LOWtests/test_http_response_text.py604 def test_selector_shortcuts_kwargs(self):CODE
LOWtests/test_contracts.py112 def returns_request_cb_kwargs(self, response, url):CODE
LOWtests/test_contracts.py128 def returns_item_cb_kwargs_error_unexpected_keyword(self, response):CODE
LOWtests/test_contracts.py136 def returns_item_cb_kwargs_error_missing_argument(self, response, arg):CODE
LOWtests/test_contracts.py157 def returns_request_range_fail(self, response):CODE
LOWtests/test_contracts.py219 def scrapes_multiple_missing_fields(self, response):CODE
LOWtests/test_contracts.py243 def invalid_regex_with_valid_contract(self, response):CODE
LOWtests/test_contracts.py267 def returns_error_missing_meta(self, response):CODE
LOWtests/test_contracts.py484 def test_returns_invalid_argument_count(self):CODE
LOWtests/test_contracts.py491 def test_returns_default_bounds(self):CODE
LOWtests/test_contracts.py506 def test_returns_and_scrapes_ignore_other_types(self):CODE
LOWtests/test_contracts.py626 def test_custom_tagged_request_contract(self):CODE
LOWtests/test_contracts.py698 def test_pre_hook_keyboard_interrupt(self):CODE
LOWtests/test_contracts.py712 def test_post_hook_keyboard_interrupt(self):CODE
LOWtests/test_contracts.py739 def test_pre_hook_assertion_failure(self):CODE
LOWtests/test_contracts.py764 def test_pre_hook_async_callback(self):CODE
LOWtests/test_contracts.py774 def test_pre_hook_async_generator(self):CODE
LOWtests/test_contracts.py784 def test_post_hook_async_generator(self):CODE
LOWtests/test_command_parse.py116 def parse_request_with_cb_kwargs(self, response, foo=None, key=None):CODE
LOWtests/test_command_parse.py122 def parse_request_without_meta(self, response):CODE
LOWtests/test_command_parse.py483 def test_crawlspider_matching_rule_default_callback(CODE
LOWtests/test_command_parse.py497 def test_spider_with_no_rules_attribute(CODE
LOWtests/test_command_parse.py512 def test_crawlspider_missing_callback(CODE
LOWtests/test_command_parse.py525 def test_crawlspider_no_matching_rule(CODE
LOWtests/test_command_parse.py540 def test_crawlspider_not_exists_with_not_matched_url(CODE
LOWtests/test_command_parse.py272 def test_request_with_cb_kwargs(STRING
LOWtests/test_command_parse.py294 def test_request_without_meta(STRING
LOWtests/test_command_parse.py323 def test_async_def_asyncio_parse_items_list(STRING
LOWtests/test_command_parse.py339 def test_async_def_asyncio_parse_items_single_element(STRING
LOWtests/test_command_parse.py354 def test_async_def_asyncgen_parse_loop(STRING
LOWtests/test_command_parse.py370 def test_async_def_asyncgen_parse_exc(STRING
LOWtests/test_command_parse.py386 def test_async_def_asyncio_parse(STRING
LOWtests/test_command_parse.py412 def test_parse_items_no_callback_passed(STRING
LOWtests/test_command_parse.py424 def test_retry_response_request(STRING
LOWtests/test_command_parse.py439 def test_retry_response_request_with_custom_callback(STRING
LOWtests/test_command_parse.py454 def test_wrong_callback_passed(STRING
1334 more matches not shown…
Unused Imports357 hits · 353 pts
SeverityFileLineSnippetContext
LOWconftest.py1CODE
LOWtests/test_request_dict.py1CODE
LOWtests/test_utils_signal.py1CODE
LOWtests/test_http_response_text.py1CODE
LOWtests/test_contracts.py1CODE
LOWtests/test_command_parse.py1CODE
LOWtests/test_logstats.py1CODE
LOWtests/test_utils_template.py1CODE
LOWtests/test_command_genspider.py1CODE
LOWtests/test_extension_remote_control.py1CODE
LOWtests/test_utils_datatypes.py1CODE
LOWtests/test_engine.py1CODE
LOWtests/test_zz_resources.py3CODE
LOWtests/test_spidermiddleware.py1CODE
LOWtests/test_utils_benchserver.py1CODE
LOWtests/test_utils_benchserver.py9CODE
LOWtests/test_scrapy__getattr__.py1CODE
LOWtests/test_extension_debug.py1CODE
LOWtests/test_downloader_handler_twisted_http11.py3CODE
LOWtests/test_stats.py1CODE
LOWtests/test_linkextractors.py1CODE
LOWtests/test_command_runspider.py1CODE
LOWtests/test_engine_close_spider.py1CODE
LOWtests/test_engine_close_spider.py16CODE
LOWtests/test_loader.py1CODE
LOWtests/test_middleware.py1CODE
LOWtests/test_http_request_json.py1CODE
LOWtests/test_http2_client_protocol.py1CODE
LOWtests/test_utils_iterators.py1CODE
LOWtests/test_dependencies.py1CODE
LOWtests/test_signals.py1CODE
LOWtests/test_utils_log.py1CODE
LOWtests/test_link.py1CODE
LOWtests/test_spider_sitemap.py1CODE
LOWtests/test_spidermiddleware_urllength.py1CODE
LOWtests/test_downloader_handler_twisted_http2.py3CODE
LOWtests/test_feedexport_postprocess.py1CODE
LOWtests/test_feedexport_uri_params.py1CODE
LOWtests/test_utils_job.py1CODE
LOWtests/test_spidermiddleware_referer.py1CODE
LOWtests/test_utils_asyncgen.py1CODE
LOWtests/spiders.py5CODE
LOWtests/test_downloadermiddleware_stats.py1CODE
LOWtests/test_utils_project.py1CODE
LOWtests/test_downloadermiddleware_robotstxt.py1CODE
LOWtests/test_crawler_subprocess.py1CODE
LOWtests/test_robotstxt_interface.py1CODE
LOWtests/test_dupefilters.py1CODE
LOWtests/test_engine_stop_download_bytes.py1CODE
LOWtests/test_http_request_form.py1CODE
LOWtests/test_utils_trackref.py1CODE
LOWtests/test_core_downloader.py1CODE
LOWtests/test_core_downloader.py54CODE
LOWtests/test_spidermiddleware_start.py1CODE
LOWtests/test_http_cookies.py1CODE
LOWtests/test_scheduler_base.py1CODE
LOWtests/test_crawler_runners.py1CODE
LOWtests/test_downloadermiddleware.py1CODE
LOWtests/test_downloadermiddleware_useragent.py1CODE
LOWtests/test_spider.py1CODE
297 more matches not shown…
Cross-Language Confusion51 hits · 308 pts
SeverityFileLineSnippetContext
HIGHtests/test_squeues_request.py47 q.push(req)CODE
HIGHtests/test_squeues_request.py66 q.push(req1)CODE
HIGHtests/test_squeues_request.py67 q.push(req2)CODE
HIGHtests/test_squeues_request.py68 q.push(req3)CODE
HIGHtests/test_pqueues.py78 queue.push(req1)CODE
HIGHtests/test_pqueues.py79 queue.push(req2)CODE
HIGHtests/test_pqueues.py80 queue.push(req3)CODE
HIGHtests/test_pqueues.py127 queue.push(req1)CODE
HIGHtests/test_pqueues.py128 queue.push(req2)CODE
HIGHtests/test_pqueues.py129 queue.push(req3)CODE
HIGHtests/test_pqueues.py159 self.queue.push(req1)CODE
HIGHtests/test_pqueues.py160 self.queue.push(req2)CODE
HIGHtests/test_pqueues.py161 self.queue.push(req3)CODE
HIGHtests/test_pqueues.py188 self.queue.push(req1)CODE
HIGHtests/test_pqueues.py189 self.queue.push(req2)CODE
HIGHtests/test_pqueues.py190 self.queue.push(req3)CODE
HIGHtests/test_pqueues.py43 queue.push(req1)CODE
HIGHtests/test_pqueues.py58 queue.push(Request("https://example.org"))CODE
HIGHtests/test_pqueues.py102 queue.push(req)CODE
HIGHtests/test_pqueues.py174 self.queue.push(Request("https://example.org"))CODE
HIGHtests/test_pqueues.py215 self.queue.push(request)CODE
HIGHtests/test_pqueues.py239 self.queue.push(request)CODE
HIGHtests/test_pqueues.py259 self.queue.push(req)CODE
HIGHtests/test_pqueues.py271 self.queue.push(req)CODE
HIGHtests/test_pqueues.py289 queue.push(request)CODE
HIGHtests/test_pqueues.py295 queue.push(request)CODE
HIGHtests/test_pqueues.py344 queue.push(request)CODE
HIGHtests/test_squeues.py69 q.push(lambda x: x)CODE
HIGHtests/test_squeues.py75 q.push(sel)CODE
HIGHtests/test_squeues.py85 q.push("a")CODE
HIGHtests/test_squeues.py86 q.push(123)CODE
HIGHtests/test_squeues.py87 q.push({"a": "dict"})CODE
HIGHtests/test_squeues.py128 q.push(i)CODE
HIGHtests/test_squeues.py137 q.push(loader)CODE
HIGHtests/test_squeues.py148 q.push(r)CODE
HIGHtests/test_squeues.py161 q.push(lambda x: x)CODE
HIGHtests/test_squeues.py170 q.push(sel)CODE
HIGHtests/test_squeues.py197 q.push("a")CODE
HIGHtests/test_squeues.py198 q.push(123)CODE
HIGHtests/test_squeues.py199 q.push({"a": "dict"})CODE
HIGHtests/test_squeues.py220 q.push(i)CODE
HIGHtests/test_squeues.py229 q.push(loader)CODE
HIGHtests/test_squeues.py240 q.push(r)CODE
HIGHtests/test_settings/__init__.py962 {"FOO": '{"tests.test_settings.Component1": null}'},CODE
HIGHscrapy/pqueues.py187 q.push(request) # this may fail (eg. serialization error)CODE
HIGHscrapy/pqueues.py425 queue.push(request)CODE
HIGHscrapy/core/scheduler.py418 self.dqs.push(request)STRING
HIGHscrapy/core/scheduler.py439 self.mqs.push(request)STRING
HIGHscrapy/extensions/remote_control.py71 - ``start_time`` (number or null): the start time of the Scrapy processSTRING
HIGHscrapy/extensions/remote_control.py90 - ``traceback`` (string or null): the traceback if an exception wasSTRING
HIGHscrapy/spidermiddlewares/referer.py74 If url is null, return no referrer.STRING
Dead Code66 hits · 132 pts
SeverityFileLineSnippetContext
MEDIUMtests/test_engine.py223CODE
MEDIUMtests/test_spidermiddleware.py78CODE
MEDIUMtests/spiders.py679CODE
MEDIUMtests/test_utils_python.py42CODE
MEDIUMtests/test_crawler.py246CODE
MEDIUMtests/test_crawler.py324CODE
MEDIUMtests/test_crawler.py402CODE
MEDIUMtests/test_crawler.py480CODE
MEDIUMtests/test_crawler.py580CODE
MEDIUMtests/test_engine_loop.py379CODE
MEDIUMtests/test_engine_loop.py420CODE
MEDIUMtests/test_engine_loop.py479CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_simple.py15CODE
MEDIUMtests/AsyncCrawlerRunner/custom_loop_same.py20CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_custom_settings.py18CODE
MEDIUMtests/AsyncCrawlerRunner/custom_loop_different.py20CODE
MEDIUMtests/AsyncCrawlerRunner/multi_parallel.py15CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_reactor.py14CODE
MEDIUMtests/AsyncCrawlerRunner/no_reactor.py13CODE
MEDIUMtests/AsyncCrawlerRunner/simple_default_reactor.py14CODE
MEDIUMtests/AsyncCrawlerRunner/simple.py17CODE
MEDIUMtests/AsyncCrawlerRunner/multi_seq.py15CODE
MEDIUMtests/CrawlerProcess/asyncio_enabled_no_reactor.py17CODE
MEDIUMtests/CrawlerProcess/args_settings.py19CODE
MEDIUMtests/CrawlerProcess/reactor_select.py15CODE
MEDIUMtests/CrawlerProcess/multi.py10CODE
MEDIUMtests/CrawlerProcess/asyncio_enabled_reactor.py53CODE
MEDIUM…ts/CrawlerProcess/asyncio_enabled_reactor_same_loop.py22CODE
MEDIUM…awlerProcess/reactor_default_twisted_reactor_select.py13CODE
MEDIUM…awlerProcess/asyncio_enabled_reactor_different_loop.py20CODE
MEDIUM…cess/reactor_select_subclass_twisted_reactor_select.py22CODE
MEDIUMtests/CrawlerProcess/dns_resolver_deprecated.py13CODE
MEDIUMtests/CrawlerProcess/reactor_default.py13CODE
MEDIUMtests/CrawlerProcess/simple.py12CODE
MEDIUM…rawlerProcess/reactor_select_twisted_reactor_select.py14CODE
MEDIUMtests/CrawlerProcess/asyncio_custom_loop.py10CODE
MEDIUMtests/CrawlerRunner/explicit_default_reactor.py17CODE
MEDIUMtests/CrawlerRunner/custom_loop_same.py19CODE
MEDIUMtests/CrawlerRunner/change_reactor.py15CODE
MEDIUMtests/CrawlerRunner/custom_loop_different.py19CODE
MEDIUMtests/CrawlerRunner/multi_parallel.py14CODE
MEDIUMtests/CrawlerRunner/no_reactor.py13CODE
MEDIUMtests/CrawlerRunner/simple.py16CODE
MEDIUMtests/CrawlerRunner/multi_seq.py15CODE
MEDIUMtests/AsyncCrawlerProcess/asyncio_enabled_no_reactor.py17CODE
MEDIUMtests/AsyncCrawlerProcess/reactorless_simple.py12CODE
MEDIUM…erProcess/asyncio_enabled_reactor_same_loop_default.py21CODE
MEDIUMtests/AsyncCrawlerProcess/args_settings.py19CODE
MEDIUM…syncCrawlerProcess/reactorless_import_hook_multiple.py21CODE
MEDIUM…sts/AsyncCrawlerProcess/reactorless_custom_settings.py23CODE
MEDIUMtests/AsyncCrawlerProcess/multi.py10CODE
MEDIUMtests/AsyncCrawlerProcess/asyncio_enabled_reactor.py43CODE
MEDIUM…yncCrawlerProcess/asyncio_enabled_reactor_same_loop.py21CODE
MEDIUM…ncCrawlerProcess/reactorless_telnetconsole_disabled.py10CODE
MEDIUM…ocess/asyncio_custom_loop_custom_settings_different.py13CODE
MEDIUM…lerProcess/asyncio_custom_loop_custom_settings_same.py13CODE
MEDIUM…awlerProcess/asyncio_enabled_reactor_different_loop.py19CODE
MEDIUM…yncCrawlerProcess/reactorless_telnetconsole_enabled.py10CODE
MEDIUMtests/AsyncCrawlerProcess/reactorless_then_reactor.py10CODE
MEDIUMtests/AsyncCrawlerProcess/reactorless_import_hook.py12CODE
6 more matches not shown…
Modern Structural Boilerplate98 hits · 100 pts
SeverityFileLineSnippetContext
LOWtests/test_robotstxt_interface.py31 def _setUp(self, parser_cls: type[RobotParser]) -> None:CODE
LOWtests/test_scheduler.py107 def _setup(scheduler: Scheduler) -> None:CODE
LOWtests/test_scheduler.py120 def _setup(scheduler: Scheduler) -> None:CODE
LOWtests/test_scheduler.py135 def _setup(scheduler: Scheduler) -> None:CODE
LOWtests/test_scheduler.py229 def _setup(scheduler: Scheduler) -> None:CODE
LOWtests/test_extension_periodic_log.py67 def set_a(self) -> None:CODE
LOWtests/test_extension_periodic_log.py70 def set_b(self) -> None:CODE
LOW…syncCrawlerProcess/reactorless_import_hook_multiple.py8logger = logging.getLogger(__name__)CODE
LOW…yncCrawlerProcess/reactorless_import_hook_uninstall.py8logger = logging.getLogger(__name__)CODE
LOWscrapy/mail.py41logger = logging.getLogger(__name__)CODE
LOWscrapy/robotstxt.py21logger = logging.getLogger(__name__)CODE
LOWscrapy/addons.py15logger = logging.getLogger(__name__)CODE
LOWscrapy/pqueues.py21logger = logging.getLogger(__name__)CODE
LOWscrapy/pqueues.py221 def _update_curprio(self) -> None:CODE
LOWscrapy/__init__.py15__all__ = [CODE
LOWscrapy/statscollectors.py18logger = logging.getLogger(__name__)CODE
LOWscrapy/statscollectors.py69 def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py73 def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py155 def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py158 def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None:CODE
LOWscrapy/exporters.py28logger = logging.getLogger(__name__)CODE
LOWscrapy/exporters.py30__all__ = [CODE
LOWscrapy/middleware.py29logger = logging.getLogger(__name__)CODE
LOWscrapy/middleware.py70 def _set_compat_spider(self, spider: Spider | None) -> None:CODE
LOWscrapy/crawler.py56logger = logging.getLogger(__name__)CODE
LOWscrapy/crawler.py149 def _update_root_log_handler(self) -> None:CODE
LOWscrapy/crawler.py767 def _setup_reactor(self, install_signal_handlers: bool) -> None:CODE
LOWscrapy/linkextractors/lxmlhtml.py34logger = logging.getLogger(__name__)CODE
LOWscrapy/linkextractors/__init__.py130__all__ = [CODE
LOWscrapy/downloadermiddlewares/cookies.py29logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/robotstxt.py33logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/redirect.py27logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/offsite.py20logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/offsite.py73 def _update_host_regex(self, spider: Spider) -> None:CODE
LOWscrapy/downloadermiddlewares/httpcache.py32logger = logging.getLogger(__name__)CODE
LOWscrapy/settings/default_settings.py21__all__ = [CODE
LOWscrapy/selector/unified.py12__all__ = ["Selector", "SelectorList"]CODE
LOWscrapy/selector/__init__.py8__all__ = [CODE
LOWscrapy/core/scraper.py55logger = logging.getLogger(__name__)CODE
LOWscrapy/core/spidermw.py38logger = logging.getLogger(__name__)CODE
LOWscrapy/core/engine.py62logger = logging.getLogger(__name__)CODE
LOWscrapy/core/scheduler.py30logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/tls.py34logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/_base_streaming.py44logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/__init__.py30logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/http11.py80logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/http.py17__all__ = [CODE
LOWscrapy/core/_http2/protocol.py52logger = logging.getLogger(__name__)CODE
LOWscrapy/core/_http2/stream.py35logger = logging.getLogger(__name__)CODE
LOWscrapy/spiders/sitemap.py23logger = logging.getLogger(__name__)CODE
LOWscrapy/spiders/__init__.py84 def _set_crawler(self, crawler: Crawler) -> None:CODE
LOWscrapy/spiders/__init__.py167 def update_settings(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/spiders/__init__.py190__all__ = [CODE
LOWscrapy/pipelines/files.py60logger = logging.getLogger(__name__)CODE
LOWscrapy/pipelines/files.py522 def _update_stores(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/pipelines/media.py54logger = logging.getLogger(__name__)CODE
LOWscrapy/pipelines/images.py132 def _update_stores(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/utils/reactor.py98def set_asyncio_event_loop_policy() -> None:CODE
LOWscrapy/utils/signal.py33logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/log.py28logger = logging.getLogger(__name__)CODE
38 more matches not shown…
Excessive Try-Catch Wrapping60 hits · 64 pts
SeverityFileLineSnippetContext
LOWtests/test_downloader_handler_httpx.py59except Exception:CODE
LOWtests/test_downloadermiddleware_httpcache.py330 except Exception:CODE
LOWtests/test_pipeline_media.py143 except Exception as exc:CODE
LOW…awlerProcess/twisted_reactor_custom_settings_select.py23 except Exception:CODE
LOW…sts/AsyncCrawlerProcess/reactorless_custom_settings.py29 except Exception:CODE
LOWscrapy/crawler.py286 except Exception:CODE
LOWscrapy/crawler.py320 except Exception:CODE
LOWscrapy/crawler.py565 except Exception:CODE
LOWscrapy/crawler.py676 except Exception:CODE
LOWscrapy/downloadermiddlewares/robotstxt.py102 except Exception as e:CODE
LOWscrapy/downloadermiddlewares/httpcache.py88 except Exception:CODE
LOWscrapy/core/scraper.py229 except Exception:CODE
LOWscrapy/core/scraper.py261 except Exception:CODE
LOWscrapy/core/scraper.py270 except Exception as spider_exc:CODE
LOWscrapy/core/scraper.py293 except Exception:CODE
LOWscrapy/core/scraper.py529 except Exception as ex:CODE
LOWscrapy/core/spidermw.py96 except Exception:CODE
LOWscrapy/core/spidermw.py110 except Exception as ex:CODE
LOWscrapy/core/spidermw.py226 except Exception as ex:CODE
LOWscrapy/core/engine.py152 except Exception:CODE
LOWscrapy/core/engine.py293 except Exception as exception:CODE
LOWscrapy/core/engine.py342 except Exception:CODE
LOWscrapy/core/engine.py436 except Exception:CODE
LOWscrapy/core/engine.py657 except Exception:CODE
LOWscrapy/core/engine.py662 except Exception:CODE
LOWscrapy/core/engine.py669 except Exception:CODE
LOWscrapy/core/engine.py678 except Exception:CODE
LOWscrapy/core/engine.py689 except Exception:CODE
LOWscrapy/core/engine.py708 except Exception:CODE
LOWscrapy/core/engine.py722 except Exception:CODE
LOWscrapy/core/downloader/__init__.py221 except Exception:CODE
LOWscrapy/core/downloader/__init__.py242 except Exception:CODE
LOWscrapy/core/downloader/middleware.py83 except Exception as ex:CODE
LOWscrapy/core/downloader/handlers/_base_streaming.py222 except Exception as e:CODE
LOWscrapy/core/downloader/handlers/__init__.py122 except Exception as ex:CODE
LOWscrapy/contracts/__init__.py57 except Exception:CODE
LOWscrapy/contracts/__init__.py91 except Exception:CODE
LOWscrapy/contracts/__init__.py151 except Exception:CODE
LOWscrapy/contracts/__init__.py207 except Exception:CODE
LOWscrapy/pipelines/files.py428 except Exception:CODE
LOWscrapy/pipelines/files.py680 except Exception as exc:CODE
LOWscrapy/pipelines/media.py211 except Exception:CODE
LOWscrapy/pipelines/media.py234 except Exception:CODE
LOWscrapy/utils/deprecate.py131 except Exception as e:CODE
LOWscrapy/utils/signal.py64 except Exception:CODE
LOWscrapy/utils/signal.py199 except Exception as ex:CODE
LOWscrapy/utils/asyncio.py206 except Exception:CODE
LOWscrapy/utils/engine.py36 except Exception as e:CODE
LOWscrapy/utils/defer.py151 except Exception:CODE
LOWscrapy/utils/defer.py357 except Exception:CODE
LOWscrapy/utils/defer.py376 except Exception:CODE
LOWscrapy/utils/ssl.py43 except Exception as ex:CODE
LOWscrapy/extensions/feedexport.py618 except Exception:CODE
LOWscrapy/extensions/remote_control.py180 except Exception:CODE
LOWscrapy/extensions/remote_control.py198 except Exception:CODE
LOWscrapy/extensions/remote_control.py221 except Exception:CODE
LOWscrapy/extensions/remote_control.py276 except Exception:CODE
LOWscrapy/extensions/httpcache.py416 except Exception:CODE
MEDIUMscrapy/commands/startproject.py56 print(f"Error: Module {project_name!r} already exists")CODE
MEDIUMscrapy/commands/startproject.py102 print(f"Error: scrapy.cfg already exists in {project_dir.resolve()}")CODE
Hallucination Indicators5 hits · 50 pts
SeverityFileLineSnippetContext
CRITICALtests/test_engine_loop.py372 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py413 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py462 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py475 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALscrapy/extensions/throttle.py100 return key, self.crawler.engine.downloader.slots.get(key)CODE
Over-Commented Block30 hits · 30 pts
SeverityFileLineSnippetContext
LOWconftest.py121COMMENT
LOWpyproject.toml301 # message wording varies across Python implementations and versionsCOMMENT
LOWtests/utils/bases/http_request.py141 url="http://www.scrapy.org/price/µ?currency=£", encoding="latin1"COMMENT
LOWtests/benchmarks/urls.txt1# Link targets for the URL benchmarks, as they would appear in the hrefCOMMENT
LOWdocs/requirements.txt1# This file was autogenerated by uv via the following command:COMMENT
LOWdocs/requirements.txt21constantly==23.10.4COMMENT
LOWdocs/requirements.txt41 # via tldextractCOMMENT
LOWdocs/requirements.txt61 # itemloadersCOMMENT
LOWdocs/requirements.txt81 # scrapyCOMMENT
LOWdocs/requirements.txt101 # -r requirements.inCOMMENT
LOWdocs/requirements.txt121 # via sphinxCOMMENT
LOWdocs/requirements.txt141 # sphinxcontrib-jqueryCOMMENT
LOWdocs/requirements.txt181 # via scrapyCOMMENT
LOWdocs/requirements.txt201 # scrapyCOMMENT
LOWdocs/conf.py121 r"\w+Contract\.(adjust_request_args|(pre|post)_process)$",COMMENT
LOWscrapy/shell.py41# official ways to use it:COMMENT
LOWscrapy/shell.py61# which initializes the engine but doesn't start processing of requests.COMMENT
LOWscrapy/shell.py81# Other architecture problems:COMMENT
LOWscrapy/crawler.py1001 # because crawl() returns a Task.COMMENT
LOWscrapy/crawler.py1021 # Workflow with Ctrl-C pressed once:COMMENT
LOWscrapy/core/_http2/protocol.py121COMMENT
LOWscrapy/pipelines/media.py261 # Response and by extension, the original Request) gets encapsulatedCOMMENT
LOWscrapy/http/request/__init__.py161COMMENT
LOWscrapy/http/request/__init__.py181 #: :meth:`~scrapy.Spider.parse` method of the spider must be used.COMMENT
LOWscrapy/http/request/__init__.py201 #:COMMENT
LOWscrapy/http/request/__init__.py221 #: duplicate request filtering (seeCOMMENT
LOWscrapy/http/request/__init__.py441 # We need to use __func__ to access the original function object because instanceCOMMENT
LOWscrapy/templates/project/module/spiders/__init__.py1# This package will contain the spiders of your Scrapy projectCOMMENT
LOW.github/workflows/flag-prs-for-triage.yml1name: Flag PRs for triageCOMMENT
LOW.github/workflows/flag-prs-for-triage.yml21# - Agent branch: the branch name carries an agent prefix.COMMENT
Decorative Section Separators8 hits · 24 pts
SeverityFileLineSnippetContext
MEDIUMtests/test_spidermiddleware_output_chain.py36# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py77# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py118# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py145# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py156# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py174# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py185# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py257# ================================================================================COMMENT
Deep Nesting21 hits · 21 pts
SeverityFileLineSnippetContext
LOWtests/test_downloadermiddleware_httpcache.py504CODE
LOWtests/utils/bases/engine.py104CODE
LOWdocs/utils/linkfix.py20CODE
LOWscrapy/pqueues.py191CODE
LOWscrapy/pqueues.py364CODE
LOWscrapy/downloadermiddlewares/httpproxy.py81CODE
LOWscrapy/selector/unified.py76CODE
LOWscrapy/core/_http2/protocol.py379CODE
LOWscrapy/core/_http2/stream.py217CODE
LOWscrapy/core/_http2/stream.py421CODE
LOWscrapy/spiders/feed.py78CODE
LOWscrapy/pipelines/media.py146CODE
LOWscrapy/pipelines/media.py317CODE
LOWscrapy/utils/sitemap.py107CODE
LOWscrapy/utils/sitemap.py62CODE
LOWscrapy/utils/request.py39CODE
LOWscrapy/utils/iterators.py21CODE
LOWscrapy/http/request/form.py173CODE
LOWscrapy/commands/check.py86CODE
LOWscrapy/commands/settings.py49CODE
LOWscrapy/commands/parse.py306CODE
Self-Referential Comments9 hits · 17 pts
SeverityFileLineSnippetContext
MEDIUMscrapy/core/scraper.py1"""This module implements the Scraper component which parses responses andSTRING
MEDIUMscrapy/core/spidermw.py261 # This method is only needed until _async compatibility methods are removed.COMMENT
MEDIUMscrapy/core/downloader/tls.py147 """This method is needed to override the verify callback."""STRING
MEDIUMscrapy/core/_http2/agent.py49 # Create a deferred which will fire with the H2ClientProtocolCOMMENT
MEDIUMscrapy/spiders/feed.py57 """This method is called for the nodes matching the provided tag nameSTRING
MEDIUMscrapy/spiders/feed.py70 """This method is called for the nodes matching the provided tag nameSTRING
MEDIUMscrapy/utils/trackref.py1"""This module provides some functions and classes to record and reportSTRING
MEDIUMscrapy/commands/shell.py98 # Create the engine and run start_async() in the main threadCOMMENT
MEDIUMscrapy/commands/shell.py104 # Create the engine and run start_async() in the event loop threadCOMMENT
Fake / Example Data11 hits · 16 pts
SeverityFileLineSnippetContext
LOWtests/test_downloadermiddleware_httpproxy.py287 request.meta["proxy"] = "https://user1:password1@example.org"CODE
LOWtests/test_downloadermiddleware_httpproxy.py304 request.meta["proxy"] = "https://user1:password1@example.org"CODE
LOWtests/test_downloadermiddleware_httpproxy.py327 request.meta["proxy"] = "https://user2:password2@example.org"CODE
LOWtests/test_item.py67 i["name"] = "John Doe"CODE
LOWtests/test_item.py71 assert itemrepr == "{'name': 'John Doe', 'number': 123}"CODE
LOWtests/test_item.py74 assert i2["name"] == "John Doe"CODE
LOWtests/test_settings/__init__.py687 dummy_config_value = settings.pop("DUMMY_CONFIG", "dummy_value")CODE
LOWtests/test_settings/__init__.py688 assert dummy_config_value == "dummy_value"CODE
LOWtests/test_settings/__init__.py692 {"DUMMY_CONFIG": "dummy_value", "OTHER_DUMMY_CONFIG": "other_dummy_value"}CODE
LOWtests/test_settings/__init__.py695 assert settings.pop("DUMMY_CONFIG") == "dummy_value"CODE
LOWscrapy/http/request/form.py70 formdata={"name": "John Doe", "age": "27"},STRING
Cross-File Repetition3 hits · 15 pts
SeverityFileLineSnippetContext
HIGHtests/test_command_runspider.py0import scrapy class myspider(scrapy.spider): name = 'myspider' async def start(self): self.logger.debug('it works!') retSTRING
HIGHtests/test_command_crawl.py0import scrapy class myspider(scrapy.spider): name = 'myspider' async def start(self): self.logger.debug('it works!') retSTRING
HIGHtests/test_commands.py0import scrapy class myspider(scrapy.spider): name = 'myspider' async def start(self): self.logger.debug('it works!') retSTRING
AI Structural Patterns10 hits · 10 pts
SeverityFileLineSnippetContext
LOWscrapy/mail.py64CODE
LOWscrapy/linkextractors/lxmlhtml.py305CODE
LOWscrapy/extensions/feedexport.py206CODE
LOWscrapy/http/response/__init__.py71CODE
LOWscrapy/http/response/__init__.py227CODE
LOWscrapy/http/response/__init__.py280CODE
LOWscrapy/http/response/text.py180CODE
LOWscrapy/http/response/text.py236CODE
LOWscrapy/http/request/__init__.py124CODE
LOWscrapy/http/request/form.py116CODE
Redundant / Tautological Comments6 hits · 9 pts
SeverityFileLineSnippetContext
LOWtests/test_http2_client_protocol.py401 # Check if headers were sent successfullyCOMMENT
LOWscrapy/core/_http2/stream.py393 # Check if we exceed the allowed max data size which can be receivedCOMMENT
LOWscrapy/core/_http2/agent.py55 # Check if we already have a connection to the remoteCOMMENT
LOWscrapy/pipelines/media.py193 # Check if request is downloading right now to avoid doing it twiceCOMMENT
LOWscrapy/utils/asyncio.py73 # Check if there is a running asyncio loop.COMMENT
LOWscrapy/utils/asyncio.py84 # Check if there is an installed asyncio reactor (it doesn't need to beCOMMENT
AI Slop Vocabulary4 hits · 5 pts
SeverityFileLineSnippetContext
LOWtests/mockserver/http_resources.py440 # just set this to trigger a test failure if no valid accept-encoding header was setCOMMENT
MEDIUMscrapy/spiders/feed.py50 """You can override this function in order to make any changes you wantSTRING
MEDIUMscrapy/utils/signal.py42 """Like ``pydispatch.robust.sendRobust()`` but it also logs errors and returnsSTRING
LOWscrapy/http/request/form.py296 # If we don't have clickdata, we just use the first clickable elementCOMMENT
Overly Generic Function Names3 hits · 2 pts
SeverityFileLineSnippetContext
LOW…ils_misc/test_return_with_argument_inside_generator.py82 def helper() -> int:STRING
LOW…ils_misc/test_return_with_argument_inside_generator.py136 def helper() -> int:STRING
LOW…ils_misc/test_return_with_argument_inside_generator.py202 def helper() -> int:STRING