Repository Analysis

scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

27.2 Moderate AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of scrapy/scrapy, a Python project with 63,130 GitHub stars. SynthScan v2.0 examined 83,426 lines of code across 484 source files, recording 1846 pattern matches distributed across 16 syntactic categories. The overall adjusted score of 27.2 places this repository in the Moderate AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

27.2
Adjusted Score
27.2
Raw Score
100%
Time Factor
2026-07-13
Last Push
63.1K
Stars
Python
Language
83.4K
Lines of Code
484
Files
1.8K
Pattern Hits
2026-07-14
Scan Date
0.12
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 10HIGH 47MEDIUM 83LOW 1706

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 1846 distinct pattern matches across 16 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers1196 hits · 1156 pts
SeverityFileLineSnippetContext
LOWtests/test_request_dict.py75 def test_callback_serialization(self):CODE
LOWtests/test_request_dict.py83 def test_reference_callback_serialization(self):CODE
LOWtests/test_request_dict.py94 def test_private_reference_callback_serialization(self):CODE
LOWtests/test_request_dict.py105 def test_private_callback_serialization(self):CODE
LOWtests/test_request_dict.py113 def test_mixin_private_callback_serialization(self):CODE
LOWtests/test_request_dict.py121 def test_delegated_callback_serialization(self):CODE
LOWtests/test_request_dict.py156 def test_callback_not_available(self):CODE
LOWtests/test_utils_signal.py140 def test_error_logged_if_deferred_not_supported(self):CODE
LOWtests/test_http_response_text.py154 def test_declared_encoding_invalid(self):CODE
LOWtests/test_http_response_text.py186 def test_bom_is_removed_from_body(self):CODE
LOWtests/test_http_response_text.py213 def test_replace_wrong_encoding(self):CODE
LOWtests/test_http_response_text.py264 def test_selector_shortcuts_kwargs(self):CODE
LOWtests/test_http_response_text.py285 def test_urljoin_with_base_url(self):CODE
LOWtests/test_http_response_text.py344 def test_follow_selector_list(self):CODE
LOWtests/test_http_response_text.py349 def test_follow_selector_invalid(self):CODE
LOWtests/test_http_response_text.py354 def test_follow_selector_attribute(self):CODE
LOWtests/test_http_response_text.py359 def test_follow_selector_no_href(self):CODE
LOWtests/test_http_response_text.py367 def test_follow_whitespace_selector(self):CODE
LOWtests/test_http_response_text.py433 def test_follow_all_css_skip_invalid(self):CODE
LOWtests/test_http_response_text.py454 def test_follow_all_xpath_skip_invalid(self):CODE
LOWtests/test_http_response_text.py471 def test_follow_all_too_many_arguments(self):CODE
LOWtests/test_http_response_text.py597 def test_selector_shortcuts_kwargs(self):CODE
LOWtests/test_contracts.py91 def returns_request_cb_kwargs(self, response, url):CODE
LOWtests/test_contracts.py107 def returns_item_cb_kwargs_error_unexpected_keyword(self, response):CODE
LOWtests/test_contracts.py115 def returns_item_cb_kwargs_error_missing_argument(self, response, arg):CODE
LOWtests/test_contracts.py136 def returns_request_range_fail(self, response):CODE
LOWtests/test_contracts.py198 def scrapes_multiple_missing_fields(self, response):CODE
LOWtests/test_contracts.py222 def invalid_regex_with_valid_contract(self, response):CODE
LOWtests/test_contracts.py246 def returns_error_missing_meta(self, response):CODE
LOWtests/test_contracts.py463 def test_returns_invalid_argument_count(self):CODE
LOWtests/test_contracts.py470 def test_returns_default_bounds(self):CODE
LOWtests/test_contracts.py487 def test_returns_and_scrapes_ignore_other_types(self):CODE
LOWtests/test_contracts.py607 def test_custom_tagged_request_contract(self):CODE
LOWtests/test_contracts.py677 def test_pre_hook_keyboard_interrupt(self):CODE
LOWtests/test_contracts.py691 def test_post_hook_keyboard_interrupt(self):CODE
LOWtests/test_contracts.py718 def test_pre_hook_assertion_failure(self):CODE
LOWtests/test_contracts.py743 def test_pre_hook_async_callback(self):CODE
LOWtests/test_contracts.py753 def test_pre_hook_async_generator(self):CODE
LOWtests/test_contracts.py763 def test_post_hook_async_generator(self):CODE
LOWtests/test_command_parse.py115 def parse_request_with_cb_kwargs(self, response, foo=None, key=None):CODE
LOWtests/test_command_parse.py121 def parse_request_without_meta(self, response):CODE
LOWtests/test_command_parse.py232 def test_request_with_cb_kwargs(STRING
LOWtests/test_command_parse.py254 def test_request_without_meta(STRING
LOWtests/test_command_parse.py283 def test_async_def_asyncio_parse_items_list(STRING
LOWtests/test_command_parse.py299 def test_async_def_asyncio_parse_items_single_element(STRING
LOWtests/test_command_parse.py314 def test_async_def_asyncgen_parse_loop(STRING
LOWtests/test_command_parse.py330 def test_async_def_asyncgen_parse_exc(STRING
LOWtests/test_command_parse.py346 def test_async_def_asyncio_parse(STRING
LOWtests/test_command_parse.py372 def test_parse_items_no_callback_passed(STRING
LOWtests/test_command_parse.py384 def test_wrong_callback_passed(STRING
LOWtests/test_command_parse.py399 def test_crawlspider_matching_rule_callback_set(STRING
LOWtests/test_command_parse.py413 def test_crawlspider_matching_rule_default_callback(CODE
LOWtests/test_command_parse.py427 def test_spider_with_no_rules_attribute(CODE
LOWtests/test_command_parse.py442 def test_crawlspider_missing_callback(CODE
LOWtests/test_command_parse.py455 def test_crawlspider_no_matching_rule(CODE
LOWtests/test_command_parse.py470 def test_crawlspider_not_exists_with_not_matched_url(CODE
LOWtests/test_logstats.py59 def test_stats_calculations_no_time(self):CODE
LOWtests/test_logstats.py68 def test_stats_calculation_no_elapsed_time(self):CODE
LOWtests/test_http_request.py247 def test_copy_inherited_classes(self):CODE
LOWtests/test_http_request.py291 def test_immutable_attributes(self):CODE
1136 more matches not shown…
Cross-Language Confusion47 hits · 298 pts
SeverityFileLineSnippetContext
HIGHtests/test_squeues_request.py54 q.push(req)CODE
HIGHtests/test_squeues_request.py87 q.push(req1)CODE
HIGHtests/test_squeues_request.py88 q.push(req2)CODE
HIGHtests/test_squeues_request.py89 q.push(req3)CODE
HIGHtests/test_pqueues.py65 queue.push(req1)CODE
HIGHtests/test_pqueues.py66 queue.push(req2)CODE
HIGHtests/test_pqueues.py67 queue.push(req3)CODE
HIGHtests/test_pqueues.py112 queue.push(req1)CODE
HIGHtests/test_pqueues.py113 queue.push(req2)CODE
HIGHtests/test_pqueues.py114 queue.push(req3)CODE
HIGHtests/test_pqueues.py142 self.queue.push(req1)CODE
HIGHtests/test_pqueues.py143 self.queue.push(req2)CODE
HIGHtests/test_pqueues.py144 self.queue.push(req3)CODE
HIGHtests/test_pqueues.py171 self.queue.push(req1)CODE
HIGHtests/test_pqueues.py172 self.queue.push(req2)CODE
HIGHtests/test_pqueues.py173 self.queue.push(req3)CODE
HIGHtests/test_pqueues.py30 queue.push(req1)CODE
HIGHtests/test_pqueues.py45 queue.push(Request("https://example.org"))CODE
HIGHtests/test_pqueues.py88 queue.push(req)CODE
HIGHtests/test_pqueues.py157 self.queue.push(Request("https://example.org"))CODE
HIGHtests/test_pqueues.py198 self.queue.push(request)CODE
HIGHtests/test_pqueues.py222 self.queue.push(request)CODE
HIGHtests/test_pqueues.py244 self.queue.push(req)CODE
HIGHtests/test_pqueues.py256 self.queue.push(req)CODE
HIGHtests/test_pqueues.py305 queue.push(request)CODE
HIGHtests/test_squeues.py38 q.push(lambda x: x)CODE
HIGHtests/test_squeues.py44 q.push(sel)CODE
HIGHtests/test_squeues.py51 q.push("a")CODE
HIGHtests/test_squeues.py52 q.push(123)CODE
HIGHtests/test_squeues.py53 q.push({"a": "dict"})CODE
HIGHtests/test_squeues.py94 q.push(i)CODE
HIGHtests/test_squeues.py103 q.push(loader)CODE
HIGHtests/test_squeues.py114 q.push(r)CODE
HIGHtests/test_squeues.py127 q.push(lambda x: x)CODE
HIGHtests/test_squeues.py136 q.push(sel)CODE
HIGHtests/test_squeues.py160 q.push("a")CODE
HIGHtests/test_squeues.py161 q.push(123)CODE
HIGHtests/test_squeues.py162 q.push({"a": "dict"})CODE
HIGHtests/test_squeues.py183 q.push(i)CODE
HIGHtests/test_squeues.py192 q.push(loader)CODE
HIGHtests/test_squeues.py203 q.push(r)CODE
HIGHtests/test_settings/__init__.py955 {"FOO": '{"tests.test_settings.Component1": null}'},CODE
HIGHscrapy/pqueues.py185 q.push(request) # this may fail (eg. serialization error)CODE
HIGHscrapy/pqueues.py419 queue.push(request)CODE
HIGHscrapy/core/scheduler.py418 self.dqs.push(request)STRING
HIGHscrapy/core/scheduler.py439 self.mqs.push(request)STRING
HIGHscrapy/spidermiddlewares/referer.py74 If url is null, return no referrer.STRING
Unused Imports295 hits · 291 pts
SeverityFileLineSnippetContext
LOWconftest.py1CODE
LOWconftest.py54CODE
LOWtests/test_utils_signal.py1CODE
LOWtests/test_http_response_text.py1CODE
LOWtests/test_command_parse.py1CODE
LOWtests/test_command_genspider.py1CODE
LOWtests/test_engine.py1CODE
LOWtests/test_engine.py45CODE
LOWtests/test_zz_resources.py3CODE
LOWtests/test_spidermiddleware.py1CODE
LOWtests/test_scrapy__getattr__.py1CODE
LOWtests/test_downloader_handler_twisted_http11.py3CODE
LOWtests/test_stats.py1CODE
LOWtests/test_linkextractors.py1CODE
LOWtests/test_command_runspider.py1CODE
LOWtests/test_loader.py1CODE
LOWtests/test_middleware.py1CODE
LOWtests/test_http_request_json.py1CODE
LOWtests/test_http2_client_protocol.py1CODE
LOWtests/test_utils_iterators.py1CODE
LOWtests/test_dependencies.py1CODE
LOWtests/test_signals.py1CODE
LOWtests/test_utils_log.py1CODE
LOWtests/test_spider_sitemap.py1CODE
LOWtests/test_spidermiddleware_urllength.py1CODE
LOWtests/test_downloader_handler_twisted_http2.py3CODE
LOWtests/test_feedexport_postprocess.py1CODE
LOWtests/test_feedexport_uri_params.py1CODE
LOWtests/test_spidermiddleware_referer.py1CODE
LOWtests/spiders.py5CODE
LOWtests/test_downloadermiddleware_robotstxt.py1CODE
LOWtests/test_crawler_subprocess.py1CODE
LOWtests/test_robotstxt_interface.py15CODE
LOWtests/test_engine_stop_download_bytes.py1CODE
LOWtests/test_http_request_form.py1CODE
LOWtests/test_core_downloader.py1CODE
LOWtests/test_core_downloader.py39CODE
LOWtests/test_spidermiddleware_start.py1CODE
LOWtests/test_http_cookies.py1CODE
LOWtests/test_scheduler_base.py1CODE
LOWtests/test_downloadermiddleware.py1CODE
LOWtests/test_downloadermiddleware_useragent.py1CODE
LOWtests/test_spider.py1CODE
LOWtests/test_squeues_request.py5CODE
LOWtests/test_spidermiddleware_base.py1CODE
LOWtests/test_scheduler.py1CODE
LOWtests/test_http_response.py1CODE
LOWtests/test_downloader_handler_httpx.py3CODE
LOWtests/test_extension_periodic_log.py1CODE
LOWtests/test_utils_defer.py1CODE
LOWtests/test_spider_crawl.py1CODE
LOWtests/test_downloadermiddleware_httpcompression.py69CODE
LOWtests/test_downloadermiddleware_httpcompression.py230CODE
LOWtests/test_extension_memusage.py1CODE
LOWtests/test_crawl.py1CODE
LOWtests/test_downloadermiddleware_redirect_metarefresh.py1CODE
LOWtests/test_downloader_handlers.py3CODE
LOWtests/test_downloader_handlers.py200CODE
LOWtests/test_downloader_handlers_http_base.py3CODE
LOWtests/test_utils_python.py1CODE
235 more matches not shown…
Hallucination Indicators10 hits · 125 pts
SeverityFileLineSnippetContext
CRITICALtests/test_engine_loop.py38 self.crawler.engine._slot.scheduler.pause()CODE
CRITICALtests/test_engine_loop.py39 self.crawler.engine._slot.scheduler.enqueue_request(Request("data:,b"))CODE
CRITICALtests/test_engine_loop.py45 self.crawler.engine._slot.scheduler.unpause()CODE
CRITICALtests/test_engine_loop.py54 self.crawler.engine._slot.scheduler.pause()CODE
CRITICALtests/test_engine_loop.py55 self.crawler.engine._slot.scheduler.enqueue_request(Request("data:,d"))CODE
CRITICALtests/test_engine_loop.py203 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py243 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py291 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALtests/test_engine_loop.py304 spider.crawler.engine._slot.scheduler.enqueue_request(request)CODE
CRITICALscrapy/extensions/throttle.py102 return key, self.crawler.engine.downloader.slots.get(key)CODE
Dead Code62 hits · 124 pts
SeverityFileLineSnippetContext
MEDIUMtests/test_engine.py630CODE
MEDIUMtests/test_spidermiddleware.py77CODE
MEDIUMtests/test_utils_python.py41CODE
MEDIUMtests/test_crawler.py865CODE
MEDIUMtests/test_crawler.py201CODE
MEDIUMtests/test_crawler.py279CODE
MEDIUMtests/test_crawler.py357CODE
MEDIUMtests/test_crawler.py435CODE
MEDIUMtests/test_crawler.py537CODE
MEDIUMtests/test_engine_loop.py210CODE
MEDIUMtests/test_engine_loop.py250CODE
MEDIUMtests/test_engine_loop.py308CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_simple.py15CODE
MEDIUMtests/AsyncCrawlerRunner/custom_loop_same.py20CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_custom_settings.py18CODE
MEDIUMtests/AsyncCrawlerRunner/custom_loop_different.py20CODE
MEDIUMtests/AsyncCrawlerRunner/multi_parallel.py15CODE
MEDIUMtests/AsyncCrawlerRunner/reactorless_reactor.py14CODE
MEDIUMtests/AsyncCrawlerRunner/no_reactor.py13CODE
MEDIUMtests/AsyncCrawlerRunner/simple_default_reactor.py14CODE
MEDIUMtests/AsyncCrawlerRunner/simple.py17CODE
MEDIUMtests/AsyncCrawlerRunner/multi_seq.py15CODE
MEDIUMtests/CrawlerProcess/asyncio_enabled_no_reactor.py17CODE
MEDIUMtests/CrawlerProcess/args_settings.py19CODE
MEDIUMtests/CrawlerProcess/reactor_select.py15CODE
MEDIUMtests/CrawlerProcess/multi.py10CODE
MEDIUMtests/CrawlerProcess/asyncio_enabled_reactor.py53CODE
MEDIUM…ts/CrawlerProcess/asyncio_enabled_reactor_same_loop.py22CODE
MEDIUM…awlerProcess/reactor_default_twisted_reactor_select.py13CODE
MEDIUM…awlerProcess/asyncio_enabled_reactor_different_loop.py20CODE
MEDIUM…cess/reactor_select_subclass_twisted_reactor_select.py22CODE
MEDIUMtests/CrawlerProcess/dns_resolver_deprecated.py13CODE
MEDIUMtests/CrawlerProcess/reactor_default.py13CODE
MEDIUMtests/CrawlerProcess/simple.py12CODE
MEDIUM…rawlerProcess/reactor_select_twisted_reactor_select.py14CODE
MEDIUMtests/CrawlerProcess/asyncio_custom_loop.py10CODE
MEDIUMtests/CrawlerRunner/explicit_default_reactor.py17CODE
MEDIUMtests/CrawlerRunner/custom_loop_same.py19CODE
MEDIUMtests/CrawlerRunner/change_reactor.py15CODE
MEDIUMtests/CrawlerRunner/custom_loop_different.py19CODE
MEDIUMtests/CrawlerRunner/multi_parallel.py14CODE
MEDIUMtests/CrawlerRunner/no_reactor.py13CODE
MEDIUMtests/CrawlerRunner/simple.py16CODE
MEDIUMtests/CrawlerRunner/multi_seq.py15CODE
MEDIUMtests/AsyncCrawlerProcess/asyncio_enabled_no_reactor.py17CODE
MEDIUMtests/AsyncCrawlerProcess/reactorless_simple.py12CODE
MEDIUM…erProcess/asyncio_enabled_reactor_same_loop_default.py21CODE
MEDIUMtests/AsyncCrawlerProcess/args_settings.py19CODE
MEDIUM…sts/AsyncCrawlerProcess/reactorless_custom_settings.py23CODE
MEDIUMtests/AsyncCrawlerProcess/multi.py10CODE
MEDIUMtests/AsyncCrawlerProcess/asyncio_enabled_reactor.py43CODE
MEDIUM…yncCrawlerProcess/asyncio_enabled_reactor_same_loop.py21CODE
MEDIUM…ncCrawlerProcess/reactorless_telnetconsole_disabled.py10CODE
MEDIUM…ocess/asyncio_custom_loop_custom_settings_different.py13CODE
MEDIUM…lerProcess/asyncio_custom_loop_custom_settings_same.py13CODE
MEDIUM…awlerProcess/asyncio_enabled_reactor_different_loop.py19CODE
MEDIUM…yncCrawlerProcess/reactorless_telnetconsole_enabled.py10CODE
MEDIUMtests/AsyncCrawlerProcess/reactorless_import_hook.py12CODE
MEDIUMtests/AsyncCrawlerProcess/dns_resolver_deprecated.py13CODE
MEDIUMtests/AsyncCrawlerProcess/reactor_default.py12CODE
2 more matches not shown…
Modern Structural Boilerplate85 hits · 86 pts
SeverityFileLineSnippetContext
LOWtests/test_scheduler.py306 def _setup(scheduler: Scheduler) -> None:CODE
LOWtests/test_extension_periodic_log.py61 def set_a(self) -> None:CODE
LOWtests/test_extension_periodic_log.py64 def set_b(self) -> None:CODE
LOWscrapy/mail.py39logger = logging.getLogger(__name__)CODE
LOWscrapy/robotstxt.py21logger = logging.getLogger(__name__)CODE
LOWscrapy/addons.py15logger = logging.getLogger(__name__)CODE
LOWscrapy/pqueues.py19logger = logging.getLogger(__name__)CODE
LOWscrapy/pqueues.py219 def _update_curprio(self) -> None:CODE
LOWscrapy/__init__.py15__all__ = [CODE
LOWscrapy/statscollectors.py18logger = logging.getLogger(__name__)CODE
LOWscrapy/statscollectors.py65 def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py68 def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py119 def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None:CODE
LOWscrapy/statscollectors.py122 def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None:CODE
LOWscrapy/exporters.py28logger = logging.getLogger(__name__)CODE
LOWscrapy/exporters.py30__all__ = [CODE
LOWscrapy/middleware.py29logger = logging.getLogger(__name__)CODE
LOWscrapy/middleware.py70 def _set_compat_spider(self, spider: Spider | None) -> None:CODE
LOWscrapy/crawler.py52logger = logging.getLogger(__name__)CODE
LOWscrapy/crawler.py89 def _update_root_log_handler(self) -> None:CODE
LOWscrapy/crawler.py667 def _setup_reactor(self, install_signal_handlers: bool) -> None:CODE
LOWscrapy/linkextractors/lxmlhtml.py34logger = logging.getLogger(__name__)CODE
LOWscrapy/linkextractors/__init__.py130__all__ = [CODE
LOWscrapy/downloadermiddlewares/cookies.py29logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/robotstxt.py31logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/redirect.py27logger = logging.getLogger(__name__)CODE
LOWscrapy/downloadermiddlewares/offsite.py20logger = logging.getLogger(__name__)CODE
LOWscrapy/settings/default_settings.py21__all__ = [CODE
LOWscrapy/selector/unified.py12__all__ = ["Selector", "SelectorList"]CODE
LOWscrapy/selector/__init__.py8__all__ = [CODE
LOWscrapy/core/scraper.py51logger = logging.getLogger(__name__)CODE
LOWscrapy/core/spidermw.py38logger = logging.getLogger(__name__)CODE
LOWscrapy/core/engine.py62logger = logging.getLogger(__name__)CODE
LOWscrapy/core/scheduler.py30logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/tls.py34logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/_base_streaming.py44logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/__init__.py30logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/http11.py71logger = logging.getLogger(__name__)CODE
LOWscrapy/core/downloader/handlers/http.py17__all__ = [CODE
LOWscrapy/core/http2/protocol.py51logger = logging.getLogger(__name__)CODE
LOWscrapy/core/http2/stream.py31logger = logging.getLogger(__name__)CODE
LOWscrapy/spiders/sitemap.py23logger = logging.getLogger(__name__)CODE
LOWscrapy/spiders/__init__.py77 def _set_crawler(self, crawler: Crawler) -> None:CODE
LOWscrapy/spiders/__init__.py144 def update_settings(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/spiders/__init__.py167__all__ = [CODE
LOWscrapy/pipelines/files.py54logger = logging.getLogger(__name__)CODE
LOWscrapy/pipelines/files.py510 def _update_stores(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/pipelines/media.py55logger = logging.getLogger(__name__)CODE
LOWscrapy/pipelines/images.py137 def _update_stores(cls, settings: BaseSettings) -> None:CODE
LOWscrapy/utils/reactor.py98def set_asyncio_event_loop_policy() -> None:CODE
LOWscrapy/utils/signal.py33logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/log.py25logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/asyncio.py31logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/request.py31logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/_download_handlers.py47 def set_cookie(self, cookie: Cookie) -> None:CODE
LOWscrapy/utils/iterators.py20logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/ssl.py27logger = logging.getLogger(__name__)CODE
LOWscrapy/utils/spider.py21logger = logging.getLogger(__name__)CODE
LOWscrapy/extensions/telnet.py35logger = logging.getLogger(__name__)CODE
LOWscrapy/extensions/memusage.py32logger = logging.getLogger(__name__)CODE
25 more matches not shown…
Excessive Try-Catch Wrapping54 hits · 58 pts
SeverityFileLineSnippetContext
LOWtests/test_downloadermiddleware_httpcache.py263 except Exception:CODE
LOWtests/test_pipeline_media.py114 except Exception as exc:CODE
LOW…awlerProcess/twisted_reactor_custom_settings_select.py23 except Exception:CODE
LOW…sts/AsyncCrawlerProcess/reactorless_custom_settings.py29 except Exception:CODE
LOWscrapy/crawler.py194 except Exception:CODE
LOWscrapy/crawler.py224 except Exception:CODE
LOWscrapy/crawler.py470 except Exception:CODE
LOWscrapy/crawler.py581 except Exception:CODE
LOWscrapy/downloadermiddlewares/robotstxt.py102 except Exception as e:CODE
LOWscrapy/core/scraper.py226 except Exception:CODE
LOWscrapy/core/scraper.py258 except Exception:CODE
LOWscrapy/core/scraper.py267 except Exception as spider_exc:CODE
LOWscrapy/core/scraper.py290 except Exception:CODE
LOWscrapy/core/scraper.py513 except Exception as ex:CODE
LOWscrapy/core/spidermw.py96 except Exception:CODE
LOWscrapy/core/spidermw.py110 except Exception as ex:CODE
LOWscrapy/core/spidermw.py226 except Exception as ex:CODE
LOWscrapy/core/engine.py150 except Exception:CODE
LOWscrapy/core/engine.py280 except Exception as exception:CODE
LOWscrapy/core/engine.py320 except Exception:CODE
LOWscrapy/core/engine.py414 except Exception:CODE
LOWscrapy/core/engine.py618 except Exception:CODE
LOWscrapy/core/engine.py623 except Exception:CODE
LOWscrapy/core/engine.py630 except Exception:CODE
LOWscrapy/core/engine.py639 except Exception:CODE
LOWscrapy/core/engine.py650 except Exception:CODE
LOWscrapy/core/engine.py671 except Exception:CODE
LOWscrapy/core/engine.py685 except Exception:CODE
LOWscrapy/core/downloader/__init__.py238 except Exception:CODE
LOWscrapy/core/downloader/__init__.py259 except Exception:CODE
LOWscrapy/core/downloader/middleware.py83 except Exception as ex:CODE
LOWscrapy/core/downloader/handlers/_base_streaming.py222 except Exception as e:CODE
LOWscrapy/core/downloader/handlers/__init__.py110 except Exception as ex:CODE
LOWscrapy/contracts/__init__.py48 except Exception:CODE
LOWscrapy/contracts/__init__.py82 except Exception:CODE
LOWscrapy/contracts/__init__.py135 except Exception:CODE
LOWscrapy/contracts/__init__.py191 except Exception:CODE
LOWscrapy/pipelines/files.py416 except Exception:CODE
LOWscrapy/pipelines/files.py667 except Exception as exc:CODE
LOWscrapy/pipelines/media.py194 except Exception:CODE
LOWscrapy/pipelines/media.py216 except Exception:CODE
LOWscrapy/utils/deprecate.py131 except Exception as e:CODE
LOWscrapy/utils/signal.py64 except Exception:CODE
LOWscrapy/utils/signal.py199 except Exception as ex:CODE
LOWscrapy/utils/asyncio.py206 except Exception:CODE
LOWscrapy/utils/engine.py36 except Exception as e:CODE
LOWscrapy/utils/defer.py154 except Exception:CODE
LOWscrapy/utils/defer.py360 except Exception:CODE
LOWscrapy/utils/defer.py379 except Exception:CODE
LOWscrapy/utils/ssl.py43 except Exception as ex:CODE
LOWscrapy/extensions/feedexport.py595 except Exception:CODE
LOWscrapy/extensions/httpcache.py418 except Exception:CODE
MEDIUMscrapy/commands/startproject.py56 print(f"Error: Module {project_name!r} already exists")CODE
MEDIUMscrapy/commands/startproject.py102 print(f"Error: scrapy.cfg already exists in {project_dir.resolve()}")CODE
Over-Commented Block26 hits · 26 pts
SeverityFileLineSnippetContext
LOWtests/test_http_request.py141 # percent-escaping sequences that do not match valid UTF-8 sequencesCOMMENT
LOWdocs/requirements.txt1# This file was autogenerated by uv via the following command:COMMENT
LOWdocs/requirements.txt21constantly==23.10.4COMMENT
LOWdocs/requirements.txt41h2==4.3.0COMMENT
LOWdocs/requirements.txt61 # scrapyCOMMENT
LOWdocs/requirements.txt81 # scrapy-spider-metadataCOMMENT
LOWdocs/requirements.txt101 # scrapy-spider-metadataCOMMENT
LOWdocs/requirements.txt121scrapy==2.14.2COMMENT
LOWdocs/requirements.txt141 # via sphinx-scrapyCOMMENT
LOWdocs/requirements.txt181 # viaCOMMENT
LOWdocs/conf.py101COMMENT
LOWdocs/conf.py121 # Base classes of downloader middlewares are implementation details thatCOMMENT
LOWscrapy/shell.py41# official ways to use it:COMMENT
LOWscrapy/shell.py61# which initializes the engine but doesn't start processing of requests.COMMENT
LOWscrapy/shell.py81# Other architecture problems:COMMENT
LOWscrapy/cmdline.py241 # Twisted prints errors in DebugInfo.__del__, but PyPy does not run gc.collect() on exit:COMMENT
LOWscrapy/crawler.py901 # So we reproduce a part of asyncio.runners.Runner that is useful to us.COMMENT
LOWscrapy/crawler.py921 # 1. While loop.run_until_complete() blocks, _signal_shutdown_reactorless()COMMENT
LOWscrapy/core/downloader/contextfactory.py221 #COMMENT
LOWscrapy/core/http2/protocol.py121COMMENT
LOWscrapy/pipelines/media.py241 # the Response status code is not 200 OK, is that the originalCOMMENT
LOWscrapy/http/request/__init__.py161 )COMMENT
LOWscrapy/http/request/__init__.py181 #: processing, i.e. by a :ref:`spider middlewareCOMMENT
LOWscrapy/http/request/__init__.py201 #: .. seealso:: :ref:`topics-request-response-ref-errbacks`COMMENT
LOWscrapy/http/request/__init__.py221 #: to be scheduled multiple times.COMMENT
LOWscrapy/templates/project/module/spiders/__init__.py1# This package will contain the spiders of your Scrapy projectCOMMENT
Decorative Section Separators8 hits · 24 pts
SeverityFileLineSnippetContext
MEDIUMtests/test_spidermiddleware_output_chain.py25# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py65# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py104# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py130# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py141# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py158# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py169# ================================================================================COMMENT
MEDIUMtests/test_spidermiddleware_output_chain.py240# ================================================================================COMMENT
Deep Nesting21 hits · 21 pts
SeverityFileLineSnippetContext
LOWtests/test_engine.py324CODE
LOWtests/test_downloadermiddleware_httpcache.py436CODE
LOWdocs/utils/linkfix.py20CODE
LOWscrapy/pqueues.py189CODE
LOWscrapy/pqueues.py363CODE
LOWscrapy/downloadermiddlewares/httpproxy.py81CODE
LOWscrapy/core/http2/protocol.py376CODE
LOWscrapy/core/http2/stream.py209CODE
LOWscrapy/core/http2/stream.py397CODE
LOWscrapy/spiders/feed.py78CODE
LOWscrapy/pipelines/media.py129CODE
LOWscrapy/utils/sitemap.py107CODE
LOWscrapy/utils/sitemap.py62CODE
LOWscrapy/utils/request.py39CODE
LOWscrapy/utils/iterators.py23CODE
LOWscrapy/utils/iterators.py81CODE
LOWscrapy/utils/ssl.py156CODE
LOWscrapy/http/request/form.py173CODE
LOWscrapy/commands/check.py73CODE
LOWscrapy/commands/settings.py49CODE
LOWscrapy/commands/parse.py305CODE
Self-Referential Comments9 hits · 17 pts
SeverityFileLineSnippetContext
MEDIUMscrapy/core/scraper.py1"""This module implements the Scraper component which parses responses andSTRING
MEDIUMscrapy/core/spidermw.py249 # This method is only needed until _async compatibility methods are removed.COMMENT
MEDIUMscrapy/core/downloader/tls.py147 """This method is needed to override the verify callback."""STRING
MEDIUMscrapy/core/http2/agent.py55 # Create a deferred which will fire with the H2ClientProtocolCOMMENT
MEDIUMscrapy/spiders/feed.py57 """This method is called for the nodes matching the provided tag nameSTRING
MEDIUMscrapy/spiders/feed.py70 """This method is called for the nodes matching the provided tag nameSTRING
MEDIUMscrapy/utils/trackref.py1"""This module provides some functions and classes to record and reportSTRING
MEDIUMscrapy/commands/shell.py98 # Create the engine and run start_async() in the main threadCOMMENT
MEDIUMscrapy/commands/shell.py104 # Create the engine and run start_async() in the event loop threadCOMMENT
Fake / Example Data11 hits · 16 pts
SeverityFileLineSnippetContext
LOWtests/test_downloadermiddleware_httpproxy.py284 request.meta["proxy"] = "https://user1:password1@example.org"CODE
LOWtests/test_downloadermiddleware_httpproxy.py301 request.meta["proxy"] = "https://user1:password1@example.org"CODE
LOWtests/test_downloadermiddleware_httpproxy.py324 request.meta["proxy"] = "https://user2:password2@example.org"CODE
LOWtests/test_item.py69 i["name"] = "John Doe"CODE
LOWtests/test_item.py73 assert itemrepr == "{'name': 'John Doe', 'number': 123}"CODE
LOWtests/test_item.py76 assert i2["name"] == "John Doe"CODE
LOWtests/test_settings/__init__.py689 dummy_config_value = settings.pop("DUMMY_CONFIG", "dummy_value")CODE
LOWtests/test_settings/__init__.py690 assert dummy_config_value == "dummy_value"CODE
LOWtests/test_settings/__init__.py694 {"DUMMY_CONFIG": "dummy_value", "OTHER_DUMMY_CONFIG": "other_dummy_value"}CODE
LOWtests/test_settings/__init__.py697 assert settings.pop("DUMMY_CONFIG") == "dummy_value"CODE
LOWscrapy/http/request/form.py70 formdata={"name": "John Doe", "age": "27"},STRING
Redundant / Tautological Comments6 hits · 9 pts
SeverityFileLineSnippetContext
LOWtests/test_http2_client_protocol.py381 # Check if headers were sent successfullyCOMMENT
LOWscrapy/core/http2/stream.py372 # Check if we exceed the allowed max data size which can be receivedCOMMENT
LOWscrapy/core/http2/agent.py61 # Check if we already have a connection to the remoteCOMMENT
LOWscrapy/pipelines/media.py176 # Check if request is downloading right now to avoid doing it twiceCOMMENT
LOWscrapy/utils/asyncio.py73 # Check if there is a running asyncio loop.COMMENT
LOWscrapy/utils/asyncio.py84 # Check if there is an installed asyncio reactor (it doesn't need to beCOMMENT
AI Structural Patterns9 hits · 9 pts
SeverityFileLineSnippetContext
LOWscrapy/mail.py62CODE
LOWscrapy/linkextractors/lxmlhtml.py299CODE
LOWscrapy/http/response/__init__.py71CODE
LOWscrapy/http/response/__init__.py220CODE
LOWscrapy/http/response/__init__.py270CODE
LOWscrapy/http/response/text.py168CODE
LOWscrapy/http/response/text.py221CODE
LOWscrapy/http/request/__init__.py123CODE
LOWscrapy/http/request/form.py116CODE
AI Slop Vocabulary4 hits · 5 pts
SeverityFileLineSnippetContext
LOWtests/mockserver/http_resources.py368 # just set this to trigger a test failure if no valid accept-encoding header was setCOMMENT
MEDIUMscrapy/spiders/feed.py50 """You can override this function in order to make any changes you wantSTRING
MEDIUMscrapy/utils/signal.py42 """Like ``pydispatch.robust.sendRobust()`` but it also logs errors and returnsSTRING
LOWscrapy/http/request/form.py296 # If we don't have clickdata, we just use the first clickable elementCOMMENT
Overly Generic Function Names3 hits · 2 pts
SeverityFileLineSnippetContext
LOW…ils_misc/test_return_with_argument_inside_generator.py74 def helper():STRING
LOW…ils_misc/test_return_with_argument_inside_generator.py135 def helper():STRING
LOW…ils_misc/test_return_with_argument_inside_generator.py200 def helper():STRING