Scrapy, a fast high-level web crawling & scraping framework for Python.
This report presents the forensic synthetic code analysis of scrapy/scrapy, a Python project with 63,130 GitHub stars. SynthScan v2.0 examined 83,426 lines of code across 484 source files, recording 1846 pattern matches distributed across 16 syntactic categories. The overall adjusted score of 27.2 places this repository in the Moderate AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 1846 distinct pattern matches across 16 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_request_dict.py | 75 | def test_callback_serialization(self): | CODE |
| LOW | tests/test_request_dict.py | 83 | def test_reference_callback_serialization(self): | CODE |
| LOW | tests/test_request_dict.py | 94 | def test_private_reference_callback_serialization(self): | CODE |
| LOW⚡ | tests/test_request_dict.py | 105 | def test_private_callback_serialization(self): | CODE |
| LOW⚡ | tests/test_request_dict.py | 113 | def test_mixin_private_callback_serialization(self): | CODE |
| LOW⚡ | tests/test_request_dict.py | 121 | def test_delegated_callback_serialization(self): | CODE |
| LOW | tests/test_request_dict.py | 156 | def test_callback_not_available(self): | CODE |
| LOW | tests/test_utils_signal.py | 140 | def test_error_logged_if_deferred_not_supported(self): | CODE |
| LOW | tests/test_http_response_text.py | 154 | def test_declared_encoding_invalid(self): | CODE |
| LOW | tests/test_http_response_text.py | 186 | def test_bom_is_removed_from_body(self): | CODE |
| LOW | tests/test_http_response_text.py | 213 | def test_replace_wrong_encoding(self): | CODE |
| LOW | tests/test_http_response_text.py | 264 | def test_selector_shortcuts_kwargs(self): | CODE |
| LOW | tests/test_http_response_text.py | 285 | def test_urljoin_with_base_url(self): | CODE |
| LOW⚡ | tests/test_http_response_text.py | 344 | def test_follow_selector_list(self): | CODE |
| LOW⚡ | tests/test_http_response_text.py | 349 | def test_follow_selector_invalid(self): | CODE |
| LOW⚡ | tests/test_http_response_text.py | 354 | def test_follow_selector_attribute(self): | CODE |
| LOW⚡ | tests/test_http_response_text.py | 359 | def test_follow_selector_no_href(self): | CODE |
| LOW⚡ | tests/test_http_response_text.py | 367 | def test_follow_whitespace_selector(self): | CODE |
| LOW | tests/test_http_response_text.py | 433 | def test_follow_all_css_skip_invalid(self): | CODE |
| LOW | tests/test_http_response_text.py | 454 | def test_follow_all_xpath_skip_invalid(self): | CODE |
| LOW | tests/test_http_response_text.py | 471 | def test_follow_all_too_many_arguments(self): | CODE |
| LOW | tests/test_http_response_text.py | 597 | def test_selector_shortcuts_kwargs(self): | CODE |
| LOW | tests/test_contracts.py | 91 | def returns_request_cb_kwargs(self, response, url): | CODE |
| LOW | tests/test_contracts.py | 107 | def returns_item_cb_kwargs_error_unexpected_keyword(self, response): | CODE |
| LOW | tests/test_contracts.py | 115 | def returns_item_cb_kwargs_error_missing_argument(self, response, arg): | CODE |
| LOW | tests/test_contracts.py | 136 | def returns_request_range_fail(self, response): | CODE |
| LOW | tests/test_contracts.py | 198 | def scrapes_multiple_missing_fields(self, response): | CODE |
| LOW | tests/test_contracts.py | 222 | def invalid_regex_with_valid_contract(self, response): | CODE |
| LOW | tests/test_contracts.py | 246 | def returns_error_missing_meta(self, response): | CODE |
| LOW | tests/test_contracts.py | 463 | def test_returns_invalid_argument_count(self): | CODE |
| LOW | tests/test_contracts.py | 470 | def test_returns_default_bounds(self): | CODE |
| LOW | tests/test_contracts.py | 487 | def test_returns_and_scrapes_ignore_other_types(self): | CODE |
| LOW | tests/test_contracts.py | 607 | def test_custom_tagged_request_contract(self): | CODE |
| LOW | tests/test_contracts.py | 677 | def test_pre_hook_keyboard_interrupt(self): | CODE |
| LOW | tests/test_contracts.py | 691 | def test_post_hook_keyboard_interrupt(self): | CODE |
| LOW | tests/test_contracts.py | 718 | def test_pre_hook_assertion_failure(self): | CODE |
| LOW⚡ | tests/test_contracts.py | 743 | def test_pre_hook_async_callback(self): | CODE |
| LOW⚡ | tests/test_contracts.py | 753 | def test_pre_hook_async_generator(self): | CODE |
| LOW⚡ | tests/test_contracts.py | 763 | def test_post_hook_async_generator(self): | CODE |
| LOW | tests/test_command_parse.py | 115 | def parse_request_with_cb_kwargs(self, response, foo=None, key=None): | CODE |
| LOW | tests/test_command_parse.py | 121 | def parse_request_without_meta(self, response): | CODE |
| LOW | tests/test_command_parse.py | 232 | def test_request_with_cb_kwargs( | STRING |
| LOW | tests/test_command_parse.py | 254 | def test_request_without_meta( | STRING |
| LOW | tests/test_command_parse.py | 283 | def test_async_def_asyncio_parse_items_list( | STRING |
| LOW | tests/test_command_parse.py | 299 | def test_async_def_asyncio_parse_items_single_element( | STRING |
| LOW | tests/test_command_parse.py | 314 | def test_async_def_asyncgen_parse_loop( | STRING |
| LOW | tests/test_command_parse.py | 330 | def test_async_def_asyncgen_parse_exc( | STRING |
| LOW | tests/test_command_parse.py | 346 | def test_async_def_asyncio_parse( | STRING |
| LOW | tests/test_command_parse.py | 372 | def test_parse_items_no_callback_passed( | STRING |
| LOW | tests/test_command_parse.py | 384 | def test_wrong_callback_passed( | STRING |
| LOW | tests/test_command_parse.py | 399 | def test_crawlspider_matching_rule_callback_set( | STRING |
| LOW | tests/test_command_parse.py | 413 | def test_crawlspider_matching_rule_default_callback( | CODE |
| LOW | tests/test_command_parse.py | 427 | def test_spider_with_no_rules_attribute( | CODE |
| LOW | tests/test_command_parse.py | 442 | def test_crawlspider_missing_callback( | CODE |
| LOW | tests/test_command_parse.py | 455 | def test_crawlspider_no_matching_rule( | CODE |
| LOW | tests/test_command_parse.py | 470 | def test_crawlspider_not_exists_with_not_matched_url( | CODE |
| LOW | tests/test_logstats.py | 59 | def test_stats_calculations_no_time(self): | CODE |
| LOW | tests/test_logstats.py | 68 | def test_stats_calculation_no_elapsed_time(self): | CODE |
| LOW | tests/test_http_request.py | 247 | def test_copy_inherited_classes(self): | CODE |
| LOW | tests/test_http_request.py | 291 | def test_immutable_attributes(self): | CODE |
| 1136 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/test_squeues_request.py | 54 | q.push(req) | CODE |
| HIGH⚡ | tests/test_squeues_request.py | 87 | q.push(req1) | CODE |
| HIGH⚡ | tests/test_squeues_request.py | 88 | q.push(req2) | CODE |
| HIGH⚡ | tests/test_squeues_request.py | 89 | q.push(req3) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 65 | queue.push(req1) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 66 | queue.push(req2) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 67 | queue.push(req3) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 112 | queue.push(req1) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 113 | queue.push(req2) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 114 | queue.push(req3) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 142 | self.queue.push(req1) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 143 | self.queue.push(req2) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 144 | self.queue.push(req3) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 171 | self.queue.push(req1) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 172 | self.queue.push(req2) | CODE |
| HIGH⚡ | tests/test_pqueues.py | 173 | self.queue.push(req3) | CODE |
| HIGH | tests/test_pqueues.py | 30 | queue.push(req1) | CODE |
| HIGH | tests/test_pqueues.py | 45 | queue.push(Request("https://example.org")) | CODE |
| HIGH | tests/test_pqueues.py | 88 | queue.push(req) | CODE |
| HIGH | tests/test_pqueues.py | 157 | self.queue.push(Request("https://example.org")) | CODE |
| HIGH | tests/test_pqueues.py | 198 | self.queue.push(request) | CODE |
| HIGH | tests/test_pqueues.py | 222 | self.queue.push(request) | CODE |
| HIGH | tests/test_pqueues.py | 244 | self.queue.push(req) | CODE |
| HIGH | tests/test_pqueues.py | 256 | self.queue.push(req) | CODE |
| HIGH | tests/test_pqueues.py | 305 | queue.push(request) | CODE |
| HIGH⚡ | tests/test_squeues.py | 38 | q.push(lambda x: x) | CODE |
| HIGH⚡ | tests/test_squeues.py | 44 | q.push(sel) | CODE |
| HIGH⚡ | tests/test_squeues.py | 51 | q.push("a") | CODE |
| HIGH⚡ | tests/test_squeues.py | 52 | q.push(123) | CODE |
| HIGH⚡ | tests/test_squeues.py | 53 | q.push({"a": "dict"}) | CODE |
| HIGH⚡ | tests/test_squeues.py | 94 | q.push(i) | CODE |
| HIGH⚡ | tests/test_squeues.py | 103 | q.push(loader) | CODE |
| HIGH⚡ | tests/test_squeues.py | 114 | q.push(r) | CODE |
| HIGH | tests/test_squeues.py | 127 | q.push(lambda x: x) | CODE |
| HIGH | tests/test_squeues.py | 136 | q.push(sel) | CODE |
| HIGH⚡ | tests/test_squeues.py | 160 | q.push("a") | CODE |
| HIGH⚡ | tests/test_squeues.py | 161 | q.push(123) | CODE |
| HIGH⚡ | tests/test_squeues.py | 162 | q.push({"a": "dict"}) | CODE |
| HIGH⚡ | tests/test_squeues.py | 183 | q.push(i) | CODE |
| HIGH⚡ | tests/test_squeues.py | 192 | q.push(loader) | CODE |
| HIGH⚡ | tests/test_squeues.py | 203 | q.push(r) | CODE |
| HIGH | tests/test_settings/__init__.py | 955 | {"FOO": '{"tests.test_settings.Component1": null}'}, | CODE |
| HIGH | scrapy/pqueues.py | 185 | q.push(request) # this may fail (eg. serialization error) | CODE |
| HIGH | scrapy/pqueues.py | 419 | queue.push(request) | CODE |
| HIGH | scrapy/core/scheduler.py | 418 | self.dqs.push(request) | STRING |
| HIGH | scrapy/core/scheduler.py | 439 | self.mqs.push(request) | STRING |
| HIGH | scrapy/spidermiddlewares/referer.py | 74 | If url is null, return no referrer. | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | conftest.py | 1 | CODE | |
| LOW | conftest.py | 54 | CODE | |
| LOW | tests/test_utils_signal.py | 1 | CODE | |
| LOW | tests/test_http_response_text.py | 1 | CODE | |
| LOW | tests/test_command_parse.py | 1 | CODE | |
| LOW | tests/test_command_genspider.py | 1 | CODE | |
| LOW | tests/test_engine.py | 1 | CODE | |
| LOW | tests/test_engine.py | 45 | CODE | |
| LOW | tests/test_zz_resources.py | 3 | CODE | |
| LOW | tests/test_spidermiddleware.py | 1 | CODE | |
| LOW | tests/test_scrapy__getattr__.py | 1 | CODE | |
| LOW | tests/test_downloader_handler_twisted_http11.py | 3 | CODE | |
| LOW | tests/test_stats.py | 1 | CODE | |
| LOW | tests/test_linkextractors.py | 1 | CODE | |
| LOW | tests/test_command_runspider.py | 1 | CODE | |
| LOW | tests/test_loader.py | 1 | CODE | |
| LOW | tests/test_middleware.py | 1 | CODE | |
| LOW | tests/test_http_request_json.py | 1 | CODE | |
| LOW | tests/test_http2_client_protocol.py | 1 | CODE | |
| LOW | tests/test_utils_iterators.py | 1 | CODE | |
| LOW | tests/test_dependencies.py | 1 | CODE | |
| LOW | tests/test_signals.py | 1 | CODE | |
| LOW | tests/test_utils_log.py | 1 | CODE | |
| LOW | tests/test_spider_sitemap.py | 1 | CODE | |
| LOW | tests/test_spidermiddleware_urllength.py | 1 | CODE | |
| LOW | tests/test_downloader_handler_twisted_http2.py | 3 | CODE | |
| LOW | tests/test_feedexport_postprocess.py | 1 | CODE | |
| LOW | tests/test_feedexport_uri_params.py | 1 | CODE | |
| LOW | tests/test_spidermiddleware_referer.py | 1 | CODE | |
| LOW | tests/spiders.py | 5 | CODE | |
| LOW | tests/test_downloadermiddleware_robotstxt.py | 1 | CODE | |
| LOW | tests/test_crawler_subprocess.py | 1 | CODE | |
| LOW | tests/test_robotstxt_interface.py | 15 | CODE | |
| LOW | tests/test_engine_stop_download_bytes.py | 1 | CODE | |
| LOW | tests/test_http_request_form.py | 1 | CODE | |
| LOW | tests/test_core_downloader.py | 1 | CODE | |
| LOW | tests/test_core_downloader.py | 39 | CODE | |
| LOW | tests/test_spidermiddleware_start.py | 1 | CODE | |
| LOW | tests/test_http_cookies.py | 1 | CODE | |
| LOW | tests/test_scheduler_base.py | 1 | CODE | |
| LOW | tests/test_downloadermiddleware.py | 1 | CODE | |
| LOW | tests/test_downloadermiddleware_useragent.py | 1 | CODE | |
| LOW | tests/test_spider.py | 1 | CODE | |
| LOW | tests/test_squeues_request.py | 5 | CODE | |
| LOW | tests/test_spidermiddleware_base.py | 1 | CODE | |
| LOW | tests/test_scheduler.py | 1 | CODE | |
| LOW | tests/test_http_response.py | 1 | CODE | |
| LOW | tests/test_downloader_handler_httpx.py | 3 | CODE | |
| LOW | tests/test_extension_periodic_log.py | 1 | CODE | |
| LOW | tests/test_utils_defer.py | 1 | CODE | |
| LOW | tests/test_spider_crawl.py | 1 | CODE | |
| LOW | tests/test_downloadermiddleware_httpcompression.py | 69 | CODE | |
| LOW | tests/test_downloadermiddleware_httpcompression.py | 230 | CODE | |
| LOW | tests/test_extension_memusage.py | 1 | CODE | |
| LOW | tests/test_crawl.py | 1 | CODE | |
| LOW | tests/test_downloadermiddleware_redirect_metarefresh.py | 1 | CODE | |
| LOW | tests/test_downloader_handlers.py | 3 | CODE | |
| LOW | tests/test_downloader_handlers.py | 200 | CODE | |
| LOW | tests/test_downloader_handlers_http_base.py | 3 | CODE | |
| LOW | tests/test_utils_python.py | 1 | CODE | |
| 235 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| CRITICAL⚡ | tests/test_engine_loop.py | 38 | self.crawler.engine._slot.scheduler.pause() | CODE |
| CRITICAL⚡ | tests/test_engine_loop.py | 39 | self.crawler.engine._slot.scheduler.enqueue_request(Request("data:,b")) | CODE |
| CRITICAL⚡ | tests/test_engine_loop.py | 45 | self.crawler.engine._slot.scheduler.unpause() | CODE |
| CRITICAL⚡ | tests/test_engine_loop.py | 54 | self.crawler.engine._slot.scheduler.pause() | CODE |
| CRITICAL⚡ | tests/test_engine_loop.py | 55 | self.crawler.engine._slot.scheduler.enqueue_request(Request("data:,d")) | CODE |
| CRITICAL | tests/test_engine_loop.py | 203 | spider.crawler.engine._slot.scheduler.enqueue_request(request) | CODE |
| CRITICAL | tests/test_engine_loop.py | 243 | spider.crawler.engine._slot.scheduler.enqueue_request(request) | CODE |
| CRITICAL | tests/test_engine_loop.py | 291 | spider.crawler.engine._slot.scheduler.enqueue_request(request) | CODE |
| CRITICAL | tests/test_engine_loop.py | 304 | spider.crawler.engine._slot.scheduler.enqueue_request(request) | CODE |
| CRITICAL | scrapy/extensions/throttle.py | 102 | return key, self.crawler.engine.downloader.slots.get(key) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_engine.py | 630 | CODE | |
| MEDIUM | tests/test_spidermiddleware.py | 77 | CODE | |
| MEDIUM | tests/test_utils_python.py | 41 | CODE | |
| MEDIUM | tests/test_crawler.py | 865 | CODE | |
| MEDIUM | tests/test_crawler.py | 201 | CODE | |
| MEDIUM | tests/test_crawler.py | 279 | CODE | |
| MEDIUM | tests/test_crawler.py | 357 | CODE | |
| MEDIUM | tests/test_crawler.py | 435 | CODE | |
| MEDIUM | tests/test_crawler.py | 537 | CODE | |
| MEDIUM | tests/test_engine_loop.py | 210 | CODE | |
| MEDIUM | tests/test_engine_loop.py | 250 | CODE | |
| MEDIUM | tests/test_engine_loop.py | 308 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/reactorless_simple.py | 15 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/custom_loop_same.py | 20 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/reactorless_custom_settings.py | 18 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/custom_loop_different.py | 20 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/multi_parallel.py | 15 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/reactorless_reactor.py | 14 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/no_reactor.py | 13 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/simple_default_reactor.py | 14 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/simple.py | 17 | CODE | |
| MEDIUM | tests/AsyncCrawlerRunner/multi_seq.py | 15 | CODE | |
| MEDIUM | tests/CrawlerProcess/asyncio_enabled_no_reactor.py | 17 | CODE | |
| MEDIUM | tests/CrawlerProcess/args_settings.py | 19 | CODE | |
| MEDIUM | tests/CrawlerProcess/reactor_select.py | 15 | CODE | |
| MEDIUM | tests/CrawlerProcess/multi.py | 10 | CODE | |
| MEDIUM | tests/CrawlerProcess/asyncio_enabled_reactor.py | 53 | CODE | |
| MEDIUM | …ts/CrawlerProcess/asyncio_enabled_reactor_same_loop.py | 22 | CODE | |
| MEDIUM | …awlerProcess/reactor_default_twisted_reactor_select.py | 13 | CODE | |
| MEDIUM | …awlerProcess/asyncio_enabled_reactor_different_loop.py | 20 | CODE | |
| MEDIUM | …cess/reactor_select_subclass_twisted_reactor_select.py | 22 | CODE | |
| MEDIUM | tests/CrawlerProcess/dns_resolver_deprecated.py | 13 | CODE | |
| MEDIUM | tests/CrawlerProcess/reactor_default.py | 13 | CODE | |
| MEDIUM | tests/CrawlerProcess/simple.py | 12 | CODE | |
| MEDIUM | …rawlerProcess/reactor_select_twisted_reactor_select.py | 14 | CODE | |
| MEDIUM | tests/CrawlerProcess/asyncio_custom_loop.py | 10 | CODE | |
| MEDIUM | tests/CrawlerRunner/explicit_default_reactor.py | 17 | CODE | |
| MEDIUM | tests/CrawlerRunner/custom_loop_same.py | 19 | CODE | |
| MEDIUM | tests/CrawlerRunner/change_reactor.py | 15 | CODE | |
| MEDIUM | tests/CrawlerRunner/custom_loop_different.py | 19 | CODE | |
| MEDIUM | tests/CrawlerRunner/multi_parallel.py | 14 | CODE | |
| MEDIUM | tests/CrawlerRunner/no_reactor.py | 13 | CODE | |
| MEDIUM | tests/CrawlerRunner/simple.py | 16 | CODE | |
| MEDIUM | tests/CrawlerRunner/multi_seq.py | 15 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/asyncio_enabled_no_reactor.py | 17 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/reactorless_simple.py | 12 | CODE | |
| MEDIUM | …erProcess/asyncio_enabled_reactor_same_loop_default.py | 21 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/args_settings.py | 19 | CODE | |
| MEDIUM | …sts/AsyncCrawlerProcess/reactorless_custom_settings.py | 23 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/multi.py | 10 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/asyncio_enabled_reactor.py | 43 | CODE | |
| MEDIUM | …yncCrawlerProcess/asyncio_enabled_reactor_same_loop.py | 21 | CODE | |
| MEDIUM | …ncCrawlerProcess/reactorless_telnetconsole_disabled.py | 10 | CODE | |
| MEDIUM | …ocess/asyncio_custom_loop_custom_settings_different.py | 13 | CODE | |
| MEDIUM | …lerProcess/asyncio_custom_loop_custom_settings_same.py | 13 | CODE | |
| MEDIUM | …awlerProcess/asyncio_enabled_reactor_different_loop.py | 19 | CODE | |
| MEDIUM | …yncCrawlerProcess/reactorless_telnetconsole_enabled.py | 10 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/reactorless_import_hook.py | 12 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/dns_resolver_deprecated.py | 13 | CODE | |
| MEDIUM | tests/AsyncCrawlerProcess/reactor_default.py | 12 | CODE | |
| 2 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_scheduler.py | 306 | def _setup(scheduler: Scheduler) -> None: | CODE |
| LOW | tests/test_extension_periodic_log.py | 61 | def set_a(self) -> None: | CODE |
| LOW | tests/test_extension_periodic_log.py | 64 | def set_b(self) -> None: | CODE |
| LOW | scrapy/mail.py | 39 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/robotstxt.py | 21 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/addons.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/pqueues.py | 19 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/pqueues.py | 219 | def _update_curprio(self) -> None: | CODE |
| LOW | scrapy/__init__.py | 15 | __all__ = [ | CODE |
| LOW | scrapy/statscollectors.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/statscollectors.py | 65 | def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None: | CODE |
| LOW | scrapy/statscollectors.py | 68 | def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None: | CODE |
| LOW | scrapy/statscollectors.py | 119 | def set_value(self, key: str, value: Any, spider: Spider | None = None) -> None: | CODE |
| LOW | scrapy/statscollectors.py | 122 | def set_stats(self, stats: StatsT, spider: Spider | None = None) -> None: | CODE |
| LOW | scrapy/exporters.py | 28 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/exporters.py | 30 | __all__ = [ | CODE |
| LOW | scrapy/middleware.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/middleware.py | 70 | def _set_compat_spider(self, spider: Spider | None) -> None: | CODE |
| LOW | scrapy/crawler.py | 52 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/crawler.py | 89 | def _update_root_log_handler(self) -> None: | CODE |
| LOW | scrapy/crawler.py | 667 | def _setup_reactor(self, install_signal_handlers: bool) -> None: | CODE |
| LOW | scrapy/linkextractors/lxmlhtml.py | 34 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/linkextractors/__init__.py | 130 | __all__ = [ | CODE |
| LOW | scrapy/downloadermiddlewares/cookies.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/downloadermiddlewares/robotstxt.py | 31 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/downloadermiddlewares/redirect.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/downloadermiddlewares/offsite.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/settings/default_settings.py | 21 | __all__ = [ | CODE |
| LOW | scrapy/selector/unified.py | 12 | __all__ = ["Selector", "SelectorList"] | CODE |
| LOW | scrapy/selector/__init__.py | 8 | __all__ = [ | CODE |
| LOW | scrapy/core/scraper.py | 51 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/spidermw.py | 38 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/engine.py | 62 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/scheduler.py | 30 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/downloader/tls.py | 34 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/downloader/handlers/_base_streaming.py | 44 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/downloader/handlers/__init__.py | 30 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/downloader/handlers/http11.py | 71 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/downloader/handlers/http.py | 17 | __all__ = [ | CODE |
| LOW | scrapy/core/http2/protocol.py | 51 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/core/http2/stream.py | 31 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/spiders/sitemap.py | 23 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/spiders/__init__.py | 77 | def _set_crawler(self, crawler: Crawler) -> None: | CODE |
| LOW | scrapy/spiders/__init__.py | 144 | def update_settings(cls, settings: BaseSettings) -> None: | CODE |
| LOW | scrapy/spiders/__init__.py | 167 | __all__ = [ | CODE |
| LOW | scrapy/pipelines/files.py | 54 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/pipelines/files.py | 510 | def _update_stores(cls, settings: BaseSettings) -> None: | CODE |
| LOW | scrapy/pipelines/media.py | 55 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/pipelines/images.py | 137 | def _update_stores(cls, settings: BaseSettings) -> None: | CODE |
| LOW | scrapy/utils/reactor.py | 98 | def set_asyncio_event_loop_policy() -> None: | CODE |
| LOW | scrapy/utils/signal.py | 33 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/log.py | 25 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/asyncio.py | 31 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/request.py | 31 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/_download_handlers.py | 47 | def set_cookie(self, cookie: Cookie) -> None: | CODE |
| LOW | scrapy/utils/iterators.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/ssl.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/utils/spider.py | 21 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/extensions/telnet.py | 35 | logger = logging.getLogger(__name__) | CODE |
| LOW | scrapy/extensions/memusage.py | 32 | logger = logging.getLogger(__name__) | CODE |
| 25 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_downloadermiddleware_httpcache.py | 263 | except Exception: | CODE |
| LOW | tests/test_pipeline_media.py | 114 | except Exception as exc: | CODE |
| LOW | …awlerProcess/twisted_reactor_custom_settings_select.py | 23 | except Exception: | CODE |
| LOW | …sts/AsyncCrawlerProcess/reactorless_custom_settings.py | 29 | except Exception: | CODE |
| LOW | scrapy/crawler.py | 194 | except Exception: | CODE |
| LOW | scrapy/crawler.py | 224 | except Exception: | CODE |
| LOW | scrapy/crawler.py | 470 | except Exception: | CODE |
| LOW | scrapy/crawler.py | 581 | except Exception: | CODE |
| LOW | scrapy/downloadermiddlewares/robotstxt.py | 102 | except Exception as e: | CODE |
| LOW | scrapy/core/scraper.py | 226 | except Exception: | CODE |
| LOW | scrapy/core/scraper.py | 258 | except Exception: | CODE |
| LOW | scrapy/core/scraper.py | 267 | except Exception as spider_exc: | CODE |
| LOW | scrapy/core/scraper.py | 290 | except Exception: | CODE |
| LOW | scrapy/core/scraper.py | 513 | except Exception as ex: | CODE |
| LOW | scrapy/core/spidermw.py | 96 | except Exception: | CODE |
| LOW | scrapy/core/spidermw.py | 110 | except Exception as ex: | CODE |
| LOW | scrapy/core/spidermw.py | 226 | except Exception as ex: | CODE |
| LOW | scrapy/core/engine.py | 150 | except Exception: | CODE |
| LOW | scrapy/core/engine.py | 280 | except Exception as exception: | CODE |
| LOW | scrapy/core/engine.py | 320 | except Exception: | CODE |
| LOW | scrapy/core/engine.py | 414 | except Exception: | CODE |
| LOW⚡ | scrapy/core/engine.py | 618 | except Exception: | CODE |
| LOW⚡ | scrapy/core/engine.py | 623 | except Exception: | CODE |
| LOW⚡ | scrapy/core/engine.py | 630 | except Exception: | CODE |
| LOW⚡ | scrapy/core/engine.py | 639 | except Exception: | CODE |
| LOW | scrapy/core/engine.py | 650 | except Exception: | CODE |
| LOW | scrapy/core/engine.py | 671 | except Exception: | CODE |
| LOW | scrapy/core/engine.py | 685 | except Exception: | CODE |
| LOW | scrapy/core/downloader/__init__.py | 238 | except Exception: | CODE |
| LOW | scrapy/core/downloader/__init__.py | 259 | except Exception: | CODE |
| LOW | scrapy/core/downloader/middleware.py | 83 | except Exception as ex: | CODE |
| LOW | scrapy/core/downloader/handlers/_base_streaming.py | 222 | except Exception as e: | CODE |
| LOW | scrapy/core/downloader/handlers/__init__.py | 110 | except Exception as ex: | CODE |
| LOW | scrapy/contracts/__init__.py | 48 | except Exception: | CODE |
| LOW | scrapy/contracts/__init__.py | 82 | except Exception: | CODE |
| LOW | scrapy/contracts/__init__.py | 135 | except Exception: | CODE |
| LOW | scrapy/contracts/__init__.py | 191 | except Exception: | CODE |
| LOW | scrapy/pipelines/files.py | 416 | except Exception: | CODE |
| LOW | scrapy/pipelines/files.py | 667 | except Exception as exc: | CODE |
| LOW | scrapy/pipelines/media.py | 194 | except Exception: | CODE |
| LOW | scrapy/pipelines/media.py | 216 | except Exception: | CODE |
| LOW | scrapy/utils/deprecate.py | 131 | except Exception as e: | CODE |
| LOW | scrapy/utils/signal.py | 64 | except Exception: | CODE |
| LOW | scrapy/utils/signal.py | 199 | except Exception as ex: | CODE |
| LOW | scrapy/utils/asyncio.py | 206 | except Exception: | CODE |
| LOW | scrapy/utils/engine.py | 36 | except Exception as e: | CODE |
| LOW | scrapy/utils/defer.py | 154 | except Exception: | CODE |
| LOW | scrapy/utils/defer.py | 360 | except Exception: | CODE |
| LOW | scrapy/utils/defer.py | 379 | except Exception: | CODE |
| LOW | scrapy/utils/ssl.py | 43 | except Exception as ex: | CODE |
| LOW | scrapy/extensions/feedexport.py | 595 | except Exception: | CODE |
| LOW | scrapy/extensions/httpcache.py | 418 | except Exception: | CODE |
| MEDIUM | scrapy/commands/startproject.py | 56 | print(f"Error: Module {project_name!r} already exists") | CODE |
| MEDIUM | scrapy/commands/startproject.py | 102 | print(f"Error: scrapy.cfg already exists in {project_dir.resolve()}") | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_http_request.py | 141 | # percent-escaping sequences that do not match valid UTF-8 sequences | COMMENT |
| LOW | docs/requirements.txt | 1 | # This file was autogenerated by uv via the following command: | COMMENT |
| LOW | docs/requirements.txt | 21 | constantly==23.10.4 | COMMENT |
| LOW | docs/requirements.txt | 41 | h2==4.3.0 | COMMENT |
| LOW | docs/requirements.txt | 61 | # scrapy | COMMENT |
| LOW | docs/requirements.txt | 81 | # scrapy-spider-metadata | COMMENT |
| LOW | docs/requirements.txt | 101 | # scrapy-spider-metadata | COMMENT |
| LOW | docs/requirements.txt | 121 | scrapy==2.14.2 | COMMENT |
| LOW | docs/requirements.txt | 141 | # via sphinx-scrapy | COMMENT |
| LOW | docs/requirements.txt | 181 | # via | COMMENT |
| LOW | docs/conf.py | 101 | COMMENT | |
| LOW | docs/conf.py | 121 | # Base classes of downloader middlewares are implementation details that | COMMENT |
| LOW | scrapy/shell.py | 41 | # official ways to use it: | COMMENT |
| LOW | scrapy/shell.py | 61 | # which initializes the engine but doesn't start processing of requests. | COMMENT |
| LOW | scrapy/shell.py | 81 | # Other architecture problems: | COMMENT |
| LOW | scrapy/cmdline.py | 241 | # Twisted prints errors in DebugInfo.__del__, but PyPy does not run gc.collect() on exit: | COMMENT |
| LOW | scrapy/crawler.py | 901 | # So we reproduce a part of asyncio.runners.Runner that is useful to us. | COMMENT |
| LOW | scrapy/crawler.py | 921 | # 1. While loop.run_until_complete() blocks, _signal_shutdown_reactorless() | COMMENT |
| LOW | scrapy/core/downloader/contextfactory.py | 221 | # | COMMENT |
| LOW | scrapy/core/http2/protocol.py | 121 | COMMENT | |
| LOW | scrapy/pipelines/media.py | 241 | # the Response status code is not 200 OK, is that the original | COMMENT |
| LOW | scrapy/http/request/__init__.py | 161 | ) | COMMENT |
| LOW | scrapy/http/request/__init__.py | 181 | #: processing, i.e. by a :ref:`spider middleware | COMMENT |
| LOW | scrapy/http/request/__init__.py | 201 | #: .. seealso:: :ref:`topics-request-response-ref-errbacks` | COMMENT |
| LOW | scrapy/http/request/__init__.py | 221 | #: to be scheduled multiple times. | COMMENT |
| LOW | scrapy/templates/project/module/spiders/__init__.py | 1 | # This package will contain the spiders of your Scrapy project | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 25 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 65 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 104 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 130 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 141 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 158 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 169 | # ================================================================================ | COMMENT |
| MEDIUM | tests/test_spidermiddleware_output_chain.py | 240 | # ================================================================================ | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_engine.py | 324 | CODE | |
| LOW | tests/test_downloadermiddleware_httpcache.py | 436 | CODE | |
| LOW | docs/utils/linkfix.py | 20 | CODE | |
| LOW | scrapy/pqueues.py | 189 | CODE | |
| LOW | scrapy/pqueues.py | 363 | CODE | |
| LOW | scrapy/downloadermiddlewares/httpproxy.py | 81 | CODE | |
| LOW | scrapy/core/http2/protocol.py | 376 | CODE | |
| LOW | scrapy/core/http2/stream.py | 209 | CODE | |
| LOW | scrapy/core/http2/stream.py | 397 | CODE | |
| LOW | scrapy/spiders/feed.py | 78 | CODE | |
| LOW | scrapy/pipelines/media.py | 129 | CODE | |
| LOW | scrapy/utils/sitemap.py | 107 | CODE | |
| LOW | scrapy/utils/sitemap.py | 62 | CODE | |
| LOW | scrapy/utils/request.py | 39 | CODE | |
| LOW | scrapy/utils/iterators.py | 23 | CODE | |
| LOW | scrapy/utils/iterators.py | 81 | CODE | |
| LOW | scrapy/utils/ssl.py | 156 | CODE | |
| LOW | scrapy/http/request/form.py | 173 | CODE | |
| LOW | scrapy/commands/check.py | 73 | CODE | |
| LOW | scrapy/commands/settings.py | 49 | CODE | |
| LOW | scrapy/commands/parse.py | 305 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | scrapy/core/scraper.py | 1 | """This module implements the Scraper component which parses responses and | STRING |
| MEDIUM | scrapy/core/spidermw.py | 249 | # This method is only needed until _async compatibility methods are removed. | COMMENT |
| MEDIUM | scrapy/core/downloader/tls.py | 147 | """This method is needed to override the verify callback.""" | STRING |
| MEDIUM | scrapy/core/http2/agent.py | 55 | # Create a deferred which will fire with the H2ClientProtocol | COMMENT |
| MEDIUM | scrapy/spiders/feed.py | 57 | """This method is called for the nodes matching the provided tag name | STRING |
| MEDIUM | scrapy/spiders/feed.py | 70 | """This method is called for the nodes matching the provided tag name | STRING |
| MEDIUM | scrapy/utils/trackref.py | 1 | """This module provides some functions and classes to record and report | STRING |
| MEDIUM | scrapy/commands/shell.py | 98 | # Create the engine and run start_async() in the main thread | COMMENT |
| MEDIUM | scrapy/commands/shell.py | 104 | # Create the engine and run start_async() in the event loop thread | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_downloadermiddleware_httpproxy.py | 284 | request.meta["proxy"] = "https://user1:password1@example.org" | CODE |
| LOW⚡ | tests/test_downloadermiddleware_httpproxy.py | 301 | request.meta["proxy"] = "https://user1:password1@example.org" | CODE |
| LOW⚡ | tests/test_downloadermiddleware_httpproxy.py | 324 | request.meta["proxy"] = "https://user2:password2@example.org" | CODE |
| LOW⚡ | tests/test_item.py | 69 | i["name"] = "John Doe" | CODE |
| LOW⚡ | tests/test_item.py | 73 | assert itemrepr == "{'name': 'John Doe', 'number': 123}" | CODE |
| LOW⚡ | tests/test_item.py | 76 | assert i2["name"] == "John Doe" | CODE |
| LOW⚡ | tests/test_settings/__init__.py | 689 | dummy_config_value = settings.pop("DUMMY_CONFIG", "dummy_value") | CODE |
| LOW⚡ | tests/test_settings/__init__.py | 690 | assert dummy_config_value == "dummy_value" | CODE |
| LOW⚡ | tests/test_settings/__init__.py | 694 | {"DUMMY_CONFIG": "dummy_value", "OTHER_DUMMY_CONFIG": "other_dummy_value"} | CODE |
| LOW⚡ | tests/test_settings/__init__.py | 697 | assert settings.pop("DUMMY_CONFIG") == "dummy_value" | CODE |
| LOW | scrapy/http/request/form.py | 70 | formdata={"name": "John Doe", "age": "27"}, | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_http2_client_protocol.py | 381 | # Check if headers were sent successfully | COMMENT |
| LOW | scrapy/core/http2/stream.py | 372 | # Check if we exceed the allowed max data size which can be received | COMMENT |
| LOW | scrapy/core/http2/agent.py | 61 | # Check if we already have a connection to the remote | COMMENT |
| LOW | scrapy/pipelines/media.py | 176 | # Check if request is downloading right now to avoid doing it twice | COMMENT |
| LOW | scrapy/utils/asyncio.py | 73 | # Check if there is a running asyncio loop. | COMMENT |
| LOW | scrapy/utils/asyncio.py | 84 | # Check if there is an installed asyncio reactor (it doesn't need to be | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | scrapy/mail.py | 62 | CODE | |
| LOW | scrapy/linkextractors/lxmlhtml.py | 299 | CODE | |
| LOW | scrapy/http/response/__init__.py | 71 | CODE | |
| LOW | scrapy/http/response/__init__.py | 220 | CODE | |
| LOW | scrapy/http/response/__init__.py | 270 | CODE | |
| LOW | scrapy/http/response/text.py | 168 | CODE | |
| LOW | scrapy/http/response/text.py | 221 | CODE | |
| LOW | scrapy/http/request/__init__.py | 123 | CODE | |
| LOW | scrapy/http/request/form.py | 116 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/mockserver/http_resources.py | 368 | # just set this to trigger a test failure if no valid accept-encoding header was set | COMMENT |
| MEDIUM | scrapy/spiders/feed.py | 50 | """You can override this function in order to make any changes you want | STRING |
| MEDIUM | scrapy/utils/signal.py | 42 | """Like ``pydispatch.robust.sendRobust()`` but it also logs errors and returns | STRING |
| LOW | scrapy/http/request/form.py | 296 | # If we don't have clickdata, we just use the first clickable element | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ils_misc/test_return_with_argument_inside_generator.py | 74 | def helper(): | STRING |
| LOW | …ils_misc/test_return_with_argument_inside_generator.py | 135 | def helper(): | STRING |
| LOW | …ils_misc/test_return_with_argument_inside_generator.py | 200 | def helper(): | STRING |