Home / D4Vinci / Scrapling
Repository Analysis

D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

22.2 Moderate AI signal View on GitHub
22.2
Adjusted Score
22.2
Raw Score
100%
Time Factor
2026-05-27
Last Push
55,329
Stars
Python
Language
43,435
Lines of Code
201
Files
793
Pattern Hits
2026-05-31
Scan Date

Score History

Severity Breakdown

CRITICAL 0HIGH 11MEDIUM 62LOW 720

Pattern Findings

793 matches across 11 categories. Click a row to expand file-level details.

Hyper-Verbose Identifiers536 hits · 524 pts
SeverityFileLineSnippet
LOWtests/core/test_storage_core.py19 def test_returns_default_when_url_is_none(self):
LOWtests/core/test_storage_core.py23 def test_returns_default_when_url_is_empty(self):
LOWtests/core/test_storage_core.py27 def test_returns_fld_for_valid_url(self):
LOWtests/core/test_storage_core.py43 def test_deterministic_output(self):
LOWtests/core/test_storage_core.py48 def test_different_input_different_output(self):
LOWtests/core/test_storage_core.py53 def test_strips_and_lowercases(self):
LOWtests/core/test_storage_core.py58 def test_includes_length_suffix(self):
LOWtests/core/test_storage_core.py70 def test_sqlite_storage_creation(self):
LOWtests/core/test_storage_core.py75 def test_sqlite_storage_with_file(self):
LOWtests/core/test_storage_core.py91 def test_sqlite_storage_initialization_args(self):
LOWtests/core/test_storage_core.py125 def test_retrieve_nonexistent_returns_none(self):
LOWtests/core/test_storage_core.py129 def test_save_overwrites_existing(self):
LOWtests/core/test_storage_core.py167 def test_element_path_is_stored(self):
LOWtests/core/test_storage_core.py178 def test_element_with_children_and_siblings(self):
LOWtests/core/test_storage_core.py254 def test_element_no_attributes(self):
LOWtests/core/test_storage_core.py260 def test_element_strips_whitespace_attributes(self):
LOWtests/core/test_shell_core.py14 def test_simple_cookie_parsing(self):
LOWtests/core/test_shell_core.py21 def test_multiple_cookies_parsing(self):
LOWtests/core/test_shell_core.py31 def test_cookie_with_attributes(self):
LOWtests/core/test_shell_core.py43 def test_malformed_cookie_handling(self):
LOWtests/core/test_shell_core.py67 def test_headers_with_cookies(self):
LOWtests/core/test_shell_core.py80 def test_headers_without_colons(self):
LOWtests/core/test_shell_core.py92 def test_invalid_header_format(self):
LOWtests/core/test_shell_core.py102 def test_headers_with_multiple_colons(self):
LOWtests/core/test_shell_core.py113 def test_headers_with_whitespace(self):
LOWtests/core/test_shell_core.py125 def test_parse_cookies_disabled(self):
LOWtests/core/test_shell_core.py187 def test_request_field_access(self):
LOWtests/core/test_shell_core.py212 def test_known_logging_levels(self):
LOWtests/core/test_shell_core.py220 def test_logging_level_values(self):
LOWtests/fetchers/test_utils.py316 def test_continues_non_blocked_domain(self):
LOWtests/fetchers/test_utils.py322 def test_resource_blocking_takes_priority_over_domain(self):
LOWtests/fetchers/test_utils.py329 def test_domain_blocking_with_resources_disabled(self):
LOWtests/fetchers/test_utils.py336 def test_no_blocking_continues(self):
LOWtests/fetchers/test_utils.py342 def test_does_not_block_partial_domain_match(self):
LOWtests/fetchers/test_utils.py349 def test_multiple_blocked_domains(self):
LOWtests/fetchers/test_utils.py437 def test_ad_domains_is_frozenset(self):
LOWtests/fetchers/test_utils.py442 def test_ad_domains_has_entries(self):
LOWtests/fetchers/test_utils.py447 def test_ad_domains_contains_known_entries(self):
LOWtests/fetchers/test_utils.py457 def test_block_ads_populates_blocked_domains(self):
LOWtests/fetchers/test_utils.py465 def test_block_ads_false_leaves_blocked_domains_none(self):
LOWtests/fetchers/test_utils.py471 def test_block_ads_merges_with_user_domains(self):
LOWtests/fetchers/test_utils.py480 def test_block_ads_does_not_modify_original_set(self):
LOWtests/fetchers/test_utils.py131def test_parsing_response_status(status_map):
LOWtests/fetchers/test_utils.py152 def test_proxy_string_with_auth(self):
LOWtests/fetchers/test_utils.py174 def test_invalid_proxy_string(self):
LOWtests/fetchers/test_utils.py196 def test_generate_headers_basic(self):
LOWtests/fetchers/test_utils.py204 def test_generate_headers_browser_mode(self):
LOWtests/fetchers/test_utils.py233 def test_response_with_bytes_content(self):
LOWtests/fetchers/test_utils.py292 def test_blocks_disabled_resource_types(self):
LOWtests/fetchers/test_utils.py298 def test_continues_allowed_resource_types(self):
LOWtests/fetchers/test_utils.py366 async def test_blocks_disabled_resource_types(self):
LOWtests/fetchers/test_utils.py380 async def test_continues_non_blocked(self):
LOWtests/fetchers/test_utils.py394 async def test_does_not_block_partial_domain_match(self):
LOWtests/fetchers/test_utils.py412 def test_deep_subdomain_match(self):
LOWtests/fetchers/test_impersonate_list.py14 def test_select_random_browser_with_single_string(self):
LOWtests/fetchers/test_impersonate_list.py19 def test_select_random_browser_with_none(self):
LOWtests/fetchers/test_impersonate_list.py24 def test_select_random_browser_with_list(self):
LOWtests/fetchers/test_impersonate_list.py30 def test_select_random_browser_with_empty_list(self):
LOWtests/fetchers/test_impersonate_list.py35 def test_select_random_browser_with_single_item_list(self):
LOWtests/fetchers/test_impersonate_list.py50 def test_get_with_impersonate_list(self):
476 more matches not shown…
Decorative Section Separators54 hits · 184 pts
SeverityFileLineSnippet
MEDIUMtests/spiders/test_engine.py174# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py176# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py190# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py192# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py227# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py229# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py268# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py270# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py304# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py306# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py326# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py328# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py524# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py526# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py554# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py556# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py587# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py589# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py627# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py629# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py787# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py789# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py808# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py810# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py862# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py864# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py18# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py20# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py925# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_engine.py927# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_force_stop_checkpoint.py25# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_force_stop_checkpoint.py27# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_force_stop_checkpoint.py121# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_force_stop_checkpoint.py123# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py296# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py298# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py390# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py392# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py469# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py471# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py10# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py12# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py40# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py42# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py356# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py358# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py68# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py70# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py151# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py153# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py227# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_robotstxt.py229# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_cache.py108# ---------------------------------------------------------------------------
MEDIUMtests/spiders/test_cache.py110# ---------------------------------------------------------------------------
Unused Imports93 hits · 86 pts
SeverityFileLineSnippet
LOWtests/fetchers/test_impersonate_list.py4
LOWtests/fetchers/test_impersonate_list.py4
LOWtests/fetchers/test_merge_request_args.py6
LOWtests/spiders/test_result.py7
LOWtests/spiders/test_engine.py6
LOWtests/spiders/test_engine.py14
LOWtests/spiders/test_request.py5
LOWtests/spiders/test_session.py3
LOWtests/spiders/test_session.py5
LOWtests/spiders/test_spider.py12
LOWtests/spiders/test_force_stop_checkpoint.py22
LOWtests/spiders/test_links.py5
LOWtests/spiders/test_checkpoint.py8
LOWtests/parser/test_parser_advanced.py5
LOWscrapling/__init__.py8
LOWscrapling/__init__.py8
LOWscrapling/__init__.py9
LOWscrapling/__init__.py9
LOWscrapling/__init__.py10
LOWscrapling/__init__.py10
LOWscrapling/__init__.py10
LOWscrapling/__init__.py10
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py5
LOWscrapling/core/_types.py34
LOWscrapling/core/_types.py34
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/core/utils/__init__.py1
LOWscrapling/fetchers/__init__.py2
LOWscrapling/fetchers/__init__.py5
LOWscrapling/fetchers/__init__.py5
LOWscrapling/fetchers/__init__.py5
33 more matches not shown…
Excessive Try-Catch Wrapping54 hits · 55 pts
SeverityFileLineSnippet
LOWcleanup.py29 except Exception as e:
LOWcleanup.py37 except Exception as e:
LOWtests/core/test_storage_core.py213 except Exception as e:
MEDIUMtests/core/test_storage_core.py207def save_element(idx):
LOWscrapling/cli.py72 except Exception as err:
LOWscrapling/core/ai.py307 except Exception as exc:
LOWscrapling/core/shell.py171 except Exception as e: # pragma: no cover
LOWscrapling/core/shell.py203 except Exception as e: # pragma: no cover
LOWscrapling/core/shell.py217 except Exception as e:
LOWscrapling/core/shell.py234 except Exception as e:
LOWscrapling/core/shell.py321 except Exception as e: # pragma: no cover
LOWscrapling/core/shell.py363 except Exception: # pragma: no cover
LOWscrapling/core/shell.py380 except Exception as e:
LOWscrapling/core/shell.py566 except Exception as e:
LOWscrapling/core/utils/_shell.py41 except Exception as e: # pragma: no cover
LOWscrapling/spiders/robotstxt.py31 except Exception as e:
LOWscrapling/spiders/robotstxt.py36 except Exception as e:
LOWscrapling/spiders/checkpoint.py56 except Exception as e:
LOWscrapling/spiders/checkpoint.py79 except Exception as e:
LOWscrapling/spiders/checkpoint.py89 except Exception as e:
LOWscrapling/spiders/cache.py42 except Exception as e:
LOWscrapling/spiders/cache.py68 except Exception as e:
LOWscrapling/spiders/engine.py167 except Exception as e:
LOWscrapling/spiders/engine.py210 except Exception as e:
LOWscrapling/spiders/spider.py140 except Exception as e:
LOWscrapling/spiders/templates/sitemap.py88 except Exception as e:
LOWscrapling/engines/static.py724 except Exception:
LOWscrapling/engines/static.py750 except Exception:
LOWscrapling/engines/toolbelt/convertor.py71 except Exception as e: # pragma: no cover
LOWscrapling/engines/toolbelt/convertor.py76 except Exception as e: # pragma: no cover
LOWscrapling/engines/toolbelt/convertor.py127 except Exception as e: # pragma: no cover
LOWscrapling/engines/toolbelt/convertor.py187 except Exception as e: # pragma: no cover
LOWscrapling/engines/toolbelt/convertor.py192 except Exception as e: # pragma: no cover
LOWscrapling/engines/toolbelt/convertor.py274 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py94 except Exception:
LOWscrapling/engines/_browsers/_controllers.py160 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py173 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py181 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py196 except Exception as e:
LOWscrapling/engines/_browsers/_controllers.py282 except Exception:
LOWscrapling/engines/_browsers/_controllers.py349 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py362 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py370 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_controllers.py385 except Exception as e:
LOWscrapling/engines/_browsers/_stealth.py99 except Exception:
LOWscrapling/engines/_browsers/_stealth.py243 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py261 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py269 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py284 except Exception as e:
LOWscrapling/engines/_browsers/_stealth.py374 except Exception:
LOWscrapling/engines/_browsers/_stealth.py519 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py537 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py545 except Exception as e: # pragma: no cover
LOWscrapling/engines/_browsers/_stealth.py560 except Exception as e:
Deep Nesting33 hits · 33 pts
SeverityFileLineSnippet
LOWcleanup.py6
LOWscrapling/parser.py519
LOWscrapling/parser.py628
LOWscrapling/parser.py698
LOWscrapling/parser.py1094
LOWscrapling/core/mixins.py15
LOWscrapling/core/shell.py333
LOWscrapling/core/shell.py149
LOWscrapling/core/shell.py615
LOWscrapling/core/shell.py656
LOWscrapling/core/utils/_shell.py19
LOWscrapling/spiders/request.py71
LOWscrapling/spiders/links.py220
LOWscrapling/spiders/engine.py141
LOWscrapling/spiders/engine.py324
LOWscrapling/spiders/spider.py34
LOWscrapling/spiders/spider.py255
LOWscrapling/spiders/spider.py297
LOWscrapling/spiders/templates/sitemap.py94
LOWscrapling/spiders/templates/sitemap.py105
LOWscrapling/spiders/templates/sitemap.py125
LOWscrapling/engines/static.py223
LOWscrapling/engines/static.py437
LOWscrapling/engines/_browsers/_controllers.py72
LOWscrapling/engines/_browsers/_controllers.py102
LOWscrapling/engines/_browsers/_controllers.py261
LOWscrapling/engines/_browsers/_controllers.py290
LOWscrapling/engines/_browsers/_stealth.py76
LOWscrapling/engines/_browsers/_stealth.py107
LOWscrapling/engines/_browsers/_stealth.py184
LOWscrapling/engines/_browsers/_stealth.py352
LOWscrapling/engines/_browsers/_stealth.py382
LOWscrapling/engines/_browsers/_stealth.py459
Cross-File Repetition6 hits · 30 pts
SeverityFileLineSnippet
HIGHtests/fetchers/async/test_dynamic.py0test doing a basic fetch request with multiple statuses
HIGHtests/fetchers/async/test_stealth.py0test doing a basic fetch request with multiple statuses
HIGHtests/fetchers/sync/test_dynamic.py0test doing a basic fetch request with multiple statuses
HIGHscrapling/parser.py0using the identifier, we search the storage and return the unique properties of the element :param identifier: this is t
HIGHscrapling/core/storage.py0using the identifier, we search the storage and return the unique properties of the element :param identifier: this is t
HIGHscrapling/core/storage.py0using the identifier, we search the storage and return the unique properties of the element :param identifier: this is t
Synthetic Comment Markers5 hits · 28 pts
SeverityFileLineSnippet
HIGH…-skill/Scrapling-Skill/references/fetching/stealthy.md163This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Sc
HIGHtests/fetchers/test_utils.py16 # A map generated by ChatGPT for most possible `content_type` values and the expected outcome
HIGHdocs/overview.md17Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
HIGHdocs/development/adaptive_storage_system.md26Here's a more practical example generated by AI using Redis:
HIGHdocs/fetching/stealthy.md175This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Sc
Self-Referential Comments4 hits · 10 pts
SeverityFileLineSnippet
MEDIUMtests/parser/test_attributes_handler.py279 # Create an element with many attributes
MEDIUMscrapling/core/shell.py55# Define the structure for parsed context - Simplified for Fetcher args
MEDIUMscrapling/core/shell.py516 # Create the namespace dictionary
MEDIUMscrapling/engines/toolbelt/custom.py29 """This class is returned by all engines as a way to unify the response type between different libraries.
Dead Code3 hits · 5 pts
SeverityFileLineSnippet
MEDIUMtests/spiders/test_engine.py473
MEDIUMscrapling/spiders/spider.py169
MEDIUMscrapling/spiders/templates/sitemap.py81
Redundant / Tautological Comments3 hits · 4 pts
SeverityFileLineSnippet
LOWscrapling/core/shell.py238 # Check if raw data looks like JSON, prefer 'json' param if so
LOWscrapling/core/shell.py343 # Check if this function has known parameters
LOWscrapling/engines/_browsers/_base.py572 # Check if turnstile captcha is embedded inside the page (Usually inside a closed Shadow iframe)
Fake / Example Data2 hits · 2 pts
SeverityFileLineSnippet
LOW…nt-skill/Scrapling-Skill/references/fetching/static.md350 'username': 'user@example.com',
LOWdocs/fetching/static.md357 'username': 'user@example.com',