newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
This report summarizes scanner evidence for codelucas/newspaper, a Python project with 15,181 GitHub stars. SynthScan examined 297,753 lines of code across 399 source files, recording 127 pattern matches distributed across 14 syntactic categories. The legacy adjusted pattern signal is 0.6 (Low pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Payload schema: 1.0.0 · Engine: 2.0.0 · Scoring: heuristic-density-v1 · Scan date: 2026-10-31.
Can maintainers explain intent, assumptions, and trade-offs? 90 finding(s).
Are imports declared and package/API names verified? 25 finding(s).
Are wrappers, guards, and repeated structures necessary? 46 finding(s).
Which changes need focused human review? 127 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 127 distinct pattern matches across 14 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | tests/data/html/thedailybeast.com2.html | 1 | <!DOCTYPE html><!--[if lt IE 7]> <html class="ie ie6 lte9 lte8 lte7" lang="en"> <![endif]--><!--[if IE 7]> <html class=" | CODE |
| MEDIUM⚡ | tests/data/html/thedailybeast.com2.html | 1 | <!DOCTYPE html><!--[if lt IE 7]> <html class="ie ie6 lte9 lte8 lte7" lang="en"> <![endif]--><!--[if IE 7]> <html class=" | CODE |
| MEDIUM⚡ | tests/data/html/thedailybeast.com2.html | 1 | <!DOCTYPE html><!--[if lt IE 7]> <html class="ie ie6 lte9 lte8 lte7" lang="en"> <![endif]--><!--[if IE 7]> <html class=" | CODE |
| MEDIUM | tests/data/html/lifebuzz.com1.html | 306 | <img alt="30 World Festivals That You'll Definitely Want To Add To Your Bucket List. " src="http: | CODE |
| MEDIUM | tests/data/html/lifebuzz.com1.html | 307 | </a> <h3><a href="/fun-festivals/">30 World Festivals That You'll Definitely Want To Add To Your Buck | CODE |
| MEDIUM | tests/data/html/bostonherald.com1.html | 587 | <div class="field-item even" property="content:encoded"><p><body.content></body.content></p><p><person>Annie L | CODE |
| MEDIUM | tests/data/html/slate.com1.html | 1493 | <p>These contemporary <a href="http://en.wikipedia.org/wiki/Appian_Way">Appian Ways</a> are the vital back end of the In | CODE |
| MEDIUM | tests/data/html/slate.com1.html | 1649 | <p>That’s in part because the state-owned and private telecom companies that have built and run the global network | CODE |
| MEDIUM | tests/data/html/slate.com1.html | 1649 | <p>That’s in part because the state-owned and private telecom companies that have built and run the global network | CODE |
| MEDIUM | tests/data/html/slate.com1.html | 2048 | <p>And in the cold rock deep underground, engineers are toiling to fix a pretty big flaw in the system. The <em>New | CODE |
| MEDIUM | tests/data/html/glamour.com2.html | 11 | <meta name="description" content="A ton of brides are going for the pretty pretty princess vibe on their wedding day, so | CODE |
| MEDIUM | tests/data/html/glamour.com2.html | 23 | <meta property="og:description" content="A ton of brides are going for the pretty pretty princess vibe on their wedding | CODE |
| MEDIUM | tests/data/html/glamour.com2.html | 71 | meta: {"contributor":"Kim Fusaro","description":"A ton of brides are going for the pretty pretty princess vibe o | CODE |
| MEDIUM | tests/data/html/glamour.com2.html | 731 | <p>A ton of brides are going for the <a href="http://smile.amazon.com/Pretty-Princess-Dr | CODE |
| MEDIUM | tests/data/html/lifebuzz.com2.html | 306 | <img alt="30 World Festivals That You'll Definitely Want To Add To Your Bucket List. " src="http: | CODE |
| MEDIUM | tests/data/html/lifebuzz.com2.html | 307 | </a> <h3><a href="/fun-festivals/">30 World Festivals That You'll Definitely Want To Add To Your Buck | CODE |
| MEDIUM⚡ | tests/data/html/thedailybeast.com1.html | 3 | <p>The report was the fruit of an investigation into 341 American congregations guided by Mother Superior Marie Clare Mi | CODE |
| MEDIUM⚡ | tests/data/html/thedailybeast.com1.html | 3 | <p>The report was the fruit of an investigation into 341 American congregations guided by Mother Superior Marie Clare Mi | CODE |
| MEDIUM⚡ | tests/data/html/thedailybeast.com1.html | 3 | <p>The report was the fruit of an investigation into 341 American congregations guided by Mother Superior Marie Clare Mi | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/unit_tests.py | 382 | def test_get_canonical_link_rel_canonical(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 387 | def test_get_canonical_link_rel_canonical_absolute_url(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 393 | def test_get_canonical_link_og_url_absolute_url(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 399 | def test_get_canonical_link_hostname_og_url_absolute_url(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 405 | def test_get_top_image_from_meta(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 639 | def test_article_default_params(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 648 | def test_article_custom_params(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 657 | def test_source_default_params(self): | CODE |
| LOW⚡ | tests/unit_tests.py | 665 | def test_source_custom_params(self): | CODE |
| LOW | tests/unit_tests.py | 186 | def test_meta_refresh_redirect(self): | CODE |
| LOW | tests/unit_tests.py | 199 | def test_meta_refresh_no_url_redirect(self): | CODE |
| LOW | tests/unit_tests.py | 248 | def test_meta_type_extraction(self): | CODE |
| LOW | tests/unit_tests.py | 332 | def test_download_file_success(self): | CODE |
| LOW | tests/unit_tests.py | 341 | def test_download_file_failure(self): | CODE |
| LOW | tests/unit_tests.py | 369 | def test_get_title_split_escaped(self): | CODE |
| LOW | tests/unit_tests.py | 450 | def test_source_url_input_none(self): | CODE |
| LOW | tests/unit_tests.py | 676 | def test_chinese_fulltext_extract(self): | CODE |
| LOW | tests/unit_tests.py | 687 | def test_arabic_fulltext_extract(self): | CODE |
| LOW | tests/unit_tests.py | 700 | def test_spanish_fulltext_extract(self): | CODE |
| LOW | tests/unit_tests.py | 712 | def test_japanese_fulltext_extract(self): | CODE |
| LOW | tests/unit_tests.py | 734 | def test_thai_fulltext_extract(self): | CODE |
| LOW | tests/unit_tests.py | 754 | def test_article_pdf_ignoring(self): | CODE |
| LOW | tests/unit_tests.py | 765 | def test_article_pdf_fetching(self): | CODE |
| LOW | tests/data/html/nypost.com1.html | 182 | </script><script type="text/javascript">function janrainCaptureWidgetOnLoad() { | CODE |
| LOW | tests/data/html/youbeauty.com2.html | 1547 | function highlightUserAlreadyExistMessage() { | CODE |
| LOW | tests/data/html/youbeauty.com1.html | 1637 | function highlightUserAlreadyExistMessage() { | CODE |
| LOW | tests/data/html/nypost.com2.html | 182 | </script><script type="text/javascript">function janrainCaptureWidgetOnLoad() { | CODE |
| LOW | newspaper/outputformatters.py | 101 | def remove_negativescores_nodes(self): | CODE |
| LOW | newspaper/outputformatters.py | 141 | def remove_trailing_media_div(self): | CODE |
| LOW | newspaper/extractors.py | 1022 | def is_table_and_no_para_exist(self, e): | CODE |
| LOW | newspaper/extractors.py | 1034 | def is_nodescore_threshold_met(self, node, e): | CODE |
| LOW | newspaper/utils.py | 354 | def print_available_languages(): | CODE |
| LOW | newspaper/article.py | 552 | def throw_if_not_downloaded_verbose(self): | CODE |
| LOW | newspaper/article.py | 562 | def throw_if_not_parsed_verbose(self): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| CRITICAL | tests/data/html/nypost.com1.html | 414 | fyre.conv.BackplaneAuthDelegate.prototype.logout.call(this, delegate); | CODE |
| CRITICAL | tests/data/html/nypost.com2.html | 413 | fyre.conv.BackplaneAuthDelegate.prototype.logout.call(this, delegate); | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/data/html/bhg.com2.html | 141 | .socialToolbar li div.gig-button-container { padding: 0; } | COMMENT |
| LOW | tests/data/html/bhg.com2.html | 161 | COMMENT | |
| LOW | tests/data/html/reuters.com1.html | 801 | Reuters.utils.addLoadEvent(function() { Reuters.utils.replaceContent('searchModule', '/assets/searchIntercept?blob='+hig | COMMENT |
| LOW | newspaper/resources/text/stopwords-fr.txt | 1 | # Licensed to the Apache Software Foundation (ASF) under one or more | COMMENT |
| LOW | docs/conf.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | docs/conf.py | 21 | # If extensions (or modules to document with autodoc) are in another directory, | COMMENT |
| LOW | docs/conf.py | 41 | # The suffix of source filenames. | COMMENT |
| LOW | docs/conf.py | 61 | release = '0.0.2' | COMMENT |
| LOW | docs/conf.py | 81 | # If true, '()' will be appended to :func: etc. cross-reference text. | COMMENT |
| LOW | docs/conf.py | 101 | COMMENT | |
| LOW | docs/conf.py | 121 | # A shorter title for the navigation bar. Default is the same as html_title. | COMMENT |
| LOW | docs/conf.py | 141 | #html_extra_path = [] | COMMENT |
| LOW | docs/conf.py | 161 | COMMENT | |
| LOW | docs/conf.py | 181 | # contain a <link> tag referring to it. The value of this option must be the | COMMENT |
| LOW | docs/conf.py | 201 | # Additional stuff for the LaTeX preamble. | COMMENT |
| LOW | docs/conf.py | 221 | # If true, show page references after internal links. | COMMENT |
| LOW | docs/conf.py | 261 | # If false, no module index is generated. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/benchmarks.py | 11 | CODE | |
| LOW | tests/benchmarks.py | 14 | CODE | |
| LOW | tests/benchmarks.py | 15 | CODE | |
| LOW | tests/benchmarks.py | 16 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 10 | CODE | |
| LOW | newspaper/__init__.py | 12 | CODE | |
| LOW | newspaper/__init__.py | 12 | CODE | |
| LOW | newspaper/__init__.py | 14 | CODE | |
| LOW | newspaper/__init__.py | 15 | CODE | |
| LOW | newspaper/settings.py | 15 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/unit_tests.py | 549 | CODE | |
| LOW | newspaper/configuration.py | 106 | CODE | |
| LOW | newspaper/extractors.py | 237 | CODE | |
| LOW | newspaper/extractors.py | 492 | CODE | |
| LOW | newspaper/extractors.py | 629 | CODE | |
| LOW | newspaper/extractors.py | 773 | CODE | |
| LOW | newspaper/extractors.py | 885 | CODE | |
| LOW | newspaper/parsers.py | 193 | CODE | |
| LOW | newspaper/nlp.py | 180 | CODE | |
| LOW | newspaper/images.py | 86 | CODE | |
| LOW | newspaper/images.py | 228 | CODE | |
| LOW | newspaper/cleaners.py | 221 | CODE | |
| LOW | newspaper/urls.py | 102 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | newspaper/settings.py | 55 | except FileExistsError: | CODE |
| HIGH | newspaper/article.py | 422 | except OSError: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | newspaper/resources/text/stopwords-fr.txt | 16 | #----------------------------------------------------------------------- | COMMENT |
| MEDIUM | newspaper/resources/text/stopwords-sv.txt | 1 | #----------------------------------------------------------------------- | COMMENT |
| MEDIUM | newspaper/resources/text/stopwords-sv.txt | 3 | #----------------------------------------------------------------------- | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/data/html/upworthy.com2.html | 765 | <img src="#" id="placeholder" alt="Image Detail"> | CODE |
| LOW | tests/data/html/youbeauty.com2.html | 381 | $("#newsletter_subscription_email").attr("placeholder", "Please enter a valid email address") | CODE |
| LOW | tests/data/html/upworthy.com1.html | 749 | <img src="#" id="placeholder" alt="Image Detail"> | CODE |
| LOW | tests/data/html/youbeauty.com1.html | 381 | $("#newsletter_subscription_email").attr("placeholder", "Please enter a valid email address") | CODE |
| LOW⚡ | tests/data/html/nola.com1.html | 1221 | jQuery('#SearchKeyword').val(jQuery(this).attr('placeholder')).addClass('grayTxt').focus(function(){ | CODE |
| LOW⚡ | tests/data/html/nola.com1.html | 1224 | if(jQuery(this).val()==='')jQuery(this).addClass('grayTxt').val(jQuery(this).attr('placeholder')); | CODE |
| LOW⚡ | tests/data/html/nola.com1.html | 1226 | jQuery('#SearchKeyword').val(jQuery('#SearchKeyword').attr('placeholder')); | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/unit_tests.py | 99 | except Exception: | CODE |
| LOW | newspaper/parsers.py | 70 | except Exception: | CODE |
| LOW | newspaper/api.py | 66 | except Exception as e: | CODE |
| LOW | newspaper/images.py | 127 | except Exception as e: | CODE |
| LOW | newspaper/mthreading.py | 47 | except Exception: | CODE |
| LOW | newspaper/article.py | 442 | except Exception as e: | CODE |
| MEDIUM | newspaper/article.py | 165 | def _parse_scheme_file(self, path): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/data/html/teenvogue.com2.html | 2738 | <p>Ahead, we've rounded up our top 10 break-out style star picks for 2014. Click through to see who we chose, and don't | CODE |
| LOW | tests/data/html/teenvogue.com2.html | 3628 | <p>Ahead, we've rounded up our top 10 break-out style star picks for 2014. Click through to see who we chose, and don't | CODE |
| LOW | tests/data/html/teenvogue.com2.html | 5384 | introData.body = {"xml-fragment":{"text":"<p>Every year, we love looking back and picking ou | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | newspaper/configuration.py | 44 | # Set this to false if you don't care about getting images | COMMENT |
| LOW | newspaper/configuration.py | 76 | # Set this to False if you want to recompute the categories | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | docs/conf.py | 6 | # This file is execfile()d with the current directory set to its | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | newspaper/article.py | 417 | # TODO: implement in entirety | COMMENT |