Repository Analysis

Unstructured-IO/unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

8.4 Low AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of Unstructured-IO/unstructured, a HTML project with 15,129 GitHub stars. SynthScan v2.0 examined 467,013 lines of code across 1043 source files, recording 3198 pattern matches distributed across 18 syntactic categories. The overall adjusted score of 8.4 places this repository in the Low AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

8.4
Adjusted Score
8.4
Raw Score
100%
Time Factor
2026-07-13
Last Push
15.1K
Stars
HTML
Language
467.0K
Lines of Code
1.0K
Files
3.2K
Pattern Hits
2026-07-14
Scan Date
0.03
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 29MEDIUM 323LOW 2846

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 3198 distinct pattern matches across 18 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers2048 hits · 1860 pts
SeverityFileLineSnippetContext
LOWtest_unstructured/test_safe_http.py88 def test_allows_public_addresses(self, ip: str):CODE
LOWtest_unstructured/test_safe_http.py91 def test_fails_closed_on_unparseable_input(self):CODE
LOWtest_unstructured/test_safe_http.py102 def test_returns_lowercase_for_ascii_input(self):CODE
LOWtest_unstructured/test_safe_http.py111 def test_idn_collapses_to_punycode(self):CODE
LOWtest_unstructured/test_safe_http.py132 def test_rejects_non_http_scheme(self, url: str):CODE
LOWtest_unstructured/test_safe_http.py136 def test_rejects_missing_hostname(self):CODE
LOWtest_unstructured/test_safe_http.py145 def test_rejects_out_of_range_port(self, url: str):CODE
LOWtest_unstructured/test_safe_http.py177 def test_rejects_blocked_ip_literals(self, url: str):CODE
LOWtest_unstructured/test_safe_http.py182 def test_rejects_hostname_resolving_to_non_routable(self, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py188 def test_allows_public_hostname(self, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py193 def test_rejects_when_any_resolved_ip_is_blocked(self, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py201 def test_skips_validation_with_allow_private(self):CODE
LOWtest_unstructured/test_safe_http.py204 def test_skips_validation_with_env_var(self, monkeypatch):CODE
LOWtest_unstructured/test_safe_http.py216 def test_rejects_blocked_resolved_address_at_connect(self, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py223 def test_connects_to_validated_public_address(self, mock_getaddrinfo, MockSocket):CODE
LOWtest_unstructured/test_safe_http.py245 def test_returns_response_for_public_url(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py375 def test_rejects_proxies_kwarg(self, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py389 def test_mounts_safe_adapter_and_disables_trust_env(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py420 def test_proxy_manager_is_refused(self):CODE
LOWtest_unstructured/test_safe_http.py432 def test_same_origin_not_cross(self):CODE
LOWtest_unstructured/test_safe_http.py435 def test_different_host_is_cross(self):CODE
LOWtest_unstructured/test_safe_http.py438 def test_http_to_https_upgrade_is_not_cross(self):CODE
LOWtest_unstructured/test_safe_http.py441 def test_downgrade_on_same_nondefault_port_is_cross(self):CODE
LOWtest_unstructured/test_safe_http.py444 def test_explicit_default_port_same_origin_not_cross(self):CODE
LOWtest_unstructured/test_safe_http.py447 def test_unparseable_port_fails_safe(self):CODE
LOWtest_unstructured/test_safe_http.py452 def test_removes_credential_headers(self):CODE
LOWtest_unstructured/test_safe_http.py74 def test_blocks_sixtofour_embedded(self):CODE
LOWtest_unstructured/test_safe_http.py162 def test_rejects_blocked_hostname(self, hostname: str):CODE
LOWtest_unstructured/test_safe_http.py260 def test_applies_default_timeout(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py272 def test_respects_explicit_timeout(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py284 def test_forces_allow_redirects_false(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py296 def test_validates_redirect_targets(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py312 def test_follows_valid_redirect_chain(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py329 def test_enforces_max_redirects(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py342 def test_returns_response_on_missing_location_header(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py353 def test_rejects_non_routable_url_without_fetch(self):CODE
LOWtest_unstructured/test_safe_http.py359 def test_session_closed_on_validation_error(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_safe_http.py403 def test_allow_private_skips_adapter_and_permits_proxies(self, MockSession):CODE
LOWtest_unstructured/test_safe_http.py464 def test_removes_auth_and_cookies_kwargs(self):CODE
LOWtest_unstructured/test_safe_http.py480 def test_strips_credentials_on_cross_origin_redirect(self, MockSession, mock_getaddrinfo):CODE
LOWtest_unstructured/test_utils.py46def test_requires_dependencies_decorator():CODE
LOWtest_unstructured/test_utils.py54def test_requires_dependencies_decorator_multiple():CODE
LOWtest_unstructured/test_utils.py63def test_requires_dependencies_decorator_import_error():CODE
LOWtest_unstructured/test_utils.py72def test_requires_dependencies_decorator_import_error_multiple():CODE
LOWtest_unstructured/test_utils.py82def test_requires_dependencies_decorator_in_class():CODE
LOWtest_unstructured/test_utils.py315def test_only_returns_singleton_iterable():CODE
LOWtest_unstructured/test_utils.py321def test_only_raises_on_non_singleton_iterable():CODE
LOWtest_unstructured/test_utils.py327def test_calculate_shared_ngram_percentage_returns_null_vals_for_empty_str():CODE
LOWtest_unstructured/test_utils.py97def test_first_raises_if_empty(iterator):CODE
LOWtest_unstructured/test_utils.py114def test_only_raises_if_empty(iterator):CODE
LOWtest_unstructured/test_utils.py265def test_catch_overlapping_and_nested_bboxes(CODE
LOWtest_unstructured/test_utils.py291def test_catch_overlapping_and_nested_bboxes_non_overlapping_case():CODE
LOWtest_unstructured/test_utils.py339 def it_groups_elements_by_parent_id_with_orphans_in_none_group(self):CODE
LOWtest_unstructured/test_utils.py357 def it_assigns_orphans_to_previous_element_group_when_assign_orphans_is_true(self):CODE
LOWtest_unstructured/test_utils.py374 def it_keeps_first_orphan_in_none_group_when_assign_orphans_is_true(self):CODE
LOWtest_unstructured/test_cli_doctor.py31def test_resolve_specifier_pdf() -> None:CODE
LOWtest_unstructured/test_cli_doctor.py36def test_resolve_specifier_jpg_exact() -> None:CODE
LOWtest_unstructured/test_cli_doctor.py40def test_resolve_specifier_image_family() -> None:CODE
LOWtest_unstructured/test_cli_doctor.py46def test_resolve_specifier_audio_family() -> None:CODE
LOWtest_unstructured/test_cli_doctor.py52def test_resolve_specifier_email_partitioner_shortname() -> None:CODE
1988 more matches not shown…
Decorative Section Separators292 hits · 1048 pts
SeverityFileLineSnippetContext
MEDIUM…ingest/src/local-single-file-chunk-no-orig-elements.sh3# ------------------------------------------------------------------------------------------------COMMENT
MEDIUM…ingest/src/local-single-file-chunk-no-orig-elements.sh7# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py96# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py98# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py116# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py118# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py209# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py211# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py237# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py239# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py381# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py383# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py426# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py428# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py22# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/test_safe_http.py24# ---------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/unit_utils.py111# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/unit_utils.py113# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/unit_utils.py116# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/chunking/test_basic.py224# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/chunking/test_basic.py226# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/chunking/test_title.py30# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py32# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py35# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py546# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py548# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py551# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py705# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_title.py707# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py46# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py48# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py292# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py294# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py327# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py329# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py460# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py462# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py931# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py933# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py3299# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py3301# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py674# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py676# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py2820# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py2822# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py3572# ================================================================================================COMMENT
MEDIUMtest_unstructured/chunking/test_base.py3574# ================================================================================================COMMENT
MEDIUMtest_unstructured/partition/test_ndjson.py305# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_xml.py175# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_docx.py112# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_docx.py202# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_docx.py271# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_docx.py708# ================================================================================================COMMENT
MEDIUMtest_unstructured/partition/test_docx.py710# ================================================================================================COMMENT
MEDIUMtest_unstructured/partition/test_docx.py713# ================================================================================================COMMENT
MEDIUMtest_unstructured/partition/test_docx.py469# ------------------------------------------------------------------------------------------------COMMENT
MEDIUMtest_unstructured/partition/test_docx.py1121 # ┌───────┐COMMENT
MEDIUMtest_unstructured/partition/test_docx.py1151 # ┌───────┬───┬───┐COMMENT
MEDIUMtest_unstructured/partition/test_docx.py1159 # └───────┘COMMENT
MEDIUMtest_unstructured/partition/test_email.py392# ================================================================================================COMMENT
232 more matches not shown…
Fake / Example Data422 hits · 403 pts
SeverityFileLineSnippetContext
LOWREADME.md150elements = partition(filename="example-docs/eml/fake-email.eml")CODE
LOW…t-with-permissions/SitePages/This-is-a-title.aspx.json49 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales cCODE
LOW…t-with-permissions/SitePages/This-is-a-title.aspx.json49 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales cCODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.json21 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.json21 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.json23 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n\n\n2023-08-26T14:00:00.000Z\nName3\n#CODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.json23 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n\n\n2023-08-26T14:00:00.000Z\nName3\n#CODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.json20 "text_as_html": "<table border=\"1\" class=\"dataframe\">\n <tbody>\n <tr>\n <td></td>\n <td>AssigneCODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.json22 "text": "\n\n\n\nAssignee\nCheckboxes\nDates\nName\nNotes 1\nStatus\n\n\n0.0\n{'id': 'usrNdSxRQjglQU4eR', 'email': 'CODE
LOW…expected-structured-output/google-drive/fake.docx.json5 "text": "Lorem ipsum dolor sit amet.",CODE
LOW…expected-structured-output/google-drive/fake.docx.json5 "text": "Lorem ipsum dolor sit amet.",CODE
LOW…d-structured-output/google-drive/nested/fake.docx.json5 "text": "Lorem ipsum dolor sit amet.",CODE
LOW…d-structured-output/google-drive/nested/fake.docx.json5 "text": "Lorem ipsum dolor sit amet.",CODE
LOW…gest/expected-structured-output/jira-diff/1/10000.json219 "text": "Unstructured Devops My comment 1 Unstructured Devops My attachment image lorem ipsum:",CODE
LOW…gest/expected-structured-output/jira-diff/1/10000.json329 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. [Nam quid possumus facere melius?|http://loripsum.CODE
LOW…gest/expected-structured-output/jira-diff/1/10000.json329 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. [Nam quid possumus facere melius?|http://loripsum.CODE
LOW…-output/Sharepoint/SitePages/This-is-a-title.aspx.json49 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales cCODE
LOW…-output/Sharepoint/SitePages/This-is-a-title.aspx.json49 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales cCODE
LOW…int-with-permissions/SitePages/This-is-a-title.aspx.md3Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales consequat turpCODE
LOW…int-with-permissions/SitePages/This-is-a-title.aspx.md3Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam ex tellus, sodales non nulla et, sodales consequat turpCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md27 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md27 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md36 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblbj2vBlL2dN2xqq.md36 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md36 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/appJ43QmP8I17zu88/tblfu7DzEcCWNKwP4.md36 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md18 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md36 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblBoUk54tWXGqYai.md36 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md18 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md18 <td># Quis est enim, in quo sit cupiditas, quin recte cupidus dici possit?\nLorem ipsum dolor sit amet, consectetuCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md27 <td># Nescio quo modo praetervolavit oratio.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Quid ad utiCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md36 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…n/airtable-diff/app5YQxSfp220fWtm/tblxdPc7L2meGIZLE.md36 <td># Nec lapathi suavitatem acupenseri Galloni Laelius anteponebat, sed suavitatem ipsam neglegebat;\nLorem ipsumCODE
LOW…d-structured-output-markdown/google-drive/fake.docx.md1Lorem ipsum dolor sit amet.CODE
LOW…d-structured-output-markdown/google-drive/fake.docx.md1Lorem ipsum dolor sit amet.CODE
LOW…tured-output-markdown/google-drive/nested/fake.docx.md1Lorem ipsum dolor sit amet.CODE
LOW…tured-output-markdown/google-drive/nested/fake.docx.md1Lorem ipsum dolor sit amet.CODE
LOW…pected-structured-output-markdown/jira-diff/1/10000.md10Unstructured Devops My comment 1 Unstructured Devops My attachment image lorem ipsum:CODE
LOW…pected-structured-output-markdown/jira-diff/1/10000.md15Lorem ipsum dolor sit amet, consectetur adipiscing elit. [Nam quid possumus facere melius?|http://loripsum.net/] Ita relCODE
362 more matches not shown…
Unused Imports151 hits · 132 pts
SeverityFileLineSnippetContext
LOWtest_unstructured/test_safe_http.py3CODE
LOWtest_unstructured/test_utils.py1CODE
LOWtest_unstructured/test_utils.py49CODE
LOWtest_unstructured/test_utils.py57CODE
LOWtest_unstructured/test_utils.py58CODE
LOWtest_unstructured/test_utils.py66CODE
LOWtest_unstructured/test_utils.py75CODE
LOWtest_unstructured/test_utils.py76CODE
LOWtest_unstructured/test_utils.py86CODE
LOWtest_unstructured/test_cli_doctor.py3CODE
LOWtest_unstructured/test_telemetry.py8CODE
LOWtest_unstructured/unit_utils.py3CODE
LOWtest_unstructured/unit_utils.py10CODE
LOWtest_unstructured/unit_utils.py10CODE
LOWtest_unstructured/unit_utils.py10CODE
LOWtest_unstructured/unit_utils.py21CODE
LOWtest_unstructured/unit_utils.py21CODE
LOWtest_unstructured/unit_utils.py21CODE
LOWtest_unstructured/metrics/test_element_type.py1CODE
LOWtest_unstructured/staging/test_base.py14CODE
LOWtest_unstructured/chunking/test_dispatch.py5CODE
LOWtest_unstructured/chunking/test_basic.py7CODE
LOWtest_unstructured/chunking/test_table_isolation.py10CODE
LOWtest_unstructured/chunking/test_title.py5CODE
LOWtest_unstructured/chunking/test_base.py5CODE
LOWtest_unstructured/common/test_html_table.py5CODE
LOWtest_unstructured/documents/test_elements.py5CODE
LOWtest_unstructured/partition/test_ndjson.py3CODE
LOWtest_unstructured/partition/test_xml.py3CODE
LOWtest_unstructured/partition/test_docx.py5CODE
LOWtest_unstructured/partition/test_email.py3CODE
LOWtest_unstructured/partition/test_odt.py5CODE
LOWtest_unstructured/partition/test_tsv.py3CODE
LOWtest_unstructured/partition/test_md.py1CODE
LOWtest_unstructured/partition/test_audio.py5CODE
LOWtest_unstructured/partition/test_org.py1CODE
LOWtest_unstructured/partition/test_ppt.py3CODE
LOWtest_unstructured/partition/test_xlsx.py5CODE
LOWtest_unstructured/partition/test_msg.py3CODE
LOWtest_unstructured/partition/test_text.py3CODE
LOWtest_unstructured/partition/test_rst.py1CODE
LOWtest_unstructured/partition/test_rtf.py1CODE
LOWtest_unstructured/partition/test_auto.py3CODE
LOWtest_unstructured/partition/test_json.py3CODE
LOWtest_unstructured/partition/test_pptx.py5CODE
LOWtest_unstructured/partition/test_doc.py5CODE
LOWtest_unstructured/partition/test_csv.py3CODE
LOWtest_unstructured/partition/test_epub.py1CODE
LOW…tured/partition/utils/ocr_models/test_ocr_interface.py5CODE
LOW…t_unstructured/partition/html/test_xss_sanitization.py12CODE
LOWtest_unstructured/partition/html/test_partition.py5CODE
LOWtest_unstructured/partition/common/test_lang.py5CODE
LOWtest_unstructured/partition/common/test_metadata.py5CODE
LOW…nstructured/partition/common/test_json_partitioning.py3CODE
LOWtest_unstructured/partition/pdf_image/test_image.py1CODE
LOWtest_unstructured/file_utils/test_model.py3CODE
LOWtest_unstructured/file_utils/test_filetype.py5CODE
LOWscripts/performance/benchmark_partition.py23CODE
LOWscripts/performance/compare_benchmark.py38CODE
LOWunstructured/cli.py3CODE
91 more matches not shown…
Cross-File Repetition22 hits · 110 pts
SeverityFileLineSnippetContext
HIGHtest_unstructured/partition/test_docx.py0all default arguments for `htmlpartitioneroptions`. individual argument values can be changed to suit each test. makes cSTRING
HIGHtest_unstructured/partition/test_xlsx.py0all default arguments for `htmlpartitioneroptions`. individual argument values can be changed to suit each test. makes cSTRING
HIGHtest_unstructured/partition/test_pptx.py0all default arguments for `htmlpartitioneroptions`. individual argument values can be changed to suit each test. makes cSTRING
HIGHtest_unstructured/partition/test_msg.py0all default arguments for `htmlpartitioneroptions`. individual argument values can be changed to suit each test. makes cSTRING
HIGHtest_unstructured/partition/html/test_partition.py0all default arguments for `htmlpartitioneroptions`. individual argument values can be changed to suit each test. makes cSTRING
HIGHunstructured/partition/docx.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/email.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/xlsx.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/msg.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/pptx.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/html/partition.py0encapsulates partitioning option validation, computation, and application of defaults.STRING
HIGHunstructured/partition/docx.py0the best last-modified date available, none if no sources are available.STRING
HIGHunstructured/partition/csv.py0the best last-modified date available, none if no sources are available.STRING
HIGHunstructured/partition/xlsx.py0the best last-modified date available, none if no sources are available.STRING
HIGHunstructured/partition/pptx.py0the best last-modified date available, none if no sources are available.STRING
HIGHunstructured/partition/html/partition.py0the best last-modified date available, none if no sources are available.STRING
HIGHunstructured/partition/docx.py0the best available file-path for this document or `none` if unavailable.STRING
HIGHunstructured/partition/xlsx.py0the best available file-path for this document or `none` if unavailable.STRING
HIGHunstructured/partition/pptx.py0the best available file-path for this document or `none` if unavailable.STRING
HIGHunstructured/partition/org.py0partitions an rst document. the document is first converted to html and then partitioned using partition_html. parameterSTRING
HIGHunstructured/partition/rtf.py0partitions an rst document. the document is first converted to html and then partitioned using partition_html. parameterSTRING
HIGHunstructured/partition/rst.py0partitions an rst document. the document is first converted to html and then partitioned using partition_html. parameterSTRING
Over-Commented Block70 hits · 70 pts
SeverityFileLineSnippetContext
LOWtest_unstructured_ingest/json-to-text.sh1#!/usr/bin/env bashCOMMENT
LOW…ed-output-markdown/s3/2023-Jan-economic-outlook.pdf.md61The balance of risks to the global outlook remains tilted to the downside, with scope for lower growth and higher inflatCOMMENT
LOW…-structured-output-markdown/s3/Silent-Giant-(1).pdf.md61However, markets fail to give due credit to electricity generators, such as nuclear energy, that are able to meet these COMMENT
LOW…-structured-output-markdown/s3/Silent-Giant-(1).pdf.md81# iCOMMENT
LOW…ed-output-markdown/s3/recalibrating-risk-report.pdf.md418COMMENT
LOW…uctured-output-markdown/confluence-diff/MFS/1605928.md1# \uD83D\uDDD3 DateCOMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"COMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md21<body> <div class="header"> <div class="header-content"> <a href="http://www.weather.gov" class="header-nws"><img src="/COMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md41<p><strong>First, take steps to better prepare for the seasonal hazards weather can throw at you.</strong><br /> This coCOMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md61# widthCOMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md81<!-COMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md101# <div class="divCOMMENT
LOW…tructured-output-markdown/azure/spring-weather.html.md121>COMMENT
LOW…nedrive/utic-test-ingest-fixtures/tests-example.xls.md1<table><tr><td>MC</td><td>What is 2+2?</td><td>4</td><td>correct</td><td>3</td><td>incorrect</td><td/><td/><td/></tr><trCOMMENT
LOW…ctured-output-markdown/opensearch/movies-4-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-1-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-8-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-2-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-7-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-9-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-5-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-6-57554198.md1# AmericanCOMMENT
LOW…ctured-output-markdown/opensearch/movies-3-57554198.md1# AmericanCOMMENT
LOW…ed-output-markdown/mongodb/659daefa21dd8c9054b084b7.md11901COMMENT
LOW…ed-output-markdown/mongodb/659daefa21dd8c9054b084b6.md11901COMMENT
LOW…ed-output-markdown/mongodb/659daefa21dd8c9054b084b9.md11908COMMENT
LOW…ed-output-markdown/mongodb/659daefa21dd8c9054b084b8.md11908COMMENT
LOW…uctured-output-markdown/hubspot/products/2362691415.md1Example product description. This is a text containing relevant information pertaining to ProductA. The text represents COMMENT
LOW…uctured-output-markdown/hubspot/products/2362691417.md1Example product description. This is a text containing relevant information pertaining to ProductC. The text represents COMMENT
LOW…uctured-output-markdown/hubspot/products/2362691416.md1Example product description. This is a text containing relevant information pertaining to ProductB. The text represents COMMENT
LOW…rkdown/local-single-file/UDHR_first_article_all.txt.md81Chin, Matu Thlangboeih he rhimomna, vanpitna, yalpona hamhmoel ka tawn thlang la cuun la ng’om u. Thlanghing he athae-thCOMMENT
LOW…rkdown/local-single-file/UDHR_first_article_all.txt.md101# Chinese, Mandarin (Tianjin)COMMENT
LOW…pected-structured-output-markdown/jira-diff/1/10002.md1# IssueID_IssueKey:10002 JCTP1-3COMMENT
LOW…pected-structured-output-markdown/jira-diff/1/10013.md1# IssueID_IssueKey:10013 JCTP1-4COMMENT
LOW…pected-structured-output-markdown/jira-diff/1/10000.md1# IssueID_IssueKey:10000 JCTP1-1COMMENT
LOW…pected-structured-output-markdown/jira-diff/1/10000.md21# test.atlassian.net/rest/api/2/attachment/10000COMMENT
LOW…pected-structured-output-markdown/jira-diff/1/10001.md1# IssueID_IssueKey:10001 JCTP1-2COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP3/10014.md1# IssueID_IssueKey:10014 JCTP3-1COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP2/10009.md1# IssueID_IssueKey:10009 JCTP2-7COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP2/10006.md1# IssueID_IssueKey:10006 JCTP2-4COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP2/10012.md1# IssueID_IssueKey:10012 JCTP2-10COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP2/10010.md1# IssueID_IssueKey:10010 JCTP2-8COMMENT
LOW…ed-structured-output-markdown/jira-diff/JCTP2/10015.md1# IssueID_IssueKey:10015 JCTP2-11COMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9FIAU.xml.md1# attributes.type: CampaignCOMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9FIAU.xml.md21# NumberOfResponses: 0COMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9GIAU.xml.md1# attributes.type: CampaignCOMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9GIAU.xml.md21# NumberOfResponses: 0COMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9EIAU.xml.md1# attributes.type: CampaignCOMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9EIAU.xml.md21# NumberOfResponses: 0COMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9HIAU.xml.md1# attributes.type: CampaignCOMMENT
LOW…markdown/salesforce/Campaign/701Hu000001eX9HIAU.xml.md21# NumberOfResponses: 0COMMENT
LOW…red-output-markdown/elasticsearch/movies-4-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-1-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-8-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-2-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-7-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-9-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-5-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-6-57554198.md1# AmericanCOMMENT
LOW…red-output-markdown/elasticsearch/movies-3-57554198.md1# AmericanCOMMENT
10 more matches not shown…
Deep Nesting70 hits · 59 pts
SeverityFileLineSnippetContext
LOWtest_unstructured/partition/test_audio.py190CODE
LOWtest_unstructured/partition/test_auto.py1451CODE
LOWtest_unstructured/partition/pdf_image/test_pdf.py286CODE
LOWtest_unstructured/partition/pdf_image/test_pdf.py969CODE
LOWtest_unstructured/partition/pdf_image/test_pdf.py1352CODE
LOWtest_unstructured/partition/pdf_image/test_pdf.py1874CODE
LOWtest_unstructured/file_utils/test_encoding.py15CODE
LOWunstructured/utils.py366CODE
LOWunstructured/safe_http.py190CODE
LOWunstructured/metrics/text_extraction.py123CODE
LOWunstructured/metrics/evaluate.py510CODE
LOWunstructured/metrics/evaluate.py599CODE
LOWunstructured/metrics/evaluate.py282CODE
LOWunstructured/metrics/table/table_alignment.py59CODE
LOWunstructured/metrics/table/table_extraction.py15CODE
LOWunstructured/staging/weaviate.py89CODE
LOWunstructured/staging/base.py196CODE
LOWunstructured/staging/base.py359CODE
LOWunstructured/staging/base.py557CODE
LOWunstructured/staging/base.py711CODE
LOWunstructured/chunking/base.py734CODE
LOWunstructured/chunking/base.py887CODE
LOWunstructured/chunking/base.py1462CODE
LOWunstructured/chunking/base.py896CODE
LOWunstructured/nlp/tokenize.py52CODE
LOWunstructured/documents/elements.py367CODE
LOWunstructured/partition/md.py48CODE
LOWunstructured/partition/docx.py393CODE
LOWunstructured/partition/docx.py483CODE
LOWunstructured/partition/docx.py785CODE
LOWunstructured/partition/docx.py833CODE
LOWunstructured/partition/docx.py502CODE
LOWunstructured/partition/docx.py838CODE
LOWunstructured/partition/pdf.py253CODE
LOWunstructured/partition/pdf.py480CODE
LOWunstructured/partition/pdf.py618CODE
LOWunstructured/partition/pdf.py779CODE
LOWunstructured/partition/pdf.py1400CODE
LOWunstructured/partition/api.py232CODE
LOWunstructured/partition/text.py111CODE
LOWunstructured/partition/xlsx.py45CODE
LOWunstructured/partition/xlsx.py441CODE
LOWunstructured/partition/pptx.py141CODE
LOWunstructured/partition/pptx.py217CODE
LOWunstructured/partition/utils/sorting.py99CODE
LOWunstructured/partition/utils/sorting.py138CODE
LOWunstructured/partition/html/transformations.py302CODE
LOWunstructured/partition/common/lang.py328CODE
LOWunstructured/partition/common/lang.py412CODE
LOWunstructured/partition/common/common.py34CODE
LOWunstructured/partition/pdf_image/pdf_image_utils.py117CODE
LOWunstructured/partition/pdf_image/pdf_image_utils.py313CODE
LOWunstructured/partition/pdf_image/pdfminer_processing.py467CODE
LOWunstructured/partition/pdf_image/pdfminer_processing.py772CODE
LOWunstructured/partition/pdf_image/pdfminer_processing.py1234CODE
LOWunstructured/partition/pdf_image/ocr.py105CODE
LOWunstructured/partition/pdf_image/ocr.py210CODE
LOWunstructured/partition/pdf_image/pdfminer_utils.py46CODE
LOWunstructured/partition/pdf_image/pdfminer_utils.py155CODE
LOWunstructured/partition/pdf_image/pdfminer_utils.py461CODE
10 more matches not shown…
Self-Referential Comments19 hits · 57 pts
SeverityFileLineSnippetContext
MEDIUMtest_unstructured_ingest/structured-json-to-html.sh3# Define the input and output top directoriesCOMMENT
MEDIUMtest_unstructured_ingest/structured-json-to-markdown.sh3# Define the input and output top directoriesCOMMENT
MEDIUMtest_unstructured/partition/utils/test_xycut.py114 # Define the expected image with the square drawnCOMMENT
MEDIUMtest_unstructured/partition/html/test_partition.py1505 # Create a file-like object with empty contentCOMMENT
MEDIUMtest_unstructured/partition/html/test_partition.py1509 # Create a temporary file with the given contentCOMMENT
MEDIUMtest_unstructured/partition/pdf_image/test_pdf.py1249 # Create an exception that will be raised directly after OCR is called to stop executionCOMMENT
MEDIUM…nstructured/partition/pdf_image/test_merge_elements.py30 # Create a PageLayout for the inferred layoutCOMMENT
MEDIUMexample-docs/logger.py7# Create a custom logging levelCOMMENT
MEDIUMexample-docs/logger.py12# Create a custom log method for the "DETAIL" levelCOMMENT
MEDIUM…rs/destination_connector/create-opensearch-instance.sh7# Create the Opensearch clusterCOMMENT
MEDIUM…elpers/source_connector/create-and-check-opensearch.sh7# Create the Opensearch clusterCOMMENT
MEDIUM…destination_connector/create-elasticsearch-instance.sh8# Create the Elasticsearch clusterCOMMENT
MEDIUM…t-helpers/source_connector/create-fill-and-check-es.sh8# Create the Elasticsearch clusterCOMMENT
MEDIUMscripts/kafka-test-helpers/create-kafka-instance.sh7# Create the Weaviate instanceCOMMENT
MEDIUMscripts/sql-test-helpers/create-sql-instance.sh9# Create the SQL instanceCOMMENT
MEDIUM…ipts/weaviate-test-helpers/create-weaviate-instance.sh7# Create the Weaviate instanceCOMMENT
MEDIUMscripts/performance/benchmark-local.sh4# This file is separated out to allow us to easily execute this part of the test script inside a Docker container.COMMENT
MEDIUMunstructured/logger.py6# Create a custom logging levelCOMMENT
MEDIUMunstructured/logger.py11# Create a custom log method for the "DETAIL" levelCOMMENT
Redundant / Tautological Comments26 hits · 42 pts
SeverityFileLineSnippetContext
LOWtest_unstructured_ingest/test-ingest-src.sh61 # Check if the test is in tests_to_ignoreCOMMENT
LOW…structured_ingest/python/test-produce-kafka-message.py47 # Read the file in binary mode and encode content in base64COMMENT
LOW…structured_ingest/python/test-ingest-astradb-output.py38 # Print the resultsCOMMENT
LOWtest_unstructured/partition/utils/test_xycut.py132 # Check if the resulting image contains the expected shapes and labelsCOMMENT
LOWtest_unstructured/partition/utils/test_xycut.py178 # Check if vis_points was called with the correct argumentsCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py352 # Check if the final layout contains the original layout elementsCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py356 # Check if the final layout contains the OCR-derived elementsCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py359 # Check if the OCR-derived elements that are subregions of layout elements are removedCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py381 # Check if the out layout's text attribute is updated with aggregated OCR textCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py384 # Check if the final layout contains both original elements and OCR-derived elementsCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py530 # Check if the out layout's text attribute is updated with aggregated OCR textCOMMENT
LOWtest_unstructured/partition/pdf_image/test_ocr.py533 # Check if the final layout contains both original elements and OCR-derived elementsCOMMENT
LOW…structured/partition/pdf_image/test_pdf_image_utils.py370 # Check if images for both layouts were savedCOMMENT
LOW…structured/partition/pdf_image/test_pdf_image_utils.py391 # Check if only the inferred layout image was saved if extracted layout is NoneCOMMENT
LOWscripts/check-new-release-version.sh22# Check if the current version is a non-dev version and not matching the main versionCOMMENT
LOWscripts/user/u-tables-inspect.sh33# Check if the directory exists, if not create itCOMMENT
LOWscripts/user/u-tables-inspect.sh65 # Open the file in a new browser windowCOMMENT
LOWscripts/user/process-pdf-parallel-through-api.sh31# Check if UNST_API_KEY is setCOMMENT
LOWscripts/user/process-pdf-parallel-through-api.sh49# Check if PDF parts directory existsCOMMENT
LOWscripts/user/unstructured-get-json.sh198# Check if vlm-provider or vlm-model are provided without --vlmCOMMENT
LOWunstructured/metrics/utils.py238 # Check if the file existsCOMMENT
LOWunstructured/staging/base.py181 # Check if we hit the size limit or if data is actually incompleteCOMMENT
LOWunstructured/embed/voyageai.py113 # Check if adding this text would exceed limitsCOMMENT
LOWunstructured/partition/docx.py907 # Check if category depth can be determined from style ilvlCOMMENT
LOWunstructured/partition/pdf_image/pdfminer_processing.py444 # Check if text is low_fidelity:COMMENT
LOWunstructured/partition/pdf_image/pdfminer_utils.py447 # Check if this is a duplicate of the last LTCharCOMMENT
Excessive Try-Catch Wrapping23 hits · 30 pts
SeverityFileLineSnippetContext
LOWtest_unstructured/partition/pdf_image/test_pdf.py1154 except Exception as e:CODE
LOWscripts/performance/quick_partition_bench.py41 except Exception as exc: # noqa: BLE001CODE
LOWunstructured/doctor.py54 except Exception as exc: # noqa: BLE001 — surface libmagic load/binary issuesCODE
MEDIUMunstructured/doctor.py47def _libmagic_status() -> tuple[Status, str]:CODE
LOWunstructured/metrics/evaluate.py197 except Exception as e:CODE
LOWunstructured/metrics/table/table_extraction.py163 except Exception as e:CODE
MEDIUMunstructured/metrics/table/table_extraction.py164 print(f"Error converting ground truth data: {e}")CODE
LOWunstructured/metrics/table/table_extraction.py254 except Exception as e:CODE
MEDIUMunstructured/metrics/table/table_extraction.py255 print(f"Error converting Unstructured table data: {e}")CODE
LOWunstructured/partition/docx.py534 except Exception as e:CODE
LOWunstructured/partition/docx.py794 except Exception as e:CODE
LOWunstructured/partition/pdf.py326 except Exception as e:CODE
LOWunstructured/partition/pdf.py713 except Exception:CODE
LOWunstructured/partition/pdf.py740 except Exception as e:CODE
LOW…ition/utils/speech_to_text/speech_to_text_interface.py82 except Exception as e:CODE
LOW…ructured/partition/utils/speech_to_text/whisper_stt.py48 except Exception as exc:CODE
LOWunstructured/partition/common/json_partitioning.py69 except Exception as e:CODE
LOWunstructured/partition/pdf_image/pdf_image_utils.py389 except Exception as e:CODE
LOWunstructured/partition/pdf_image/pdfminer_processing.py1052 except Exception:CODE
LOWunstructured/partition/pdf_image/pdfminer_processing.py1074 except Exception:CODE
LOWunstructured/partition/pdf_image/ocr.py202 except Exception as e:CODE
LOW…red/partition/pdf_image/analysis/bbox_visualisation.py682 except Exception as ex: # noqa: E722CODE
LOW…red/partition/pdf_image/analysis/bbox_visualisation.py704 except Exception as ex: # noqa: E722CODE
Cross-Language Confusion3 hits · 22 pts
SeverityFileLineSnippetContext
HIGHtest_unstructured/staging/test_base.py152 assert df.type.equals(expected_df.type) is True # type: ignoreCODE
HIGHtest_unstructured/staging/test_base.py153 assert df.text.equals(expected_df.text) is True # type: ignoreCODE
HIGHunstructured/partition/common/json_partitioning.py102 # -- scalar (str / number / bool / null) --COMMENT
Docstring Block Structure4 hits · 20 pts
SeverityFileLineSnippetContext
HIGHunstructured/metrics/text_extraction.py76 Calculates edit distance using Levenshtein distance between two strings. Args: output (str): The targeSTRING
HIGHunstructured/partition/html/transformations.py378 Parses the given HTML code and converts it into an Element object. Args: html_code (str): The HTML codSTRING
HIGHunstructured/partition/pdf_image/pdf_image_utils.py35 Write an image to a specified file path, supporting both PIL Image and numpy ndarray formats. Parameters: STRING
HIGHunstructured/file_utils/filetype.py75Determine file-type of specified file using libmagic and/or fallback methods. One of `file_path` or `file` must be STRING
AI Structural Patterns18 hits · 16 pts
SeverityFileLineSnippetContext
LOWunstructured/staging/datasaur.py24CODE
LOWunstructured/embed/vertexai.py64CODE
LOWunstructured/chunking/basic.py24CODE
LOWunstructured/chunking/title.py23CODE
LOWunstructured/documents/elements.py238CODE
LOWunstructured/partition/pdf.py127CODE
LOWunstructured/partition/pdf.py253CODE
LOWunstructured/partition/pdf.py779CODE
LOWunstructured/partition/pdf.py1133CODE
LOWunstructured/partition/pdf.py1400CODE
LOWunstructured/partition/api.py24CODE
LOWunstructured/partition/image.py17CODE
LOWunstructured/partition/auto.py26CODE
LOWunstructured/partition/html/partition.py28CODE
LOWunstructured/partition/common/common.py157CODE
LOWunstructured/partition/pdf_image/ocr.py35CODE
LOWunstructured/partition/pdf_image/ocr.py105CODE
LOWunstructured/partition/pdf_image/analysis/tools.py57CODE
AI Slop Vocabulary7 hits · 15 pts
SeverityFileLineSnippetContext
MEDIUMunstructured/chunking/base.py685 # -- efficient and definitely more robust than hoping two different computations of combinedCOMMENT
MEDIUMunstructured/common/html_table.py66 # -- root is always a `<table>` element so far but let's be robust --COMMENT
LOWunstructured/partition/docx.py404 # -- functions like `._iter_paragraph_elements()` where the "just return when done"COMMENT
LOWunstructured/partition/pptx.py152 # -- functions like `._iter_shape_elements()` where the "just return when done"COMMENT
MEDIUMunstructured/partition/pptx.py432 # -- any documents, however there's no reason not to make the mechanism robust againstCOMMENT
LOWunstructured/partition/pdf_image/pdfminer_processing.py89 # TODO: refactor to just use np array as inputCOMMENT
LOWunstructured/partition/pdf_image/ocr.py391 # results. Can we just use ocr bounding boxes (gonna be many but at least we saveCOMMENT
Modern Structural Boilerplate13 hits · 13 pts
SeverityFileLineSnippetContext
LOWscripts/performance/benchmark_partition.py35logger = logging.getLogger(__name__)CODE
LOWscripts/performance/benchmark_partition.py86def _set_github_output(key: str, value: str) -> None:CODE
LOWscripts/performance/compare_benchmark.py51logger = logging.getLogger(__name__)CODE
LOWscripts/convert/elements_json_to_format.py10logger = logging.getLogger(__name__)CODE
LOWscripts/convert/rendered_html_from_elements.py24logger = logging.getLogger(__name__)CODE
LOWunstructured/safe_http.py23logger = logging.getLogger(__name__)CODE
LOWunstructured/chunking/__init__.py15__all__ = [CODE
LOWunstructured/nlp/tokenize.py19logger = logging.getLogger(__name__)CODE
LOWunstructured/partition/utils/config.py53 def _setup_tmpdir(self, tmpdir: str) -> None:CODE
LOWunstructured/partition/utils/speech_to_text/__init__.py8__all__ = ["SpeechToTextAgent", "TranscriptionSegment"]CODE
LOWunstructured/partition/html/convert.py11logger = logging.getLogger(__name__)CODE
LOWunstructured/partition/html/convert.py80 def set_children(self, children: list["ElementHtml"]) -> None:STRING
LOWunstructured/partition/html/__init__.py3__all__ = ["partition_html"]CODE
Dead Code5 hits · 10 pts
SeverityFileLineSnippetContext
MEDIUM…/airtable-test-helpers/create_scale_test_components.py47CODE
MEDIUMunstructured/partition/docx.py990CODE
MEDIUMunstructured/partition/pptx.py512CODE
MEDIUMunstructured/partition/html/parser.py579CODE
MEDIUMunstructured/partition/html/parser.py901CODE
Example Usage Blocks4 hits · 6 pts
SeverityFileLineSnippetContext
LOWscripts/performance/benchmark.sh3# Usage:COMMENT
LOWscripts/performance/profile.sh8# Usage:COMMENT
LOWscripts/performance/profile.sh23# Usage example:COMMENT
LOWunstructured/metrics/object_detection.py705 # Example usageCOMMENT
Slop Phrases1 hit · 3 pts
SeverityFileLineSnippetContext
MEDIUM…ctured/partition/pdf_image/test_pdfminer_processing.py579# Now you can use it in testsCOMMENT