Toolkit for linearizing PDFs for LLM datasets/training
This report summarizes scanner evidence for allenai/olmocr, a Python project with 19,734 GitHub stars. SynthScan examined 77,441 lines of code across 345 source files, recording 1500 pattern matches distributed across 22 syntactic categories. The legacy adjusted pattern signal is 31.3 (High pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 860 finding(s).
Are imports declared and package/API names verified? 96 finding(s).
Are wrappers, guards, and repeated structures necessary? 857 finding(s).
Which changes need focused human review? 1500 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 1500 distinct pattern matches across 22 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_tests.py | 1578 | except Exception as e: | CODE |
| LOW | tests/test_tests.py | 1600 | except Exception as e: | CODE |
| LOW | tests/test_tests.py | 1612 | except Exception as e: | CODE |
| LOW | tests/test_tests.py | 1625 | except Exception as e: | CODE |
| LOW | tests/test_tests.py | 1645 | except Exception as e: | CODE |
| LOW | tests/test_tests.py | 2214 | except Exception as e: | CODE |
| LOW⚡ | olmocr/image_utils.py | 49 | except Exception as e: | CODE |
| MEDIUM⚡ | olmocr/image_utils.py | 50 | print(f"Error: {e}") | CODE |
| LOW⚡ | olmocr/image_utils.py | 59 | except Exception as e: | CODE |
| MEDIUM⚡ | olmocr/image_utils.py | 60 | print(f"Error: {e}") | CODE |
| MEDIUM | olmocr/image_utils.py | 44 | def is_png(file_path): | CODE |
| MEDIUM | olmocr/image_utils.py | 54 | def is_jpeg(file_path): | CODE |
| LOW | olmocr/s3_utils.py | 141 | except Exception as e: | CODE |
| LOW | olmocr/s3_utils.py | 243 | except Exception as e: | CODE |
| LOW | olmocr/s3_utils.py | 281 | except Exception as e: | CODE |
| LOW | olmocr/s3_utils.py | 323 | except Exception as e: | CODE |
| LOW | olmocr/s3_utils.py | 349 | except Exception as e: | CODE |
| MEDIUM | olmocr/s3_utils.py | 277 | def download_blob(blob, local_file_path): | CODE |
| MEDIUM | olmocr/pipeline.py | 845 | def _kill_proc(): | CODE |
| MEDIUM | olmocr/pipeline.py | 1115 | def process_output_file(s3_path): | CODE |
| LOW | olmocr/pipeline.py | 228 | except Exception as e: | CODE |
| LOW | olmocr/pipeline.py | 466 | except Exception as e: | CODE |
| LOW | olmocr/pipeline.py | 570 | except Exception as e: | CODE |
| LOW | olmocr/pipeline.py | 803 | except Exception as e: | CODE |
| LOW | olmocr/pipeline.py | 890 | except Exception as ex: | CODE |
| LOW | olmocr/pipeline.py | 948 | except Exception: | CODE |
| LOW | olmocr/pipeline.py | 974 | except Exception: | CODE |
| LOW | olmocr/pipeline.py | 1149 | except Exception as e: | CODE |
| LOW | olmocr/pipeline.py | 1395 | except Exception as e: | CODE |
| LOW | olmocr/work_queue.py | 213 | except Exception as e: | CODE |
| LOW | olmocr/bench/benchmark.py | 111 | except Exception as e: | CODE |
| LOW | olmocr/bench/benchmark.py | 121 | except Exception as e: | CODE |
| MEDIUM | olmocr/bench/benchmark.py | 203 | print("Error: /pdfs folder must exist in your data directory.", file=sys.stderr) | CODE |
| MEDIUM | olmocr/bench/benchmark.py | 209 | print(f"Error: No PDF files found in {pdf_folder}", file=sys.stderr) | CODE |
| MEDIUM | olmocr/bench/benchmark.py | 220 | print(f"Error: No .jsonl files found in {input_folder}.", file=sys.stderr) | CODE |
| MEDIUM | olmocr/bench/benchmark.py | 275 | print("Error: No candidate pipeline folders found (subdirectories besides 'pdfs').", file=sys.stderr) | CODE |
| LOW | olmocr/bench/convert.py | 84 | except Exception as ex: | CODE |
| LOW | olmocr/bench/convert.py | 172 | except Exception as e: | CODE |
| LOW | olmocr/bench/convert.py | 215 | except Exception as e: | CODE |
| LOW | olmocr/bench/convert.py | 226 | except Exception as e: | CODE |
| MEDIUM⚡ | olmocr/bench/review_app.py | 267 | print(f"Error: Dataset not found: {args.dataset_file}") | CODE |
| MEDIUM⚡ | olmocr/bench/review_app.py | 276 | print(f"Error: PDF directory not found: {pdf_dir}") | CODE |
| LOW⚡ | olmocr/bench/review_app.py | 282 | except Exception as e: | CODE |
| MEDIUM⚡ | olmocr/bench/review_app.py | 283 | print(f"Error loading dataset: {str(e)}") | CODE |
| MEDIUM⚡ | olmocr/bench/review_app_latex.py | 689 | print(f"Error: Dataset not found: {args.dataset_file}") | CODE |
| MEDIUM⚡ | olmocr/bench/review_app_latex.py | 697 | print(f"Error: PDF directory not found: {pdf_dir}") | CODE |
| LOW⚡ | olmocr/bench/review_app_latex.py | 702 | except Exception as e: | CODE |
| MEDIUM⚡ | olmocr/bench/review_app_latex.py | 703 | print(f"Error loading dataset: {str(e)}") | CODE |
| MEDIUM⚡ | olmocr/bench/tests.py | 834 | print(f"Error parsing JSON on line {line_number}: {e}") | CODE |
| MEDIUM⚡ | olmocr/bench/tests.py | 837 | print(f"Error on line {line_number}: {e}") | CODE |
| LOW⚡ | olmocr/bench/tests.py | 839 | except Exception as e: | CODE |
| LOW | olmocr/bench/report.py | 267 | except Exception as e: | CODE |
| LOW | olmocr/bench/report.py | 279 | except Exception as e: | CODE |
| LOW | olmocr/bench/katex/render.py | 179 | except Exception: | CODE |
| LOW | olmocr/bench/katex/render.py | 183 | except Exception: | CODE |
| LOW | olmocr/bench/katex/render.py | 224 | except Exception: | CODE |
| MEDIUM | olmocr/bench/katex/render.py | 304 | print(f"Error rendering equation: '{equation}'") | STRING |
| LOW | olmocr/bench/katex/render.py | 431 | except Exception as e: | STRING |
| MEDIUM | olmocr/bench/katex/render.py | 432 | print("Error parsing MathML with BeautifulSoup:", e) | STRING |
| LOW | olmocr/bench/scripts/screen_pdfs.py | 93 | except Exception as e: | CODE |
| 406 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_mine_html_templates_mine_tests.py | 931 | # Create a temporary file to store the rendered PDF | STRING |
| MEDIUM⚡ | tests/test_tests.py | 570 | # Create a table with irregular pipe placement at the end | COMMENT |
| MEDIUM⚡ | tests/test_tests.py | 603 | # Create a very long text before the table | COMMENT |
| MEDIUM | tests/test_grpo.py | 44 | # Create a minimal valid PDF | COMMENT |
| MEDIUM | tests/test_grpo.py | 189 | # Create an empty JSONL file | COMMENT |
| MEDIUM | tests/test_grpo.py | 208 | # Create a JSONL with some malformed entries | COMMENT |
| MEDIUM | tests/test_grpo.py | 275 | # Create a sample JSONL test file with different test types | COMMENT |
| MEDIUM | tests/test_pipeline.py | 172 | # Create a more complex test image | COMMENT |
| MEDIUM | olmocr/metrics.py | 159 | # Create the table header | COMMENT |
| MEDIUM | olmocr/metrics.py | 163 | # Create the table rows | COMMENT |
| MEDIUM | olmocr/pipeline.py | 691 | # Create the output markdown path | COMMENT |
| MEDIUM | olmocr/pipeline.py | 750 | # Define the output S3 path using the work_hash | COMMENT |
| MEDIUM | olmocr/pipeline.py | 775 | # Create the directory structure if it doesn't exist | COMMENT |
| MEDIUM | olmocr/pipeline.py | 1070 | # Create the experiment spec | COMMENT |
| MEDIUM | olmocr/bench/benchmark.py | 79 | # Define an inner function to evaluate a single test | COMMENT |
| MEDIUM | olmocr/bench/review_app_latex.py | 248 | # Create the review_latex.html template with MathJax support | COMMENT |
| MEDIUM | olmocr/bench/review_app_latex.py | 611 | # Create the all_done_latex.html template | STRING |
| MEDIUM | olmocr/bench/katex/render.py | 243 | # Create a new page. | COMMENT |
| MEDIUM | olmocr/bench/scripts/split_table_tests.py | 91 | # Create a new test dict for each relationship | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_reading_order.py | 93 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_reading_order.py | 34 | # Create a thread-safe lock for writing to output files | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_reading_order.py | 343 | # Create a thread-specific temp directory to avoid conflicts | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_reading_order.py | 442 | # Create a summary file | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_reading_order.py | 448 | # Create a ThreadPoolExecutor | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_tables_gpt.py | 89 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_tables_gemini.py | 97 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_tables_gemini.py | 37 | # Create a thread-safe lock for writing to the output file | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_tables_gemini.py | 238 | # Create an image part from the rendered pdf image | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_tables_gemini.py | 308 | # Create a thread-specific temp directory to avoid conflicts | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_tables_gemini.py | 414 | # Create a ThreadPoolExecutor | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_long_tiny_text.py | 57 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM | olmocr/bench/miners/mine_long_tiny_text.py | 214 | # Create a placeholder test to ensure the PDF is represented | COMMENT |
| MEDIUM | olmocr/bench/miners/cleanup_data.py | 56 | # Create a new single-page PDF | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/cleanup_data.py | 93 | # Create a dataset.jsonl with only the tests for pages we're keeping | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_headers_footers.py | 89 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM⚡ | olmocr/bench/miners/mine_old_scans_math.py | 57 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM | olmocr/bench/miners/pick_mediod.py | 141 | # Create the output path, preserving the directory structure relative to input_dir | COMMENT |
| MEDIUM⚡ | olmocr/bench/runners/run_mistral.py | 32 | # Create a new PDF with just the requested page | COMMENT |
| MEDIUM⚡ | olmocr/bench/runners/run_mineru.py | 26 | # Create a new PDF with just the requested page | COMMENT |
| MEDIUM | olmocr/bench/runners/run_marker.py | 17 | # Create a configuration dictionary with the necessary settings | COMMENT |
| MEDIUM⚡ | olmocr/bench/runners/run_marker.py | 47 | # Create a new PDF with just the requested page | COMMENT |
| MEDIUM⚡ | olmocr/synth/mine_html_templates.py | 530 | # Create a temporary PDF file | COMMENT |
| MEDIUM⚡ | olmocr/synth/mine_html_templates.py | 670 | # Create a new PDF with just the selected page | COMMENT |
| MEDIUM | olmocr/synth/mine_html_templates.py | 224 | # Create a temporary soup with just this element to get its HTML | COMMENT |
| MEDIUM | olmocr/synth/mine_html_templates.py | 311 | # Create a new soup with just the body content | COMMENT |
| MEDIUM | olmocr/synth/mine_html_templates.py | 328 | # Create an img tag with placeholder src and appropriate alt text | COMMENT |
| MEDIUM | olmocr/synth/mine_html_templates.py | 1825 | # Create a unique folder for each PDF in the temp directory; include PID and PDF path hash to avoid cross-run collis | COMMENT |
| MEDIUM | olmocr/synth/mine_html_templates.py | 972 | # Create a TableTest with relevant relationships | STRING |
| MEDIUM | olmocr/synth/mine_html_templates.py | 1016 | # Create the actual test object | STRING |
| MEDIUM | olmocr/synth/mine_html_templates.py | 1441 | # Create a format test | STRING |
| MEDIUM | olmocr/synth/mine_html_templates.py | 1694 | # Create a signature for the test by using all fields except 'id' | STRING |
| MEDIUM | olmocr/synth/mine_html_templates.py | 1717 | # Create a baseline test object to check if disallowed characters are present | STRING |
| MEDIUM | olmocr/synth/rotate_html_templates.py | 54 | # Create the same symlink in destination | COMMENT |
| MEDIUM | olmocr/synth/rotate_html_templates.py | 283 | # Create a temporary file for the rotated PDF | COMMENT |
| MEDIUM | olmocr/train/config.py | 330 | # Create main config | COMMENT |
| MEDIUM | olmocr/train/compress_checkpoint.py | 254 | # Define a oneshot data collator for multimodal inputs. | COMMENT |
| MEDIUM | olmocr/train/grpo_train.py | 1429 | # Create a wrapper function that calls olmocr_bench_reward with macro_average=True | COMMENT |
| MEDIUM | olmocr/train/grpo_train.py | 1474 | # Create a wrapper function with proper __name__ attribute | COMMENT |
| MEDIUM | olmocr/train/dataloader.py | 926 | # Create a new PageResponse with the updated text (since it's frozen) | COMMENT |
| MEDIUM | olmocr/train/dataloader.py | 1075 | # Create a new PageResponse with the updated text (since it's frozen) | COMMENT |
| 59 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_olmocrmix.py | 10 | def test_repackage_and_prepare_olmocrmix(): | CODE |
| LOW | tests/test_s3_work_queue.py | 29 | def test_compute_workgroup_hash(self): | CODE |
| LOW | tests/test_s3_work_queue.py | 73 | async def test_populate_queue_new_items(self): | CODE |
| LOW | tests/test_s3_work_queue.py | 94 | async def test_populate_queue_existing_items(self): | CODE |
| LOW | tests/test_mine_html_templates_markdown.py | 17 | def test_title_tag_excluded_from_markdown(self): | CODE |
| LOW | tests/test_mine_html_templates_markdown.py | 44 | def test_image_with_data_description(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 65 | def test_image_without_data_description(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 80 | def test_headers_footers_excluded(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 109 | def test_no_body_tag_fallback(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 124 | def test_removes_triple_dashes_from_content(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 222 | def test_mixed_super_and_subscripts(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 314 | def test_frontmatter_preserved(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 351 | def test_empty_super_subscripts(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 369 | def test_complex_math_expression(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 391 | def test_non_numerical_footnote(self): | STRING |
| LOW | tests/test_mine_html_templates_markdown.py | 406 | def test_line_numbers_markdown(self): | STRING |
| LOW | tests/test_dataloader.py | 25 | def test_markdown_table_detection(self): | CODE |
| LOW | tests/test_dataloader.py | 99 | def test_malformed_html_table_unclosed_tags(self): | STRING |
| LOW | tests/test_dataloader.py | 128 | def test_malformed_html_table_missing_closing(self): | STRING |
| LOW | tests/test_dataloader.py | 197 | def test_complex_markdown_table_variations(self): | CODE |
| LOW | tests/test_dataloader.py | 247 | def test_br_tags_outside_tables(self): | STRING |
| LOW | tests/test_dataloader.py | 277 | def test_simple_bold_conversion(self): | CODE |
| LOW | tests/test_dataloader.py | 293 | def test_simple_italic_conversion(self): | CODE |
| LOW | tests/test_dataloader.py | 309 | def test_bold_and_italic_conversion(self): | CODE |
| LOW | tests/test_dataloader.py | 325 | def test_latex_in_math_equation_preserved(self): | CODE |
| LOW | tests/test_dataloader.py | 373 | def test_multiple_occurrences(self): | CODE |
| LOW | tests/test_dataloader.py | 389 | def test_latex_in_parenthesis_delimiter(self): | CODE |
| LOW | tests/test_dataloader.py | 405 | def test_latex_in_bracket_delimiter(self): | CODE |
| LOW | tests/test_dataloader.py | 460 | def test_complex_latex_with_parenthesis_delimiters(self): | CODE |
| LOW | tests/test_dataloader.py | 536 | def test_rotation_correction_needed(self): | CODE |
| LOW | tests/test_dataloader.py | 567 | def test_dollar_to_parentheses(self): | CODE |
| LOW | tests/test_dataloader.py | 584 | def test_double_dollar_to_brackets(self): | CODE |
| LOW | tests/test_dataloader.py | 601 | def test_mixed_latex_delimiters(self): | CODE |
| LOW | tests/test_dataloader.py | 667 | def test_parse_yaml_front_matter(self, mock_read_text): | CODE |
| LOW | tests/test_dataloader.py | 715 | def test_preserve_existing_markdown_content(self, mock_read_text): | STRING |
| LOW | tests/test_mine_html_templates_mine_tests.py | 1044 | def test_complex_markdown_formatting(self): | CODE |
| LOW | tests/test_mine_html_templates_mine_tests.py | 1104 | def test_complex_formatting_two(self): | STRING |
| LOW | tests/test_mine_html_templates_footnotes.py | 46 | def test_single_marker_generates_marker_only(self): | CODE |
| LOW | tests/test_mine_html_templates_footnotes.py | 71 | def test_marker_with_definition_includes_text(self): | STRING |
| LOW | tests/test_mine_html_templates_footnotes.py | 99 | def test_multiple_markers_mixed_occurrences(self): | STRING |
| LOW | tests/test_mine_html_templates_footnotes.py | 211 | def test_footnotes_in_footer_ptags(self): | STRING |
| LOW | tests/test_mine_html_templates_footnotes.py | 314 | def test_footnotes_footer_again(self): | CODE |
| LOW⚡ | tests/test_tests.py | 136 | def test_present_text_exact_match(self): | CODE |
| LOW⚡ | tests/test_tests.py | 142 | def test_present_text_not_found(self): | CODE |
| LOW⚡ | tests/test_tests.py | 149 | def test_present_text_with_max_diffs(self): | CODE |
| LOW⚡ | tests/test_tests.py | 182 | def test_absent_text_not_found(self): | CODE |
| LOW⚡ | tests/test_tests.py | 188 | def test_case_insensitive_present(self): | CODE |
| LOW⚡ | tests/test_tests.py | 194 | def test_case_insensitive_absent(self): | CODE |
| LOW⚡ | tests/test_tests.py | 526 | def test_with_stripped_content(self): | CODE |
| LOW⚡ | tests/test_tests.py | 534 | def test_table_at_end_of_file(self): | CODE |
| LOW⚡ | tests/test_tests.py | 542 | def test_table_at_end_with_no_trailing_newline(self): | CODE |
| LOW⚡ | tests/test_tests.py | 550 | def test_table_at_end_with_extra_spaces(self): | CODE |
| LOW⚡ | tests/test_tests.py | 559 | def test_table_at_end_with_mixed_whitespace(self): | CODE |
| LOW⚡ | tests/test_tests.py | 567 | def test_malformed_table_at_end(self): | CODE |
| LOW⚡ | tests/test_tests.py | 580 | def test_incomplete_table_at_end(self): | CODE |
| LOW⚡ | tests/test_tests.py | 592 | def test_table_with_excessive_blank_lines_at_end(self): | CODE |
| LOW⚡ | tests/test_tests.py | 600 | def test_table_at_end_after_long_text(self): | CODE |
| LOW⚡ | tests/test_tests.py | 609 | def test_valid_table_at_eof_without_newline(self): | CODE |
| LOW⚡ | tests/test_tests.py | 2077 | def test_empty_appears_before_validation(self): | CODE |
| LOW⚡ | tests/test_tests.py | 2082 | def test_empty_appears_after_validation(self): | CODE |
| 252 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_mine_html_templates_markdown.py | 288 | # Check if tags are preserved in table cells | STRING |
| LOW | tests/test_mine_html_templates_mine_tests.py | 938 | # Check if rendering was successful | STRING |
| LOW | olmocr/s3_utils.py | 76 | # Check if it's actually a folder with contents | COMMENT |
| LOW | olmocr/pipeline.py | 1347 | # Check if this is a tar.gz file (local) | COMMENT |
| LOW | olmocr/pipeline.py | 1410 | # Add tarballs to the queue - each tarball is one work item | COMMENT |
| LOW | olmocr/bench/tests.py | 319 | # Check if the text appears in any of the formatted texts using fuzzy matching | COMMENT |
| LOW | olmocr/bench/tests.py | 715 | # Check if appears_before_marker is at the end of this text (using fuzzy matching) | COMMENT |
| LOW | olmocr/bench/tests.py | 741 | # Check if appears_after_marker is at the beginning of this text (using fuzzy matching) | COMMENT |
| LOW | olmocr/bench/table_parsing.py | 284 | # Check if this line has pipe characters (a table row indicator) | COMMENT |
| LOW | olmocr/bench/table_parsing.py | 359 | # Check if this looks like a separator row (contains mostly dashes) | COMMENT |
| LOW | olmocr/bench/scripts/convert_all.sh | 121 | # Check if environment exists | COMMENT |
| LOW | olmocr/bench/scripts/convert_all.sh | 149 | # Check if the server process is running | COMMENT |
| LOW | olmocr/bench/scripts/rotate_pdfs_random.sh | 6 | # Check if directory exists | COMMENT |
| LOW | olmocr/bench/scripts/rotate_pdfs_random.sh | 12 | # Check if qpdf is installed (preferred for PDF rotation) | COMMENT |
| LOW | olmocr/bench/scripts/rotate_pdfs_random.sh | 28 | # Check if any PDF files exist | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_reading_order.py | 88 | # Check if page number is valid | COMMENT |
| LOW | olmocr/bench/miners/mine_reading_order.py | 470 | # Check if we've reached the maximum number of documents | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_tables_gpt.py | 84 | # Check if page number is valid | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_blank_pages_gpt.py | 119 | # Check if natural_text is null | COMMENT |
| LOW | olmocr/bench/miners/mine_blank_pages_gpt.py | 172 | # Check if the page has null natural_text | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_tables_gemini.py | 92 | # Check if page number is valid | COMMENT |
| LOW | olmocr/bench/miners/mine_tables_gemini.py | 432 | # Check if we've reached the maximum number of tests | COMMENT |
| LOW | olmocr/bench/miners/mine_multilingual_gpt.py | 169 | # Check if the page contains a table | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_long_tiny_text.py | 52 | # Check if page number is valid | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_headers_footers.py | 84 | # Check if page number is valid | COMMENT |
| LOW | olmocr/bench/miners/delete_rejected.py | 38 | # Check if the test is marked as rejected | COMMENT |
| LOW | olmocr/bench/miners/delete_rejected.py | 95 | # Check if all tests for this PDF are in the rejected list | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_footnotes_gpt.py | 125 | # Check if there are any footnotes in the list | COMMENT |
| LOW | olmocr/bench/miners/mine_footnotes_gpt.py | 179 | # Check if the page contains footnotes | COMMENT |
| LOW⚡ | olmocr/bench/miners/mine_old_scans_math.py | 52 | # Check if page number is valid | COMMENT |
| LOW | olmocr/bench/miners/mine_tables_gpt_simple.py | 200 | # Check if the page contains a table | COMMENT |
| LOW⚡ | olmocr/bench/runners/run_mistral.py | 28 | # Check if the requested page exists | COMMENT |
| LOW⚡ | olmocr/bench/runners/run_mistral.py | 39 | temp_file.close() # Close the file but keep the name | CODE |
| LOW⚡ | olmocr/bench/runners/run_mineru.py | 22 | # Check if the requested page exists | COMMENT |
| LOW⚡ | olmocr/bench/runners/run_mineru.py | 33 | temp_file.close() # Close the file but keep the name | CODE |
| LOW⚡ | olmocr/bench/runners/run_marker.py | 43 | # Check if the requested page exists | COMMENT |
| LOW⚡ | olmocr/bench/runners/run_marker.py | 54 | temp_file.close() # Close the file but keep the name | CODE |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 665 | # Check if page number is valid | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 157 | # Check if it's a local path | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 461 | # Check if response was complete | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 508 | # Check if response was complete | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 603 | # Check if refinement response was complete | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 1866 | # Check if this PDF has already been processed (skip logic for resume) | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 885 | # Check if this element has no children from our target tags | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1194 | # Check if it's a number | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1198 | # Check if it's a word (has at least one alphabetic character) | STRING |
| LOW | olmocr/synth/rotate_html_templates.py | 50 | # Check if it's a symlink | COMMENT |
| LOW | olmocr/synth/rotate_html_templates.py | 117 | # Check if file starts with FrontMatter | COMMENT |
| LOW | olmocr/train/train.py | 119 | # Check if we have any valid samples | COMMENT |
| LOW | olmocr/train/train.py | 654 | # Check if we should do a gradient update | COMMENT |
| LOW | olmocr/train/train.py | 712 | # Check if we've reached our training limit | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 439 | # Check if line looks like a table row | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 441 | # Check if next line is a separator (for header rows) | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 446 | # Check if previous line is a separator (for data rows) | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 662 | # Check if there was an error | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 683 | # Check if there are any tables in the text | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 698 | # Check if cell content contains <br> tags (any variation) | COMMENT |
| LOW⚡ | olmocr/train/dataloader.py | 713 | # Check if there are any <table> tags at all | COMMENT |
| LOW | olmocr/train/dataloader.py | 346 | # Check if image exists | COMMENT |
| LOW | olmocr/train/dataloader.py | 350 | # Check if page_data exists (we need to update it) | COMMENT |
| 84 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/test_mine_html_templates_markdown.py | 444 | <div class="line-text">unknown and undefined, so the present study aimed to investigate characteristics of the DHLA-</di | CODE |
| HIGH | olmocr/pipeline.py | 181 | r"---\nprimary_language: (?:[a-z]{2}|null)\nis_rotation_valid: (?:True|False|true|false)\nrotation_corre | CODE |
| HIGH | olmocr/bench/review_app_latex.py | 442 | let pdfDoc = null; | CODE |
| HIGH | olmocr/bench/review_app_latex.py | 444 | let canvas = null; | CODE |
| HIGH | olmocr/bench/katex/render.py | 290 | return null; | CODE |
| HIGH | olmocr/bench/katex/render.py | 329 | list.push({ | CODE |
| HIGH | olmocr/bench/miners/mine_blank_pages_gpt.py | 79 | Optional[bool]: True if natural_text is null, False otherwise, None if detection fails | STRING |
| HIGH | olmocr/bench/miners/mine_blank_pages_gpt.py | 79 | Optional[bool]: True if natural_text is null, False otherwise, None if detection fails | STRING |
| HIGH⚡ | olmocr/bench/miners/mine_blank_pages_gpt.py | 119 | # Check if natural_text is null | COMMENT |
| HIGH | olmocr/synth/cutoff_detection.py | 61 | if (child.nodeType === Node.TEXT_NODE && child.textContent.trim()) { | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 68 | leaves.push(el); | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 83 | if (rect.width <= 0 || rect.height <= 0) continue; | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 109 | let clippingAncestorTag = null; | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 123 | if (clipsX || clipsY) { | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 142 | (visLeft !== prevLeft || visTop !== prevTop || | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 143 | visRight !== prevRight || visBottom !== prevBottom)) { | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 166 | results.push({ | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 200 | if (child.nodeType === Node.TEXT_NODE && child.textContent.trim()) { | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 205 | if (hasDirectText) leaves.push(el); | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 221 | if (!bgColor || bgColor === 'rgba(0, 0, 0, 0)' || bgColor === 'transparent') return 0; | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 231 | if (rect.width <= 0 || rect.height <= 0) continue; | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 238 | if (!directText || directText.length < 3) continue; | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 246 | let blockerTag = null; | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 252 | if (px < 0 || py < 0) continue; | CODE |
| HIGH⚡ | olmocr/synth/cutoff_detection.py | 256 | if (topEl === el || isRelated(topEl, el)) continue; | CODE |
| HIGH | olmocr/synth/cutoff_detection.py | 269 | results.push({ | CODE |
| HIGH⚡ | scripts/scan_dolmadocs.py | 1160 | const primaryOption = activePrimaryButton ? activePrimaryButton.getAttribute('data-value') : null; | CODE |
| HIGH⚡ | scripts/scan_dolmadocs.py | 1165 | publicPiiOptions.push(checkbox.getAttribute('data-value')); | CODE |
| HIGH⚡ | scripts/scan_dolmadocs.py | 1171 | privatePiiOptions.push(checkbox.getAttribute('data-value')); | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1058 | status.onclick = null; | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1343 | if (data.publicPiiOptions && data.publicPiiOptions.length > 0) { | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1357 | if (data.privatePiiOptions && data.privatePiiOptions.length > 0) { | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1528 | if (event.target && event.target.id === 'start-button') { | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1539 | if (event.target && event.target.id === 'view-instructions-button') { | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_s3_work_queue.py | 224 | CODE | |
| LOW | tests/test_integration.py | 11 | CODE | |
| LOW | olmocr/s3_utils.py | 36 | CODE | |
| LOW | olmocr/s3_utils.py | 199 | CODE | |
| LOW | olmocr/s3_utils.py | 250 | CODE | |
| LOW | olmocr/s3_utils.py | 378 | CODE | |
| LOW | olmocr/pipeline.py | 106 | CODE | |
| LOW | olmocr/pipeline.py | 281 | CODE | |
| LOW | olmocr/pipeline.py | 382 | CODE | |
| LOW | olmocr/pipeline.py | 485 | CODE | |
| LOW | olmocr/pipeline.py | 698 | CODE | |
| LOW | olmocr/pipeline.py | 956 | CODE | |
| LOW | olmocr/pipeline.py | 1199 | CODE | |
| LOW | olmocr/bench/benchmark.py | 162 | CODE | |
| LOW | olmocr/bench/convert.py | 19 | CODE | |
| LOW | olmocr/bench/convert.py | 103 | CODE | |
| LOW | olmocr/bench/review_app.py | 38 | CODE | |
| LOW | olmocr/bench/review_app.py | 219 | CODE | |
| LOW | olmocr/bench/review_app_latex.py | 36 | CODE | |
| LOW | olmocr/bench/review_app_latex.py | 217 | CODE | |
| LOW | olmocr/bench/utils.py | 6 | CODE | |
| LOW | olmocr/bench/tests.py | 766 | CODE | |
| LOW | olmocr/bench/tests.py | 253 | CODE | |
| LOW | olmocr/bench/tests.py | 376 | CODE | |
| LOW | olmocr/bench/tests.py | 655 | CODE | |
| LOW | olmocr/bench/table_parsing.py | 83 | CODE | |
| LOW | olmocr/bench/table_parsing.py | 263 | CODE | |
| LOW | olmocr/bench/table_parsing.py | 390 | CODE | |
| LOW | olmocr/bench/report.py | 12 | CODE | |
| LOW | olmocr/bench/report.py | 42 | CODE | |
| LOW | olmocr/bench/katex/render.py | 414 | CODE | |
| LOW | olmocr/bench/katex/render.py | 475 | CODE | |
| LOW | olmocr/bench/katex/render.py | 517 | CODE | |
| LOW | olmocr/bench/scripts/workspace_to_bench.py | 25 | CODE | |
| LOW | olmocr/bench/scripts/difference_viewer.py | 15 | CODE | |
| LOW | olmocr/bench/scripts/difference_viewer.py | 64 | CODE | |
| LOW | olmocr/bench/scripts/url_matcher.py | 10 | CODE | |
| LOW | olmocr/bench/scripts/url_matcher.py | 29 | CODE | |
| LOW | olmocr/bench/scripts/rotate_pdfs.py | 10 | CODE | |
| LOW | olmocr/bench/miners/mine_reading_order.py | 330 | CODE | |
| LOW | olmocr/bench/miners/mine_reading_order.py | 426 | CODE | |
| LOW | olmocr/bench/miners/mine_reading_order.py | 483 | CODE | |
| LOW | olmocr/bench/miners/mine_tables_gpt.py | 189 | CODE | |
| LOW | olmocr/bench/miners/mine_diffs.py | 90 | CODE | |
| LOW | olmocr/bench/miners/mine_multi_column.py | 181 | CODE | |
| LOW | olmocr/bench/miners/mine_multi_column.py | 330 | CODE | |
| LOW | olmocr/bench/miners/mine_blank_pages_gpt.py | 204 | CODE | |
| LOW | olmocr/bench/miners/mine_tables_gemini.py | 205 | CODE | |
| LOW | olmocr/bench/miners/mine_tables_gemini.py | 398 | CODE | |
| LOW | olmocr/bench/miners/mine_tables_gemini.py | 445 | CODE | |
| LOW | olmocr/bench/miners/mine_old_scan_pdf.py | 9 | CODE | |
| LOW | olmocr/bench/miners/mine_multilingual_gpt.py | 201 | CODE | |
| LOW | olmocr/bench/miners/mine_long_tiny_text.py | 151 | CODE | |
| LOW | olmocr/bench/miners/cleanup_data.py | 10 | CODE | |
| LOW | olmocr/bench/miners/cleanup_data.py | 89 | CODE | |
| LOW | olmocr/bench/miners/delete_rejected.py | 10 | CODE | |
| LOW | olmocr/bench/miners/delete_rejected.py | 53 | CODE | |
| LOW | olmocr/bench/miners/mine_math.py | 84 | CODE | |
| LOW | olmocr/bench/miners/check_old_scans_math.py | 86 | CODE | |
| LOW | olmocr/bench/miners/mine_footnotes_gpt.py | 211 | CODE | |
| 118 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/test_tests.py | 2214 | except Exception as e: | CODE |
| HIGH | olmocr/pipeline.py | 1050 | except BeakerSecretNotFound: | CODE |
| HIGH | olmocr/pipeline.py | 475 | except: | CODE |
| HIGH | olmocr/bench/katex/render.py | 179 | except Exception: | CODE |
| HIGH | olmocr/bench/katex/render.py | 183 | except Exception: | CODE |
| HIGH | olmocr/bench/katex/render.py | 224 | except Exception: | CODE |
| HIGH | olmocr/bench/miners/mine_reading_order.py | 314 | except (ValueError, TypeError): | CODE |
| HIGH | olmocr/synth/mine_html_templates.py | 1735 | except Exception: | CODE |
| HIGH | olmocr/synth/mine_html_templates.py | 1166 | except Exception: | CODE |
| HIGH | olmocr/synth/mine_html_templates.py | 1286 | except Exception: | CODE |
| HIGH | olmocr/synth/mine_html_templates.py | 1593 | except Exception: | CODE |
| HIGH | olmocr/synth/mine_html_templates.py | 1470 | except Exception: | CODE |
| HIGH | olmocr/filter/filter.py | 202 | except StopIteration: | CODE |
| HIGH | olmocr/data/prepare_national_archive_transcripts.py | 403 | except: | CODE |
| HIGH | olmocr/data/prepare_loc_transcripts.py | 282 | except: | CODE |
| HIGH | olmocr/data/prepare_olmocrmix.py | 84 | except (TypeError, ValueError): | CODE |
| HIGH | olmocr/data/prepare_olmocrmix.py | 122 | except json.JSONDecodeError: | CODE |
| HIGH | olmocr/data/repackage_olmocrmix.py | 126 | except (TypeError, ValueError): | CODE |
| HIGH | olmocr/data/runopenaibatch.py | 74 | except (TypeError, ValueError): | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1675 | except (ValueError, TypeError): | CODE |
| HIGH | scripts/scan_dolmadocs.py | 1608 | except (ValueError, TypeError): | CODE |
| HIGH | scripts/pii/rich_tagging_pipeline.py | 785 | except BeakerSecretNotFound: | CODE |
| HIGH | scripts/pii/rich_tagging_pipeline.py | 646 | except asyncio.CancelledError: | CODE |
| HIGH | scripts/pii/rich_tagging_pipeline.py | 306 | except: | CODE |
| HIGH | scripts/pii/pii_rule_comparison.py | 1324 | except (ValueError, TypeError): | CODE |
| HIGH | scripts/pii/tagging_pipeline.py | 617 | except BeakerSecretNotFound: | CODE |
| HIGH | scripts/pii/tagging_pipeline.py | 478 | except asyncio.CancelledError: | CODE |
| HIGH | scripts/pii/tagging_pipeline.py | 232 | except: | CODE |
| HIGH | scripts/eval/runeval.py | 95 | except json.JSONDecodeError: | CODE |
| HIGH | scripts/data/runopenaibatch.py | 74 | except (TypeError, ValueError): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | olmocr/bench/miners/mine_reading_order.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gpt.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_blank_pages_gpt.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gemini.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_multilingual_gpt.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_headers_footers.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_footnotes_gpt.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_length_gpt_simple.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gpt_simple.py | 0 | download a pdf file from s3. args: s3_path: the s3 path (s3://bucket/path/to/file.pdf) local_path: the local path to sav | STRING |
| HIGH | olmocr/bench/miners/mine_reading_order.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gpt.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gemini.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_long_tiny_text.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_headers_footers.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_old_scans_math.py | 0 | extract a specific page from a pdf and save it as a new pdf. args: input_path: path to the input pdf output_path: path t | STRING |
| HIGH | olmocr/bench/miners/mine_blank_pages_gpt.py | 0 | process a single pdf from s3. args: s3_path: s3 path to the pdf temp_dir: directory for temporary files output_dir: dire | STRING |
| HIGH | olmocr/bench/miners/mine_multilingual_gpt.py | 0 | process a single pdf from s3. args: s3_path: s3 path to the pdf temp_dir: directory for temporary files output_dir: dire | STRING |
| HIGH | olmocr/bench/miners/mine_tables_gpt_simple.py | 0 | process a single pdf from s3. args: s3_path: s3 path to the pdf temp_dir: directory for temporary files output_dir: dire | STRING |
| HIGH | olmocr/bench/miners/mine_footnotes_gpt.py | 0 | process a single pdf from s3. args: s3_path: s3 path to the pdf temp_dir: directory for temporary files output_dir: dire | STRING |
| HIGH | olmocr/bench/miners/mine_length_gpt_simple.py | 0 | process a single pdf from s3. args: s3_path: s3 path to the pdf temp_dir: directory for temporary files output_dir: dire | STRING |
| HIGH | olmocr/bench/runners/run_transformers.py | 0 | convert page of a pdf file to markdown by calling a request running against an openai compatible server. you can use thi | STRING |
| HIGH | olmocr/bench/runners/run_nanonetsocr_2.py | 0 | convert page of a pdf file to markdown by calling a request running against an openai compatible server. you can use thi | STRING |
| HIGH | olmocr/bench/runners/run_server.py | 0 | convert page of a pdf file to markdown by calling a request running against an openai compatible server. you can use thi | STRING |
| HIGH | scripts/pii/rich_tagging_pipeline.py | 0 | you are a document analyzer that identifies personally identifiable information (pii) in documents. your task is to anal | STRING |
| HIGH | scripts/pii/autoscan_dolmadocs.py | 0 | you are a document analyzer that identifies personally identifiable information (pii) in documents. your task is to anal | STRING |
| HIGH | scripts/pii/chatgpt_tag_dolmadocs_v1.py | 0 | you are a document analyzer that identifies personally identifiable information (pii) in documents. your task is to anal | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_mine_html_templates_markdown.py | 1 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_markdown.py | 8 | CODE | |
| LOW | tests/test_dataloader.py | 6 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_mine_tests.py | 8 | CODE | |
| LOW | tests/test_filter.py | 4 | CODE | |
| LOW | tests/test_cutoff_detection.py | 6 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 1 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_metadata.py | 8 | CODE | |
| LOW | tests/test_grpo.py | 13 | CODE | |
| LOW | tests/test_anchor.py | 16 | CODE | |
| LOW | tests/test_anchor.py | 17 | CODE | |
| LOW | tests/test_anchor.py | 69 | CODE | |
| LOW | tests/test_anchor.py | 70 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_formatting.py | 8 | CODE | |
| LOW | tests/test_table_parsing.py | 3 | CODE | |
| LOW | tests/test_mine_html_templates_incomplete_content.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_incomplete_content.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_incomplete_content.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_incomplete_content.py | 9 | CODE | |
| LOW | tests/test_mine_html_templates_introduce_text_errors.py | 4 | CODE | |
| LOW | tests/test_mine_html_templates_math_extraction.py | 5 | CODE | |
| LOW | tests/test_mine_html_templates_math_extraction.py | 7 | CODE | |
| LOW | tests/test_mine_html_templates_math_extraction.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_math_extraction.py | 8 | CODE | |
| LOW | tests/test_mine_html_templates_math_extraction.py | 8 | CODE | |
| LOW | olmocr/__init__.py | 1 | CODE | |
| LOW | olmocr/__init__.py | 1 | CODE | |
| LOW | olmocr/bench/katex/__init__.py | 1 | CODE | |
| LOW | olmocr/bench/katex/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/prompts/__init__.py | 1 | CODE | |
| LOW | olmocr/filter/__init__.py | 1 | CODE | |
| LOW | olmocr/data/repackage_olmocrmix.py | 12 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_olmocrmix.py | 17 | # Step 1: Repackage the sample dataset into parquet + tarballs | COMMENT |
| LOW | tests/test_olmocrmix.py | 44 | # Step 2: Repackage the sample dataset into parquet + tarballs | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 35 | # Step 1: Get all local JSONL files | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 39 | # Step 2: Extract URLs from local files | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 58 | # Step 3: Load Hugging Face dataset | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 64 | # Step 4: Extract URLs from Hugging Face dataset | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 79 | # Step 5: Find matches | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 82 | # Step 6: Group matches by local file with metadata | COMMENT |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 529 | # Step 3: Render the initial HTML to PDF and then back to PNG for comparison | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 390 | # Step 0: Check that the orientation of the original document is right-side-up. If not, we will | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 436 | # Step 1: Initial analysis and column detection | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 476 | # Step 2: Initial HTML generation with detailed layout instructions | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 562 | # Step 4: Refinement - Show both images to Claude and ask for corrections | COMMENT |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 861 | # Step 1: Process headers, footers, and page numbers for TextAbsenceTests | STRING |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 1476 | # Step 6: Generate FootnoteTests for footnotes on the page | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 943 | # Step 2: Generate tests from tables using parse_html_tables | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1054 | # Step 3: Generate TextPresenceTests and OrderingTests from markdown content | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1368 | # Step 4: Generate Math tests for LaTeX equations from the markdown | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1409 | # Step 5: Generate FormatTests for headings, bold, and italic text | STRING |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 234 | # Step 1: Copy entire directory structure | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 241 | # Step 2: Find all PDFs in the destination bench_data/pdfs directory | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 262 | # Step 3: Select PDFs to rotate based on percentage | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 271 | # Step 4: Rotate selected PDFs and update corresponding markdown files | COMMENT |
| LOW | olmocr/data/prepare_olmocrmix.py | 148 | # Step 1: Download dataset using hugging face hub snapshot_download to destination/hugging_face folder | COMMENT |
| LOW | olmocr/data/prepare_olmocrmix.py | 197 | # Step 2: Create destination folder structure for processed markdown files | COMMENT |
| LOW⚡ | olmocr/data/prepare_olmocrmix.py | 216 | # Step 3: Extract PDF tarballs | COMMENT |
| LOW⚡ | olmocr/data/prepare_olmocrmix.py | 265 | # Step 4: Process parquet files | COMMENT |
| LOW⚡ | scripts/check_contamination.py | 197 | # Step 1: Get all URLs from bench data | COMMENT |
| LOW⚡ | scripts/check_contamination.py | 202 | # Step 2: Read metadata JSONL and process source URLs | COMMENT |
| LOW | scripts/check_contamination.py | 216 | # Step 3: Map URLs to hashes and query database | COMMENT |
| LOW | scripts/check_contamination.py | 285 | # Step 4: Check for contamination | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_olmocrmix.py | 17 | # Step 1: Repackage the sample dataset into parquet + tarballs | COMMENT |
| LOW | tests/test_olmocrmix.py | 44 | # Step 2: Repackage the sample dataset into parquet + tarballs | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 35 | # Step 1: Get all local JSONL files | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 39 | # Step 2: Extract URLs from local files | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 58 | # Step 3: Load Hugging Face dataset | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 64 | # Step 4: Extract URLs from Hugging Face dataset | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 79 | # Step 5: Find matches | COMMENT |
| LOW⚡ | olmocr/bench/scripts/url_matcher.py | 82 | # Step 6: Group matches by local file with metadata | COMMENT |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 529 | # Step 3: Render the initial HTML to PDF and then back to PNG for comparison | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 390 | # Step 0: Check that the orientation of the original document is right-side-up. If not, we will | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 436 | # Step 1: Initial analysis and column detection | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 476 | # Step 2: Initial HTML generation with detailed layout instructions | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 562 | # Step 4: Refinement - Show both images to Claude and ask for corrections | COMMENT |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 861 | # Step 1: Process headers, footers, and page numbers for TextAbsenceTests | STRING |
| LOW⚡ | olmocr/synth/mine_html_templates.py | 1476 | # Step 6: Generate FootnoteTests for footnotes on the page | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 943 | # Step 2: Generate tests from tables using parse_html_tables | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1054 | # Step 3: Generate TextPresenceTests and OrderingTests from markdown content | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1368 | # Step 4: Generate Math tests for LaTeX equations from the markdown | STRING |
| LOW | olmocr/synth/mine_html_templates.py | 1409 | # Step 5: Generate FormatTests for headings, bold, and italic text | STRING |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 234 | # Step 1: Copy entire directory structure | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 241 | # Step 2: Find all PDFs in the destination bench_data/pdfs directory | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 262 | # Step 3: Select PDFs to rotate based on percentage | COMMENT |
| LOW⚡ | olmocr/synth/rotate_html_templates.py | 271 | # Step 4: Rotate selected PDFs and update corresponding markdown files | COMMENT |
| LOW | olmocr/data/prepare_olmocrmix.py | 148 | # Step 1: Download dataset using hugging face hub snapshot_download to destination/hugging_face folder | COMMENT |
| LOW | olmocr/data/prepare_olmocrmix.py | 197 | # Step 2: Create destination folder structure for processed markdown files | COMMENT |
| LOW⚡ | olmocr/data/prepare_olmocrmix.py | 216 | # Step 3: Extract PDF tarballs | COMMENT |
| LOW⚡ | olmocr/data/prepare_olmocrmix.py | 265 | # Step 4: Process parquet files | COMMENT |
| LOW⚡ | scripts/check_contamination.py | 197 | # Step 1: Get all URLs from bench data | COMMENT |
| LOW⚡ | scripts/check_contamination.py | 202 | # Step 2: Read metadata JSONL and process source URLs | COMMENT |
| LOW | scripts/check_contamination.py | 216 | # Step 3: Map URLs to hashes and query database | COMMENT |
| LOW | scripts/check_contamination.py | 285 | # Step 4: Check for contamination | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | olmocr/bench/scripts/convert_all.sh | 221 | # Run raw server benchmarks with generic server function | COMMENT |
| LOW | olmocr/bench/scripts/convert_all.sh | 241 | COMMENT | |
| LOW | olmocr/synth/mine_html_templates.py | 1281 | # Run the test against the markdown content | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 1301 | COMMENT | |
| LOW | olmocr/synth/mine_html_templates.py | 1321 | # # Calculate similarity scores for each absent word | COMMENT |
| LOW | olmocr/synth/mine_html_templates.py | 1341 | COMMENT | |
| LOW | …igs/v0.5.0/qwen3_vl_olmocrv5_1epoch_mix_1025_mini.yaml | 61 | # - name: processed_00_documents_eval_s2pdf | COMMENT |
| LOW | olmocr/data/prepare_national_archive_transcripts.py | 1 | # This script prepares transcriptions from the National Archives into a format usable by olmOCR | COMMENT |
| LOW | olmocr/data/prepare_national_archive_transcripts.py | 21 | # "objectFileSize": 9496446, | COMMENT |
| LOW | olmocr/data/prepare_national_archive_transcripts.py | 41 | # "userId": "dff3eed0-38e5-35fc-b7e7-d2d58b023262", | COMMENT |
| LOW | olmocr/data/prepare_national_archive_transcripts.py | 61 | # } | COMMENT |
| LOW | scripts/remove_paths_from_olmocrmix.py | 1 | # This code takes in a path to an olmocr-mix processed data folder | COMMENT |
| LOW | scripts/run_transformers_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_server_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/check_contamination.py | 1 | #!/usr/bin/env python3 | COMMENT |
| LOW | scripts/run_dots_mocr_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_api_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_mineru_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_qianfan_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_infrapartner_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | scripts/run_chandra_benchmark.sh | 1 | #!/bin/bash | COMMENT |
| LOW | .github/workflows/main.yml | 301 | # with: | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | olmocr/image_utils.py | 7 | Convert one or multiple image files to PDF bytes. Args: image_files: A single image file path (str) or | STRING |
| HIGH | olmocr/bench/tests.py | 767 | Load a single test from a JSON line string or JSON object. Args: data: Either a JSON string to parse o | STRING |
| HIGH | olmocr/data/renderpdf.py | 74 | Returns the (width, height) of a PNG image given its base64-encoded data, without base64-decoding the entire da | STRING |
| HIGH | scripts/pii/pii_rule_comparison.py | 1411 | Generate HTML section with CDF plots for all numeric attributes. Args: numeric_attributes: Dictionary | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | olmocr/check.py | 6 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/s3_utils.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/pipeline.py | 55 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/work_queue.py | 23 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/bench/runners/run_gemini.py | 24 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/bench/runners/run_chatgpt.py | 28 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/train/front_matter.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/train/train.py | 37 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/train/grpo_train.py | 46 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/train/dataloader.py | 39 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/train/compare_vllm_checkpoint.py | 28 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/filter/filter.py | 10 | logger = logging.getLogger(__name__) | CODE |
| LOW | olmocr/data/prepare_workspace.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | scripts/pii/rich_tagging_pipeline.py | 42 | logger = logging.getLogger(__name__) | CODE |
| LOW | scripts/pii/pii_rule_comparison.py | 59 | logger = logging.getLogger(__name__) | CODE |
| LOW | scripts/pii/chatgpt_tag_dolmadocs_v1.py | 32 | logger = logging.getLogger(__name__) | CODE |
| LOW | scripts/pii/chatgpt_tag_dolmadocs_v2.py | 32 | logger = logging.getLogger(__name__) | CODE |
| LOW | scripts/pii/tagging_pipeline.py | 42 | logger = logging.getLogger(__name__) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | olmocr/pipeline.py | 470 | # But just make sure to close the socket on your way out | COMMENT |
| MEDIUM | olmocr/data/buildsilver.py | 39 | # Ask for logprobs, it's not any more expensive and you can use them later to help identify problematic responses | COMMENT |
| LOW | scripts/pii/rich_tagging_pipeline.py | 301 | # But just make sure to close the socket on your way out | STRING |
| LOW | scripts/pii/tagging_pipeline.py | 227 | # But just make sure to close the socket on your way out | COMMENT |
| MEDIUM | scripts/data/buildsilver.py | 39 | # Ask for logprobs, it's not any more expensive and you can use them later to help identify problematic responses | COMMENT |
| MEDIUM | .github/workflows/main.yml | 298 | # jakep: (you can try by setting load: true above, but you'll need a larger runner) | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | olmocr/bench/runners/run_olmocr_pipeline.py | 100 | # Example usage | COMMENT |
| LOW | scripts/run_dots_mocr_benchmark.sh | 5 | # Usage: | COMMENT |
| LOW⚡ | scripts/jsonl_to_markdown.py | 40 | # Example usage | COMMENT |
| LOW | scripts/run_qianfan_benchmark.sh | 8 | # Usage: | COMMENT |
| LOW | scripts/run_chandra_benchmark.sh | 4 | # Usage: | COMMENT |
| LOW | scripts/eval/scoreelo.py | 288 | # Example usage | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | olmocr/data/buildsilver.py | 37 | # Also, structured outputs let you cheat, because the order in which fields are in your schema, is the order in whic | COMMENT |
| MEDIUM | scripts/data/buildsilver.py | 37 | # Also, structured outputs let you cheat, because the order in which fields are in your schema, is the order in whic | COMMENT |
| MEDIUM | scripts/plots/plot_olmocr2_timeline.py | 210 | # "descriptor": "Fixing bug with blank page hallucinations", | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | scripts/pii/rich_tagging_pipeline.py | 511 | # ------------------------------------------------------------------ | STRING |
| MEDIUM | scripts/pii/rich_tagging_pipeline.py | 513 | # ------------------------------------------------------------------ | STRING |
| MEDIUM | scripts/pii/tagging_pipeline.py | 346 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM | scripts/pii/tagging_pipeline.py | 348 | # ------------------------------------------------------------------ | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | olmocr/data/prepare_national_archive_transcripts.py | 246 | # Skip if AI generated tags | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_tests.py | 604 | long_text = "Lorem ipsum dolor sit amet, " * 100 | CODE |
| LOW⚡ | tests/test_tests.py | 604 | long_text = "Lorem ipsum dolor sit amet, " * 100 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_mine_html_templates_introduce_text_errors.py | 194 | <script>function doSomething() { return value; }</script> | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | olmocr/bench/runners/run_dotsocr.py | 147 | CODE |