SWE-bench: Can Language Models Resolve Real-world Github Issues?
This report summarizes scanner evidence for SWE-bench/SWE-bench, a Python project with 6,001 GitHub stars. SynthScan examined 67,487 lines of code across 651 source files, recording 433 pattern matches distributed across 15 syntactic categories. The legacy adjusted pattern signal is 8.8 (Moderate pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 322 finding(s).
Are imports declared and package/API names verified? 38 finding(s).
Are wrappers, guards, and repeated structures necessary? 314 finding(s).
Which changes need focused human review? 433 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 433 distinct pattern matches across 15 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_log_parsers_python.py | 23 | def test_bare_status_lines_do_not_crash(self): | STRING |
| LOW⚡ | tests/test_grading_truncated_ids.py | 17 | def test_exact_match_still_exact(): | CODE |
| LOW⚡ | tests/test_grading_truncated_ids.py | 23 | def test_missing_balanced_id_is_failed_no_fallback(): | CODE |
| LOW⚡ | tests/test_grading_truncated_ids.py | 30 | def test_truncated_id_unique_prefix_match(): | CODE |
| LOW⚡ | tests/test_grading_truncated_ids.py | 39 | def test_truncated_id_no_match_is_failed(): | CODE |
| LOW⚡ | tests/test_grading_truncated_ids.py | 45 | def test_truncated_id_ambiguous_same_status_ok(): | CODE |
| LOW | tests/test_grading_truncated_ids.py | 56 | def test_truncated_id_ambiguous_mixed_status_conservative(): | CODE |
| LOW | tests/test_grading_truncated_ids.py | 67 | def test_truncated_failed_id_reports_failed(): | CODE |
| LOW | tests/test_grading_truncated_ids.py | 74 | def test_mixed_passed_and_xfail_matches_resolve(): | CODE |
| LOW | tests/test_grading_truncated_ids.py | 93 | def test_surplus_closing_bracket_is_not_a_truncation(): | CODE |
| LOW | tests/test_inference_usage.py | 6 | def test_usage_to_dict_preserves_openai_cached_tokens(): | CODE |
| LOW | tests/test_inference_usage.py | 24 | def test_usage_to_dict_preserves_anthropic_cache_tokens(): | CODE |
| LOW | tests/test_inference_usage.py | 45 | def test_usage_to_dict_handles_dict_details(): | CODE |
| LOW⚡ | tests/test_grading_marker_fallback.py | 32 | def test_results_inside_markers_are_used(tmp_path): | CODE |
| LOW⚡ | tests/test_grading_marker_fallback.py | 41 | def test_results_outside_markers_fall_back_to_whole_log(tmp_path): | CODE |
| LOW⚡ | tests/test_grading_marker_fallback.py | 51 | def test_missing_markers_still_reports_not_found(tmp_path): | CODE |
| LOW⚡ | tests/test_grading_marker_fallback.py | 58 | def test_empty_results_without_evidence_is_not_found(tmp_path): | CODE |
| LOW | tests/test_grading_marker_fallback.py | 73 | def test_empty_results_with_runner_evidence_is_a_clean_run(tmp_path): | CODE |
| LOW | tests/test_grading_marker_fallback.py | 84 | def test_a_clean_jasmine_run_counts_as_having_run(tmp_path): | CODE |
| LOW | tests/test_grading_marker_fallback.py | 101 | def test_a_suite_that_never_started_is_still_rejected(tmp_path): | CODE |
| LOW | tests/test_grading_marker_fallback.py | 110 | def test_a_zero_count_summary_is_not_evidence_the_suite_ran(tmp_path): | CODE |
| LOW | tests/test_grading_marker_fallback.py | 127 | def test_a_real_karma_run_is_still_evidence(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 72 | def test_publish_omits_trajs_asset_when_there_are_none(packaged, tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 80 | def test_publish_includes_trajs_asset_when_present(packaged, tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 90 | def test_publish_preserves_header_comments(packaged, tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 99 | def test_publish_requires_a_destination(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 105 | def test_the_error_suggests_a_name(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 110 | def test_publish_without_a_repo_dir_is_a_clear_error(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 116 | def test_repo_name_comes_from_the_generated_header(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 133 | def test_register_refuses_unfilled_todos(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 140 | def test_unfilled_todos_names_dotted_paths(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 146 | def test_unfilled_todos_checks_the_readme_too(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 151 | def test_clean_entry_has_no_todos(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 155 | def test_write_entry_lands_at_the_split_path(packaged, tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 196 | def test_verify_passes_an_honest_entry(packaged, graded): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 204 | def test_verify_catches_a_claim_that_does_not_hold(packaged, graded): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 212 | def test_verify_catches_a_claim_with_no_log_at_all(packaged, graded): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 261 | def test_verify_needs_somewhere_to_look(packaged, graded): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 267 | def test_verify_rejects_unknown_split(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 272 | def test_verify_needs_results_json(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 281 | def test_init_commit_is_idempotent(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 291 | def test_to_https_normalizes_ssh_remotes(): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 301 | def test_register_split_comes_from_the_sidecar(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 305 | def test_register_split_falls_back_to_the_run(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 318 | def test_register_explicit_split_wins(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 322 | def test_register_split_unresolvable_is_a_clear_error(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 328 | def test_repo_name_comes_from_the_sidecar(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 334 | def test_verify_split_from_the_sidecar(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 338 | def test_verify_split_from_an_experiments_entry(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 345 | def test_verify_split_unresolvable_is_a_clear_error(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 355 | def test_resolvers_accept_a_run_directory(tmp_path, packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 367 | def test_resolvers_still_accept_the_deeper_paths(packaged): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 375 | def test_resolvers_name_where_they_looked(tmp_path): | CODE |
| LOW⚡ | tests/test_submit_flow.py | 383 | def test_register_always_forks(monkeypatch, packaged): | CODE |
| LOW | tests/test_submit_flow.py | 58 | def test_publish_pushes_and_records_assets(packaged, tmp_path): | CODE |
| LOW | tests/test_submit_flow.py | 166 | def test_write_entry_refuses_to_overwrite(packaged, tmp_path): | CODE |
| LOW | tests/test_submit_flow.py | 224 | def test_verify_reports_an_unclaimed_resolving_instance(packaged, graded): | CODE |
| LOW | tests/test_submit_flow.py | 237 | def test_missing_log_for_an_unclaimed_instance_is_not_a_failure(packaged, graded): | CODE |
| LOW | tests/test_submit_flow.py | 247 | def test_verify_reads_gzipped_test_output(packaged, graded): | CODE |
| LOW | tests/test_submit_flow.py | 401 | def test_register_reports_each_step(monkeypatch, packaged): | CODE |
| 174 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/collect/get_top_pypi.py | 20 | CODE | |
| LOW | swebench/collect/build_dataset_ft.py | 13 | CODE | |
| LOW | swebench/collect/utils.py | 353 | CODE | |
| LOW | swebench/collect/utils.py | 65 | CODE | |
| LOW | swebench/collect/utils.py | 127 | CODE | |
| LOW | swebench/collect/build_dataset.py | 97 | CODE | |
| LOW | swebench/submit/verify.py | 118 | CODE | |
| LOW | swebench/submit/publish.py | 87 | CODE | |
| LOW | swebench/cli/submit.py | 140 | CODE | |
| LOW | swebench/harness/run_evaluation.py | 229 | CODE | |
| LOW | swebench/harness/utils.py | 92 | CODE | |
| LOW | swebench/harness/utils.py | 137 | CODE | |
| LOW | swebench/harness/reporting.py | 16 | CODE | |
| LOW | swebench/harness/log_parsers/java.py | 6 | CODE | |
| LOW | swebench/harness/log_parsers/java.py | 68 | CODE | |
| LOW | swebench/harness/log_parsers/java.py | 85 | CODE | |
| LOW | swebench/harness/log_parsers/ruby.py | 7 | CODE | |
| LOW | swebench/harness/log_parsers/ruby.py | 30 | CODE | |
| LOW | swebench/harness/log_parsers/ruby.py | 50 | CODE | |
| LOW | swebench/harness/log_parsers/ruby.py | 69 | CODE | |
| LOW | swebench/harness/log_parsers/c.py | 8 | CODE | |
| LOW | swebench/harness/log_parsers/c.py | 35 | CODE | |
| LOW | swebench/harness/log_parsers/c.py | 57 | CODE | |
| LOW | swebench/harness/log_parsers/c.py | 94 | CODE | |
| LOW | swebench/harness/log_parsers/c.py | 113 | CODE | |
| LOW | swebench/harness/log_parsers/python.py | 45 | CODE | |
| LOW | swebench/harness/log_parsers/python.py | 82 | CODE | |
| LOW | swebench/harness/log_parsers/python.py | 162 | CODE | |
| LOW | swebench/harness/log_parsers/python.py | 192 | CODE | |
| LOW | swebench/harness/log_parsers/go.py | 6 | CODE | |
| LOW | swebench/harness/log_parsers/php.py | 6 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 9 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 109 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 192 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 324 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 390 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 453 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 627 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 654 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 675 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 696 | CODE | |
| LOW | swebench/harness/log_parsers/javascript.py | 745 | CODE | |
| LOW | swebench/harness/log_parsers/rust.py | 7 | CODE | |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 402 | CODE | |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 142 | CODE | |
| LOW | …/harness/modal_eval/run_evaluation_modal_entrypoint.py | 16 | CODE | |
| LOW | …/harness/modal_eval/run_evaluation_modal_entrypoint.py | 27 | CODE | |
| LOW | swebench/task/checks.py | 72 | CODE | |
| LOW | swebench/task/publish.py | 142 | CODE | |
| LOW | swebench/inference/run_api.py | 559 | CODE | |
| LOW | swebench/inference/run_llama.py | 28 | CODE | |
| LOW | swebench/inference/run_llama.py | 246 | CODE | |
| LOW | swebench/inference/llamao/modeling_flash_llama.py | 935 | CODE | |
| LOW | swebench/inference/make_datasets/create_instance.py | 340 | CODE | |
| LOW | swebench/inference/make_datasets/utils.py | 237 | CODE | |
| LOW | swebench/inference/make_datasets/utils.py | 277 | CODE | |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 81 | CODE | |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 104 | CODE | |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 276 | CODE | |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 309 | CODE | |
| 1 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/collect/get_tasks_pipeline.py | 89 | except Exception as e: | CODE |
| MEDIUM | swebench/collect/get_tasks_pipeline.py | 91 | print(f"Something went wrong for {repo}, skipping: {e}") | CODE |
| LOW | swebench/collect/utils.py | 168 | except Exception as e: | CODE |
| MEDIUM | swebench/collect/make_repo/call_make_repo.py | 16 | print(f"Error making mirror repo for {repo}") | CODE |
| MEDIUM | swebench/collect/cleanup/remove_envs.py | 56 | print(f"Error: {e}") | CODE |
| MEDIUM | swebench/collect/cleanup/remove_envs.py | 57 | print(f"Error output: {e.stderr}") | CODE |
| MEDIUM | swebench/collect/cleanup/remove_envs.py | 83 | print(f"Error: {e}") | CODE |
| MEDIUM | swebench/collect/cleanup/remove_envs.py | 84 | print(f"Error output: {e.stderr.decode('utf-8')}") | CODE |
| LOW | swebench/image_builder/docker_utils.py | 27 | except Exception as e: | CODE |
| LOW | swebench/image_builder/docker_build.py | 108 | except Exception as e: | CODE |
| LOW | swebench/cli/images.py | 118 | except Exception: | CODE |
| LOW | swebench/harness/run_evaluation.py | 113 | except Exception: | CODE |
| LOW | swebench/harness/run_evaluation.py | 147 | except Exception as e: | CODE |
| LOW | swebench/harness/run_evaluation.py | 166 | except Exception as e: | CODE |
| LOW | swebench/harness/run_evaluation.py | 418 | except Exception as e: | CODE |
| LOW | swebench/harness/docker_utils.py | 85 | except Exception as e2: | CODE |
| LOW | swebench/harness/docker_utils.py | 124 | except Exception: | CODE |
| LOW | swebench/harness/docker_utils.py | 150 | except Exception as e: | CODE |
| LOW⚡ | swebench/harness/utils.py | 106 | except Exception as e: | CODE |
| LOW | swebench/harness/utils.py | 128 | except Exception: | CODE |
| LOW | swebench/harness/remove_containers.py | 46 | except Exception as e: | CODE |
| MEDIUM | swebench/harness/remove_containers.py | 47 | print(f"Error removing container {name}: {e}") | CODE |
| MEDIUM | swebench/harness/modal_eval/run_evaluation_modal.py | 102 | print(f"Error reading stream: {e}") | CODE |
| MEDIUM | swebench/harness/modal_eval/run_evaluation_modal.py | 138 | print(f"Error during command execution: {e}") | CODE |
| MEDIUM | swebench/harness/modal_eval/run_evaluation_modal.py | 265 | print(f"Error creating sandbox: {e}") | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 101 | except Exception as e: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 137 | except Exception as e: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 153 | except Exception: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 157 | except Exception: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 264 | except Exception as e: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 384 | except Exception as e: | CODE |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 466 | except Exception: | CODE |
| LOW | swebench/task/publish.py | 159 | except Exception as e: # noqa: BLE001 - datasets raises many types when a dataset or split is absent | CODE |
| LOW | swebench/inference/run_api.py | 293 | except Exception as e: | CODE |
| LOW | swebench/inference/run_api.py | 334 | except Exception as e: | CODE |
| LOW | swebench/inference/run_api.py | 409 | except Exception as e: | CODE |
| LOW | swebench/inference/run_api.py | 505 | except Exception: | CODE |
| LOW | swebench/inference/run_llama.py | 330 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/create_instance.py | 484 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 58 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 96 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 138 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 332 | except Exception as e: | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 343 | except Exception: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | swebench/collect/get_top_pypi.py | 74 | except: | CODE |
| HIGH | swebench/image_builder/docker_build.py | 172 | except docker.errors.ImageNotFound: | CODE |
| HIGH | swebench/harness/run_evaluation.py | 111 | except docker.errors.NotFound: | CODE |
| HIGH | swebench/harness/run_evaluation.py | 113 | except Exception: | CODE |
| HIGH | swebench/harness/docker_utils.py | 124 | except Exception: | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 99 | except asyncio.CancelledError: | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 116 | except asyncio.CancelledError: | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 157 | except Exception: | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 151 | except asyncio.TimeoutError: | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 153 | except Exception: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 170 | "conda activate testbed && python -m pip install -r $HOME/requirements.txt", | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 171 | "conda activate testbed && python -m pip install --trusted-host pypi-mirror.modal.local -r $HOME/requirement | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 216 | "echo 'source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed' >> /root/.bashrc", | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 279 | "cd /testbed && git apply -v /tmp/patch.diff", | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 286 | "cd /testbed && patch --batch --fuzz=5 -p1 -i /tmp/patch.diff", | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 303 | "cd /testbed && git diff", | CODE |
| HIGH | swebench/harness/modal_eval/run_evaluation_modal.py | 337 | git_diff_output_after, returncode = runner.exec("cd /testbed && git diff") | CODE |
| HIGH | swebench/task/publish.py | 51 | rejected by the hub. Filled empty rather than null, so the column keeps the | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_gold_patch_source.py | 10 | CODE | |
| LOW | swebench/collect/print_pulls.py | 5 | CODE | |
| LOW | swebench/collect/utils.py | 1 | CODE | |
| LOW | swebench/image_builder/docker_utils.py | 1 | CODE | |
| LOW | swebench/image_builder/__init__.py | 9 | CODE | |
| LOW | swebench/image_builder/__init__.py | 9 | CODE | |
| LOW | swebench/image_builder/__init__.py | 9 | CODE | |
| LOW | swebench/image_builder/__init__.py | 14 | CODE | |
| LOW | swebench/image_builder/__init__.py | 14 | CODE | |
| LOW | swebench/image_builder/__init__.py | 14 | CODE | |
| LOW | swebench/image_builder/__init__.py | 19 | CODE | |
| LOW | swebench/image_builder/docker_build.py | 1 | CODE | |
| LOW | swebench/harness/run_evaluation.py | 1 | CODE | |
| LOW | swebench/harness/docker_utils.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/__init__.py | 1 | CODE | |
| LOW | swebench/harness/infra_failure.py | 16 | CODE | |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 3 | CODE | |
| LOW | swebench/harness/modal_eval/__init__.py | 1 | CODE | |
| LOW | swebench/harness/modal_eval/__init__.py | 2 | CODE | |
| LOW | swebench/harness/constants/__init__.py | 3 | CODE | |
| LOW | swebench/task/checks.py | 11 | CODE | |
| LOW | swebench/task/repo.py | 23 | CODE | |
| LOW | swebench/task/publish.py | 8 | CODE | |
| LOW | swebench/inference/usage.py | 3 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/__init__.py | 27 | __all__ = [ | CODE |
| LOW | swebench/collect/print_pulls.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/collect/utils.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/collect/build_dataset.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/image_builder/__init__.py | 23 | __all__ = [ | CODE |
| LOW | swebench/harness/__init__.py | 11 | __all__ = [ | CODE |
| LOW | swebench/harness/log_parsers/__init__.py | 138 | __all__ = [ | CODE |
| LOW | swebench/harness/modal_eval/__init__.py | 5 | __all__ = [ | CODE |
| LOW | swebench/inference/run_api.py | 30 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/run_llama.py | 23 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/run_live.py | 44 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/make_datasets/eval_retrieval.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/make_datasets/create_instance.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/make_datasets/tokenize_dataset.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/make_datasets/bm25_retrieval.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | swebench/inference/make_datasets/create_text_dataset.py | 22 | logger = logging.getLogger(__name__) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | swebench/harness/log_parsers/ruby.py | 0 | args: log (str): log content returns: dict: test case to test status mapping | STRING |
| HIGH | swebench/harness/log_parsers/c.py | 0 | args: log (str): log content returns: dict: test case to test status mapping | STRING |
| HIGH | swebench/harness/log_parsers/rust.py | 0 | args: log (str): log content returns: dict: test case to test status mapping | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/collect/make_repo/make_repo.sh | 12 | # Check if the target repository exists | COMMENT |
| LOW | swebench/collect/make_repo/make_repo.sh | 27 | # Check if the new repository already exists | COMMENT |
| LOW | swebench/collect/make_repo/make_repo.sh | 37 | # Check if the repository creation was successful | COMMENT |
| LOW | swebench/collect/make_repo/make_repo.sh | 49 | # Check if the local repository directory already exists | COMMENT |
| LOW | swebench/harness/run_evaluation.py | 89 | # Check if the image exists | COMMENT |
| LOW | swebench/harness/run_evaluation.py | 392 | # Check if git diff changed after running eval script | COMMENT |
| LOW | swebench/harness/grading.py | 358 | # Check if the model patch exists | COMMENT |
| LOW⚡ | swebench/harness/utils.py | 103 | # Check if instance ran successfully | COMMENT |
| LOW | swebench/harness/modal_eval/run_evaluation_modal.py | 339 | # Check if git diff changed after running eval script | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | swebench/logger.py | 9 | Set up a logger for evaluation and build processes. This logger is used for logging the evaluation process, bu | STRING |
| HIGH | swebench/inference/run_llama.py | 106 | Loads a base model and optionally PEFT adapters. Args: model_name_or_path (str): The name or path of t | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | pyproject.toml | 115 | # the harness writes test_output.txt files -- these match pytest's default | COMMENT |
| MEDIUM | swebench/submit/package.py | 44 | # the harness identified by metadata tags rather than by a separate directory. | COMMENT |
| MEDIUM | swebench/harness/grading.py | 41 | # openlayers' rendering harness logs "<case>': ok" per passing case at | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | docs/installation.md | 27 | ### Install dependencies for dataset generation or RAG inference | COMMENT |
| MEDIUM | docs/guides/create_rag_datasets.md | 1 | # Creating RAG Datasets for SWE-bench | COMMENT |
| MEDIUM | swebench/inference/run_api.py | 497 | # Determine the context window from LiteLLM's model registry so we can drop | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | swebench/harness/utils.py | 98 | # Create a future for running each instance | COMMENT |
| MEDIUM | swebench/harness/modal_eval/run_evaluation_modal.py | 1 | # This file contains logic for running evaluations on Modal: <https://modal.com/>. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/cli/submit.py | 400 | CODE | |
| LOW | swebench/cli/images.py | 16 | CODE | |
| LOW | swebench/cli/infer.py | 12 | CODE | |
| LOW | swebench/cli/evaluate.py | 16 | CODE | |
| LOW | swebench/inference/llamao/modeling_flash_llama.py | 590 | CODE | |
| LOW | swebench/inference/llamao/modeling_flash_llama.py | 767 | CODE | |
| LOW | swebench/inference/llamao/modeling_flash_llama.py | 935 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | swebench/inference/llamao/modeling_flash_llama.py | 1 | # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. | COMMENT |