One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
This report summarizes scanner evidence for EvolvingLMMs-Lab/lmms-eval, a Python project with 4,450 GitHub stars. SynthScan examined 337,150 lines of code across 3311 source files, recording 6426 pattern matches distributed across 24 syntactic categories. The legacy adjusted pattern signal is 35.4 (High pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 3898 finding(s).
Are imports declared and package/API names verified? 410 finding(s).
Are wrappers, guards, and repeated structures necessary? 4796 finding(s).
Which changes need focused human review? 6426 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 6426 distinct pattern matches across 24 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/prepare_physics_benchmarks.py | 296 | def _normalize_explanation_steps(problem: dict) -> list[dict]: | CODE |
| LOW | tools/prepare_physics_benchmarks.py | 364 | def _normalize_image_captions(problem: dict) -> str: | CODE |
| LOW | tools/task_input_capture.py | 172 | def _capture_instance_boundary(task_obj, instance, *, capture_mode: str) -> dict[str, Any]: | CODE |
| LOW | test/conftest.py | 143 | def pytest_collection_modifyitems(config, items): | CODE |
| LOW | test/README.md | 77 | def test_is_legacy_invocation(argv, expected): | CODE |
| LOW | test/README.md | 111 | def test_chat_precedence_and_force_simple(): | CODE |
| LOW | test/README.md | 154 | def test_task_output_type_is_generate_until(task_name, tm): | CODE |
| LOW | test/README.md | 256 | def test_configurable_task_generate_until_constructs_simple_layout(): | CODE |
| LOW | test/README.md | 292 | def test_extract_media_mixed_content(sample_image): | CODE |
| LOW | test/README.md | 326 | def test_agentic_single_round_terminal(): | CODE |
| LOW | test/README.md | 342 | def test_cache_hit_miss_lifecycle(self): | CODE |
| LOW | test/README.md | 420 | def test_unwrap_generation_result(): | CODE |
| LOW | test/README.md | 448 | def test_efficiency_metrics_tokens_per_correct(base_results): | CODE |
| LOW | test/tools/test_batch_watchdog.py | 15 | def test_find_stale_heartbeats_filters_by_phase_and_age(self): | CODE |
| LOW | test/tools/test_batch_watchdog.py | 33 | def test_write_timeout_snapshot_persists_payload(self): | CODE |
| LOW | test/cache/test_agentic_response_cache.py | 68 | def test_agentic_path_uses_response_cache(self): | CODE |
| LOW | test/cache/test_agentic_response_cache.py | 79 | def test_agentic_same_doc_different_prompt_not_collide(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 62 | def test_conditional_vs_unconditional_differ(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 71 | def test_different_idx_differ(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 81 | def test_task_fingerprint_invalidates(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 86 | def test_model_fingerprint_invalidates(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 91 | def test_generate_until_same_doc_idx_different_prompt_differ(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 107 | def test_no_dict_returns_empty(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 116 | def test_empty_string_rejected(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 120 | def test_malformed_loglikelihood_rejected(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 124 | def test_valid_responses_accepted(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 202 | def test_skipped_on_positive_temperature(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 211 | def test_temp_positive_repeats_all_bypass(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 270 | def test_nondeterministic_responses_logged(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 326 | def test_invalid_responses_stay_observable_but_are_not_recovered(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 471 | def test_separate_ranks_write_independently(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 712 | def test_fingerprint_stored_in_meta(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 718 | def test_different_models_use_separate_db_files(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 738 | def test_fingerprint_hash_and_schema_stored_in_meta(self): | CODE |
| LOW⚡ | test/cache/test_response_cache.py | 754 | def test_total_cached_entries_across_close_reopen(self): | CODE |
| LOW | test/cache/test_response_cache.py | 175 | def test_same_doc_idx_different_prompts_do_not_collide(self): | CODE |
| LOW | test/cache/test_response_cache.py | 222 | def test_temp_zero_repeats_all_hit_same_entry(self): | CODE |
| LOW | test/cache/test_response_cache.py | 237 | def test_do_sample_repeats_bypass(self): | CODE |
| LOW | test/cache/test_response_cache.py | 248 | def test_no_cross_run_poisoning(self): | CODE |
| LOW | test/cache/test_response_cache.py | 285 | def test_deterministic_responses_logged(self): | CODE |
| LOW | test/cache/test_response_cache.py | 346 | def test_legacy_audit_skips_invalid_rows_and_recovers_later_valid_rows(self): | CODE |
| LOW | test/cache/test_response_cache.py | 368 | def test_existing_invalid_database_rows_are_misses(self): | CODE |
| LOW | test/cache/test_response_cache.py | 381 | def test_invalid_local_row_falls_through_to_valid_shared_cache(self): | CODE |
| LOW | test/cache/test_response_cache.py | 398 | def test_valid_audit_repairs_invalid_existing_database_row(self): | CODE |
| LOW | test/cache/test_response_cache.py | 411 | def test_retry_repairs_shared_cache_after_merge(self): | CODE |
| LOW | test/cache/test_response_cache.py | 439 | def test_jsonl_replay_after_simulated_crash(self): | CODE |
| LOW | test/cache/test_response_cache.py | 491 | def test_merge_shards_combines_ranks(self): | CODE |
| LOW | test/cache/test_response_cache.py | 513 | def test_merge_shards_deduplicates_overlapping_keys(self): | CODE |
| LOW | test/cache/test_response_cache.py | 555 | def test_create_sets_up_run_directory(self): | CODE |
| LOW | test/cache/test_response_cache.py | 573 | def test_create_with_db_suffix_normalizes_to_directory(self): | CODE |
| LOW | test/cache/test_response_cache.py | 587 | def test_create_remote_uses_scratch(self): | CODE |
| LOW | test/cache/test_response_cache.py | 605 | def test_finalize_merges_rank_dbs_into_root(self): | CODE |
| LOW | test/cache/test_response_cache.py | 642 | def test_finalize_multi_rank_merge(self): | CODE |
| LOW | test/cache/test_response_cache.py | 680 | def test_finalize_skips_merge_on_failure(self): | CODE |
| LOW | test/cache/test_response_cache.py | 772 | def test_hit_miss_counters_reset_on_new_instance(self): | CODE |
| LOW | test/tui/test_server.py | 24 | def test_default_tui_origins_are_exact_local_origins(monkeypatch): | CODE |
| LOW | test/tui/test_server.py | 50 | def test_invalid_tui_origin_override_fails_closed(monkeypatch, value): | CODE |
| LOW | test/tui/test_server.py | 57 | def test_cors_allows_only_configured_origin(): | CODE |
| LOW | test/tui/test_server.py | 76 | def test_disallowed_origin_cannot_stop_job(monkeypatch): | CODE |
| LOW | test/tui/test_server.py | 99 | def test_preview_and_start_reject_request_environment_text(field, value): | CODE |
| 2492 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | test/conftest.py | 17 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 19 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 30 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 32 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 50 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 52 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 82 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 84 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 115 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/conftest.py | 117 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 129 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 131 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 196 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 198 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 264 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 266 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 320 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 322 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 465 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 467 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 706 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 708 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 748 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/cache/test_response_cache.py | 750 | # =========================================================================== | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 20 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 22 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 537 | # =========================================================================== | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 539 | # =========================================================================== | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 785 | # =========================================================================== | COMMENT |
| MEDIUM | test/cache/test_response_cache.py | 788 | # =========================================================================== | COMMENT |
| MEDIUM | test/cli/test_cli_dispatch_parametrized.py | 13 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/cli/test_cli_dispatch_parametrized.py | 15 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | test/cli/test_cli_dispatch_parametrized.py | 45 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | test/cli/test_cli_dispatch_parametrized.py | 47 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/cli/test_cli_dispatch_parametrized.py | 64 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/cli/test_cli_dispatch_parametrized.py | 66 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/entrypoints/test_job_scheduler_subprocess.py | 217 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | test/entrypoints/test_job_scheduler_subprocess.py | 219 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 30 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 32 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 53 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 55 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 78 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 80 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 101 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 103 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 124 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 126 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 183 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 185 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 224 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 226 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 385 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_protocol.py | 387 | # ============================================================================ | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 113 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 115 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 149 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 151 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 172 | # =========================================================================== | COMMENT |
| MEDIUM⚡ | test/eval/test_evaluator.py | 174 | # =========================================================================== | COMMENT |
| 685 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | docs/releases/lmms-eval-0.3.md | 0 | [question] {question} [reference answer] {ground_truth} [model answer] {model_response} [task] rate the model's answer b | STRING |
| HIGH | lmms_eval/tasks/openhermes/utils.py | 0 | [question] {question} [reference answer] {ground_truth} [model answer] {model_response} [task] rate the model's answer b | STRING |
| HIGH | lmms_eval/tasks/clotho_aqa/utils.py | 0 | [question] {question} [reference answer] {ground_truth} [model answer] {model_response} [task] rate the model's answer b | STRING |
| HIGH | lmms_eval/tasks/wavcaps/utils.py | 0 | [question] {question} [reference answer] {ground_truth} [model answer] {model_response} [task] rate the model's answer b | STRING |
| HIGH | lmms_eval/tasks/alpaca_audio/utils.py | 0 | [question] {question} [reference answer] {ground_truth} [model answer] {model_response} [task] rate the model's answer b | STRING |
| HIGH | lmms_eval/filters/extraction.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/muirbench/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/mmlu_pro/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/ai2d/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/worldqa/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/realworldqa/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/sat/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/mmlu/flan_cot_zeroshot/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/mmlu/flan_n_shot/generative/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/tasks/illusionvqa/utils.py | 0 | regex_pattern: the basic regex pattern to use. if fails to match, we will use the customized match procedure - step 1 : | STRING |
| HIGH | lmms_eval/filters/selection.py | 0 | can define custom behavior here, if an individual instantiation of a filter class should have state. | STRING |
| HIGH | lmms_eval/tasks/mix_evals/video2text/utils.py | 0 | can define custom behavior here, if an individual instantiation of a filter class should have state. | STRING |
| HIGH | lmms_eval/tasks/mix_evals/image2text/utils.py | 0 | can define custom behavior here, if an individual instantiation of a filter class should have state. | STRING |
| HIGH | lmms_eval/api/filter.py | 0 | can define custom behavior here, if an individual instantiation of a filter class should have state. | STRING |
| HIGH | lmms_eval/tasks/egothink/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mme/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/vl_rewardbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/camerabench_vqa/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/conbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mme_cot/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/naturalbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/llava-in-the-wild/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/llava-bench-coco/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/screenspot/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/refcoco/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/ferret/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/llava_wilder/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/refcocog/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/refcoco+/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | …al/tasks/multilingual-llava-bench-in-the-wild/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/xlrs/mcq_utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/vcr_wiki/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/timescope/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/longtimescope/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mmvu/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/lsdbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/lvbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/video-tt/gpt_utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/videomme/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mmworld/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mlvu/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/nocaps/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/cinepile/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/worldsense/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/av_odyssey/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/tomato/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/video-tt/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/capability/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/videomathqa/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mvbench/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/coco_cap/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/vatex/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/detailcaps/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| HIGH | lmms_eval/tasks/mme_realworld/utils.py | 0 | args: doc: a instance of the eval dataset results: [pred] returns: a dictionary with key: metric name (in this case text | STRING |
| 236 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH⚡ | README.md | 225 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | README.md | 225 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | README.md | 227 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | README.md | 227 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | README.md | 229 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | README.md | 229 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/README.md | 52 | export OPENAI_API_KEY="your-api-key" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 9 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 9 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 11 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 11 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 13 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/getting-started/run_examples.md | 13 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 255 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 255 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 308 | export GOOGLE_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 308 | export GOOGLE_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 333 | export ANTHROPIC_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH | docs/getting-started/run_examples.md | 333 | export ANTHROPIC_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 153 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 153 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 155 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 155 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 157 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_id.md | 157 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 146 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 146 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 148 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 148 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 150 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_ja.md | 150 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 171 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 171 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 173 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 173 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 175 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pt-BR.md | 175 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 135 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 135 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 137 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 137 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 139 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-TW.md | 139 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 139 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 139 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 141 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 141 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 143 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_vi.md | 143 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 136 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 136 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 138 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 138 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 140 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_pl.md | 140 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-CN.md | 163 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-CN.md | 163 | export OPENAI_API_KEY="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-CN.md | 165 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-CN.md | 165 | export HF_TOKEN="<YOUR_API_KEY>" | CODE |
| HIGH⚡ | docs/i18n/README_zh-CN.md | 167 | export REKA_API_KEY="<YOUR_API_KEY>" | CODE |
| 152 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/bench_video_decode_backends.py | 97 | except Exception as exc: | CODE |
| LOW | tools/score_physreason_psas_a.py | 80 | except Exception as error: # Network/provider failures are retried. | CODE |
| LOW | tools/score_physreason_psas_a.py | 291 | except Exception as error: | CODE |
| LOW | tools/task_input_capture.py | 74 | except Exception: | CODE |
| LOW | tools/task_input_capture.py | 265 | except Exception: | CODE |
| MEDIUM | tools/task_input_capture.py | 262 | def _norm(v): | CODE |
| LOW | test/eval/utils.py | 145 | except Exception: | CODE |
| LOW | test/eval/utils.py | 342 | except Exception as e: | CODE |
| MEDIUM | test/eval/run_cicd.py | 31 | print(f"Error: Model directory '{model_name}' not found in {test_dir}") | CODE |
| LOW | examples/mcp_server/crop_video_mcp_server.py | 118 | except Exception as e: | CODE |
| LOW | lmms_eval/utils.py | 307 | except Exception as e: | CODE |
| LOW | lmms_eval/utils.py | 587 | except Exception: | CODE |
| LOW⚡ | lmms_eval/utils.py | 861 | except Exception: | CODE |
| LOW | lmms_eval/utils.py | 956 | except Exception as ex: | CODE |
| LOW | lmms_eval/utils.py | 1051 | except Exception as ex: | CODE |
| MEDIUM | lmms_eval/utils.py | 50 | def is_json(string): | CODE |
| LOW | lmms_eval/evaluator.py | 799 | except Exception as e: | CODE |
| LOW | lmms_eval/__main__.py | 507 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/__main__.py | 597 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/__main__.py | 607 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/__main__.py | 610 | except Exception as e: | CODE |
| LOW | lmms_eval/logging_utils.py | 22 | except Exception as e: | CODE |
| LOW | lmms_eval/logging_utils.py | 76 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/openhermes/utils.py | 100 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/openhermes/utils.py | 127 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/egothink/utils.py | 114 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 62 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 82 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 121 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 178 | except Exception as e: | CODE |
| MEDIUM | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 72 | def parse_score(review): | CODE |
| MEDIUM | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 161 | def llava_aggregation(results, category): | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils.py | 57 | except Exception: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils.py | 128 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/llava-in-the-wild/utils.py | 185 | except Exception as e: | CODE |
| MEDIUM | lmms_eval/tasks/llava-in-the-wild/utils.py | 168 | def llava_aggregation(results, category): | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/image_utils.py | 22 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 46 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 83 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 109 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 148 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 153 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 224 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 227 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/mmsearch/utils/utils.py | 287 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/mmsearch/utils/utils.py | 290 | except Exception as e: | CODE |
| MEDIUM⚡ | lmms_eval/tasks/mmsearch/utils/utils.py | 291 | print(f"An error occurred: {e}") | CODE |
| LOW | lmms_eval/tasks/mmsearch/score/result_summary.py | 47 | except Exception: | CODE |
| LOW | lmms_eval/tasks/ami/utils.py | 103 | except Exception as e: | CODE |
| LOW | lmms_eval/tasks/herbench/utils.py | 219 | except Exception as e: | CODE |
| MEDIUM⚡ | lmms_eval/tasks/voicebench/utils.py | 98 | print(f"Error converting AudioDecoder object: {e}") | CODE |
| MEDIUM⚡ | lmms_eval/tasks/voicebench/utils.py | 107 | print(f"Error converting audio object: {e}") | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 97 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 106 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1474 | except Exception: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1479 | except Exception as e: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1525 | except Exception: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1530 | except Exception: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1535 | except Exception: | CODE |
| LOW⚡ | lmms_eval/tasks/voicebench/utils.py | 1540 | except Exception as e: | CODE |
| 816 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tools/prepare_physics_benchmarks.py | 115 | except (OSError, TimeoutError, urllib.error.URLError, http.client.IncompleteRead): | CODE |
| HIGH | tools/task_input_capture.py | 74 | except Exception: | CODE |
| HIGH | test/conftest.py | 153 | except ImportError: | CODE |
| HIGH | test/entrypoints/test_job_scheduler_subprocess.py | 116 | except asyncio.CancelledError: | CODE |
| HIGH | test/eval/utils.py | 98 | except (httpx.ConnectError, httpx.ReadTimeout, httpx.ConnectTimeout): | CODE |
| HIGH | test/eval/utils.py | 211 | except ProcessLookupError: | CODE |
| HIGH | test/eval/utils.py | 221 | except ProcessLookupError: | CODE |
| HIGH | lmms_eval/utils.py | 128 | except ImportError: | CODE |
| HIGH | lmms_eval/utils.py | 823 | except (subprocess.CalledProcessError, FileNotFoundError): | CODE |
| HIGH | lmms_eval/utils.py | 834 | except (subprocess.CalledProcessError, FileNotFoundError): | CODE |
| HIGH | lmms_eval/evaluator.py | 955 | except (TypeError, json.JSONDecodeError): | CODE |
| HIGH | lmms_eval/tasks/kaleidoscope/utils.py | 418 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/herbench/utils.py | 195 | except FileExistsError: | CODE |
| HIGH | lmms_eval/tasks/voicebench/utils.py | 1525 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/voicebench/utils.py | 1535 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/mmvu/utils.py | 252 | except ValueError: | CODE |
| HIGH | lmms_eval/tasks/mathvision/eval_utils.py | 175 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/gedit_bench/viescore/openai_backend.py | 43 | except ImportError: | CODE |
| HIGH | lmms_eval/tasks/mathvista/mathvista_evals.py | 234 | except ValueError: | CODE |
| HIGH | lmms_eval/tasks/mathvista/mathvista_evals.py | 241 | except ValueError: | CODE |
| HIGH | lmms_eval/tasks/mathvista/mathvista_evals.py | 253 | except re.error: | CODE |
| HIGH | lmms_eval/tasks/visualwebbench/utils.py | 217 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/vsibench/utils.py | 243 | except ValueError: | CODE |
| HIGH | lmms_eval/tasks/auxsolidmath/utils.py | 83 | except (SyntaxError, NameError, ValueError, ZeroDivisionError): | CODE |
| HIGH | lmms_eval/tasks/kris_bench/utils.py | 280 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/kris_bench/utils.py | 422 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/prismm_bench/utils.py | 97 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/prismm_bench/utils.py | 69 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/sunrgbd/utils.py | 87 | except json.JSONDecodeError: | CODE |
| HIGH | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 443 | except (ValueError, IndexError): | CODE |
| HIGH | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 770 | except (ValueError, SyntaxError): | CODE |
| HIGH | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 1053 | except (ValueError, SyntaxError): | CODE |
| HIGH | …atialtreebench/metrics/mindcube_cogmap/parse_output.py | 223 | except Exception: | CODE |
| HIGH | …ebench/metrics/mindcube_cogmap/src/utils/text_utils.py | 30 | except json.JSONDecodeError: | CODE |
| HIGH | …ebench/metrics/mindcube_cogmap/src/utils/text_utils.py | 39 | except json.JSONDecodeError: | CODE |
| HIGH | …rics/mindcube_cogmap/src/evaluation/core/extractors.py | 177 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/refcoco_grounding/utils.py | 82 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/pointbench/utils.py | 105 | except (UnicodeEncodeError, UnicodeDecodeError): | CODE |
| HIGH | lmms_eval/tasks/wm_abench/utils.py | 68 | except (ValueError, TypeError): | CODE |
| HIGH | …s_eval/tasks/olympiadbench_mimo/olympiadbench_evals.py | 152 | except Exception: | CODE |
| HIGH | …s_eval/tasks/olympiadbench_mimo/olympiadbench_evals.py | 159 | except Exception: | CODE |
| HIGH | …s_eval/tasks/olympiadbench_mimo/olympiadbench_evals.py | 166 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/mmsi_video/utils.py | 147 | except Exception: | CODE |
| HIGH | …ms_eval/tasks/mdpbench/cdm/modules/latex2bbox_color.py | 209 | except Exception: | CODE |
| HIGH | …ms_eval/tasks/mdpbench/cdm/modules/latex2bbox_color.py | 235 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/captionqa/utils.py | 216 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_utils.py | 56 | except (ValueError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 279 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 300 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 313 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 324 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 358 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/seephys/seephys_evals.py | 113 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/spatial_dise/utils.py | 134 | except Exception: | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 96 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 100 | except (ValueError, SyntaxError): | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 162 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 229 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 279 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | lmms_eval/tasks/uni_mmmu/utils.py | 330 | except (json.JSONDecodeError, TypeError): | CODE |
| 112 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/regression.py | 90 | CODE | |
| LOW | tools/prepare_physics_benchmarks.py | 372 | CODE | |
| LOW | tools/task_input_capture.py | 172 | CODE | |
| LOW | test/models/test_decord_lazy_imports.py | 22 | CODE | |
| LOW | test/eval/test_request_construction_contract.py | 212 | CODE | |
| LOW | test/eval/test_evaluator_sample_limits.py | 207 | CODE | |
| LOW | test/eval/utils.py | 130 | CODE | |
| LOW | test/eval/mcq_extract/probe_benchmarks.py | 52 | CODE | |
| LOW | lmms_eval/protocol.py | 49 | CODE | |
| LOW | lmms_eval/protocol.py | 65 | CODE | |
| LOW | lmms_eval/protocol.py | 95 | CODE | |
| LOW | lmms_eval/protocol.py | 147 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 230 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 298 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 386 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 462 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 134 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 164 | CODE | |
| LOW | lmms_eval/utils.py | 157 | CODE | |
| LOW | lmms_eval/utils.py | 541 | CODE | |
| LOW | lmms_eval/evaluator.py | 195 | CODE | |
| LOW | lmms_eval/evaluator.py | 249 | CODE | |
| LOW | lmms_eval/evaluator.py | 298 | CODE | |
| LOW | lmms_eval/evaluator.py | 366 | CODE | |
| LOW | lmms_eval/evaluator.py | 810 | CODE | |
| LOW | lmms_eval/evaluator.py | 964 | CODE | |
| LOW | lmms_eval/evaluator.py | 563 | CODE | |
| LOW | lmms_eval/__main__.py | 477 | CODE | |
| LOW | lmms_eval/__main__.py | 639 | CODE | |
| LOW | lmms_eval/logging_utils.py | 155 | CODE | |
| LOW | lmms_eval/logging_utils.py | 319 | CODE | |
| LOW | lmms_eval/logging_utils.py | 169 | CODE | |
| LOW | lmms_eval/filters/extraction.py | 105 | CODE | |
| LOW | lmms_eval/tasks/__init__.py | 96 | CODE | |
| LOW | lmms_eval/tasks/__init__.py | 233 | CODE | |
| LOW | lmms_eval/tasks/__init__.py | 383 | CODE | |
| LOW | lmms_eval/tasks/egoplan/utils.py | 39 | CODE | |
| LOW | lmms_eval/tasks/mme/utils.py | 48 | CODE | |
| LOW | lmms_eval/tasks/hipho/utils.py | 13 | CODE | |
| LOW | lmms_eval/tasks/hipho/utils.py | 36 | CODE | |
| LOW | lmms_eval/tasks/hipho/utils.py | 63 | CODE | |
| LOW | lmms_eval/tasks/hipho/utils.py | 166 | CODE | |
| LOW | lmms_eval/tasks/mmsearch/lmms_eval_utils.py | 392 | CODE | |
| LOW | lmms_eval/tasks/mmsearch/utils/utils.py | 74 | CODE | |
| LOW | lmms_eval/tasks/vmcbench/utils.py | 55 | CODE | |
| LOW | lmms_eval/tasks/osworld_g/utils.py | 115 | CODE | |
| LOW | lmms_eval/tasks/ami/utils.py | 35 | CODE | |
| LOW | lmms_eval/tasks/ami/utils.py | 316 | CODE | |
| LOW | lmms_eval/tasks/herbench/utils.py | 54 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 30 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 196 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 319 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 1162 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 1466 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 320 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 344 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 770 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 834 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 905 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 992 | CODE | |
| 670 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | lmms_eval/utils.py | 663 | # Check if v is not empty (handle numpy array safely) | COMMENT |
| LOW | lmms_eval/tasks/__init__.py | 336 | # Check if this is a duplicate. | COMMENT |
| LOW | lmms_eval/tasks/mmsearch/utils/image_utils.py | 42 | # Check if the image is grayscale or color | COMMENT |
| LOW | lmms_eval/tasks/ami/utils.py | 60 | # Check if samples itself has a data attribute | COMMENT |
| LOW⚡ | lmms_eval/tasks/mmvu/utils.py | 200 | # Check if choice text appears in prediction | COMMENT |
| LOW⚡ | lmms_eval/tasks/mmvu/utils.py | 203 | # Check if prediction contains both the letter and key words from choice | COMMENT |
| LOW⚡ | lmms_eval/tasks/mmvu/utils.py | 208 | # Check if at least one key word appears in prediction | COMMENT |
| LOW | lmms_eval/tasks/mmvu/utils.py | 260 | # Check if all answer words appear in prediction (order-independent) | COMMENT |
| LOW | lmms_eval/tasks/temporalbench/utils.py | 144 | # Print the results | COMMENT |
| LOW⚡ | lmms_eval/tasks/mathvision/eval_utils.py | 60 | # Check if string is a tuple representation and has more than one element | COMMENT |
| LOW⚡ | lmms_eval/tasks/mathvision/eval_utils.py | 67 | # Check if string is a list representation and has more than one element | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 158 | # Check if the ID contains the specified area or if it matches the pattern | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 222 | # Check if there are any occurrences of '\frac' in the string. | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 244 | # Check if the denominator (b) is already braced. | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 264 | # Check if the string contains exactly one slash, which may indicate it's a fraction. | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 276 | # Check if the string is in the expected format after conversion. | COMMENT |
| LOW | lmms_eval/tasks/mathvision/eval_utils.py | 297 | # Check if "\sqrt" is not in the string. If not, return the string as is. | COMMENT |
| LOW | lmms_eval/tasks/voxpopuli/utils.py | 57 | # Check if samples itself has a data attribute | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 18 | # Check if value is an integer or float | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 21 | # Check if value is a boolean or null | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 118 | # Check if the parsed object is a list | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 122 | # Check if all elements are integers and between 0 to 10 | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 220 | # Check if we found two delimiters | COMMENT |
| LOW⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 358 | # Check if the key exists at the top level of the JSON structure | COMMENT |
| LOW | lmms_eval/tasks/gedit_bench/viescore/utils.py | 101 | # Check if the target sequence appears exactly twice | COMMENT |
| LOW | lmms_eval/tasks/gedit_bench/viescore/utils.py | 144 | # Check if the key starts with 'score' and the value is in the correct range | COMMENT |
| LOW | lmms_eval/tasks/gedit_bench/viescore/utils.py | 151 | # Check if at least 'score1' and 'score2' are present | COMMENT |
| LOW | lmms_eval/tasks/gedit_bench/viescore/utils.py | 305 | # Check if we found two delimiters | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 197 | # Check if control counts are reasonable | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 322 | # Check if cell is unchanged | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 437 | # Check if filled | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 545 | # Check if symbols are evenly spaced | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 572 | # Check if colors maintain relative order | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 692 | # Check if a new symbol was added | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 707 | # Check if original symbols are still present (by color) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 930 | # Check if correct number of symbols were inserted | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 947 | # Check if new symbols in final match the target type | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1169 | # Check if acceleration is roughly constant (gravity) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1188 | # Check if the frames are nearly identical (GT vs GT case) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1207 | # Check if ball is near ground (bottom of frame) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1342 | # Check if animals moved from left to right side | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1380 | # Check if animals are on right side (target area) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1472 | # Check if sizes increase left to right | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1491 | # Check if bottom y-coordinates are similar (aligned on baseline) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part5.py | 1688 | # Check if rotation is consistent (all same direction) | COMMENT |
| LOW⚡ | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1939 | # Check if marking is on second largest | COMMENT |
| LOW⚡ | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1956 | # Check if marking forms a proper border | COMMENT |
| LOW⚡ | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 2125 | # Check if circle is in a reasonable position | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 252 | # Check if agent reached the door (green agent at door position) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 660 | # Check if there's a continuous path of this color between them | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 729 | # Check if this pixel is colored (high saturation) and not background | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 865 | # Check if marked candidate matches expected type | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 905 | # Check if sizes increase smoothly | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1286 | # Check if outline is around topmost shape | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1305 | # Check if outline forms a closed shape | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1459 | # Check if circle marks the unique shape | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1478 | # Check if circle is in a reasonable position (not at edges) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1498 | # Check if circle is reasonable size | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 1721 | # Check if circle is in reasonable position (not at edges) | COMMENT |
| LOW | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 2106 | # Check if circle is near the longest edge midpoint | COMMENT |
| 153 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/task_input_capture.py | 15 | CODE | |
| LOW | test/models/test_vqtoken.py | 1 | CODE | |
| LOW | test/models/test_openai.py | 1 | CODE | |
| LOW | test/models/test_litellm.py | 3 | CODE | |
| LOW | test/models/test_vllm_sampling_params.py | 1 | CODE | |
| LOW | test/models/test_llava_hf_generate_until.py | 13 | CODE | |
| LOW | test/eval/test_messages_text_ctx.py | 17 | CODE | |
| LOW | lmms_eval/imports.py | 24 | CODE | |
| LOW | lmms_eval/filters/__init__.py | 1 | CODE | |
| LOW | lmms_eval/tasks/vstat/utils.py | 9 | CODE | |
| LOW | lmms_eval/tasks/cc_ocr/evaluators/__init__.py | 1 | CODE | |
| LOW | lmms_eval/tasks/cc_ocr/evaluators/__init__.py | 1 | CODE | |
| LOW | lmms_eval/tasks/cc_ocr/evaluators/__init__.py | 1 | CODE | |
| LOW | lmms_eval/tasks/vantage_vqa/utils.py | 1 | CODE | |
| LOW | lmms_eval/tasks/vbvr/utils.py | 21 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 10 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 11 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 13 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 16 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 19 | CODE | |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/__init__.py | 22 | CODE | |
| LOW | lmms_eval/tasks/vsibench/utils.py | 3 | CODE | |
| LOW | lmms_eval/tasks/kris_bench/utils.py | 30 | CODE | |
| LOW | lmms_eval/tasks/mvp/utils.py | 32 | CODE | |
| LOW | lmms_eval/tasks/vbench/build_dataset.py | 19 | CODE | |
| LOW | lmms_eval/tasks/multi_crit/utils.py | 3 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 6 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 6 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 6 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 6 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 6 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 7 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 7 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 7 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 12 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 12 | CODE | |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 12 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 23 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 23 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 23 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 26 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 34 | CODE | |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 34 | CODE | |
| LOW | …rics/mindcube_cogmap/src/evaluation/cogmap/__init__.py | 11 | CODE | |
| LOW | …rics/mindcube_cogmap/src/evaluation/cogmap/__init__.py | 11 | CODE | |
| LOW | …rics/mindcube_cogmap/src/evaluation/cogmap/__init__.py | 11 | CODE | |
| LOW | …rics/mindcube_cogmap/src/evaluation/cogmap/__init__.py | 11 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 9 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 10 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 10 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 11 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 11 | CODE | |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 11 | CODE | |
| LOW | lmms_eval/tasks/rvos/eval_sam2.py | 28 | CODE | |
| LOW | lmms_eval/tasks/rvos/utils.py | 23 | CODE | |
| 131 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | test/cache/test_response_cache.py | 661 | # Create a rank-0 instance with finalize metadata and call finalize | COMMENT |
| MEDIUM | test/eval/utils.py | 19 | # Create a temporary directory | COMMENT |
| MEDIUM | examples/models/plm.sh | 24 | # Define the tasks you want to evaluate PLM on. We support all the tasks present in lmms-eval, however have tested the f | COMMENT |
| MEDIUM | lmms_eval/logging_utils.py | 27 | # Define the pattern to match ',none' at the end of the string | COMMENT |
| MEDIUM | lmms_eval/tasks/llava-in-the-wild/utils_ko.py | 51 | # Create a Request object for the unified judge API | COMMENT |
| MEDIUM | lmms_eval/tasks/mmsearch/utils/utils.py | 173 | # Create a 512x512 pixel blank image | COMMENT |
| MEDIUM | lmms_eval/tasks/mmsearch/utils/utils.py | 344 | # Create an array of Promises, each corresponding to the loading of an image | STRING |
| MEDIUM | lmms_eval/tasks/ami/utils.py | 28 | # Create a translation table that removes all punctuation except apostrophe | COMMENT |
| MEDIUM | lmms_eval/tasks/gedit_bench/viescore/vie_prompts.py | 1 | # This file is generated automatically through parse_prompt.py | COMMENT |
| MEDIUM | lmms_eval/tasks/gedit_bench/viescore/parse_prompt.py | 6 | out_file.write("# This file is generated automatically through parse_prompt.py\n\n") | CODE |
| MEDIUM | lmms_eval/tasks/gedit_bench/viescore/utils.py | 170 | # Define the delimiters | COMMENT |
| MEDIUM | lmms_eval/tasks/gedit_bench/viescore/utils.py | 257 | # Define the delimiters | COMMENT |
| MEDIUM | lmms_eval/tasks/mathvista/mathvista_evals.py | 173 | # Create a custom server config for this specific request with different parameters | STRING |
| MEDIUM | lmms_eval/tasks/vbvr/vbvr_bench/utils.py | 110 | # Create a mask of pixels that are NOT background | COMMENT |
| MEDIUM | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part2.py | 1879 | # Create a mask with the triangle contour | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcoco/utils_rec.py | 35 | # Create a new dataset from the exploded rows | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcoco/utils_rec.py | 66 | # Define the regex pattern to find the first instance of four floats within square brackets | COMMENT |
| MEDIUM | lmms_eval/tasks/mmmu/utils.py | 231 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | lmms_eval/tasks/mmmu/utils_group_img.py | 20 | # Create a drawing context | COMMENT |
| MEDIUM | lmms_eval/tasks/mmmu/utils_group_img.py | 198 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | lmms_eval/tasks/mmmu/reasoning/utils.py | 145 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | …indcube_cogmap/src/evaluation/cogmap/cogmap_metrics.py | 282 | # Create an empty result structure to build upon | COMMENT |
| MEDIUM | lmms_eval/tasks/mdpbench/table_utils.py | 190 | # Define a regex pattern to find Markdown tables with newlines | COMMENT |
| MEDIUM | lmms_eval/tasks/jmmmu/utils.py | 93 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | lmms_eval/tasks/llava_wilder/utils.py | 57 | # Create a Request object for the unified judge API | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcocog/utils_rec.py | 35 | # Create a new dataset from the exploded rows | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcocog/utils_rec.py | 66 | # Define the regex pattern to find the first instance of four floats within square brackets | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcoco+/utils_rec.py | 35 | # Create a new dataset from the exploded rows | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/refcoco+/utils_rec.py | 66 | # Define the regex pattern to find the first instance of four floats within square brackets | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/nextqa/utils.py | 164 | # The following code copied from # | COMMENT |
| MEDIUM | lmms_eval/tasks/jmmmu_pro/utils.py | 93 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | lmms_eval/tasks/emma/utils.py | 170 | # Create a Request object for the unified judge API | COMMENT |
| MEDIUM | lmms_eval/tasks/videommmu/utils.py | 31 | # Define the mapping for subjects to their respective directories | COMMENT |
| MEDIUM | lmms_eval/tasks/videommmu/utils.py | 123 | # Define the option letters based on the number of options | COMMENT |
| MEDIUM | lmms_eval/tasks/videommmu/utils.py | 156 | # Define a regex pattern to match "validation_" at the beginning and "_<number>" at the end | COMMENT |
| MEDIUM | lmms_eval/tasks/wild_vision_bench/utils.py | 273 | # Create a DataFrame with "model" and "score" columns | STRING |
| MEDIUM | lmms_eval/tasks/mmvetv2/utils.py | 15 | # Create a drawing context | COMMENT |
| MEDIUM | lmms_eval/tasks/mmvetv2/utils.py | 220 | # Create a Request object for the unified judge API | STRING |
| MEDIUM | lmms_eval/tasks/mmvet/utils.py | 47 | # Create a Request object for the unified judge API | STRING |
| MEDIUM | …s/megabench/metrics/scoring/general_numerical_match.py | 34 | # Create a multiprocessing.Queue to receive the output | COMMENT |
| MEDIUM | …_eval/tasks/megabench/metrics/scoring/program_judge.py | 18 | # Create a CodeTester instance with the response and the found test cases | COMMENT |
| MEDIUM | …/tasks/megabench/metrics/scoring/common/conversions.py | 216 | # Create a new image with the calculated size | COMMENT |
| MEDIUM | lmms_eval/tasks/megabench/breakdown/analysis_utils.py | 90 | meta = all_meta[task_name].copy() # Create a copy to avoid modifying original | CODE |
| MEDIUM | lmms_eval/tasks/stare/utils.py | 126 | # Create a Request object for the unified judge API | COMMENT |
| MEDIUM | lmms_eval/tasks/tomato/utils.py | 122 | # Create a VideoCapture object | STRING |
| MEDIUM | …al/tasks/multilingual-llava-bench-in-the-wild/utils.py | 41 | # Create a Request object for the unified judge API | COMMENT |
| MEDIUM | lmms_eval/tasks/ocrbench_v2/TEDS_metric.py | 365 | # Create a dictionary for each combination of values. | COMMENT |
| MEDIUM | lmms_eval/tasks/ocrbench_v2/TEDS_metric.py | 377 | # Create a dictionary for each combination of values. | COMMENT |
| MEDIUM | lmms_eval/loggers/utils.py | 24 | # Define the pattern to match ',none' at the end of the string | COMMENT |
| MEDIUM | lmms_eval/models/model_utils/thyme/sandbox.py | 65 | # Define the temporary folder for processed images | COMMENT |
| MEDIUM | lmms_eval/models/model_utils/thyme/sandbox.py | 186 | # Define a minimum dimension for the crop | COMMENT |
| MEDIUM | lmms_eval/models/simple/llama_vid.py | 131 | # Create the folder if it doesn't exist | COMMENT |
| MEDIUM | lmms_eval/models/simple/minicpm_o.py | 12 | # Create a dedicated environment (recommended) | STRING |
| MEDIUM | lmms_eval/api/task.py | 1121 | # Create a new symbolic link | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | test/conftest.py | 59 | CODE | |
| LOW | test/cache/test_response_cache.py | 29 | CODE | |
| LOW | examples/mcp_server/sample_mcp_server.py | 40 | CODE | |
| LOW | lmms_eval/evaluator_utils.py | 54 | CODE | |
| LOW | lmms_eval/evaluator.py | 366 | CODE | |
| LOW | lmms_eval/evaluator.py | 964 | CODE | |
| LOW | lmms_eval/tasks/asr_wer_utils.py | 102 | CODE | |
| LOW | lmms_eval/tasks/voicebench/utils.py | 342 | CODE | |
| LOW | lmms_eval/tasks/vcr_wiki/utils.py | 185 | CODE | |
| LOW | lmms_eval/tasks/plm_videobench/eval_utils.py | 27 | CODE | |
| LOW | lmms_eval/tasks/cc_ocr/evaluators/common.py | 32 | CODE | |
| LOW | lmms_eval/tasks/aime/utils.py | 126 | CODE | |
| LOW | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 108 | CODE | |
| LOW | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 928 | CODE | |
| LOW | …ms_eval/tasks/spatialtreebench/metrics/rule_metrics.py | 1133 | CODE | |
| LOW | …indcube_cogmap/src/evaluation/cogmap/cogmap_metrics.py | 243 | CODE | |
| LOW | lmms_eval/tasks/mmupd/mmupd_evals.py | 64 | CODE | |
| LOW | lmms_eval/tasks/librispeech/cn_tn.py | 645 | CODE | |
| LOW | lmms_eval/tasks/librispeech/cn_tn.py | 1040 | CODE | |
| LOW | lmms_eval/tasks/nextqa/utils.py | 244 | CODE | |
| LOW | lmms_eval/tasks/worldqa/utils.py | 175 | CODE | |
| LOW | lmms_eval/tasks/openai_math/utils.py | 123 | CODE | |
| LOW | lmms_eval/tasks/gpqa/openai/utils.py | 90 | CODE | |
| LOW | …tasks/megabench/metrics/scoring/latex_expr_equality.py | 74 | CODE | |
| LOW | …eval/tasks/megabench/metrics/scoring/sacrebleu_bleu.py | 21 | CODE | |
| LOW | …eval/tasks/megabench/metrics/parsing/common/parsers.py | 140 | CODE | |
| LOW | lmms_eval/tasks/tomato/utils.py | 214 | CODE | |
| LOW | lmms_eval/tasks/ocrbench_v2/TEDS_metric.py | 327 | CODE | |
| LOW | lmms_eval/tasks/ocrbench_v2/TEDS_metric.py | 885 | CODE | |
| LOW | lmms_eval/loggers/evaluation_tracker.py | 106 | CODE | |
| LOW | lmms_eval/verifiers/gemini.py | 68 | CODE | |
| LOW | lmms_eval/verifiers/openai.py | 76 | CODE | |
| LOW | lmms_eval/mcp/tools.py | 174 | CODE | |
| LOW | lmms_eval/models/whisper_tt.py | 36 | CODE | |
| LOW | lmms_eval/models/chat/llava_onevision2.py | 70 | CODE | |
| LOW | lmms_eval/models/chat/async_hf_model.py | 39 | CODE | |
| LOW | lmms_eval/models/chat/nanovlm.py | 47 | CODE | |
| LOW | lmms_eval/models/chat/vllm_omni.py | 421 | CODE | |
| LOW | lmms_eval/models/chat/vllm_generate.py | 41 | CODE | |
| LOW | lmms_eval/models/chat/fastvideo.py | 171 | CODE | |
| LOW | lmms_eval/models/chat/vllm.py | 25 | CODE | |
| LOW | lmms_eval/models/chat/vllm_omni_api.py | 82 | CODE | |
| LOW | lmms_eval/models/chat/longvila.py | 24 | CODE | |
| LOW | lmms_eval/models/chat/async_openai.py | 41 | CODE | |
| LOW | lmms_eval/models/chat/aero_realtime_vllm.py | 318 | CODE | |
| LOW | lmms_eval/models/chat/neo_ov.py | 155 | CODE | |
| LOW | lmms_eval/models/chat/huggingface.py | 39 | CODE | |
| LOW | lmms_eval/models/chat/sglang_diffusion.py | 181 | CODE | |
| LOW | lmms_eval/models/chat/sglang.py | 125 | CODE | |
| LOW | lmms_eval/models/chat/bagel_lmms_engine.py | 46 | CODE | |
| LOW | lmms_eval/models/chat/internvl_hf.py | 60 | CODE | |
| LOW | lmms_eval/models/simple/llama_vid.py | 39 | CODE | |
| LOW | lmms_eval/models/simple/cambrians_vsc.py | 56 | CODE | |
| LOW | lmms_eval/models/simple/llama_vision.py | 28 | CODE | |
| LOW | lmms_eval/models/simple/cambrians_vsc_streaming.py | 36 | CODE | |
| LOW | lmms_eval/models/simple/vila.py | 45 | CODE | |
| LOW | lmms_eval/models/simple/ovis_u1.py | 81 | CODE | |
| LOW | lmms_eval/models/simple/aria.py | 47 | CODE | |
| LOW | lmms_eval/models/simple/internvl.py | 103 | CODE | |
| LOW | lmms_eval/models/simple/internvl.py | 405 | CODE | |
| 66 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | docs/releases/lmms-eval-0.4.md | 341 | # "mme": { | COMMENT |
| LOW | examples/test_reasoning_tag_stripping.sh | 21 | # --tasks mme \ | COMMENT |
| LOW | examples/models/plm.sh | 1 | # # STEP # 1: Install lmms-eval | COMMENT |
| LOW | examples/models/llava_onevision2.sh | 1 | export HF_HOME="~/.cache/huggingface" | COMMENT |
| LOW | examples/models/llava_onevision2.sh | 21 | # canvas packing driven by motion vectors / bit-cost. It typically | COMMENT |
| LOW | examples/models/llava_onevision2.sh | 41 | # codec model_args reference: | COMMENT |
| LOW | examples/models/bagel.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/models/vqtoken.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 61 | # ============================================================================ | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 121 | # 1. Create an MCP server that exposes tools (e.g., image processing, web search) | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 141 | # ────────────────────────────────────────────── | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 161 | # | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 181 | # max_frame_num : Max frames for videos (default: 768) | COMMENT |
| LOW | examples/models/sglang_qwen3vl.sh | 201 | # 2. MCP SERVER: | COMMENT |
| LOW | examples/models/async_openai_compatible.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/models/async_openai_compatible.sh | 21 | # ============================================================================ | COMMENT |
| LOW | examples/models/async_openai_compatible.sh | 41 | COMMENT | |
| LOW | examples/models/async_openai_compatible.sh | 61 | # 7. The result is sent back to the model for continuation | COMMENT |
| LOW | examples/models/async_openai_compatible.sh | 81 | # | COMMENT |
| LOW | examples/models/async_openai_compatible.sh | 101 | # max_frames : Max frames for videos (default: 768) | COMMENT |
| LOW | examples/models/openrouter_molmo.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/models/openrouter_molmo.sh | 41 | # 1. OpenRouter free models have rate limits - use small --limit for testing | COMMENT |
| LOW | examples/models/cambrians.sh | 1 | # tested with | COMMENT |
| LOW | examples/models/qwen25vl.sh | 1 | # Run and exactly reproduce qwen2vl results! | COMMENT |
| LOW | examples/models/vllm_qwen3vl.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/models/vllm_qwen3vl.sh | 21 | # Available Qwen3-VL models: | COMMENT |
| LOW | .github/workflows/claude.yml | 41 | actions: read | COMMENT |
| LOW | lmms_eval/tasks/kaleidoscope/utils.py | 181 | # --------------------------------------------------------------------------- | COMMENT |
| LOW | lmms_eval/tasks/saco/utils.py | 281 | # Both have F1 scores — compare mean F1 | COMMENT |
| LOW | …/voicebench/instruction_following_eval/instructions.py | 1 | # coding=utf-8 | COMMENT |
| LOW | …ebench/instruction_following_eval/instructions_util.py | 1 | # coding=utf-8 | COMMENT |
| LOW | …ch/instruction_following_eval/instructions_registry.py | 1 | # coding=utf-8 | COMMENT |
| LOW | lmms_eval/tasks/ifeval/instructions.py | 1 | # Copyright 2023 The Google Research Authors. | COMMENT |
| LOW | lmms_eval/tasks/ifeval/instructions_util.py | 1 | # Copyright 2023 The Google Research Authors. | COMMENT |
| LOW | lmms_eval/tasks/ifeval/instructions_registry.py | 1 | # Copyright 2023 The Google Research Authors. | COMMENT |
| LOW | lmms_eval/tasks/chartqa/upload_chartqa.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor. | COMMENT |
| LOW | lmms_eval/tasks/kris_bench/__init__.py | 1 | # KRIS-Bench task | COMMENT |
| LOW | lmms_eval/tasks/mmmu/mmmu_val_reasoning.yaml | 1 | dataset_path: lmms-lab-encoder/MMMU | COMMENT |
| LOW | lmms_eval/tasks/hallusion_bench/utils.py | 281 | COMMENT | |
| LOW | lmms_eval/tasks/vlmsareblind/__init__.py | 1 | # VLMs Are Blind benchmark task | COMMENT |
| LOW | …atialtreebench/metrics/mindcube_cogmap/parse_output.py | 241 | # else: | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 21 | get_pred_category_type, | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 41 | # # 只处理满足条件的字典 | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 61 | # output.append(item) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 81 | COMMENT | |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 101 | # idx_in_body = body.find(ln, search_from) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 501 | # print(cost_matrix) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 521 | COMMENT | |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 621 | COMMENT | |
| LOW | lmms_eval/tasks/mdpbench/match_quick.py | 641 | # ignore_type = ['figure_caption', 'figure_footnote', 'table_caption', 'table_footnote', 'code_algorithm | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/data_preprocess.py | 61 | # "\u2018": "'", # Left single quote to straight quote | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/data_preprocess.py | 381 | COMMENT | |
| LOW | lmms_eval/tasks/mdpbench/data_preprocess.py | 401 | # # inline_array.append(match.group(0)) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 121 | # markdown_table_content[i, j] = LatexNodes2Text().latex_to_text(content_str) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 161 | # position = [match.start(), match.end()] | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 201 | single_line = re.sub(dollar_pattern, r"\\[\1\\]", single_line) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 221 | # pred_all.append({ | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 261 | # title_matches = title_reg.finditer(content) | COMMENT |
| LOW | lmms_eval/tasks/mdpbench/extract.py | 361 | # return content | COMMENT |
| 45 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tools/regression.py | 1 | # code from https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/regression.py | COMMENT |
| MEDIUM | lmms_eval/evaluator_utils.py | 1 | # credit to https://github.com/EleutherAI/lm-evaluation-harness | COMMENT |
| MEDIUM | lmms_eval/evaluator.py | 473 | # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412 | COMMENT |
| MEDIUM | lmms_eval/logging_utils.py | 1 | # Code mostly from: https://github.com/EleutherAI/lm-evaluation-harness/pull/1339, credit to: https://github.com/ayulock | COMMENT |
| MEDIUM | lmms_eval/tasks/__init__.py | 1 | # credit to https://github.com/EleutherAI/lm-evaluation-harness | COMMENT |
| LOW | lmms_eval/tasks/mmvu/utils.py | 349 | # For open-ended, just use the prediction as-is (truncated for logging) | COMMENT |
| MEDIUM | …s/vbvr/vbvr_bench/evaluators/Out_of_Domain_50_part1.py | 101 | # Sort by area to match shapes (more robust than position) | COMMENT |
| MEDIUM⚡ | lmms_eval/tasks/longvideo_reason/utils.py | 226 | """Headline accuracy over all 1,000 rows, with the robust extractor. | STRING |
| MEDIUM | lmms_eval/tasks/charades_sta/eval_tvg.py | 90 | # More comprehensive pattern to match various formats like: | COMMENT |
| MEDIUM | lmms_eval/tasks/mia_bench/utils.py | 168 | # More robust regex patterns to extract scores | COMMENT |
| MEDIUM | lmms_eval/tasks/realunify/realunify.yaml | 2 | # Evaluates bidirectional capability synergy in unified multimodal models | COMMENT |
| MEDIUM | lmms_eval/tasks/mmbench/mmbench_evals.py | 221 | # Use robust MCQ extraction instead of GPT API | COMMENT |
| LOW | …s/megabench/metrics/scoring/xml_norm_point_distance.py | 36 | # Instead of normalizing by 1/sqrt(2), we just set it to 0 if the distance is above 1. | COMMENT |
| LOW | …s_eval/tasks/megabench/metrics/parsing/common/utils.py | 151 | # the result overflows, simply return an empty string | COMMENT |
| MEDIUM | lmms_eval/tasks/multimodal_rewardbench/utils.py | 34 | # TODO: we should use a more robust method to extract the judgment -- what if both "[[A]]" and "[[B]]" are in the ju | COMMENT |
| MEDIUM | lmms_eval/loggers/evaluation_tracker.py | 1 | # code is adapted from https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lmms_eval/loggers/evaluation_tracke | COMMENT |
| MEDIUM | lmms_eval/models/chat/aero_realtime_vllm.py | 338 | # Accept (and ignore) unknown harness kwargs gracefully — be lenient | COMMENT |
| LOW | lmms_eval/models/simple/internvl.py | 86 | # The above steps can be optional, I add snapshot download, so now can just use hf repo_id | COMMENT |
| LOW | lmms_eval/models/simple/minicpm_v.py | 199 | # ominicpm does not give much information on how they do eval so I just use the chat format. | COMMENT |
| LOW | lmms_eval/models/simple/qwen2_vl.py | 256 | # Update the original contexts list as well if needed elsewhere, otherwise just use current_context | COMMENT |
| MEDIUM | lmms_eval/models/simple/qwen2_vl.py | 264 | # A more robust approach might map visuals back to their original context index. | COMMENT |
| LOW | lmms_eval/models/simple/vllm.py | 173 | # Here we just use the same token as llava for convenient | COMMENT |
| LOW | lmms_eval/models/simple/batch_gpt4.py | 60 | # Here we just use the same token as llava for convenient | COMMENT |
| LOW | lmms_eval/models/simple/srt_api.py | 50 | # Here we just use the same token as llava for convenient | COMMENT |
| MEDIUM | lmms_eval/models/simple/uni_moe_2_omni.py | 45 | SYSTEM_PROMPT = """You are Uni-MoE-2, a helpful multi-modal model. Your role as an assistant involves thoroughly explori | CODE |
| LOW | lmms_eval/models/simple/gpt4v.py | 47 | # Here we just use the same token as llava for convenient | COMMENT |
| LOW | lmms_eval/models/simple/qwen2_audio.py | 226 | # This placeholder is just use to make chat template work | COMMENT |
| MEDIUM | lmms_eval/api/metrics.py | 1 | # the code is adapted from https://github.com/EleutherAI/lm-evaluation-harness | COMMENT |
| MEDIUM | lmms_eval/api/metrics.py | 641 | # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1390 for more documentation. | COMMENT |
| MEDIUM | lmms_eval/api/task.py | 1807 | # built by build_all_requests and the simple adapters (lm-eval-harness | COMMENT |
| LOW | lmms_eval/api/task.py | 1623 | # we just use the argmin here | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | docs/releases/CHANGELOG.md | 14 | - **Agentic task evaluation**: new `generate_until_agentic` output type with iterative tool-call loop, deterministic sim | CODE |
| MEDIUM | docs/releases/CHANGELOG.md | 91 | - **`generate_until_agentic` output type**: iterative evaluator loop where the model emits `<tool_call>` or `<submit>` t | CODE |
| MEDIUM | docs/releases/lmms-eval-0.7.md | 11 | - [4. Agentic Task Evaluation](#4-agentic-task-evaluation) | CODE |
| MEDIUM | cicd/run_evalcicd.sh | 4 | # This allows for better GPU management and test orchestration | COMMENT |
| MEDIUM | lmms_eval/evaluator.py | 691 | # add info about the model and few shot config | COMMENT |
| MEDIUM | lmms_eval/tasks/mmsearch/score/f1_score.py | 33 | # from https://github.com/chanchimin/RQ-RAG/blob/96b4ec981d4a4399e8402da1b75e16f7812aedfe/retrieval_lm/output/sample_fro | COMMENT |
| MEDIUM | lmms_eval/tasks/kaleidoscope/utils.py | 137 | # Zero-shot chain-of-thought system message, translated into every evaluation language. | COMMENT |
| MEDIUM | lmms_eval/tasks/kris_bench/utils.py | 351 | # Clamp score to valid 1-5 range (judge may hallucinate out-of-range values) | COMMENT |
| MEDIUM | lmms_eval/tasks/longvt/README.md | 3 | This directory contains evaluation tasks for [LongVT](https://github.com/EvolvingLMMs-Lab/LongVT), an agentic framework | CODE |
| MEDIUM | lmms_eval/tasks/mmsi_video/mmsi_video_u50.yaml | 4 | # Suitable for models with limited context window. | COMMENT |
| MEDIUM | lmms_eval/tasks/coco_cap_chair/utils.py | 995 | # calculate the number of hallucination | COMMENT |
| MEDIUM | lmms_eval/tasks/coco_cap_chair/utils.py | 1014 | # calculate the number of hallucination | COMMENT |
| MEDIUM | lmms_eval/models/chat/aero_realtime_vllm.py | 559 | # no video item, no extra prompt scaffolding. | COMMENT |
| MEDIUM⚡ | lmms_eval/models/simple/cosmos_wm.py | 1358 | # Public: extract last frame (for agentic use) | COMMENT |
| MEDIUM⚡ | lmms_eval/models/simple/wan2_2.py | 94 | # Standard tasks pass a real doc; agentic rollout rounds 2+ pass a | COMMENT |
| MEDIUM | lmms_eval/api/task.py | 1298 | # if few-shot - append examples after the system prompt | COMMENT |
| MEDIUM | lmms_eval/api/task.py | 1806 | # (description + few-shot examples + current question), matching the ctx | COMMENT |
| MEDIUM | lmms_eval/api/task.py | 1848 | # Reuse the complete few-shot ctx and the existing choice expansion. | COMMENT |
| MEDIUM | lmms_eval/api/samplers.py | 17 | if fewshot_indices: # subset few-shot docs from | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | lmms_eval/tasks/mmsearch/utils/utils.py | 319 | if (k * scrollStep < height && k * scrollStep < maxHeight) {{ | CODE |
| HIGH | lmms_eval/tasks/mmsearch/utils/utils.py | 353 | if (!img.src && img.dataset.src) { | CODE |
| HIGH⚡ | lmms_eval/tasks/gedit_bench/viescore/utils.py | 22 | elif re.match(r"^(true|false|null)$", value, re.IGNORECASE): | CODE |
| HIGH | lmms_eval/tasks/rvos/utils.py | 226 | # File is {"<internal_obj_id>": [rle, rle, ..., null, ...]} | COMMENT |
| HIGH | …tasks/megabench/metrics/scoring/latex_expr_equality.py | 35 | result = 1 if responses_expr.equals(targets_expr) else 0 | CODE |
| HIGH⚡ | lmms_eval/tasks/groundingme/utils.py | 136 | PROMPT = "All spatial relationships are defined from the viewer's perspective, where 'front' means closer to the viewer | CODE |
| HIGH⚡ | lmms_eval/tasks/groundingme/utils.py | 146 | """Extract bounding box from JSON format: {"bbox_2d": [x1, y1, x2, y2]} or {"bbox_2d": null}.""" | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | lmms_eval/imports.py | 43 | Import a module or attribute optionally, returning fallback if unavailable. Args: module_name: Full module | STRING |
| HIGH | lmms_eval/tasks/mathvision/eval_utils.py | 131 | Determine if a given ID falls within a specified area. This function checks if a provided ID contains the specified | STRING |
| HIGH | lmms_eval/tasks/gedit_bench/viescore/utils.py | 76 | Search for a file across a directory and return its absolute path. Args: filename (Union[str, os.PathL | STRING |
| HIGH | …ch/metrics/mindcube_cogmap/src/evaluation/evaluator.py | 119 | Unified evaluation interface with automatic mode selection. Args: jsonl_path: Path to JSONL file with model | STRING |
| HIGH | lmms_eval/models/chat/thyme.py | 37 | Context manager that extracts the user's image and saves it to a temporary file if needed. Args: messa | STRING |
| HIGH | lmms_eval/models/simple/gemma3.py | 351 | Generate text in a multi-round conversation format. Args: requests: List of Instance objects for mu | STRING |
| HIGH | lmms_eval/entrypoints/client.py | 228 | Wait for a job to complete and return results. Args: job_id: The job ID to wait for | STRING |
| HIGH | lmms_eval/entrypoints/client.py | 276 | Merge FSDP2 sharded checkpoint into a single consolidated checkpoint. Args: checkpoint_pat | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | lmms_eval/tasks/voicebench/utils.py | 232 | "As a language model", | CODE |
| MEDIUM⚡ | lmms_eval/tasks/voicebench/utils.py | 238 | "I'm unable to", | CODE |
| MEDIUM | lmms_eval/tasks/plm_videobench/eval_utils.py | 177 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. | CODE |
| MEDIUM | lmms_eval/tasks/moviechat/utils.py | 105 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningful | CODE |
| MEDIUM | lmms_eval/tasks/cvrr/utils.py | 106 | "Your task is to compare the predicted answer with the ground-truth answer and determine if the predicted an | CODE |
| MEDIUM | lmms_eval/tasks/video-tt/gpt_utils.py | 47 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. | CODE |
| MEDIUM | lmms_eval/tasks/activitynetqa/utils.py | 92 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningful | CODE |
| MEDIUM⚡ | lmms_eval/tasks/llava_interleave_bench/utils.py | 46 | # Optionally, you can add a placeholder image or just continue | COMMENT |
| MEDIUM | lmms_eval/tasks/vdc/utils.py | 197 | "Here's how you can accomplish the task:" | CODE |
| MEDIUM | lmms_eval/tasks/vdc/utils.py | 224 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. | CODE |
| MEDIUM | lmms_eval/tasks/FALCONBench/utils.py | 368 | "Your task is to compare the predicted answer with the correct answer and determine if they match meaningful | STRING |
| MEDIUM | lmms_eval/tasks/video_detail_description/utils.py | 90 | "Your task is to compare the predicted answer with the correct answer and determine its level of detail, con | CODE |
| MEDIUM | lmms_eval/tasks/videochatgpt/utils.py | 222 | "Your task is to compare the predicted answer with the correct answer and determine if they are factuall | CODE |
| MEDIUM | lmms_eval/tasks/videochatgpt/utils.py | 247 | "Your task is to compare the predicted answer with the correct answer and determine its level of detail, | CODE |
| MEDIUM | lmms_eval/tasks/videochatgpt/utils.py | 272 | "Your task is to compare the predicted answer with the correct answer and determine if the generated res | CODE |
| MEDIUM | lmms_eval/tasks/videochatgpt/utils.py | 297 | "Your task is to compare the predicted answer with the correct answer and determine if they correctly re | CODE |
| MEDIUM | lmms_eval/tasks/videochatgpt/utils.py | 363 | "Your task is to compare the predicted answers for two very similar question, with a common correct answer a | CODE |
| MEDIUM | lmms_eval/tasks/holisafe/utils.py | 110 | "i cannot provide", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | docs/advanced/mmmu-eval-discrepancy.md | 169 | # Step 1: Strip punctuation, pad with spaces | COMMENT |
| LOW⚡ | docs/advanced/mmmu-eval-discrepancy.md | 174 | # Step 2: Hierarchical pattern matching | COMMENT |
| LOW⚡ | docs/advanced/mmmu-eval-discrepancy.md | 219 | # Step 1: Replace punctuation with spaces | COMMENT |
| LOW⚡ | docs/advanced/mmmu-eval-discrepancy.md | 225 | # Step 2: Count how many choice letters appear | COMMENT |
| LOW⚡ | docs/advanced/mmmu-eval-discrepancy.md | 228 | # Step 3: If exactly one choice found, verify it's near the end | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 467 | # Step 2: Detect which lights are ON in GT final (expected states) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 470 | # Step 3: Detect which lights are ON in generated final | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 473 | # Step 4: Compare states - STRICT rule-based | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1139 | # Step 1: Detect elements in GT first (the pattern) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1142 | # Step 2: Detect elements in GT final (includes correct answer) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1145 | # Step 3: Detect elements in generated final | COMMENT |
| LOW | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 454 | # Step 1: Find all light positions from GT first frame | COMMENT |
| LOW | lmms_eval/tasks/longvt/README.md | 65 | #### Step 1: Start MCP Server | COMMENT |
| LOW | lmms_eval/tasks/longvt/README.md | 86 | #### Step 2: Run Evaluation | COMMENT |
| LOW | lmms_eval/tasks/phyx/phyx_evals.py | 142 | Predicted answer: ### Step 1: Calculate ( l_1 ) | CODE |
| LOW | lmms_eval/models/simple/ovis_u1.py | 400 | # Step 1: Unconditional baseline | COMMENT |
| LOW | lmms_eval/models/simple/ovis_u1.py | 412 | # Step 2: Conditional with original image | COMMENT |
| LOW | lmms_eval/models/simple/ovis_u1.py | 440 | # Step 3: Generate the image | COMMENT |
| LOW | lmms_eval/cli/wizard.py | 251 | # Step 4: confirm | COMMENT |
| LOW⚡ | lmms_eval/entrypoints/http_server.py | 33 | # WARNING: This server is intended for use in trusted environments only. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/prepare_physics_benchmarks.py | 540 | def _update_card(api: HfApi, dataset_name: str, target_repo: str) -> None: | CODE |
| LOW | examples/mcp_server/crop_video_mcp_server.py | 21 | logger = logging.getLogger(__name__) | CODE |
| LOW | …s/mmsearch/retrieve_content/tokenization/tokenizers.py | 27 | def update_lexeme(self, lex: str) -> None: | CODE |
| LOW | lmms_eval/tasks/cc_ocr/evaluators/__init__.py | 3 | __all__ = ["ocr_metric", "kie_metric", "doc_parsing_metric"] | CODE |
| LOW | lmms_eval/tasks/ifeval/instructions.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | lmms_eval/tasks/vbvr/vbvr_bench/evaluators/__init__.py | 640 | __all__ = [ | CODE |
| LOW | …reebench/metrics/mindcube_cogmap/src/utils/__init__.py | 14 | __all__ = [ | CODE |
| LOW | …nch/metrics/mindcube_cogmap/src/evaluation/__init__.py | 36 | __all__ = [ | CODE |
| LOW | …rics/mindcube_cogmap/src/evaluation/cogmap/__init__.py | 18 | __all__ = [ | CODE |
| LOW | …etrics/mindcube_cogmap/src/evaluation/core/__init__.py | 13 | __all__ = [ | CODE |
| LOW | lmms_eval/tasks/worldvqa/utils.py | 85 | __all__ = [ | CODE |
| LOW | lmms_eval/tasks/megabench/metrics/__init__.py | 5 | __all__ = [AggregationType, MetricType, ResponseParseType] | CODE |
| LOW | lmms_eval/tui/__init__.py | 15 | __all__ = ["run_tui", "server_app"] | CODE |
| LOW | lmms_eval/verifiers/__init__.py | 137 | __all__ = [ | CODE |
| LOW | lmms_eval/verifiers/gemini.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | lmms_eval/verifiers/openai.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | lmms_eval/mcp/__init__.py | 10 | __all__ = ["MCPClient", "main"] | CODE |
| LOW | lmms_eval/models/model_utils/usage_metrics.py | 52 | def set_budget(max_tokens: Optional[int] = None) -> None: | CODE |
| LOW | lmms_eval/models/model_utils/usage_metrics.py | 65 | def set_task_context(task_name: Optional[str]) -> None: | CODE |
| LOW⚡ | lmms_eval/models/simple/vllm.py | 265 | def _setup_tp_group_for_request_sync(self) -> None: | CODE |
| LOW | lmms_eval/cli/__init__.py | 5 | __all__ = ["main"] | CODE |
| LOW | lmms_eval/entrypoints/__init__.py | 16 | __all__ = [ | CODE |
| LOW | lmms_eval/api/task.py | 706 | def set_config(self, key: str, value: Any, update: bool = False) -> None: | CODE |
| LOW | lmms_eval/api/task.py | 742 | def set_fewshot_seed(self, seed: Optional[int] = None) -> None: | CODE |
| LOW | lmms_eval/api/model.py | 161 | def set_cache_hook(self, cache_hook) -> None: | CODE |
| LOW | lmms_eval/baselines/__init__.py | 52 | __all__ = ["BASELINE_REGISTRY", "get_baseline_display_name", "load_baseline"] | CODE |
| LOW | lmms_eval/llm_judge/__init__.py | 21 | __all__ = [ | CODE |
| LOW | lmms_eval/llm_judge/launcher/__init__.py | 33 | __all__ = [ | CODE |
| LOW | lmms_eval/llm_judge/providers/__init__.py | 9 | __all__ = [ | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 467 | # Step 2: Detect which lights are ON in GT final (expected states) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 470 | # Step 3: Detect which lights are ON in generated final | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 473 | # Step 4: Compare states - STRICT rule-based | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1139 | # Step 1: Detect elements in GT first (the pattern) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1142 | # Step 2: Detect elements in GT final (includes correct answer) | COMMENT |
| LOW⚡ | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 1145 | # Step 3: Detect elements in generated final | COMMENT |
| LOW | …tasks/vbvr/vbvr_bench/evaluators/In_Domain_50_part5.py | 454 | # Step 1: Find all light positions from GT first frame | COMMENT |
| LOW | lmms_eval/tasks/phyx/phyx_evals.py | 142 | Predicted answer: ### Step 1: Calculate ( l_1 ) | CODE |
| LOW | lmms_eval/models/simple/ovis_u1.py | 400 | # Step 1: Unconditional baseline | COMMENT |
| LOW | lmms_eval/models/simple/ovis_u1.py | 412 | # Step 2: Conditional with original image | COMMENT |
| LOW | lmms_eval/models/simple/ovis_u1.py | 440 | # Step 3: Generate the image | COMMENT |
| LOW | lmms_eval/cli/wizard.py | 251 | # Step 4: confirm | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | lmms_eval/tasks/moviechat/README.md | 5 | This repository contains an evaluation dataset designed for assessing the long video understanding performance of video | CODE |
| HIGH | lmms_eval/tasks/video_detail_description/README.md | 5 | This repository contains an evaluation dataset designed for assessing the performance of video models. The dataset inclu | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/regression.py | 34 | # TODO: implement num_fewshot and limit per task, e.g. task1:5,task2:1:100,task3::1000 | COMMENT |
| LOW | tools/regression.py | 37 | # TODO: implement hf-auto to pick between causal and seq2seq models so we don't need this | COMMENT |
| LOW | tools/regression.py | 139 | # TODO: implement proper timing for each task | COMMENT |
| LOW | lmms_eval/filters/__init__.py | 17 | # TODO: implement this filter. either it should take in an arbitrary "scoring"/reward function | COMMENT |
| LOW | lmms_eval/filters/selection.py | 6 | # TODO: implement "arg_max" filter. either it should take in an arbitrary "scoring"/reward function | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | examples/models/bagel.sh | 16 | # Usage: | COMMENT |
| LOW | lmms_eval/tasks/gedit_bench/viescore/parse_prompt.py | 17 | # Example usage | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | lmms_eval/tasks/ocrbench_v2/TEDS_metric.py | 317 | def process_data(obj): | CODE |