Inspect: A framework for large language model evaluations
This report summarizes scanner evidence for UKGovernmentBEIS/inspect_ai, a Python project with 2,983 GitHub stars. SynthScan examined 710,576 lines of code across 2094 source files, recording 18058 pattern matches distributed across 23 syntactic categories. The legacy adjusted pattern signal is 31.9 (High pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 13966 finding(s).
Are imports declared and package/API names verified? 1730 finding(s).
Are wrappers, guards, and repeated structures necessary? 14938 finding(s).
Which changes need focused human review? 18058 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 18058 distinct pattern matches across 23 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | design/ctl/config-log-persistence.md | 161 | def effective_generate_config(log: EvalLog) -> GenerateConfig: ... | CODE |
| LOW | tests/test_task_cancel.py | 34 | def test_abort_cancel_produces_error_status() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 93 | def test_abort_cancel_not_retried_without_task_retries() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 166 | def test_score_resolution_cancel_completes_eval() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 223 | def test_error_resolution_cancel_completes_eval() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 265 | def test_error_resolution_rejected_when_samples_fail_on_error() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 274 | def error_resolution_rejected_solver(): | CODE |
| LOW | tests/test_task_cancel.py | 303 | def test_error_resolution_downgraded_for_materializing_fail_on_error_sample() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 329 | def downgrade_resolution_solver(): | CODE |
| LOW | tests/test_task_cancel.py | 366 | def test_drain_resolution_cancels_materializing_sample() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 446 | def test_sample_cancelled_interrupt_action() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 455 | def cancelled_interrupt_solver(): | CODE |
| LOW | tests/test_task_cancel.py | 493 | def test_score_resolution_sweep_preserves_cancelled_sample() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 506 | def cancelled_then_score_solver(): | CODE |
| LOW | tests/test_task_cancel.py | 543 | def test_interrupt_in_retry_drain_window_resolves_cancelled() -> None: | CODE |
| LOW | tests/test_task_cancel.py | 571 | def drain_window_error_solver(): | CODE |
| LOW | tests/test_task_cancel.py | 642 | def test_external_interrupt_with_pending_resolution_logs_cancelled( | CODE |
| LOW | tests/test_task_cancel.py | 828 | def test_sample_cancel_while_initializing_resolves_at_start( | CODE |
| LOW | tests/test_task_cancel.py | 909 | def test_sample_cancel_while_initializing_wins_over_later_task_score( | CODE |
| LOW | tests/test_task_cancel.py | 958 | def test_sample_cancel_while_initializing_then_init_failure_abandons( | CODE |
| LOW | tests/test_task_cancel.py | 1036 | def test_errored_attempt_marked_retry_pending() -> None: | CODE |
| LOW | tests/test_eval_set_capture.py | 68 | def test_eval_set_capture_manifest( | CODE |
| LOW | tests/test_eval_set_capture.py | 108 | def test_eval_set_capture_limit_and_epochs( | CODE |
| LOW | tests/test_eval_set_capture.py | 126 | def test_eval_set_capture_max_samples( | CODE |
| LOW | tests/test_eval_set_capture.py | 137 | def test_eval_set_capture_model_roles( | CODE |
| LOW | tests/test_eval_set_capture.py | 148 | def test_eval_set_capture_model_roles_list( | CODE |
| LOW | tests/test_eval_set_capture.py | 162 | def test_eval_set_capture_adhoc_task( | CODE |
| LOW | tests/test_eval_set_capture.py | 191 | def test_eval_set_capture_task_source_error( | CODE |
| LOW | tests/test_eval_set_capture.py | 204 | def test_eval_set_capture_subprocess_protocol(tmp_path: Path) -> None: | CODE |
| LOW | tests/test_eval_set_capture.py | 347 | def test_eval_set_capture_schema_stability() -> None: | CODE |
| LOW⚡ | tests/test_sample_limits.py | 141 | def test_message_limit_generate(): | CODE |
| LOW⚡ | tests/test_sample_limits.py | 145 | def test_message_limit_append(): | CODE |
| LOW⚡ | tests/test_sample_limits.py | 149 | def test_message_limit_overwrite(): | CODE |
| LOW⚡ | tests/test_sample_limits.py | 153 | def test_message_limit_reached_before_assistant_message(): | CODE |
| LOW | tests/test_sample_limits.py | 70 | def looping_concurrecy_solver(): | CODE |
| LOW | tests/test_sample_limits.py | 169 | def test_message_limit_does_not_apply_to_scorer(): | CODE |
| LOW | tests/test_sample_limits.py | 218 | def test_token_limit_reached_exactly_refuses_next_generate(): | CODE |
| LOW | tests/test_sample_limits.py | 239 | def test_token_limit_does_not_apply_to_scorer(): | CODE |
| LOW | tests/test_sample_limits.py | 285 | def test_output_token_limit_string_form(): | CODE |
| LOW | tests/test_sample_limits.py | 320 | def test_token_limit_type_absent_for_int_limit(): | CODE |
| LOW | tests/test_sample_limits.py | 367 | def test_turn_limit_does_not_apply_to_scorer(): | CODE |
| LOW | tests/test_sample_limits.py | 415 | def test_sample_limits_available_to_scorer(): | CODE |
| LOW | tests/test_sample_limits.py | 446 | def test_solver_scorer_combined_timeout(): | CODE |
| LOW | tests/test_sample_limits.py | 455 | def test_solver_scorer_combined_timeout_exceeded(): | CODE |
| LOW | tests/test_sample_limits.py | 466 | def test_solver_timeout_scored(): | CODE |
| LOW | tests/test_sample_limits.py | 475 | def test_solver_timeout_not_scored(): | CODE |
| LOW | tests/test_sample_limits.py | 494 | def test_working_limit_reporting(): | CODE |
| LOW | tests/test_sample_limits.py | 511 | def test_working_limit_does_not_raise_during_sandbox_teardown() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 526 | def check_working_limit_event(log: EvalLog, working_limit: int): | CODE |
| LOW | tests/test_sample_limits.py | 580 | def test_cost_limit_without_cost_data_errors() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 592 | def test_model_without_cost_data_errors() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 609 | def test_cost_data_without_cost_limit_tracks_cost() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 640 | def test_cost_data_keyed_by_full_model_string_tracks_cost() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 677 | def test_two_models_both_with_cost_data_tracks_cost() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 728 | def test_task_level_cost_limit_without_cost_data_errors() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 740 | def test_task_level_cost_limit() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 771 | def test_model_cost_config_file() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 803 | def test_model_cost_config_dict() -> None: | CODE |
| LOW | tests/test_sample_limits.py | 831 | def test_operator_limit_records_reason() -> None: | CODE |
| LOW | tests/test_retry.py | 67 | def test_eval_retry_with_task_name(): | CODE |
| 12440 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | design/s3-incremental-log-flush.md | 762 | #480 (independent) ──────────────────────────────────────┐ | COMMENT |
| MEDIUM | tests/test_task_cancel.py | 721 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/test_task_cancel.py | 724 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/conftest.py | 35 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/conftest.py | 49 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/conftest.py | 370 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/conftest.py | 388 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 138 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 140 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 312 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 314 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 525 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 527 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 601 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 603 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 696 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 698 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 826 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_release_workflow.py | 828 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_ctl.py | 8989 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_ctl.py | 8991 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 102 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 104 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 213 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 215 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 276 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 278 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 437 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 439 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 579 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 581 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 950 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 952 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1028 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1030 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1356 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1358 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1395 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_pause.py | 1397 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 31 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 33 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 132 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 134 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 217 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_buffer.py | 219 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 381 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 383 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 462 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 464 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 1189 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_score.py | 1191 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 193 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 195 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 1342 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 1344 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 1712 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/_control/test_score.py | 1714 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_server.py | 1756 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_server.py | 1758 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/_control/test_cancel.py | 134 | # --------------------------------------------------------------------------- | COMMENT |
| 1255 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/test_task_cancel.py | 708 | except KeyboardInterrupt: | CODE |
| HIGH | tests/conftest.py | 1113 | except BaseException: | CODE |
| HIGH | tests/test_eval_set.py | 205 | except Exception: | CODE |
| HIGH | tests/test_eval_set.py | 3164 | except KeyboardInterrupt: | CODE |
| HIGH | tests/test_eval_set.py | 874 | except KeyboardInterrupt: | CODE |
| HIGH | tests/test_cancellation_logging.py | 289 | except KeyboardInterrupt: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 101 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 279 | except (ValueError, PrerequisiteError): | CODE |
| HIGH | tests/test_model_env_mismatch.py | 50 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 136 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 168 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 194 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 221 | except Exception: | CODE |
| HIGH | tests/test_model_env_mismatch.py | 247 | except Exception: | CODE |
| HIGH | tests/tools/test_tools_bridge.py | 879 | except Exception: | CODE |
| HIGH | tests/tools/test_tools_bridge.py | 1198 | except Exception: | CODE |
| HIGH | tests/tools/test_mcp_tools.py | 180 | except anyio.ClosedResourceError: | CODE |
| HIGH | tests/tools/test_tool_with.py | 34 | except Exception: | CODE |
| HIGH | tests/util/test_media_resolver.py | 57 | except ValueError: | CODE |
| HIGH | tests/util/test_asyncfiles.py | 391 | except EndOfStream: | CODE |
| HIGH | tests/util/test_asyncfiles.py | 417 | except EndOfStream: | CODE |
| HIGH | tests/input/test_input_acp.py | 104 | except ValueError: | CODE |
| HIGH | tests/agent/deepagent/test_agent_tool.py | 266 | except Exception: | CODE |
| HIGH | tests/agent/test_channel/test_channel.py | 202 | except BaseException: | CODE |
| HIGH | tests/agent/test_acp/test_span_boundary.py | 184 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_server_dispatch.py | 71 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_dispatch.py | 221 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_server_dispatch.py | 226 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_server_dispatch.py | 64 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 775 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 779 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 89 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 203 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 208 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 82 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 843 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_standard_acp_client.py | 81 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_standard_acp_client.py | 192 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_standard_acp_client.py | 197 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_standard_acp_client.py | 74 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_raw_events.py | 87 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_raw_events.py | 174 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_raw_events.py | 179 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_raw_events.py | 80 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_raw_events.py | 351 | except asyncio.TimeoutError: | CODE |
| HIGH | tests/agent/test_acp/test_action_methods.py | 555 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_action_methods.py | 610 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_action_methods.py | 615 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_action_methods.py | 548 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_discovery.py | 72 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_discovery.py | 68 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_elicitation_e2e.py | 87 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_elicitation_e2e.py | 163 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_elicitation_e2e.py | 168 | except Exception: | CODE |
| HIGH | tests/agent/test_acp/test_elicitation_e2e.py | 83 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_stdio_bridge.py | 69 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_stdio_bridge.py | 65 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_session_info.py | 59 | except OSError: | CODE |
| HIGH | tests/agent/test_acp/test_server_session_info.py | 153 | except (asyncio.CancelledError, Exception): | CODE |
| HIGH | tests/agent/test_acp/test_server_session_info.py | 158 | except Exception: | CODE |
| 252 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/conftest.py | 30 | CODE | |
| LOW | tests/test_release_workflow.py | 10 | CODE | |
| LOW | tests/test_eval_set_scanner.py | 3166 | CODE | |
| LOW | tests/test_disk_sample_store.py | 23 | CODE | |
| LOW | tests/test_disk_sample_store.py | 24 | CODE | |
| LOW | tests/_control/test_state.py | 20 | CODE | |
| LOW | tests/_control/test_cancel.py | 22 | CODE | |
| LOW | tests/tools/test_mcp_session_isolation.py | 14 | CODE | |
| LOW | tests/tools/test_sandbox_docker_and_local.py | 20 | CODE | |
| LOW | …ools/sandbox_tools_utils/test_sandbox_tools_digests.py | 9 | CODE | |
| LOW | tests/checkpoint/test_eval_checkpoints.py | 3 | CODE | |
| LOW | tests/checkpoint/test_sample_checkpoints.py | 3 | CODE | |
| LOW | tests/checkpoint/test_resume_copy.py | 11 | CODE | |
| LOW | tests/checkpoint/test_resolve.py | 3 | CODE | |
| LOW | tests/checkpoint/docker_projects.py | 3 | CODE | |
| LOW | tests/checkpoint/conftest.py | 11 | CODE | |
| LOW | tests/checkpoint/hydrate_interrupt_harness.py | 27 | CODE | |
| LOW | tests/checkpoint/test_checkpoint_scoring_resume_e2e.py | 23 | CODE | |
| LOW | tests/checkpoint/resume_kill_thinking_harness.py | 35 | CODE | |
| LOW | tests/checkpoint/test_parse.py | 3 | CODE | |
| LOW | tests/checkpoint/test_staging_dir.py | 3 | CODE | |
| LOW | tests/checkpoint/test_snapshot_strategy.py | 15 | CODE | |
| LOW | tests/checkpoint/test_sandbox_dir.py | 12 | CODE | |
| LOW | tests/checkpoint/test_current_checkpointer_post_exit.py | 31 | CODE | |
| LOW | tests/checkpoint/test_sandbox_restic_repo.py | 20 | CODE | |
| LOW | tests/checkpoint/test_sample_runtime.py | 3 | CODE | |
| LOW | tests/checkpoint/test_layout_paths.py | 3 | CODE | |
| LOW | tests/checkpoint/test_host_egress.py | 9 | CODE | |
| LOW | tests/checkpoint/test_checkpoint_e2e.py | 40 | CODE | |
| LOW | tests/checkpoint/test_checkpointer.py | 10 | CODE | |
| LOW | tests/checkpoint/test_fs_copy_s3.py | 12 | CODE | |
| LOW | tests/checkpoint/resume_scoring_kill_harness.py | 24 | CODE | |
| LOW | tests/checkpoint/test_host_egress_s3.py | 9 | CODE | |
| LOW | tests/checkpoint/resume_kill_harness.py | 21 | CODE | |
| LOW | tests/checkpoint/test_sandbox_egress_restic.py | 24 | CODE | |
| LOW | tests/checkpoint/test_restore_repo.py | 11 | CODE | |
| LOW | tests/checkpoint/test_schemas.py | 9 | CODE | |
| LOW | tests/checkpoint/test_sandbox_egress_extract.py | 14 | CODE | |
| LOW | tests/checkpoint/test_sandbox_paths.py | 3 | CODE | |
| LOW | tests/checkpoint/test_normalize.py | 10 | CODE | |
| LOW | tests/checkpoint/test_restore_scope.py | 9 | CODE | |
| LOW | tests/util/test_restic_ops.py | 15 | CODE | |
| LOW | tests/util/test_zipfile_multiframe.py | 8 | CODE | |
| LOW | tests/util/test_download.py | 3 | CODE | |
| LOW | tests/util/test_restic_summary.py | 9 | CODE | |
| LOW | tests/util/test_store_from_events.py | 7 | CODE | |
| LOW | tests/util/test_async_zip.py | 7 | CODE | |
| LOW | tests/util/test_async_zip.py | 22 | CODE | |
| LOW | tests/util/test_limit_working.py | 1 | CODE | |
| LOW | tests/util/test_compression_transcoding.py | 3 | CODE | |
| LOW | tests/util/test_restic_binary.py | 19 | CODE | |
| LOW | tests/input/test_input_acp.py | 11 | CODE | |
| LOW | tests/input/test_input_panel.py | 11 | CODE | |
| LOW | tests/agent/test_bridge_additional_tools.py | 16 | CODE | |
| LOW | tests/agent/test_bridge_agent_message.py | 25 | CODE | |
| LOW | tests/agent/test_bridge_google_messages.py | 18 | CODE | |
| LOW | tests/agent/test_bridge_function_call_id.py | 11 | CODE | |
| LOW | tests/agent/test_bridge_tool_search.py | 9 | CODE | |
| LOW | tests/agent/test_bridge_provider_errors.py | 10 | CODE | |
| LOW | tests/agent/test_react_checkpoint_resume.py | 1 | CODE | |
| 1509 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH⚡ | tests/test_eval_set_overrides.py | 659 | """`model_dump_json()` writes every unset field as null, and that has to be readable. | STRING |
| HIGH | tests/test_eval_set_overrides.py | 683 | Read as an instruction, `{"max_connections": null}` replaced a definition's | STRING |
| HIGH | tests/test_docker_workflow.py | 44 | 'inspect-tool-support && touch "$CANARY"', | CODE |
| HIGH⚡ | tests/_control/test_ctl.py | 662 | # a running sample with nothing pending (activity null) keeps the | COMMENT |
| HIGH | tests/_control/test_log_dir.py | 301 | # live-only state is null, not a false zero | COMMENT |
| HIGH | tests/tools/sandbox_tools_utils/test_sandbox_tools.py | 800 | "input": f"echo {flag} > /flag && chmod 400 /flag; ls -al /flag && cat /flag", | CODE |
| HIGH | tests/tools/sandbox_tools_utils/test_sandbox_tools.py | 904 | f"touch {SANDBOX_TOOLS_DIR}/planted && echo CREATED; " | CODE |
| HIGH | tests/tools/sandbox_tools_utils/test_sandbox_tools.py | 905 | f"mv {SANDBOX_TOOLS_DIR} {SANDBOX_TOOLS_DIR}.moved && echo MOVED; " | CODE |
| HIGH | tests/tools/sandbox_tools_utils/test_sandbox_tools.py | 997 | "input": "command -v gzip || echo no-gzip; echo probe-done", | CODE |
| HIGH | …ls/sandbox_tools_utils/test_sandbox_tools_injection.py | 152 | TAR_XZF_STDIN = "tar xzf - || { cat >/dev/null; exit 1; }" | CODE |
| HIGH | …ls/sandbox_tools_utils/test_sandbox_tools_injection.py | 153 | TAR_XF_STDIN = "tar xf - || { cat >/dev/null; exit 1; }" | CODE |
| HIGH⚡ | tests/util/test_store_from_events.py | 516 | """Add operation should include value even when it's None (explicit null).""" | STRING |
| HIGH | tests/util/sandbox/test_privileged.py | 282 | 'mkdir -p -- "$1" && tar --version >/dev/null && sh -c \'printf %s "$PATH"\'', | CODE |
| HIGH | tests/util/sandbox/test_privileged.py | 593 | 'tar --version >/dev/null && rm -rf /root/pinned-check && printf %s "$PATH"', | CODE |
| HIGH | tests/util/sandbox/test_privileged.py | 607 | ["sh", "-c", "tar --version >/dev/null && cat > ./ok"], | CODE |
| HIGH⚡ | tests/util/sandbox/test_framework_directory.py | 732 | local, str(target), ["sh", "-c", "pwd -P && touch marker"], user=None | CODE |
| HIGH | tests/util/sandbox/test_framework_directory.py | 762 | 'mv -- "$1" "$2" && ln -s -- "$3" "$1" && touch ./marker && pwd -P', | CODE |
| HIGH | tests/util/sandbox/test_sandbox_service.py | 1139 | _DECOY = "mkdir -p /tmp/decoy && chmod 777 /tmp/decoy && touch /tmp/decoy/sentinel" | CODE |
| HIGH | tests/util/sandbox/test_sandbox_service.py | 1233 | f"mkdir -m 700 {svc}/responses && echo stale > {svc}/responses/stale.json", | CODE |
| HIGH | tests/agent/test_bridge_function_call_id.py | 21 | """Each function_call item gets a non-null, unique id; call_id round-trips.""" | STRING |
| HIGH⚡ | tests/agent/test_agent_human.py | 985 | "touch /root/target && chmod 644 /root/target && " | CODE |
| HIGH⚡ | tests/agent/test_agent_human.py | 986 | "rm /home/nonroot/.bashrc && ln -s /root/target /home/nonroot/.bashrc", | CODE |
| HIGH | tests/agent/test_agent_human.py | 913 | docker_sandbox, "cat /root/.bashrc 2>/dev/null || true" | CODE |
| HIGH | tests/agent/test_agent_human.py | 952 | f"mkdir -p /home/nonroot/decoy && ln -s /home/nonroot/decoy {HUMAN_AGENT_DIR}", | CODE |
| HIGH | tests/agent/test_agent_human.py | 968 | f"ls -A {HUMAN_AGENT_DIR} /home/nonroot/decoy 2>/dev/null || true", | CODE |
| HIGH | tests/scorer/test_metric.py | 210 | Serialized as JSON constants (NaN/Infinity) rather than null, via both | STRING |
| HIGH⚡ | tests/scorer/test_reducers.py | 755 | # With fewer than k scored epochs the estimator is undefined, so the | COMMENT |
| HIGH⚡ | tests/scorer/test_math.py | 630 | assert plain.equals(latex) | CODE |
| HIGH | tests/_cli/test_util.py | 71 | "critic={model: mockllm/model, temperature: 0.2, model_args: {custom_outputs: null}}", | CODE |
| HIGH | tests/_cli/test_util.py | 223 | "{model: mockllm/model, model_args: {custom_outputs: null}, " | CODE |
| HIGH | tests/model/providers/test_anthropic.py | 3200 | 'head -c 9 /dev/urandom | base64 > /tmp/seed && echo "seed-$(cat /tmp/seed)"' | CODE |
| HIGH | tests/model/providers/test_anthropic.py | 3202 | pending_cmd = 'sleep 5 && echo "marker-$(cat /tmp/seed)"' | CODE |
| HIGH | tests/model/providers/test_bedrock_structured_output.py | 38 | nickname: Optional[str] = None # renders as anyOf:[string, null] | CODE |
| HIGH | …ts/model/providers/test_anthropic_cache_diagnostics.py | 364 | """`{cache_miss_reason: null}` means comparison still pending — no warning.""" | STRING |
| HIGH | tests/model/providers/test_litellm_proxy.py | 1897 | '"invalid_request_error", "param": null, "code": "invalid_prompt"}}\n' | CODE |
| HIGH | tests/model/providers/test_openai_responses.py | 360 | Meta's Model API streams a refusal-only response whose `model` is null; | STRING |
| HIGH⚡ | tests/log/test_message_pool.py | 1052 | Regression: if persistence stores call_refs as [] instead of null, | STRING |
| HIGH | tests/log/test_eval_log.py | 1640 | null_value = re.compile(r'"value":\s*(null|\[\s*null|\{"a":\s*null)') | CODE |
| HIGH | tests/_eval/task/test_hf.py | 382 | # hard-fails under the C locale (ascii) and cp1252 (0x81 undefined); | COMMENT |
| HIGH⚡ | …/_remote_tools/_web_browser/playwright_page_crawler.py | 231 | "(document.scrollingElement || document.body).scrollTop =" | CODE |
| HIGH⚡ | …/_remote_tools/_web_browser/playwright_page_crawler.py | 232 | " (document.scrollingElement || document.body).scrollTop -" | CODE |
| HIGH⚡ | …/_remote_tools/_web_browser/playwright_page_crawler.py | 237 | "(document.scrollingElement || document.body).scrollTop =" | CODE |
| HIGH⚡ | …/_remote_tools/_web_browser/playwright_page_crawler.py | 238 | " (document.scrollingElement || document.body).scrollTop +" | CODE |
| HIGH⚡ | …sandbox_tools/tests/test_text_editor_validated_path.py | 329 | b'{"/x": [null]}', | CODE |
| HIGH | src/inspect_sandbox_tools/tests/test_exec_remote.py | 292 | "command": "echo hello_from_kill && echo err_from_kill >&2 && sleep 100" | CODE |
| HIGH | src/inspect_sandbox_tools/tests/test_exec_remote.py | 412 | "params": {"command": "echo stdout_msg && echo stderr_msg >&2"}, | CODE |
| HIGH | …_sandbox_tools/tests/test_mcp_server_session_reader.py | 31 | b'{"jsonrpc":"2.0","id":null,' | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 10 | sd.push("A") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 19 | sd.push("A") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 25 | sd.push("B") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 34 | sd.push("A") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 39 | sd.push("B") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 48 | sd.push("A") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 51 | sd.push("B") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 54 | sd.push("C") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 63 | sd.push("data") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 66 | sd.push("") # retransmit, no new data | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 76 | sd.push("A") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 80 | sd.push("B") | CODE |
| HIGH⚡ | …inspect_sandbox_tools/tests/test_acked_chunk_buffer.py | 90 | sd.push(("out1", "err1")) | CODE |
| 55 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/conftest.py | 529 | CODE | |
| LOW | tests/test_eval_set_scanner.py | 1104 | CODE | |
| LOW | tests/_control/test_pause.py | 1400 | CODE | |
| LOW | tests/_control/test_pause.py | 1430 | CODE | |
| LOW | tests/_control/test_pause.py | 1431 | CODE | |
| LOW | tests/_control/test_limits.py | 1257 | CODE | |
| LOW | tests/_control/test_requeue.py | 1621 | CODE | |
| LOW | tests/tools/test_mcp_tools.py | 87 | CODE | |
| LOW | tests/tools/test_mcp_tools.py | 646 | CODE | |
| LOW | tests/tools/test_mcp_tools.py | 128 | CODE | |
| LOW | tests/tools/test_mcp_tools.py | 137 | CODE | |
| LOW | tests/tools/test_skill.py | 559 | CODE | |
| LOW | tests/checkpoint/test_checkpoint_e2e.py | 179 | CODE | |
| LOW | tests/util/sandbox/test_docker_cleanup.py | 272 | CODE | |
| LOW | tests/util/sandbox/test_local_sandbox.py | 29 | CODE | |
| LOW | tests/agent/test_agent_compaction.py | 208 | CODE | |
| LOW | tests/agent/test_agent_react.py | 153 | CODE | |
| LOW | tests/agent/test_agent_react.py | 154 | CODE | |
| LOW | tests/agent/test_channel/test_channel.py | 134 | CODE | |
| LOW | tests/agent/test_acp/test_server_dispatch.py | 45 | CODE | |
| LOW | tests/agent/test_acp/test_server_dispatch.py | 173 | CODE | |
| LOW | tests/agent/test_acp/test_server_forwarding.py | 63 | CODE | |
| LOW | tests/agent/test_acp/test_server_forwarding.py | 157 | CODE | |
| LOW | tests/agent/test_acp/test_standard_acp_client.py | 55 | CODE | |
| LOW | tests/agent/test_acp/test_standard_acp_client.py | 225 | CODE | |
| LOW | tests/agent/test_acp/test_standard_acp_client.py | 149 | CODE | |
| LOW | tests/agent/test_acp/test_raw_events.py | 62 | CODE | |
| LOW | tests/agent/test_acp/test_raw_events.py | 313 | CODE | |
| LOW | tests/agent/test_acp/test_raw_events.py | 133 | CODE | |
| LOW | tests/agent/test_acp/test_action_methods.py | 529 | CODE | |
| LOW | tests/agent/test_acp/test_action_methods.py | 572 | CODE | |
| LOW | tests/agent/test_acp/test_discovery.py | 36 | CODE | |
| LOW | tests/agent/test_acp/test_elicitation_e2e.py | 61 | CODE | |
| LOW | tests/agent/test_acp/test_elicitation_e2e.py | 116 | CODE | |
| LOW | tests/agent/test_acp/test_stdio_bridge.py | 43 | CODE | |
| LOW | tests/agent/test_acp/test_server_session_info.py | 115 | CODE | |
| LOW | tests/agent/test_acp/test_server.py | 29 | CODE | |
| LOW | tests/agent/test_acp/test_approval.py | 1445 | CODE | |
| LOW | tests/agent/test_acp/test_approval.py | 1499 | CODE | |
| LOW | tests/agent/test_acp/test_tui/test_client.py | 89 | CODE | |
| LOW | tests/_cli/test_acp_cli.py | 71 | CODE | |
| LOW | tests/model/test_compaction_edit_server_tools.py | 60 | CODE | |
| LOW | tests/model/test_compaction_edit_thinking.py | 48 | CODE | |
| LOW | tests/model/test_inline_media_serialization.py | 154 | CODE | |
| LOW | tests/model/test_compaction.py | 783 | CODE | |
| LOW | tests/model/test_model_info.py | 357 | CODE | |
| LOW | tests/model/test_model_info.py | 432 | CODE | |
| LOW | tests/model/providers/test_openai_prompt_cache.py | 129 | CODE | |
| LOW | tests/model/providers/test_openai_prompt_cache.py | 516 | CODE | |
| LOW | tests/model/providers/test_anthropic.py | 74 | CODE | |
| LOW | tests/model/providers/test_anthropic_cache_control.py | 87 | CODE | |
| LOW | tests/model/providers/test_perplexity.py | 31 | CODE | |
| LOW | tests/model/providers/test_litellm_proxy.py | 2051 | CODE | |
| LOW | tests/view/test_sample_list_states.py | 368 | CODE | |
| LOW | tests/view/test_sample_list_states.py | 371 | CODE | |
| LOW | tests/log/test_log_attachments.py | 57 | CODE | |
| LOW | tests/log/test_log_attachments.py | 67 | CODE | |
| LOW | tests/log/test_task_log.py | 1717 | CODE | |
| LOW | tests/log/test_eval_log.py | 450 | CODE | |
| LOW | tests/log/test_eval_log.py | 1594 | CODE | |
| 568 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …ort/src/inspect_tool_support/_util/json_rpc_helpers.py | 43 | Validates RPC method parameters and handles the method execution. This function validates the provided paramet | STRING |
| HIGH | src/inspect_sandbox_tools/tests/conftest.py | 80 | Execute an RPC request via the CLI and return the parsed response. Args: request: The JSON-RPC request dict | STRING |
| HIGH | src/inspect_sandbox_tools/tests/test_exec_remote.py | 34 | Poll an exec_remote job until it completes or times out. Args: rpc_client: The RPC client function to use f | STRING |
| HIGH | …ls/src/inspect_sandbox_tools/_util/json_rpc_helpers.py | 46 | Validates RPC method parameters and handles the method execution. This function validates the provided paramet | STRING |
| HIGH | src/inspect_ai/_util/registry.py | 426 | Create a registry object. Creates objects registered via decorator (e.g. `@task`, `@solver`). Note that this ca | STRING |
| HIGH | src/inspect_ai/_util/registry.py | 502 | Lookup RegistryInfo for an object. Args: o (object): Object to lookup info for Returns: Regist | STRING |
| HIGH | src/inspect_ai/_util/flag_values.py | 29 | An option that is a bare flag or an integer. Args: value: The text given, or `None` for the bare flag. | STRING |
| HIGH | src/inspect_ai/_util/download.py | 44 | Download a file and verify its SHA256 checksum. If `dest` already exists and its checksum matches, the download is | STRING |
| HIGH | src/inspect_ai/_util/download.py | 112 | Download a Google Drive file via `gdown` and verify SHA256. Useful for fetching public-link Google Drive assets (da | STRING |
| HIGH | src/inspect_ai/_util/format.py | 49 | Format a template string, optionally preserving unknown placeholders. Args: template: A string containing { | STRING |
| HIGH | src/inspect_ai/_util/_json_rpc.py | 159 | Execute a JSON-RPC command expecting a scalar result. Args: method (str): The JSON-RPC method to call. | STRING |
| HIGH | src/inspect_ai/_util/_json_rpc.py | 199 | Execute a JSON-RPC command to a sandbox environment expecting a model result. Args: method (str): The JS | STRING |
| HIGH | src/inspect_ai/_util/_json_rpc.py | 233 | Execute a JSON-RPC notification to a sandbox environment. A notification is a JSON-RPC request that doesn't ex | STRING |
| HIGH | src/inspect_ai/_util/_json_rpc.py | 342 | Generate a string description of an RPC call. Args: method (str): The name of the RPC method. | STRING |
| HIGH | src/inspect_ai/_util/local_server.py | 294 | Start a server with the given command and handle potential errors. Args: base_cmd: List of base comman | STRING |
| HIGH | src/inspect_ai/_util/file.py | 587 | Convert a string into a safe filename by removing or replacing unsafe characters. Args: s (str): The i | STRING |
| HIGH | src/inspect_ai/_util/dateutil.py | 148 | Parse and normalize ISO datetime string or datetime instance to UTC. For legacy string temporal fields that cannot | STRING |
| HIGH | src/inspect_ai/_util/_async.py | 41 | Runs all of the passed async functions and collects their results. The results will be returned in the same order a | STRING |
| HIGH | src/inspect_ai/_util/text.py | 141 | Convert a str to float, handling exponent characters and Unicode fractions. The Python isnumeric() function returns | STRING |
| HIGH | src/inspect_ai/_util/images.py | 183 | Validate an inline image returned by a model provider. Provider output is untrusted, so only inline data URIs are a | STRING |
| HIGH | src/inspect_ai/_util/async_zip.py | 438 | Open a ZIP member and stream its decompressed contents. Must be used as an async context manager to ensure prop | STRING |
| HIGH | src/inspect_ai/_util/atomic_write.py | 103 | Context manager for atomic local-file writes with durability guarantees. Implements the write-to-temporary-then-ato | STRING |
| HIGH | src/inspect_ai/core/_imports.py | 41 | Check every import in a module or package against an allowlist. Reads source files with `ast` rather than importing | STRING |
| HIGH | src/inspect_ai/util/_json.py | 325 | Convert JSON schema to Pydantic BaseModel. Handles nested objects, arrays, and basic validations. Args: | STRING |
| HIGH | src/inspect_ai/util/_subprocess.py | 112 | Execute and wait for a subprocess. Convenience method for solvers, scorers, and tools to launch subprocesses. A | STRING |
| HIGH | src/inspect_ai/util/_store.py | 227 | Convert a JsonChange to a jsonpatch operation dict with validation. Args: change: The JsonChange to convert | STRING |
| HIGH | src/inspect_ai/util/_notify.py | 62 | Build an `Apprise` instance for the eval's notification config. Notification URLs frequently carry secrets (Slack t | STRING |
| HIGH | src/inspect_ai/util/_panel.py | 82 | Create an input panel in the task display. There can only be a single instance of an InputPanel with a given 't | STRING |
| HIGH | src/inspect_ai/util/_restic/resolver.py | 129 | Return a path to a usable restic binary for the given platform. Downloads the archive on cache miss, retrying trans | STRING |
| HIGH | src/inspect_ai/util/_sandbox/exec_remote.py | 674 | Run a command and return the result without streaming. Submits the command, polls until completion, and returns Exe | STRING |
| HIGH | src/inspect_ai/util/_sandbox/compose.py | 459 | Parse a Docker Compose file into a ComposeConfig. Args: file: Path to the compose file. multiple_se | STRING |
| HIGH | src/inspect_ai/util/_sandbox/context.py | 187 | Get a SandboxEnvironment that satisfies all the given injection requirements. Args: injectables: Single San | STRING |
| HIGH | src/inspect_ai/util/_sandbox/context.py | 449 | Find the best sandbox for injection and return it with needed injections. Args: injectables: List of detect | STRING |
| HIGH | src/inspect_ai/util/_sandbox/environment.py | 215 | Execute a command within a sandbox environment. The current working directory for execution will be the per-sam | STRING |
| HIGH | src/inspect_ai/util/_sandbox/environment.py | 311 | Read a file from the sandbox environment. By default, file size is limited to 100 MiB. You may change this by s | STRING |
| HIGH | src/inspect_ai/util/_sandbox/environment.py | 342 | Information required to connect to sandbox environment. Args: user: User to login as. Return | STRING |
| HIGH | src/inspect_ai/util/_sandbox/environment.py | 381 | Start a command and return a process handle or result. In streaming mode (stream=True), the function returns on | STRING |
| HIGH | src/inspect_ai/util/_sandbox/environment.py | 454 | Verify and return a reference to a subclass of SandboxEnvironment. Args: sandbox_cls: Class of sandb | STRING |
| HIGH | src/inspect_ai/util/_sandbox/_privileged.py | 139 | Run ``cmd`` in ``sandbox`` as ``user`` with utilities resolved via :data:`SYSTEM_PATH`. A drop-in for ``sandbox.exe | STRING |
| HIGH | src/inspect_ai/util/_sandbox/_privileged.py | 222 | Look ``name`` up on the *image's* ``PATH`` with a ``which`` resolved via :data:`SYSTEM_PATH`. For the questions tha | STRING |
| HIGH | src/inspect_ai/util/_sandbox/_framework_directory.py | 573 | Create or adopt ``path`` as a root-owned framework directory, if root works. Runs :func:`ensure_framework_directory | STRING |
| HIGH | src/inspect_ai/util/_sandbox/_framework_directory.py | 715 | Verify ``path`` and then run ``cmd`` with the verified directory as cwd. The verification and the command run in th | STRING |
| HIGH | src/inspect_ai/util/_sandbox/_json_rpc_transport.py | 72 | Execute an RPC request using the sandbox transport. Args: method: The JSON-RPC method to call. | STRING |
| HIGH | src/inspect_ai/_lfs/_client.py | 39 | Get download URLs for LFS objects via the batch API. Chunks requests to avoid exceeding GitHub's payload size limit | STRING |
| HIGH | src/inspect_ai/_lfs/resolver.py | 20 | Resolve a directory that may contain LFS pointer files. Recursively checks source_dir for LFS pointers. If none are | STRING |
| HIGH | src/inspect_ai/solver/_solver.py | 85 | Contribute to solving an evaluation task. Transform a `TaskState`, returning the new state. Solvers may | STRING |
| HIGH | src/inspect_ai/solver/_solver.py | 163 | Decorator for registering solvers. Args: name: Optional name for solver. If the decorator has n | STRING |
| HIGH | src/inspect_ai/scorer/_scorer.py | 139 | Decorator for registering scorers. Args: metrics: One or more metrics to calculate over the sco | STRING |
| HIGH | src/inspect_ai/scorer/_metric.py | 335 | Compute a metric on a list of scores. Args: scores: List of scores. Returns: Metri | STRING |
| HIGH | src/inspect_ai/scorer/_score.py | 15 | Score a model conversation. Score a model conversation (you may pass `TaskState` or `AgentState` as the value f | STRING |
| HIGH | src/inspect_ai/scorer/_metrics/aggregate.py | 25 | Apply `agg` to a single key extracted from each dict-valued `Score.value`. Many scorers emit dict-valued scores (mu | STRING |
| HIGH | src/inspect_ai/_cli/util.py | 189 | Parse model roles from CLI args. Supports key-value, YAML, and JSON formats. Args: model_roles: Tuple of st | STRING |
| HIGH | src/inspect_ai/_cli/util.py | 251 | Parse model specs from CLI args. Each spec is one YAML or JSON mapping. A spec holds `model` (required), `base_url` | STRING |
| HIGH | src/inspect_ai/_cli/eval.py | 1874 | Resolve `--model-spec` into one model per spec. A spec builds a `Model`, and `get_model()` returns a `Model` unchan | STRING |
| HIGH | src/inspect_ai/model/_internal.py | 15 | Extracts and removes a smuggled <internal>...</internal> tag from the content string, if present. Note: | STRING |
| HIGH | src/inspect_ai/model/_openai_responses.py | 2456 | Convert CompactedResponse to a list of ChatMessages. The compact endpoint returns the complete new context window, | STRING |
| HIGH | src/inspect_ai/model/_google_convert.py | 76 | Convert Google GenerateContentResponse into Inspect ModelOutput. Args: response: Google GenerateContentResp | STRING |
| HIGH | src/inspect_ai/model/_model_info.py | 289 | Get model information including context window, output tokens, etc. Looks up model information from a local databas | STRING |
| HIGH | src/inspect_ai/model/_model.py | 514 | Tokenize text into token IDs using the model's tokenizer. Override in providers that support server-side tokeni | STRING |
| HIGH | src/inspect_ai/model/_model.py | 647 | Compact messages using provider-native compaction. Some model providers (e.g., OpenAI Codex models) support nat | STRING |
| 42 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | design/bridge-host-tool-events.md | 663 | except Exception as ex: | CODE |
| LOW | design/deepagent-background.md | 137 | except Exception as ex: | CODE |
| LOW | design/recover.md | 320 | except Exception: | CODE |
| LOW | design/recover.md | 401 | except Exception: | CODE |
| LOW | tests/test_retry.py | 394 | except Exception: | CODE |
| LOW | tests/conftest.py | 490 | except Exception as ex: | CODE |
| LOW | tests/conftest.py | 574 | except Exception as ex: | CODE |
| LOW⚡ | tests/test_eval_set.py | 198 | except Exception: | CODE |
| LOW⚡ | tests/test_eval_set.py | 205 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 50 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 101 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 136 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 168 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 194 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 221 | except Exception: | CODE |
| LOW | tests/test_model_env_mismatch.py | 247 | except Exception: | CODE |
| LOW | tests/tools/test_tools_bridge.py | 879 | except Exception: | CODE |
| LOW | tests/tools/test_tools_bridge.py | 1162 | except Exception as ex: | CODE |
| LOW | tests/tools/test_tools_bridge.py | 1198 | except Exception: | CODE |
| LOW | tests/tools/test_sandbox_docker_and_local.py | 135 | except Exception: | CODE |
| LOW | tests/tools/test_tool_with.py | 34 | except Exception: | CODE |
| MEDIUM | tests/tools/test_tool_with.py | 27 | def test_tool_with_validation(): | CODE |
| LOW | tests/tools/sandbox_tools_utils/test_sandbox_tools.py | 646 | except Exception as ex: | CODE |
| LOW | tests/util/test_async.py | 44 | except Exception as group: | CODE |
| LOW | tests/util/test_async.py | 57 | except Exception as group: | CODE |
| MEDIUM | tests/util/test_async.py | 30 | def fail(): | CODE |
| LOW⚡ | tests/util/test_anyio.py | 66 | except Exception as inner_tool_error: | CODE |
| LOW⚡ | tests/util/test_anyio.py | 72 | except Exception as e: | CODE |
| LOW⚡ | tests/util/test_anyio.py | 74 | except Exception as wow: | CODE |
| LOW | tests/util/sandbox/test_sandbox_service.py | 1103 | except Exception as ex: | CODE |
| LOW | tests/util/sandbox/test_sandbox_service.py | 115 | except Exception as e: | STRING |
| LOW | tests/agent/test_bridge_refusals.py | 243 | except Exception as ex: | CODE |
| MEDIUM | tests/agent/deepagent/test_agent_tool.py | 258 | def import ToolDef | CODE |
| LOW | tests/agent/deepagent/test_agent_tool.py | 266 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_span_boundary.py | 184 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_server_dispatch.py | 173 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_server_dispatch.py | 226 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_server_forwarding.py | 157 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_server_forwarding.py | 208 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_server_forwarding.py | 779 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_server_forwarding.py | 843 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_standard_acp_client.py | 149 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_standard_acp_client.py | 197 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_raw_events.py | 133 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_raw_events.py | 179 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_action_methods.py | 572 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_action_methods.py | 615 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_elicitation_e2e.py | 168 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_elicitation_e2e.py | 116 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_server_session_info.py | 158 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_server_session_info.py | 115 | def _read_loop(self) -> None: | CODE |
| LOW⚡ | tests/agent/test_acp/test_server.py | 266 | except Exception: | CODE |
| LOW⚡ | tests/agent/test_acp/test_server.py | 318 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_server.py | 238 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_server.py | 387 | except Exception: | CODE |
| LOW | tests/agent/test_acp/test_server.py | 432 | except Exception: | CODE |
| MEDIUM | tests/agent/test_acp/test_approval.py | 1499 | def _read_loop(self) -> None: | CODE |
| LOW | tests/agent/test_acp/test_approval.py | 1552 | except Exception: | CODE |
| LOW⚡ | tests/agent/test_acp/test_tui/test_defer_body.py | 147 | except Exception: | CODE |
| LOW⚡ | tests/agent/test_acp/test_tui/test_defer_body.py | 155 | except Exception: | CODE |
| 375 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/tools/test_tools_bridge.py | 0 | add two numbers. args: x: first number to add. y: second number to add. | STRING |
| HIGH | tests/sentinel/test_sentinel_config.py | 0 | add two numbers. args: x: first number to add. y: second number to add. | STRING |
| HIGH | tests/sentinel/test_sentinel_human.py | 0 | add two numbers. args: x: first number to add. y: second number to add. | STRING |
| HIGH | tests/sentinel/test_sentinel_dispatch.py | 0 | add two numbers. args: x: first number to add. y: second number to add. | STRING |
| HIGH | tests/model/providers/test_deepseek.py | 0 | add two numbers. args: x: first number to add. y: second number to add. | STRING |
| HIGH | tests/tools/test_tools.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/agent/test_react_review.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/agent/test_agent_react.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/solver/test_basic_agent.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/review/test_review.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/model/test_reasoning_claude.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/test_helpers/tools.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/approval/test_approval.py | 0 | add two numbers. args: x (int): first number to add. y (int): second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | tests/agent/test_acp/test_server_forwarding.py | 0 | short /tmp data dir so af_unix paths fit in 104 chars on macos. | STRING |
| HIGH | tests/agent/test_acp/test_action_methods.py | 0 | short /tmp data dir so af_unix paths fit in 104 chars on macos. | STRING |
| HIGH | tests/agent/test_acp/test_elicitation_e2e.py | 0 | short /tmp data dir so af_unix paths fit in 104 chars on macos. | STRING |
| HIGH | tests/agent/test_acp/test_stdio_bridge.py | 0 | short /tmp data dir so af_unix paths fit in 104 chars on macos. | STRING |
| HIGH | tests/agent/test_acp/test_approval.py | 0 | short /tmp data dir so af_unix paths fit in 104 chars on macos. | STRING |
| HIGH | tests/_cli/test_token_limit_flag.py | 0 | run the test command and eval its echoed repr back to a python value. | STRING |
| HIGH | tests/_cli/test_ctl_server_flag.py | 0 | run the test command and eval its echoed repr back to a python value. | STRING |
| HIGH | tests/_cli/test_acp_server_flag.py | 0 | run the test command and eval its echoed repr back to a python value. | STRING |
| HIGH | tests/model/providers/test_anthropic.py | 0 | completes each batch at once, answering each request with its id. | STRING |
| HIGH | tests/model/providers/test_openai_compatible.py | 0 | completes each batch at once, answering each request with its id. | STRING |
| HIGH | tests/model/providers/util/test_batch.py | 0 | completes each batch at once, answering each request with its id. | STRING |
| HIGH | tests/model/providers/test_mistral.py | 0 | unset stream is "auto": stream iff the caller passed on_stream. | STRING |
| HIGH | tests/model/providers/test_azureai.py | 0 | unset stream is "auto": stream iff the caller passed on_stream. | STRING |
| HIGH | tests/model/providers/test_bedrock_streaming.py | 0 | unset stream is "auto": stream iff the caller passed on_stream. | STRING |
| HIGH | tests/model/providers/test_groq.py | 0 | unset stream is "auto": stream iff the caller passed on_stream. | STRING |
| HIGH | tests/model/providers/test_openai_compatible.py | 0 | unset stream is "auto": stream iff the caller passed on_stream. | STRING |
| HIGH | tests/model/providers/test_mistral.py | 0 | the stream accumulator reconstructs the completion and reports deltas. | STRING |
| HIGH | tests/model/providers/test_azureai.py | 0 | the stream accumulator reconstructs the completion and reports deltas. | STRING |
| HIGH | tests/model/providers/test_groq.py | 0 | the stream accumulator reconstructs the completion and reports deltas. | STRING |
| HIGH | tests/model/providers/test_mistral.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_azureai.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_grok.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_bedrock_streaming.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_groq.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_sagemaker.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_openai_responses.py | 0 | without an on_stream consumer only heartbeat/usage progress runs. explicit streaming=true callers stream without asking | STRING |
| HIGH | tests/model/providers/test_google.py | 0 | add two numbers. args: x: first number to add. y: second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | docs/_tools-basics.md | 0 | add two numbers. args: x: first number to add. y: second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | examples/tool_use.py | 0 | add two numbers. args: x: first number to add. y: second number to add. returns: the sum of the two numbers. | STRING |
| HIGH | …ort/src/inspect_tool_support/_util/json_rpc_helpers.py | 0 | check whether an exception raised from httpx should be retried. implements the strategy described here: https://cloud.go | STRING |
| HIGH | …ls/src/inspect_sandbox_tools/_util/json_rpc_helpers.py | 0 | check whether an exception raised from httpx should be retried. implements the strategy described here: https://cloud.go | STRING |
| HIGH | src/inspect_ai/_util/httpx.py | 0 | check whether an exception raised from httpx should be retried. implements the strategy described here: https://cloud.go | STRING |
| HIGH | …andbox_tools/_remote_tools/_bash_session/tool_types.py | 0 | user to run as: a username, or the sandbox default user's identity as captured by the host. switching requires the serve | STRING |
| HIGH | …sandbox_tools/_remote_tools/_exec_remote/tool_types.py | 0 | user to run as: a username, or the sandbox default user's identity as captured by the host. switching requires the serve | STRING |
| HIGH | …inspect_sandbox_tools/_remote_tools/_mcp/tool_types.py | 0 | user to run as: a username, or the sandbox default user's identity as captured by the host. switching requires the serve | STRING |
| HIGH | src/inspect_ai/util/_store_model.py | 0 | pydantic model interface to the store. args: model_cls: pydantic model type (must derive from storemodel) instance: opti | STRING |
| HIGH | src/inspect_ai/solver/_task_state.py | 0 | pydantic model interface to the store. args: model_cls: pydantic model type (must derive from storemodel) instance: opti | STRING |
| HIGH | src/inspect_ai/log/_log.py | 0 | pydantic model interface to the store. args: model_cls: pydantic model type (must derive from storemodel) instance: opti | STRING |
| HIGH | src/inspect_ai/model/_model.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/mistral.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/grok.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/google.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/openai.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/anthropic.py | 0 | canonical model name for model info database lookup. | STRING |
| HIGH | src/inspect_ai/model/_providers/mistral.py | 0 | scope adaptive concurrency per (key, model). a pool shared across models lets the faster model's signals push the adapti | STRING |
| HIGH | src/inspect_ai/model/_providers/grok.py | 0 | scope adaptive concurrency per (key, model). a pool shared across models lets the faster model's signals push the adapti | STRING |
| HIGH | src/inspect_ai/model/_providers/openai_compatible.py | 0 | scope adaptive concurrency per (key, model). a pool shared across models lets the faster model's signals push the adapti | STRING |
| 8 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | CHANGELOG.md | 2813 | - ReAct Agent: [truncation](https://inspect.aisi.org.uk/agents.html#truncation) option to trim conversation messages whe | CODE |
| MEDIUM | design/bridge-host-tool-events.md | 151 | teardown #5464 triggers may pre-empt delivery of the reply; the scaffold's | CODE |
| MEDIUM | design/ctl/pause-resume.md | 140 | - **Hard pause (hold at next model call) — implemented.** `pause --now`: in addition to quiesce, hold in-flight samples | CODE |
| MEDIUM | design/ctl/control-channel.md | 992 | - **Self-targeting guard hardening.** Open question #8 in this doc — an LLM agent running *inside* an eval shouldn't be | CODE |
| MEDIUM⚡ | tests/tools/test_tools_bridge.py | 922 | # the scaffold reads as tool output, so an unexpected one has to be signalled to | COMMENT |
| MEDIUM | tests/tools/test_tools_bridge.py | 1152 | # the scaffold carries on regardless; the monitor tears the group down | COMMENT |
| MEDIUM | tests/checkpoint/test_checkpointer.py | 651 | # leading scaffolding before the first wrap — dropped by the slice | COMMENT |
| MEDIUM | tests/checkpoint/test_checkpointer.py | 671 | # seeded wrap and still open at fire time — trailing scaffolding | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_compaction_native.py | 49 | # the context window. Reasoning + compaction is tested separately in | COMMENT |
| MEDIUM | tests/agent/test_bridge_generation_config.py | 405 | # would leak the scaffold's value to the served model whenever the eval config | COMMENT |
| MEDIUM⚡ | tests/agent/test_bridge_track_state.py | 1387 | # the scaffold restarts: title call fires first again | COMMENT |
| MEDIUM | tests/agent/test_bridge_track_state.py | 259 | # main loop, then scaffold compaction: the compacted loop is promoted, so | COMMENT |
| MEDIUM | tests/agent/test_bridge_track_state.py | 390 | # the scaffold re-creates every message: different ids and metadata, and | COMMENT |
| MEDIUM | tests/agent/test_bridge_track_state.py | 427 | # scaffold runs the task under its own (different) system prompt | COMMENT |
| MEDIUM | tests/agent/test_bridge_track_state.py | 604 | # title call fires first; the scaffold's store holds the condensed turn | COMMENT |
| MEDIUM | tests/agent/test_bridge_track_state.py | 1461 | # main loop: turn 2 (scaffold round-trips through its own store) | COMMENT |
| MEDIUM | tests/agent/test_bridge_refusals.py | 236 | # the scaffold carries on regardless; the monitor tears the group down | COMMENT |
| MEDIUM | tests/agent/test_agent_bridge.py | 1108 | # backend; the scaffold's second call replays the searched assistant turn | STRING |
| MEDIUM | tests/agent/test_agent_bridge.py | 1119 | # both scaffold calls completed (the second replays the first turn) | STRING |
| MEDIUM⚡ | tests/agent/test_bridge_approval.py | 817 | # the scaffold re-issues the approved call with the keys in a different order | COMMENT |
| MEDIUM⚡ | tests/agent/test_bridge_approval.py | 2081 | # the rejected call never reached the scaffold | COMMENT |
| MEDIUM⚡ | tests/agent/test_bridge_approval.py | 2215 | # should behave identically whichever API the scaffold speaks. These drive each | COMMENT |
| MEDIUM⚡ | tests/agent/test_bridge_approval.py | 2216 | # dialect impl the way a scaffold's request arrives. | COMMENT |
| MEDIUM | tests/agent/test_bridge_approval.py | 384 | # the scaffold gets the approved arguments... | COMMENT |
| MEDIUM | tests/agent/test_bridge_approval.py | 452 | # the model is told and generates again; the scaffold sees only the replacement | COMMENT |
| MEDIUM | tests/agent/test_bridge_approval.py | 1537 | # a second call for the same proposal (e.g. a scaffold retrying after a | COMMENT |
| MEDIUM | tests/agent/test_bridge_approval.py | 1806 | # and so does the response handed to the scaffold | COMMENT |
| MEDIUM | tests/agent/test_bridge_approval.py | 2040 | # what the scaffold would go on to execute | COMMENT |
| MEDIUM⚡ | tests/agent/test_acp/test_operator_provenance.py | 55 | # note_operator_message — the scaffold's signal | COMMENT |
| MEDIUM | tests/agent/test_acp/test_router_bridge_tools.py | 150 | # the scaffold-provided view renders richly (not args-only) | COMMENT |
| MEDIUM⚡ | tests/agent/test_acp/test_tui/test_widgets.py | 915 | # Used-only — context window denominator dropped. | COMMENT |
| MEDIUM | tests/scorer/test_math.py | 289 | # An integer answer on the final line of a reasoning trace is extracted. | COMMENT |
| MEDIUM | tests/model/test_compaction.py | 1197 | # small context window so a modest tool output overflows it | COMMENT |
| MEDIUM | tests/model/test_model_length.py | 24 | # model context window sizes | COMMENT |
| MEDIUM | tests/model/test_model_length.py | 49 | # cloudflare serves this model with a reduced context window | COMMENT |
| MEDIUM | tests/model/test_model_length.py | 61 | # context window for model | COMMENT |
| MEDIUM | tests/model/test_model_length.py | 65 | # create message that exceeds context window | COMMENT |
| MEDIUM | tests/model/providers/test_anthropic.py | 526 | # scaffold for should_retry | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_vllm.py | 34 | # this allows us to run base models with the chat message scaffolding: | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_openai_model_names.py | 89 | # input_tokens_name() aliases to the current frontier so the context window | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_grok_model_names.py | 84 | # input_tokens_name() aliases to the current frontier so the context window | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_grok_model_names.py | 98 | # frontier would overstate its context window | COMMENT |
| MEDIUM | tests/model/providers/test_anthropic_fallback.py | 542 | # scaffold echoes the assistant turn as dicts (incl. the fallback block) | COMMENT |
| MEDIUM | tests/model/providers/test_hf.py | 36 | # this allows us to run base models with the chat message scaffolding: | COMMENT |
| MEDIUM | tests/model/providers/test_hf.py | 55 | # this allows us to run base models with the chat message scaffolding: | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_google_model_names.py | 169 | # input_tokens_name() aliases to the current frontier so the context window | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 2300 | # Bridge -> scaffold (assistant text) -> scaffold echoes it back as input. | COMMENT |
| MEDIUM | docs/CHANGELOG.md | 2813 | - ReAct Agent: [truncation](https://inspect.aisi.org.uk/agents.html#truncation) option to trim conversation messages whe | CODE |
| MEDIUM | docs/evals/evals.json | 4400 | "paper": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/", | CODE |
| MEDIUM | docs/evals/evals.json | 6246 | "paper": "https://www.anthropic.com/research/agentic-misalignment", | CODE |
| MEDIUM | src/inspect_ai/_control/pause.py | 256 | # at the hard gate — the interim-scoring pass's event-driven park ack (no | COMMENT |
| MEDIUM | src/inspect_ai/_control/pause.py | 425 | # event-driven park ack (see sample_park_waiter) | COMMENT |
| MEDIUM | src/inspect_ai/util/_checkpoint/hydrate.py | 748 | # Slice away the live session's scaffolding ahead of the first | COMMENT |
| MEDIUM | src/inspect_ai/util/_checkpoint/hydrate.py | 783 | # resumed session's own structural scaffolding (`init` / `solvers` / | COMMENT |
| MEDIUM | src/inspect_ai/agent/_react.py | 275 | # check for context window overflow | COMMENT |
| MEDIUM | src/inspect_ai/agent/_react.py | 515 | # check for context window overflow | COMMENT |
| MEDIUM | src/inspect_ai/agent/_react.py | 796 | # exceed the context window; _handle_overflow's forced | COMMENT |
| MEDIUM⚡ | src/inspect_ai/agent/_acp/inspect_ext.py | 132 | # (claude_code, codex, …): their tools are run by the bridged scaffold, not by | STRING |
| MEDIUM⚡ | src/inspect_ai/agent/_acp/connection.py | 1170 | # Bind / unbind orchestration | COMMENT |
| MEDIUM | src/inspect_ai/agent/_acp/event_mapping.py | 723 | # and a known context window. ACP semantics: "Tokens currently in | COMMENT |
| 54 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_eval_set.py | 3268 | # Create a log with the exact task identity eval_set expects, then | COMMENT |
| MEDIUM | tests/test_model_env_mismatch.py | 85 | # Create a minimal task with proper solver and scorer | COMMENT |
| MEDIUM⚡ | tests/tools/test_web_search_tavily.py | 55 | # Create a client with our mock transport | COMMENT |
| MEDIUM | tests/analysis/test_large_string.py | 11 | # Create a DataFrame with a small PyArrow string column | COMMENT |
| MEDIUM | tests/analysis/test_large_string.py | 32 | # Create a DataFrame with a large PyArrow string column | COMMENT |
| MEDIUM | tests/analysis/test_large_string.py | 55 | # Create a DataFrame with both small and large columns (same length) | COMMENT |
| MEDIUM⚡ | tests/util/test_store_nested_types.py | 89 | # Create a store with dict data | COMMENT |
| MEDIUM⚡ | tests/util/test_anyio.py | 84 | # Create a circular reference: exc1.__context__ -> exc2 -> exc1 | COMMENT |
| MEDIUM | tests/util/test_anyio.py | 107 | # Create a circular reference in exception groups | COMMENT |
| MEDIUM | tests/util/sandbox/test_docker_compatibility.py | 530 | # Create a task with compose.yaml config (use None for dataset to get a dummy sample) | COMMENT |
| MEDIUM | tests/util/sandbox/sandbox_setup_symlink.sh | 12 | # Create a nested directory and symlink via “..” | COMMENT |
| MEDIUM | tests/util/sandbox/test_docker_compose_config.py | 160 | # Create a ComposeService with x-default extension | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_react.py | 661 | # Create a tool call for submit | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_react.py | 668 | # Create a tool call for a regular tool | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_react.py | 675 | # Create an assistant message with both reasoning content and tool calls | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_react.py | 684 | # Create an assistant message without reasoning content but with submit tool call | COMMENT |
| MEDIUM⚡ | tests/agent/test_agent_react.py | 690 | # Create an assistant message with only regular tool calls (should be unchanged) | COMMENT |
| MEDIUM | tests/agent/test_acp/test_action_methods.py | 973 | # Create the pending nested ToolEvent — same construction the | COMMENT |
| MEDIUM⚡ | tests/model/test_web_search_exclude_none.py | 84 | # Create a web search with ActionSearch that has sources=None (the bug condition) | COMMENT |
| MEDIUM | tests/model/test_compaction_summary.py | 100 | # Create a previous summary message | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 72 | # Create a longer conversation | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 140 | # Create a scenario with a tool response without a corresponding assistant message | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 241 | # Create a longer conversation | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 305 | # Create a longer conversation with tool calls | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 349 | # Create a conversation with alternating user-assistant pairs | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 432 | # Create an assistant message with 3 tool calls, but only 1 tool result | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 465 | # Create an assistant message with tool calls, but no tool results at all | COMMENT |
| MEDIUM | tests/model/test_trim_messages.py | 492 | # Create a conversation where trimming will create orphan tool_calls | COMMENT |
| MEDIUM | tests/model/test_token_counting.py | 120 | # Create a conversation with an assistant message containing multiple tool calls | COMMENT |
| MEDIUM | tests/model/test_prompt_logprobs.py | 464 | # Create a minimal instance without calling __init__ | COMMENT |
| MEDIUM | tests/model/providers/test_anthropic.py | 699 | # Create an assistant message with a single tool call (no tool result) | COMMENT |
| MEDIUM | tests/model/providers/test_anthropic.py | 721 | # Create an assistant message with multiple tool calls (no tool results) | COMMENT |
| MEDIUM | tests/model/providers/test_anthropic.py | 753 | # Create a tool result message without a preceding assistant message | COMMENT |
| MEDIUM | tests/model/providers/test_mistral.py | 111 | # Create a simple prompt | COMMENT |
| MEDIUM | tests/model/providers/test_transformer_lens.py | 18 | # Create a small HookedTransformer model for testing | COMMENT |
| MEDIUM | tests/model/providers/test_transformer_lens.py | 34 | # Create the model using Inspect | COMMENT |
| MEDIUM | tests/model/providers/test_transformer_lens.py | 143 | # Create a small HookedTransformer model for testing | COMMENT |
| MEDIUM | tests/model/providers/test_transformer_lens.py | 159 | # Create a simple task for testing | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_google.py | 456 | # Create a mock client | COMMENT |
| MEDIUM⚡ | tests/model/providers/test_google.py | 459 | # Create a message with reasoning + text + tool call | COMMENT |
| MEDIUM | tests/model/providers/test_google.py | 1028 | # Create an assistant message with a single tool call (no tool result) | COMMENT |
| MEDIUM | tests/model/providers/test_google.py | 1053 | # Create an assistant message with multiple tool calls (no tool results) | COMMENT |
| MEDIUM | tests/model/providers/test_google.py | 1088 | # Create a tool result message without a preceding assistant message | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 733 | # Create a mock CompactedResponse with a ResponseCompactionItem | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 786 | # Create a CompactedResponse without a ResponseCompactionItem | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 953 | # Create a CompactedResponse with compaction item followed by an output message | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 1025 | # Create a compact response matching real API output: | COMMENT |
| MEDIUM | tests/model/providers/test_openai_responses.py | 1211 | # Create a message with phase stored in ContentText.internal | COMMENT |
| MEDIUM | tests/model/providers/util/test_batch_intake.py | 208 | # Create a request that's too large for the available size | COMMENT |
| MEDIUM | tests/model/providers/util/test_batch.py | 630 | # Create a batcher with a low max_consecutive_check_failures value | COMMENT |
| MEDIUM | tests/model/providers/util/test_batch.py | 701 | # Create a batcher that will fail initially then succeed | COMMENT |
| MEDIUM | tests/model/providers/util/test_batch.py | 760 | # Create a request that's close to but under the limit | COMMENT |
| MEDIUM⚡ | tests/log/test_log_file_info_fallback.py | 204 | # Create a native-named eval (timestamp prefix -> fast path) | COMMENT |
| MEDIUM⚡ | tests/log/test_log_file_info_fallback.py | 211 | # Create a custom-named eval (no timestamp -> header fallback) | COMMENT |
| MEDIUM⚡ | tests/log/test_log_attachments.py | 463 | # # Create a long text that should be condensed (> 100 chars) | COMMENT |
| MEDIUM⚡ | tests/log/test_log_attachments.py | 467 | # # Create a model event with long content | COMMENT |
| MEDIUM | tests/log/test_log_attachments.py | 539 | # # Create the same long text that will appear in multiple events | COMMENT |
| MEDIUM | tests/log/test_log_attachments.py | 620 | # Create an event with long content | COMMENT |
| MEDIUM | tests/log/test_recover_e2e.py | 283 | # Create an empty buffer DB so recovery can proceed | COMMENT |
| MEDIUM | tests/log/test_recover_e2e.py | 402 | # Create a crashed log | COMMENT |
| 42 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/conftest.py | 361 | COMMENT | |
| LOW | tests/conftest.py | 381 | # OS-thread dump (async-signal-safe; catches hard C-level wedges), then — for | COMMENT |
| LOW | tests/test_eval_set_selection.py | 1301 | # Additive though they are, the models forbid extra fields, so a v1 reader | COMMENT |
| LOW | tests/checkpoint/resume_kill_harness.py | 101 | COMMENT | |
| LOW | tests/checkpoint/resume_kill_harness.py | 221 | return "crashed" | COMMENT |
| LOW | tests/util/test_text.py | 21 | # Stripping it to face value would make match(numeric=True) | COMMENT |
| LOW | tests/agent/test_acp/test_session.py | 221 | COMMENT | |
| LOW | tests/agent/test_acp/test_router.py | 1881 | tg.start_soon(lambda: agent(_state())) | COMMENT |
| LOW | tests/agent/test_acp/test_standard_acp_client.py | 241 | k.startswith("inspect.") or k.startswith("inspect/") | COMMENT |
| LOW | tests/agent/test_acp/test_approval.py | 561 | shim_task.cancel() | COMMENT |
| LOW | tests/agent/test_acp/test_tui/test_client.py | 61 | active.agent_name = agent_name | COMMENT |
| LOW | tests/agent/test_acp/test_tui/test_session_screen.py | 1461 | # by one refresh — pump another pause so focus has landed | COMMENT |
| LOW | tests/model/test_reasoning_claude.py | 21 | @skip_if_no_anthropic | COMMENT |
| LOW | tests/model/test_adaptive_connections.py | 961 | # Controller's adaptive bound max should be 50 — the int shorthand. | COMMENT |
| LOW | tests/view/test_openapi_schema_generation.py | 241 | def test_default_pydantic_makes_defaulted_fields_not_required() -> None: | COMMENT |
| LOW | tests/log/test_log_attachments.py | 461 | # transcript = Transcript() | COMMENT |
| LOW | tests/log/test_log_attachments.py | 481 | # stored_event = transcript.events[0] | COMMENT |
| LOW | tests/log/test_log_attachments.py | 501 | # input=[initial_message], | COMMENT |
| LOW | tests/log/test_log_attachments.py | 521 | # # Verify the output was condensed | COMMENT |
| LOW | tests/log/test_log_attachments.py | 541 | # message1 = ChatMessageUser(content=repeated_text) | COMMENT |
| LOW | tests/log/test_log_attachments.py | 561 | # ) | COMMENT |
| LOW | docs/evals/model_cards.yml | 1 | # Benchmarks referenced in recent model cards/announcements. | COMMENT |
| LOW | examples/computer/compose.yaml | 1 | services: | COMMENT |
| LOW | scripts/release-sandbox-tools.sh | 181 | COMMENT | |
| LOW | .github/workflows/release.yml | 1 | name: Release | COMMENT |
| LOW | .github/workflows/suppressions-comment.yml | 1 | # Posts the sticky suppression-ledger delta comment on PRs, fork PRs | COMMENT |
| LOW | .github/workflows/release-pr-checks.yml | 1 | name: Release PR checks | COMMENT |
| LOW | .github/workflows/build.yml | 181 | fetch-tags: true | COMMENT |
| LOW | .github/workflows/build.yml | 261 | # (~2.5 min median, ~8 min p90 of PR wall clock under runner | COMMENT |
| LOW | .github/workflows/build.yml | 321 | # published binaries were downloaded into. | COMMENT |
| LOW | .github/workflows/build.yml | 601 | # Digest file must be in lockstep with the version bump: four non-blank | COMMENT |
| LOW | .github/scripts/check_suppressions.py | 41 | COMMENT | |
| LOW | …ool_support/_remote_tools/_web_browser/scale_factor.py | 21 | # | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 41 | # AXNode, | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 61 | # def test_getitem(self): | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 81 | # } | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 101 | # } | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 121 | COMMENT | |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 141 | # node_data = { | COMMENT |
| LOW | …ote_tools/_web_browser/test_accessibility_tree_node.py | 161 | # } | COMMENT |
| LOW | …t_sandbox_tools/_remote_tools/_exec_remote/__init__.py | 1 | # TODO: Consider renaming _remote_tools directory to _remote_services or _json_rpc_services | COMMENT |
| LOW | …sandbox_tools/src/inspect_sandbox_tools/_cli/server.py | 41 | # | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 201 | # Keep-alive intent | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 801 | active_since: float | None = None, | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1041 | # naturally (scored on their own terms, no interrupts), then | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1201 | if result.get("ok") is False: | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1321 | # the ambiguity rejection carries the pending calls | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1381 | # after a `/`); `key`/`key_limit` retune a named concurrency() registry | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1621 | # Pause / resume the whole run (the eval-set spelling — one | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1641 | # retries/min, backoff ratio, and active retry waits per model, plus | COMMENT |
| LOW | src/inspect_ai/_control/server.py | 1801 | exc_info=True, | COMMENT |
| LOW | src/inspect_ai/_control/limits.py | 361 | COMMENT | |
| LOW | src/inspect_ai/_control/cancel.py | 341 | pending = state.task_cancel.cancel_type | COMMENT |
| LOW | src/inspect_ai/_control/cancel.py | 401 | if stamp == "abort": | COMMENT |
| LOW | src/inspect_ai/_control/scoring.py | 81 | # gate (and then quiesce) before reporting it un-scored. A constant to start | COMMENT |
| LOW | src/inspect_ai/_control/__init__.py | 1 | """Control channel — HTTP server embedded in each running eval process. | COMMENT |
| LOW | src/inspect_ai/_control/__init__.py | 21 | # non-GET route with a 400 (`_control/strict.py`), atomically — so a | COMMENT |
| LOW | src/inspect_ai/_control/__init__.py | 41 | # must not 400 their retune on an older strict server, so the CLI includes | COMMENT |
| LOW | src/inspect_ai/_control/__init__.py | 61 | # server advertises >= 5 (an explicit --author/--reason against an | COMMENT |
| LOW | src/inspect_ai/_control/__init__.py | 81 | # `ctl config`), joining the version-4 retry knobs. No CLI gate — the | COMMENT |
| 228 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/input/test_input_panel.py | 129 | def _set_input_value(input_widget: Input, value: str) -> None: | CODE |
| LOW | tests/input/test_input_panel.py | 53 | def set_title(self, title: str) -> None: | CODE |
| LOW⚡ | tests/agent/test_acp/test_tui/test_elicitation_card.py | 68 | def _set_input(input_widget: Input, value: str) -> None: | CODE |
| LOW | tests/test_package/inspect_package/__init__.py | 3 | __all__ = ["simple_score"] | CODE |
| LOW | …ridianlabs-ai/inspect-docs/filters/reference/filter.py | 230 | def set_default_title(elem: pf.Element, doc: pf.Doc) -> None: | CODE |
| LOW | examples/hooks/mlflow_tracking.py | 44 | logger = logging.getLogger(__name__) | CODE |
| LOW | examples/hooks/trackio_tracking.py | 41 | logger = logging.getLogger(__name__) | CODE |
| LOW | examples/hooks/mlflow_tracing.py | 51 | logger = logging.getLogger(__name__) | CODE |
| LOW | …pect_tool_support/src/inspect_tool_support/__init__.py | 12 | __all__ = ["__version__"] | CODE |
| LOW | …ool_support/_remote_tools/_web_browser/scale_factor.py | 5 | logger = logging.getLogger(__name__) | CODE |
| LOW | …sandbox_tools/tests/test_text_editor_validated_path.py | 193 | def _set_history_path(monkeypatch: pytest.MonkeyPatch, history_path: Path) -> None: | CODE |
| LOW | src/inspect_sandbox_tools/tests/conftest.py | 19 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_sandbox_tools/tests/test_exec_remote.py | 90 | def _setup_and_teardown(sandbox_server_cleanup: None) -> None: | CODE |
| LOW | …ct_sandbox_tools/src/inspect_sandbox_tools/__init__.py | 12 | __all__ = ["__version__"] | CODE |
| LOW | …ox_tools/_in_process_tools/_text_editor/text_editor.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | …box_tools/src/inspect_sandbox_tools/_util/constants.py | 11 | __all__ = [ | CODE |
| LOW | …x_tools/src/inspect_sandbox_tools/_util/user_switch.py | 54 | def set_oom_score_adj() -> None: | CODE |
| LOW | src/inspect_ai/__init__.py | 32 | __all__ = [ | CODE |
| LOW | src/inspect_ai/_control/eval_state.py | 742 | def set_sample_requeue(eval_id: str, handle: "SampleRequeue | None") -> None: | CODE |
| LOW | src/inspect_ai/_control/eval_state.py | 755 | def set_task_scoring(eval_id: str, handle: "TaskScoring | None") -> None: | CODE |
| LOW | src/inspect_ai/_control/max_tasks.py | 58 | def set_max_tasks_override(value: int | None) -> None: | CODE |
| LOW | src/inspect_ai/_util/zipfile.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/zipfile.py | 199 | __all__ = ["zipfile_compress_kwargs"] | CODE |
| LOW | src/inspect_ai/_util/log_context.py | 8 | def set_run_shape(task_names: Iterable[str], max_epochs: int) -> None: | CODE |
| LOW | src/inspect_ai/_util/error.py | 116 | def set_exception_hook() -> None: | CODE |
| LOW | src/inspect_ai/_util/transcript.py | 66 | def set_transcript_markdown_options(markdown: Markdown) -> None: | CODE |
| LOW | src/inspect_ai/_util/registry.py | 92 | def set_annotations(wrapper: Callable[..., Any], annotations: dict[str, Any]) -> None: | CODE |
| LOW | src/inspect_ai/_util/registry.py | 123 | def set_return_annotation(wrapper: Callable[..., Any], return_type: type[Any]) -> None: | CODE |
| LOW | src/inspect_ai/_util/httpx.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/background.py | 50 | def set_background_task_group(tg: TaskGroup | None) -> None: | CODE |
| LOW | src/inspect_ai/_util/local_server.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/file.py | 668 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/event_loop_monitor.py | 38 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/asyncfiles.py | 55 | logger = logging.getLogger(__name__) | CODE |
| LOW | src/inspect_ai/_util/future.py | 14 | def set_result(self, result: T) -> None: | CODE |
| LOW | src/inspect_ai/_util/future.py | 18 | def set_exception(self, ex: Exception) -> None: | CODE |
| LOW | src/inspect_ai/_util/future.py | 32 | def set_future_result(future: "Future[T]", result: T) -> None: | CODE |
| LOW | src/inspect_ai/_util/future.py | 36 | def set_future_exception(future: "Future[T]", error: Exception) -> None: | CODE |
| LOW | src/inspect_ai/_util/strenum.py | 49 | __all__ = ["StrEnum"] | CODE |
| LOW | src/inspect_ai/_util/textual/__init__.py | 3 | __all__ = ["ElicitationForm"] | CODE |
| LOW | src/inspect_ai/_display/__init__.py | 14 | __all__ = [ | CODE |
| LOW | src/inspect_ai/_display/core/results.py | 38 | def set_retry_args_suffix(suffix: str) -> None: | CODE |
| LOW | src/inspect_ai/_display/core/display.py | 170 | def update_metrics(self, scores: list[TaskDisplayMetric]) -> None: ... | CODE |
| LOW | src/inspect_ai/_display/core/display.py | 198 | def update_task_count(self, n: int) -> None: | CODE |
| LOW | src/inspect_ai/_display/core/progress.py | 70 | def update_count(self, complete: int, total: int) -> None: | CODE |
| LOW | src/inspect_ai/_display/core/progress.py | 77 | def update_score(self, score: str) -> None: | CODE |
| LOW | src/inspect_ai/_display/core/panel.py | 23 | def set_eval_set_id_display(value: str | None) -> None: | CODE |
| LOW | src/inspect_ai/_display/plain/display.py | 75 | def update_task_count(self, n: int) -> None: | CODE |
| LOW | src/inspect_ai/_display/plain/display.py | 222 | def update_metrics(self, metrics: list[TaskDisplayMetric]) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/display.py | 85 | def update_task_count(self, n: int) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 59 | def set_flow_content(content: RenderableType) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 244 | def update_task_count(self, n: int) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 300 | def update_display(self) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 315 | def update_scan(self) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 340 | def update_title(self) -> None: | CODE |
| LOW⚡ | src/inspect_ai/_display/textual/app.py | 365 | def update_tasks(self) -> None: | CODE |
| LOW⚡ | src/inspect_ai/_display/textual/app.py | 374 | def update_samples(self) -> None: | CODE |
| LOW⚡ | src/inspect_ai/_display/textual/app.py | 381 | def update_footer(self) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 394 | def set_unread(unread: int | None) -> None: | CODE |
| LOW | src/inspect_ai/_display/textual/app.py | 408 | async def set_active_tab(active: str) -> None: | CODE |
| 163 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_sample_source.py | 322 | # once epoch 4 completed, every time under its own id (an RL harness | COMMENT |
| MEDIUM⚡ | tests/checkpoint/test_current_checkpointer_post_exit.py | 224 | # harness faithfully reproduces the normal span/checkpoint pairing — and that | COMMENT |
| MEDIUM | tests/checkpoint/resume_kill_harness.py | 253 | # this harness drives — so stamp a fixed amount, giving the budget-carry | COMMENT |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 234 | "content": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov.uk/government/news/tack | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 257 | "content": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov.uk/government/news/tack | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 1545 | "result": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov.uk/government/news/tackl | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 1577 | "output": "{\"jsonrpc\": \"2.0\", \"result\": {\"web_url\": \"https://www.aisi.gov.uk/work\", \"main_content\" | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 2102 | "result": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov.uk/government/news/tackl | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 2134 | "output": "{\"jsonrpc\": \"2.0\", \"result\": {\"web_url\": \"https://www.aisi.gov.uk/work\", \"main_content\" | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 2732 | "replaced": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov.uk/government/news | CODE |
| MEDIUM | …lysis/test_logs/2025-05-12T20-27-36-04-00_browser.json | 3577 | "cb6d9b3382c23a2729d389c44be2fb26": "[793] link \"We are now the AI Security Institute\" [url: https://www.gov. | CODE |
| MEDIUM | tests/agent/deepagent/test_deepagent_background.py | 979 | # deterministically without a scheduling barrier the harness lacks.) | COMMENT |
| MEDIUM | tests/agent/test_acp/test_deepagent_integration.py | 479 | # ACP harness. The cancel pathway itself is tested in | COMMENT |
| MEDIUM | tests/agent/test_acp/test_server_dispatch.py | 102 | # mirrors what the eval harness produces from the default | COMMENT |
| MEDIUM | tests/agent/test_acp/test_action_methods.py | 957 | # Set up the cancel-propagation harness in a real task group. | COMMENT |
| MEDIUM | tests/agent/test_acp/test_approval.py | 1204 | # Scan all blocks for the view markdown — robust to layout | COMMENT |
| MEDIUM | tests/agent/test_acp/test_tui/test_cancel_card.py | 242 | # Don't call original_notify — the harness can't render toasts. | COMMENT |
| MEDIUM⚡ | tests/agent/test_acp/test_tui/test_widgets.py | 63 | # Single-widget harness | COMMENT |
| MEDIUM | tests/agent/test_acp/test_tui/test_widgets.py | 1099 | # comprehensive fingerprint should detect the change and the | COMMENT |
| MEDIUM | examples/skills/skills/system-info/scripts/sysinfo.sh | 3 | # Outputs comprehensive system details in a structured format | COMMENT |
| MEDIUM | .github/workflows/test.yml | 1 | # Test harness shell. GitHub only exposes workflow_dispatch for workflows that | COMMENT |
| MEDIUM | …x_tools/src/inspect_sandbox_tools/_util/user_switch.py | 39 | """The identity a child process must switch to in order to run as ``user``. | STRING |
| LOW | src/inspect_ai/_util/notebook.py | 53 | # notebook cells, if not just return an empty module | COMMENT |
| LOW | src/inspect_ai/analysis/_dataframe/record.py | 391 | # For now, I'll just return an empty dict | COMMENT |
| LOW | src/inspect_ai/util/_limit.py | 228 | # gone out of scope for the sample so we just return that snapshot | COMMENT |
| MEDIUM | src/inspect_ai/util/_checkpoint/checkpointer_impl.py | 581 | # checkpoint span. Skip on the harness-driven final fire — | COMMENT |
| MEDIUM | src/inspect_ai/util/_sandbox/service.py | 568 | # under an eval harness -- a sandbox-awareness leak that was | COMMENT |
| MEDIUM | src/inspect_ai/agent/_react.py | 228 | # to clean exit and the harness will run scoring next. State | COMMENT |
| MEDIUM | src/inspect_ai/agent/_react.py | 472 | # to clean exit and the harness will run scoring next. State | COMMENT |
| LOW | src/inspect_ai/agent/_acp/tui/session_screen.py | 184 | # …); duplicating them in the footer would just add noise. | COMMENT |
| MEDIUM | src/inspect_ai/agent/_acp/tui/widgets/cancel_card.py | 232 | # ``fails_on_error`` is True, but stay robust if a future | COMMENT |
| MEDIUM | src/inspect_ai/agent/_acp/tui/widgets/cancel_card.py | 267 | # but stay robust if a future refactor changes the | COMMENT |
| MEDIUM | src/inspect_ai/agent/_acp/tui/widgets/transcript.py | 158 | # Gate on fingerprint change — the comprehensive | COMMENT |
| LOW | src/inspect_ai/_view/common.py | 647 | # then just return that file | COMMENT |
| MEDIUM | src/inspect_ai/model/_call_tools.py | 1100 | # harness injects the real AgentState below) | COMMENT |
| LOW | src/inspect_ai/model/_call_tools.py | 1324 | # if the function takes **kwargs: Any then just pass the tool arguments through | COMMENT |
| LOW | src/inspect_ai/model/_providers/openai.py | 354 | # side step that complexity and just use two different batchers. | COMMENT |
| LOW | src/inspect_ai/log/_recorders/buffer/database.py | 745 | # this will just return no events and no attachments until the | COMMENT |
| LOW | src/inspect_ai/log/_recorders/buffer/filestore.py | 469 | # we'll just return None | COMMENT |
| LOW | src/inspect_ai/_eval/loader.py | 664 | # if there is no solver file then just create from the registry by name | COMMENT |
| MEDIUM | src/inspect_ai/tool/_sandbox_tools_utils/sandbox.py | 637 | # TODO: once we get the github CI/CD actions robust, this should be fatal | COMMENT |
| LOW | src/inspect_ai/tool/_mcp/_local.py | 360 | # to the session, we can just return nit | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/model/test_message_ids.py | 177 | # Output message should have an ID | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 117 | # Check if the model used server-side tools | COMMENT |
| LOW | tests/model/test_stable_message_ids.py | 157 | # Output message should have ID | COMMENT |
| LOW | tests/model/providers/test_perplexity.py | 83 | # Check if citations are present | COMMENT |
| LOW | tests/model/providers/util/test_batch.py | 102 | # Check if batch should fail | COMMENT |
| LOW | tests/model/providers/util/test_batch.py | 108 | # Check if batch is "complete" based on elapsed time | COMMENT |
| LOW | tests/log/test_eval_log.py | 194 | # Set timestamp to a timezone-aware datetime object because when serializing to | COMMENT |
| LOW | examples/structured.py | 170 | # Check if the output matches the expected RGB format | STRING |
| LOW | scripts/pypi-release.py | 745 | # Check if there are any other versions present | COMMENT |
| LOW | scripts/pypi-release.py | 805 | # Check if .pypirc exists or environment variables are set | COMMENT |
| LOW | scripts/pypi-release.py | 929 | # Check if we're ahead of remote | COMMENT |
| LOW | scripts/pypi-release.py | 972 | # Check if we're in a git repository | COMMENT |
| LOW | scripts/pypi-release.py | 1022 | # Check if branch is up to date | COMMENT |
| LOW | scripts/pypi-release.py | 1033 | # Check if tag already exists | COMMENT |
| LOW | scripts/pypi-release.py | 1165 | # Check if correct version already exists | COMMENT |
| LOW | …t_tool_support/src/inspect_tool_support/_cli/server.py | 27 | # Set umask to handle dynamic user switching scenarios: | COMMENT |
| LOW | …ox_tools/_in_process_tools/_text_editor/text_editor.py | 107 | # Read the file content | COMMENT |
| LOW | …ox_tools/_in_process_tools/_text_editor/text_editor.py | 112 | # Check if old_str is unique in the file | COMMENT |
| LOW | …ox_tools/_in_process_tools/_text_editor/text_editor.py | 215 | # Check if path exists | COMMENT |
| LOW | …ox_tools/_in_process_tools/_text_editor/text_editor.py | 224 | # Check if the path points to a directory | COMMENT |
| LOW | …ols/src/inspect_sandbox_tools/_util/pseudo_terminal.py | 68 | # Close the file descriptors | COMMENT |
| LOW | …spect_sandbox_tools/_remote_tools/_exec_remote/_job.py | 154 | # Check if process has finished | COMMENT |
| LOW | src/inspect_ai/_util/generate_config_args.py | 129 | # Check if it's a file path | COMMENT |
| LOW | src/inspect_ai/_util/generate_config_args.py | 149 | # Check if it looks like a file path that doesn't exist | COMMENT |
| LOW | src/inspect_ai/_util/platform.py | 53 | # Check if running in VS Code Jupyter notebook or interactive window | COMMENT |
| LOW | src/inspect_ai/_util/platform.py | 61 | # Check if running in a VS Code terminal | COMMENT |
| LOW | src/inspect_ai/_util/local_server.py | 162 | os.kill(pid, 0) # Check if process exists | CODE |
| LOW | src/inspect_ai/_util/local_server.py | 254 | # Check if the process is still alive | COMMENT |
| LOW | src/inspect_ai/_util/file.py | 456 | # Check if it's already a URI | COMMENT |
| LOW | src/inspect_ai/_util/file.py | 572 | # Open the file using fsspec and retrieve the file's information | COMMENT |
| LOW | src/inspect_ai/analysis/_dataframe/samples/table.py | 220 | # Check if it's specifically the pyarrow offset overflow error | COMMENT |
| LOW | src/inspect_ai/analysis/_dataframe/samples/table.py | 612 | # Check if this is a PyArrow-backed column | COMMENT |
| LOW | src/inspect_ai/util/_json.py | 385 | # Check if it's already Optional/Union with None | COMMENT |
| LOW | src/inspect_ai/util/_sandbox/self_check.py | 381 | # Write the file byte-by-byte from ASCII-only octal escapes, then cat it, so | COMMENT |
| LOW | src/inspect_ai/_cli/score.py | 311 | # Print the results panel | COMMENT |
| LOW | src/inspect_ai/model/_openai_responses.py | 363 | # Check if this is a compaction marker message | COMMENT |
| LOW | src/inspect_ai/model/_openai_responses.py | 1444 | # Check if this is a valid OpenAI action (correct type + required fields) | COMMENT |
| LOW | src/inspect_ai/model/_openai_responses.py | 1618 | # Check if result was cleared during compaction | COMMENT |
| LOW | src/inspect_ai/model/_model_info.py | 162 | # Check if the second-to-last part is a known service prefix | COMMENT |
| LOW | src/inspect_ai/model/_model_info.py | 168 | # Check if this is a hosting provider that needs org detection | COMMENT |
| LOW | src/inspect_ai/model/_providers/mockllm.py | 51 | # Check if it's a callable function | COMMENT |
| LOW | src/inspect_ai/model/_providers/google.py | 2046 | # Check if this block has an associated thought_signature and | COMMENT |
| LOW | src/inspect_ai/model/_providers/anthropic.py | 1127 | # Check if model doesn't support compaction | COMMENT |
| LOW | src/inspect_ai/model/_providers/anthropic.py | 5589 | # Check if result was cleared during compaction | COMMENT |
| LOW | src/inspect_ai/model/_providers/anthropic.py | 5806 | # Check if previous message has corresponding tool_use blocks | COMMENT |
| LOW | src/inspect_ai/model/_providers/anthropic.py | 5848 | # Check if next message has corresponding tool_result blocks | COMMENT |
| LOW | src/inspect_ai/model/_providers/sglang.py | 181 | # Check if process is still alive | COMMENT |
| LOW | src/inspect_ai/log/_bundle.py | 204 | # Open the file for writing to save the updated content | STRING |
| LOW | src/inspect_ai/event/_timeline_repr.py | 202 | # Check if span overlaps with any span in the current cluster | COMMENT |
| LOW | …spect_ai/tool/_sandbox_tools_utils/validate_distros.py | 122 | # Check if it's for a supported architecture | COMMENT |
| LOW | src/inspect_ai/tool/_sandbox_tools_utils/sandbox.py | 858 | # Check if we're in a git repo | COMMENT |
| LOW | src/inspect_ai/tool/_tools/_memory.py | 281 | # Read file | COMMENT |
| LOW | src/inspect_ai/tool/_tools/_memory.py | 349 | # Read file content | COMMENT |
| LOW | src/inspect_ai/tool/_tools/_memory.py | 354 | # Check if old_str is unique | COMMENT |
| LOW | …_tools/_computer/_resources/entrypoint/xvfb_startup.sh | 30 | # Check if Xvfb is already running | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/log/test_eval_log.py | 2648 | CODE | |
| LOW | …/_remote_tools/_web_browser/accessibility_tree_node.py | 224 | CODE | |
| LOW | …sandbox_tools/_remote_tools/_mcp/mcp_server_session.py | 143 | CODE | |
| LOW | src/inspect_ai/_control/server.py | 882 | CODE | |
| LOW | src/inspect_ai/_control/server.py | 1395 | CODE | |
| LOW | src/inspect_ai/_control/server.py | 1487 | CODE | |
| LOW | src/inspect_ai/_control/limits.py | 155 | CODE | |
| LOW | src/inspect_ai/_control/limits.py | 242 | CODE | |
| LOW | src/inspect_ai/_control/events.py | 141 | CODE | |
| LOW | src/inspect_ai/_control/events.py | 204 | CODE | |
| LOW | src/inspect_ai/_control/eval_state.py | 446 | CODE | |
| LOW | src/inspect_ai/_control/eval_state.py | 503 | CODE | |
| LOW | src/inspect_ai/_control/log_dir/samples.py | 114 | CODE | |
| LOW | src/inspect_ai/util/_subprocess.py | 74 | CODE | |
| LOW | src/inspect_ai/util/_subprocess.py | 88 | CODE | |
| LOW | src/inspect_ai/util/_subprocess.py | 101 | CODE | |
| LOW | src/inspect_ai/util/_subprocess.py | 159 | CODE | |
| LOW | src/inspect_ai/util/_subprocess.py | 200 | CODE | |
| LOW | src/inspect_ai/util/_subtask.py | 155 | CODE | |
| LOW | src/inspect_ai/util/_checkpoint/hydrate.py | 512 | CODE | |
| LOW | src/inspect_ai/util/_sandbox/local.py | 112 | CODE | |
| LOW | src/inspect_ai/util/_sandbox/docker/compose.py | 75 | CODE | |
| LOW | src/inspect_ai/util/_sandbox/docker/compose.py | 342 | CODE | |
| LOW | src/inspect_ai/dataset/_dataset.py | 32 | CODE | |
| LOW | src/inspect_ai/dataset/_sources/file.py | 14 | CODE | |
| LOW | src/inspect_ai/dataset/_sources/csv.py | 53 | CODE | |
| LOW | src/inspect_ai/dataset/_sources/json.py | 27 | CODE | |
| LOW | src/inspect_ai/dataset/_sources/hf.py | 123 | CODE | |
| LOW | src/inspect_ai/agent/_react.py | 57 | CODE | |
| LOW | src/inspect_ai/agent/_bridge/bridge.py | 101 | CODE | |
| LOW | src/inspect_ai/agent/_bridge/types.py | 57 | CODE | |
| LOW | src/inspect_ai/agent/_bridge/sandbox/bridge.py | 51 | CODE | |
| LOW | src/inspect_ai/agent/_bridge/sandbox/types.py | 42 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/agent_tool.py | 277 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/plan.py | 35 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/subagent.py | 56 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/research.py | 35 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/prompt.py | 234 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/deepagent.py | 50 | CODE | |
| LOW | src/inspect_ai/agent/_deepagent/general.py | 32 | CODE | |
| LOW | src/inspect_ai/solver/_basic_agent.py | 52 | CODE | |
| LOW | src/inspect_ai/solver/_task_state.py | 144 | CODE | |
| LOW | src/inspect_ai/scorer/_model.py | 35 | CODE | |
| LOW | src/inspect_ai/scorer/_model.py | 115 | CODE | |
| LOW | src/inspect_ai/_view/fastapi_server.py | 148 | CODE | |
| LOW | src/inspect_ai/_view/fastapi_server.py | 808 | CODE | |
| LOW | src/inspect_ai/_view/view.py | 24 | CODE | |
| LOW | src/inspect_ai/_cli/_scanner.py | 23 | CODE | |
| LOW | src/inspect_ai/_cli/_scanner.py | 91 | CODE | |
| LOW | src/inspect_ai/_cli/eval.py | 1936 | CODE | |
| LOW | src/inspect_ai/_cli/ctl/_fetch.py | 485 | CODE | |
| LOW | src/inspect_ai/_cli/ctl/_sample_read.py | 448 | CODE | |
| LOW | src/inspect_ai/_cli/ctl/_config.py | 421 | CODE | |
| LOW | src/inspect_ai/_cli/ctl/_config.py | 933 | CODE | |
| LOW | src/inspect_ai/model/_model.py | 2236 | CODE | |
| LOW | src/inspect_ai/model/_providers/vllm.py | 161 | CODE | |
| LOW | src/inspect_ai/model/_providers/nnterp.py | 54 | CODE | |
| LOW | src/inspect_ai/model/_providers/openai_compatible.py | 79 | CODE | |
| LOW | src/inspect_ai/model/_providers/openai.py | 119 | CODE | |
| LOW | src/inspect_ai/log/_condense.py | 260 | CODE | |
| 16 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| CRITICAL | tests/_view/test_view_server.py | 1413 | buffer = inspect_ai.log._recorders.buffer.filestore.SampleBufferFilestore( | CODE |
| CRITICAL | tests/_view/test_view_server.py | 1768 | inspect_ai.log._recorders.buffer.filestore.SampleManifest( | CODE |
| CRITICAL | tests/_view/test_view_server.py | 1779 | inspect_ai.log._recorders.buffer.filestore.Segment( | CODE |
| CRITICAL | tests/_view/test_view_server.py | 1786 | manifest = inspect_ai.log._recorders.buffer.filestore.Manifest( | CODE |
| CRITICAL | tests/model/test_should_retry_classification.py | 797 | from azure.core.exceptions import HttpResponseError | CODE |
| CRITICAL | tests/model/test_should_retry_classification.py | 812 | from azure.core.exceptions import HttpResponseError | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | design/recover.md | 434 | ### Step 1: Read crashed .eval files (done: `162628a3f`) | COMMENT |
| LOW⚡ | design/recover.md | 444 | ### Step 2: Read recovery data from sample buffer database (done: `664a9a43b`) | COMMENT |
| LOW⚡ | design/recover.md | 454 | ### Step 3: Reconstruct EvalSample from buffer DB data (done: `a05ce7fda`) | COMMENT |
| LOW | design/recover.md | 468 | ### Step 4: Write recovered .eval file (done: `493de0a1d`) | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 101 | # Step 1: Initial request with server-side tools | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 129 | # Step 2: Follow-up to build more history | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 143 | # Step 3: Another follow-up | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 158 | # Step 4: Compact history - clear older tool results | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 181 | # Step 5: Final request with compacted history - THIS SHOULD NOT CAUSE 400 ERROR | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 90 | # Step 2: Execute tool calls if present | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 95 | # Step 3: Ask follow-up question | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 112 | # Step 4: Add thank you message | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 115 | # Step 5: Compact history - remove ALL thinking blocks | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 145 | # Step 6: Call model with compacted history | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 152 | # Step 7: This should NOT raise an error | COMMENT |
| LOW | tests/model/test_compaction_edit_thinking.py | 61 | # Step 1: System prompt that STRONGLY requires tool use + user prompt | COMMENT |
| LOW | tests/model/test_web_search_exclude_none.py | 97 | # Step 1: Convert OpenAI Response to ChatMessageAssistant | COMMENT |
| LOW | tests/model/test_web_search_exclude_none.py | 136 | # Step 2: Convert ChatMessageAssistant back to OpenAI Input Items | COMMENT |
| LOW⚡ | tests/model/providers/test_google.py | 1473 | # Step 1: Simulate model returning an image via inline_data | COMMENT |
| LOW | tests/model/providers/test_google.py | 1496 | # Step 2: Build assistant message and convert back to Google Content | COMMENT |
| LOW⚡ | tests/model/providers/test_openai_responses.py | 1490 | # Step 2: Build an assistant message with the image content | COMMENT |
| LOW⚡ | tests/model/providers/test_openai_responses.py | 1497 | # Step 3: Convert back to input items (simulating the next turn) | COMMENT |
| LOW | tests/model/providers/test_openai_responses.py | 1473 | # Step 1: Model returns an image generation call | COMMENT |
| LOW | src/inspect_ai/model/_providers/_anthropic_citations.py | 109 | # NOTE: This function cannot assume that this module was responsible for the | COMMENT |
| LOW | src/inspect_ai/log/_log.py | 1306 | # WARNING: The order of these fields is important for the log file format. | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 191 | # Step 1: Read the crashed .eval file metadata | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 228 | # Step 2: Read buffer DB metadata (lightweight — just summaries) | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 260 | # Step 3: Determine recovery path and write | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 303 | # Step 4: Stream all samples into the recovered file. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | design/litellm-proxy.md | 503 | - The `get_model_info()` dummy-key path already catches exceptions and returns | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 280 | "fake-key" if key == "TAVILY_API_KEY" else default | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 289 | "fake-key" if key == "TAVILY_API_KEY" else default | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 298 | "fake-key" if key == "TAVILY_API_KEY" else default | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 307 | return_value=("fake-key", "fake-cse-id"), | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 314 | return_value=("fake-key", "fake-cse-id"), | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 322 | "fake-key" if key == "EXA_API_KEY" else default | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 331 | "fake-key" if key == "EXA_API_KEY" else default | CODE |
| LOW⚡ | tests/tools/test_web_search.py | 340 | "fake-key" if key == "EXA_API_KEY" else default | CODE |
| LOW | tests/tools/test_web_search.py | 365 | return_value=("fake-key", "fake-cse-id"), | CODE |
| LOW | tests/tools/test_web_search.py | 396 | return_value=("fake-key", "fake-cse-id"), | CODE |
| LOW | tests/tools/test_web_search_exa.py | 40 | with patch.dict("os.environ", {"EXA_API_KEY": "dummy-key"}): | CODE |
| LOW⚡ | tests/tools/test_web_search_tavily.py | 59 | with patch.dict("os.environ", {"TAVILY_API_KEY": "dummy-key"}): | CODE |
| LOW | tests/tools/test_web_search_tavily.py | 113 | with patch.dict("os.environ", {"TAVILY_API_KEY": "dummy-key"}): | CODE |
| LOW⚡ | tests/tools/test_web_search_tavily.py | 127 | with patch.dict("os.environ", {"TAVILY_API_KEY": "dummy-key"}): | CODE |
| LOW⚡ | tests/tools/test_web_search_tavily.py | 144 | with patch.dict("os.environ", {"TAVILY_API_KEY": "dummy-key"}): | CODE |
| LOW | tests/tools/test_web_search_google.py | 94 | mock_get_keys.return_value = ("dummy-key", "dummy-cse-id") | CODE |
| LOW | tests/tools/test_web_search_google.py | 141 | mock_get_keys.return_value = ("dummy-key", "dummy-cse-id") | CODE |
| LOW | tests/tools/test_web_search_google.py | 174 | mock_get_keys.return_value = ("dummy-key", "dummy-cse-id") | CODE |
| LOW⚡ | tests/util/test_store_nested_types.py | 97 | "street": "123 Main St", | CODE |
| LOW | tests/util/test_format_template.py | 171 | address=Address(street="123 Main St", city="Anytown"), | CODE |
| LOW⚡ | tests/util/test_json_schema.py | 169 | instance = Model(name="Alice", address={"street": "123 Main St", "city": "NYC"}) | CODE |
| LOW⚡ | tests/util/test_json_schema.py | 171 | assert instance.address.street == "123 Main St" | CODE |
| LOW | tests/util/test_json_schema.py | 200 | "contact": {"email": "bob@example.com", "phone": "555-1234"}, | CODE |
| LOW | tests/agent/test_sandbox_agent_bridge_media.py | 27 | api_key="local-fake-key", | CODE |
| LOW | tests/model/test_token_counting_integration.py | 72 | assert result[0]["name"] == "placeholder" | CODE |
| LOW⚡ | tests/model/providers/test_vllm.py | 37 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW | tests/model/providers/test_vllm.py | 318 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW | tests/model/providers/test_litellm_proxy_reasoning.py | 167 | id="placeholder", | CODE |
| LOW⚡ | tests/model/providers/test_hf.py | 291 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW⚡ | tests/model/providers/test_hf.py | 293 | assert chat == "user: Lorem ipsum dolor\n" | CODE |
| LOW | tests/model/providers/test_hf.py | 67 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW | tests/model/providers/test_hf.py | 93 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW | tests/model/providers/test_hf.py | 373 | message = ChatMessageUser(content="Lorem ipsum dolor") | CODE |
| LOW | tests/model/providers/test_hf.py | 375 | assert chat == "[user] Lorem ipsum dolor" | CODE |
| LOW | examples/http_proxy/claude.py | 56 | "FUTUREMODEL_API_KEY": "fm-fake-key-for-demo", | CODE |
| LOW | …ct_tool_support/_remote_tools/_web_browser/cdp/a11y.py | 94 | "attribute", "implicit", "stylet", "contents", "placeholder", "relatedElement" | CODE |
| LOW⚡ | src/inspect_ai/model/_openai_responses.py | 1224 | param["id"] = "dummy-id" | CODE |
| LOW | src/inspect_ai/model/_openai_responses.py | 2625 | "name": "placeholder", | CODE |
| LOW | src/inspect_ai/model/_providers/anthropic.py | 5827 | name="placeholder", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/model/test_compaction_edit_server_tools.py | 101 | # Step 1: Initial request with server-side tools | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 129 | # Step 2: Follow-up to build more history | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 143 | # Step 3: Another follow-up | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 158 | # Step 4: Compact history - clear older tool results | COMMENT |
| LOW | tests/model/test_compaction_edit_server_tools.py | 181 | # Step 5: Final request with compacted history - THIS SHOULD NOT CAUSE 400 ERROR | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 90 | # Step 2: Execute tool calls if present | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 95 | # Step 3: Ask follow-up question | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 112 | # Step 4: Add thank you message | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 115 | # Step 5: Compact history - remove ALL thinking blocks | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 145 | # Step 6: Call model with compacted history | COMMENT |
| LOW⚡ | tests/model/test_compaction_edit_thinking.py | 152 | # Step 7: This should NOT raise an error | COMMENT |
| LOW | tests/model/test_compaction_edit_thinking.py | 61 | # Step 1: System prompt that STRONGLY requires tool use + user prompt | COMMENT |
| LOW | tests/model/test_web_search_exclude_none.py | 97 | # Step 1: Convert OpenAI Response to ChatMessageAssistant | COMMENT |
| LOW | tests/model/test_web_search_exclude_none.py | 136 | # Step 2: Convert ChatMessageAssistant back to OpenAI Input Items | COMMENT |
| LOW⚡ | tests/model/providers/test_google.py | 1473 | # Step 1: Simulate model returning an image via inline_data | COMMENT |
| LOW | tests/model/providers/test_google.py | 1496 | # Step 2: Build assistant message and convert back to Google Content | COMMENT |
| LOW⚡ | tests/model/providers/test_openai_responses.py | 1490 | # Step 2: Build an assistant message with the image content | COMMENT |
| LOW⚡ | tests/model/providers/test_openai_responses.py | 1497 | # Step 3: Convert back to input items (simulating the next turn) | COMMENT |
| LOW | tests/model/providers/test_openai_responses.py | 1473 | # Step 1: Model returns an image generation call | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 191 | # Step 1: Read the crashed .eval file metadata | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 228 | # Step 2: Read buffer DB metadata (lightweight — just summaries) | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 260 | # Step 3: Determine recovery path and write | COMMENT |
| LOW | src/inspect_ai/log/_recover/_api.py | 303 | # Step 4: Stream all samples into the recovered file. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/tools/test_tool_error.py | 75 | CODE | |
| MEDIUM | …ts/agent/test_acp/test_tui/test_inline_request_card.py | 129 | CODE | |
| MEDIUM | tests/model/providers/test_mistral.py | 956 | CODE | |
| MEDIUM | tests/model/providers/test_sagemaker.py | 1470 | CODE | |
| MEDIUM | tests/model/providers/test_openai.py | 714 | CODE | |
| MEDIUM | src/inspect_ai/agent/_acp/tui/widgets/cancel_card.py | 162 | CODE | |
| MEDIUM | src/inspect_ai/agent/_acp/tui/widgets/approval_card.py | 112 | CODE | |
| MEDIUM | …spect_ai/agent/_acp/tui/widgets/inline_request_card.py | 115 | CODE | |
| MEDIUM | …spect_ai/agent/_acp/tui/widgets/inline_request_card.py | 126 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/model/providers/test_bedrock_adaptive_thinking.py | 360 | # the base Sonnet 5 keeps forced tool choice as requested | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_flaky_retry.py | 168 | def test_function(): | CODE |
| LOW | tests/review/test_review.py | 540 | def helper() -> Agent: | CODE |
| LOW | …t_tool_support/src/inspect_tool_support/_cli/server.py | 16 | async def handle_request(request: Request) -> Response: | CODE |
| LOW | …sandbox_tools/src/inspect_sandbox_tools/_cli/server.py | 138 | async def handle_request(request: Request) -> Response: | CODE |
| LOW | src/inspect_ai/_util/_async.py | 64 | async def run_task(func: Callable[[], Awaitable[T]], index: int) -> None: | CODE |
| LOW | src/inspect_ai/util/_collect.py | 38 | async def run_task(index: int, task: Awaitable[T]) -> None: | CODE |