Our library for RL environments + evals
This report summarizes scanner evidence for PrimeIntellect-ai/verifiers, a Python project with 4,694 GitHub stars. SynthScan examined 45,905 lines of code across 345 source files, recording 955 pattern matches distributed across 19 syntactic categories. The legacy adjusted pattern signal is 31.1 (High pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 305 finding(s).
Are imports declared and package/API names verified? 434 finding(s).
Are wrappers, guards, and repeated structures necessary? 343 finding(s).
Which changes need focused human review? 955 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 955 distinct pattern matches across 19 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ronments/nemo_gym_weather/nemo_gym_weather/__init__.py | 1 | CODE | |
| LOW | …ronments/nemo_gym_weather/nemo_gym_weather/__init__.py | 2 | CODE | |
| LOW | …yscale_interception/grayscale_interception/__init__.py | 1 | CODE | |
| LOW | environments/deepwiki/deepwiki/__init__.py | 1 | CODE | |
| LOW | environments/compact/compact/__init__.py | 4 | CODE | |
| LOW | environments/compact/compact/__init__.py | 4 | CODE | |
| LOW | …vironments/proposer_solver/proposer_solver/__init__.py | 1 | CODE | |
| LOW | …vironments/proposer_solver/proposer_solver/__init__.py | 1 | CODE | |
| LOW | …vironments/proposer_solver/proposer_solver/__init__.py | 1 | CODE | |
| LOW | environments/kuhn_poker/kuhn_poker/__init__.py | 1 | CODE | |
| LOW | environments/kuhn_poker/kuhn_poker/__init__.py | 1 | CODE | |
| LOW | environments/interception/interception/__init__.py | 1 | CODE | |
| LOW | environments/openenv_wordle/openenv_wordle/taskset.py | 6 | CODE | |
| LOW | environments/openenv_wordle/openenv_wordle/__init__.py | 1 | CODE | |
| LOW | environments/openenv_wordle/openenv_wordle/__init__.py | 1 | CODE | |
| LOW | environments/wiki_search/wiki_search/__init__.py | 1 | CODE | |
| LOW | environments/wiki_search/wiki_search/__init__.py | 2 | CODE | |
| LOW | environments/scratchpad/scratchpad/__init__.py | 1 | CODE | |
| LOW | …earch_interception/web_search_interception/__init__.py | 1 | CODE | |
| LOW | …earch_interception/web_search_interception/__init__.py | 1 | CODE | |
| LOW | …earch_interception/web_search_interception/__init__.py | 2 | CODE | |
| LOW | environments/reverse_text/reverse_text/__init__.py | 1 | CODE | |
| LOW | environments/wordle/wordle/__init__.py | 1 | CODE | |
| LOW | environments/wordle/wordle/__init__.py | 2 | CODE | |
| LOW | environments/alphabet_sort/alphabet_sort/__init__.py | 1 | CODE | |
| LOW | environments/alphabet_sort/alphabet_sort/__init__.py | 1 | CODE | |
| LOW | environments/gsm8k/gsm8k/__init__.py | 1 | CODE | |
| LOW | …nments/bash_interception/bash_interception/__init__.py | 1 | CODE | |
| LOW | environments/color_codeword/color_codeword/__init__.py | 1 | CODE | |
| LOW | environments/color_codeword/color_codeword/__init__.py | 1 | CODE | |
| LOW | environments/glossary/glossary/__init__.py | 1 | CODE | |
| LOW | environments/code_golf/code_golf/__init__.py | 1 | CODE | |
| LOW | environments/code_golf/code_golf/__init__.py | 1 | CODE | |
| LOW | verifiers/v1/trace.py | 1 | CODE | |
| LOW | verifiers/v1/task.py | 10 | CODE | |
| LOW | verifiers/v1/harness.py | 1 | CODE | |
| LOW | verifiers/v1/semantic.py | 3 | CODE | |
| LOW | verifiers/v1/graph.py | 21 | CODE | |
| LOW | verifiers/v1/taskset.py | 22 | CODE | |
| LOW | verifiers/v1/session.py | 17 | CODE | |
| LOW | verifiers/v1/__init__.py | 3 | CODE | |
| LOW | verifiers/v1/__init__.py | 5 | CODE | |
| LOW | verifiers/v1/__init__.py | 5 | CODE | |
| LOW | verifiers/v1/__init__.py | 5 | CODE | |
| LOW | verifiers/v1/__init__.py | 6 | CODE | |
| LOW | verifiers/v1/__init__.py | 6 | CODE | |
| LOW | verifiers/v1/__init__.py | 6 | CODE | |
| LOW | verifiers/v1/__init__.py | 6 | CODE | |
| LOW | verifiers/v1/__init__.py | 6 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 7 | CODE | |
| LOW | verifiers/v1/__init__.py | 16 | CODE | |
| LOW | verifiers/v1/__init__.py | 17 | CODE | |
| LOW | verifiers/v1/__init__.py | 17 | CODE | |
| LOW | verifiers/v1/__init__.py | 21 | CODE | |
| 330 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …onments/bash_interception/bash_interception/taskset.py | 8 | REWRITE_COMMAND = f"git --version && touch {REWRITE_SENTINEL}" | CODE |
| HIGH | verifiers/v1/agent.py | 649 | "one (e.g. bash or null)." | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 11 | node_ok() { "$node/bin/node" -e 'const [a,b]=process.versions.node.split(".").map(Number); process.exit(a>22 || a===22 & | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 11 | node_ok() { "$node/bin/node" -e 'const [a,b]=process.versions.node.split(".").map(Number); process.exit(a>22 || a===22 & | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 15 | if ! node -e 'const [a,b]=process.versions.node.split(".").map(Number); process.exit(a>22 || a===22 && b>=19 ? 0 : 1 | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 15 | if ! node -e 'const [a,b]=process.versions.node.split(".").map(Number); process.exit(a>22 || a===22 && b>=19 ? 0 : 1 | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 25 | && ! (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null); then | CODE |
| HIGH⚡ | verifiers/v1/harnesses/node.py | 25 | && ! (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null); then | CODE |
| HIGH | verifiers/v1/harnesses/node.py | 69 | if [ ! -x "$node/bin/node" ] || [ "$("$node/bin/node" --version 2>/dev/null)" != "v$VF_NODE_VERSION" ]; then | CODE |
| HIGH | verifiers/v1/harnesses/prime_agent/harness.py | 77 | fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 22 | command -v curl >/dev/null || (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null) | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 22 | command -v curl >/dev/null || (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null) | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 73 | [ -z "$acp_pid" ] || kill -TERM -"$acp_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 74 | [ -z "$gateway_pid" ] || kill -TERM -"$gateway_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 76 | [ -z "$acp_pid" ] || kill -KILL -"$acp_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 77 | [ -z "$gateway_pid" ] || kill -KILL -"$gateway_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 78 | [ -z "$acp_pid" ] || wait "$acp_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/openclaw/harness.py | 79 | [ -z "$gateway_pid" ] || wait "$gateway_pid" 2>/dev/null || true | CODE |
| HIGH | verifiers/v1/harnesses/openclaw/harness.py | 93 | if ! kill -0 "$gateway_pid" 2>/dev/null; then | CODE |
| HIGH | verifiers/v1/harnesses/openclaw/harness.py | 94 | wait "$gateway_pid" 2>/dev/null || true | CODE |
| HIGH⚡ | verifiers/v1/harnesses/kimi_code/harness.py | 26 | if [ -x "$bin" ] && [ "$("$bin" --version 2>/dev/null)" = "{version}" ]; then | CODE |
| HIGH⚡ | verifiers/v1/harnesses/kimi_code/harness.py | 29 | command -v curl >/dev/null || { apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null; } | CODE |
| HIGH⚡ | verifiers/v1/harnesses/kimi_code/harness.py | 29 | command -v curl >/dev/null || { apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null; } | CODE |
| HIGH⚡ | verifiers/v1/harnesses/utils/install.py | 42 | f'until ln -s "$me" {lock} 2>/dev/null; do owner=$(readlink {lock} 2>/dev/null); ' | CODE |
| HIGH⚡ | verifiers/v1/harnesses/utils/install.py | 45 | f'[ "$(readlink {lock} 2>/dev/null)" != "$owner" ] || rm -f {lock}; fi; ' | CODE |
| HIGH⚡ | verifiers/v1/harnesses/utils/install.py | 47 | f'trap \'[ "$(readlink {lock} 2>/dev/null)" != "$me" ] || rm -f {lock}\' EXIT; {run}' | CODE |
| HIGH⚡ | verifiers/v1/harnesses/utils/install.py | 51 | f'if l=$(command -v flock || command -v lockf); then "$l" {lock} {run}; else {spinlock}; fi' | CODE |
| HIGH | verifiers/v1/harnesses/pi/harness.py | 39 | if [ "$(cat "$packages/.versions" 2>/dev/null)" != "$versions" ]; then | CODE |
| HIGH | verifiers/v1/harnesses/rlm/harness.py | 164 | "{ apt-get update -qq && apt-get install -y -qq git; } && " | CODE |
| HIGH | verifiers/v1/harnesses/hermes_agent/harness.py | 20 | command -v curl >/dev/null || (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null) | CODE |
| HIGH | verifiers/v1/harnesses/hermes_agent/harness.py | 20 | command -v curl >/dev/null || (apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null) | CODE |
| HIGH | verifiers/v1/harnesses/terminus_2/harness.py | 69 | 'tmux kill-server >/dev/null 2>&1 || true; rm -rf "$TMUX_TMPDIR"', | CODE |
| HIGH | verifiers/v1/runtimes/container.py | 125 | 'kill -"$1" "-$2" 2>/dev/null || kill -"$1" "$2" 2>/dev/null', | CODE |
| HIGH | verifiers/v1/runtimes/container.py | 145 | 'kill -KILL "-$pid" 2>/dev/null || kill -KILL "$pid" 2>/dev/null || true; ' | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 33 | "{ command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1; } " | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 34 | "|| { apt-get update -qq && apt-get install -y -qq curl ca-certificates; } " | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 38 | "{ command -v curl >/dev/null 2>&1 && curl -LsSf https://astral.sh/uv/install.sh | sh; } " | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 39 | "|| { command -v wget >/dev/null 2>&1 && wget -qO- https://astral.sh/uv/install.sh | sh; }" | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 47 | "|| { pip install -q -U --user uv 2>/dev/null && hash -r && uv_supports_scripts; } " | CODE |
| HIGH⚡ | verifiers/v1/runtimes/base.py | 48 | f"|| {{ {{ {_INSTALL_CURL}; {_DOWNLOAD_UV}; }} && hash -r && uv_supports_scripts; }}" | CODE |
| HIGH | verifiers/v1/tasksets/harbor/compose.py | 52 | "apt-get update -qq && apt-get install -y -qq --no-install-recommends docker.io " | CODE |
| HIGH | verifiers/v1/tasksets/harbor/taskset.py | 223 | "tar --no-same-owner -xzf /tmp/environment.tgz && rm /tmp/environment.tgz", | CODE |
| HIGH | verifiers/v1/tasksets/harbor/taskset.py | 348 | "mkdir -p /tests && tar -xzf /tmp/tests.tgz -C /tests" | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …ayscale_interception/grayscale_interception/taskset.py | 13 | # The last request message is the new user input, before the harness stores it. | COMMENT |
| MEDIUM | environments/interception/interception/taskset.py | 11 | # A response stop runs while the response is buffered, before the harness | COMMENT |
| MEDIUM | …earch_interception/web_search_interception/__init__.py | 4 | # Exporting Codex beside the taskset makes it this example's default harness, so | COMMENT |
| MEDIUM | tests/v1/conftest.py | 23 | uv run pytest tests/v1 -n auto -m bash # only the bash harness | STRING |
| MEDIUM | tests/v1/conftest.py | 79 | # `compact` (an example harness) and `terminus-2` (drives the host tmux) are excluded from e2e. | COMMENT |
| MEDIUM | tests/v1/test_serve_delta.py | 126 | # the harness sends its next request: the tool result is previewed at once | COMMENT |
| MEDIUM | tests/v1/test_trace.py | 444 | # The base ACP layer resolves the generic edge set before harness-owned metadata. | COMMENT |
| MEDIUM⚡ | tests/v1/test_e2e.py | 245 | # harness runtime x tool placement: every axis value once plus the two-container case | COMMENT |
| MEDIUM⚡ | tests/v1/test_e2e.py | 246 | # (harness and tool in separate docker boxes) and a prime-colocated row (a tool in its | COMMENT |
| MEDIUM⚡ | tests/v1/test_e2e.py | 247 | # OWN prime sandbox needs port exposure; colocated rides the harness's box). | COMMENT |
| MEDIUM⚡ | tests/v1/test_e2e.py | 821 | # The tool was advertised to the masked chat, regardless of harness qualification. | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 129 | # harness x harness runtime: every harness once, both local runtimes hit (subprocess | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 148 | # harness x harness runtime for the shell task: every coding agent once (null is a chat | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 183 | # The scripted user runs in the eval process itself (no placement axis); the harness | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 200 | # retain MCP access after resuming. Cover every harness in the local container runtime, | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 508 | # The interception server captured the harness's advertised name and schema. Harnesses | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 783 | # assistant's harness while the task's rewards score off non-prompt fields | COMMENT |
| MEDIUM⚡ | tests/v1/test_e2e.py | 832 | harness=None, # both seats pin the null harness themselves | CODE |
| MEDIUM | tests/v1/test_e2e.py | 633 | harness=None, # both duet seats pin their own harness | CODE |
| MEDIUM | tests/v1/test_e2e.py | 662 | harness=None, # a multi-agent env refuses the run-level harness | CODE |
| MEDIUM | tests/v1/test_e2e.py | 864 | harness=None, # both duet seats pin their own harness | CODE |
| MEDIUM | tests/v1/test_delivery_behaviors.py | 191 | # Let the harness enter its native wait after reading the response. | COMMENT |
| MEDIUM | verifiers/v1/trace.py | 566 | # sampled response is the logical request result consumed by the harness. | COMMENT |
| MEDIUM | verifiers/v1/env.py | 119 | # Same harness config -> one loaded object (harnesses are stateless values). | COMMENT |
| MEDIUM | verifiers/v1/env.py | 130 | # Warn once per distinct harness; tool-less chat loops are exempt. | COMMENT |
| MEDIUM | verifiers/v1/env.py | 210 | # Unpinned fields fall back to the run's ctx / the taskset's harness. | COMMENT |
| MEDIUM | verifiers/v1/session.py | 472 | # The harness would run the call regardless; only ending the rollout keeps | COMMENT |
| MEDIUM | verifiers/v1/__init__.py | 292 | # taskset / harness / runtime / environment | COMMENT |
| MEDIUM | verifiers/v1/rollout.py | 267 | # Setup hooks and harness session preparation share one deadline. | COMMENT |
| MEDIUM | verifiers/v1/rollout.py | 286 | # The harness reaches the model at `{base_url}/v1`; tool servers reach this | COMMENT |
| MEDIUM | verifiers/v1/rollout.py | 481 | # Prefer an intercepted model/tool error to the harness exit it caused. | COMMENT |
| MEDIUM⚡ | verifiers/v1/rollout.py | 497 | # timeout stop. A TimeoutError from the harness's own I/O with no | COMMENT |
| MEDIUM⚡ | verifiers/v1/rollout.py | 504 | except Exception as e: # noqa: BLE001 - harness boundary records every rollout failure | CODE |
| MEDIUM | verifiers/v1/rollout.py | 523 | # A harness that completes cleanly after a failed model call handled it (e.g. it | COMMENT |
| MEDIUM | verifiers/v1/rollout.py | 525 | # harness that dies on it surfaces the stashed error through the except above. | COMMENT |
| MEDIUM | verifiers/v1/agent.py | 325 | # The harness may already have accepted input when the caller is | COMMENT |
| MEDIUM | verifiers/v1/agent.py | 636 | # Multi-turn capability is derived: a harness needs transcript replay, a | COMMENT |
| MEDIUM | verifiers/v1/agent.py | 976 | # `Agent.interaction()` may fail before yielding (e.g. task/harness setup). | COMMENT |
| MEDIUM | verifiers/v1/clients/train.py | 460 | # deterministic 400, so the harness SDK never retries it. | COMMENT |
| MEDIUM | verifiers/v1/clients/eval.py | 28 | # The harness uses this rollout secret to authenticate with the localhost server. | COMMENT |
| MEDIUM | verifiers/v1/clients/eval.py | 93 | # failure — map these to a retryable 502 so the harness SDK retries the | COMMENT |
| MEDIUM⚡ | verifiers/v1/interception/server.py | 75 | # Each session proxies one rollout's own harness requests, so aiohttp's default 1 MiB body | COMMENT |
| MEDIUM⚡ | verifiers/v1/interception/server.py | 77 | # and the harness gets a 413. Allow large bodies; the upstream provider and the model's | COMMENT |
| MEDIUM⚡ | verifiers/v1/interception/server.py | 82 | # its HTTP status (so the harness SDK can retry 5xx/429), and a longer one is kept alive | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 546 | # A malformed harness knob must not kill recording (this runs in the | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 640 | # This key identifies the harness-to-interception hop. The server owns its | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 765 | # The tail is what the harness added since the last turn (tool results, user | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 864 | # Stash the real cause; the rollout re-raises it after the harness returns. | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 865 | # Relay the provider's status so the harness SDK retries 5xx/429 and not 4xx. | COMMENT |
| MEDIUM | verifiers/v1/interception/server.py | 888 | # A cancelled exchange (harness disconnect, shutdown) is still | COMMENT |
| MEDIUM | verifiers/v1/harnesses/utils/core.py | 32 | # The harness bundles this module into the generated script before execution. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/utils/mcp.py | 121 | # Task.cancelling() is 3.11+; harness programs may run on 3.10. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/utils/compaction.py | 8 | # The harness bundles this module into the generated script before execution. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/bash/harness.py | 18 | # harness-injected prompt). The edit clause is appended only when the `edit` tool is enabled. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/bash/harness.py | 99 | # inheriting it via $SERPER_API_KEY / /proc/self/environ. Prefer a key set in the harness | COMMENT |
| MEDIUM | verifiers/v1/harnesses/bash/harness.py | 100 | # env (harness config env / forward_env); fall back to the host env only when the key is | COMMENT |
| MEDIUM | verifiers/v1/harnesses/browser_use/harness.py | 79 | # prompt, that transcript already contains this harness prompt. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/browser_use/program.py | 4 | # "browser-harness==0.1.13", | COMMENT |
| MEDIUM | verifiers/v1/harnesses/browser_use/program.py | 38 | # The harness bundles this module into the generated script before execution. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/browser_use/program.py | 106 | # browser-harness uses five seconds for this same DevTools HTTP probe. | COMMENT |
| 25 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/v1/conftest.py | 99 | def pytest_collection_modifyitems(config, items) -> None: | CODE |
| LOW | tests/v1/conftest.py | 110 | def _configure_prime_runtimes(config: dict) -> None: | CODE |
| LOW⚡ | tests/v1/test_judges.py | 159 | def test_judges_entry_requires_id(): | CODE |
| LOW⚡ | tests/v1/test_judges.py | 164 | def test_rubric_config_requires_path(): | CODE |
| LOW⚡ | tests/v1/test_judges.py | 170 | def test_judges_reject_shared_reward_keys(): | CODE |
| LOW | tests/v1/test_judges.py | 119 | def test_judge_plugin_resolution(): | CODE |
| LOW | tests/v1/test_judges.py | 137 | def test_taskset_config_narrows_judges(tmp_path): | CODE |
| LOW | tests/v1/test_judges.py | 200 | def test_reward_name_fallback(): | CODE |
| LOW | tests/v1/test_judges.py | 216 | async def test_base_judge_score_raises(): | CODE |
| LOW | tests/v1/test_judges.py | 288 | async def test_reference_score_messages_prompt(fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 312 | async def test_reference_question_field(fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 397 | def test_agentic_judge_trace_hidden_reasoning_toggle(): | CODE |
| LOW | tests/v1/test_judges.py | 430 | async def test_rubric_view_full_trace(tmp_path, fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 439 | async def test_config_prompt_overrides_class_template(tmp_path, fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 458 | async def test_reference_list_answer(fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 544 | def test_rubric_criteria_toml_and_json(tmp_path): | CODE |
| LOW | tests/v1/test_judges.py | 563 | def test_rubric_config_weight_overrides_file(tmp_path): | CODE |
| LOW | tests/v1/test_judges.py | 568 | def test_rubric_rejects_bad_files(tmp_path): | CODE |
| LOW | tests/v1/test_judges.py | 593 | def test_judge_composition_weights_are_finite(): | CODE |
| LOW | tests/v1/test_judges.py | 612 | async def test_rubric_verdict_mismatch_raises(tmp_path, monkeypatch): | CODE |
| LOW | tests/v1/test_judges.py | 629 | async def test_rubric_choices_normalize(tmp_path, monkeypatch): | CODE |
| LOW | tests/v1/test_judges.py | 642 | async def test_rubric_off_menu_answer_raises(tmp_path, monkeypatch): | CODE |
| LOW | tests/v1/test_judges.py | 655 | def test_rubric_choices_validation(tmp_path): | CODE |
| LOW | tests/v1/test_judges.py | 667 | async def test_rubric_reference_answer_optional(tmp_path, fake_judge_model): | CODE |
| LOW | tests/v1/test_judges.py | 695 | async def test_task_score_runs_plugged_judges(tmp_path, fake_judge_model): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 63 | async def test_connected_is_left_alone(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 67 | async def test_one_disconnected_check_is_not_enough(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 72 | async def test_disconnected_twice_restarts_once(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 77 | async def test_exited_frpc_restarts_without_asking(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 82 | async def test_unreachable_service_is_no_verdict(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 88 | async def test_gone_registration_stops_watching(registration, monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 94 | async def test_gone_reported_by_frpc_stops_watching(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 101 | async def test_restart_refused_as_gone_stops_watching(monkeypatch): | CODE |
| LOW⚡ | tests/v1/test_tunnel_watchdog.py | 107 | async def test_failed_restart_is_tried_again(monkeypatch): | CODE |
| LOW | tests/v1/test_platform_run.py | 42 | def test_a_run_without_attach_is_created_on_the_platform(init_calls): | CODE |
| LOW | tests/v1/test_platform_run.py | 53 | def test_attach_hands_the_launchers_id_to_the_sdk(init_calls): | CODE |
| LOW | tests/v1/test_platform_run.py | 66 | def test_a_failed_attach_fails_the_eval_instead_of_falling_back(monkeypatch): | CODE |
| LOW | tests/v1/test_platform_run.py | 78 | def test_a_failed_open_without_attach_still_falls_back_to_a_local_run(monkeypatch): | CODE |
| LOW | tests/v1/test_platform_run.py | 98 | def test_the_sdk_kill_switch_refuses_an_attach(init_calls, monkeypatch): | CODE |
| LOW | tests/v1/test_taskset.py | 59 | def test_iteration_sets_idx_to_the_load_position() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 63 | def test_include_keeps_tasks_that_any_list_names() -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 75 | def test_keys_match_the_keys_traces_record(tmp_path) -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 84 | def test_exclude_drops_tasks_that_any_list_names() -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 88 | def test_unmatched_entries_warn(caplog) -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 94 | def test_shuffle_is_seeded_and_reproducible() -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 99 | def test_skip_and_take_slice_the_stream() -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 105 | def test_views_chain_in_call_order_and_leave_the_base_alone() -> None: | CODE |
| LOW⚡ | tests/v1/test_taskset.py | 112 | def test_views_read_load_lazily() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 128 | def test_only_a_bounded_view_can_shuffle() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 148 | def test_task_match_config_parses_idx() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 157 | def test_select_config_defaults_and_short_aliases() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 174 | def test_default_select_yields_every_task() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 178 | def test_select_applies_its_steps_in_a_fixed_order() -> None: | CODE |
| LOW | tests/v1/test_taskset.py | 195 | def test_select_bounds_an_infinite_taskset() -> None: | CODE |
| LOW | tests/v1/test_scoring.py | 46 | async def test_config_plugged_fns_merge_and_override(tmp_path) -> None: | STRING |
| LOW | tests/v1/test_scoring.py | 92 | async def test_defer_scoring_defers_only_task_scoring() -> None: | STRING |
| LOW⚡ | tests/v1/test_scoring.py | 105 | def test_compare_stdout_results_accepts_token_equal_text() -> None: | STRING |
| LOW⚡ | tests/v1/test_scoring.py | 109 | def test_compare_stdout_results_keeps_numeric_tolerance() -> None: | STRING |
| LOW⚡ | tests/v1/test_scoring.py | 113 | def test_parse_pytest_outcomes_strips_xfail_xpass_reasons() -> None: | STRING |
| LOW | tests/v1/test_scoring.py | 129 | def test_parse_judge_choice_prefers_final_marker_then_boxed() -> None: | STRING |
| 66 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ronments/nemo_gym_weather/nemo_gym_weather/__init__.py | 4 | __all__ = ["NeMoGymEnv", "NeMoGymWeatherTaskset"] | CODE |
| LOW | …yscale_interception/grayscale_interception/__init__.py | 3 | __all__ = ["GrayscaleInterceptionTaskset"] | CODE |
| LOW | environments/deepwiki/deepwiki/__init__.py | 3 | __all__ = ["DeepWikiTaskset"] | CODE |
| LOW | environments/compact/compact/__init__.py | 6 | __all__ = ["CompactingHarness", "CompactingHarnessConfig"] | CODE |
| LOW | …vironments/proposer_solver/proposer_solver/__init__.py | 7 | __all__ = ["ProposerSolverEnv", "ProposerSolverEnvConfig", "ProposerSolverTaskset"] | CODE |
| LOW | environments/kuhn_poker/kuhn_poker/__init__.py | 3 | __all__ = ["KuhnPokerEnv", "KuhnPokerTaskset"] | CODE |
| LOW | environments/interception/interception/__init__.py | 3 | __all__ = ["InterceptionTaskset"] | CODE |
| LOW | environments/openenv_wordle/openenv_wordle/taskset.py | 13 | __all__ = ["OpenEnvEnv", "OpenEnvWordleConfig", "OpenEnvWordleTaskset"] | CODE |
| LOW | environments/openenv_wordle/openenv_wordle/__init__.py | 3 | __all__ = ["OpenEnvEnv", "OpenEnvWordleTaskset"] | CODE |
| LOW | environments/wiki_search/wiki_search/__init__.py | 4 | __all__ = ["WikiSearchJudge", "WikiSearchTaskset"] | CODE |
| LOW | environments/wiki_search/wiki_search/judge.py | 20 | __all__ = ["WikiSearchJudge", "WikiSearchJudgeConfig"] | CODE |
| LOW | environments/scratchpad/scratchpad/__init__.py | 3 | __all__ = ["ScratchpadTaskset"] | CODE |
| LOW | …earch_interception/web_search_interception/__init__.py | 6 | __all__ = [ | CODE |
| LOW | environments/reverse_text/reverse_text/__init__.py | 3 | __all__ = ["ReverseTextTaskset"] | CODE |
| LOW | environments/wordle/wordle/__init__.py | 4 | __all__ = ["TextArenaEnv", "WordleTaskset"] | CODE |
| LOW | environments/alphabet_sort/alphabet_sort/__init__.py | 3 | __all__ = ["AlphabetSortEnv", "AlphabetSortTaskset"] | CODE |
| LOW | environments/gsm8k/gsm8k/__init__.py | 3 | __all__ = ["GSM8KTaskset"] | CODE |
| LOW | …nments/bash_interception/bash_interception/__init__.py | 3 | __all__ = ["BashInterceptionTaskset"] | CODE |
| LOW | environments/color_codeword/color_codeword/__init__.py | 3 | __all__ = ["ColorCodewordEnv", "ColorCodewordTaskset"] | CODE |
| LOW | environments/glossary/glossary/__init__.py | 3 | __all__ = ["GlossaryTaskset"] | CODE |
| LOW | environments/code_golf/code_golf/__init__.py | 3 | __all__ = ["CodeGolfEnv", "CodeGolfTaskset"] | CODE |
| LOW | verifiers/v1/task.py | 43 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harness.py | 34 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/env.py | 39 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/taskset.py | 42 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/session.py | 47 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/__init__.py | 204 | __all__ = [ # noqa: RUF022 - grouped by public API area | CODE |
| LOW | verifiers/v1/rollout.py | 39 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/agent.py | 60 | __all__ = ["Agent", "AgentConfig", "Agents", "TimeoutConfig", "make_agent"] | CODE |
| LOW | verifiers/v1/agent.py | 62 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/clients/__init__.py | 11 | __all__ = [ | CODE |
| LOW | verifiers/v1/clients/train.py | 35 | logger = logging.getLogger(__name__) | CODE |
| LOW⚡ | verifiers/v1/interception/server.py | 72 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/interception/__init__.py | 104 | __all__ = [ | CODE |
| LOW | verifiers/v1/interception/pool.py | 32 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/interception/tunnel/prime.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/interception/tunnel/__init__.py | 22 | __all__ = [ | CODE |
| LOW | verifiers/v1/gepa/runner.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/gepa/__init__.py | 5 | __all__ = ["GEPAAdapter", "GEPAConfig", "run_gepa"] | CODE |
| LOW | verifiers/v1/harnesses/__init__.py | 30 | __all__ = [ | CODE |
| LOW | verifiers/v1/harnesses/prime_agent/harness.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/prime_agent/__init__.py | 6 | __all__ = ["PrimeAgentHarness", "PrimeAgentHarnessConfig"] | CODE |
| LOW⚡ | verifiers/v1/harnesses/openclaw/harness.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/openclaw/__init__.py | 6 | __all__ = ["OpenClawHarness", "OpenClawHarnessConfig"] | CODE |
| LOW⚡ | verifiers/v1/harnesses/kimi_code/harness.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/kimi_code/__init__.py | 6 | __all__ = ["KimiCodeHarness", "KimiCodeHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/codex/harness.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/codex/__init__.py | 3 | __all__ = ["CodexHarness", "CodexHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/claude_code/__init__.py | 6 | __all__ = ["ClaudeCodeHarness", "ClaudeCodeHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/bash/__init__.py | 6 | __all__ = ["BashHarness", "BashHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/pi/harness.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/pi/__init__.py | 3 | __all__ = ["PiHarness", "PiHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/null/__init__.py | 3 | __all__ = ["NullHarness", "NullHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/rlm/harness.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/rlm/__init__.py | 3 | __all__ = ["RLMHarness", "RLMHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/mini_swe_agent/__init__.py | 6 | __all__ = ["MiniSWEAgentHarness", "MiniSWEAgentHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/browser_use/__init__.py | 6 | __all__ = ["BrowserUseHarness", "BrowserUseHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/hermes_agent/__init__.py | 6 | __all__ = ["HermesAgentHarness", "HermesAgentHarnessConfig"] | CODE |
| LOW | verifiers/v1/harnesses/terminus_2/harness.py | 12 | logger = logging.getLogger(__name__) | CODE |
| LOW | verifiers/v1/harnesses/terminus_2/__init__.py | 6 | __all__ = ["Terminus2Harness", "Terminus2HarnessConfig"] | CODE |
| 51 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | environments/gsm8k/gsm8k/verify.py | 29 | except Exception: # noqa: BLE001 - malformed answers can fail anywhere in math-verify | CODE |
| LOW | verifiers/v1/env.py | 271 | except Exception as e: # noqa: BLE001 - episode boundary records every hook failure | CODE |
| LOW | verifiers/v1/env.py | 286 | except Exception as e: # noqa: BLE001 - episode boundary records every hook failure | CODE |
| LOW | verifiers/v1/session.py | 347 | except Exception as error: | CODE |
| LOW | verifiers/v1/session.py | 436 | except Exception as error: | CODE |
| LOW | verifiers/v1/rollout.py | 433 | except Exception as e: # noqa: BLE001 - setup boundary records every rollout failure | CODE |
| LOW⚡ | verifiers/v1/rollout.py | 504 | except Exception as e: # noqa: BLE001 - harness boundary records every rollout failure | CODE |
| LOW | verifiers/v1/rollout.py | 564 | except Exception: | CODE |
| LOW | verifiers/v1/rollout.py | 615 | except Exception as e: # noqa: BLE001 - finalize boundary records every rollout failure | CODE |
| LOW | verifiers/v1/rollout.py | 639 | except Exception: | CODE |
| LOW | verifiers/v1/rollout.py | 649 | except Exception: | CODE |
| LOW | verifiers/v1/agent.py | 795 | except Exception as e: | CODE |
| MEDIUM | verifiers/v1/agent.py | 382 | def _drain_messages(self) -> None: | CODE |
| LOW | verifiers/v1/errors.py | 94 | except Exception as e: | CODE |
| LOW | verifiers/v1/interception/server.py | 233 | except Exception as e: # a malformed provider stream | CODE |
| LOW | verifiers/v1/interception/server.py | 732 | except Exception as error: # noqa: BLE001 - surface task hook failures | CODE |
| LOW | verifiers/v1/interception/server.py | 878 | except Exception as e: # noqa: BLE001 - surface as an API error | CODE |
| LOW | verifiers/v1/interception/server.py | 946 | except Exception as e: # noqa: BLE001 - surface auxiliary relay failures | CODE |
| LOW | verifiers/v1/interception/tunnel/prime.py | 66 | except Exception as e: # noqa: BLE001 - the service is unreachable: no verdict | CODE |
| LOW | verifiers/v1/interception/tunnel/prime.py | 85 | except Exception as e: # noqa: BLE001 - tried again on the next check | CODE |
| LOW | verifiers/v1/interception/tunnel/prime.py | 123 | except Exception as e: | CODE |
| LOW | verifiers/v1/harnesses/utils/core.py | 156 | except Exception as e: # noqa: BLE001 - tool failures are returned to the model | CODE |
| LOW | verifiers/v1/harnesses/utils/core.py | 170 | except Exception as e: # noqa: BLE001 - tool failures are returned to the model | CODE |
| LOW | verifiers/v1/harnesses/utils/core.py | 192 | except Exception as e: # noqa: BLE001 - tool failures are returned to the model | CODE |
| LOW | verifiers/v1/harnesses/utils/core.py | 199 | except Exception as e: # noqa: BLE001 - tool failures are returned to the model | CODE |
| LOW | verifiers/v1/harnesses/utils/mcp.py | 84 | except Exception as error: # noqa: BLE001 - deliver owner failures to the caller | CODE |
| LOW | verifiers/v1/harnesses/browser_use/program.py | 172 | except Exception as e: # noqa: BLE001 - tool failures are returned to the model | CODE |
| LOW | verifiers/v1/harnesses/terminus_2/harness.py | 73 | except Exception: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 288 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 351 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 373 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 402 | except Exception as exc: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 416 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 431 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/prime.py | 454 | except Exception as e: # noqa: BLE001 - provider teardown is best-effort | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 265 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 288 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 313 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 434 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 444 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 492 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 506 | except Exception: # noqa: BLE001 - unreadable means "not finished yet" | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 521 | except Exception as reconnect_error: # noqa: BLE001 - re-raised after the loop | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 584 | except Exception as e: # noqa: BLE001 - a dropped stream; _reconnect re-raises | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 602 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 629 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 661 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 671 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 693 | except Exception: # noqa: BLE001 - a later cleanup call may retry | CODE |
| LOW | verifiers/v1/runtimes/e2b.py | 708 | except Exception as e: # noqa: BLE001 - provider teardown is best-effort | CODE |
| MEDIUM | verifiers/v1/runtimes/e2b.py | 390 | def start(self) -> None: | CODE |
| LOW | verifiers/v1/runtimes/modal.py | 273 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/modal.py | 290 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/modal.py | 352 | except Exception: | CODE |
| LOW | verifiers/v1/runtimes/modal.py | 368 | except Exception as e: | CODE |
| LOW⚡ | verifiers/v1/runtimes/modal.py | 395 | except Exception as e: | CODE |
| LOW⚡ | verifiers/v1/runtimes/modal.py | 406 | except Exception as e: | CODE |
| LOW | verifiers/v1/runtimes/modal.py | 430 | except Exception as e: # noqa: BLE001 - provider teardown is best-effort | CODE |
| MEDIUM | verifiers/v1/runtimes/modal.py | 185 | def start(self) -> None: | CODE |
| LOW | verifiers/v1/runtimes/base.py | 246 | except Exception: # noqa: BLE001 - failing to exec at all means the box is gone | CODE |
| 25 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ayscale_interception/grayscale_interception/taskset.py | 12 | CODE | |
| LOW | environments/wiki_search/wiki_search/servers/wiki.py | 77 | CODE | |
| LOW | environments/alphabet_sort/alphabet_sort/taskset.py | 116 | CODE | |
| LOW | tests/v1/test_delivery_behaviors.py | 53 | CODE | |
| LOW | verifiers/v1/graph.py | 241 | CODE | |
| LOW | verifiers/v1/session.py | 234 | CODE | |
| LOW | verifiers/v1/rollout.py | 217 | CODE | |
| LOW | verifiers/v1/rollout.py | 550 | CODE | |
| LOW | verifiers/v1/agent.py | 382 | CODE | |
| LOW | verifiers/v1/agent.py | 465 | CODE | |
| LOW | verifiers/v1/interception/server.py | 584 | CODE | |
| LOW | verifiers/v1/interception/server.py | 789 | CODE | |
| LOW | verifiers/v1/gepa/adapter.py | 84 | CODE | |
| LOW | verifiers/v1/harnesses/utils/core.py | 212 | CODE | |
| LOW | verifiers/v1/harnesses/utils/core.py | 357 | CODE | |
| LOW | verifiers/v1/harnesses/browser_use/program.py | 205 | CODE | |
| LOW | verifiers/v1/runtimes/prime.py | 376 | CODE | |
| LOW | verifiers/v1/runtimes/e2b.py | 530 | CODE | |
| LOW | verifiers/v1/runtimes/modal.py | 315 | CODE | |
| LOW | verifiers/v1/runtimes/docker/egress.py | 181 | CODE | |
| LOW | verifiers/v1/utils/generic.py | 74 | CODE | |
| LOW | verifiers/v1/utils/generic.py | 102 | CODE | |
| LOW | verifiers/v1/utils/platform.py | 47 | CODE | |
| LOW | verifiers/v1/utils/artifacts.py | 212 | CODE | |
| LOW | verifiers/v1/utils/artifacts.py | 240 | CODE | |
| LOW | verifiers/v1/utils/score.py | 153 | CODE | |
| LOW | verifiers/v1/mcp/launch.py | 76 | CODE | |
| LOW | verifiers/v1/mcp/launch.py | 101 | CODE | |
| LOW | verifiers/v1/cli/validate.py | 118 | CODE | |
| LOW | verifiers/v1/cli/dashboard/eval.py | 288 | CODE | |
| LOW | verifiers/v1/cli/dashboard/eval.py | 562 | CODE | |
| LOW | verifiers/v1/cli/dashboard/base.py | 81 | CODE | |
| LOW | verifiers/v1/cli/eval/resume.py | 18 | CODE | |
| LOW | verifiers/v1/tasksets/nemo_gym/response.py | 11 | CODE | |
| LOW | verifiers/v1/tasksets/harbor/compose.py | 66 | CODE | |
| LOW | verifiers/v1/dialects/responses.py | 376 | CODE | |
| LOW | verifiers/v1/dialects/responses.py | 522 | CODE | |
| LOW | verifiers/v1/dialects/chat.py | 292 | CODE | |
| LOW | verifiers/v1/dialects/chat.py | 540 | CODE | |
| LOW | verifiers/v1/dialects/anthropic.py | 108 | CODE | |
| LOW | verifiers/v1/dialects/anthropic.py | 287 | CODE | |
| LOW | verifiers/v1/dialects/anthropic.py | 354 | CODE | |
| LOW | verifiers/v1/dialects/anthropic.py | 547 | CODE | |
| LOW | verifiers/v1/configs/runtime.py | 90 | CODE | |
| LOW | verifiers/v1/serve/server.py | 125 | CODE | |
| LOW | verifiers/v1/serve/encoding.py | 27 | CODE | |
| LOW | verifiers/v1/serve/client.py | 66 | CODE | |
| LOW | verifiers/v1/serve/delta.py | 196 | CODE | |
| LOW | verifiers/v1/serve/pool.py | 166 | CODE | |
| LOW | verifiers/v1/acp/runner.py | 364 | CODE | |
| LOW | verifiers/v1/acp/runner.py | 315 | CODE | |
| LOW | verifiers/v1/acp/runner.py | 374 | CODE | |
| LOW | verifiers/v1/acp/__init__.py | 375 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | verifiers/v1/harnesses/utils/mcp.py | 82 | except asyncio.CancelledError: | CODE |
| HIGH | verifiers/v1/utils/git.py | 169 | except Exception: # noqa: BLE001, S110 - cleanup must never fail the rollout | CODE |
| HIGH | verifiers/v1/utils/version.py | 21 | except (PackageNotFoundError, ValueError): | CODE |
| HIGH | verifiers/v1/utils/aio.py | 27 | except BaseException: # noqa: BLE001, S110 - re-raised or chained below | CODE |
| HIGH | verifiers/v1/tasksets/nemo_gym/taskset.py | 153 | except (FileNotFoundError, ValueError, httpx.HTTPError): | CODE |
| HIGH | verifiers/v1/tasksets/harbor/taskset.py | 250 | except TimeoutError: | CODE |
| HIGH | verifiers/v1/serve/server.py | 74 | except KeyboardInterrupt: | CODE |
| HIGH | verifiers/v1/serve/server.py | 217 | except (asyncio.CancelledError, KeyboardInterrupt): | CODE |
| HIGH | verifiers/v1/serve/pool.py | 370 | except KeyboardInterrupt: | CODE |
| HIGH | verifiers/v1/serve/pool.py | 264 | except (asyncio.CancelledError, KeyboardInterrupt): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | verifiers/v1/task.py | 0 | timeout (in seconds) for the agent's solve attempt. | STRING |
| HIGH | verifiers/v1/rollout.py | 0 | timeout (in seconds) for the agent's solve attempt. | STRING |
| HIGH | verifiers/v1/configs/agent.py | 0 | timeout (in seconds) for the agent's solve attempt. | STRING |
| HIGH | verifiers/v1/runtimes/prime.py | 0 | working directory override; none uses the task's workdir, or /app. | STRING |
| HIGH | verifiers/v1/runtimes/e2b.py | 0 | working directory override; none uses the task's workdir, or /app. | STRING |
| HIGH | verifiers/v1/runtimes/container.py | 0 | working directory override; none uses the task's workdir, or /app. | STRING |
| HIGH | verifiers/v1/runtimes/modal.py | 0 | working directory override; none uses the task's workdir, or /app. | STRING |
| HIGH | verifiers/v1/configs/cli/validate.py | 0 | which of the taskset's tasks to evaluate, under `--select.*` (`-n` sets `select.limit`, `-s` sets `select.shuffle`). | STRING |
| HIGH | verifiers/v1/configs/cli/debug.py | 0 | which of the taskset's tasks to evaluate, under `--select.*` (`-n` sets `select.limit`, `-s` sets `select.shuffle`). | STRING |
| HIGH | verifiers/v1/configs/cli/eval.py | 0 | which of the taskset's tasks to evaluate, under `--select.*` (`-n` sets `select.limit`, `-s` sets `select.shuffle`). | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/v1/conftest.py | 52 | # Fixture tasksets (echo-v1, echo-agentic-v1) live in tests/v1/fixtures, added to the | COMMENT |
| MEDIUM | tests/v1/test_e2e.py | 662 | harness=None, # a multi-agent env refuses the run-level harness | CODE |
| MEDIUM | tests/v1/test_e2e.py | 769 | harness=None, # multi-agent: each seat pins its own | CODE |
| MEDIUM | tests/v1/test_e2e.py | 806 | harness=None, # multi-agent: each seat pins its own | CODE |
| MEDIUM | verifiers/v1/env.py | 150 | # --- the multi-agent surface (override these) ------------------------------ | COMMENT |
| MEDIUM⚡ | verifiers/v1/interception/server.py | 78 | # context window are the real limits, this is just a host-OOM backstop. | COMMENT |
| MEDIUM | verifiers/v1/harnesses/utils/compaction.py | 43 | # OpenAI Responses/Completions: "Your input exceeds the context window of this model". | COMMENT |
| MEDIUM | verifiers/v1/judges/rubric.py | 26 | # criterion carries a one-sentence `reason` written *before* the verdict (chain-of-thought), so the | COMMENT |
| MEDIUM | verifiers/v1/cli/gepa.py | 65 | # Refuse multi-agent before the dry-run return, so --dry-run can't write a | COMMENT |
| MEDIUM | verifiers/v1/cli/replay.py | 60 | # Refuse multi-agent up front, from the SOURCE run's saved config — the replay | COMMENT |
| MEDIUM | verifiers/v1/cli/dashboard/eval.py | 569 | # trace (a multi-agent episode shows each role's trace), braced per task. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 31 | # Step 1: Login to Docker Hub | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 36 | # Step 2: Build the binary | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 41 | # Step 3: Extract binary from builder | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 55 | # Step 4: Build runtime image | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 60 | # Step 5: Push to Docker Hub | COMMENT |
| LOW | …sets/templates/browserbase/cua/scripts/build-binary.sh | 27 | # Step 1: Bundle with esbuild | COMMENT |
| LOW⚡ | …sets/templates/browserbase/cua/scripts/build-binary.sh | 40 | # Step 2: Generate SEA blob | COMMENT |
| LOW⚡ | …sets/templates/browserbase/cua/scripts/build-binary.sh | 46 | # Step 3: Create the binary | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 31 | # Step 1: Login to Docker Hub | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 36 | # Step 2: Build the binary | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 41 | # Step 3: Extract binary from builder | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 55 | # Step 4: Build runtime image | COMMENT |
| LOW⚡ | assets/templates/browserbase/cua/build-and-push.sh | 60 | # Step 5: Push to Docker Hub | COMMENT |
| LOW | …sets/templates/browserbase/cua/scripts/build-binary.sh | 27 | # Step 1: Bundle with esbuild | COMMENT |
| LOW⚡ | …sets/templates/browserbase/cua/scripts/build-binary.sh | 40 | # Step 2: Generate SEA blob | COMMENT |
| LOW⚡ | …sets/templates/browserbase/cua/scripts/build-binary.sh | 46 | # Step 3: Create the binary | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | verifiers/v1/types.py | 241 | # Transient carrier (excluded): per-message token spans into `prompt_ids` from the renderer, | COMMENT |
| LOW | verifiers/v1/harnesses/utils/compaction.py | 41 | # OpenAI error code "context_length_exceeded"; OpenRouter relays the raw body. | COMMENT |
| LOW | verifiers/v1/harnesses/browser_use/program.py | 1 | # /// script | COMMENT |
| LOW | verifiers/v1/utils/git.py | 41 | # reports: a failure in add, diff, reset, or head fails the capture. A failed | COMMENT |
| LOW | verifiers/v1/tasksets/nemo_gym/server.py | 1 | # /// script | COMMENT |
| LOW | assets/templates/browserbase/cua/setup.sh | 1 | #!/bin/bash | COMMENT |
| LOW | assets/templates/browserbase/cua/setup-binary.sh | 1 | #!/bin/bash | COMMENT |
| LOW | assets/templates/browserbase/cua/build-and-push.sh | 1 | #!/bin/bash | COMMENT |
| LOW | …sets/templates/browserbase/cua/scripts/build-binary.sh | 1 | #!/bin/bash | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | verifiers/v1/runtimes/modal.py | 399 | # Create the parent first (Modal's write does not mkdir). | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/v1/test_judges.py | 498 | response = JudgeResponse(text="as an AI language model I cannot grade this") | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | assets/templates/browserbase/cua/build-and-push.sh | 9 | # Usage: | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/v1/test_configs.py | 32 | "user@example.com", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/v1/conftest.py | 123 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | verifiers/v1/interception/server.py | 584 | async def handle_request( | CODE |