Democratizing Reinforcement Learning for LLMs
This report presents the forensic synthetic code analysis of rllm-org/rllm, a Python project with 5,694 GitHub stars. SynthScan v2.0 examined 108,416 lines of code across 646 source files, recording 3084 pattern matches distributed across 23 syntactic categories. The overall adjusted score of 50.3 places this repository in the Strong AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 3084 distinct pattern matches across 23 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_store.py | 11 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/test_store.py | 13 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 71 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 73 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 116 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 118 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 168 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_store.py | 170 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 11 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 13 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 73 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 75 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 95 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 97 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 138 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 140 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 160 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 162 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 202 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 204 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 228 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 230 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_remote_engine.py | 23 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_remote_engine.py | 25 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_remote_engine.py | 66 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_remote_engine.py | 68 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_sampling_e2e.py | 24 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_sampling_e2e.py | 26 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_sampling_e2e.py | 98 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/integration/test_sampling_e2e.py | 101 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/integration/test_sampling_e2e.py | 130 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/integration/test_sampling_e2e.py | 132 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/integration/test_sampling_e2e.py | 158 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/integration/test_sampling_e2e.py | 160 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 145 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 147 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 199 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 201 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 267 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 269 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 323 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 325 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 361 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_tinker_transform.py | 363 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 21 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 23 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 78 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 80 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 464 | # ============================================================================= | COMMENT |
| MEDIUM | tests/unified_trainer/test_tinker_transform.py | 466 | # ============================================================================= | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 105 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 107 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 196 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 198 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 262 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 264 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 302 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/unified_trainer/test_config_sync.py | 304 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/harnesses/test_cli_harness.py | 79 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/harnesses/test_cli_harness.py | 82 | # --------------------------------------------------------------------------- | COMMENT |
| 602 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/test_logging.py | 28 | def test_get_log_level_from_env_uses_default(monkeypatch): | CODE |
| LOW⚡ | tests/test_logging.py | 35 | def test_get_log_level_from_env_reads_level(monkeypatch): | CODE |
| LOW⚡ | tests/test_logging.py | 41 | def test_get_log_level_from_env_falls_back_for_invalid_level(monkeypatch): | CODE |
| LOW⚡ | tests/test_logging.py | 47 | def test_configure_logging_from_env_sets_rllm_logger(monkeypatch): | CODE |
| LOW⚡ | tests/test_logging.py | 55 | def test_configure_logging_from_env_initializes_basic_logging(monkeypatch): | CODE |
| LOW | tests/test_logging.py | 68 | def test_import_configures_rllm_logger_from_env(monkeypatch): | CODE |
| LOW | tests/test_verl_imports.py | 47 | def test_verl_module_paths_exist(module_path: str, name: str) -> None: | CODE |
| LOW | tests/test_verl_metrics.py | 24 | def test_calculate_debug_metrics_compat_passes_through_upstream_metrics(monkeypatch: pytest.MonkeyPatch) -> None: | CODE |
| LOW | tests/test_verl_metrics.py | 49 | def test_calculate_debug_metrics_compat_keeps_newer_upstream_empty_mask_fallback( | CODE |
| LOW | tests/test_verl_metrics.py | 72 | def test_calculate_debug_metrics_compat_single_token_std_is_zero(monkeypatch: pytest.MonkeyPatch) -> None: | CODE |
| LOW | tests/test_verl_metrics.py | 90 | def test_calculate_debug_metrics_compat_backfills_legacy_empty_mask_error( | CODE |
| LOW | tests/test_verl_metrics.py | 113 | def test_calculate_debug_metrics_compat_reraises_unrelated_runtime_errors( | CODE |
| LOW | tests/test_store.py | 27 | def test_get_missing_key_returns_default(self, store): | CODE |
| LOW⚡ | tests/test_store.py | 64 | def test_stores_arbitrary_types(self, store): | CODE |
| LOW⚡ | tests/test_store.py | 77 | def test_concurrent_writes_no_corruption(self, store): | CODE |
| LOW | tests/test_store.py | 94 | def test_concurrent_reads_during_writes(self, store): | CODE |
| LOW⚡ | tests/test_store.py | 122 | def test_two_instances_share_state(self, store): | CODE |
| LOW | tests/test_store.py | 138 | def test_interleaved_read_write(self, store): | CODE |
| LOW⚡ | tests/test_store.py | 174 | def test_workflow_receives_store(self): | CODE |
| LOW⚡ | tests/test_store.py | 192 | def test_workflow_without_store(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 17 | def test_bare_values_unchanged(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 79 | def test_unknown_loss_triggers_warning(self, caplog): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 88 | def test_known_losses_complete(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 108 | def test_override_applied_and_restored(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 130 | def test_no_override_passes_through(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 151 | def test_default_loss_is_known(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 166 | def test_roles_with_same_loss_grouped(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 208 | def test_accumulated_data_tracks_group_roles(self): | CODE |
| LOW⚡ | tests/test_verl_policy_loss.py | 234 | def test_no_loss_fn_map_means_no_splitting(self): | CODE |
| LOW | tests/test_verl_policy_loss.py | 53 | def test_invalid_tuple_length(self): | CODE |
| LOW | tests/test_verl_policy_loss.py | 62 | def test_backward_compat_no_tuples(self): | CODE |
| LOW | tests/test_verl_policy_loss.py | 179 | def test_all_same_loss_single_group(self): | CODE |
| LOW | tests/test_verl_policy_loss.py | 190 | def test_each_role_different_loss(self): | CODE |
| LOW | tests/test_verl_policy_loss.py | 245 | def test_advantage_computation_unaffected(self): | CODE |
| LOW | tests/tasks/test_loader_memory.py | 8 | def test_vlm_instruction_encodes_images_lazily(tmp_path, monkeypatch): | CODE |
| LOW | tests/integration/test_remote_engine.py | 105 | async def test_batch_tasks_session_correlation(self, engine): | CODE |
| LOW | tests/integration/test_remote_engine.py | 130 | async def test_timeout_produces_error_episode(self, engine): | CODE |
| LOW⚡ | tests/integration/test_sampling_e2e.py | 135 | def test_eval_run_dataset_enforces_sampling(mock_upstream): | CODE |
| LOW⚡ | tests/integration/test_sampling_e2e.py | 163 | def test_train_rollout_enforces_train_sampling(mock_upstream): | CODE |
| LOW | tests/integration/test_minimax_integration.py | 27 | def test_minimax_litellm_config_generation(self): | CODE |
| LOW | tests/integration/test_minimax_integration.py | 41 | def test_minimax_config_save_and_load(self, tmp_path, monkeypatch): | CODE |
| LOW | tests/integration/test_minimax_integration.py | 60 | def test_minimax_litellm_completion(self): | CODE |
| LOW | tests/integration/test_tinker_gateway_integration.py | 124 | def test_vllm_fields_stripped(self, tinker_gateway): | CODE |
| LOW | tests/integration/test_tinker_gateway_integration.py | 147 | def test_trace_has_token_data(self, tinker_gateway): | CODE |
| LOW | tests/integration/test_tinker_gateway_integration.py | 177 | def test_trace_round_trips_to_step(self, tinker_gateway): | CODE |
| LOW | tests/integration/test_tinker_gateway_integration.py | 211 | def test_streaming_delivers_content_and_trace(self, tinker_gateway): | CODE |
| LOW | tests/integration/test_tinker_gateway_integration.py | 265 | def test_tool_calling_multi_turn(self, tinker_gateway): | CODE |
| LOW | tests/unified_trainer/test_warm_queue_wiring.py | 15 | def test_detached_warm_queue_detaches_and_restores(): | CODE |
| LOW | tests/unified_trainer/test_warm_queue_wiring.py | 53 | def test_start_warm_queue_guard_returns_none(warm_queue_size, sandbox_backend, hooks_present): | CODE |
| LOW⚡ | tests/unified_trainer/test_algorithm_config.py | 48 | def test_norm_adv_by_std_in_grpo_true_from_algorithm(): | CODE |
| LOW⚡ | tests/unified_trainer/test_algorithm_config.py | 55 | def test_norm_adv_by_std_in_grpo_false_from_algorithm(): | CODE |
| LOW⚡ | tests/unified_trainer/test_algorithm_config.py | 62 | def test_warmup_steps_from_algorithm(): | CODE |
| LOW | tests/unified_trainer/test_termination_metrics.py | 6 | def test_collect_workflow_metrics_counts_unknown_termination_reason(): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 37 | def test_auto_no_checkpoint_returns_none(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 41 | def test_auto_resumes_latest_from_tracker(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 47 | def test_auto_tracker_points_at_missing_dir_returns_none(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 53 | def test_disable_returns_none_even_with_checkpoint(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 58 | def test_resume_path_explicit_folder(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 63 | def test_resume_path_missing_checkpoint_raises(tmp_path): | CODE |
| LOW⚡ | tests/unified_trainer/test_tinker_checkpoint.py | 69 | def test_resume_path_requires_global_step_token(tmp_path): | CODE |
| 951 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/conftest.py | 22 | except Exception: | CODE |
| LOW | tests/conftest.py | 31 | except Exception: | CODE |
| LOW | tests/conftest.py | 51 | except Exception: | CODE |
| LOW | tests/conftest.py | 121 | except Exception: | CODE |
| MEDIUM | tests/tools/test_tools.py | 143 | print(f"Error in tool execution: {result.error}") | CODE |
| MEDIUM | tests/tools/test_tools.py | 175 | print(f"Error: {result.error}") | CODE |
| MEDIUM | tests/tools/test_tools.py | 204 | print(f"Error: {result.error}") | CODE |
| MEDIUM | tests/tools/test_tools.py | 245 | print(f"Error: {result.error}") | CODE |
| LOW | tests/parser/test_harmony_chat_parser.py | 17 | except Exception as exc: | CODE |
| LOW | tests/parser/test_chat_parser.py | 111 | except Exception as e: | CODE |
| LOW | rllm/hooks.py | 237 | except Exception: | CODE |
| LOW | rllm/hooks.py | 267 | except Exception as e: | CODE |
| LOW | rllm/hooks.py | 277 | except Exception: | CODE |
| LOW | rllm/hooks.py | 288 | except Exception: | CODE |
| LOW⚡ | …ental/rllm_telemetry/examples/experiment_comparison.py | 107 | except Exception as e: | CODE |
| LOW | …erimental/rllm_telemetry/src/rllm_telemetry/scorers.py | 250 | except Exception as exc: | CODE |
| LOW⚡ | …rllm_telemetry/src/rllm_telemetry/trajectory_export.py | 50 | except Exception: | CODE |
| LOW⚡ | …rllm_telemetry/src/rllm_telemetry/trajectory_export.py | 55 | except Exception: | CODE |
| LOW⚡ | …rllm_telemetry/src/rllm_telemetry/trajectory_export.py | 59 | except Exception: | CODE |
| MEDIUM | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 425 | def _flush_loop(self) -> None: | CODE |
| LOW⚡ | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 230 | except Exception: | CODE |
| LOW⚡ | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 378 | except Exception as exc: | CODE |
| LOW⚡ | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 385 | except Exception as exc: | CODE |
| LOW⚡ | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 394 | except Exception: | CODE |
| LOW⚡ | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 646 | except Exception as exc: | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 407 | except Exception: | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 457 | except Exception as exc: | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 474 | except Exception: | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 563 | except Exception as exc: | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 584 | except Exception as exc: | CODE |
| MEDIUM | …perimental/rllm_telemetry/src/rllm_telemetry/plugin.py | 63 | def wrapper(self, **kwargs): | CODE |
| LOW⚡ | …perimental/rllm_telemetry/src/rllm_telemetry/plugin.py | 66 | except Exception: | CODE |
| LOW⚡ | …perimental/rllm_telemetry/src/rllm_telemetry/plugin.py | 159 | except Exception: | CODE |
| LOW⚡ | …perimental/rllm_telemetry/src/rllm_telemetry/plugin.py | 163 | except Exception: | CODE |
| LOW | …perimental/rllm_telemetry/src/rllm_telemetry/plugin.py | 91 | except Exception: | CODE |
| LOW | …experimental/rllm_telemetry/src/rllm_telemetry/eval.py | 285 | except Exception as exc: | CODE |
| LOW | …experimental/rllm_telemetry/src/rllm_telemetry/eval.py | 300 | except Exception as exc: | CODE |
| LOW | rllm/experimental/fully_async/runner.py | 246 | except Exception as e: | CODE |
| LOW | rllm/experimental/fully_async/runner.py | 254 | except Exception as e: | CODE |
| LOW | rllm/experimental/fully_async/rollout_executor.py | 283 | except Exception as e: | STRING |
| LOW | rllm/experimental/fully_async/rollout_executor.py | 303 | except Exception as e: | STRING |
| LOW | rllm/experimental/fully_async/rollout_executor.py | 310 | except Exception: | STRING |
| LOW | rllm/experimental/fully_async/rollout_executor.py | 388 | except Exception: | CODE |
| LOW | rllm/experimental/fully_async/utils.py | 209 | except Exception: | CODE |
| LOW | rllm/tasks/loader.py | 210 | except Exception: | CODE |
| LOW | rllm/tasks/loader.py | 385 | except Exception: | CODE |
| LOW | rllm/tools/utils.py | 208 | except Exception as e: | CODE |
| MEDIUM | rllm/tools/utils.py | 213 | print(f"Error in test case {{i+1}}: {{str(e)}}") | CODE |
| LOW | rllm/tools/utils.py | 284 | except Exception as e: | CODE |
| LOW | rllm/tools/mcp_tool.py | 37 | except Exception as e: | CODE |
| MEDIUM | rllm/tools/mcp_tool.py | 20 | def async_forward(self, **kwargs) -> ToolOutput: | CODE |
| LOW | rllm/tools/tool_base.py | 122 | except Exception as e: | CODE |
| LOW | rllm/tools/code_tools/e2b_tool.py | 39 | except Exception as e: | CODE |
| MEDIUM | rllm/tools/code_tools/e2b_tool.py | 40 | print(f"Error killing sandbox: {e}") | CODE |
| LOW | rllm/tools/code_tools/e2b_tool.py | 77 | except Exception: | CODE |
| LOW | rllm/tools/code_tools/together_tool.py | 90 | except Exception as e: | CODE |
| LOW | rllm/tools/code_tools/lcb_tool.py | 52 | except Exception as e: | CODE |
| LOW | rllm/tools/code_tools/lcb_tool.py | 99 | except Exception as e: | CODE |
| LOW | rllm/tools/code_tools/lcb_tool.py | 110 | except Exception: | CODE |
| LOW | rllm/tools/code_tools/lcb_tool.py | 203 | except Exception as e: | CODE |
| 353 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_verl_metrics.py | 3 | CODE | |
| LOW | tests/test_store.py | 3 | CODE | |
| LOW | tests/test_verl_policy_loss.py | 3 | CODE | |
| LOW | tests/tasks/test_loader_memory.py | 1 | CODE | |
| LOW | tests/integration/test_sampling_e2e.py | 10 | CODE | |
| LOW | tests/harnesses/test_cli_harness.py | 13 | CODE | |
| LOW | tests/cli/test_sampling_config.py | 3 | CODE | |
| LOW | tests/cli/test_sampling_resolution.py | 3 | CODE | |
| LOW | tests/sandbox/test_warm_queue.py | 9 | CODE | |
| LOW | tests/sandbox/test_sandboxed_flow.py | 3 | CODE | |
| LOW | tests/sandbox/test_snapshot.py | 10 | CODE | |
| LOW | tests/sandbox/test_daytona_backend.py | 8 | CODE | |
| LOW | tests/eval/test_unified_runner.py | 17 | CODE | |
| LOW | tests/eval/test_pull.py | 3 | CODE | |
| LOW | tests/eval/test_evaluator_loader.py | 3 | CODE | |
| LOW | tests/eval/test_agent_loader.py | 3 | CODE | |
| LOW | tests/eval/test_env_join.py | 12 | CODE | |
| LOW | tests/eval/test_script_module_evaluators.py | 11 | CODE | |
| LOW | tests/eval/test_agentic.py | 9 | CODE | |
| LOW | tests/eval/test_vlm_agents.py | 3 | CODE | |
| LOW | tests/eval/test_eval_engine.py | 16 | CODE | |
| LOW | tests/eval/test_ifeval.py | 9 | CODE | |
| LOW | tests/eval/test_rollout_decorator.py | 3 | CODE | |
| LOW | tests/eval/test_vlm_transforms.py | 3 | CODE | |
| LOW | tests/eval/test_transforms.py | 3 | CODE | |
| LOW | tests/eval/test_materialize.py | 13 | CODE | |
| LOW | tests/eval/test_resolver.py | 8 | CODE | |
| LOW | tests/eval/test_eval_types.py | 11 | CODE | |
| LOW | tests/eval/test_mcq.py | 9 | CODE | |
| LOW | tests/data/test_swesmith_builder.py | 3 | CODE | |
| LOW | tests/data/test_skillsbench_builder.py | 8 | CODE | |
| LOW | rllm/hooks.py | 24 | CODE | |
| LOW | rllm/env.py | 19 | CODE | |
| LOW | rllm/paths.py | 18 | CODE | |
| LOW | rllm/types.py | 18 | CODE | |
| LOW | …erimental/rllm_telemetry/src/rllm_telemetry/scorers.py | 3 | CODE | |
| LOW | …rllm_telemetry/src/rllm_telemetry/trajectory_export.py | 19 | CODE | |
| LOW | …perimental/rllm_telemetry/src/rllm_telemetry/config.py | 3 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 13 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 3 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 3 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 5 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 6 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 6 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 6 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 7 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 9 | CODE | |
| 315 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/harnesses/test_cli_harness.py | 98 | invocations like ``cd /work && claude ...`` are compound commands and | STRING |
| HIGH | rllm/harnesses/qwen_code.py | 31 | -o Acquire::Check-Valid-Until=false 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/qwen_code.py | 92 | f". $HOME/.nvm/nvm.sh 2>/dev/null; " | CODE |
| HIGH | rllm/harnesses/zeroclaw.py | 46 | apt-get update -qq 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/zeroclaw.py | 47 | apt-get install -y -qq --no-install-recommends curl ca-certificates tar coreutils 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/zeroclaw.py | 146 | f"mkdir -p $HOME/.zeroclaw && cat > {_CONFIG_PATH} << 'ZC_CONFIG_EOF'\n{content}\nZC_CONFIG_EOF", | CODE |
| HIGH | rllm/harnesses/codex.py | 44 | -o Acquire::Check-Valid-Until=false 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/codex.py | 155 | f". $HOME/.nvm/nvm.sh 2>/dev/null; " | CODE |
| HIGH | rllm/harnesses/kimi_cli.py | 42 | -o Acquire::Check-Valid-Until=false 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/kimi_cli.py | 158 | f"done; kill 0 2>/dev/null)" | CODE |
| HIGH | rllm/harnesses/mini_swe_agent.py | 55 | apt-get update -qq && apt-get install -y -qq curl ca-certificates git | CODE |
| HIGH | rllm/harnesses/mini_swe_agent.py | 147 | f"mkdir -p $HOME/.config/mini-swe-agent && cat > {path} << 'MSWEA_DOTENV_EOF'\n{content}\nMSWEA_DOTENV_EOF", | CODE |
| HIGH | rllm/harnesses/aider.py | 44 | -o Acquire::Check-Valid-Until=false 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/opencode.py | 47 | apt-get update -qq && apt-get install -y -qq curl ca-certificates | CODE |
| HIGH | rllm/harnesses/opencode.py | 148 | cmd = f"mkdir -p $HOME/.config/opencode && cat > $HOME/.config/opencode/opencode.json << '{marker}'\n{content}\n | CODE |
| HIGH | rllm/harnesses/opencode.py | 168 | f". $HOME/.nvm/nvm.sh 2>/dev/null; " | CODE |
| HIGH | rllm/harnesses/cli_harness.py | 83 | invocations like ``cd /workspace && claude ...`` are compound | STRING |
| HIGH | rllm/harnesses/terminus2.py | 63 | apt-get update -qq && apt-get install -y -qq tmux util-linux procps curl ca-certificates git | CODE |
| HIGH | rllm/harnesses/claude_code.py | 65 | apt-get "${_APT_OPTS[@]}" update -qq 2>/dev/null || true | CODE |
| HIGH | rllm/harnesses/tools/submit_tool.py | 31 | diff = _safe_exec(sandbox, "git diff 2>/dev/null || echo 'No git repo'") | CODE |
| HIGH⚡ | rllm/eval/script_evaluator.py | 119 | check = sandbox.exec(f"test -f {path} && echo yes || echo no", timeout=10, user=user).strip() | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 370 | if (v == null) return ""; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 372 | try { return JSON.stringify(v, null, 2); } catch (e) { return String(v); } | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 375 | if (s == null) return "text-gray-400"; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 472 | if (av == null && bv == null) return 0; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 472 | if (av == null && bv == null) return 0; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 473 | if (av == null) return 1; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 474 | if (bv == null) return -1; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 722 | if (!task || typeof task !== "object") return fieldBox("task", task); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 724 | const metadata = task.metadata && Object.keys(task.metadata).length ? task.metadata : null; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 724 | const metadata = task.metadata && Object.keys(task.metadata).length ? task.metadata : null; | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 737 | for (const [ti, traj] of (ep.trajectories || []).entries()) parts.push(renderTrajectory(traj, ti)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 738 | if (ep.artifacts && Object.keys(ep.artifacts).length) parts.push(fieldBox("artifacts", ep.artifacts)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 738 | if (ep.artifacts && Object.keys(ep.artifacts).length) parts.push(fieldBox("artifacts", ep.artifacts)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 739 | if (ep.metrics && Object.keys(ep.metrics).length) parts.push(fieldBox("metrics", ep.metrics)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 739 | if (ep.metrics && Object.keys(ep.metrics).length) parts.push(fieldBox("metrics", ep.metrics)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 740 | if (ep.metadata && Object.keys(ep.metadata).length) parts.push(fieldBox("metadata", ep.metadata)); | CODE |
| HIGH⚡ | rllm/eval/visualizer.py | 740 | if (ep.metadata && Object.keys(ep.metadata).length) parts.push(fieldBox("metadata", ep.metadata)); | CODE |
| HIGH | rllm/eval/visualizer.py | 524 | let runState = { id:null, meta:null, index:[], filter:"all", search:"" }; | CODE |
| HIGH | rllm/eval/visualizer.py | 556 | if (summary.total != null) { | CODE |
| HIGH | rllm/eval/visualizer.py | 620 | if (r == null) return ""; | CODE |
| HIGH | rllm/eval/visualizer.py | 645 | if (v == null) return ""; | CODE |
| HIGH | rllm/eval/visualizer.py | 677 | const meta = step.metadata && Object.keys(step.metadata).length ? step.metadata : null; | CODE |
| HIGH | rllm/eval/visualizer.py | 677 | const meta = step.metadata && Object.keys(step.metadata).length ? step.metadata : null; | CODE |
| HIGH | rllm/data/r2egym_builder.py | 291 | bash "$RUN_TESTS" > /tmp/test_output.txt 2>&1 || log "run_tests.sh exited non-zero (parser inspects log)" | CODE |
| HIGH | rllm/data/r2egym_builder.py | 296 | PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' "$PY" -W ignore -m pytest -rA r2e_tests > /tmp/test_outpu | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 313 | git add -A . >/dev/null 2>&1 || true | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 314 | git diff --cached --binary "$BASE_COMMIT" > "$MODEL_PATCH" 2>/dev/null || true | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 315 | git reset >/dev/null 2>&1 || true | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 317 | PATCH_BYTES=$(wc -c < "$MODEL_PATCH" 2>/dev/null || echo 0) | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 334 | git reset --hard "$BASE_COMMIT" >/dev/null 2>&1 || log "git reset --hard failed (continuing)" | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 335 | git checkout "$BASE_COMMIT" >/dev/null 2>&1 || log "git checkout failed (continuing)" | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 336 | git clean -fd >/dev/null 2>&1 || log "git clean -fd failed (continuing)" | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 370 | bash /tests/run_script.sh "$SELECTED" > /tmp/stdout.log 2> /tmp/stderr.log || log "run_script.sh exited non-zero (parser | CODE |
| HIGH⚡ | rllm/data/swebench_pro_builder.py | 373 | python3 /tests/parser.py /tmp/stdout.log /tmp/stderr.log /tmp/output.json 2>>/tmp/stderr.log || log "parser.py failed" | CODE |
| HIGH | rllm/data/swebench_pro_builder.py | 289 | git config --global --add safe.directory "$(pwd)" 2>/dev/null || true | CODE |
| HIGH | rllm/data/swebench_pro_builder.py | 169 | - ``requirements``: explicit deliverables (sometimes null) | STRING |
| HIGH | rllm/data/swebench_pro_builder.py | 456 | "git config --global --add safe.directory /app 2>/dev/null || true\n" | STRING |
| HIGH | rllm/trainer/fireworks/fireworks_backend.py | 270 | raise ValueError(f"rllm.algorithm.loss_agg_mode must be null, 'token-mean', 'seq-mean-token-sum', or 'seq-me | CODE |
| HIGH | rllm/trainer/fireworks/fireworks_backend.py | 278 | raise ValueError(f"rollout_correction.tis_mode must be null, 'token', or 'sequence', got '{tis_mode}'") | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/hooks.py | 168 | CODE | |
| LOW | rllm/hooks.py | 245 | CODE | |
| LOW | …lm/experimental/rllm_telemetry/examples/hello_world.py | 69 | CODE | |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/exporter.py | 440 | CODE | |
| LOW | …erimental/rllm_telemetry/src/rllm_telemetry/dataset.py | 48 | CODE | |
| LOW | …experimental/rllm_telemetry/src/rllm_telemetry/eval.py | 159 | CODE | |
| LOW | …erimental/rllm_telemetry/src/rllm_telemetry/compare.py | 88 | CODE | |
| LOW | …erimental/rllm_telemetry/src/rllm_telemetry/compare.py | 165 | CODE | |
| LOW | rllm/experimental/fully_async/runner.py | 228 | CODE | |
| LOW | rllm/experimental/fully_async/metric_utils.py | 122 | CODE | |
| LOW | rllm/experimental/fully_async/rollout_executor.py | 326 | CODE | |
| LOW | rllm/experimental/fully_async/utils.py | 251 | CODE | |
| LOW | rllm/experimental/fully_async/utils.py | 350 | CODE | |
| LOW | rllm/experimental/fully_async/utils.py | 480 | CODE | |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 166 | CODE | |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 524 | CODE | |
| LOW | rllm/tasks/loader.py | 169 | CODE | |
| LOW | rllm/tools/utils.py | 43 | CODE | |
| LOW | rllm/tools/mcp_tool.py | 20 | CODE | |
| LOW | rllm/tools/code_tools/together_tool.py | 40 | CODE | |
| LOW | rllm/tools/code_tools/lcb_tool.py | 58 | CODE | |
| LOW | rllm/tools/code_tools/lcb_tool.py | 122 | CODE | |
| LOW | rllm/harnesses/tool_calling.py | 33 | CODE | |
| LOW | rllm/harnesses/tools/file_editor_tool.py | 61 | CODE | |
| LOW | rllm/rewards/code_reward.py | 425 | CODE | |
| LOW | rllm/rewards/countdown_reward.py | 156 | CODE | |
| LOW | rllm/rewards/math_utils/utils.py | 42 | CODE | |
| LOW | rllm/rewards/math_utils/utils.py | 496 | CODE | |
| LOW | rllm/rewards/math_utils/utils.py | 43 | CODE | |
| LOW | rllm/rewards/code_utils/taco.py | 79 | CODE | |
| LOW | rllm/rewards/code_utils/taco.py | 214 | CODE | |
| LOW | rllm/rewards/code_utils/taco.py | 292 | CODE | |
| LOW | rllm/rewards/code_utils/taco.py | 598 | CODE | |
| LOW | rllm/rewards/code_utils/codeforces.py | 61 | CODE | |
| LOW | rllm/rewards/code_utils/codeforces.py | 193 | CODE | |
| LOW | rllm/rewards/code_utils/codeforces.py | 271 | CODE | |
| LOW | rllm/rewards/code_utils/livecodebench.py | 268 | CODE | |
| LOW | rllm/rewards/code_utils/livecodebench.py | 382 | CODE | |
| LOW | rllm/rewards/code_utils/humanevalplus.py | 16 | CODE | |
| LOW | rllm/workflows/distillation_workflow.py | 43 | CODE | |
| LOW | rllm/workflows/simple_workflow.py | 32 | CODE | |
| LOW | rllm/workflows/timing_mixin.py | 134 | CODE | |
| LOW | rllm/utils/visualization.py | 69 | CODE | |
| LOW | rllm/utils/tracking.py | 442 | CODE | |
| LOW | rllm/utils/tracking.py | 487 | CODE | |
| LOW | rllm/utils/tracking.py | 608 | CODE | |
| LOW | rllm/cli/_ui.py | 74 | CODE | |
| LOW | rllm/cli/model_cmd.py | 35 | CODE | |
| LOW | rllm/cli/dataset.py | 47 | CODE | |
| LOW | rllm/cli/dataset.py | 224 | CODE | |
| LOW | rllm/cli/train.py | 146 | CODE | |
| LOW | rllm/cli/_pull.py | 153 | CODE | |
| LOW | rllm/cli/_pull.py | 316 | CODE | |
| LOW | rllm/cli/eval.py | 80 | CODE | |
| LOW | rllm/cli/eval.py | 578 | CODE | |
| LOW | rllm/parser/chat_template_parser.py | 87 | CODE | |
| LOW | rllm/parser/chat_template_parser.py | 206 | CODE | |
| LOW | rllm/parser/chat_template_parser.py | 263 | CODE | |
| LOW | rllm/parser/chat_template_parser.py | 319 | CODE | |
| LOW | rllm/parser/chat_template_parser.py | 396 | CODE | |
| 109 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | rllm/hooks.py | 42 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/hooks.py | 345 | __all__ = [ | CODE |
| LOW | rllm/__init__.py | 10 | __all__ = ["BaseAgent", "Action", "Step", "Trajectory", "Episode", "rollout", "evaluator", "Task"] | CODE |
| LOW | …rimental/rllm_telemetry/src/rllm_telemetry/__init__.py | 34 | __all__ = [ | CODE |
| LOW | rllm/experimental/fully_async/param_sync.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/experimental/fully_async/message_queue.py | 23 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/tasks/__init__.py | 16 | __all__ = [ | CODE |
| LOW | rllm/tasks/loader.py | 38 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/tools/__init__.py | 25 | __all__ = ["PythonInterpreter", "GoogleSearchTool", "FirecrawlTool", "TavilyExtractTool", "TavilySearchTool", "ToolRegis | CODE |
| LOW | rllm/tools/mcp_tool.py | 6 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/tools/code_tools/__init__.py | 6 | __all__ = [ | CODE |
| LOW | rllm/tools/web_tools/__init__.py | 5 | __all__ = ["GoogleSearchTool", "FirecrawlTool", "TavilyExtractTool", "TavilySearchTool"] | CODE |
| LOW | rllm/environments/__init__.py | 3 | __all__ = ["BaseEnv"] | CODE |
| LOW | rllm/environments/base/__init__.py | 3 | __all__ = ["BaseEnv"] | CODE |
| LOW | rllm/harnesses/react.py | 19 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/bash.py | 23 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/cli_harness.py | 41 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/terminus2.py | 33 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/oracle.py | 34 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/tool_calling.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/harnesses/tools/__init__.py | 7 | __all__ = ["BashTool", "FileEditorTool", "SubmitTool"] | CODE |
| LOW | rllm/rewards/__init__.py | 6 | __all__ = ["RewardInput", "RewardOutput", "RewardType", "RewardConfig", "RewardFunction", "zero_reward"] | CODE |
| LOW | rllm/rewards/countdown_reward.py | 8 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/rewards/math_utils/__init__.py | 11 | __all__ = ["extract_answer", "grade_answer_sympy", "grade_answer_mathd"] | CODE |
| LOW | rllm/rewards/code_utils/pyext2.py | 5 | __all__ = [ | CODE |
| LOW | rllm/workflows/__init__.py | 8 | __all__ = [ | CODE |
| LOW | rllm/agents/__init__.py | 13 | __all__ = [ | CODE |
| LOW | rllm/agents/agent.py | 29 | __all__ = [ | CODE |
| LOW | rllm/utils/__init__.py | 8 | __all__ = ["EpisodeLogger", "compute_pass_at_k", "extract_source_metadata", "visualize_trajectories", "VisualizationConf | CODE |
| LOW | rllm/cli/_pull.py | 12 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/cli/eval.py | 24 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/parser/chat_template_parser.py | 10 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/parser/__init__.py | 3 | __all__ = [ | CODE |
| LOW⚡ | rllm/integrations/harbor/atif_trajectory_bridge.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/integrations/harbor/runtime.py | 24 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/integrations/harbor/dataset_loader.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/integrations/harbor/evaluator.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/integrations/harbor/trial_helper.py | 21 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/train_schedule.py | 55 | __all__ = ["build_train_schedule"] | CODE |
| LOW | rllm/sandbox/sandboxed_flow.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/warm_queue.py | 48 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/warm_queue.py | 258 | __all__ = ["WarmQueue"] | CODE |
| LOW | rllm/sandbox/snapshot.py | 26 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/snapshot.py | 412 | __all__ = ["env_key", "env_key_for", "install_script_for", "keys_for_tasks", "get_sandbox", "SnapshotRegistry"] | CODE |
| LOW | rllm/sandbox/backends/daytona.py | 29 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/backends/modal_backend.py | 27 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/backends/local.py | 11 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/sandbox/backends/docker.py | 10 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/module_evaluator.py | 28 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/vlm_utils.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW⚡ | rllm/eval/rollout_decorator.py | 21 | logger = logging.getLogger(__name__) | CODE |
| LOW⚡ | rllm/eval/rollout_decorator.py | 31 | __all__ = ["_coerce_to_episode"] | CODE |
| LOW | rllm/eval/runner.py | 26 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/proxy.py | 33 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/proxy.py | 192 | def set_limits() -> None: | CODE |
| LOW | rllm/eval/script_evaluator.py | 22 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/materialize.py | 39 | logger = logging.getLogger(__name__) | CODE |
| LOW⚡ | rllm/eval/_resolution.py | 37 | logger = logging.getLogger(__name__) | CODE |
| LOW | rllm/eval/_resolution.py | 329 | def _setup_task_environment(task: Task, sandbox: Sandbox) -> None: | CODE |
| LOW | rllm/eval/reward_fns/llm_equality.py | 19 | logger = logging.getLogger(__name__) | CODE |
| 86 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | tests/test_store.py | 184 | # Create a minimal concrete subclass. | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 116 | # Create a mock actor instance with a config | COMMENT |
| MEDIUM⚡ | tests/test_verl_policy_loss.py | 121 | # Create a mock DataProto with override | COMMENT |
| MEDIUM | tests/data/test_registry_migration.py | 44 | # Create a fake v1 registry at the new location | COMMENT |
| MEDIUM | tests/data/test_registry_migration.py | 48 | # Create a fake parquet file | COMMENT |
| MEDIUM | rllm/experimental/fully_async/runner.py | 291 | # Create a configured TaskRunner class with rollout_fn baked in | COMMENT |
| MEDIUM | rllm/tools/__init__.py | 21 | # Create the singleton registry instance and register all default tools | COMMENT |
| MEDIUM | rllm/tools/utils.py | 161 | # Create an underlying bytes buffer initialized with the encoded input. | COMMENT |
| MEDIUM | rllm/tools/code_tools/e2b_tool.py | 83 | # Create a CodeToolOutput object instead of a dictionary | COMMENT |
| MEDIUM | rllm/tools/code_tools/together_tool.py | 114 | # Create a StringIO object with some test data | COMMENT |
| MEDIUM | rllm/tools/code_tools/lcb_tool.py | 211 | # Create a Python interpreter instance | COMMENT |
| MEDIUM | rllm/rewards/code_reward.py | 113 | # Create a new dict with only the selected test cases | COMMENT |
| MEDIUM⚡ | rllm/rewards/countdown_reward.py | 49 | # Define a regex pattern that only allows numbers, operators, parentheses, and whitespace | COMMENT |
| MEDIUM | rllm/rewards/code_utils/kodcode.py | 18 | # Create a preexec_fn function to set resource limits | COMMENT |
| MEDIUM | rllm/rewards/code_utils/humanevalplus.py | 45 | # Create a preexec_fn function to set resource limits | COMMENT |
| MEDIUM | rllm/utils/compute_pass_at_k.py | 11 | # Create a map to store correct answers per problem | COMMENT |
| MEDIUM⚡ | rllm/utils/tracking.py | 865 | # Create a new table with same columns and existing data | COMMENT |
| MEDIUM⚡ | rllm/trainer/verl/train_agent_ppo.py | 38 | # Create a remote instance of the TaskRunner class, and | COMMENT |
| MEDIUM⚡ | rllm/trainer/verl/train_agent_ppo.py | 48 | # This file is used for performance analysis | COMMENT |
| MEDIUM | examples/archive/deepresearch/deepresearch_workflow.py | 54 | # Create the DeepResearch agent | COMMENT |
| MEDIUM | examples/archive/deepresearch/deepresearch_workflow.py | 103 | # Create a failed episode | COMMENT |
| MEDIUM | examples/countdown/run_countdown.py | 53 | # Create a map to store correct answers per problem | COMMENT |
| MEDIUM⚡ | scripts/data/download_gaia.py | 7 | # Define the Google Drive file IDs for the Gaia files | COMMENT |
| MEDIUM⚡ | scripts/data/download_gaia.py | 16 | # Define the destination paths | COMMENT |
| MEDIUM⚡ | scripts/data/download_gaia.py | 22 | # Create the necessary directories | COMMENT |
| MEDIUM | scripts/data/download_datasets.py | 8 | # Define the Google Drive file IDs for the JSON files | COMMENT |
| MEDIUM | scripts/data/download_datasets.py | 21 | # Define the destination paths | COMMENT |
| MEDIUM | scripts/data/download_datasets.py | 31 | # Create the necessary directories | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | rllm/experimental/fully_async/utils.py | 481 | Assemble gen_batch_output from TrajectoryGroup objects. Optimized version: pre-allocates tensors and fills in s | STRING |
| HIGH | rllm/tools/registry.py | 123 | Get a tool class by name using dictionary-like syntax. Args: name: The name of the tool to | STRING |
| HIGH | rllm/environments/base/base_env.py | 60 | Creates an environment instance from a dictionary. This method should be implemented by concrete subclasses to | STRING |
| HIGH | rllm/parser/chat_template_parser.py | 51 | Verify that parsing messages together is equivalent to parsing them individually. Args: messages (l | STRING |
| HIGH | rllm/parser/chat_template_parser.py | 88 | Factory method to get the appropriate parser based on a string identifier. Args: parser_type (str): | STRING |
| HIGH | rllm/parser/tool_parser.py | 21 | Factory method to get the appropriate tool parser based on a string identifier. Args: tokenizer: Th | STRING |
| HIGH | rllm/data/dataset.py | 136 | Load dataset directly from a file path. Args: path: Path to the dataset file Returns: | STRING |
| HIGH | rllm/data/utils.py | 43 | Load a dataset from a JSON file based on the dataset enum. This function takes a dataset enum value and loads the c | STRING |
| HIGH | rllm/engine/unified_workflow_engine.py | 121 | Process a single task rollout with retry logic based on termination reasons. Args: task: Task dicti | STRING |
| HIGH | rllm/engine/agent_workflow_engine.py | 99 | Process a single task rollout with retry logic based on termination reasons. Args: task: Task dicti | STRING |
| HIGH | rllm/engine/rollout/completer.py | 25 | Basic completer that takes in messages and returns a single step. Args: rollout_engine: The rollout en | STRING |
| HIGH | rllm/trainer/distill/alignment.py | 211 | Align teacher logprobs to student tokens using byte-level alignment. Maps teacher logprobs to student tokens b | STRING |
| HIGH | examples/fully_async/deepresearch/refine_agent.py | 311 | Parse out the <think>...</think> tags and return only the result part. Args: content: The raw model re | STRING |
| HIGH | examples/fully_async/deepresearch/refine_agent.py | 340 | Parse out the content inside <information>...</information> tags. Args: content: The model response th | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | rllm/rewards/code_utils/taco.py | 0 | if test(generated_code) is not none it'll try to run the code. otherwise it'll just return an input and output pair. | STRING |
| HIGH | rllm/rewards/code_utils/codeforces.py | 0 | if test(generated_code) is not none it'll try to run the code. otherwise it'll just return an input and output pair. | STRING |
| HIGH | rllm/rewards/code_utils/livecodebench.py | 0 | if test(generated_code) is not none it'll try to run the code. otherwise it'll just return an input and output pair. | STRING |
| HIGH | rllm/rewards/code_utils/taco.py | 0 | this disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bo | STRING |
| HIGH | rllm/rewards/code_utils/codeforces.py | 0 | this disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bo | STRING |
| HIGH | rllm/rewards/code_utils/livecodebench.py | 0 | this disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bo | STRING |
| HIGH | rllm/rewards/code_utils/kodcode.py | 0 | this disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bo | STRING |
| HIGH | rllm/rewards/code_utils/humanevalplus.py | 0 | this disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bo | STRING |
| HIGH | scripts/data/gaia_dataset.py | 0 | create a mapping function to process dataset examples. args: split: dataset split name ('train' or 'test') returns: func | STRING |
| HIGH | scripts/data/code_dataset.py | 0 | create a mapping function to process dataset examples. args: split: dataset split name ('train' or 'test') returns: func | STRING |
| HIGH | scripts/data/deepscaler_dataset.py | 0 | create a mapping function to process dataset examples. args: split: dataset split name ('train' or 'test') returns: func | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | tests/harnesses/test_cli_harness.py | 258 | # Cross-harness: ANTHROPIC_BASE_URL must not end in /v1 | COMMENT |
| LOW | rllm/tools/web_tools/firecrawl_tool.py | 130 | # For now, just call the synchronous version | COMMENT |
| MEDIUM | rllm/harnesses/zeroclaw.py | 158 | # startup before any LLM call in testing. The symlink is robust.) | COMMENT |
| MEDIUM | rllm/harnesses/claude_code.py | 127 | # harness doesn't pollute $HOME between runs. | COMMENT |
| MEDIUM | rllm/rewards/code_utils/swebench.py | 49 | # from swebench.harness.reporting import make_run_report | COMMENT |
| MEDIUM | rllm/cli/train.py | 312 | # Harbor datasets on an rllm-native harness skip ``harbor_reward_fn`` | COMMENT |
| MEDIUM | rllm/cli/eval.py | 162 | # Resolve agent name (display + harness lookup) | COMMENT |
| MEDIUM | rllm/cli/eval.py | 276 | # harbor-sourced dataset through an rllm-native harness (oracle, | COMMENT |
| MEDIUM | rllm/data/skillsbench_builder.py | 240 | # materialization time; the harness-layer hook is the right place for them. | COMMENT |
| MEDIUM⚡ | rllm/data/swebench_pro_builder.py | 310 | # regardless of what the harness or the image left HEAD pointing at. | COMMENT |
| MEDIUM | rllm/data/swebench_pro_builder.py | 648 | # solution/ (gold patch + apply script for the oracle harness) | COMMENT |
| LOW | rllm/engine/rollout/tinker_engine.py | 305 | sampling_params.pop("max_tokens", None) # TODO(listar2000): later we should simply use this as the max_tokens | CODE |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 195 | # No rejection, just return filtered groups with metrics | COMMENT |
| LOW | rllm/trainer/verl/transform.py | 184 | # The padding is done after the transform (in `_pad_dataproto_to_world_size`), so we simply set all to False her | COMMENT |
| MEDIUM | rllm/trainer/verl/transform.py | 600 | # Match advantages to batch entries by step_id (robust to batch reordering and padding) | COMMENT |
| MEDIUM⚡ | rllm/trainer/verl/verl_backend.py | 841 | # Async hook methods - leverage RayPPOTrainer utilities where possible | COMMENT |
| MEDIUM | examples/fully_async/deepresearch/train.py | 91 | # Be robust: if structure is unexpected, fall back to zero | COMMENT |
| MEDIUM | examples/fully_async/deepresearch/train.py | 147 | # Be robust: if structure is unexpected, fall back to zero | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/experimental/fully_async/utils.py | 418 | # Check if all correct or all incorrect | COMMENT |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 235 | # Check if the ESI (Elastic Server Instance)/training plan is close to expiration. | COMMENT |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 240 | # Check if the conditions for saving a checkpoint are met. | COMMENT |
| LOW | rllm/rewards/code_reward.py | 62 | # Check if we're out of the block (less indentation) | COMMENT |
| LOW⚡ | rllm/rewards/countdown_reward.py | 36 | # Check if all numbers in equation are available | COMMENT |
| LOW | rllm/rewards/countdown_reward.py | 180 | # Check if all numbers used are from the available set | COMMENT |
| LOW | rllm/rewards/code_utils/swebench.py | 203 | # Check if git diff changed after running eval script | COMMENT |
| LOW | rllm/cli/agent.py | 45 | # Check if it's a plugin agent | COMMENT |
| LOW | rllm/cli/login.py | 21 | # Check if already logged in | COMMENT |
| LOW | rllm/cli/eval.py | 491 | # Print results | COMMENT |
| LOW | rllm/parser/chat_template_parser.py | 73 | # Check if results are equivalent | COMMENT |
| LOW | rllm/integrations/harbor/dataset_loader.py | 176 | # Check if it's a local dataset directory | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 201 | # Check if we have enough partial solves | COMMENT |
| LOW | rllm/trainer/verl/agent_workflow_trainer.py | 228 | # Check if all episodes are correct or incorrect | COMMENT |
| LOW⚡ | rllm/trainer/verl/train_agent_ppo.py | 29 | # Check if Ray is not initialized | COMMENT |
| LOW | rllm/trainer/deprecated/tinker_data_processor.py | 466 | # Check if current step extends previous sequence | COMMENT |
| LOW | rllm/trainer/deprecated/tinker_policy_trainer.py | 74 | # Check if a Tinker model ID is provided in config | COMMENT |
| LOW | examples/archive/sft_tinker/train_norobots_tinker.sh | 17 | # Check if TINKER_API_KEY is set | COMMENT |
| LOW | examples/archive/deepresearch/deepresearch_tools.py | 634 | # Check if the module or its parent is allowed | COMMENT |
| LOW | examples/archive/deepresearch/deepresearch_agent.py | 568 | # Check if we've exceeded call limit | COMMENT |
| LOW | examples/archive/deepresearch/evaluate_hle.py | 355 | # Check if model is O3/O1 (use model_name which is already determined above) | COMMENT |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 326 | # Check if there's a <think> tag | COMMENT |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 330 | # Check if there's a </think> tag | COMMENT |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 355 | # Check if there's an <information> tag | COMMENT |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 359 | # Check if there's a </information> tag | COMMENT |
| LOW | examples/fully_async/deepresearch/search_agent.py | 338 | # Check if duplicate search was detected - if so, set reward to 0 | COMMENT |
| LOW | examples/fully_async/deepresearch/search_agent.py | 347 | # Check if excessive parallel tool calls detected (>= 5) - if so, set reward to 0 | COMMENT |
| LOW | examples/fully_async/deepresearch/tool.py | 116 | self._maybe_refresh_urls() # Check if URLs need refreshing | CODE |
| LOW | examples/fully_async/deepresearch/rag/launch_rag.sh | 61 | # Check if data directory exists | COMMENT |
| LOW⚡ | scripts/benchmark/cf_elo_calc.py | 212 | # Display results | COMMENT |
| LOW⚡ | cookbooks/workflow/solver_judge/solver_judge_flow.py | 118 | # Assign rewards to solver trajectories | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 280 | # Step 1: prompt=[1,2], response=[3,4] | COMMENT |
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 287 | # Step 2: prompt=[1,2,3,4,5] (extends [1,2,3,4]), response=[6,7] | COMMENT |
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 418 | # Step 1: [img_chunk, 1, 2] -> response [3, 4] | COMMENT |
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 424 | # Step 2: extends with [img_chunk, 1, 2, 3, 4, 5] -> response [6, 7] | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 303 | # Step 1: capture the agent's diff vs base_commit. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 320 | # Step 2: reset to base_commit so the patch applies on a clean tree. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 338 | # Step 3: re-apply the agent's patch. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 347 | # Step 4: run the last non-blank line of before_repo_set_cmd. Upstream | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 362 | # Step 5: run the per-instance run_script.sh with the selected test files. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 372 | # Step 6: parse the run_script.sh output into the standard JSON shape. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 375 | # Step 7: compare passed set against fail_to_pass ∪ pass_to_pass. | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 186 | # Step 1: Filter groups and episodes based on config | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 190 | # Step 2: Compute episode-level correctness metrics (always, for logging) | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 193 | # Step 3: Apply mode-specific logic (TODO(listar2000): implement a group-level rejection sampling) | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 239 | # Step 1: Name imputation | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 247 | # Step 2: Invoke the trajectory grouping hook | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 250 | # Step 3: Get metrics | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 228 | # Step 1: Process to datums (includes filtering and advantage computation) | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 236 | # Step 3: Remove mask from datums (not needed by forward_backward) | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 239 | # Step 4: Forward-backward pass | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 245 | # Step 5: Optimizer step | COMMENT |
| LOW⚡ | cookbooks/workflow/solver_judge/solver_judge_flow.py | 115 | # Step 1: Solver generates multiple solutions in parallel | COMMENT |
| LOW⚡ | cookbooks/workflow/solver_judge/solver_judge_flow.py | 127 | # Step 2: Judge selects the best solution | COMMENT |
| LOW⚡ | cookbooks/solver_judge_flow/solver_judge_flow.py | 27 | # Step 1: Solver generates N solutions in parallel | COMMENT |
| LOW⚡ | cookbooks/solver_judge_flow/solver_judge_flow.py | 30 | # Step 2: Judge selects the best solution | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 280 | # Step 1: prompt=[1,2], response=[3,4] | COMMENT |
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 287 | # Step 2: prompt=[1,2,3,4,5] (extends [1,2,3,4]), response=[6,7] | COMMENT |
| LOW⚡ | tests/unified_trainer/test_tinker_transform.py | 424 | # Step 2: extends with [img_chunk, 1, 2, 3, 4, 5] -> response [6, 7] | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 303 | # Step 1: capture the agent's diff vs base_commit. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 320 | # Step 2: reset to base_commit so the patch applies on a clean tree. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 338 | # Step 3: re-apply the agent's patch. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 347 | # Step 4: run the last non-blank line of before_repo_set_cmd. Upstream | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 362 | # Step 5: run the per-instance run_script.sh with the selected test files. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 372 | # Step 6: parse the run_script.sh output into the standard JSON shape. | COMMENT |
| LOW⚡ | rllm/data/swebench_pro_builder.py | 375 | # Step 7: compare passed set against fail_to_pass ∪ pass_to_pass. | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 186 | # Step 1: Filter groups and episodes based on config | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 190 | # Step 2: Compute episode-level correctness metrics (always, for logging) | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/rejection_sampling.py | 193 | # Step 3: Apply mode-specific logic (TODO(listar2000): implement a group-level rejection sampling) | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 239 | # Step 1: Name imputation | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 247 | # Step 2: Invoke the trajectory grouping hook | COMMENT |
| LOW⚡ | rllm/trainer/algorithms/transform.py | 250 | # Step 3: Get metrics | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 228 | # Step 1: Process to datums (includes filtering and advantage computation) | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 236 | # Step 3: Remove mask from datums (not needed by forward_backward) | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 239 | # Step 4: Forward-backward pass | COMMENT |
| LOW⚡ | rllm/trainer/deprecated/tinker_policy_trainer.py | 245 | # Step 5: Optimizer step | COMMENT |
| LOW⚡ | cookbooks/workflow/solver_judge/solver_judge_flow.py | 115 | # Step 1: Solver generates multiple solutions in parallel | COMMENT |
| LOW⚡ | cookbooks/workflow/solver_judge/solver_judge_flow.py | 127 | # Step 2: Judge selects the best solution | COMMENT |
| LOW⚡ | cookbooks/solver_judge_flow/solver_judge_flow.py | 27 | # Step 1: Solver generates N solutions in parallel | COMMENT |
| LOW⚡ | cookbooks/solver_judge_flow/solver_judge_flow.py | 30 | # Step 2: Judge selects the best solution | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/tools/test_tools.py | 121 | "google_search": [ | COMMENT |
| LOW | rllm/experimental/fully_async/runner.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/metric_utils.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/param_sync.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/message_queue.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/inference_manager.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 1 | # Copyright 2025 Meituan Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 241 | # The conditions include a mandatory condition (1) and | COMMENT |
| LOW | rllm/experimental/fully_async/fully_async_trainer.py | 601 | if rollout_corr_config is not None and "rollout_log_probs" in batch.batch: | COMMENT |
| LOW | rllm/harnesses/zeroclaw.py | 141 | content = self._config_toml(config) | COMMENT |
| LOW | rllm/harnesses/kimi_cli.py | 121 | self._exec_agent(sandbox, self._heredoc_write(_KIMI_CONFIG_PATH, content), env=env) | COMMENT |
| LOW | rllm/harnesses/claude_code.py | 21 | """ | COMMENT |
| LOW | rllm/harnesses/claude_code.py | 121 | "ANTHROPIC_MODEL": model, | COMMENT |
| LOW | rllm/rewards/code_utils/taco.py | 421 | # result = subprocess.run(['python', temp_program_path], stdin=inputs_tmp_file, text=True, captu | COMMENT |
| LOW | rllm/rewards/code_utils/codeforces.py | 361 | exec_code = -2 | COMMENT |
| LOW | rllm/utils/tracking.py | 1 | # Copyright 2024 Bytedance Ltd. and/or its affiliates | COMMENT |
| LOW | rllm/cli/eval.py | 261 | # catalog reward_fn > None. When ``evaluator`` is None ``SandboxTaskHooks`` | COMMENT |
| LOW | rllm/data/r2egym_builder.py | 261 | # R2E-Gym's grader runs the repo's own ``run_tests.sh`` from the repo root. | COMMENT |
| LOW | rllm/data/skillsbench_builder.py | 221 | _SKILLS_COPY_MARKER = "# rllm-skillsbench: skills injection" | COMMENT |
| LOW | rllm/data/swebench_pro_builder.py | 321 | # | COMMENT |
| LOW | rllm/trainer/tinker/transform.py | 161 | datums = [] | COMMENT |
| LOW | rllm/trainer/fireworks/fireworks_backend.py | 21 | from training.provision import FireworksProvisionInfra, init_fireworks_infra | COMMENT |
| LOW | rllm/trainer/algorithms/advantage.py | 241 | COMMENT | |
| LOW | rllm/trainer/algorithms/advantage.py | 261 | # rewards bucketed sensibly rather than silently dropped. | COMMENT |
| LOW | rllm/trainer/verl/transform.py | 301 | # ------------------------------------------------------------------ | COMMENT |
| LOW | rllm/trainer/sft/config/fireworks.yaml | 1 | # Native config for the Fireworks SFT backend. | COMMENT |
| LOW | examples/archive/sft_tinker/train_norobots_tinker.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/fully_async/deepresearch/rag/launch_rag.sh | 1 | #!/bin/bash | COMMENT |
| LOW | examples/verifiers_env/train_verifiers_tinker.sh | 1 | #!/bin/bash | COMMENT |
| LOW | .github/workflows/claude.yml | 41 | COMMENT | |
| LOW | cookbooks/math/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/frozenlake/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/deepcoder/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/agent_frameworks/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/agent_frameworks/train_tinker.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/agent_frameworks/train_verl.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/finqa/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/workflow/solver_judge/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/workflow/geo3k/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/geo3k/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/math_tool_agent/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | cookbooks/solver_judge_flow/train_fireworks.sh | 1 | #!/usr/bin/env bash | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_store.py | 134 | CODE | |
| LOW | rllm/cli/eval.py | 80 | CODE | |
| LOW | rllm/integrations/harbor/runtime.py | 44 | CODE | |
| LOW | rllm/integrations/harbor/trial_helper.py | 107 | CODE | |
| LOW | rllm/integrations/harbor/trial_helper.py | 303 | CODE | |
| LOW | rllm/eval/runner.py | 29 | CODE | |
| LOW | rllm/data/claw_eval_builder.py | 92 | CODE | |
| LOW | rllm/data/r2egym_builder.py | 532 | CODE | |
| LOW | rllm/data/skillsbench_builder.py | 372 | CODE | |
| LOW | rllm/data/swebench_pro_builder.py | 667 | CODE | |
| LOW | rllm/data/deepswe_builder.py | 99 | CODE | |
| LOW | rllm/data/swesmith_builder.py | 160 | CODE | |
| LOW | rllm/engine/rollout/openai_engine.py | 21 | CODE | |
| LOW | rllm/engine/rollout/fireworks_engine.py | 89 | CODE | |
| LOW | rllm/engine/rollout/tinker_engine.py | 150 | CODE | |
| LOW | rllm/trainer/unified_trainer.py | 123 | CODE | |
| LOW | rllm/trainer/unified_trainer.py | 969 | CODE | |
| LOW | rllm/trainer/distill/advantage.py | 77 | CODE | |
| LOW | examples/archive/ocr/process_ocr_data.py | 24 | CODE | |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 336 | CODE | |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 371 | CODE | |
| LOW | examples/fully_async/deepresearch/search_agent.py | 68 | CODE | |
| LOW | examples/fully_async/deepresearch/tool.py | 15 | CODE | |
| LOW | examples/fully_async/deepresearch/rag/rag_server.py | 59 | CODE | |
| LOW | scripts/data/webarena_dataset.py | 56 | CODE | |
| LOW | scripts/data/miniwob_dataset.py | 57 | CODE | |
| LOW | scripts/data/gaia_dataset.py | 59 | CODE | |
| LOW | scripts/data/code_dataset.py | 67 | CODE | |
| LOW | scripts/data/deepscaler_dataset.py | 66 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …rllm_telemetry/src/rllm_telemetry/trajectory_export.py | 11 | # After the agentic loop finishes: | STRING |
| MEDIUM⚡ | rllm/cli/eval.py | 114 | # whenever --agent is set to anything other than a harbor scaffold | COMMENT |
| MEDIUM | rllm/cli/eval.py | 251 | # plus user-registered + plugin agents; ``harbor:<scaffold>`` resolves | COMMENT |
| MEDIUM | rllm/integrations/harbor/trial_helper.py | 164 | # agent scaffolds read. Each scaffold checks a different one: | COMMENT |
| MEDIUM | rllm/engine/remote_runtime/protocol.py | 92 | # Pass-through to harbor AgentConfig.kwargs (scaffold-specific flags). | COMMENT |
| MEDIUM | rllm/trainer/config/rllm/base.yaml | 140 | agent: mini-swe-agent # scaffold name (mini-swe-agent | aider | openhands | swe-agent | qwen-coder | ...) | CODE |
| MEDIUM | rllm/trainer/config/rllm/base.yaml | 142 | agent_kwargs: {} # pass-through to harbor's AgentConfig.kwargs (scaffold-specific flags) | CODE |
| MEDIUM | rllm/trainer/algorithms/advantage.py | 300 | # but role-keyed for multi-agent correctness (single-agent -> "default"). | COMMENT |
| MEDIUM | rllm/trainer/verl/verl_backend.py | 446 | # and can grow up to the full context window - so use max_total_length to | COMMENT |
| MEDIUM | rllm/trainer/deprecated/tinker_data_processor.py | 705 | # Group by (task_id, trajectory_name) - for multi-agent workflows like solver-judge | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | examples/archive/sft_tinker/README_NOROBOTS.md | 162 | export TINKER_API_KEY=your_api_key | CODE |
| HIGH | examples/archive/sft_tinker/train_norobots_tinker.sh | 20 | echo "Please set it with: export TINKER_API_KEY=your_api_key" | CODE |
| HIGH | examples/fully_async/deepresearch/search_agent.py | 8 | export OPENAI_API_KEY="your-api-key" | STRING |
| HIGH | examples/verifiers_env/README.md | 77 | echo 'TINKER_API_KEY="your-api-key"' > examples/verifiers_env/.env | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/data/utils.py | 111 | # Example usage | COMMENT |
| LOW | examples/fully_async/deepresearch/refine_agent.py | 501 | # Example usage | COMMENT |
| LOW | examples/fully_async/deepresearch/rag/launch_rag.sh | 8 | # Usage: | COMMENT |
| LOW | cookbooks/agent_frameworks/train_fireworks.sh | 4 | # Usage: | COMMENT |
| LOW | cookbooks/agent_frameworks/train_tinker.sh | 4 | # Usage: | COMMENT |
| LOW | cookbooks/agent_frameworks/train_verl.sh | 4 | # Usage: | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/trainer/verl/dataclass.py | 93 | if step_data.advantage is not None: # make sure to not add None to the list | CODE |
| MEDIUM | rllm/trainer/verl/train_agent_ppo.py | 111 | # TODO Here you can use the new registration method to support dynamic registration of roles | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/engine/rollout/verl_engine.py | 140 | # TODO: implement parse_completion for the standard parser | COMMENT |
| LOW | rllm/trainer/verl/agent_workflow_trainer.py | 530 | # TODO: implement actual tflpo and theoretical tflpo | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | examples/fully_async/deepresearch/search_agent.py | 105 | async def helper(tool_call): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | rllm/data/dataset.py | 730 | "prompt": [{"role": "user", "content": "placeholder"}], | CODE |