🚀 An open-source, hands-on curriculum bridging the gap from basic RL concepts to LLM alignment, RLVR, and advanced Agentic systems.
This report summarizes scanner evidence for walkinglabs/hands-on-modern-rl, a Python project with 4,565 GitHub stars. SynthScan examined 206,964 lines of code across 601 source files, recording 1510 pattern matches distributed across 19 syntactic categories. The legacy adjusted pattern signal is 19.5 (Elevated pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 482 finding(s).
Are imports declared and package/API names verified? 94 finding(s).
Are wrappers, guards, and repeated structures necessary? 1043 finding(s).
Which changes need focused human review? 1510 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 1510 distinct pattern matches across 19 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | …de/appendix_common_pitfalls/debug_training_collapse.py | 47 | # ========================================== | COMMENT |
| MEDIUM⚡ | …de/appendix_common_pitfalls/debug_training_collapse.py | 49 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 97 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 99 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 208 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 210 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 312 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 314 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 436 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 438 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 538 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 540 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 646 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 648 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 747 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 749 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 45 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 47 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 163 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 165 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 248 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 250 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 331 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 333 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 407 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 409 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 478 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 480 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 543 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 545 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 612 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 614 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/sft_pipeline.py | 24 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/sft_pipeline.py | 26 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 100 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 102 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 126 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 128 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 38 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 40 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 105 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 107 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 235 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 237 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 449 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 451 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 497 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 499 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 32 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 34 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 153 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 155 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 213 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 215 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 266 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 268 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 391 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 393 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 428 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 430 | # ========================================== | COMMENT |
| 596 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | modelscope-space/_shared/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 0 | return a compact, proxy-independent task-detail image data url. | STRING |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | modelscope-space/_shared/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 0 | <main class="app-shell"> <section class="hero"> <div class="brand-lockup"> <a href="{html.escape(organization)}" target= | STRING |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | modelscope-space/_shared/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 0 | <section class="training-guide"> <div class="training-guide__intro"> <span class="task-kicker">result checklist</span> < | STRING |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | modelscope-space/_shared/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 0 | <section class="task-brief"> <div class="task-brief__visual"><img src="{html.escape(preview)}" alt="{html.escape(str(tas | STRING |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | modelscope-space/_shared/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 0 | return explicit epoch controls or derive them from a legacy total budget. | STRING |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 0 | <section class="baseline-preset"> <div class="baseline-preset__head"> <div><span>{html.escape(copy['baseline_badge'])}</ | STRING |
| 97 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 256 | # Step 1: determine the query type | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 259 | # Step 2: sample a tool according to the policy | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 263 | # Step 3: execute the tool and get the reward | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 272 | # Step 4: update the policy | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 428 | # Step 1: generate the dataset | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 438 | # Step 2: print statistics | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 441 | # Step 3: display sample images | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/multi_modal_reward.py | 561 | # Step 1: run the reward function tests | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/multi_modal_reward.py | 564 | # Step 2: visualize the reward weights | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 600 | # Step 1: print the VLM GRPO overview | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 603 | # Step 2: GRPO core computation demo | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 606 | # Step 3: create training data | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 620 | # Step 4: simulate GRPO training | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 630 | # Step 5: before/after training comparison | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 633 | # Step 6: plot training curves | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 253 | # Step 1: collect a full episode trajectory with the current policy | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 256 | # Step 2: compute the discounted cumulative returns | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 259 | # Step 3: perform the policy gradient update | COMMENT |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 232 | # Step 1: expand all nodes at the current level, generating children | COMMENT |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 247 | # Step 2: sort by score, keep the top-k | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 208 | # Step 1: use the Q network to select the optimal action | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 212 | # Step 2: use the target network to evaluate those actions' Q values | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 216 | # Step 3: compute the target value | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 439 | # Step 1: generate a group of responses for the current question | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 446 | # Step 2: compute the reward for each response | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 452 | # Step 3: GRPO group-relative normalization | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 455 | # Step 4: policy gradient update | COMMENT |
| LOW | code/chapter07_ppo/ppo_from_scratch.py | 430 | # Step 1: collect trajectories | COMMENT |
| LOW | code/chapter07_ppo/ppo_from_scratch.py | 441 | # Step 2: PPO update | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 510 | # Step 1: 把对话历史拼成模型能理解的 prompt | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 512 | # Step 2: 模型生成动作（推理，不计算梯度） | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 514 | # Step 3: 从自由文本输出中解析结构化动作 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 529 | # Step 4: 环境执行动作，返回观测和终止状态 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 532 | # Step 5: 记录本轮交互到轨迹 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 541 | # Step 6: 把本轮交互加入对话历史，供下一轮使用 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 549 | # Step 7: 计算整条轨迹的 reward（只有轨迹结束时才给） | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 348 | # Step 1: student 生成回答 | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 360 | # Step 2: 计算 student 和 teacher 的逐 token log-prob | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 381 | # Step 3: 计算逐 token reward（teacher 认可度 - student 自信度） | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 399 | # Step 1: student rollout | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 404 | # Step 2: 计算 per-token advantage | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 408 | # Step 3: 策略梯度更新 | COMMENT |
| LOW | docs/en/chapter01_cartpole/principles.md | 45 | ### Step 1: Install Dependencies | COMMENT |
| LOW | docs/en/chapter01_cartpole/principles.md | 55 | ### Step 2: Run Training | COMMENT |
| LOW | docs/en/chapter01_cartpole/principles.md | 97 | ### Step 3: Where to View SwanLab Training Curves | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/metrics.md | 375 | # Step 1: Collect experience data (2048 steps) | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/metrics.md | 378 | # Step 2: Compute GAE advantages | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/metrics.md | 381 | # Step 3: PPO update (train on the same data for 10 epochs) | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 556 | ### Step 1: Define Multi-Dimensional Rubrics for AI Search | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 570 | ### Step 2: From Rubrics to Reward Model | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 599 | ### Step 3: Train Search Agent with RL | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 639 | ### Step 4: Reward Hacking Detection and Mitigation | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 651 | ### Step 5: Search Quality Evaluation and Iteration | COMMENT |
| LOW | docs/en/chapter24_deep_research/browser-rl-harness.md | 735 | #### Step 1: Create Training Environment | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 28 | # Step 1: sample G completions for one prompt, score each | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 31 | # Step 2: within-group normalization (subtract mean, divide by std) -> advantage | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 34 | # Step 3: PPO clipped loss (advantage comes from step 2, not a critic) | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 40 | # Step 4: k3 KL penalty (pull back, don't drift too far from reference) | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 44 | # Step 5: total loss | COMMENT |
| LOW | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 129 | # Step 1: the RM assigns a scalar score to each answer | COMMENT |
| 80 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | planning/outline-gap-analysis.md | 96 | | #1 | Anthropic's multi-agent research system (orchestrator-worker + OODA + independent context, 90.2% speedup) | CODE |
| MEDIUM | planning/outline-gap-analysis.md | 97 | | #3 | Agentic rollout infrastructure (Qwen3-Coder's 20,000 envs, Kimi K2's agentic rollout infra) | CODE |
| MEDIUM | planning/outline-final-0622.md | 1131 | ### 22.6 Multi-Agent Collaboration and Agent Swarms → `chapter10_agentic_rl/multi-agent-swarm.md` `[v5.2 new]` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1577 | ### 32.3 In-Context RL and the Next Decade → `chapter12_future_trends/llm-multi-agent-rl/index.md` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1613 | ### A.3 An Agent / Long-Trace Troubleshooting Checklist → `appendix_common_pitfalls/agentic-failure.md` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1635 | ### B.2 Agent Sandbox and Evaluation Engineering → `appendix_industrial_training/agentic-rl-infra.md` | COMMENT |
| MEDIUM | …8_grpo/verl_code_rlvr/run_qwen_coder_ppo_single_gpu.sh | 35 | # because a function implementation is usually longer than the reasoning trace for a math problem | COMMENT |
| MEDIUM | docs/chapter23_rl_based_swe/meta-swe-rl.md | 148 | ### RAG（Retrieval-Augmented Generation） | COMMENT |
| MEDIUM | …cs/chapter23_rl_based_swe/self-play-ssr-and-summary.md | 105 | # 2. Fixer 用 agentic 方式修复 | STRING |
| MEDIUM | docs/chapter24_deep_research/browser-rl-harness.md | 956 | [GRPO from Scratch](https://github.com/rasbt/reasoning-from-scratch)（Sebastian Raschka）训练 0.6B 模型在 MATH 数据集上做数学推理，从零实现 G | CODE |
| MEDIUM | docs/chapter24_deep_research/browser-rl-harness.md | 968 | [^tongyi_dr]: Tongyi DeepResearch Team. "Tongyi DeepResearch Technical Report." [arXiv:2510.24701](https://arxiv.org/abs | CODE |
| MEDIUM | docs/chapter24_deep_research/deep-research-eval.md | 353 | ### OpenResearcher 的文献 RAG | COMMENT |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 58 | > **LinkedIn 团队** 在使用 GPT-OSS（一个 MoE 架构的开源模型）进行 RL 训练时，遇到了梯度爆炸和奖励不增长的问题。经过排查，他们发现根本原因是训练框架中 **Attention Sink 参数的反向传播未被实现 | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 239 | > **LinkedIn 团队** 在使用 GPT-OSS 进行 RL 训练时发现，MoE 模型的路由网络（Gating Network）在两次前向传播中，可能为同一个 Token 选择不同的专家（Expert），导致 $\log \pi( | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 245 | > **Salesforce** 在其 SFR-RL 系统中提出了 **流水线同步 RL（Pipelined Synchronous）** 方案：所有 GPU 在 Rollout 和 Training 两个阶段之间交替切换，而非将 GPU | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 287 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 291 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/projects.md | 1602 | [^tongyi_dr]: Tongyi DeepResearch Team. "Tongyi DeepResearch Technical Report." [arXiv:2510.24701](https://arxiv.org/abs | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 58 | > **LinkedIn 团队** 在使用 GPT-OSS（一个 MoE 架构的开源模型）进行 RL 训练时，遇到了梯度爆炸和奖励不增长的问题。经过排查，他们发现根本原因是训练框架中 **Attention Sink 参数的反向传播未被实现 | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 239 | > **LinkedIn 团队** 在使用 GPT-OSS 进行 RL 训练时发现，MoE 模型的路由网络（Gating Network）在两次前向传播中，可能为同一个 Token 选择不同的专家（Expert），导致 $\log \pi( | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 245 | > **Salesforce** 在其 SFR-RL 系统中提出了 **流水线同步 RL（Pipelined Synchronous）** 方案：所有 GPU 在 Rollout 和 Training 两个阶段之间交替切换，而非将 GPU | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 287 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 291 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 1192 | [^benchmark-exploit]: Berkeley RDI. "[Trustworthy Benchmarks for Contamination](https://rdi.berkeley.edu/blog/trustworth | STRING |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 1194 | [^abc]: Zhu J, et al. "[Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://arxiv.org/abs/2507 | STRING |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 330 | [StepPO](https://arxiv.org/abs/2604.18401)（2026-04）指出：现有 LLM RL 算法继承了 RLHF/RLVR 的 **token-centric paradigm**——token 是建模和 | CODE |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 334 | [AT²PO](https://arxiv.org/abs/2601.04767)（2026-01）进一步把 tree search 引入 turn-level optimization。它针对多轮 agentic RL 的三个核心挑战—— | CODE |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 706 | [^casurvey]: Zhang C. "[From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-agents.md | 755 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-agents.md | 759 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-and-trajectory.md | 1146 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-and-trajectory.md | 1150 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter16_llm_rl_industrial/distributed-sync.md | 126 | [SGLang](https://arxiv.org/abs/2312.07104) 由 LMSYS 团队开发，在 agentic 场景下比 vLLM 更快：RadixAttention 用基数树管理 KV cache 支持跨请求复用，Pr | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 354 | Qwen 的公开资料分为四条线（[Qwen2.5](https://arxiv.org/abs/2412.15115) 通用指令后训练、[Qwen2.5-Math](https://arxiv.org/abs/2409.12122) 数学自 | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 386 | Kimi 的公开资料分三条线（[Kimi k1.5](https://arxiv.org/abs/2501.12599) 长思考 RL 简化框架、[Kimi K2](https://arxiv.org/abs/2507.20534) Age | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 414 | 字节 Seed 的公开材料（[Seed1.5-Thinking](https://arxiv.org/abs/2504.13914)、[VAPO](https://arxiv.org/abs/2504.05118)、[DAPO](https | CODE |
| MEDIUM⚡ | …hapter16_llm_rl_industrial/industrial-post-training.md | 446 | #### [Seed Prover 1.5](https://seed.bytedance.com/en/blog/seed-prover-1-5-advanced-mathematical-reasoning-through-a-nove | COMMENT |
| MEDIUM⚡ | …hapter16_llm_rl_industrial/industrial-post-training.md | 446 | #### [Seed Prover 1.5](https://seed.bytedance.com/en/blog/seed-prover-1-5-advanced-mathematical-reasoning-through-a-nove | COMMENT |
| MEDIUM⚡ | …hapter16_llm_rl_industrial/industrial-post-training.md | 448 | 形式化数学的环境是 theorem prover，而不是浏览器。动作是选择 tactic、生成 lemma、调用搜索器；reward 是 proof 是否通过、证明长度、搜索步数和中间 lemma 是否复用。它给 agent RL 的启发是 | CODE |
| MEDIUM⚡ | …hapter16_llm_rl_industrial/industrial-post-training.md | 452 | DeepSeek 的公开材料（[DeepSeekMath](https://arxiv.org/abs/2402.03300)、[DeepSeek-R1](https://arxiv.org/abs/2501.12948)、[DeepSee | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 678 | Operator 和 Codex 类模型把后训练扩展到 browser / software engineering episode。coding agent 的环境要包含仓库状态、测试命令、patch verifier、lint、用户指令 | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 786 | **AI2 Tulu 3**（[论文](https://openreview.net/forum?id=i1uGbfHHpH)）的价值在透明度而非 SOTA：数据、代码、recipe 和评测全公开，覆盖 SFT→偏好学习→RLVR 三阶段。 | CODE |
| MEDIUM | …s/appendix_industrial_training/industrial-exercises.md | 279 | [^openai-agentic]: OpenAI, "Researcher, Agentic Post-Training", <https://openai.com/careers/researcher-agentic-post-trai | CODE |
| MEDIUM⚡ | docs/appendix_paper_reading/learning-resources.md | 592 | | 现代后训练工程 | [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL)、[verl](https://github.com/verl-project/verl)、[Open | CODE |
| MEDIUM⚡ | docs/appendix_paper_reading/learning-resources.md | 594 | | 开源大模型底座 | [Qwen3.6](https://github.com/QwenLM/Qwen3.6)、[Qwen3](https://github.com/QwenLM/Qwen3)、[Meta Llama Mode | CODE |
| MEDIUM⚡ | docs/appendix_paper_reading/learning-resources.md | 596 | | Agent 框架与工具调用 | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python)、[Google ADK](https://github.com/go | CODE |
| MEDIUM | docs/appendix_paper_reading/learning-resources.md | 607 | - [Agentic RL 与工具调用](#agentic-rl-与工具调用) | CODE |
| MEDIUM | docs/appendix_paper_reading/learning-resources.md | 737 | | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) | tools、hand | CODE |
| MEDIUM | docs/appendix_paper_reading/learning-resources.md | 741 | | [AReaL](https://github.com/inclusionAI/AReaL) | 异步 LLM RL | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/game-projects.md | 423 | | Modern post-training engineering | [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL), [verl](https://github.com/verl | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/game-projects.md | 425 | | Open-source base models | [Qwen3.6](https://github.com/QwenLM/Qwen3.6), [Qwen3](https://github.com/QwenLM/Qwe | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/game-projects.md | 427 | | Agent frameworks & tool calling | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python), [Google ADK](h | CODE |
| MEDIUM | docs/en/chapter23_rl_based_swe/meta-swe-rl.md | 148 | ### RAG (Retrieval-Augmented Generation) | COMMENT |
| MEDIUM | …en/chapter23_rl_based_swe/self-play-ssr-and-summary.md | 118 | # 2. Fixer fixes the bug in an agentic manner | STRING |
| MEDIUM | …/en/chapter23_rl_based_swe/world-model-and-deep-swe.md | 11 | [Code World Model](https://arxiv.org/abs/2510.02387) (CWM, 2025.09) releases a 32B dense decoder-only model. It learns f | CODE |
| MEDIUM | docs/en/chapter24_deep_research/browser-rl-harness.md | 1267 | | **Agentic AI Researcher** (Owkin) | Qwen3-8B generates innovative drug target hypotheses, reward from 5-dimens | CODE |
| MEDIUM | docs/en/chapter24_deep_research/deep-research-eval.md | 329 | ### OpenResearcher: Literature RAG | COMMENT |
| MEDIUM | docs/en/chapter24_deep_research/deep-research-eval.md | 339 | - [Tongyi DeepResearch](https://github.com/Alibaba-NLP/DeepResearch) spans agentic continued pretraining, SFT, RL, and h | CODE |
| MEDIUM | docs/en/chapter22_agentic/overview.md | 337 | The agent loop above does not include training. [Search-R1](https://github.com/PeterGriffinJin/Search-R1) provides a com | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 58 | > When the **LinkedIn team** used GPT-OSS, an open-source model with an MoE architecture, for RL training, they encounte | CODE |
| 33 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2027 | let active = null, follow = true, saved = 0, internal = false, scheduled = false; | CODE |
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2028 | let goalPage = 0, goalSignature = "", goalPagerBound = null; | CODE |
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2029 | let catalogBefore = "", catalogBusyTimer = null, catalogBusyStarted = 0; | CODE |
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2097 | if (!button || button.disabled) return; | CODE |
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2106 | if (catalog?.dataset.catalogBusy === "true" && performance.now() - catalogBusyStarted > 120 && catalogResult() !== c | CODE |
| HIGH⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2111 | if (selectionPending && selectionDone && selectionDone !== selectionBefore) { | CODE |
| HIGH | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2055 | if (label || pageButton) { | CODE |
| HIGH | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2074 | const signature = labels.map(label => label.querySelector("input")?.value || label.textContent.trim()).join("|"); | CODE |
| HIGH | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2075 | const pageSize = Number(pager.dataset.pageSize || 6); | CODE |
| HIGH | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2133 | if (element && element !== active) { | CODE |
| HIGH | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 1036 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | modelscope-space/_shared/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 1209 | let active=null,follow=true,saved=0,internal=false; | CODE |
| HIGH | …de/appendix_common_pitfalls/debug_training_collapse.py | 253 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/appendix_common_pitfalls/debug_reward_hacking.py | 298 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/chapter08_rlhf/rlhf_ppo_train.py | 255 | Total loss = -L_CLIP + β * KL(π_θ || π_ref) | STRING |
| HIGH | code/chapter08_rlhf/rlhf_ppo_train.py | 295 | KL(π_θ || π_ref) = Σ π_θ * log(π_θ / π_ref) | STRING |
| HIGH | code/chapter04_dqn/double_dqn_cartpole.py | 256 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/chapter04_dqn/dqn_cartpole.py | 264 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 2023 | function initializeGymPlaygroundUi() { | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 278 | def warm_environment_runtimes() -> dict[str, str]: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 358 | def discover_environment_catalog() -> list[str]: | CODE |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 28 | def _configure_software_opengl() -> str: | CODE |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 26 | def _configure_software_vulkan() -> str | None: | CODE |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 121 | def _prepare_persistent_sapien_home() -> Path: | CODE |
| LOW | …pace/hands-on-modern-rl-experiment10-minestudio/app.py | 8 | def ensure_minestudio_simulator() -> None: | CODE |
| LOW⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 546 | def print_reward_design_principles(): | CODE |
| LOW⚡ | code/chapter08_rlhf/reward_model_training.py | 36 | def generate_preference_pairs(): | CODE |
| LOW⚡ | code/chapter08_rlhf/reward_model_training.py | 395 | def visualize_reward_distributions(chosen_scores, rejected_scores, save_path="output/reward_distribution.png"): | CODE |
| LOW⚡ | code/chapter03_mdp/bellman_equation_verify.py | 80 | def manual_bellman_expectation(): | CODE |
| LOW⚡ | code/chapter03_mdp/bellman_equation_verify.py | 412 | def print_value_iteration_steps(n_steps=10): | CODE |
| LOW | code/chapter10_agentic_rl/multi_turn_rl.py | 206 | def compute_discounted_returns(rewards, gamma=0.99): | CODE |
| LOW | code/chapter10_agentic_rl/mini_deep_research_grpo.py | 88 | def search_validity_reward_func(completions, **kwargs) -> list[float]: | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 97 | def generate_correct_response(gt): | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 395 | def print_before_after_comparison(samples, history, seed=42): | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 529 | def demonstrate_grpo_normalization(): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 115 | def generate_short_correct_response(gt): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 134 | def generate_partially_correct_response(gt): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 148 | def generate_low_quality_response(gt): | CODE |
| LOW⚡ | …e/chapter05_policy_gradient/reinforce_with_baseline.py | 211 | def train_reinforce_with_baseline(num_episodes=500, gamma=0.99, lr=1e-3): | CODE |
| LOW | code/chapter05_policy_gradient/actor_critic_cartpole.py | 155 | def compute_actor_critic_losses( | CODE |
| LOW | code/chapter12_future_trends/multi_agent_marl.py | 747 | def print_cooperation_statistics(ind_results, shared_results, n_agents): | CODE |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 88 | def _require_execution_opt_in() -> None: | CODE |
| LOW⚡ | code/chapter09_alignment/dpo_math_reward.py | 36 | def create_math_preference_data(): | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 49 | def maybe_make_swanlab_callback(args: argparse.Namespace): | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 211 | def ensure_discrete_action_space(env_id: str) -> None: | CODE |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 53 | def maybe_make_swanlab_callback(args: argparse.Namespace): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 89 | def generate_synthetic_rewards(group_size=8, seed=42): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 141 | def demonstrate_grpo_step_by_step(): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 239 | def plot_reward_normalization(rewards, grpo_advantages): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 310 | def plot_advantage_comparison(grpo_advantages, ppo_advantages): | CODE |
| LOW | code/chapter09_grpo_rlvr/grpo_mechanism.py | 114 | def simulate_critic_predictions(rewards, noise_scale=0.08, seed=123): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 129 | def extract_answer_from_response(response): | CODE |
| LOW | docs/chapter28_vla/embodied-intelligence/index.md | 176 | def apply_domain_randomization(self): | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 369 | function updateSidebarEdgePosition() { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 379 | function getSidebarScrollContainer() { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 404 | function syncSidebarGroupsExpanded() { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 672 | function handleMermaidViewerImageLoad(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 692 | function handleMermaidViewerPointerDown(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 709 | function handleMermaidViewerPointerMove(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 724 | function handleMermaidViewerKeydown(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 814 | function updateLineHeightFromRange(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 832 | function updateSidebarWidthFromPointer(clientX) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 886 | function scrollOutlineToActiveItem(activeLink) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 907 | function scrollSidebarToActiveItem(activeItem) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 947 | function getSidebarNavigationGroup(path) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 965 | function scrollSidebarToNavigationGroup(path) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 1029 | function scheduleRenderOutlineMath() { | CODE |
| LOW | docs/chapter23_rl_based_swe/swe-bench-and-rlvr.md | 42 | def test_invalid_input_is_not_hidden(): | CODE |
| LOW | docs/chapter24_deep_research/browser-rl-harness.md | 552 | def train_search_reward_model(preference_data, base_model): | CODE |
| LOW | docs/chapter32_selfplay/llm-multi-agent-rl/index.md | 291 | def compute_multi_agent_reward(self, trajectories): | CODE |
| LOW | docs/appendix_code_cheatsheet/attention-mha.md | 41 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/appendix_code_cheatsheet/attention-mha.md | 66 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/chapter15_rlhf/imitation-learning-pipeline.md | 400 | def construct_preference_pairs(prompt, responses, judge_outputs): | STRING |
| LOW | docs/chapter22_agentic/industrial-practice.md | 182 | def hallucination_aware_reward(answer, tool_results, citations): | CODE |
| LOW | docs/chapter22_agentic/projects.md | 1295 | def train_search_reward_model(preference_data, base_model): | CODE |
| LOW | docs/chapter22_agentic/industrial-evaluation.md | 182 | def hallucination_aware_reward(answer, tool_results, citations): | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 541 | async def rollout_single_trajectory(model, task, sandbox, max_turns=10): | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 609 | def comprehensive_agent_reward(trajectory, final_result, task): | CODE |
| 48 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 240 | except Exception: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 297 | except Exception as exc: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1515 | except Exception as exc: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1568 | except Exception as exc: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1666 | except Exception as exc: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1736 | except Exception as exc: | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1848 | except Exception as exc: last_preview=result_preview_image(experiment,"Training complete",f"{score:.1f}","Re | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1868 | except Exception as exc: last_preview=result_preview_image(MOUNTAINCAR,"Training complete",f"{score:.1f}","R | CODE |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1902 | except Exception as exc: | CODE |
| LOW | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 130 | except Exception as exc: | CODE |
| LOW | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 325 | except Exception: | CODE |
| LOW | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 328 | except Exception: | CODE |
| MEDIUM | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 122 | def runtime_status() -> str: | CODE |
| LOW | …-on-modern-rl-experiment11-unity-mlagents/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …hands-on-modern-rl-experiment03-atari/space_runtime.py | 106 | except Exception as exc: | CODE |
| MEDIUM | …hands-on-modern-rl-experiment03-atari/space_runtime.py | 92 | def runtime_status(): | CODE |
| LOW | …ace/hands-on-modern-rl-experiment03-atari/sb3_tools.py | 380 | except Exception: | CODE |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 874 | except Exception as exc: | CODE |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 924 | except Exception as exc: | CODE |
| LOW | …dern-rl-experiment05-multiagent-games/space_runtime.py | 87 | except Exception as exc: | CODE |
| MEDIUM | …dern-rl-experiment05-multiagent-games/space_runtime.py | 78 | def runtime_status(): | CODE |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 93 | except Exception as exc: | CODE |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 208 | except Exception: | CODE |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 211 | except Exception: | CODE |
| MEDIUM | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 87 | def runtime_status() -> str: | CODE |
| LOW | …n-modern-rl-experiment12-ai2thor-embodied/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …rn-rl-experiment06-minigrid-adventure/space_runtime.py | 73 | except Exception as exc: | CODE |
| MEDIUM | …rn-rl-experiment06-minigrid-adventure/space_runtime.py | 64 | def runtime_status(): | CODE |
| LOW | …modern-rl-experiment06-minigrid-adventure/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …nds-on-modern-rl-experiment02-vizdoom/space_runtime.py | 74 | except Exception as exc: | CODE |
| MEDIUM | …nds-on-modern-rl-experiment02-vizdoom/space_runtime.py | 64 | def runtime_status(): | CODE |
| LOW | …e/hands-on-modern-rl-experiment02-vizdoom/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …modern-rl-experiment04-board-selfplay/space_runtime.py | 74 | except Exception as exc: | CODE |
| LOW | …modern-rl-experiment04-board-selfplay/space_runtime.py | 174 | except Exception: | CODE |
| LOW | …modern-rl-experiment04-board-selfplay/space_runtime.py | 177 | except Exception: | CODE |
| MEDIUM | …modern-rl-experiment04-board-selfplay/space_runtime.py | 67 | def runtime_status(): | CODE |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | modelscope-space/_shared/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | modelscope-space/_shared/game_ui.py | 1033 | except Exception as exc: | STRING |
| LOW | modelscope-space/_shared/game_ui.py | 1098 | except Exception as exc: | STRING |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 205 | except Exception as exc: | CODE |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 250 | except Exception as exc: | CODE |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 439 | except Exception as exc: | CODE |
| MEDIUM | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 178 | def _warmup_runtime() -> None: | CODE |
| MEDIUM | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 239 | def runtime_status() -> str: | CODE |
| LOW | …hands-on-modern-rl-experiment08-maniskill/sb3_tools.py | 215 | except Exception: | CODE |
| LOW | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 1033 | except Exception as exc: | STRING |
| 48 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | …de/appendix_common_pitfalls/debug_training_collapse.py | 39 | # Create the output directory | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 37 | # Create the output directory | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/sft_pipeline.py | 21 | # Create the output directory | COMMENT |
| MEDIUM⚡ | code/chapter09_continuous_control/sac_halfcheetah.py | 213 | # Create the training monitoring callback | COMMENT |
| MEDIUM⚡ | code/chapter09_continuous_control/sac_halfcheetah.py | 325 | # Create a separate environment for evaluation | COMMENT |
| MEDIUM⚡ | code/chapter10_agentic_rl/multi_turn_rl.py | 27 | # Create the output directory | COMMENT |
| MEDIUM⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 30 | # Create the output directory | COMMENT |
| MEDIUM⚡ | …e/chapter05_policy_gradient/reinforce_with_baseline.py | 28 | # Create the output directory | COMMENT |
| MEDIUM⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 29 | # Create the output directory | COMMENT |
| MEDIUM | code/chapter09_alignment/dpo_math_reward.py | 289 | # Create the DPOTrainer | COMMENT |
| MEDIUM | code/chapter09_alignment/dpo_hands_on.py | 157 | # Create the DPOTrainer | COMMENT |
| MEDIUM | code/chapter04_dqn/double_dqn_cartpole.py | 291 | # Create an environment to get dimension info | COMMENT |
| MEDIUM | code/chapter04_dqn/dqn_cartpole.py | 233 | # Create the environment and the agent | COMMENT |
| MEDIUM | code/chapter09_grpo_rlvr/rule_based_reward.py | 372 | # Define the test cases | COMMENT |
| MEDIUM⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 29 | # Create the output directory | COMMENT |
| MEDIUM | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 407 | # Create the optimizer | COMMENT |
| MEDIUM⚡ | code/chapter07_ppo/ppo_from_scratch.py | 392 | # Create the environment | COMMENT |
| MEDIUM⚡ | code/chapter07_ppo/ppo_from_scratch.py | 397 | # Create the model and optimizer | COMMENT |
| MEDIUM⚡ | code/chapter07_ppo/ppo_lunar_lander.py | 138 | # Create the training monitor callback | COMMENT |
| MEDIUM⚡ | code/chapter07_ppo/ppo_lunar_lander.py | 219 | # Create a separate environment for evaluation | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 753 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 794 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1472 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1577 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1713 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1776 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1796 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1816 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1836 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1856 | CODE | |
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 1876 | CODE | |
| LOW | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 540 | CODE | |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 741 | CODE | |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 879 | CODE | |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 1061 | CODE | |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 589 | CODE | |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 739 | CODE | |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 741 | CODE | |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 879 | CODE | |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 1061 | CODE | |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 115 | CODE | |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 741 | CODE | |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 879 | CODE | |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 1061 | CODE | |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 741 | CODE | |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 879 | CODE | |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 1061 | CODE | |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 741 | CODE | |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 879 | CODE | |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 1061 | CODE | |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 741 | CODE | |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 879 | CODE | |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 1061 | CODE | |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 741 | CODE | |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 879 | CODE | |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 1061 | CODE | |
| LOW | modelscope-space/_shared/game_ui.py | 741 | CODE | |
| LOW | modelscope-space/_shared/game_ui.py | 879 | CODE | |
| LOW | modelscope-space/_shared/game_ui.py | 1061 | CODE | |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 178 | CODE | |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 421 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 741 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 879 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 1061 | CODE | |
| LOW | …-on-modern-rl-experiment10-minestudio/space_runtime.py | 95 | CODE | |
| LOW | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 741 | CODE | |
| LOW | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 879 | CODE | |
| LOW | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 1061 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 741 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 879 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 1061 | CODE | |
| LOW | …de/appendix_common_pitfalls/debug_training_collapse.py | 211 | CODE | |
| LOW | code/chapter08_rlhf/rlhf_ppo_train.py | 191 | CODE | |
| LOW | code/chapter03_mdp/gridworld_q_learning.py | 253 | CODE | |
| LOW | code/chapter03_mdp/gridworld_q_learning.py | 282 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 146 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 192 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 412 | CODE | |
| LOW | code/chapter01_cartpole/2-pytorch_ppo.py | 310 | CODE | |
| LOW | code/chapter11_vlm_rl/geometry_counting_dataset.py | 126 | CODE | |
| 14 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …e-space/hands-on-modern-rl-experiment-gymnasium/app.py | 3 | CODE | |
| LOW | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 1 | CODE | |
| LOW | …-on-modern-rl-experiment11-unity-mlagents/sb3_tools.py | 1 | CODE | |
| LOW | …ds-on-modern-rl-experiment11-unity-mlagents/game_ui.py | 1 | CODE | |
| LOW | …n-rl-experiment11-unity-mlagents/bootstrap_mlagents.py | 1 | CODE | |
| LOW | …xperiment11-unity-mlagents/bootstrap_native_runtime.py | 1 | CODE | |
| LOW | …hands-on-modern-rl-experiment03-atari/space_runtime.py | 1 | CODE | |
| LOW | …ace/hands-on-modern-rl-experiment03-atari/sb3_tools.py | 1 | CODE | |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 1 | CODE | |
| LOW | …space/hands-on-modern-rl-experiment03-atari/game_ui.py | 20 | CODE | |
| LOW | …dern-rl-experiment05-multiagent-games/space_runtime.py | 1 | CODE | |
| LOW | …-on-modern-rl-experiment05-multiagent-games/game_ui.py | 1 | CODE | |
| LOW | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 1 | CODE | |
| LOW | …n-modern-rl-experiment12-ai2thor-embodied/sb3_tools.py | 1 | CODE | |
| LOW | …-on-modern-rl-experiment12-ai2thor-embodied/game_ui.py | 1 | CODE | |
| LOW | …rn-rl-experiment06-minigrid-adventure/space_runtime.py | 1 | CODE | |
| LOW | …modern-rl-experiment06-minigrid-adventure/sb3_tools.py | 1 | CODE | |
| LOW | …n-modern-rl-experiment06-minigrid-adventure/game_ui.py | 1 | CODE | |
| LOW | …nds-on-modern-rl-experiment02-vizdoom/space_runtime.py | 1 | CODE | |
| LOW | …nds-on-modern-rl-experiment02-vizdoom/space_runtime.py | 80 | CODE | |
| LOW | …nds-on-modern-rl-experiment02-vizdoom/space_runtime.py | 69 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment02-vizdoom/sb3_tools.py | 1 | CODE | |
| LOW | …ace/hands-on-modern-rl-experiment02-vizdoom/game_ui.py | 1 | CODE | |
| LOW | …ds-on-modern-rl-experiment01-cartpole/space_runtime.py | 1 | CODE | |
| LOW | …ce/hands-on-modern-rl-experiment01-cartpole/game_ui.py | 1 | CODE | |
| LOW | …pace/hands-on-modern-rl-experiment01-cartpole/train.py | 8 | CODE | |
| LOW | …-space/hands-on-modern-rl-experiment01-cartpole/app.py | 1 | CODE | |
| LOW | …modern-rl-experiment04-board-selfplay/space_runtime.py | 1 | CODE | |
| LOW | …modern-rl-experiment04-board-selfplay/space_runtime.py | 3 | CODE | |
| LOW | …ds-on-modern-rl-experiment04-board-selfplay/game_ui.py | 1 | CODE | |
| LOW | modelscope-space/_shared/sb3_tools.py | 1 | CODE | |
| LOW | modelscope-space/_shared/game_ui.py | 1 | CODE | |
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 1 | CODE | |
| LOW | …hands-on-modern-rl-experiment08-maniskill/sb3_tools.py | 1 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment08-maniskill/game_ui.py | 1 | CODE | |
| LOW | …-on-modern-rl-experiment10-minestudio/space_runtime.py | 1 | CODE | |
| LOW | …ands-on-modern-rl-experiment10-minestudio/sb3_tools.py | 1 | CODE | |
| LOW | …/hands-on-modern-rl-experiment10-minestudio/game_ui.py | 1 | CODE | |
| LOW | …pace/hands-on-modern-rl-experiment10-minestudio/app.py | 1 | CODE | |
| LOW | …s-on-modern-rl-experiment07-jax-games/space_runtime.py | 1 | CODE | |
| LOW | …e/hands-on-modern-rl-experiment07-jax-games/game_ui.py | 1 | CODE | |
| LOW | code/device_utils.py | 3 | CODE | |
| LOW | code/chapter03_mdp/gridworld_q_learning.py | 18 | CODE | |
| LOW | code/chapter10_agentic_rl/mini_deep_research_grpo.py | 26 | CODE | |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 28 | CODE | |
| LOW | code/chapter05_policy_gradient/reinforce_cartpole.py | 27 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 23 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 24 | CODE | |
| LOW | code/chapter17_dpo/2-test_before.py | 3 | CODE | |
| LOW | code/chapter17_dpo/4-test_after.py | 3 | CODE | |
| LOW | code/chapter09_alignment/dpo_math_reward.py | 23 | CODE | |
| LOW | code/chapter09_alignment/dpo_math_reward.py | 25 | CODE | |
| LOW | code/chapter09_alignment/dpo_hands_on.py | 17 | CODE | |
| LOW | code/chapter09_alignment/dpo_hands_on.py | 18 | CODE | |
| LOW | code/chapter04_dqn/render_atari.py | 17 | CODE | |
| LOW | code/chapter04_dqn/dqn_pokemon_red_pyboy.py | 17 | CODE | |
| LOW | code/chapter04_dqn/export_dqn_curves.py | 8 | CODE | |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 14 | CODE | |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 14 | CODE | |
| LOW | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 30 | CODE | |
| 5 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …modern-rl-experiment11-unity-mlagents/space_runtime.py | 328 | except Exception: | CODE |
| HIGH | …-on-modern-rl-experiment11-unity-mlagents/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | …hands-on-modern-rl-experiment03-atari/space_runtime.py | 249 | except (OSError, json.JSONDecodeError): | CODE |
| HIGH | …ace/hands-on-modern-rl-experiment03-atari/sb3_tools.py | 380 | except Exception: | CODE |
| HIGH | …dern-rl-experiment12-ai2thor-embodied/space_runtime.py | 211 | except Exception: | CODE |
| HIGH | …n-modern-rl-experiment12-ai2thor-embodied/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | …modern-rl-experiment06-minigrid-adventure/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | …e/hands-on-modern-rl-experiment02-vizdoom/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | modelscope-space/_shared/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | …hands-on-modern-rl-experiment08-maniskill/sb3_tools.py | 215 | except Exception: | CODE |
| HIGH | …-on-modern-rl-experiment10-minestudio/space_runtime.py | 225 | except Exception: | CODE |
| HIGH | …ands-on-modern-rl-experiment10-minestudio/sb3_tools.py | 215 | except Exception: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 256 | # Step 1: determine the query type | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 259 | # Step 2: sample a tool according to the policy | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 263 | # Step 3: execute the tool and get the reward | COMMENT |
| LOW⚡ | code/chapter10_agentic_rl/tool_use_agent.py | 272 | # Step 4: update the policy | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 428 | # Step 1: generate the dataset | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 438 | # Step 2: print statistics | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/geometry_counting_dataset.py | 441 | # Step 3: display sample images | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/multi_modal_reward.py | 561 | # Step 1: run the reward function tests | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/multi_modal_reward.py | 564 | # Step 2: visualize the reward weights | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 600 | # Step 1: print the VLM GRPO overview | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 603 | # Step 2: GRPO core computation demo | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 606 | # Step 3: create training data | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 620 | # Step 4: simulate GRPO training | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 630 | # Step 5: before/after training comparison | COMMENT |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 633 | # Step 6: plot training curves | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 253 | # Step 1: collect a full episode trajectory with the current policy | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 256 | # Step 2: compute the discounted cumulative returns | COMMENT |
| LOW⚡ | code/chapter05_policy_gradient/reinforce_cartpole.py | 259 | # Step 3: perform the policy gradient update | COMMENT |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 232 | # Step 1: expand all nodes at the current level, generating children | COMMENT |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 247 | # Step 2: sort by score, keep the top-k | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 208 | # Step 1: use the Q network to select the optimal action | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 212 | # Step 2: use the target network to evaluate those actions' Q values | COMMENT |
| LOW⚡ | code/chapter04_dqn/double_dqn_cartpole.py | 216 | # Step 3: compute the target value | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 439 | # Step 1: generate a group of responses for the current question | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 446 | # Step 2: compute the reward for each response | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 452 | # Step 3: GRPO group-relative normalization | COMMENT |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 455 | # Step 4: policy gradient update | COMMENT |
| LOW | code/chapter07_ppo/ppo_from_scratch.py | 430 | # Step 1: collect trajectories | COMMENT |
| LOW | code/chapter07_ppo/ppo_from_scratch.py | 441 | # Step 2: PPO update | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 21 | COMMENT | |
| LOW | …chapter09_continuous_control/ppo_td3_sac_comparison.py | 201 | # ---- Algorithm 3: SAC ---- | COMMENT |
| LOW | code/chapter09_continuous_control/sac_halfcheetah.py | 141 | # Breakdown of SAC's key hyperparameters: | COMMENT |
| LOW | code/chapter18_grpo/verl_code_rlvr/prepare_data.py | 1 | # prepare_data.py | COMMENT |
| LOW | …8_grpo/verl_code_rlvr/run_qwen_coder_ppo_single_gpu.sh | 1 | #!/bin/bash | COMMENT |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 1 | # code_reward.py | COMMENT |
| LOW | docs/chapter28_vla/embodied-intelligence/index.md | 521 | # 1. 创建 MuJoCo 仿真环境（半猎豹机器人，状态连续，动作连续） | COMMENT |
| LOW | docs/chapter32_selfplay/rl-scaling-outlook.md | 61 | COMMENT | |
| LOW | docs/chapter32_selfplay/rl-scaling-outlook.md | 161 | # 奖励 = 答案是否正确（规则验证） | COMMENT |
| LOW | docs/en/chapter28_vla/embodied-intelligence/index.md | 481 | COMMENT | |
| LOW | docs/en/chapter32_selfplay/rl-scaling-outlook.md | 61 | # ========================================== | COMMENT |
| LOW | docs/chapter10_ppo/gae-reward-model.md | 301 | ```python | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | docs/en/chapter24_deep_research/browser-rl-harness.md | 356 | **Problem**: Synthetic data often faces the "not hard enough" problem — research trajectories generated by GPT-4 level m | CODE |
| HIGH | docs/en/chapter22_agentic/projects.md | 1069 | **Problem**: Synthetic data often suffers from insufficient difficulty. Research trajectories generated by GPT-4-class m | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | code/chapter10_agentic_rl/generate_synthetic_data.py | 30 | api_key=os.environ.get("OPENAI_API_KEY", "your-api-key-here"), | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter18_grpo/verl_code_rlvr/prepare_data.py | 26 | # Usage: | COMMENT |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 16 | # Usage: | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | package-lock.json | 5786 | "resolved": "https://registry.npmmirror.com/robust-predicates/-/robust-predicates-3.0.3.tgz", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …s-on-modern-rl-experiment08-maniskill/space_runtime.py | 111 | def _set_warmup_state(phase: str, detail: str, progress: float = 0.0, error: str | None = None) -> None: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter09_alignment/dpo_hands_on.py | 100 | CODE |