🚀 An open-source, hands-on curriculum bridging the gap from basic RL concepts to LLM alignment, RLVR, and advanced Agentic systems.
This report presents the forensic synthetic code analysis of walkinglabs/hands-on-modern-rl, a Python project with 3,670 GitHub stars. SynthScan v2.0 examined 165,476 lines of code across 494 source files, recording 1067 pattern matches distributed across 14 syntactic categories. The overall adjusted score of 16.8 places this repository in the Moderate AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
Longitudinal tracking requires multiple scan runs. Once this repository is re-scanned after new commits land, this chart will visualise how the synthetic code signal evolves over time — enabling you to detect whether AI authorship is growing, stabilising, or being actively corrected by human engineers.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 1067 distinct pattern matches across 14 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 46 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 48 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 96 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 98 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 207 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 209 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 311 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 313 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 435 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 437 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 537 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 539 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 645 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 647 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 746 | # ========================================== | COMMENT |
| MEDIUM | …de/appendix_common_pitfalls/debug_training_collapse.py | 748 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 41 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 43 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 157 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 159 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 234 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 236 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 315 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 317 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 389 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 391 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 460 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 462 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 524 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 526 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 590 | # ========================================== | COMMENT |
| MEDIUM | code/appendix_common_pitfalls/debug_reward_hacking.py | 592 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 23 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 25 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 99 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 101 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 125 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/sft_pipeline.py | 127 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 38 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 40 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 98 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 100 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 219 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 221 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 419 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 421 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 467 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/rlhf_ppo_train.py | 469 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 32 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 34 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 153 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 155 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 213 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 215 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 265 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 267 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 390 | # ========================================== | COMMENT |
| MEDIUM⚡ | code/chapter08_rlhf/reward_model_training.py | 392 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 427 | # ========================================== | COMMENT |
| MEDIUM | code/chapter08_rlhf/reward_model_training.py | 429 | # ========================================== | COMMENT |
| 652 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 498 | # Step 1: 把对话历史拼成模型能理解的 prompt | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 500 | # Step 2: 模型生成动作(推理,不计算梯度) | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 502 | # Step 3: 从自由文本输出中解析结构化动作 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 516 | # Step 4: 环境执行动作,返回观测和终止状态 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 519 | # Step 5: 记录本轮交互到轨迹 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 527 | # Step 6: 把本轮交互加入对话历史,供下一轮使用 | COMMENT |
| LOW⚡ | docs/chapter22_agentic/build-agentic-training-system.md | 535 | # Step 7: 计算整条轨迹的 reward(只有轨迹结束时才给) | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 293 | # Step 1: student 生成回答 | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 304 | # Step 2: 计算 student 和 teacher 的逐 token log-prob | COMMENT |
| LOW | docs/chapter18_grpo/on-policy-distillation.md | 319 | # Step 3: 计算逐 token reward(teacher 认可度 - student 自信度) | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 342 | # Step 1: student rollout | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 347 | # Step 2: 计算 per-token advantage | COMMENT |
| LOW⚡ | docs/chapter18_grpo/on-policy-distillation.md | 351 | # Step 3: 策略梯度更新 | COMMENT |
| LOW | docs/chapter29_visual_generation/intro.md | 742 | # Step 3: score final images | COMMENT |
| LOW | docs/chapter29_visual_generation/intro.md | 746 | # Step 4: turn rewards into advantages | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/principles.md | 360 | # Step 1: Collect experience data (2048 steps) | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/principles.md | 363 | # Step 2: Compute GAE advantages | COMMENT |
| LOW⚡ | docs/en/chapter01_cartpole/principles.md | 366 | # Step 3: PPO update (train on the same data for 10 epochs) | COMMENT |
| LOW | docs/en/chapter01_cartpole/intro.md | 41 | ### Step 1: Install Dependencies | COMMENT |
| LOW | docs/en/chapter01_cartpole/intro.md | 51 | ### Step 2: Run Training | COMMENT |
| LOW | docs/en/chapter01_cartpole/intro.md | 80 | ### Step 3: Where to View SwanLab Training Curves | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 28 | # Step 1: sample G completions for one prompt, score each | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 31 | # Step 2: within-group normalization (subtract mean, divide by std) -> advantage | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 34 | # Step 3: PPO clipped loss (advantage comes from step 2, not a critic) | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 40 | # Step 4: k3 KL penalty (pull back, don't drift too far from reference) | COMMENT |
| LOW⚡ | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 44 | # Step 5: total loss | COMMENT |
| LOW | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 129 | # Step 1: the RM assigns a scalar score to each answer | COMMENT |
| LOW | docs/en/appendix_code_cheatsheet/grpo-rlvr.md | 133 | # Step 2: we want r_w > r_l; sigmoid the diff and take the negative log | COMMENT |
| LOW | docs/en/chapter15_rlhf/standard-rlhf-pipeline.md | 103 | ## Step 0: Choose a Base Checkpoint | COMMENT |
| LOW | docs/en/chapter15_rlhf/standard-rlhf-pipeline.md | 125 | ## Step 1: SFT Teaches "How To Answer" | COMMENT |
| LOW | docs/en/chapter15_rlhf/standard-rlhf-pipeline.md | 137 | ## Step 2: The Reward Model Teaches "What Is Better" | COMMENT |
| LOW | docs/en/chapter15_rlhf/standard-rlhf-pipeline.md | 169 | ## Step 3: PPO-RLHF Optimizes the Policy Under Constraints | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 453 | ### Step 1: Define Multi-Dimensional Rubrics for AI Search | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 467 | ### Step 2: From Rubrics to Reward Model | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 496 | ### Step 3: Train Search Agent with RL | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 536 | ### Step 4: Reward Hacking Detection and Mitigation | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 548 | ### Step 5: Search Quality Evaluation and Iteration | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 632 | #### Step 1: Create Training Environment | COMMENT |
| LOW | docs/en/chapter22_agentic/agent-data-swe-smith.md | 45 | ## Step 1: Environment Construction: One Repository, One Docker Image | COMMENT |
| LOW | docs/en/chapter22_agentic/agent-data-swe-smith.md | 59 | ## Step 2: Bug Injection: Four Strategies | COMMENT |
| LOW | docs/en/chapter22_agentic/agent-data-swe-smith.md | 172 | ## Step 3: Test Validation: Use the Test Suite as a Sieve | COMMENT |
| LOW | docs/en/chapter22_agentic/agent-data-swe-smith.md | 181 | ## Step 4: Generate Issue Text | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 316 | ## Step 2: Agent Prompt and Tool-Call Parsing | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 368 | ## Step 3: Multi-Tool Agent Rollout | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 518 | ## Step 4: Baseline Evaluation: Single-Turn No-Tool vs Single-Tool vs Multi-Tool | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 585 | ## Step 5: Batch Rollout and GRPO RL Training | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 720 | ## Step 6: Post-Training Evaluation: Did the Model Really Learn a Multi-Tool Strategy? | COMMENT |
| LOW⚡ | docs/en/chapter22_agentic/projects.md | 1398 | ### Step 1: Build the Agent Environment | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 94 | ## Step 1: Load the Model, Build the Tool Environment, and Load Data | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1269 | ### Step 1: Define Multidimensional Rubrics for AI Search | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1283 | ### Step 2: From Rubrics to Reward Model | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1312 | ### Step 3: Train the Search Agent with RL | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1352 | ### Step 4: Detect and Mitigate Reward Hacking | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1364 | ### Step 5: Evaluate and Iterate on Search Quality | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1455 | ### Step 2: Define the Tool-Call Format | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1546 | ### Step 4: Run Training | COMMENT |
| LOW | docs/en/chapter22_agentic/projects.md | 1485 | ### Step 3: GRPO Training Framework | STRING |
| LOW | docs/en/chapter22_agentic/rllm-deepcoder-lab.md | 106 | ## Step 1: Designing Rewards with rLLM | COMMENT |
| LOW | docs/en/chapter22_agentic/rllm-deepcoder-lab.md | 285 | ## Step 2: What Does the Data Look Like? | COMMENT |
| LOW | docs/en/chapter22_agentic/rllm-deepcoder-lab.md | 387 | ## Step 3: What Does Model Output Look Like? How Does the Evaluator Score It? | STRING |
| 51 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | planning/outline-gap-analysis.md | 97 | | #3 | Agentic Rollout 基础设施(Qwen3-Coder 20000 envs,Kimi K2 agentic rollout infra) | | CODE |
| MEDIUM | planning/outline-final-0622.md | 1131 | ### 22.6 多智能体协作与 Agent Swarm → `chapter10_agentic_rl/multi-agent-swarm.md` `[v5.2 新增]` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1577 | ### 32.3 In-Context RL 与未来十年 → `chapter12_future_trends/llm-multi-agent-rl/index.md` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1613 | ### A.3 Agent / Long-trace 排查清单 → `appendix_common_pitfalls/agentic-failure.md` | COMMENT |
| MEDIUM | planning/outline-final-0622.md | 1635 | ### B.2 Agent 沙箱与评测工程 → `appendix_industrial_training/agentic-rl-infra.md` | COMMENT |
| MEDIUM⚡ | docs/appendix_game_projects/intro.md | 426 | | 现代后训练工程 | [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL)、[verl](https://github.com/verl-project/verl)、[Open | CODE |
| MEDIUM⚡ | docs/appendix_game_projects/intro.md | 428 | | 开源大模型底座 | [Qwen3.6](https://github.com/QwenLM/Qwen3.6)、[Qwen3](https://github.com/QwenLM/Qwen3)、[Meta Llama Mode | CODE |
| MEDIUM⚡ | docs/appendix_game_projects/intro.md | 430 | | Agent 框架与工具调用 | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python)、[Google ADK](https://github.com/go | CODE |
| MEDIUM | docs/appendix_game_projects/intro.md | 441 | - [Agentic RL 与工具调用](#agentic-rl-与工具调用) | CODE |
| MEDIUM | docs/appendix_game_projects/intro.md | 571 | | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) | tools、hand | CODE |
| MEDIUM | docs/appendix_game_projects/intro.md | 575 | | [AReaL](https://github.com/inclusionAI/AReaL) | 异步 LLM RL | CODE |
| MEDIUM | docs/chapter19_reasoning/adaptive-thinking.md | 173 | ### 推理与 agentic 的融合 | COMMENT |
| MEDIUM | docs/chapter23_rl_based_swe/meta-swe-rl.md | 148 | ### RAG(Retrieval-Augmented Generation) | COMMENT |
| MEDIUM | …cs/chapter23_rl_based_swe/self-play-ssr-and-summary.md | 114 | # 2. Fixer 用 agentic 方式修复 | STRING |
| MEDIUM | docs/chapter24_deep_research/intro.md | 971 | | **Agentic AI 研究员**(Owkin) | Qwen3-8B 生成创新药物靶点假说,reward 由 5 维 LLM judge 面板给出(新颖性 / 有效性 / 可药性 / 可行性 / 商业价值) | CODE |
| MEDIUM | docs/chapter24_deep_research/intro.md | 983 | [^tongyi_dr]: Tongyi DeepResearch Team. "Tongyi DeepResearch Technical Report." [arXiv:2510.24701](https://arxiv.org/abs | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 62 | > **LinkedIn 团队** 在使用 GPT-OSS(一个 MoE 架构的开源模型)进行 RL 训练时,遇到了梯度爆炸和奖励不增长的问题。经过排查,他们发现根本原因是训练框架中 **Attention Sink 参数的反向传播未被实现 | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 243 | > **LinkedIn 团队** 在使用 GPT-OSS 进行 RL 训练时发现,MoE 模型的路由网络(Gating Network)在两次前向传播中,可能为同一个 Token 选择不同的专家(Expert),导致 $\log \pi( | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 249 | > **Salesforce** 在其 SFR-RL 系统中提出了 **流水线同步 RL(Pipelined Synchronous)** 方案:所有 GPU 在 Rollout 和 Training 两个阶段之间交替切换,而非将 GPU | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 291 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-practice.md | 295 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/projects.md | 1602 | [^tongyi_dr]: Tongyi DeepResearch Team. "Tongyi DeepResearch Technical Report." [arXiv:2510.24701](https://arxiv.org/abs | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 58 | > **LinkedIn 团队** 在使用 GPT-OSS(一个 MoE 架构的开源模型)进行 RL 训练时,遇到了梯度爆炸和奖励不增长的问题。经过排查,他们发现根本原因是训练框架中 **Attention Sink 参数的反向传播未被实现 | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 239 | > **LinkedIn 团队** 在使用 GPT-OSS 进行 RL 训练时发现,MoE 模型的路由网络(Gating Network)在两次前向传播中,可能为同一个 Token 选择不同的专家(Expert),导致 $\log \pi( | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 245 | > **Salesforce** 在其 SFR-RL 系统中提出了 **流水线同步 RL(Pipelined Synchronous)** 方案:所有 GPU 在 Rollout 和 Training 两个阶段之间交替切换,而非将 GPU | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 287 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 291 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 1192 | [^benchmark-exploit]: Berkeley RDI. "[Trustworthy Benchmarks for Contamination](https://rdi.berkeley.edu/blog/trustworth | STRING |
| MEDIUM | docs/chapter22_agentic/industrial-evaluation.md | 1194 | [^abc]: Zhu J, et al. "[Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://arxiv.org/abs/2507 | STRING |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 330 | [StepPO](https://arxiv.org/abs/2604.18401)(2026-04)指出:现有 LLM RL 算法继承了 RLHF/RLVR 的 **token-centric paradigm**——token 是建模和 | CODE |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 334 | [AT²PO](https://arxiv.org/abs/2601.04767)(2026-01)进一步把 tree search 引入 turn-level optimization。它针对多轮 agentic RL 的三个核心挑战—— | CODE |
| MEDIUM | docs/chapter22_agentic/credit-assignment.md | 706 | [^casurvey]: Zhang C. "[From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-agents.md | 759 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-agents.md | 763 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-and-trajectory.md | 1146 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/chapter22_agentic/tool-use-and-trajectory.md | 1150 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/chapter16_llm_rl_industrial/distributed-sync.md | 311 | [SGLang](https://github.com/sgl-project/sglang) 由 LMSYS 团队开发,在 agentic 场景下比 vLLM 更快: | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 153 | #### 4. Kimi K2 与 agentic intelligence 的数据和工具闭环。 | COMMENT |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 197 | #### 6. Seed Prover 1.5 与 形式化证明环境里的 agentic RL。 | COMMENT |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 225 | #### 4. DeepSeek-V3.2 与 从答案 verifier 到 agentic verifier。 | COMMENT |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 647 | #### 1. 企业小模型 与 RAG、工具、安全和低成本推理。 | COMMENT |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 807 | [^seed_prover]: [Seed Prover 1.5: Advanced Mathematical Reasoning through a Novel Agentic Architecture](https://seed.byt | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 809 | [^seed1_8]: [Official Release of Seed1.8: A Generalized Agentic Model](https://seed.bytedance.com/en/blog/official-relea | CODE |
| MEDIUM | …hapter16_llm_rl_industrial/industrial-post-training.md | 1017 | [^sfr_rl]: [Salesforce SFR-RL](https://www.salesforce.com/blog/efficient-rl-training-agentic-era/) | CODE |
| MEDIUM | docs/chapter13_imitation_meta_rl/meta-rl.md | 78 | ### 元 RL 与 few-shot 学习 | COMMENT |
| MEDIUM | …s/appendix_industrial_training/industrial-exercises.md | 279 | [^openai-agentic]: OpenAI, "Researcher, Agentic Post-Training", <https://openai.com/careers/researcher-agentic-post-trai | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/intro.md | 423 | | Modern post-training engineering | [NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL), [verl](https://github.com/verl | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/intro.md | 425 | | Open-source base models | [Qwen3.6](https://github.com/QwenLM/Qwen3.6), [Qwen3](https://github.com/QwenLM/Qwe | CODE |
| MEDIUM⚡ | docs/en/appendix_game_projects/intro.md | 427 | | Agent frameworks & tool calling | [OpenAI Agents SDK](https://github.com/openai/openai-agents-python), [Google ADK](h | CODE |
| MEDIUM | docs/en/chapter22_agentic/deep-research-agent.md | 953 | | **Agentic AI Researcher** (Owkin) | Qwen3-8B generates innovative drug target hypotheses, reward from 5-dimens | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 62 | > When the **LinkedIn team** used GPT-OSS, an open-source model with an MoE architecture, for RL training, they encounte | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 243 | > When the **LinkedIn team** used GPT-OSS for RL training, they found that the routing network, or gating network, of th | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 249 | > **Salesforce** proposed a **pipelined synchronous RL** design in its SFR-RL system. All GPUs alternate between the rol | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 291 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-practice.md | 295 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/en/chapter22_agentic/projects.md | 1331 | reward -= 0.3 * hallucination_penalty(result) # hallucination penalty | CODE |
| MEDIUM | docs/en/chapter22_agentic/projects.md | 1603 | [^tongyi_dr]: Tongyi DeepResearch Team. "Tongyi DeepResearch Technical Report." [arXiv:2510.24701](https://arxiv.org/abs | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-evaluation.md | 238 | - Zhu J, Sang H, et al. "[Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://huggingface.co/b | CODE |
| MEDIUM | docs/en/chapter22_agentic/industrial-evaluation.md | 242 | - Salesforce AI Research. "[Building Efficient RL Training for the Agentic Era](https://www.salesforce.com/blog/efficien | CODE |
| MEDIUM | docs/en/chapter22_agentic/tool-use-agents.md | 748 | - Cheng M, Ouyang J, Yu S, et al. "[Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning](https | CODE |
| 15 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | code/appendix_common_pitfalls/debug_reward_hacking.py | 527 | def print_reward_design_principles(): | CODE |
| LOW⚡ | code/chapter08_rlhf/reward_model_training.py | 36 | def generate_preference_pairs(): | CODE |
| LOW⚡ | code/chapter08_rlhf/reward_model_training.py | 394 | def visualize_reward_distributions(chosen_scores, rejected_scores, save_path="output/reward_distribution.png"): | CODE |
| LOW⚡ | code/chapter03_mdp/bellman_equation_verify.py | 80 | def manual_bellman_expectation(): | CODE |
| LOW⚡ | code/chapter03_mdp/bellman_equation_verify.py | 412 | def print_value_iteration_steps(n_steps=10): | CODE |
| LOW | code/chapter10_agentic_rl/multi_turn_rl.py | 206 | def compute_discounted_returns(rewards, gamma=0.99): | CODE |
| LOW | code/chapter10_agentic_rl/mini_deep_research_grpo.py | 83 | def search_validity_reward_func(completions, **kwargs) -> list[float]: | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 93 | def generate_correct_response(gt): | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 391 | def print_before_after_comparison(samples, history, seed=42): | CODE |
| LOW⚡ | code/chapter11_vlm_rl/vlm_grpo_train.py | 525 | def demonstrate_grpo_normalization(): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 111 | def generate_short_correct_response(gt): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 130 | def generate_partially_correct_response(gt): | CODE |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 144 | def generate_low_quality_response(gt): | CODE |
| LOW⚡ | …e/chapter05_policy_gradient/reinforce_with_baseline.py | 211 | def train_reinforce_with_baseline(num_episodes=500, gamma=0.99, lr=1e-3): | CODE |
| LOW | code/chapter12_future_trends/multi_agent_marl.py | 746 | def print_cooperation_statistics(ind_results, shared_results, n_agents): | CODE |
| LOW⚡ | code/chapter09_alignment/dpo_math_reward.py | 36 | def create_math_preference_data(): | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 49 | def maybe_make_swanlab_callback(args: argparse.Namespace): | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 211 | def ensure_discrete_action_space(env_id: str) -> None: | CODE |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 53 | def maybe_make_swanlab_callback(args: argparse.Namespace): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 88 | def generate_synthetic_rewards(group_size=8, seed=42): | CODE |
| LOW | code/chapter09_grpo_rlvr/grpo_mechanism.py | 113 | def simulate_critic_predictions(rewards, noise_scale=0.08, seed=123): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 139 | def demonstrate_grpo_step_by_step(): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 235 | def plot_reward_normalization(rewards, grpo_advantages): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_mechanism.py | 306 | def plot_advantage_comparison(grpo_advantages, ppo_advantages): | CODE |
| LOW⚡ | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 127 | def extract_answer_from_response(response): | CODE |
| LOW | docs/chapter28_vla/embodied-intelligence/index.md | 174 | def apply_domain_randomization(self): | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 306 | function updateSidebarEdgePosition() { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 512 | function handleMermaidViewerImageLoad(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 532 | function handleMermaidViewerPointerDown(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 549 | function handleMermaidViewerPointerMove(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 564 | function handleMermaidViewerKeydown(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 654 | function updateLineHeightFromRange(event) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 672 | function updateSidebarWidthFromPointer(clientX) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 726 | function scrollOutlineToActiveItem(activeLink) { | CODE |
| LOW | docs/.vitepress/theme/Layout.vue | 747 | function scrollSidebarToActiveItem(activeItem) { | CODE |
| LOW | docs/chapter23_rl_based_swe/swe-bench-and-rlvr.md | 36 | def test_empty_in_lookup_short_circuits(self): | CODE |
| LOW | docs/chapter24_deep_research/intro.md | 476 | def train_search_reward_model(preference_data, base_model): | CODE |
| LOW | docs/chapter32_selfplay/llm-multi-agent-rl/index.md | 287 | def compute_multi_agent_reward(self, trajectories): | CODE |
| LOW | docs/appendix_code_cheatsheet/attention-mha.md | 41 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/appendix_code_cheatsheet/attention-mha.md | 66 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/chapter15_rlhf/imitation-learning-pipeline.md | 363 | def construct_preference_pairs(prompt, responses, judge_outputs): | STRING |
| LOW | docs/chapter22_agentic/industrial-practice.md | 186 | def hallucination_aware_reward(answer, tool_results, citations): | CODE |
| LOW | docs/chapter22_agentic/projects.md | 1295 | def train_search_reward_model(preference_data, base_model): | CODE |
| LOW | docs/chapter22_agentic/industrial-evaluation.md | 182 | def hallucination_aware_reward(answer, tool_results, citations): | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 545 | async def rollout_single_trajectory(model, task, sandbox, max_turns=10): | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 613 | def comprehensive_agent_reward(trajectory, final_result, task): | CODE |
| LOW | docs/chapter22_agentic/build-agentic-training-system.md | 395 | def train_step_with_advantage(self, trajectories: list): | CODE |
| LOW | docs/chapter22_agentic/trajectory-synthesis.md | 282 | def trajectory_synthesis_pipeline( | CODE |
| LOW | docs/chapter22_agentic/tool-use-and-trajectory.md | 278 | def trajectory_synthesis_pipeline( | CODE |
| LOW | docs/chapter22_agentic/tool-use-and-trajectory.md | 932 | async def rollout_single_trajectory(model, task, sandbox, max_turns=10): | CODE |
| LOW | docs/chapter22_agentic/tool-use-and-trajectory.md | 1000 | def comprehensive_agent_reward(trajectory, final_result, task): | CODE |
| LOW | docs/chapter22_agentic/code/policy.py | 41 | def train_step_with_advantage(self, trajectories: list): | CODE |
| LOW⚡ | docs/chapter26_vlm/vlm-challenges.md | 46 | def setup_optimizer_with_lr_decay(model, text_lr=1e-6, vision_lr=1e-7): | CODE |
| LOW | docs/chapter13_imitation_meta_rl/meta-rl.md | 116 | def algorithm_distillation_data_generate(env, rl_algorithm, n_runs=1000, n_episodes_per_run=200): | CODE |
| LOW | docs/en/chapter32_selfplay/llm-multi-agent-rl/index.md | 291 | def compute_multi_agent_reward(self, trajectories): | CODE |
| LOW | …s/en/chapter32_selfplay/embodied-intelligence/index.md | 180 | def apply_domain_randomization(self): | CODE |
| LOW | docs/en/appendix_code_cheatsheet/attention-mha.md | 45 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/en/appendix_code_cheatsheet/attention-mha.md | 70 | def scaled_dot_product_attention(Q, K, V, mask=None): | CODE |
| LOW | docs/en/chapter15_rlhf/imitation-learning-pipeline.md | 366 | def construct_preference_pairs(prompt, responses, judge_outputs): | STRING |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 480 | def train_search_reward_model(preference_data, base_model): | CODE |
| 12 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter01_cartpole/2-pytorch_ppo.py | 443 | except Exception: | CODE |
| LOW | …chapter09_continuous_control/ppo_td3_sac_comparison.py | 96 | except Exception as e: | CODE |
| LOW | code/chapter10_agentic_rl/generate_synthetic_data.py | 74 | except Exception as e: | STRING |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 99 | except Exception as exc: # noqa: BLE001 | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 37 | except Exception: | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 56 | except Exception as exc: | CODE |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 104 | except Exception: | CODE |
| MEDIUM | code/chapter04_dqn/dqn_gym_sb3.py | 100 | def _on_training_start(self) -> None: | CODE |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 41 | except Exception: | CODE |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 60 | except Exception as exc: | CODE |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 108 | except Exception: | CODE |
| MEDIUM | code/chapter04_dqn/dqn_atari_sb3.py | 104 | def _on_training_start(self) -> None: | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 174 | except Exception: | CODE |
| LOW | docs/chapter22_agentic/tool-use-agents.md | 460 | except Exception as e: | CODE |
| LOW | docs/chapter22_agentic/rllm-deepcoder-lab.md | 1070 | except Exception: | CODE |
| LOW | docs/chapter22_agentic/build-agentic-training-system.md | 288 | except Exception as e: | CODE |
| LOW | docs/chapter22_agentic/tool-use-and-trajectory.md | 555 | except Exception: | CODE |
| LOW | docs/chapter22_agentic/tool-use-and-trajectory.md | 847 | except Exception as e: | CODE |
| LOW | docs/chapter22_agentic/code/environment.py | 41 | except Exception as e: | CODE |
| LOW | docs/chapter26_vlm/easyr1-geoqa.md | 201 | except Exception: | CODE |
| LOW | docs/chapter18_grpo/financial-tool-calling-grpo.md | 248 | except Exception: | CODE |
| LOW | docs/chapter18_grpo/financial-tool-calling-grpo.md | 345 | except Exception: | CODE |
| LOW | docs/en/chapter22_agentic/tool-use-agents.md | 174 | except Exception: | CODE |
| LOW | docs/en/chapter22_agentic/tool-use-agents.md | 460 | except Exception as e: | CODE |
| LOW | docs/en/chapter22_agentic/rllm-deepcoder-lab.md | 1071 | except Exception: | CODE |
| LOW | …/en/chapter22_agentic/build-agentic-training-system.md | 292 | except Exception as e: | CODE |
| LOW | docs/en/chapter22_agentic/tool-use-and-trajectory.md | 558 | except Exception: | CODE |
| LOW | docs/en/chapter22_agentic/tool-use-and-trajectory.md | 850 | except Exception as e: | CODE |
| LOW | docs/en/chapter26_vlm/easyr1-geoqa.md | 204 | except Exception: | CODE |
| LOW | docs/en/chapter18_grpo/financial-tool-calling-grpo.md | 245 | except Exception: | CODE |
| LOW | docs/en/chapter18_grpo/financial-tool-calling-grpo.md | 349 | except Exception: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …de/appendix_common_pitfalls/debug_training_collapse.py | 252 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/appendix_common_pitfalls/debug_reward_hacking.py | 282 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/chapter08_rlhf/rlhf_ppo_train.py | 239 | 总损失 = -L_CLIP + β * KL(π_θ || π_ref) | STRING |
| HIGH | code/chapter08_rlhf/rlhf_ppo_train.py | 275 | KL(π_θ || π_ref) = Σ π_θ * log(π_θ / π_ref) | STRING |
| HIGH | code/chapter04_dqn/double_dqn_cartpole.py | 252 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| HIGH | code/chapter04_dqn/dqn_cartpole.py | 250 | agent.buffer.push(state, action, reward, next_state, float(done)) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …de/appendix_common_pitfalls/debug_training_collapse.py | 210 | CODE | |
| LOW | code/chapter08_rlhf/rlhf_ppo_train.py | 175 | CODE | |
| LOW | code/chapter03_mdp/gridworld_q_learning.py | 283 | CODE | |
| LOW | code/chapter03_mdp/gridworld_q_learning.py | 55 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 146 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 192 | CODE | |
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 412 | CODE | |
| LOW | code/chapter01_cartpole/2-pytorch_ppo.py | 275 | CODE | |
| LOW | code/chapter01_cartpole/plot_curves.py | 28 | CODE | |
| LOW | code/chapter11_vlm_rl/geometry_counting_dataset.py | 125 | CODE | |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 262 | CODE | |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 525 | CODE | |
| LOW | code/chapter12_future_trends/multi_agent_marl.py | 123 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 110 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 154 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 194 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 376 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 57 | CODE | |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 53 | CODE | |
| LOW | code/chapter09_grpo_rlvr/grpo_mechanism.py | 139 | CODE | |
| LOW | scripts/fix-intro-h1.py | 50 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter10_agentic_rl/mini_deep_research_grpo.py | 21 | CODE | |
| LOW | code/chapter11_vlm_rl/vlm_grpo_train.py | 27 | CODE | |
| LOW | code/chapter05_policy_gradient/reinforce_cartpole.py | 25 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 23 | CODE | |
| LOW | code/chapter12_future_trends/tree_of_thought.py | 24 | CODE | |
| LOW | code/chapter17_dpo/2-test_before.py | 3 | CODE | |
| LOW | code/chapter17_dpo/4-test_after.py | 3 | CODE | |
| LOW | code/chapter09_alignment/dpo_math_reward.py | 23 | CODE | |
| LOW | code/chapter09_alignment/dpo_math_reward.py | 25 | CODE | |
| LOW | code/chapter09_alignment/dpo_hands_on.py | 17 | CODE | |
| LOW | code/chapter09_alignment/dpo_hands_on.py | 18 | CODE | |
| LOW | code/chapter04_dqn/render_atari.py | 17 | CODE | |
| LOW | code/chapter04_dqn/dqn_pokemon_red_pyboy.py | 17 | CODE | |
| LOW | code/chapter04_dqn/export_dqn_curves.py | 8 | CODE | |
| LOW | code/chapter04_dqn/dqn_gym_sb3.py | 14 | CODE | |
| LOW | code/chapter04_dqn/dqn_atari_sb3.py | 14 | CODE | |
| LOW | code/chapter09_grpo_rlvr/grpo_math_reasoning.py | 28 | CODE | |
| LOW | …s/chapter22_agentic/code/deep_research_rl_benchmark.py | 13 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …de/appendix_common_pitfalls/debug_training_collapse.py | 0 | q 网络:将状态映射到每个动作的 q 值 结构:state_dim → 128 → 128 → action_dim | STRING |
| HIGH | code/appendix_common_pitfalls/debug_reward_hacking.py | 0 | q 网络:将状态映射到每个动作的 q 值 结构:state_dim → 128 → 128 → action_dim | STRING |
| HIGH | code/chapter04_dqn/double_dqn_cartpole.py | 0 | q 网络:将状态映射到每个动作的 q 值 结构:state_dim → 128 → 128 → action_dim | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter03_mdp/bellman_equation_verify.py | 21 | COMMENT | |
| LOW | …chapter09_continuous_control/ppo_td3_sac_comparison.py | 201 | print(" 特点:异策略、随机策略、熵正则化、自动温度调节") | COMMENT |
| LOW | code/chapter09_continuous_control/sac_halfcheetah.py | 141 | # | COMMENT |
| LOW | code/chapter18_grpo/verl_code_rlvr/prepare_data.py | 1 | # prepare_data.py | COMMENT |
| LOW | …8_grpo/verl_code_rlvr/run_qwen_coder_ppo_single_gpu.sh | 1 | #!/bin/bash | COMMENT |
| LOW | code/chapter18_grpo/verl_code_rlvr/code_reward.py | 1 | # code_reward.py | COMMENT |
| LOW | docs/chapter24_deep_research/intro.md | 681 | ### 数据准备 | COMMENT |
| LOW | docs/chapter24_deep_research/intro.md | 881 | # 只有模型生成的 token 参与 loss 计算 | COMMENT |
| LOW | docs/chapter32_selfplay/rl-scaling-outlook.md | 121 | COMMENT | |
| LOW | docs/en/chapter32_selfplay/rl-scaling-outlook.md | 121 | # ========================================== | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 681 | # Should output OK, no errors | COMMENT |
| LOW | docs/en/chapter22_agentic/deep-research-agent.md | 861 | COMMENT | |
| LOW | docs/chapter10_ppo/gae-reward-model.md | 301 | ```python | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | docs/en/chapter22_agentic/deep-research-agent.md | 253 | **Problem**: Synthetic data often faces the "not hard enough" problem — research trajectories generated by GPT-4 level m | CODE |
| HIGH | docs/en/chapter22_agentic/projects.md | 1069 | **Problem**: Synthetic data often suffers from insufficient difficulty. Research trajectories generated by GPT-4-class m | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | code/chapter10_agentic_rl/generate_synthetic_data.py | 25 | api_key=os.environ.get("OPENAI_API_KEY", "your-api-key-here"), | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | package-lock.json | 5786 | "resolved": "https://registry.npmmirror.com/robust-predicates/-/robust-predicates-3.0.3.tgz", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | code/chapter09_alignment/dpo_hands_on.py | 100 | CODE |