Repository Analysis

NVIDIA/Megatron-LM

Ongoing research training transformer models at scale

13.5 Low AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of NVIDIA/Megatron-LM, a Python project with 17,655 GitHub stars. SynthScan v2.0 examined 1,086,557 lines of code across 3090 source files, recording 10824 pattern matches distributed across 25 syntactic categories. The overall adjusted score of 13.5 places this repository in the Low AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

13.5
Adjusted Score
13.5
Raw Score
100%
Time Factor
2026-08-28
Last Push
17.7K
Stars
Python
Language
1.1M
Lines of Code
3.1K
Files
10.8K
Pattern Hits
2026-08-29
Scan Date
0.07
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 30HIGH 180MEDIUM 1313LOW 9301

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 10824 distinct pattern matches across 25 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers6429 hits · 6048 pts
SeverityFileLineSnippetContext
LOWpretrain_vlm.py242def train_valid_test_datasets_provider(train_val_test_num_samples):CODE
LOWpretrain_vlm.py282def _preprocess_data_for_llava(data):CODE
LOWpretrain_vlm.py480def llava_position_embedding_ranks(pp_ranks):CODE
LOWtrain_rl.py337def train_valid_test_datasets_provider(train_val_test_num_samples):CODE
LOWpretrain_gpt.py202def _build_cached_logits_loss_func(CODE
LOWpretrain_gpt.py390def core_gpt_dataset_config_from_args(args: Any) -> GPTDatasetConfig:CODE
LOWpretrain_gpt.py458def train_valid_test_datasets_provider(train_val_test_num_samples, vp_stage=None):CODE
LOWmodel_provider.py62def count_parameters_in_layer(model, layer_name):CODE
LOWpretrain_hybrid.py203def _build_cached_logits_loss_func(CODE
LOWpretrain_hybrid.py381def core_gpt_dataset_config_from_args(args: Any) -> GPTDatasetConfig:CODE
LOWpretrain_hybrid.py425def train_valid_test_datasets_provider(train_val_test_num_samples, vp_stage=None):CODE
LOWgpt_builders.py113def _get_transformer_layer_spec(use_te, config):CODE
LOWtasks/finetune_utils.py53def _cross_entropy_forward_step(batch, model):CODE
LOWtasks/finetune_utils.py95def _build_infinite_size_dataloader(dataloader):CODE
LOWtasks/finetune_utils.py106def _build_train_valid_dataloaders(train_dataset, valid_dataset,CODE
LOWtasks/data_utils.py35def build_tokens_types_paddings_from_text(text_a, text_b,CODE
LOWtasks/data_utils.py49def build_tokens_types_paddings_from_ids(text_a_ids, text_b_ids, max_seq_length,CODE
LOWtasks/eval_utils.py65def calculate_correct_answers(name, model, dataloader,CODE
LOWtasks/eval_utils.py111 def correct_answers_forward_step(batch, model):CODE
LOWtools/run_vlm_text_generation.py161def generate_and_write_samples(model):CODE
LOWtools/prepare_cache.py54def _normalize_prepare_cache_args(args: Any) -> None:CODE
LOWtools/prepare_cache.py65def _validate_prepare_cache_args(args: Any) -> None:CODE
LOWtools/prepare_cache.py82def _disable_cache_load_only_flags(args: Any) -> Dict[str, bool]:CODE
LOWtools/prepare_cache.py102def _print_effective_configuration(CODE
LOWtools/prepare_cache.py120def core_gpt_dataset_config_from_args(args: Any) -> GPTDatasetConfig:CODE
LOWtools/run_inference_performance_test.py49def add_inference_benchmarking_args(parser):CODE
LOWtools/build_sequences_per_dataset.py60def build_sequences_per_dataset(args):CODE
LOWtools/run_dynamic_text_generation_server.py56def add_text_generation_server_args(parser: argparse.ArgumentParser):CODE
LOWtools/run_dynamic_text_generation_server.py77def _build_engine_for_vlm_or_gpt(is_vlm: bool) -> DynamicInferenceEngine:CODE
LOWtools/run_dynamic_text_generation_server.py172async def run_text_generation_server(CODE
LOWtools/checkpoint/saver_hf_llava.py176 def receive_vision_projection(self):CODE
LOWtools/checkpoint/saver_hf_llava.py352 def save_state_dict_to_hf_checkpoint(self):CODE
LOWtools/checkpoint/schema_core.py10def get_core_transformer_block_key(model_key):CODE
LOWtools/checkpoint/schema_hf.py173def get_language_model_schema(CODE
LOWtools/checkpoint/loader_base.py30 def _maybe_parse_additional_megatron_args(self, margs, checkpoint_args):CODE
LOWtools/checkpoint/loader_base.py90 def _maybe_ensure_additional_required_arguments(self):CODE
LOWtools/checkpoint/loader_base.py107 def ensure_required_arguments(self):CODE
LOWtools/checkpoint/loader_base.py193 def get_models_for_pipeline_stage(count, dtype):CODE
LOWtools/checkpoint/loader_base.py425 def build_checkpoint_metadata(self, true_vocab_size):CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py404def initialize_ssm_layer_params(CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py152def parse_hybrid_layer_pattern(pattern):CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py183def build_layer_index_mapping(layer_types, direction):CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py316def validate_pattern_gpt_compatible(layer_types, direction):CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py361def validate_source_args_gpt_compatible(source_args, direction):CODE
LOWtools/checkpoint/dist_checkpoint_io.py69def resolve_checkpoint_subdir(load_dir):CODE
LOWtools/checkpoint/dist_checkpoint_io.py113def ensure_single_rank_process_group():CODE
LOWtools/checkpoint/dist_checkpoint_io.py159def load_dist_checkpoint_full(load_dir):CODE
LOWtools/checkpoint/dist_checkpoint_io.py221def save_dist_checkpoint_full(CODE
LOWtools/checkpoint/dist_checkpoint_io.py260def write_latest_iteration_marker(save_dir, iteration):CODE
LOWtools/checkpoint/loader_mixtral_hf.py28def load_args_from_checkpoint(args):CODE
LOWtools/checkpoint/loader_mixtral_hf.py63def verify_transformers_version():CODE
LOWtools/checkpoint/saver_llava.py134 def _maybe_parse_additional_megatron_args(self, margs):CODE
LOWtools/checkpoint/saver_llava.py320 def receive_vision_projection(self):CODE
LOWtools/checkpoint/checkpoint_inspector.py288def save_checkpoint_with_pickle_protocol(state_dict, output_dir, pickle_protocol=4):CODE
LOWtools/checkpoint/checkpoint_inspector.py292 def transform_object_override(write_item, obj):CODE
LOWtools/checkpoint/checkpoint_inspector.py831def convert_torch_dist_to_fsdp_dtensor(CODE
LOWtools/checkpoint/saver_base.py31 def _maybe_parse_additional_megatron_args(self, margs):CODE
LOWtools/checkpoint/saver_base.py38 def insert_megatron_path_and_check_te(self):CODE
LOWtools/checkpoint/saver_base.py277 def receive_checkpoint_metadata(self):CODE
LOWtools/checkpoint/saver_base.py341 def save_local_models_to_checkpoint(self):CODE
6369 more matches not shown…
Decorative Section Separators1087 hits · 3832 pts
SeverityFileLineSnippetContext
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py400# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py402# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py132# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py134# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py225# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py227# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py486# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py488# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py530# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py532# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py627# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py629# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py698# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py700# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py722# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/gpt_hybrid_conversion.py724# ---------------------------------------------------------------------------COMMENT
MEDIUMtools/checkpoint/saver_llava.py225 #-----------COMMENT
MEDIUMtools/checkpoint/saver_llava.py186 #-----------COMMENT
MEDIUMtools/checkpoint/saver_base.py377 #-----------COMMENT
MEDIUMtools/checkpoint/saver_base.py420 # --------------COMMENT
MEDIUMtools/checkpoint/saver_base.py432 # ------------------COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py360 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py362 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py386 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py388 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py410 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py412 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py433 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py435 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py474 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py476 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py496 # ================================================================COMMENT
MEDIUM…s/common_pile_dataset/create_common_pile_ci_dataset.py498 # ================================================================COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh24# ── Parse KEY=VALUE positional args ───────────────────────────────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh75# ── Read model_config.yaml ────────────────────────────────────────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh116# ── Build model args (substituting ${CHECKPOINT_LOAD_PATH}) ───────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh139# ── Make image-bundled extras (mamba-ssm) visible to the cog venv ─────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh179# ── Launch the inference server in the background ─────────────────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh259# ── Wait for server readiness ─────────────────────────────────────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh279# ── Benchmark sweep ───────────────────────────────────────────────────────────COMMENT
MEDIUM…ts/performance_tests/shell_test_utils/run_perf_test.sh299# ── Baseline comparison or recording ──────────────────────────────────────────COMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py313 # ------------------------------------------------------------------COMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py315 # ------------------------------------------------------------------COMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py439 # ------------------------------------------------------------------COMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py441 # ------------------------------------------------------------------COMMENT
MEDIUMtests/unit_tests/test_optimizer_param_scheduler.py302# ── get_canonical_lr_for_logging tests ──────────────────────────────────────COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py66# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py68# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1099# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1101# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1546# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1548# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1785# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1787# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1956# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py1958# ===========================================================================COMMENT
MEDIUMtests/unit_tests/test_sequence_packing.py248 # =====================================================================COMMENT
MEDIUMtests/unit_tests/test_sequence_packing.py250 # =====================================================================COMMENT
MEDIUMtests/unit_tests/test_sequence_packing.py273 # =====================================================================COMMENT
MEDIUMtests/unit_tests/test_sequence_packing.py275 # =====================================================================COMMENT
1027 more matches not shown…
Unused Imports1022 hits · 954 pts
SeverityFileLineSnippetContext
LOWtrain_rl.py4CODE
LOWmamba_builders.py14CODE
LOWmamba_builders.py15CODE
LOWpretrain_gpt.py39CODE
LOWpretrain_gpt.py80CODE
LOWpretrain_hybrid.py38CODE
LOWpretrain_hybrid.py80CODE
LOWgpt_builders.py20CODE
LOWtools/merge_datasets.py3CODE
LOWtools/preprocess_data_nmt.py6CODE
LOWtools/run_inference_performance_test.py37CODE
LOWtools/run_inference_performance_test.py41CODE
LOWtools/run_inference_performance_test.py43CODE
LOWtools/run_text_generation_server.py6CODE
LOWtools/linter.py11CODE
LOWtools/preprocess_mmdata.py12CODE
LOWtools/checkpoint/saver_core.py3CODE
LOWtools/checkpoint/saver_core.py5CODE
LOWtools/checkpoint/saver_core.py7CODE
LOWtools/checkpoint/saver_core.py8CODE
LOWtools/checkpoint/remap_gpt_dsa_to_mamba.py38CODE
LOWtools/checkpoint/remap_gpt_dsa_to_mamba.py39CODE
LOWtools/checkpoint/remap_gpt_dsa_to_mamba.py108CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py122CODE
LOWtools/checkpoint/dist_checkpoint_io.py20CODE
LOWtools/checkpoint/dist_checkpoint_io.py31CODE
LOWtools/checkpoint/utils.py3CODE
LOWtools/checkpoint/hybrid_conversion.py10CODE
LOWtools/checkpoint/loader_mixtral_hf.py3CODE
LOWtools/checkpoint/loader_mixtral_hf.py167CODE
LOWtools/checkpoint/saver_llava.py2CODE
LOWtools/checkpoint/saver_llava.py4CODE
LOWtools/checkpoint/saver_llava.py173CODE
LOWtools/checkpoint/saver_llava.py173CODE
LOWtools/checkpoint/checkpoint_inspector.py37CODE
LOWtools/checkpoint/loader_core.py3CODE
LOWtools/checkpoint/loader_core.py4CODE
LOWtools/checkpoint/loader_core.py5CODE
LOWtools/checkpoint/loader_core.py6CODE
LOWtools/checkpoint/loader_core.py7CODE
LOWtools/checkpoint/saver_base.py2CODE
LOWtools/checkpoint/loader_llava.py3CODE
LOWtools/checkpoint/loader_llava.py7CODE
LOWtools/bert_embedding/embed.py3CODE
LOWtools/bert_embedding/embed.py8CODE
LOWtools/bert_embedding/embed.py15CODE
LOWtools/bert_embedding/embed.py16CODE
LOWtools/bert_embedding/embed.py19CODE
LOWtools/bert_embedding/embed.py21CODE
LOWtools/bert_embedding/embed.py21CODE
LOWtools/bert_embedding/__init__.py3CODE
LOWtools/bert_embedding/__init__.py3CODE
LOW…formance_tests/shell_test_utils/compare_to_baseline.py23CODE
LOWtests/unit_tests/test_fp8_param.py3CODE
LOWtests/unit_tests/test_fp8_param.py29CODE
LOWtests/unit_tests/test_fp8_param.py44CODE
LOWtests/unit_tests/test_inference.py3CODE
LOWtests/unit_tests/test_basic.py2CODE
LOWtests/unit_tests/test_emerging_optimizers.py11CODE
LOWtests/unit_tests/test_imports.py14CODE
962 more matches not shown…
Cross-File Repetition123 hits · 615 pts
SeverityFileLineSnippetContext
HIGHtrain_rl.py0forward training step. args: data_iterator : input data iterator model (hybridmodel): the gpt modelSTRING
HIGHpretrain_hybrid.py0forward training step. args: data_iterator : input data iterator model (hybridmodel): the gpt modelSTRING
HIGHmegatron/elastification/pretrain_hybrid_flex.py0forward training step. args: data_iterator : input data iterator model (hybridmodel): the gpt modelSTRING
HIGHpretrain_gpt.py0loss function. args: loss_mask (torch.tensor): used to mask out some portions of the loss output_tensor (torch.tensor): STRING
HIGHpretrain_hybrid.py0loss function. args: loss_mask (torch.tensor): used to mask out some portions of the loss output_tensor (torch.tensor): STRING
HIGHmegatron/elastification/pretrain_hybrid_flex.py0loss function. args: loss_mask (torch.tensor): used to mask out some portions of the loss output_tensor (torch.tensor): STRING
HIGHpretrain_gpt.py0build the train test and validation datasets. args: train_val_test_num_samples : a list containing the number of samplesSTRING
HIGHpretrain_hybrid.py0build the train test and validation datasets. args: train_val_test_num_samples : a list containing the number of samplesSTRING
HIGHexamples/post_training/modelopt/finetune.py0build the train test and validation datasets. args: train_val_test_num_samples : a list containing the number of samplesSTRING
HIGHexamples/t5/pretrain_t5.py0build the train test and validation datasets. args: train_val_test_num_samples : a list containing the number of samplesSTRING
HIGHmegatron/elastification/pretrain_hybrid_flex.py0build the train test and validation datasets. args: train_val_test_num_samples : a list containing the number of samplesSTRING
HIGHtools/run_text_generation_server.py0utility to get the relevant backend for running inference this function will automatically chose the trtllmbackend when STRING
HIGHexamples/inference/advanced/gpt_static_inference.py0utility to get the relevant backend for running inference this function will automatically chose the trtllmbackend when STRING
HIGH…amples/inference/advanced/simple_t5_batch_inference.py0utility to get the relevant backend for running inference this function will automatically chose the trtllmbackend when STRING
HIGHtools/checkpoint/saver_hf_llava.py0required top-level function that creates the saver and calls its .save().STRING
HIGHtools/checkpoint/saver_core.py0required top-level function that creates the saver and calls its .save().STRING
HIGHtools/checkpoint/saver_llava.py0required top-level function that creates the saver and calls its .save().STRING
HIGHtools/checkpoint/loader_base.py0orchestrates loading a megatron checkpoint and sending model parameters over a given multiprocessing queue. args: args: STRING
HIGHtools/checkpoint/loader_core.py0orchestrates loading a megatron checkpoint and sending model parameters over a given multiprocessing queue. args: args: STRING
HIGHtools/checkpoint/loader_llava.py0orchestrates loading a megatron checkpoint and sending model parameters over a given multiprocessing queue. args: args: STRING
HIGHtools/checkpoint/loader_base.py0parse megatron arguments by forcibly overwriting sys.argv. populates self.margs and self.checkpoint_args.STRING
HIGHtools/checkpoint/saver_base.py0parse megatron arguments by forcibly overwriting sys.argv. populates self.margs and self.checkpoint_args.STRING
HIGHtools/checkpoint/loader_llava.py0parse megatron arguments by forcibly overwriting sys.argv. populates self.margs and self.checkpoint_args.STRING
HIGHtools/checkpoint/loader_base.py0construct a sys.argv list for megatron's argument parser. this centralizes the hack of overwriting sys.argv.STRING
HIGHtools/checkpoint/loader_core.py0construct a sys.argv list for megatron's argument parser. this centralizes the hack of overwriting sys.argv.STRING
HIGHtools/checkpoint/saver_base.py0construct a sys.argv list for megatron's argument parser. this centralizes the hack of overwriting sys.argv.STRING
HIGHtools/checkpoint/loader_llava.py0construct a sys.argv list for megatron's argument parser. this centralizes the hack of overwriting sys.argv.STRING
HIGHtests/unit_tests/test_hyper_comm_grid.py0set up distributed environment for the entire test class.STRING
HIGH…it_tests/pipeline_parallel/test_bridge_communicator.py0set up distributed environment for the entire test class.STRING
HIGH…sts/pipeline_parallel/test_multimodule_communicator.py0set up distributed environment for the entire test class.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_overlap.py0model with direct parameters and multiple child fsdpmodules.STRING
HIGH…ts/unit_tests/distributed/mfsdp_v2/test_fully_shard.py0model with direct parameters and multiple child fsdpmodules.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_context.py0model with direct parameters and multiple child fsdpmodules.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_memory.py0model with direct parameters and multiple child fsdpmodules.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_overlap.py0run through every child layer with a root-owned bias.STRING
HIGH…ts/unit_tests/distributed/mfsdp_v2/test_fully_shard.py0run through every child layer with a root-owned bias.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_context.py0run through every child layer with a root-owned bias.STRING
HIGHtests/unit_tests/distributed/mfsdp_v2/test_memory.py0run through every child layer with a root-owned bias.STRING
HIGH…sts/unit_tests/distributed/mfsdp_v2/test_cuda_graph.py0run through every child layer with a root-owned bias.STRING
HIGH…it_tests/pipeline_parallel/test_bridge_communicator.py0destroy all tracked grids and bridge communicator pgs.STRING
HIGH…_tests/pipeline_parallel/test_multimodule_schedules.py0destroy all tracked grids and bridge communicator pgs.STRING
HIGHtests/unit_tests/models/mimo/test_mimo_1f1b_schedule.py0destroy all tracked grids and bridge communicator pgs.STRING
HIGHtests/unit_tests/models/mimo/test_mimo_1f1b_schedule.py0return a transformerconfig for the vision projection mlp.STRING
HIGHexamples/mimo/configs/llava_vlm.py0return a transformerconfig for the vision projection mlp.STRING
HIGHexamples/mimo/configs/llava_avlm.py0return a transformerconfig for the vision projection mlp.STRING
HIGH…unit_tests/dist_checkpointing/models/test_gpt_model.py0test model loading with different vocab size (caused by tp padding).STRING
HIGH…nit_tests/dist_checkpointing/models/test_bert_model.py0test model loading with different vocab size (caused by tp padding).STRING
HIGHtests/unit_tests/dist_checkpointing/models/common.py0test model loading with different vocab size (caused by tp padding).STRING
HIGH…ultimodal/evaluation/evaluate_video_phys_game_bench.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_realworldqa.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_ai2d.py0merge input files to a format compatible with the evaluator.STRING
HIGH…es/multimodal/evaluation/evaluate_video_motionbench.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_spdocvqa.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_textvqa.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_chartqa.py0merge input files to a format compatible with the evaluator.STRING
HIGH…amples/multimodal/evaluation/evaluate_video_mvbench.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_mathvista.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_vqav2.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_ocrbench.py0merge input files to a format compatible with the evaluator.STRING
HIGHexamples/multimodal/evaluation/evaluate_infovqa.py0merge input files to a format compatible with the evaluator.STRING
63 more matches not shown…
Deep Nesting664 hits · 550 pts
SeverityFileLineSnippetContext
LOWgpt_builders.py25CODE
LOWtasks/finetune_utils.py147CODE
LOWtasks/eval_utils.py65CODE
LOW.gitlab/scripts/check_imports.py66CODE
LOW.gitlab/scripts/check_imports.py123CODE
LOWtools/preprocess_data.py310CODE
LOWtools/run_vlm_text_generation.py88CODE
LOWtools/checkpoint/saver_hf_llava.py69CODE
LOWtools/checkpoint/loader_base.py254CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py631CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py726CODE
LOWtools/checkpoint/dist_checkpoint_io.py131CODE
LOWtools/checkpoint/hybrid_conversion.py59CODE
LOWtools/checkpoint/hybrid_conversion.py114CODE
LOWtools/checkpoint/hybrid_conversion.py170CODE
LOWtools/checkpoint/hybrid_conversion.py222CODE
LOWtools/checkpoint/saver_llava.py183CODE
LOWtools/checkpoint/checkpoint_inspector.py74CODE
LOWtools/checkpoint/checkpoint_inspector.py335CODE
LOWtools/checkpoint/checkpoint_inspector.py951CODE
LOWtools/checkpoint/saver_base.py365CODE
LOWtools/checkpoint/loader_llava.py161CODE
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py63CODE
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py109CODE
LOWtools/moe_routing/analyze_routing_predictability.py54CODE
LOWtools/moe_routing/analyze_routing_concentration.py70CODE
LOW…formance_tests/shell_test_utils/compare_to_baseline.py36CODE
LOW…shell_test_utils/determinism/print_nsys_leaderboard.py85CODE
LOWtests/unit_tests/test_fp8_param.py219CODE
LOWtests/unit_tests/test_optimizer.py900CODE
LOWtests/unit_tests/test_utils.py366CODE
LOWtests/unit_tests/test_imports.py63CODE
LOWtests/unit_tests/test_layer_wise_optimizer.py321CODE
LOWtests/unit_tests/test_layer_wise_optimizer.py874CODE
LOWtests/unit_tests/test_sequence_packing.py177CODE
LOWtests/unit_tests/test_sequence_packing.py348CODE
LOWtests/unit_tests/ssm/test_gdp_dynamic_inference.py727CODE
LOWtests/unit_tests/ssm/ops/mamba2/test_ssd_bmm.py68CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py296CODE
LOW…t_tests/tools/checkpoint/test_gpt_hybrid_conversion.py461CODE
LOWtests/unit_tests/extension/test_kitchen_sdpa.py60CODE
LOWtests/unit_tests/determinism/utils.py119CODE
LOW…istributed/mfsdp_v1/test_mfsdp_fused_mla_checkpoint.py234CODE
LOW…ted/mfsdp_v1/test_mcore_fully_sharded_data_parallel.py1050CODE
LOW…ted/mfsdp_v1/test_mcore_fully_sharded_data_parallel.py1125CODE
LOW…ted/mfsdp_v1/test_mcore_fully_sharded_data_parallel.py1128CODE
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py194CODE
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py1012CODE
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py1182CODE
LOW…/unit_tests/distributed/mfsdp_v2/test_mcore_adapter.py349CODE
LOWtests/unit_tests/transformer/test_cuda_graphs.py772CODE
LOWtests/unit_tests/transformer/test_cuda_graphs.py964CODE
LOWtests/unit_tests/transformer/test_cuda_graphs.py1647CODE
LOWtests/unit_tests/transformer/test_transformer_block.py502CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py2166CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py3294CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py3328CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py3414CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py3542CODE
LOW…mental_attention_variant/test_attention_variant_dsa.py3579CODE
604 more matches not shown…
Hallucination Indicators30 hits · 415 pts
SeverityFileLineSnippetContext
CRITICALtools/checkpoint/loader_mixtral_hf.py69 model.embedding.word_embeddings.weight.data.copy_(CODE
CRITICALtools/checkpoint/loader_mixtral_hf.py74 model.decoder.final_layernorm.weight.data.copy_(hf_model.model.norm.weight)CODE
CRITICALtools/checkpoint/loader_mixtral_hf.py103 layer.mlp.router.weight.data.copy_(hf_layer.block_sparse_moe.gate.weight)CODE
CRITICALtools/checkpoint/loader_mixtral_hf.py127 layer.self_attention.linear_qkv.layer_norm_weight.data.copy_(hf_layer.input_layernorm.weight)CODE
CRITICALtools/checkpoint/saver_llava.py199 model.vision_model.conv1.weight.data.copy_(vit_embeddings_msg["conv1 weight"])CODE
CRITICALtools/checkpoint/saver_llava.py201 model.vision_model.conv1.bias.data.copy_(vit_embeddings_msg["conv1 bias"])CODE
CRITICALtools/checkpoint/saver_llava.py202 model.vision_model.position_embeddings.weight.data.copy_(vit_embeddings_msg["position embeddings"])CODE
CRITICALtools/checkpoint/saver_llava.py208 model.vision_model.embedder.weight.data.copy_(embedder_weight[tp_rank])CODE
CRITICALtools/checkpoint/saver_llava.py210 model.vision_model.embedder.bias.data.copy_(embedder_bias[tp_rank])CODE
CRITICALtools/checkpoint/saver_llava.py214 model.vision_model.ln_pre.weight.data.copy_(vit_embeddings_msg["ln pre weight"])CODE
CRITICALtools/checkpoint/saver_llava.py215 model.vision_model.ln_pre.bias.data.copy_(vit_embeddings_msg["ln pre bias"])CODE
CRITICALtools/checkpoint/saver_llava.py218 model.vision_model.ln_post.weight.data.copy_(vit_embeddings_msg["ln post weight"])CODE
CRITICALtools/checkpoint/saver_llava.py219 model.vision_model.ln_post.bias.data.copy_(vit_embeddings_msg["ln post bias"])CODE
CRITICALtools/checkpoint/saver_llava.py343 model.vision_projection.encoder.linear_fc1.weight.data.copy_(CODE
CRITICALtools/checkpoint/saver_llava.py345 model.vision_projection.encoder.linear_fc2.weight.data.copy_(CODE
CRITICALtools/checkpoint/saver_llava.py348 model.vision_projection.encoder.linear_fc1.layer_norm_weight.data.copy_(CODE
CRITICALtools/checkpoint/saver_llava.py351 model.vision_projection.encoder.linear_fc1.layer_norm_bias.data.copy_(CODE
CRITICALtools/checkpoint/saver_llava.py354 model.vision_projection.encoder.linear_fc1.bias.data.copy_(CODE
CRITICALtools/checkpoint/saver_llava.py356 model.vision_projection.encoder.linear_fc2.bias.data.copy_(vision_projection_l1_bias)CODE
CRITICALtests/unit_tests/transformer/moe/test_routers.py135 assert self.sequential_mlp.router.weight.grad.abs().sum() == 0CODE
CRITICALtests/unit_tests/transformer/moe/test_routers.py141 assert self.sequential_mlp.router.weight.grad.abs().sum() > 0CODE
CRITICALtests/unit_tests/transformer/moe/test_routers.py146 self.sequential_mlp.router.weight.grad.fill_(0)CODE
CRITICALtests/unit_tests/transformer/moe/test_routers.py149 assert self.sequential_mlp.router.weight.grad.abs().sum() > 0CODE
CRITICALtests/unit_tests/models/test_gpt_model.py99 assert self.gpt_model.embedding.word_embeddings.weight.std().cpu().item() == approx(CODE
CRITICALtests/unit_tests/models/test_gpt_model.py102 assert self.gpt_model.embedding.word_embeddings.weight.mean().cpu().item() == approx(CODE
CRITICALmegatron/core/parallel_state.py2596 and torch.distributed.distributed_c10d._world.pg_map.get(_DATA_PARALLEL_GROUP_GLOO, None)CODE
CRITICALmegatron/core/parallel_state.py2605 and torch.distributed.distributed_c10d._world.pg_map.get(CODE
CRITICALmegatron/core/parallel_state.py2659 and torch.distributed.distributed_c10d._world.pg_map.get(CODE
CRITICALmegatron/core/parallel_state.py2673 and torch.distributed.distributed_c10d._world.pg_map.get(CODE
CRITICALmegatron/core/transformer/attention.py2042 self.linear_qkv.weight.main_param.data.copy_(CODE
Self-Referential Comments137 hits · 402 pts
SeverityFileLineSnippetContext
MEDIUMgpt_builders.py42 # Define the decoder block specCOMMENT
MEDIUMgpt_builders.py54 # Define the decoder layer specCOMMENT
MEDIUMgpt_builders.py67 # Define the decoder block specCOMMENT
MEDIUM.gitlab/stages/05.publish.yml149 # Define the full refspec for the branchCOMMENT
MEDIUMtools/preprocess_data_fast.py54 # Create an IndexedDatasetBuilderCOMMENT
MEDIUMtools/run_inference_performance_test.py113 # Create a list of valid token IDsCOMMENT
MEDIUMtools/common_pile_dataset/setup_common_pile_dataset.sh87# Create a virtual environment to avoid system package conflictsCOMMENT
MEDIUMtools/common_pile_dataset/setup_common_pile_dataset.sh122# Create the output directoryCOMMENT
MEDIUMtests/unit_tests/test_optimizer.py1116 # Create a new state_dict with all params set to 3.COMMENT
MEDIUMtests/unit_tests/test_optimizer.py1258 # Create a simple model for testingCOMMENT
MEDIUMtests/unit_tests/test_optimizer.py1340 # Create a simple modelCOMMENT
MEDIUMtests/unit_tests/test_utils.py509 # Create a straggler_detector with enabled set to false.COMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py637 # Create a process groupCOMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py640 # Create a tensor for communication testCOMMENT
MEDIUMtests/unit_tests/test_hyper_comm_grid.py741 # Create a unique tensor based on rankCOMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py328 # Create a simple linear model for testingCOMMENT
MEDIUMtests/unit_tests/test_emerging_optimizers.py898 # Create a model with QKV-like parameterCOMMENT
MEDIUMtests/unit_tests/test_training.py189 # Create a mock state_dict with gradients (use deterministic values for reproducibility).COMMENT
MEDIUMtests/unit_tests/ssm/ops/mamba2/test_ssm_kernel.py84 # Create the Mixer instance directlyCOMMENT
MEDIUMtests/unit_tests/fusions/test_torch_softmax.py197 # Create a padding maskCOMMENT
MEDIUM…/unit_tests/post_training/test_modelopt_module_spec.py278 # Define the expected signatureCOMMENT
MEDIUM…/unit_tests/post_training/test_modelopt_module_spec.py317 # Define the expected signatureCOMMENT
MEDIUM…tests/distributed/test_torch_fully_sharded_parallel.py78 # Create a dummy model and configs.COMMENT
MEDIUM…tests/distributed/test_torch_fully_sharded_parallel.py85 # Create the sharded model.COMMENT
MEDIUM…tests/distributed/test_torch_fully_sharded_parallel.py106 # Create a dummy model and configs.COMMENT
MEDIUM…tests/distributed/test_torch_fully_sharded_parallel.py112 # Create a custom process group (using the default world for testing)COMMENT
MEDIUM…tests/distributed/test_torch_fully_sharded_parallel.py115 # Create the sharded model with explicit process groupCOMMENT
MEDIUM…ests/distributed/mfsdp_v1/test_mfsdp_uneven_dtensor.py600 # Create a manual uneven sharding along dim 1 with possible zero-length local on some ranksCOMMENT
MEDIUM…tests/transformer/test_transformer_block_custom_pgs.py259 # Create a transformer block with default process groupsCOMMENT
MEDIUM…tests/transformer/test_transformer_block_custom_pgs.py285 # Create a transformer block with custom process groupsCOMMENT
MEDIUM…tests/transformer/test_transformer_block_custom_pgs.py737 # Create a single transformer blockCOMMENT
MEDIUMtests/unit_tests/transformer/test_cuda_graphs.py1043 # Create the CUDA graphs - this is where the is_last_layer logic is testedCOMMENT
MEDIUMtests/unit_tests/transformer/test_cuda_graphs.py1294 # Create a mapping of sample_keys to indicesCOMMENT
MEDIUMtests/unit_tests/transformer/test_transformer_block.py642 # Create a new build_layers method that uses interleaved attentionCOMMENT
MEDIUM…ts/unit_tests/transformer/moe/test_token_dispatcher.py261 # Create the answer.COMMENT
MEDIUMtests/unit_tests/transformer/moe/test_shared_experts.py435 # Create a dummy input tensor.COMMENT
MEDIUMtests/unit_tests/transformer/moe/test_aux_loss.py267 # Create a new config with updated parametersCOMMENT
MEDIUMtests/unit_tests/transformer/moe/test_aux_loss.py270 # Create the router with the updated configCOMMENT
MEDIUM…s/unit_tests/training/config/test_instantiate_utils.py334 # Create a mock that raises an error when used with functools.partialCOMMENT
MEDIUMtests/unit_tests/training/config/test_yaml_utils.py215 # Create a mock torch dtypeCOMMENT
MEDIUMtests/unit_tests/training/config/test_yaml_utils.py262 # Create a mock GenerationConfigCOMMENT
MEDIUMtests/unit_tests/utils/test_experimental_log_once.py23 # Define a fresh function with the decorator so it has its own closure state.COMMENT
MEDIUMtests/unit_tests/models/test_gpt_model.py261 # Define the expected signatureCOMMENT
MEDIUMtests/unit_tests/models/test_fastconformer_model.py45 # Create a parameter with the target dtype so ``next(self.parameters()).dtype``COMMENT
MEDIUM…s/unit_tests/models/mimo/test_mimo_audio_submodules.py212 # Create a time arrayCOMMENT
MEDIUM…s/unit_tests/models/mimo/test_mimo_audio_submodules.py215 # Create a simple sine wave at 440 Hz (A4)COMMENT
MEDIUM…nit_tests/models/mimo/test_mimo_embedding_alignment.py22 # Create a minimal MimoModelConfigCOMMENT
MEDIUM…nit_tests/models/mimo/test_mimo_embedding_alignment.py73 # Create a simple batchCOMMENT
MEDIUM…nit_tests/models/mimo/test_mimo_embedding_alignment.py309 # Create a test case with 2 batches:COMMENT
MEDIUM…nit_tests/models/mimo/test_mimo_embedding_alignment.py345 # Create the unflattened embeddings that would come from a vision encoderCOMMENT
MEDIUMtests/unit_tests/models/mimo/test_mimo_submodules.py92 # Create the main module specCOMMENT
MEDIUMtests/unit_tests/models/mimo/test_mimo_submodules.py351 # Create a data batch without imagesCOMMENT
MEDIUM…tests/dist_checkpointing/test_save_tokenizer_assets.py69 # Create a fake tokenizer model fileCOMMENT
MEDIUM…tests/dist_checkpointing/test_save_tokenizer_assets.py263 # Create a fake tokenizer model fileCOMMENT
MEDIUM…tests/dist_checkpointing/test_save_tokenizer_assets.py329 # Create a fake tokenizer fileCOMMENT
MEDIUMtests/unit_tests/dist_checkpointing/test_fp8.py33 # Create a quantizer for FP8 conversionCOMMENT
MEDIUM…s/unit_tests/dist_checkpointing/test_fully_parallel.py596 # Create a mock that will do what it's supposed to do,COMMENT
MEDIUM…/unit_tests/inference/contexts/test_dynamic_context.py476 # Initialize all variablesCOMMENT
MEDIUM…/unit_tests/inference/contexts/test_dynamic_context.py1473 # Create an active_requests_mask where requests 0, 2, and 4 are finished (0),COMMENT
MEDIUM…/unit_tests/inference/contexts/test_dynamic_context.py1563 # Create an active_requests_mask where all requests are finishedCOMMENT
77 more matches not shown…
Excessive Try-Catch Wrapping281 hits · 335 pts
SeverityFileLineSnippetContext
LOWtasks/finetune_utils.py61 except Exception:CODE
LOWtasks/eval_utils.py114 except Exception:CODE
MEDIUMtasks/eval_utils.py111def correct_answers_forward_step(batch, model):CODE
LOW.gitlab/scripts/check_imports.py117 except Exception:CODE
LOWtools/check_copyright.py22 except Exception as e:CODE
MEDIUMtools/check_copyright.py23 print(f"Error reading {file_path}: {e}")CODE
MEDIUMtools/text_generation_cli.py21 print(f"Error {response.status_code}: {response.json()['message']}")CODE
LOWtools/checkpoint/saver_hf_llava.py400 except Exception as e:CODE
LOWtools/checkpoint/loader_base.py78 except Exception as e:CODE
MEDIUMtools/checkpoint/loader_base.py79 print(f"Error validating Megatron arguments: {e}")CODE
LOWtools/checkpoint/saver_core.py70 except Exception as e:CODE
LOWtools/checkpoint/gpt_hybrid_conversion.py384 except Exception:CODE
LOWtools/checkpoint/dist_checkpoint_io.py215 except Exception:CODE
LOWtools/checkpoint/hybrid_conversion.py322 except Exception:CODE
LOWtools/checkpoint/loader_mixtral_hf.py341 except Exception:CODE
MEDIUMtools/checkpoint/loader_mixtral_hf.py338def load_checkpoint(queue, args):CODE
LOWtools/checkpoint/saver_llava.py404 except Exception as e:CODE
LOWtools/checkpoint/loader_core.py90 except Exception as e:CODE
LOWtools/checkpoint/loader_llava.py360 except Exception as e:CODE
LOWtools/bert_embedding/embed.py217 except Exception:CODE
LOWtools/bert_embedding/embed.py223 except Exception:CODE
LOWtests/unit_tests/conftest.py66 except Exception:CODE
LOWtests/unit_tests/conftest.py114 except Exception as e:CODE
LOWtests/unit_tests/test_hyper_comm_grid.py531 except Exception as e:CODE
LOWtests/unit_tests/test_api_backwards_compat_setup.py106 except Exception as e:CODE
LOWtests/unit_tests/test_api_backwards_compat_setup.py134 except Exception as e:STRING
LOWtests/unit_tests/test_imports.py58 except Exception:CODE
LOWtests/unit_tests/test_imports.py77 except Exception:CODE
LOWtests/unit_tests/test_utilities.py118 except Exception:CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py806except Exception:CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py21except Exception:CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py158 except Exception:CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py184 except Exception:CODE
LOWtests/unit_tests/tokenizers/test_tokenizer.py325 except Exception:CODE
LOWtests/unit_tests/fusions/test_mla_yarn_rope_apply.py24except Exception:CODE
LOW…istributed/mfsdp_v1/test_mfsdp_fused_mla_checkpoint.py353 except Exception:CODE
LOW…ted/mfsdp_v1/test_mcore_fully_sharded_data_parallel.py1142 except Exception:CODE
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py74 except Exception as e:CODE
LOW…sformer/moe/test_moe_single_grouped_weight_numerics.py583 except Exception:CODE
LOW…sformer/moe/test_moe_single_grouped_weight_numerics.py616 except Exception:CODE
LOW…sformer/moe/test_moe_single_grouped_weight_numerics.py643 except Exception:CODE
LOW…sformer/moe/test_moe_single_grouped_weight_numerics.py690 except Exception:CODE
MEDIUM…sformer/moe/test_moe_single_grouped_weight_numerics.py108def teardown_method(self, method):CODE
LOWtests/unit_tests/transformer/moe/test_aux_loss.py38except Exception: # pragma: no cover - defensiveCODE
LOWtests/unit_tests/transformer/moe/test_routers.py30except Exception: # pragma: no cover - defensiveCODE
LOWtests/unit_tests/models/test_hybrid_moe_model.py461 except Exception:CODE
LOW…s/unit_tests/models/mimo/test_mimo_audio_submodules.py123 except Exception as e:CODE
LOW…s/unit_tests/models/mimo/test_mimo_audio_submodules.py197 except Exception as e:CODE
LOWtests/unit_tests/models/mimo/test_mimo_submodules.py55 except Exception as e:CODE
LOWtests/unit_tests/models/mimo/test_mimo_submodules.py105 except Exception as e:CODE
LOWtests/unit_tests/models/mimo/test_mimo_submodules.py229 except Exception as e:CODE
LOWtests/unit_tests/models/mimo/test_mimo_submodules.py284 except Exception as e:CODE
MEDIUMtests/unit_tests/models/mimo/test_mimo_model.py160def setup_method(self, method):CODE
MEDIUMtests/unit_tests/models/mimo/test_mimo_model.py176def teardown_method(self, method):CODE
MEDIUMtests/unit_tests/models/mimo/test_mimo_model.py930def setup_method(self, method):CODE
MEDIUMtests/unit_tests/models/mimo/test_mimo_model.py944def teardown_method(self, method):CODE
LOWtests/unit_tests/models/mimo/test_mimo_model.py163 except Exception:CODE
LOWtests/unit_tests/models/mimo/test_mimo_model.py179 except Exception:CODE
LOWtests/unit_tests/models/mimo/test_mimo_model.py933 except Exception:CODE
LOWtests/unit_tests/models/mimo/test_mimo_model.py947 except Exception:CODE
221 more matches not shown…
Modern Structural Boilerplate248 hits · 253 pts
SeverityFileLineSnippetContext
LOWtools/check_golden_values.py12logger = logging.getLogger(__name__)CODE
LOWtools/trigger_internal_ci.py48logger = logging.getLogger(__name__)CODE
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py33logger = logging.getLogger(__name__)CODE
LOW…/unit_tests/distributed/mfsdp_v2/test_mcore_adapter.py30logger = logging.getLogger(__name__)CODE
LOW…ts/unit_tests/distributed/mfsdp_v2/test_fully_shard.py27logger = logging.getLogger(__name__)CODE
LOWtests/unit_tests/distributed/mfsdp_v2/test_memory.py23logger = logging.getLogger(__name__)CODE
LOW…sts/unit_tests/distributed/mfsdp_v2/test_annotation.py70def _setup_nvtx_recording(monkeypatch: pytest.MonkeyPatch, events: list[NvtxEvent]) -> None:CODE
LOW…sts/unit_tests/distributed/mfsdp_v2/test_cuda_graph.py20logger = logging.getLogger(__name__)CODE
LOWtests/unit_tests/models/mimo/test_mimo_1f1b_schedule.py54logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/recipe_parser.py13logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/check_status_of_main.py18logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/launch_jet_workload.py29logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/launch_jet_workload.py32logger = logging.getLogger(__name__)CODE
LOW…/test_utils/python_scripts/launch_nemo_run_workload.py17logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/notify.py37logger = logging.getLogger(__name__)CODE
LOW…st_utils/python_scripts/download_unit_tests_dataset.py18logger = logging.getLogger(__name__)CODE
LOW…/test_utils/python_scripts/compare_golden_values_kl.py63logger = logging.getLogger(__name__)CODE
LOW…test_utils/python_scripts/download_coverage_results.py15logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/auto_reminder_github.py23logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/wait_for_resources.py19logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/dashboard.py15logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/approve_merge_gate.py27logger = logging.getLogger(__name__)CODE
LOWtests/test_utils/python_scripts/swap_pr_labels.py19logger = logging.getLogger(__name__)CODE
LOW…ts/test_utils/python_scripts/download_golden_values.py19logger = logging.getLogger(__name__)CODE
LOW…l_tests/python_test_utils/compute_golden_statistics.py43logger = logging.getLogger(__name__)CODE
LOW…t_utils/test_pretraining_resume_checkpoint_pipeline.py11logger = logging.getLogger(__name__)CODE
LOW…s/python_test_utils/test_inference_regular_pipeline.py14logger = logging.getLogger(__name__)CODE
LOW…python_test_utils/test_pretraining_regular_pipeline.py11logger = logging.getLogger(__name__)CODE
LOWtests/functional_tests/python_test_utils/common.py21logger = logging.getLogger(__name__)CODE
LOW…nal_tests/python_test_utils/test_grpo_training_loop.py11logger = logging.getLogger(__name__)CODE
LOW…n_test_utils/get_test_results_from_tensorboard_logs.py13logger = logging.getLogger(__name__)CODE
LOWexamples/mimo/pretrain_mimo.py46def _set_stock_parallel_args(args: argparse.Namespace) -> None:CODE
LOWexamples/mimo/training/encoder_prefetch.py21logger = logging.getLogger(__name__)CODE
LOWexamples/mimo/training/distributed.py15__all__ = ["initialize_distributed", "print_rank_0", "shutdown_distributed"]CODE
LOWscripts/check_api_backwards_compatibility.py26logger = logging.getLogger(__name__)CODE
LOWmegatron/post_training/model_builder.py97logger = logging.getLogger(__name__)CODE
LOWmegatron/post_training/checkpointing.py32logger = logging.getLogger(__name__)CODE
LOWmegatron/core/fault_injector.py31__all__ = ["FaultInjectorConfig", "setup_fault_injection", "maybe_raise_workload_exception"]CODE
LOWmegatron/core/fault_injector.py42logger = logging.getLogger(__name__)CODE
LOWmegatron/core/parallel_state.py19logger = logging.getLogger(__name__)CODE
LOWmegatron/core/full_cuda_graph.py12logger = logging.getLogger(__name__)CODE
LOWmegatron/core/__init__.py30__all__ = [CODE
LOWmegatron/core/msc_utils.py6logger = logging.getLogger(__name__)CODE
LOWmegatron/core/msc_utils.py111__all__ = ['MultiStorageClientFeature', 'maybe_msc']CODE
LOWmegatron/core/optimizer_param_scheduler.py16logger = logging.getLogger(__name__)CODE
LOWmegatron/core/nccl_allocator.py15logger = logging.getLogger(__name__)CODE
LOWmegatron/core/utils.py54logger = logging.getLogger(__name__)CODE
LOWmegatron/core/config_logger.py126__all__ = ['has_config_logger_enabled', 'log_config_to_disk']CODE
LOWmegatron/core/num_microbatches_calculator.py9logger = logging.getLogger(__name__)CODE
LOWmegatron/core/timers.py32logger = logging.getLogger(__name__)CODE
LOWmegatron/core/rerun_state_machine.py32logger = logging.getLogger(__name__)CODE
LOWmegatron/core/rerun_state_machine.py243 def set_mode(self, mode: RerunMode) -> None:CODE
LOWmegatron/core/rerun_state_machine.py1393def _set_rerun_state_machine(rerun_state_machine) -> None:CODE
LOWmegatron/core/ssm/mamba_mixer.py110logger = logging.getLogger(__name__)CODE
LOWmegatron/core/ssm/gated_delta_product.py110logger = logging.getLogger(__name__)CODE
LOWmegatron/core/ssm/gated_delta_net/__init__.py27__all__ = [CODE
LOWmegatron/core/ssm/gated_delta_net/common.py56logger = logging.getLogger(__name__)CODE
LOWmegatron/core/ssm/gated_delta_net/gdn2.py37logger = logging.getLogger(__name__)CODE
LOWmegatron/core/ssm/ops/__init__.py30__all__ = ["mamba_chunk_scan_combined_varlen", "causal_conv1d_varlen_fn"]CODE
LOWmegatron/core/ssm/ops/gdp/__init__.py27__all__ = [CODE
188 more matches not shown…
Docstring Block Structure41 hits · 205 pts
SeverityFileLineSnippetContext
HIGHtools/checkpoint/remap_gpt_dsa_to_mamba.py46Return the HybridModel state-dict key corresponding to *key* from GPTModel. Args: key: A key from the GPTMoSTRING
HIGH…ted/mfsdp_v1/test_mcore_fully_sharded_data_parallel.py756 Run a small deterministic training loop using a mocked hybrid Mamba+MoE model and optimizer. This helpeSTRING
HIGHtests/test_utils/python_scripts/launch_jet_workload.py66Block until a downstream JET pipeline reaches a terminal status. jetclient's own ``pipeline.wait`` polls the GitLabSTRING
HIGHmegatron/core/process_groups_config.py801Get process group collection for a specific module. Args: module_name: Name of the module. STRING
HIGHmegatron/core/hyper_comm_grid.py209Create a process group based on a list of dimension names Note: The unique key used to store the process group STRING
HIGHmegatron/core/utils.py121Validates the request to the experimental function. Args: func (Callable): Callee max_lSTRING
HIGHmegatron/core/utils.py187Validates the request to the experimental function. Args: func (Callable): Callee max_lSTRING
HIGHmegatron/core/utils.py219Pass-through to callee attribute if experimental flag is enabled. Args: super (supeSTRING
HIGHmegatron/core/timers.py385Returns the output string with logged timer values according to configured options. Args: names (LiSTRING
HIGHmegatron/core/ssm/causal_conv1d.py111Apply causal Conv1d to a contiguous context-parallel shard. Args: x: Input tensor of shape ``[B, T, D]``. TSTRING
HIGH…ore/tokenizers/text/libraries/huggingface_tokenizer.py213 Adds a dictionary of special tokens (eos, pad, cls...). If special tokens are NOT in the vocabulary, thSTRING
HIGHmegatron/core/fusions/fused_bias_swiglu.py290Implementation of biased SwiGLU that handles different input shapes. This function reshapes the input if necessary,STRING
HIGHmegatron/core/fusions/fused_bias_geglu.py154Implementation of biased GEGLU that handles different input shapes. This function reshapes the input if necessary, STRING
HIGHmegatron/core/optimizer/optimizer.py597Pair each current param_group with its saved counterpart by identifier tuple. Construction order isn't part of STRING
HIGH…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py4613 Release the specified parameter bucket, freeing its associated buffer storage. This function marks or STRING
HIGH…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py5024 Creates a distributed tensor (DTensor) from a local tensor with support for Megatron-FSDP and Tensor Parallel sSTRING
HIGH…e/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py140 Validates the chunk metadata of an uneven DTensor to ensure correctness and boundary coverage. Notes: - `gSTRING
HIGH…e/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py257 Gather a DTensor with potentially uneven sharding across ranks into a full tensor. This function handles DTensSTRING
HIGH…core/distributed/fsdp/src/megatron_fsdp/fully_shard.py455 Fully shard the optimizer for Megatron-FSDP. This is an in-place operation on the optimizer instance, which modSTRING
HIGHmegatron/core/transformer/fsdp_dtensor_checkpoint.py945Report model weights that an ``fsdp_dtensor`` load cannot supply. Torch DCP loads with ``allow_partial_load=True`` STRING
HIGH…tron/core/datasets/blended_megatron_dataset_builder.py510Build the DistributedDataset Return None if and only if the underlying dataset class is not built on the currenSTRING
HIGHmegatron/core/datasets/object_storage_utils.py136Ascertain whether the object at the given S3 path exists in S3 Args: client (S3Client): The S3 client STRING
HIGHmegatron/core/datasets/indexed_dataset.py95Get the size of the dtype/code in bytes Args: key (Union[int, Type[numpy.number]]): The dtype or coSTRING
HIGHmegatron/core/datasets/indexed_dataset.py791Return from the dataset Args: idx (Union[int, numpy.integer, slice]): The index or index slice intoSTRING
HIGH…ron/core/pipeline_parallel/multimodule_communicator.py512Compute the total number of pipeline stages across a multi-module chain. Interprets ``topology`` as a directed STRING
HIGHmegatron/core/models/hybrid/hybrid_layer_allocation.py163Count layers by type across the full hybrid pattern (main + MTP). Parses the pattern to extract main and MTP componSTRING
HIGHmegatron/core/models/hybrid/hybrid_layer_allocation.py202Parse a unified hybrid pattern string into main and MTP components. The pattern uses "/" as a separator between theSTRING
HIGHmegatron/core/models/hybrid/hybrid_layer_allocation.py303Validate and convert a single pipeline segment pattern to a layer type list. This is used after the main pattern haSTRING
HIGHmegatron/core/models/hybrid/hybrid_layer_allocation.py341Select and validate the pipeline segment for the given PP rank and VP stage. When the main pattern contains '|' pipSTRING
HIGH…tron/core/dist_checkpointing/gpt_checkpoint_interop.py101Derive hybrid->GPT layer index maps from a hybrid layer pattern. Args: hybrid_layer_pattern: the run's unifSTRING
HIGHmegatron/core/dist_checkpointing/validation.py288Raises or logs an error in case missing or unexpected keys are non-empty. Args: missing_keys (Set[str]): miSTRING
HIGHmegatron/core/dist_checkpointing/validation.py366Validate if the ShardedTensors and ShardedObjects from multiple processes define correct sharding. Local ShardedTenSTRING
HIGH…tron/core/dist_checkpointing/strategies/async_utils.py669Finalizes all available calls. This method must be called on all ranks. Args: blocking (boSTRING
HIGH…ng/nvshmem_copy_service/memory/tensor_pointer_utils.py19 Extract the data pointer from a tensor. Args: tensor: Can be torch.Tensor, CuPy array, or STRING
HIGHmegatron/core/inference/headers.py102Resolve an integer wire value to its category header member. Args: value (int): The integer header value reSTRING
HIGHmegatron/core/inference/shards_spec.py101Parse + validate the ``--inference-shards`` string. Args: spec_str: Raw CLI value, e.g. ``"tp=2,dp=1+tp=1,dSTRING
HIGHmegatron/core/inference/quantization/utils.py65Resolve the canonical MXFP8 storage required by a grouped-MoE backend. Args: inference_grouped_gemm_backendSTRING
HIGHmegatron/core/export/trtllm/trtllm_layers.py85Helper function to rename model layer names to TRTLLM Layer names We go through each layer (keys) in the model STRING
HIGHmegatron/training/arguments.py173Validate model config arguments from heterogeneous config. This function takes model arguments and validates them bSTRING
HIGHmegatron/training/config/instantiate_utils.py128Instantiate an object or callable from a config object. This function takes a configuration object (dictionary, lisSTRING
HIGHmegatron/training/config/instantiate_utils.py232Recursively instantiates a node within a configuration structure. This function handles the instantiation of indiviSTRING
Redundant / Tautological Comments127 hits · 194 pts
SeverityFileLineSnippetContext
LOWtrain_rl.py61 # Check if fp8_model_init supports preserve_high_precision_init_valCOMMENT
LOWtools/check_copyright.py20 # Check if the expected header is at the start of the fileCOMMENT
LOWtools/common_pile_dataset/setup_common_pile_dataset.sh50# Check if create_common_pile_ci_dataset.py was scp'd alongside this scriptCOMMENT
LOWdocker/common/install_source_wheels.sh25# Check if required arguments are providedCOMMENT
LOWtests/unit_tests/test_optimizer.py46 # Check if FP8 block scaling is available.COMMENT
LOWtests/unit_tests/test_utils.py512 # Check if configuration was success.COMMENT
LOWtests/unit_tests/test_utils.py515 # Check if the instance is in disabled state.COMMENT
LOWtests/unit_tests/test_utils.py519 # Check if all ranks have straggler detector enabled.COMMENT
LOWtests/unit_tests/conftest.py101 # Check if data directory exists and has contentCOMMENT
LOWtests/unit_tests/fusions/test_torch_softmax.py145 # Check if output is a valid probability distributionCOMMENT
LOWtests/unit_tests/transformer/test_cuda_graphs.py838 # Check if cuda graph is correctly setting is first/last layerCOMMENT
LOWtests/unit_tests/transformer/test_attention.py416 # Check if output and bias have the correct shapeCOMMENT
LOWtests/unit_tests/transformer/test_attention.py599 # Check if the output is closeCOMMENT
LOW…/unit_tests/transformer/test_multi_latent_attention.py1603 # Check if the output is the sameCOMMENT
LOW…it_tests/transformer/moe/test_moe_layer_discrepancy.py69 # Check if parameters are the sameCOMMENT
LOW…it_tests/transformer/moe/test_moe_layer_discrepancy.py75 # Check if input is the same across all ranksCOMMENT
LOW…it_tests/transformer/moe/test_moe_layer_discrepancy.py150 # Check if output is the same across all ranksCOMMENT
LOW…it_tests/transformer/moe/test_moe_layer_discrepancy.py216 # Check if output is the same across all ranksCOMMENT
LOWtests/unit_tests/transformer/moe/test_moe_layer.py164 # Check if the moe layer is interleaved correctlyCOMMENT
LOWtests/unit_tests/models/test_llava_model.py899 # Check if output shape is as expectedCOMMENT
LOW…_tests/dist_checkpointing/test_layer_wise_optimizer.py145 # Check if optimizer is ChainedOptimizer (expected for standard setup)COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py1456 # Assign blocks to the requests (one block per request)COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py1532 # Assign blocks to the requests:COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py4668 # Check if any request was evicted during this stepCOMMENT
LOW…ts/test_utils/python_scripts/download_golden_values.py282 # Check if we should skip based on only_failing flagCOMMENT
LOW…nal_tests/test_cases/common/ckpt_converter/__main__.py895 # Print results.COMMENT
LOW…unctional_tests/shell_test_utils/run_batch_ci_tests.sh173 # Check if file is empty (job still running or not started)COMMENT
LOWtests/functional_tests/shell_test_utils/run_ci_test.sh16# Set umask to 0002 to allow group read/write permissionsCOMMENT
LOWtests/functional_tests/shell_test_utils/run_ci_test.sh276 ## Loop over the list of model configs in the params file and run each one in sequence, collectingCOMMENT
LOWdocs/conf.py43# Check if we should skip autodoc generationCOMMENT
LOWexamples/post_training/modelopt/finetune.py373 # Check if this is OpenAI chat data?COMMENT
LOWexamples/mimo/utils/logging.py40 # Print output projectionsCOMMENT
LOWexamples/gptoss/02_train.sh55# Check if checkpoint path existsCOMMENT
LOWexamples/gptoss/02_train.sh62# Check if tensorboard logs path existsCOMMENT
LOWexamples/multimodal/dataset_helpers.py121 # Check if all samples fit in the knapsack capacity.COMMENT
LOWexamples/inference/utils.py249 # Check if we have any prompts (from command line or JSONL)COMMENT
LOWexamples/inference/advanced/gpt_dynamic_inference.py272 # Check if all requests are finished.COMMENT
LOWexamples/rl/environments/countdown/countdown.py34 # Check if all numbers in equation are availableCOMMENT
LOWscripts/check_api_backwards_compatibility.py240 # Check if this breakage kind should be ignored globally (not a signature change)COMMENT
LOWscripts/check_api_backwards_compatibility.py253 # Check if this is a breakage kind we ignore for __init__ methodsCOMMENT
LOWscripts/check_api_backwards_compatibility.py263 # Check if it's a child of a filtered objectCOMMENT
LOWscripts/check_api_backwards_compatibility.py350 # Print resultsCOMMENT
LOW.github/actions/check-nvidia-sso-membership/action.yml103 # Check if SSO file is availableCOMMENT
LOW.github/actions/check-nvidia-sso-membership/action.yml112 # Check if username exists as a key in the JSON objectCOMMENT
LOWmegatron/post_training/model_builder.py325 # Set num_layers to 0 for base model in offline modeCOMMENT
LOWmegatron/core/parallel_state.py953 # Set NCCL_COLLNET_ENABLE to 1 to enable SHARP for the dp group.COMMENT
LOWmegatron/core/parallel_state.py1477 # Set NCCL_COLLNET_ENABLE to 1 to enable SHARP for the dp_replica group.COMMENT
LOWmegatron/core/parallel_state.py1506 # Set NCCL_COLLNET_ENABLE to 0 to restrict SHARP application to the dp_replica group.COMMENT
LOWmegatron/core/fp4_utils.py18# Check if Transformer Engine is installedCOMMENT
LOWmegatron/core/fp4_utils.py29# Check if Transformer Engine has class for fp4 tensors.COMMENT
LOWmegatron/core/fp8_utils.py23# Check if Transformer Engine is installedCOMMENT
LOWmegatron/core/fp8_utils.py40# Check if Transformer Engine has class for fp8 tensors.COMMENT
LOWmegatron/core/fp8_utils.py55# Check if Transformer Engine has MXFP8Tensor classCOMMENT
LOWmegatron/core/fp8_utils.py871 # Check if fp8_model_init supports setting recipeCOMMENT
LOWmegatron/core/fp8_utils.py876 # Check if fp8_model_init supports preserve_high_precision_init_valCOMMENT
LOWmegatron/core/utils.py3152 # Check if any deprecated key is present in kwargsCOMMENT
LOWmegatron/core/ssm/ops/common/causal_conv1d_triton.py243 # Check if input is 2D, temporarily treat as 3D for uniform processingCOMMENT
LOWmegatron/core/tokenizers/megatron_tokenizer.py55 # Check if metadata file existsCOMMENT
LOWmegatron/core/tensor_parallel/random.py314 # Check if we have added the stateCOMMENT
LOWmegatron/core/tensor_parallel/random.py334 # Check if the current state name is the same as the desired state name.COMMENT
67 more matches not shown…
Over-Commented Block200 hits · 176 pts
SeverityFileLineSnippetContext
LOWpretrain_hybrid.py481 # single srun, and is STALE for every restart -- re-using it backdates a restart's startup all theCOMMENT
LOW.gitlab/scripts/check_imports.py1# Copyright (c) 2025, NVIDIA CORPORATION.COMMENT
LOWtools/bisect.sh1#!/usr/bin/env bashCOMMENT
LOWtools/trigger_internal_ci.py1#!/usr/bin/env python3COMMENT
LOWtools/checkpoint/convert.py1# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWtools/checkpoint/convert.py21# full model weights, nothing split.COMMENT
LOWtools/checkpoint/convert.py41# consumed_train_samplesCOMMENT
LOWtools/checkpoint/convert.py61# "mlp l0 bias"COMMENT
LOWtools/checkpoint/gpt_hybrid_conversion.py221 else:COMMENT
LOWtools/checkpoint/gpt_hybrid_conversion.py241#COMMENT
LOWtools/common_pile_dataset/setup_common_pile_dataset.sh1#!/bin/bashCOMMENT
LOWdocker/common/install_nccl.sh1# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOW…t_cases/gpt/gpt_583m_perf_gb200_4gpu/model_config.yaml1# Inference perf test: 583M mcore-mistral checkpoint, TP=1 PP=1 DP=4 (4 GPUs).COMMENT
LOW…_cases/gpt/gpt_583m_perf_async_sched/model_config.yaml1# Inference perf test: 583M mcore-mistral checkpoint, TP=1 PP=1 DP=8 (8 GPUs),COMMENT
LOW…gpt_583m_perf_async_sched_gb200_4gpu/model_config.yaml1# Inference perf test: 583M mcore-mistral checkpoint, TP=1 PP=1 DP=4 (4 GPUs),COMMENT
LOW…rid/hybrid_nanov3_3b_perf_gb200_4gpu/model_config.yaml1# Inference perf test: nanov3 3B hybrid MoE checkpoint, TP=1 PP=1 EP=4 (4 GPUs).COMMENT
LOW…ts/performance_tests/shell_test_utils/run_perf_test.sh1#!/usr/bin/env bashCOMMENT
LOW…ts/performance_tests/shell_test_utils/run_perf_test.sh141# from /usr/lib/python3.12/dist-packages — but mamba-ssm + causal-conv1d (andCOMMENT
LOWtests/unit_tests/test_imports.py1# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWtests/unit_tests/ssm/test_gdp_dynamic_inference.py461 )COMMENT
LOW…s/checkpoint/test_gpt_hybrid_conversion_parallelism.py41# tiny synthetic DCP checkpoint and round-trips it through the converter onCOMMENT
LOWtests/unit_tests/determinism/configs.py121 # mamba-attn-mlp covers Mamba + attention + MLP paths. pure-mamba isCOMMENT
LOWtests/unit_tests/determinism/configs.py181 "TP": "tensor_model_parallel_size",COMMENT
LOW…sts/distributed/test_grad_sync_with_expert_parallel.py321 # With above conditions, the data in param_and_grad_buffer.grad_data[0] equalsCOMMENT
LOWtests/unit_tests/distributed/mfsdp_v2/test_overlap.py101COMMENT
LOW…unit_tests/transformer/test_fsdp_dtensor_checkpoint.py1# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWtests/unit_tests/transformer/test_mhc_block_manager.py401# ============================================================================COMMENT
LOW…s/unit_tests/elastification/test_hybrid_flex_router.py81 tensor_model_parallel_size=1, pipeline_model_parallel_size=1COMMENT
LOWtests/unit_tests/training/test_freeze_all_layers.py101 assert chunk.router.frozen_expert_bias is TrueCOMMENT
LOW…s/unit_tests/pipeline_parallel/test_pipeline_layout.py361# import osCOMMENT
LOW…s/unit_tests/dist_checkpointing/test_fully_parallel.py521 # main replica is in DP group with ranks [0, 2, 4, 6],COMMENT
LOW…unit_tests/generalized_tensor_parallel/test_gtp_dcp.py1141# Gated-delta-product (GDP) in_proj: gather+split under GTP_rematCOMMENT
LOW…ests/inference/contexts/test_dynamic_prefix_caching.py1241 md = ctx.mamba_metadataCOMMENT
LOW…nit_tests/inference/engines/test_gdp_cuda_graph_e2e.py101 # Batch-invariant CUDA-graph buckets align their token counts to theCOMMENT
LOW…es/test_dynamic_engine_parallel_and_chunked_prefill.py581 )COMMENT
LOWtests/test_utils/python_scripts/recipe_parser.py21# cadence remains the trigger axis.COMMENT
LOWtests/test_utils/python_scripts/test_oncall_manager.py1# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWtests/test_utils/recipes/h100/moe.yaml121 # Weekly tests: Run both DEV and LTS unless something is flaky #COMMENT
LOWtests/test_utils/recipes/h100/moe.yaml221 - test_case: [deepseek_proxy_fsdp_ep2_fsdp2_ep_overlap]COMMENT
LOWtests/test_utils/recipes/h100/gpt.yaml461 platforms: [dgx_h100]COMMENT
LOWtests/test_utils/recipes/gb200/moe.yaml101 scope: [nightly]COMMENT
LOWtests/test_utils/recipes/gb200/moe.yaml201 - environment: [dev]COMMENT
LOW…unctional_tests/test_cases/common/moe_perf/__main__.py401COMMENT
LOW…unctional_tests/test_cases/common/moe_perf/__main__.py421 # test_moe_layer_performance(case, debug_mode=True)COMMENT
LOW…_pp1_ep4_16B_mtp_sp_block_cudagraphs/model_config.yaml1# Inference functional test: MTP + sequence parallelism + block-scope CUDA graphs.COMMENT
LOW…unctional_tests/shell_test_utils/run_batch_ci_tests.sh1#!/bin/bashCOMMENT
LOWdocs/autodoc2_docstrings_parser.py1# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWdocs/conf.py1# Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOWexamples/megatron_fsdp/sbatch_mfsdp_deepseek_v3.sh201# Note: Update SBATCH parameters below according to your cluster configurationCOMMENT
LOWexamples/megatron_fsdp/train_llama3_8b_fsdp_h100_fp8.sh121 --ckpt-format fsdp_dtensorCOMMENT
LOWexamples/megatron_fsdp/sbatch_checkpoint_convert.sh21SLURM_LOGS="${OUTPUT_PATH}/slurm_logs"COMMENT
LOWexamples/mimo/train.py121 # iterator exhausted on all ranksCOMMENT
LOWexamples/inference/run_inference_server.sh1#!/bin/bashCOMMENT
LOWexamples/inference/run_offline_inference.sh1#!/bin/bashCOMMENT
LOW…nference/advanced/run_prefix_cache_lru_resume_repro.sh1#!/bin/bashCOMMENT
LOW.github/workflows/sync-team-usergroups.yml1# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOW.github/workflows/cicd-approve-test-queue.yml1# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.COMMENT
LOW.github/workflows/_build_ci_container.yml1# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.COMMENT
LOW.github/workflows/release-docs.yml1# Copyright (c) 2025, NVIDIA CORPORATION.COMMENT
LOW.github/workflows/community-bot.yml1# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.COMMENT
140 more matches not shown…
Structural Annotation Overuse87 hits · 157 pts
SeverityFileLineSnippetContext
LOWtools/checkpoint/saver_hf_llava.py17 # Step 1: Reshape back to (num_head, 3*head_dim, -1)COMMENT
LOWtools/checkpoint/saver_hf_llava.py20 # Step 2: Slice along the head_dim dimension to get q, k, vCOMMENT
LOWtools/checkpoint/saver_hf_llava.py25 # Step 3: Reshape each back to (num_head * head_dim, -1)COMMENT
LOWtools/common_pile_dataset/README.md113### Step 1: Vocabulary FilesCOMMENT
LOWtools/common_pile_dataset/README.md119### Step 2: Download Raw TextCOMMENT
LOWtools/common_pile_dataset/README.md132### Step 3: GPT PreprocessingCOMMENT
LOWtools/common_pile_dataset/README.md137### Step 4: BERT PreprocessingCOMMENT
LOWtools/common_pile_dataset/README.md150### Step 5: T5 PreprocessingCOMMENT
LOWtools/common_pile_dataset/README.md155### Step 6: VerificationCOMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py361 # Step 1: Get vocabulary filesCOMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py387 # Step 2: Get raw text dataCOMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py411 # Step 3: Preprocess for GPT (GPT2BPETokenizer)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py434 # Step 4: Preprocess for BERT (BertWordPieceLowerCase + split-sentences)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py475 # Step 5: Preprocess for T5 (BertWordPieceCase)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py497 # Step 6: Clean up and verifyCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2466 # Step 1: Forward pass for all 3 requestsCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2475 # Step 2: Forward pass where req 10 finishes, req 11 continues. Req 999 is NOT scheduled.COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2564 # Step 1: All 3 requests are active, process forward passCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2573 # Step 2: Both decode requests finish, chunked prefill NOT scheduled this step.COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2497 # Step 3: Add the next chunk. It should sit exactly at the boundary (index 1) and inherit the state.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2663 # Step 1: schedule and launch the first prefill chunk as the async primer.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2667 # Step 2: resolve the first chunk and launch the second chunk.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2734 # Step 5: resolve that batch and launch the next decode/prefill batch.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2742 # Step 6: resolve request 1's final decode and launch request 2's finalCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2685 # Step 3: resolve request 1's second chunk and launch the batch containingCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2708 # Step 4: resolve that mixed batch and launch request 1's decode withCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2766 # Step 7: resolve request 2's final prefill and launch request 3's finalCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2778 # Step 8: resolve request 3's final prefill output.COMMENT
LOW…l_tests/python_test_utils/compute_golden_statistics.py11 # Step 1: Run batch tests (from megatron-rl directory):STRING
LOW…l_tests/python_test_utils/compute_golden_statistics.py15 # Step 2: Wait for jobs to complete, then compute statistics:STRING
LOWdocs/developer/submit.md15## Step 1: Mark PR as "Ready for Review"COMMENT
LOWdocs/developer/submit.md24## Step 2: Final Review (`megatron/core` only)COMMENT
LOWdocs/developer/submit.md32## Step 3: ApprovedCOMMENT
LOWdocs/developer/submit.md37## Step 4: MergeCOMMENT
LOWdocs/user-guide/data-loading.md39### Step 1: Consolidate dataset filesCOMMENT
LOWdocs/user-guide/data-loading.md53### Step 2: Pre-build the dataset cacheCOMMENT
LOWdocs/user-guide/data-loading.md77### Step 3: Optionally pre-build per-dataset metadataCOMMENT
LOWdocs/user-guide/data-loading.md87### Step 4: Launch training with optimized data loadingCOMMENT
LOWexamples/gptoss/README.md3## Step 0: Install DependenciesCOMMENT
LOWexamples/gptoss/README.md57## Step 1: Convert HuggingFace to Megatron (Optional - skip if you already have a Megatron checkpoint)COMMENT
LOWexamples/gptoss/README.md65## Step 2: Train from ScratchCOMMENT
LOWexamples/gptoss/README.md147## Step 3: Convert Megatron to HuggingFaceCOMMENT
LOW.github/pull_request_template.md33#### Step 1: Mark PR as "Ready for Review"COMMENT
LOW.github/pull_request_template.md42#### Step 2: Final ReviewCOMMENT
LOW.github/pull_request_template.md48#### Step 3: ApprovedCOMMENT
LOW.github/scripts/oncall_manager.py135 # Step 1: Add new oncall first (include current members to avoid removing anyone yet)COMMENT
LOW.github/scripts/oncall_manager.py142 # Step 2: Now set the usergroup to contain only the new oncallCOMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1876 # Step 3: Split parameter groups to meet communication segmentation requirements.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1921 # Step 4: Generate the groups of collective buckets, where each group aggregatesCOMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1756 # Step 0: Register new FSDP unit modules.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1789 # Step 1: Group the parameters according to their execution order and attributes.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1827 # Step 2: Bucket the parameters based on the guide bucket size.COMMENT
LOWmegatron/core/transformer/moe/README.md152### Step 1: Find the feasible parallel mapping under the memory capacity of the GPUCOMMENT
LOWmegatron/core/transformer/moe/README.md170### Step 2: Select Optimal Parallelism StrategyCOMMENT
LOWmegatron/core/transformer/moe/README.md226### Step 3: Enable Performance Features Based on Profiling BottlenecksCOMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py311 # Step 1: Find maximum value in the row for numerical stabilityCOMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py323 # Step 2: Compute sum of exp(x - max_val)COMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py339 # Step 3: Compute final log_softmax values: x - max_val - log_sum_expCOMMENT
LOWmegatron/core/datasets/data_schedule.py558 # Step 1: Fetch batches and gather global sequence lengthsCOMMENT
LOWmegatron/core/datasets/data_schedule.py563 # Step 2: Check required sample keysCOMMENT
27 more matches not shown…
AI Structural Patterns177 hits · 151 pts
SeverityFileLineSnippetContext
LOWtools/bert_embedding/huggingface.py51CODE
LOWtests/unit_tests/test_layer_wise_optimizer.py210CODE
LOW…s/checkpoint/test_gpt_hybrid_conversion_parallelism.py69CODE
LOWtests/unit_tests/distributed/mfsdp_v1/utils.py39CODE
LOW…/unit_tests/transformer/test_multi_token_prediction.py780CODE
LOW…er/experimental_attention_variant/test_absorbed_mla.py83CODE
LOW…mental_attention_variant/test_attention_variant_csa.py564CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py1092CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py1239CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py1594CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py2582CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py2672CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py2759CODE
LOW…perimental_attention_variant/test_dsa_native_parity.py2910CODE
LOW…ts/unit_tests/transformer/moe/test_token_dispatcher.py44CODE
LOWtests/unit_tests/transformer/moe/test_paged_stashing.py60CODE
LOW…nit_tests/transformer/moe/test_moe_recompute_memory.py28CODE
LOW…s/elastification/test_inject_flextron_forward_logic.py33CODE
LOW…it_tests/pipeline_parallel/test_bridge_communicator.py164CODE
LOWtests/unit_tests/dist_checkpointing/utils.py179CODE
LOWtests/unit_tests/dist_checkpointing/utils.py338CODE
LOWtests/unit_tests/resharding/test_planner.py34CODE
LOW…s/inference/test_dynamic_prefix_caching_coordinator.py199CODE
LOWtests/unit_tests/inference/test_wandb_logging.py46CODE
LOWtests/unit_tests/inference/coordinator_test_utils.py19CODE
LOW…ests/inference/contexts/test_dynamic_prefix_caching.py60CODE
LOW…eration_controllers/test_text_generation_controller.py73CODE
LOW…eration_controllers/test_text_generation_controller.py3677CODE
LOW…ts/inference/engines/test_hybrid_prefix_caching_e2e.py226CODE
LOW…sts/unit_tests/inference/engines/test_static_engine.py38CODE
LOW…sts/unit_tests/inference/engines/test_static_engine.py326CODE
LOWtests/unit_tests/data/test_bin_reader.py97CODE
LOWtests/test_utils/python_scripts/recipe_parser.py363CODE
LOW…/academic_paper_scripts/detxoify_lm/perspective_api.py88CODE
LOWexamples/post_training/modelopt/utils.py76CODE
LOWexamples/post_training/modelopt/utils.py154CODE
LOWexamples/mimo/training/builder.py112CODE
LOWexamples/mimo/data/mock.py165CODE
LOWexamples/mimo/data/mock.py41CODE
LOWexamples/multimodal/model.py21CODE
LOW…ples/multimodal/model_converter/vision_model_tester.py73CODE
LOWmegatron/core/parallel_state.py600CODE
LOWmegatron/core/ssm/mamba_mixer.py174CODE
LOWmegatron/core/ssm/gated_delta_product.py198CODE
LOWmegatron/core/ssm/gated_delta_product.py713CODE
LOWmegatron/core/ssm/mamba_context_parallel.py330CODE
LOWmegatron/core/ssm/mamba_context_parallel.py357CODE
LOWmegatron/core/ssm/gated_delta_net/common.py113CODE
LOWmegatron/core/ssm/ops/gdp/chunk_h.py230CODE
LOWmegatron/core/ssm/ops/gdp/chunk.py39CODE
LOWmegatron/core/ssm/ops/gdp/fused_recurrent.py141CODE
LOWmegatron/core/ssm/ops/mamba2/ssd_combined.py23CODE
LOWmegatron/core/ssm/ops/mamba2/ssd_combined.py158CODE
LOW…ore/tokenizers/text/libraries/huggingface_tokenizer.py33CODE
LOW…n/core/tokenizers/text/libraries/tiktoken_tokenizer.py270CODE
LOWmegatron/core/fusions/fused_softmax.py176CODE
LOWmegatron/core/tensor_parallel/random.py446CODE
LOWmegatron/core/tensor_parallel/layers.py1041CODE
LOWmegatron/core/tensor_parallel/mappings.py57CODE
LOWmegatron/core/tensor_parallel/mappings.py81CODE
117 more matches not shown…
Verbosity Indicators63 hits · 116 pts
SeverityFileLineSnippetContext
LOWtools/checkpoint/saver_hf_llava.py17 # Step 1: Reshape back to (num_head, 3*head_dim, -1)COMMENT
LOWtools/checkpoint/saver_hf_llava.py20 # Step 2: Slice along the head_dim dimension to get q, k, vCOMMENT
LOWtools/checkpoint/saver_hf_llava.py25 # Step 3: Reshape each back to (num_head * head_dim, -1)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py361 # Step 1: Get vocabulary filesCOMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py387 # Step 2: Get raw text dataCOMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py411 # Step 3: Preprocess for GPT (GPT2BPETokenizer)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py434 # Step 4: Preprocess for BERT (BertWordPieceLowerCase + split-sentences)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py475 # Step 5: Preprocess for T5 (BertWordPieceCase)COMMENT
LOW…s/common_pile_dataset/create_common_pile_ci_dataset.py497 # Step 6: Clean up and verifyCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2466 # Step 1: Forward pass for all 3 requestsCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2475 # Step 2: Forward pass where req 10 finishes, req 11 continues. Req 999 is NOT scheduled.COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2564 # Step 1: All 3 requests are active, process forward passCOMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2573 # Step 2: Both decode requests finish, chunked prefill NOT scheduled this step.COMMENT
LOW…/unit_tests/inference/contexts/test_dynamic_context.py2497 # Step 3: Add the next chunk. It should sit exactly at the boundary (index 1) and inherit the state.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2663 # Step 1: schedule and launch the first prefill chunk as the async primer.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2667 # Step 2: resolve the first chunk and launch the second chunk.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2734 # Step 5: resolve that batch and launch the next decode/prefill batch.COMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2742 # Step 6: resolve request 1's final decode and launch request 2's finalCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2685 # Step 3: resolve request 1's second chunk and launch the batch containingCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2708 # Step 4: resolve that mixed batch and launch request 1's decode withCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2766 # Step 7: resolve request 2's final prefill and launch request 3's finalCOMMENT
LOW…ts/unit_tests/inference/engines/test_dynamic_engine.py2778 # Step 8: resolve request 3's final prefill output.COMMENT
LOW…l_tests/python_test_utils/compute_golden_statistics.py11 # Step 1: Run batch tests (from megatron-rl directory):STRING
LOW…l_tests/python_test_utils/compute_golden_statistics.py15 # Step 2: Wait for jobs to complete, then compute statistics:STRING
LOW.github/scripts/oncall_manager.py135 # Step 1: Add new oncall first (include current members to avoid removing anyone yet)COMMENT
LOW.github/scripts/oncall_manager.py142 # Step 2: Now set the usergroup to contain only the new oncallCOMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1876 # Step 3: Split parameter groups to meet communication segmentation requirements.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1921 # Step 4: Generate the groups of collective buckets, where each group aggregatesCOMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1756 # Step 0: Register new FSDP unit modules.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1789 # Step 1: Group the parameters according to their execution order and attributes.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1827 # Step 2: Bucket the parameters based on the guide bucket size.COMMENT
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py4476 # If use double buffer, we need to check if the next bucketCOMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py311 # Step 1: Find maximum value in the row for numerical stabilityCOMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py323 # Step 2: Compute sum of exp(x - max_val)COMMENT
LOW…e/transformer/custom_layers/batch_invariant_kernels.py339 # Step 3: Compute final log_softmax values: x - max_val - log_sum_expCOMMENT
LOWmegatron/core/datasets/data_schedule.py558 # Step 1: Fetch batches and gather global sequence lengthsCOMMENT
LOWmegatron/core/datasets/data_schedule.py563 # Step 2: Check required sample keysCOMMENT
LOWmegatron/core/datasets/data_schedule.py569 # Step 3: Schedule samples into groupsCOMMENT
LOWmegatron/core/datasets/data_schedule.py606 # Step 5: Build packed microbatchesCOMMENT
LOWmegatron/core/datasets/data_schedule.py609 # Step 6: Calculate FLOPs infoCOMMENT
LOWmegatron/core/datasets/data_schedule.py582 # Step 4: Reroute samples to DCP ranksCOMMENT
LOWmegatron/core/datasets/data_schedule.py622 # Step 7: Broadcast to PP group (for middle PP stages)COMMENT
LOWmegatron/core/datasets/data_schedule.py638 # Step 8: Broadcast to TP group (for non-TP-0 ranks)COMMENT
LOWmegatron/core/datasets/data_schedule.py652 # Step 9: create data_iterator and handle VPP if enabledCOMMENT
LOWmegatron/core/pipeline_parallel/bridge_communicator.py462 # Step 1: broadcast its shape so receivers can allocateCOMMENT
LOWmegatron/core/pipeline_parallel/bridge_communicator.py468 # Step 2: broadcast the actual tensorCOMMENT
LOW…n/core/dist_checkpointing/strategies/fully_parallel.py277 # Step 3: load part of the checkpoint.COMMENT
LOW…n/core/dist_checkpointing/strategies/fully_parallel.py335 # Step 4: exchange data between ranksCOMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py252 # Step 1: Segment tasks (break large tasks into chunks)COMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py262 # Step 2: Pack tasks into workload groupsCOMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py269 # Step 3: Schedule workloads to iterationsCOMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py278 # Step 4: Prepare iteration schedulesCOMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py284 # Step 5: Build GPU execution plansCOMMENT
LOW…gatron/core/resharding/nvshmem_copy_service/service.py293 # Step 6: Create double-buffered eventsCOMMENT
LOW…harding/nvshmem_copy_service/core/pipeline_executor.py165 # Step 1: Pack NEXT iteration (async)COMMENT
LOW…harding/nvshmem_copy_service/core/pipeline_executor.py177 # Step 2: Unpack PRIOR iteration (async)COMMENT
LOW…harding/nvshmem_copy_service/core/pipeline_executor.py192 # Step 3: Send CURRENT iterationCOMMENT
LOW…harding/nvshmem_copy_service/core/pipeline_executor.py244 # Step 5: Wait for async pack to complete (double-buffer safety)COMMENT
LOW…shmem_copy_service/planning/communication_scheduler.py40 # Step 1: Collect all batches across all PE pairsCOMMENT
LOW…shmem_copy_service/planning/communication_scheduler.py45 # Step 2: Assign batches to iterations using greedy conflict-free algorithmCOMMENT
3 more matches not shown…
AI Slop Vocabulary38 hits · 90 pts
SeverityFileLineSnippetContext
LOWtasks/finetune_utils.py120 # shuffling so we can just use a simple infinite loop.COMMENT
MEDIUMtests/unit_tests/test_inference.py107 # we are replicating what lm-eval-harness::TemplateLM::_encode_pair doesCOMMENT
MEDIUM…s/checkpoint/test_gpt_hybrid_conversion_parallelism.py43# harness launched pytest. When that default PG is multi-rank (e.g. Megatron'sCOMMENT
MEDIUM…ests/determinism/correctness/test_transformer_layer.py210 # stress harness relies on the connection count the launcherCOMMENT
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py253 # DP: Only relevant when using HSDP, where we need the flattened DP group for data parallelism. (Otherwise, just pasCOMMENT
LOW…t_tests/distributed/mfsdp_v1/test_mfsdp_fully_shard.py255 # DP-Shard-CP: Only required if using CP. Otherwise, just pass dp_shard to FSDP.COMMENT
MEDIUMtests/unit_tests/transformer/test_utils.py375 """Test a comprehensive scenario with multiple configurations."""STRING
MEDIUMtests/unit_tests/training/models/test_base.py171 """from_dict() reconstructs configs from serialized dicts, handles nested dataclasses, and is robust to unknown keysSTRING
LOW…s/unit_tests/dist_checkpointing/models/test_mlp_glu.py129 # Load happens in-place, so we can just use the same tensorsCOMMENT
MEDIUM…neralized_tensor_parallel/test_gtp_fp8_param_gather.py373 # Pure BF16 GTP reference: fp8=None overrides the harness default (recipe inert).COMMENT
MEDIUM…tests/generalized_tensor_parallel/test_gtp_muon_dcp.py268 # aliasing. The shared harness builds its own mock args, so wrap init_basic_mock_args toCOMMENT
MEDIUM…/unit_tests/inference/contexts/test_dynamic_context.py2962 # 6. Verify seamless append (no legacy offset math needed)COMMENT
MEDIUM…cipes/h100/gpt-dynamic-inference-with-coordinator.yaml98 # skills/run-performance-tests/SKILL.md for the harness it runs under.COMMENT
MEDIUM…inference_server_smoke_tp1_pp1_dp8_583m/serve_smoke.py43 # the JET harness expects at ``logs/*/*/attempt_0/*/std*.log``) while stillCOMMENT
LOW…sts/functional_tests/shell_test_utils/_run_training.sh102 # If value is "true", just use the keyCOMMENT
MEDIUM.github/workflows/claude_review.yml104 # Strict review: comprehensive Megatron-LM focused analysisCOMMENT
LOWmegatron/core/optimizer_param_scheduler.py235 # If the learning rate is constant, just return the initial value.COMMENT
LOWmegatron/core/timers.py476 # polutes the runs list, so we just add each as a scalarCOMMENT
MEDIUMmegatron/core/ssm/triton_cache_manager.py69 # use temp dir to be robust against program interruptionsCOMMENT
LOW…ore/tokenizers/text/parsers/qwen3_coder_tool_parser.py16# These map to vLLM types but we just use dictionaries for nowCOMMENT
LOWmegatron/core/tensor_parallel/random.py116 # if not using cuda graphs, just use the builtin pytorch functionCOMMENT
LOWmegatron/core/tensor_parallel/random.py197 # already graphable, just return it.COMMENT
LOWmegatron/core/tensor_parallel/random.py203 # already non-graphable, just return it.COMMENT
MEDIUM…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1931 # Set aggregate buckets by FSDP units, i.e. buckets pertaining to the sameCOMMENT
MEDIUM…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py2564 # to leverage NCCL UBR for high-precision gradient reduction withCOMMENT
MEDIUM…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py3524 # for a seamless user experience and coverage for ZeRO-1 and ZeRO-2?COMMENT
MEDIUM…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py4745 # TODO(@cspades): Clean up this logic in conjunction withCOMMENT
LOW…atron/core/distributed/fsdp/src/megatron_fsdp/utils.py199 # if not using cuda graphs, just use the builtin pytorch functionCOMMENT
LOWmegatron/core/transformer/transformer_config.py2952 # so just set both if either is specified.COMMENT
LOWmegatron/core/transformer/transformer_layer.py1128 # elements in bias_chunks are the same for all chunks, so we can just use the first oneCOMMENT
MEDIUMmegatron/core/transformer/mlp.py210 # Weight resharding across TP sizes will have aforementioned problems.COMMENT
MEDIUMmegatron/core/datasets/indexed_dataset.py6# Essentially re-written in entiretyCOMMENT
MEDIUM…e/models/common/embeddings/language_model_embedding.py149 # the original tensor from being garbage collected. Clone to facilitate GC.COMMENT
LOWmegatron/core/models/bert/bert_model.py204 # For local layer spec we just use b1ssCOMMENT
LOWmegatron/core/extensions/transformer_engine.py1062 # TODO should we ditch normalization config and just use spec to choose LayerNorm vs RMSNorm?COMMENT
LOW…tron/core/dist_checkpointing/strategies/async_utils.py479 # to simply call `sync_all_async_calls` to check if other ranks complete the writingCOMMENT
MEDIUM…nce/data_parallel_inference_coordinator/coordinator.py518 # Todo [Siddharth]: Make this more robust to handle invalid messages.COMMENT
MEDIUMmegatron/training/checkpointing.py1653# in the serialized payload is a robust, unpickle-free signal that theCOMMENT
Cross-Language Confusion15 hits · 52 pts
SeverityFileLineSnippetContext
HIGH…core/transformer/heterogeneous/heterogeneous_config.py177 "n_heads_in_group": null,STRING
HIGH…er/experimental_attention_variant/dsa_cudnn_kernels.py1666 """KL(target || predict) over selected top-K entries.STRING
HIGH…er/experimental_attention_variant/dsa_cudnn_kernels.py1718 """KL(target || predict) over the **full** KV axis, averaged over ``(B, S_q)``.STRING
HIGH…mer/experimental_attention_variant/dsa_indexer_loss.py25 """Return elementwise ``KL(target || predict)`` contributions."""STRING
HIGH…mer/experimental_attention_variant/dsa_indexer_loss.py35 """Return ``KL(target || predict)`` reduced over the key dimension."""STRING
HIGH…mer/experimental_attention_variant/dsa_indexer_loss.py43 """Return ``KL(target || predict)`` reduced over every dimension."""STRING
HIGH…mer/experimental_attention_variant/ops/tilelang_dsa.py383 """Compute KL(target || index) sum for one [s_chunk, topk] chunk."""STRING
HIGH…e/pipeline_parallel/fine_grained_activation_offload.py590 self.push(chunk)CODE
HIGH…e/pipeline_parallel/fine_grained_activation_offload.py798 self.push(cur_chunk)CODE
HIGHmegatron/core/inference/inference_request.py369 Each block hash is computed as SHA-256(parent_digest || block_bytes), whereSTRING
HIGHmegatron/core/inference/unified_memory.py113 if (device != prev_device && device >= 0) cudaSetDevice(device);CODE
HIGHmegatron/core/inference/unified_memory.py149 if (device != prev_device && prev_device >= 0) cudaSetDevice(prev_device);CODE
HIGHmegatron/core/inference/moe/activations.py187 [M, ffn_hidden] BF16 (rows >= n_rows undefined).STRING
HIGHmegatron/core/inference/moe/vllm_fused_moe.py628 `input` are undefined).STRING
HIGHmegatron/training/arguments.py3717 "n_heads_in_group": null,STRING
Fake / Example Data28 hits · 30 pts
SeverityFileLineSnippetContext
LOWtools/run_dynamic_text_generation_server.py260 "--language-model-type", "placeholder",CODE
LOWtests/unit_tests/transformer/test_full_cuda_graph.py107 dummy_data = torch.ones(1, 4)CODE
LOWtests/unit_tests/transformer/test_full_cuda_graph.py112 return model(dummy_data), loss_funcCODE
LOWtests/unit_tests/pipeline_parallel/test_schedules.py554 dummy_data = torch.ones(1, 4)CODE
LOWtests/unit_tests/pipeline_parallel/test_schedules.py559 return model(dummy_data), loss_funcCODE
LOW…sts/unit_tests/dist_checkpointing/test_safe_globals.py28 dummy_obj = Namespace(dummy_value=0)CODE
LOWtests/unit_tests/rl/test_rl_utils.py1889 # Adapter stamps pass through; bare empties are inferred to 'placeholder'.COMMENT
LOWtests/unit_tests/rl/test_rl_utils.py1891 ["ok", "ok", "placeholder"],CODE
LOWtests/unit_tests/rl/test_rl_utils.py1892 ["masked", "placeholder", "placeholder"],CODE
LOWtests/unit_tests/rl/test_rl_utils.py1653 group.append("placeholder")CODE
LOWtests/unit_tests/rl/test_rl_utils.py1654 rollout_statuses.append(["placeholder", "placeholder"])CODE
LOWtests/unit_tests/rl/test_rl_utils.py1810 rollout_statuses=[["placeholder", "placeholder"]],CODE
LOWtests/unit_tests/rl/test_rl_utils.py1846 problem_id="placeholder",CODE
LOWexamples/multimodal/text_generation_mistral_clip.sh7INPUT_IMAGE_PATH="placeholder"CODE
LOWexamples/multimodal/text_generation_mistral_clip.sh8GROUNDTRUTH_PATH="placeholder"CODE
LOW…imodal/nvlm/run_text_generation_yi_34b_internvit_6b.sh7INPUT_IMAGE_PATH="placeholder"CODE
LOW…imodal/nvlm/run_text_generation_yi_34b_internvit_6b.sh8GROUNDTRUTH_PATH="placeholder"CODE
LOW…ultimodal/nvlm/run_text_generation_qwen25_7b_siglip.sh8INPUT_IMAGE_PATH="placeholder"CODE
LOW…ultimodal/nvlm/run_text_generation_qwen25_7b_siglip.sh9GROUNDTRUTH_PATH="placeholder"CODE
LOW…al/nvlm/run_text_generation_qwen20_72b_internvit_6b.sh8INPUT_IMAGE_PATH="placeholder"CODE
LOW…al/nvlm/run_text_generation_qwen20_72b_internvit_6b.sh9GROUNDTRUTH_PATH="placeholder"CODE
LOW…/nvlm/run_text_generation_qwen25_7b_internvit_video.sh8INPUT_IMAGE_PATH="placeholder"CODE
LOW…/nvlm/run_text_generation_qwen25_7b_internvit_video.sh9GROUNDTRUTH_PATH="placeholder"CODE
LOW…al/llama_3p1_nemotron_nano_vl_8b_v1/text_generation.sh7INPUT_IMAGE_PATH="placeholder"CODE
LOW…al/llama_3p1_nemotron_nano_vl_8b_v1/text_generation.sh8GROUNDTRUTH_PATH="placeholder"CODE
LOWmegatron/rl/rl_utils.py1400 rollout_status = 'placeholder'CODE
LOWmegatron/rl/types/rollout.py12KNOWN_ROLLOUT_STATUSES = ('ok', 'placeholder', 'masked', 'graded')CODE
LOWmegatron/rl/agent/rollout_pipeline.py504 return rollout.is_placeholder and rollout.rollout_status in ('ok', 'placeholder')CODE
Modern AI Meta-Vocabulary9 hits · 27 pts
SeverityFileLineSnippetContext
MEDIUMtests/functional_tests/shell_test_utils/run_ci_test.sh369 # orchestration's resumable retrigger.COMMENT
MEDIUMmegatron/core/ssm/ssm_inference.py188 # Shared orchestration.COMMENT
MEDIUM…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py1842 # We may share the embedding model weight and the final output layer,COMMENT
MEDIUMmegatron/core/transformer/transformer_config.py368 # TODO(#6402): consumed by DSv4 Hybrid attention orchestration, which selects theCOMMENT
MEDIUMmegatron/core/inference/apis/async_llm.py235 # SIGINT or out-of-band orchestration).COMMENT
MEDIUMmegatron/core/inference/apis/llm.py228 # SIGINT or out-of-band orchestration).COMMENT
MEDIUM…t_generation_controllers/text_generation_controller.py3671 # Pick the context window that we need to pass through the network.COMMENT
MEDIUMmegatron/rl/rl_utils.py1332 # A turn must never exceed the model's context window.COMMENT
MEDIUMmegatron/rl/inference/megatron.py72 # detokenizing prompt tokens, especially expensive for long prompts in agentic RL.COMMENT
Dead Code5 hits · 10 pts
SeverityFileLineSnippetContext
MEDIUMtools/checkpoint/loader_llava.py52CODE
MEDIUM…nce_tp1_pp1_583m_cuda_graphs_validation/cuda_graphs.py183CODE
MEDIUM…nce_tp1_pp1_583m_cuda_graphs_validation/cuda_graphs.py184CODE
MEDIUM…nce_tp1_pp1_583m_cuda_graphs_validation/cuda_graphs.py185CODE
MEDIUM…nce_tp1_pp1_583m_cuda_graphs_validation/cuda_graphs.py186CODE
Example Usage Blocks4 hits · 6 pts
SeverityFileLineSnippetContext
LOWtools/bisect.sh5# Usage:COMMENT
LOWtools/common_pile_dataset/setup_common_pile_dataset.sh9# Usage:COMMENT
LOW…unctional_tests/shell_test_utils/run_batch_ci_tests.sh5# Usage:COMMENT
LOW…nference/advanced/run_prefix_cache_lru_resume_repro.sh12# Usage:COMMENT
Synthetic Comment Markers1 hit · 5 pts
SeverityFileLineSnippetContext
HIGH.github/workflows/claude-copy-to-main.yml203 echo "🤖 **This PR was auto-generated by Claude** via the Claude copy workflow."CODE
TODO Padding3 hits · 3 pts
SeverityFileLineSnippetContext
LOW…/inference/test_data_parallel_inference_coordinator.py133 self.ep_consensus_interval = 1 # TODO: add tests for interval > 1 (e.g. verify pausing is delayed by at most N CODE
LOW…ibuted/fsdp/src/megatron_fsdp/param_and_grad_buffer.py5089 # TODO: Add validation checks for the legality of DTensor.COMMENT
LOWmegatron/core/transformer/multi_token_prediction.py1855 # TODO: implement block-based recompute for MTPCOMMENT
Slop Phrases2 hits · 2 pts
SeverityFileLineSnippetContext
MEDIUMmegatron/core/process_groups_config.py793 """Check if this rank has a language model.STRING
MEDIUMmegatron/core/process_groups_config.py796 True if this rank has a language model, False otherwise.STRING
Overly Generic Function Names3 hits · 2 pts
SeverityFileLineSnippetContext
LOW…s/unit_tests/training/config/test_instantiate_utils.py52def test_function(arg1=None, arg2=None, **kwargs):CODE
LOWtests/unit_tests/training/config/test_yaml_utils.py45def test_function():CODE
LOWmegatron/core/utils.py2848 def my_function():STRING