Repository Analysis

datajuicer/data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

22.3 Moderate AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of datajuicer/data-juicer, a Python project with 6,717 GitHub stars. SynthScan v2.0 examined 211,400 lines of code across 1177 source files, recording 3507 pattern matches distributed across 20 syntactic categories. The overall adjusted score of 22.3 places this repository in the Moderate AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

22.3
Adjusted Score
22.3
Raw Score
100%
Time Factor
2026-07-14
Last Push
6.7K
Stars
Python
Language
211.4K
Lines of Code
1.2K
Files
3.5K
Pattern Hits
2026-07-14
Scan Date
0.05
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 55MEDIUM 437LOW 3015

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 3507 distinct pattern matches across 20 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers1576 hits · 1587 pts
SeverityFileLineSnippetContext
LOWservice.py38def register_objects_from_init(directory: str):CODE
LOWtools/check_ray_cluster.py6def get_ray_cluster_resources():CODE
LOWtools/evaluator/evaluator.py221 def megatron_checkpoint_exists(self, iteration):CODE
LOWtools/converter/modeling_megatron_llama.py419 def _set_gradient_checkpointing(self, module, value=False):STRING
LOWtools/converter/modeling_megatron_llama.py526 def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):STRING
LOWtools/converter/modeling_megatron_llama.py803 def prepare_inputs_for_generation(CODE
LOWtools/converter/convert_gpt_to_transformers.py60def add_transformers_checkpoint_args(parser):CODE
LOWtools/converter/convert_gpt_to_transformers.py148def megatron_to_transformers_fix_query_key_value_ordering(CODE
LOWtools/converter/convert_gpt_to_transformers.py186def transformers_to_megatron_fix_query_key_value_ordering(CODE
LOWtools/converter/convert_gpt_to_transformers.py224def merge_transformers_sharded_states(path, num_checkpoints):CODE
LOWtools/converter/convert_gpt_to_transformers.py240def get_megatron_sharded_states(args, tp_size, pp_size, pp_rank):CODE
LOWtools/converter/convert_gpt_to_transformers.py262def get_element_from_dict_by_path(d, path):CODE
LOWtools/converter/convert_gpt_to_transformers.py279def convert_checkpoint_from_megatron_to_transformers(args):CODE
LOWtools/mm_eval/inception_metrics/util.py306def get_module_dir_by_obj_name(obj_name: str) -> str:CODE
LOWtools/mm_eval/inception_metrics/util.py317def get_top_level_function_name(obj: Any) -> str:CODE
LOWtools/mm_eval/inception_metrics/util.py329def list_dir_recursively_with_ignore(dir_path: str, ignores: List[str] = None, add_base_to_relative: bool = False) -> LiCODE
LOWtools/mm_eval/inception_metrics/util.py362def copy_files_and_create_dirs(files: List[Tuple[str, str]]) -> None:CODE
LOW…m_eval/inception_metrics/video_metrics/metric_utils.py78def get_feature_detector_name(url):CODE
LOW…m_eval/inception_metrics/video_metrics/metric_utils.py269def compute_feature_stats_for_dataset(CODE
LOW…m_eval/inception_metrics/video_metrics/metric_utils.py373def compute_feature_stats_for_generator(CODE
LOW…m_eval/inception_metrics/video_metrics/metric_utils.py484def rewrite_opts_for_gen_dataset(opts):CODE
LOW…onversion/multimodal/absolute_path_to_relative_path.py76def convert_absolute_path_to_relative_path(CODE
LOWtools/fmt_conversion/multimodal/utils.py29def check_args_load_to_dj_data(CODE
LOW…ormat_to_target_format/dj_to_llama_factory_sharegpt.py71def dj_to_llama_factory_sharegpt(CODE
LOW…_to_data_juicer_format/llama_factory_sharegpt_to_dj.py72def llama_factory_sharegpt_to_dj(CODE
LOWtools/humanops/label_studio_service.py257def start_label_studio_container(CODE
LOWtools/humanops/label_studio_service.py631def get_container_error_message(container_name):CODE
LOWtools/humanops/label_studio_service.py1063def start_label_studio_with_docker(args, api_token):STRING
LOW.pre-commit-hooks/build_op_doc.py252def _compact_class_or_module_name(name: str) -> str:STRING
LOW.pre-commit-hooks/build_op_doc.py316def get_op_list_from_code_for_formatter():CODE
LOW.pre-commit-hooks/build_op_doc.py492def generate_op_table_section(op_type, op_record_list, reference_op_record_dict):CODE
LOW.pre-commit-hooks/build_op_doc.py540def get_op_desc_in_en_zh_batched(descs):CODE
LOW.pre-commit-hooks/build_op_doc.py576def parse_op_record_from_current_doc():CODE
LOW.pre-commit-hooks/build_op_doc.py619def check_and_update_op_record(old_op_record_list, new_op_record_list):CODE
LOWtests/tools/test_mcp_server.py19 def test_granular_ops_mode_default_transport(self):CODE
LOWtests/tools/test_mcp_server.py31 def test_recipe_flow_mode_default_transport(self):CODE
LOWtests/tools/test_mcp_server.py65 def test_custom_port_passed_to_create(self):CODE
LOWtests/tools/test_mcp_server.py85 def test_invalid_transport_exits(self):CODE
LOWtests/tools/test_mcp_server.py107 def test_server_transport_env_var_set(self):CODE
LOWtests/tools/test_op_search.py60 def test_search_all_returns_nonempty(self):CODE
LOWtests/tools/test_op_search.py64 def test_search_by_op_type_filter(self):CODE
LOWtests/tools/test_op_search.py70 def test_search_by_op_type_mapper(self):CODE
LOWtests/tools/test_op_search.py82 def test_search_match_all_tags(self):CODE
LOWtests/tools/test_op_search.py88 def test_search_match_any_tag(self):CODE
LOWtests/tools/test_op_search.py94 def test_search_by_type_and_tag(self):CODE
LOWtests/tools/test_op_search.py100 def test_search_invalid_type_returns_empty(self):CODE
LOWtests/tools/test_op_search.py104 def test_search_result_is_list_of_dicts(self):CODE
LOWtests/tools/test_op_search.py119 def test_regex_search_by_name(self):CODE
LOWtests/tools/test_op_search.py125 def test_regex_search_case_insensitive(self):CODE
LOWtests/tools/test_op_search.py130 def test_regex_search_with_op_type_filter(self):CODE
LOWtests/tools/test_op_search.py136 def test_regex_search_with_tag_filter(self):CODE
LOWtests/tools/test_op_search.py141 def test_regex_search_custom_fields(self):CODE
LOWtests/tools/test_op_search.py147 def test_regex_search_invalid_pattern_returns_empty(self):CODE
LOWtests/tools/test_op_search.py151 def test_regex_search_no_match_returns_empty(self):CODE
LOWtests/tools/test_op_search.py155 def test_regex_search_result_is_list_of_dicts(self):CODE
LOWtests/tools/test_op_search.py233 def test_specified_ops_only_scans_given_ops(self):CODE
LOWtests/tools/test_op_search.py241 def test_all_ops_dict_populated(self):CODE
LOWtests/tools/test_op_search.py246 def test_records_map_deprecated_still_returns_all_ops(self):CODE
LOWtests/tools/test_op_search.py16 def test_record_has_required_fields(self):CODE
LOWtests/tools/test_op_search.py21 def test_record_type_is_valid(self):CODE
1516 more matches not shown…
Decorative Section Separators321 hits · 1113 pts
SeverityFileLineSnippetContext
MEDIUMapp.py3# --------------------------------------------------------COMMENT
MEDIUMtools/check_s3_integration.py392# ============================================================================COMMENT
MEDIUMtools/check_s3_integration.py394# ============================================================================COMMENT
MEDIUMtools/check_s3_integration.py744# ============================================================================COMMENT
MEDIUMtools/check_s3_integration.py746# ============================================================================COMMENT
MEDIUMtools/evaluator/gpt_eval/gpt_evaluator.py3# --------------------------------------------------------COMMENT
MEDIUMtools/distributed_deduplication/dedup_utils.py3# --------------------------------------------------------COMMENT
MEDIUMtools/converter/modeling_megatron_llama.py5# --------------------------------------------------------COMMENT
MEDIUMtools/converter/convert_gpt_to_transformers.py3# --------------------------------------------------------COMMENT
MEDIUMtools/converter/convert_gpt_to_transformers.py7# --------------------------------------------------------COMMENT
MEDIUMtools/preprocess/raw_arxiv_to_jsonl.py3# --------------------------------------------------------COMMENT
MEDIUMtools/preprocess/raw_stackexchange_to_jsonl.py3# --------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py37# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py116# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py136# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py234# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py327# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/util.py376# ------------------------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/distributed.py16# ----------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/distributed.py23# ----------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/distributed.py30# ----------------------------------------------------------------------------COMMENT
MEDIUMtools/mm_eval/inception_metrics/distributed.py37# ----------------------------------------------------------------------------COMMENT
MEDIUM…al/inception_metrics/video_metrics/precision_recall.py18#----------------------------------------------------------------------------COMMENT
MEDIUM…al/inception_metrics/video_metrics/precision_recall.py35#----------------------------------------------------------------------------COMMENT
MEDIUM…al/inception_metrics/video_metrics/precision_recall.py75#----------------------------------------------------------------------------COMMENT
MEDIUM…on_metrics/video_metrics/frechet_inception_distance.py22#----------------------------------------------------------------------------COMMENT
MEDIUM…on_metrics/video_metrics/frechet_inception_distance.py69# ----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py51#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py74#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py106#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py221#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py266#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py370#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py482#----------------------------------------------------------------------------COMMENT
MEDIUM…m_eval/inception_metrics/video_metrics/metric_utils.py493# ----------------------------------------------------------------------------COMMENT
MEDIUM…ception_metrics/video_metrics/kernel_video_distance.py19#----------------------------------------------------------------------------COMMENT
MEDIUM…ception_metrics/video_metrics/kernel_video_distance.py76# ----------------------------------------------------------------------------COMMENT
MEDIUM…mm_eval/inception_metrics/video_metrics/metric_main.py31#----------------------------------------------------------------------------COMMENT
MEDIUM…mm_eval/inception_metrics/video_metrics/metric_main.py49#----------------------------------------------------------------------------COMMENT
MEDIUM…mm_eval/inception_metrics/video_metrics/metric_main.py88#----------------------------------------------------------------------------COMMENT
MEDIUM…mm_eval/inception_metrics/video_metrics/metric_main.py102#----------------------------------------------------------------------------COMMENT
MEDIUM…eption_metrics/video_metrics/frechet_video_distance.py23#----------------------------------------------------------------------------COMMENT
MEDIUM…eption_metrics/video_metrics/frechet_video_distance.py73# ----------------------------------------------------------------------------COMMENT
MEDIUM…eption_metrics/video_metrics/video_precision_recall.py19#----------------------------------------------------------------------------COMMENT
MEDIUM…eption_metrics/video_metrics/video_precision_recall.py36#----------------------------------------------------------------------------COMMENT
MEDIUM…eption_metrics/video_metrics/video_precision_recall.py92#----------------------------------------------------------------------------COMMENT
MEDIUM…ception_metrics/video_metrics/video_inception_score.py22#----------------------------------------------------------------------------COMMENT
MEDIUM…ception_metrics/video_metrics/video_inception_score.py66# ----------------------------------------------------------------------------COMMENT
MEDIUM…val/inception_metrics/video_metrics/inception_score.py18#----------------------------------------------------------------------------COMMENT
MEDIUM…val/inception_metrics/video_metrics/inception_score.py52# ----------------------------------------------------------------------------COMMENT
MEDIUM…ion_metrics/video_metrics/kernel_inception_distance.py18#----------------------------------------------------------------------------COMMENT
MEDIUM…ion_metrics/video_metrics/kernel_inception_distance.py58# ----------------------------------------------------------------------------COMMENT
MEDIUMtests/run.py6# --------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py206 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py208 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py255 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py257 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py296 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_exporter.py298 # ------------------------------------------------------------------COMMENT
MEDIUMtests/core/test_ray_exporter.py346 # ------------------------------------------------------------------COMMENT
261 more matches not shown…
Excessive Try-Catch Wrapping337 hits · 347 pts
SeverityFileLineSnippetContext
LOWservice.py78 except Exception as e:CODE
LOWservice.py102 except Exception as e:CODE
LOWapp.py118 except Exception as e:CODE
LOWapp.py179 except Exception as e:CODE
LOWapp.py421 except Exception as e:CODE
LOWtools/dj_install.py37 except Exception as e:CODE
LOWtools/dj_install.py65 except Exception as e:CODE
LOWtools/dj_install.py133 except Exception as e:CODE
LOWtools/check_s3_integration.py92 except Exception as e:CODE
LOWtools/check_s3_integration.py191 except Exception as e:CODE
LOWtools/check_s3_integration.py222 except Exception:CODE
LOWtools/check_s3_integration.py227 except Exception as e:CODE
LOWtools/check_s3_integration.py274 except Exception as e:CODE
LOWtools/check_s3_integration.py314 except Exception:CODE
LOWtools/check_s3_integration.py381 except Exception as e:CODE
LOWtools/check_s3_integration.py451 except Exception as e:CODE
LOWtools/check_s3_integration.py541 except Exception as e:CODE
LOWtools/check_s3_integration.py575 except Exception:CODE
LOWtools/check_s3_integration.py623 except Exception as e:CODE
LOWtools/check_s3_integration.py667 except Exception:CODE
LOWtools/check_s3_integration.py728 except Exception as e:CODE
LOWtools/check_s3_integration.py820 except Exception as e:CODE
LOWtools/check_s3_integration.py851 except Exception:CODE
LOWtools/check_s3_integration.py918 except Exception as e:CODE
LOWtools/check_ray_cluster.py80 except Exception as e:CODE
LOWtools/count_rows.py56 except Exception as e:CODE
LOWtools/count_rows.py76 except Exception as e:CODE
MEDIUMtools/count_rows.py77 print(f"Error: Could not read file with extension {extension}: {e}")CODE
LOWtools/count_rows.py127 except Exception as e:CODE
MEDIUMtools/count_rows.py160 print(f"Error: Path not found: {args.path}")CODE
MEDIUMtools/count_rows.py185 print(f"Error: Path is neither a file nor a directory: {args.path}")CODE
LOWtools/generate_uv_lock.py61 except Exception as e:CODE
MEDIUMtools/generate_uv_lock.py62 print(f"Error: {e}", file=sys.stderr)CODE
LOWtools/generate_uv_lock.py76 except Exception as e:CODE
MEDIUMtools/generate_uv_lock.py77 print(f"Error: {e}", file=sys.stderr)CODE
LOWtools/evaluator/recorder/wandb_writer.py188 except Exception as e:CODE
LOWtools/evaluator/recorder/wandb_writer.py229 except Exception:CODE
LOWtools/evaluator/recorder/wandb_writer.py233 except Exception:CODE
LOWtools/evaluator/gpt_eval/gpt_evaluator.py74 except Exception:CODE
LOWtools/evaluator/gpt_eval/gpt_evaluator.py169 except Exception as e:CODE
LOWtools/evaluator/gpt_eval/answer_generator.py85 except Exception as e:CODE
LOWtools/evaluator/gpt_eval/answer_generator.py184 except Exception as e:CODE
MEDIUMtools/evaluator/gpt_eval/answer_generator.py187 print(f"Error in megatron response: {response}, retry in 10s")CODE
LOWtools/preprocess/raw_arxiv_to_jsonl.py54 except Exception as e:CODE
LOWtools/preprocess/raw_arxiv_to_jsonl.py65 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py285 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py296 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py512 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py616 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py665 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py679 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py697 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py810 except Exception as e:CODE
LOWtools/humanops/label_studio_service.py851 except Exception as e:STRING
LOWtools/humanops/label_studio_service.py857 except Exception as cleanup_error:STRING
LOWtools/humanops/label_studio_service.py866 except Exception as e:STRING
LOW.pre-commit-hooks/build_op_doc.py553 except Exception:CODE
LOWtests/run.py118 except Exception as e:CODE
LOWtests/run.py123 except Exception as e:CODE
LOWtests/tools/test_process_data.py52 except Exception as e:CODE
277 more matches not shown…
Unused Imports413 hits · 324 pts
SeverityFileLineSnippetContext
LOW…on_metrics/video_metrics/frechet_inception_distance.py19CODE
LOWtests/core/test_exporter.py2CODE
LOWtests/core/test_exporter.py4CODE
LOWtests/core/executor/test_partition_size_optimizer.py12CODE
LOWtests/core/executor/test_partition_size_optimizer.py13CODE
LOWtests/core/executor/test_partition_size_optimizer.py15CODE
LOWtests/core/executor/test_partition_size_optimizer.py15CODE
LOWtests/core/executor/test_event_logging_mixin.py20CODE
LOWtests/core/executor/test_event_logging_mixin.py20CODE
LOWtests/core/executor/test_partitioned_integration.py19CODE
LOWtests/core/executor/test_dag_execution_strategies.py14CODE
LOWtests/core/executor/test_dag_execution_strategies.py16CODE
LOWtests/core/executor/test_ray_executor_partitioned.py3CODE
LOWtests/core/executor/test_ray_executor_partitioned.py298CODE
LOWtests/core/executor/test_ray_executor_partitioned.py298CODE
LOWtests/core/executor/test_ray_executor_partitioned.py299CODE
LOWtests/config/test_config.py2CODE
LOWtests/config/test_config.py3CODE
LOWtests/config/test_config.py14CODE
LOWtests/utils/test_model_utils.py6CODE
LOWtests/utils/test_datasets_json_compat.py7CODE
LOWtests/utils/test_logger_utils.py204CODE
LOWtests/utils/test_lazy_loader.py8CODE
LOWtests/utils/test_lazy_loader.py9CODE
LOWtests/utils/test_lazy_loader.py11CODE
LOWtests/utils/test_s3_utils.py6CODE
LOWtests/format/test_empty_formatter.py1CODE
LOWtests/format/test_text_formatter.py1CODE
LOWtests/format/test_text_formatter.py157CODE
LOWtests/ops/test_op_fusion.py4CODE
LOWtests/ops/test_op_fusion.py7CODE
LOW…mapper/test_video_whole_body_pose_estimation_mapper.py7CODE
LOWtests/ops/mapper/test_sdxl_prompt2prompt_mapper.py1CODE
LOWtests/ops/mapper/test_sdxl_prompt2prompt_mapper.py2CODE
LOWtests/ops/mapper/test_video_undistort_mapper.py3CODE
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py8CODE
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py11CODE
LOW…per/test_video_camera_calibration_droidcalib_mapper.py9CODE
LOWtests/ops/mapper/test_video_split_by_duration_mapper.py12CODE
LOW…apper/test_imgdiff_difference_area_generator_mapper.py5CODE
LOW…apper/test_imgdiff_difference_area_generator_mapper.py6CODE
LOW…apper/test_imgdiff_difference_area_generator_mapper.py7CODE
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py7CODE
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py10CODE
LOWtests/ops/mapper/test_query_intent_detection_mapper.py2CODE
LOW…ts/ops/mapper/test_video_hand_action_compute_mapper.py1CODE
LOWtests/ops/mapper/test_vggt_mapper.py7CODE
LOW…er/test_imgdiff_difference_caption_generator_mapper.py6CODE
LOW…er/test_imgdiff_difference_caption_generator_mapper.py7CODE
LOW…er/test_imgdiff_difference_caption_generator_mapper.py8CODE
LOW…er/test_imgdiff_difference_caption_generator_mapper.py10CODE
LOW…/annotation/test_human_preference_annotation_mapper.py3CODE
LOW…/annotation/test_human_preference_annotation_mapper.py5CODE
LOWtests/ops/mapper/annotation/test_annotation_mapper.py2CODE
LOWtests/ops/mapper/annotation/test_annotation_mapper.py5CODE
LOWtests/ops/filter/test_image_pair_similarity_filter.py8CODE
LOW…filter/test_instruction_following_difficulty_filter.py6CODE
LOWtests/ops/filter/test_llm_perplexity_filter.py6CODE
LOWtests/ops/deduplicator/test_ray_video_deduplicator.py4CODE
LOWtests/ops/deduplicator/test_ray_image_deduplicator.py4CODE
353 more matches not shown…
Deep Nesting326 hits · 304 pts
SeverityFileLineSnippetContext
LOWservice.py38CODE
LOWhatch_build.py21CODE
LOWhatch_build.py31CODE
LOWapp.py149CODE
LOWapp.py220CODE
LOWapp.py387CODE
LOWapp.py243CODE
LOWtools/dj_install.py18CODE
LOWtools/dj_install.py42CODE
LOWtools/dj_install.py70CODE
LOWtools/check_s3_integration.py38CODE
LOWtools/check_s3_integration.py101CODE
LOWtools/check_s3_integration.py201CODE
LOWtools/check_s3_integration.py283CODE
LOWtools/check_ray_cluster.py6CODE
LOWtools/count_rows.py34CODE
LOWtools/data_resplit.py13CODE
LOWtools/process_data.py18CODE
LOWtools/evaluator/recorder/wandb_writer.py99CODE
LOWtools/evaluator/recorder/wandb_writer.py214CODE
LOWtools/evaluator/gpt_eval/answer_generator.py202CODE
LOWtools/converter/modeling_megatron_llama.py876CODE
LOWtools/converter/convert_gpt_to_transformers.py279CODE
LOWtools/preprocess/raw_arxiv_to_jsonl.py29CODE
LOWtools/preprocess/raw_arxiv_to_jsonl.py73CODE
LOWtools/preprocess/raw_stackexchange_to_jsonl.py57CODE
LOWtools/preprocess/raw_stackexchange_to_jsonl.py84CODE
LOWtools/preprocess/serialize_meta.py20CODE
LOWtools/preprocess/reformat_jsonl_nan_value.py30CODE
LOWtools/mm_eval/inception_metrics/util.py396CODE
LOWtools/mm_eval/inception_metrics/dataset.py24CODE
LOWtools/mm_eval/inception_metrics/dataset.py101CODE
LOW…onversion/multimodal/absolute_path_to_relative_path.py76CODE
LOW…dal/data_juicer_format_to_target_format/dj_to_llava.py72CODE
LOW…data_juicer_format_to_target_format/dj_to_internvid.py41CODE
LOW…odal/data_juicer_format_to_target_format/dj_to_mmc4.py98CODE
LOW…_juicer_format_to_target_format/dj_to_video_chatgpt.py44CODE
LOW…dal/data_juicer_format_to_target_format/dj_to_youku.py65CODE
LOW…al/data_juicer_format_to_target_format/dj_to_msrvtt.py50CODE
LOW…source_format_to_data_juicer_format/internvid_to_dj.py54CODE
LOW…dal/source_format_to_data_juicer_format/llava_to_dj.py73CODE
LOW…odal/source_format_to_data_juicer_format/mmc4_to_dj.py92CODE
LOW…_to_data_juicer_format/llama_factory_sharegpt_to_dj.py72CODE
LOWtools/humanops/label_studio_service.py257CODE
LOWtools/humanops/label_studio_service.py631CODE
LOWtools/humanops/label_studio_service.py710CODE
LOWtools/humanops/label_studio_service.py815CODE
LOW.pre-commit-hooks/build_op_doc.py170CODE
LOW.pre-commit-hooks/build_op_doc.py316CODE
LOW.pre-commit-hooks/build_op_doc.py576CODE
LOW.pre-commit-hooks/build_op_doc.py619CODE
LOWtests/tools/test_process_data.py12CODE
LOWtests/tools/test_process_data.py189CODE
LOWtests/tools/test_process_data.py232CODE
LOWtests/tools/test_process_data.py275CODE
LOWtests/tools/test_process_data.py330CODE
LOWtests/utils/test_availability_utils.py86CODE
LOWtests/utils/test_availability_utils.py97CODE
LOWtests/utils/test_video_utils.py83CODE
LOW…/mapper/test_video_hand_reconstruction_hawor_mapper.py44CODE
266 more matches not shown…
Cross-File Repetition49 hits · 245 pts
SeverityFileLineSnippetContext
HIGHtools/preprocess/raw_arxiv_to_jsonl.py0find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over jsonSTRING
HIGHtools/preprocess/serialize_meta.py0find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over jsonSTRING
HIGHtools/preprocess/reformat_jsonl_nan_value.py0find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over jsonSTRING
HIGHtools/postprocess/deserialize_meta.py0find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over jsonSTRING
HIGHtools/preprocess/raw_alpaca_cot_merge_add_meta.py0find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over jsonSTRING
HIGH…data_juicer_format_to_target_format/dj_to_internvid.py0convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-juiSTRING
HIGH…_juicer_format_to_target_format/dj_to_video_chatgpt.py0convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-juiSTRING
HIGH…al/data_juicer_format_to_target_format/dj_to_msrvtt.py0convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-juiSTRING
HIGH…al/source_format_to_data_juicer_format/msrvtt_to_dj.py0convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like dSTRING
HIGH…ce_format_to_data_juicer_format/video_chatgpt_to_dj.py0convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like dSTRING
HIGH…dal/source_format_to_data_juicer_format/youku_to_dj.py0convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like dSTRING
HIGHtests/ops/mapper/test_extract_event_mapper.py0△芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到STRING
HIGHtests/ops/mapper/test_extract_support_text_mapper.py0△芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到STRING
HIGHtests/ops/mapper/test_extract_keyword_mapper.py0△芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到STRING
HIGHtests/ops/mapper/test_extract_entity_relation_mapper.py0△芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到STRING
HIGHtests/ops/mapper/test_calibrate_response_mapper.py0# 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖STRING
HIGHtests/ops/mapper/test_calibrate_query_mapper.py0# 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖STRING
HIGHtests/ops/mapper/test_calibrate_qa_mapper.py0# 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖STRING
HIGHdocs/DeveloperGuide_ZH.md0initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text lengSTRING
HIGHdocs/DeveloperGuide.md0initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text lengSTRING
HIGHdata_juicer/ops/filter/text_length_filter.py0initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text lengSTRING
HIGHdata_juicer/format/json_formatter.py0initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified STRING
HIGHdata_juicer/format/parquet_formatter.py0initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified STRING
HIGHdata_juicer/format/csv_formatter.py0initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified STRING
HIGHdata_juicer/ops/base_op.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGHdata_juicer/ops/deduplicator/video_deduplicator.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGH…icer/ops/deduplicator/document_minhash_deduplicator.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGHdata_juicer/ops/deduplicator/document_deduplicator.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGHdata_juicer/ops/deduplicator/image_deduplicator.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGH…icer/ops/deduplicator/document_simhash_deduplicator.py0for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when trSTRING
HIGH…_juicer/ops/mapper/punctuation_normalization_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/mapper/remove_bibliography_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/mapper/clean_copyright_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/mapper/clean_html_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGH…a_juicer/ops/mapper/whitespace_normalization_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/mapper/expand_macro_mapper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/grouper/naive_grouper.py0initialization method. :param args: extra args :param kwargs: extra argsSTRING
HIGHdata_juicer/ops/mapper/clean_ip_mapper.py0initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement stSTRING
HIGHdata_juicer/ops/mapper/clean_links_mapper.py0initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement stSTRING
HIGHdata_juicer/ops/mapper/clean_email_mapper.py0initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement stSTRING
HIGHdata_juicer/ops/mapper/query_intent_detection_mapper.py0initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: transSTRING
HIGHdata_juicer/ops/mapper/query_topic_detection_mapper.py0initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: transSTRING
HIGH…_juicer/ops/mapper/query_sentiment_detection_mapper.py0initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: transSTRING
HIGH…_juicer/ops/mapper/video_captioning_from_vlm_mapper.py0:param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $nSTRING
HIGH…uicer/ops/mapper/video_captioning_from_video_mapper.py0:param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $nSTRING
HIGH…icer/ops/mapper/video_captioning_from_frames_mapper.py0:param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $nSTRING
HIGHdemos/process_code_data/app.py0# refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed dataSTRING
HIGHdemos/process_sci_data/app.py0# refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed dataSTRING
HIGHdemos/process_cft_zh_data/app.py0# refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed dataSTRING
Self-Referential Comments68 hits · 225 pts
SeverityFileLineSnippetContext
MEDIUMtools/check_s3_integration.py759 # Create a small test datasetCOMMENT
MEDIUMtools/check_s3_integration.py854 # Create a small test datasetCOMMENT
MEDIUMtools/humanops/label_studio_service.py280 # Create the file if it doesn't existCOMMENT
MEDIUMtools/humanops/label_studio_service.py343 # Create a named volume for Label Studio dataCOMMENT
MEDIUMtools/humanops/label_studio_service.py823 # Create a test projectCOMMENT
MEDIUMtools/humanops/label_studio_service.py842 # Create a sample task using import_tasks instead of create_taskSTRING
MEDIUMtests/tools/test_process_data.py15 # Create a temporary file for loggingCOMMENT
MEDIUMtests/core/executor/test_event_logging_mixin.py532 # Create a mock executor with disabled loggingCOMMENT
MEDIUMtests/core/executor/test_ray_executor_partitioned.py129 # Create a simple config without loading from fileCOMMENT
MEDIUMtests/core/executor/test_ray_executor_partitioned.py301 # Create a simple config without loading from fileCOMMENT
MEDIUMtests/core/data/test_ray_dataset.py80 # Create a PyArrow table similar to what would be passed to the functionCOMMENT
MEDIUMtests/core/data/test_ray_dataset.py134 _text = "I have a very long text. " * LONG_TEXT_MULTIPLIER # Create a long text to test large JSON objectsCODE
MEDIUMtests/core/data/test_ray_dataset.py136 # Create a temporary JSONL fileCOMMENT
MEDIUMtests/core/data/test_load_strategy.py525 # Create a proper Dataset object for the mock to returnCOMMENT
MEDIUMtests/core/data/test_load_strategy.py566 # Create a proper Dataset object for the mock to returnCOMMENT
MEDIUMtests/config/test_config.py941 # Create a temporary config fileCOMMENT
MEDIUMtests/config/test_config.py977 # Create a temporary config fileCOMMENT
MEDIUMtests/config/test_config.py1014 # Create a temporary config file with invalid work_dirCOMMENT
MEDIUMtests/config/test_config.py1043 # Create a temporary config fileCOMMENT
MEDIUMtests/utils/test_model_utils.py161 # Create a mock vllm moduleCOMMENT
MEDIUMtests/utils/test_datasets_json_compat.py113 # Create a JSONL file with big integerCOMMENT
MEDIUMtests/utils/test_fingerprint_utils.py222 # Create an object that dill cannot serializeCOMMENT
MEDIUMtests/utils/test_ckpt_utils.py416 # Create a mock checkpoint fileCOMMENT
MEDIUMtests/utils/test_ckpt_utils.py495 # Create a corrupted checkpoint fileCOMMENT
MEDIUMtests/utils/test_ckpt_utils.py602 # Create a file that starts with checkpoint_op_ and ends withCOMMENT
MEDIUMtests/utils/test_ckpt_utils.py679 # Create a mock checkpoint file so path existsCOMMENT
MEDIUMtests/utils/test_lazy_loader.py305 # Create an empty uv.lockCOMMENT
MEDIUMtests/utils/test_lazy_loader.py320 # Create an invalid uv.lockCOMMENT
MEDIUMtests/utils/test_constant.py22 # Create a temporary jsonl file for testingCOMMENT
MEDIUMtests/utils/test_constant.py35 # Create a temporary config with the test dataset pathCOMMENT
MEDIUMtests/format/test_load.py25 # Create a test file with complex extensionCOMMENT
MEDIUMtests/format/test_load.py29 # Create a directory structure for relative path testingCOMMENT
MEDIUMtests/format/test_load.py33 # Create a test JSONL file in the relative path directoryCOMMENT
MEDIUMtests/format/test_load.py85 # Create a temp directory with the name that matches the errorCOMMENT
MEDIUMtests/format/test_load.py89 # Create the file with the name from the errorCOMMENT
MEDIUMtests/format/test_text_formatter.py129 # Create a minimal .docx in memoryCOMMENT
MEDIUMtests/ops/test_op_env.py780 # Create a temporary Python fileSTRING
MEDIUMtests/ops/test_op_fusion_mapper_bug.py35 # Create a NEW dict instead of mutating in-placeCOMMENT
MEDIUMtests/ops/mapper/test_python_lambda_mapper.py46 PythonLambdaMapper("lambda x, y: {'sum': x + y}") # Creating a lambda accepts two argumentsCODE
MEDIUMtests/ops/mapper/test_image_mmpose_mapper.py24 # Create a temporary file for loggingCOMMENT
MEDIUM…/ops/mapper/test_video_atomic_action_segment_mapper.py76 # Create a speed profile with clear minimaCOMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py95 # Create a dictionary version of samples (column-oriented)COMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py170 # Create a sampleCOMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py173 # Create an annotation with preference for the left option (answer1)COMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py189 # Create a sampleCOMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py192 # Create an annotation with preference for the right option (answer2)COMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py230 # Create a sample with custom keysCOMMENT
MEDIUM…/annotation/test_human_preference_annotation_mapper.py238 # Create a dictionary version of the sampleCOMMENT
MEDIUMtests/ops/mapper/annotation/test_annotation_mapper.py302 # Create a mapper with a very short timeoutCOMMENT
MEDIUMtests/ops/mapper/annotation/test_annotation_mapper.py305 # Create a task but don't add annotationsCOMMENT
MEDIUMtests/ops/mapper/annotation/test_annotation_mapper.py89 # Create a dictionary version of samples (column-oriented)COMMENT
MEDIUMtests/ops/mapper/annotation/test_annotation_mapper.py421 # Create a dictionary version of samples (column-oriented)COMMENT
MEDIUMtests/ops/mapper/annotation/test_annotation_mapper.py435 self.mock_ls_client = MagicMock() # Create a mock clientCODE
MEDIUMdata_juicer/tools/__init__.py5# This file is used for package installation. Script of tools will beCOMMENT
MEDIUMdata_juicer/core/executor/ray_executor_partitioned.py529 # Create a RayDataset wrapper for this partitionCOMMENT
MEDIUMdata_juicer/core/data/load_strategy.py96 # Create the lookup keyCOMMENT
MEDIUMdata_juicer/utils/video_utils.py464 # Create a named temporary file that will be automatically deleted when closedCOMMENT
MEDIUMdata_juicer/utils/lazy_loader.py303 # Create a temporary directory for cloningCOMMENT
MEDIUMdata_juicer/utils/registry.py22 """This class is used to register some modules to registry by a repoSTRING
MEDIUMdata_juicer/utils/constant.py270 # Create a stream reader for the file and decode theCOMMENT
8 more matches not shown…
Redundant / Tautological Comments74 hits · 113 pts
SeverityFileLineSnippetContext
LOWtools/check_s3_integration.py140 # Check if credentials are availableCOMMENT
LOWtools/check_s3_integration.py324 # Check if credentials are availableCOMMENT
LOWtools/check_s3_integration.py507 # Check if credentials are availableCOMMENT
LOWtools/check_s3_integration.py690 # Check if credentials are availableCOMMENT
LOWtools/check_s3_integration.py777 # Check if credentials are availableCOMMENT
LOWtools/check_s3_integration.py870 # Check if credentials are availableCOMMENT
LOWtools/check_ray_cluster.py43 # Check if head node participates in workloadCOMMENT
LOWtools/check_ray_cluster.py65 # Check if this is the head node by comparing node IDsCOMMENT
LOWtools/generate_smtp_cert.py157 # Check if OpenSSL is installedCOMMENT
LOW…m_eval/inception_metrics/video_metrics/metric_utils.py306 # Check if the file exists (all processes must agree).COMMENT
LOWtools/humanops/label_studio_service.py294 # Set permissions to 777 (rwxrwxrwx) to ensure the container can write to itCOMMENT
LOWtools/humanops/label_studio_service.py347 # Check if volume exists, if not create itCOMMENT
LOWtools/humanops/label_studio_service.py385 # Set user to root for ensuring write permissions inside containerCOMMENT
LOWtools/humanops/label_studio_service.py414 # Check if there's a specific error message we can extractCOMMENT
LOWtools/humanops/label_studio_service.py426 # Check if the container was partially created and clean it upCOMMENT
LOWtools/humanops/label_studio_service.py461 # Check if container is still runningCOMMENT
LOWtools/humanops/label_studio_service.py619 # Check if container is still runningCOMMENT
LOWtools/humanops/label_studio_service.py764 # Check if process is still runningCOMMENT
LOWtools/humanops/label_studio_service.py786 # Check if the server is respondingCOMMENT
LOWtools/humanops/label_studio_service.py985 # Check if Docker is installed and runningSTRING
LOWtests/core/executor/test_pipeline_dag.py60 # Assign nodes to DAGCOMMENT
LOWtests/core/executor/test_partitioned_integration.py135 # Check if deduplicator config existsCOMMENT
LOW…/annotation/test_human_preference_annotation_mapper.py293 # Check if each predefined ID is in the sample-to-task mappingCOMMENT
LOWtests/ops/mapper/annotation/test_annotation_mapper.py368 # Check if each predefined ID is in the sample-to-task mappingCOMMENT
LOW…cs/op_doc_enhance_workflow/check_tools/check_render.py86 # Check if it's just "not available"COMMENT
LOWdata_juicer/core/exporter.py89 # Check if export_path is S3 and create storage_options if neededCOMMENT
LOWdata_juicer/core/ray_exporter.py88 # Check if export_path is S3 and create filesystem if neededCOMMENT
LOWdata_juicer/core/ray_exporter.py161 # Check if dataset is empty by calling columns() regardless of columns parameterCOMMENT
LOWdata_juicer/core/tracer/tracer.py92 # Check if sample has changedCOMMENT
LOWdata_juicer/core/tracer/ray_tracer.py87 # Check if sample has changedCOMMENT
LOWdata_juicer/core/executor/ray_executor_partitioned.py375 # Check if user provided a job_id (indicating resumption attempt)COMMENT
LOWdata_juicer/core/executor/ray_executor_partitioned.py746 # Check if the current work_dir already contains the job_idCOMMENT
LOWdata_juicer/core/executor/ray_executor_partitioned.py754 # Check if this directory has events files (indicating it's a work directory)COMMENT
LOWdata_juicer/core/executor/ray_executor_partitioned.py798 # Check if config validation passed (done during config initialization)COMMENT
LOWdata_juicer/core/executor/ray_executor_partitioned.py803 # Check if job is already completedCOMMENT
LOWdata_juicer/core/executor/event_logging_mixin.py434 # Check if DAGExecutionMixin is available and has the method to get DAG nodeCOMMENT
LOWdata_juicer/core/executor/event_logging_mixin.py1054 # Check if job has any completion eventsCOMMENT
LOWdata_juicer/core/executor/event_logging_mixin.py1063 # Check if all partitions are completed (success or failure)COMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py63 # Check if DAG execution is enabled (default: True for distributed executors, False for standalone)COMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py412 # Check if we have per-op metricsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py750 # Check if all dependencies are completedCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py877 # Check if this node was already completedCOMMENT
LOWdata_juicer/core/data/data_validator.py336 # Check if fields exist in datasetCOMMENT
LOWdata_juicer/core/data/ray_dataset.py169 # Check if dataset is empty - Ray returns None for columns() on empty datasetsCOMMENT
LOWdata_juicer/core/data/ray_dataset.py422 # Check if open_json is available (PyArrow 20.0.0+)COMMENT
LOWdata_juicer/core/data/ray_dataset.py511 # Check if open_json is available (PyArrow 20.0.0+)COMMENT
LOWdata_juicer/core/data/ray_dataset.py516 js.open_json # Check if attribute existsCODE
LOWdata_juicer/analysis/correlation_analysis.py115 # Loop over the data and create a `Text` for each "pixel".COMMENT
LOWdata_juicer/config/config.py881 # Check if this is a resumption attempt by looking for existing job directoryCOMMENT
LOWdata_juicer/config/config.py1878 # Check if {job_id} is at the end of the pathCOMMENT
LOWdata_juicer/utils/model_utils.py683 # Check if we need accelerate for device_mapCOMMENT
LOWdata_juicer/utils/model_utils.py983 # Check if we need accelerate for device_mapCOMMENT
LOWdata_juicer/utils/constant.py266 # Open the file in binary mode andCOMMENT
LOWdata_juicer/utils/common_utils.py173 # Check if version is provided when using as @deprecated (without parameters)COMMENT
LOWdata_juicer/utils/job/common.py342 # Check if processes are still runningCOMMENT
LOWdata_juicer/ops/mixins.py278 # Check if notifications are enabled for this operatorCOMMENT
LOWdata_juicer/ops/mixins.py299 # Check if notifications are enabledCOMMENT
LOWdata_juicer/ops/mixins.py305 # Check if notification handlers are initializedCOMMENT
LOWdata_juicer/ops/mixins.py330 # Check if this specific channel is enabledCOMMENT
LOWdata_juicer/ops/base_op.py125 # Check if collection is already complete (early exit for performance)COMMENT
14 more matches not shown…
Over-Commented Block104 hits · 104 pts
SeverityFileLineSnippetContext
LOWtools/converter/modeling_megatron_llama.py1# Some codes are adapted from https://pypi.org/project/transformersCOMMENT
LOWtools/converter/convert_gpt_to_transformers.py1# Some code here has been modified from:COMMENT
LOWtools/preprocess/raw_arxiv_to_jsonl.py1# Part of the code here has been modified from:COMMENT
LOWtools/preprocess/raw_stackexchange_to_jsonl.py1# Part of the code here has been modified from:COMMENT
LOWtools/mm_eval/inception_metrics/util.py401 # Doesn't look like an URL scheme so interpret it as a local filename.COMMENT
LOW…onversion/multimodal/absolute_path_to_relative_path.py1# This tool is used to convert the absolute paths or relative paths inCOMMENT
LOW…onversion/multimodal/absolute_path_to_relative_path.py21# 'images': ['coco/train2017/000000033471.jpg'],COMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_llava.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_llava.py21# - in jsonCOMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_llava.py41# "value": "The back of the bus features an advertisement."COMMENT
LOW…data_juicer_format_to_target_format/dj_to_internvid.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…odal/data_juicer_format_to_target_format/dj_to_mmc4.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…odal/data_juicer_format_to_target_format/dj_to_mmc4.py21# 'doors lock/ unlock at the same time. <|__dj__eoc|>',COMMENT
LOW…odal/data_juicer_format_to_target_format/dj_to_mmc4.py41# 'time.',COMMENT
LOW…odal/data_juicer_format_to_target_format/dj_to_mmc4.py61# {'face_detections': None,COMMENT
LOW…l/data_juicer_format_to_target_format/dj_to_wavcaps.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…l/data_juicer_format_to_target_format/dj_to_wavcaps.py21# 'tags': '' }}COMMENT
LOW…l/data_juicer_format_to_target_format/dj_to_wavcaps.py41# { 'num_captions_per_audio': 1,COMMENT
LOW…_juicer_format_to_target_format/dj_to_video_chatgpt.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…_juicer_format_to_target_format/dj_to_video_chatgpt.py21# (exploring spatial, temporal, relationships, and reasoning concepts);COMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_youku.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_youku.py21# {'videos': ['videos/retrieval/14111B1211bA1F31-E-2YB57--C518FCEBC553abJYFa5541a31C8a57522AYJbYF4aTdfofa112.mp4'], # nCOMMENT
LOW…dal/data_juicer_format_to_target_format/dj_to_youku.py41#COMMENT
LOW…al/data_juicer_format_to_target_format/dj_to_msrvtt.py1# This tool is used to convert multimodal dataset in Data-Juicer format to aCOMMENT
LOW…al/data_juicer_format_to_target_format/dj_to_msrvtt.py21# - in jsonlCOMMENT
LOW…l/source_format_to_data_juicer_format/wavcaps_to_dj.py1# This tool is used to convert multimodal dataset in WavCaps format to a targetCOMMENT
LOW…l/source_format_to_data_juicer_format/wavcaps_to_dj.py21# 'caption': 'Someone is ringing a bell.',COMMENT
LOW…l/source_format_to_data_juicer_format/wavcaps_to_dj.py41# 'caption': 'An airplane is landing.',COMMENT
LOW…al/source_format_to_data_juicer_format/msrvtt_to_dj.py1# This tool is used to convert multimodal dataset in MSR-VTT format to aCOMMENT
LOW…al/source_format_to_data_juicer_format/msrvtt_to_dj.py21# - two new fields are added:COMMENT
LOW…source_format_to_data_juicer_format/internvid_to_dj.py1# This tool is used to convert multimodal dataset in InternVid format to aCOMMENT
LOW…source_format_to_data_juicer_format/internvid_to_dj.py21# - videos: video paths list, including cut videos according to their timestamps # noqa: E501COMMENT
LOW…dal/source_format_to_data_juicer_format/llava_to_dj.py1# This tool is used to convert multimodal dataset in LLaVA format to a targetCOMMENT
LOW…dal/source_format_to_data_juicer_format/llava_to_dj.py21# "from": "human",COMMENT
LOW…dal/source_format_to_data_juicer_format/llava_to_dj.py41# Corresponding Data-Juicer format:COMMENT
LOW…ce_format_to_data_juicer_format/video_chatgpt_to_dj.py1# This tool is used to convert multimodal dataset in Video_Chatgpt format to aCOMMENT
LOW…odal/source_format_to_data_juicer_format/mmc4_to_dj.py1# This tool is used to convert multimodal dataset in MMC4 format to a targetCOMMENT
LOW…odal/source_format_to_data_juicer_format/mmc4_to_dj.py21# 0.3234919607639313,COMMENT
LOW…odal/source_format_to_data_juicer_format/mmc4_to_dj.py41# {'text': 'When you lock the door using the lock tab on the driver’s door, 'COMMENT
LOW…odal/source_format_to_data_juicer_format/mmc4_to_dj.py61# 0.27694183588027954],COMMENT
LOW…dal/source_format_to_data_juicer_format/youku_to_dj.py1# This tool is used to convert multimodal dataset in Youku-mPLUG format to aCOMMENT
LOW…dal/source_format_to_data_juicer_format/youku_to_dj.py21#COMMENT
LOW…dal/source_format_to_data_juicer_format/youku_to_dj.py41#COMMENT
LOW…cer_format_to_target_format/dj_to_ms_swift_sharegpt.py1# This tool is used to convert dataset in Data-Juicer format to aCOMMENT
LOW…cer_format_to_target_format/dj_to_ms_swift_sharegpt.py21# [COMMENT
LOW…/data_juicer_format_to_target_format/dj_to_messages.py1# This tool is used to convert dataset in Data-Juicer format to aCOMMENT
LOW…/data_juicer_format_to_target_format/dj_to_messages.py21# ...COMMENT
LOW…/data_juicer_format_to_target_format/dj_to_messages.py41# },COMMENT
LOW…ormat_to_target_format/dj_to_llama_factory_sharegpt.py1# This tool is used to convert dataset in Data-Juicer format to aCOMMENT
LOW…ormat_to_target_format/dj_to_llama_factory_sharegpt.py21# ...COMMENT
LOW…ormat_to_target_format/dj_to_llama_factory_sharegpt.py41# },COMMENT
LOW…og/data_juicer_format_to_target_format/dj_to_alpaca.py1# This tool is used to convert dataset in Data-Juicer format to aCOMMENT
LOW…og/data_juicer_format_to_target_format/dj_to_alpaca.py21# {COMMENT
LOW…og/source_format_to_data_juicer_format/alpaca_to_dj.py1# This tool is used to convert dataset in Alpaca format to aCOMMENT
LOW…og/source_format_to_data_juicer_format/alpaca_to_dj.py21# {COMMENT
LOW…ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py1# This tool is used to convert dataset in ModelScope-Swift ShareGPT format to aCOMMENT
LOW…ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py21#COMMENT
LOW…/source_format_to_data_juicer_format/messages_to_dj.py1# This tool is used to convert dataset in ModelScope-Swift Messages format to aCOMMENT
LOW…/source_format_to_data_juicer_format/messages_to_dj.py21# },COMMENT
LOW…/source_format_to_data_juicer_format/messages_to_dj.py41# {COMMENT
44 more matches not shown…
AI Structural Patterns102 hits · 100 pts
SeverityFileLineSnippetContext
LOWtools/converter/modeling_megatron_llama.py58CODE
LOWtools/converter/modeling_megatron_llama.py550CODE
LOWtools/converter/modeling_megatron_llama.py716CODE
LOWtools/converter/modeling_megatron_llama.py876CODE
LOW…s/mm_eval/inception_metrics/calc_metrics_for_videos.py20CODE
LOW…source_format_to_data_juicer_format/internvid_to_dj.py54CODE
LOW…dal/source_format_to_data_juicer_format/llava_to_dj.py73CODE
LOWscripts/dlc/partition_data_dlc.py20CODE
LOWdata_juicer/core/exporter.py16CODE
LOWdata_juicer/core/data/ray_dataset.py493CODE
LOWdata_juicer/utils/model_utils.py1642CODE
LOWdata_juicer/utils/llm_semantic_ops.py309CODE
LOWdata_juicer/download/wikipedia.py722CODE
LOW…r/ops/pipeline/llm_inference_with_ray_vllm_pipeline.py26CODE
LOW…a_juicer/ops/aggregator/entity_attribute_aggregator.py63CODE
LOW…er/ops/aggregator/most_relevant_entities_aggregator.py49CODE
LOW…er/ops/aggregator/most_relevant_entities_aggregator.py129CODE
LOWdata_juicer/ops/aggregator/meta_tags_aggregator.py91CODE
LOWdata_juicer/ops/aggregator/nested_aggregator.py58CODE
LOWdata_juicer/ops/mapper/extract_support_text_mapper.py51CODE
LOWdata_juicer/ops/mapper/pii_redaction_mapper.py107CODE
LOWdata_juicer/ops/mapper/image_tagging_vlm_mapper.py61CODE
LOW…juicer/ops/mapper/dialog_sentiment_detection_mapper.py66CODE
LOW…a_juicer/ops/mapper/video_trajectory_overlay_mapper.py46CODE
LOWdata_juicer/ops/mapper/sentence_augmentation_mapper.py33CODE
LOWdata_juicer/ops/mapper/image_sam_3d_body_mapper.py30CODE
LOWdata_juicer/ops/mapper/relation_identity_mapper.py48CODE
LOW…ta_juicer/ops/mapper/dialog_intent_detection_mapper.py71CODE
LOW…juicer/ops/mapper/dialog_sentiment_intensity_mapper.py79CODE
LOWdata_juicer/ops/mapper/video_split_by_scene_mapper.py60CODE
LOWdata_juicer/ops/mapper/agent_insight_llm_mapper.py198CODE
LOWdata_juicer/ops/mapper/pii_llm_suspect_mapper.py346CODE
LOW…ta_juicer/ops/mapper/extract_entity_relation_mapper.py162CODE
LOW…r/ops/mapper/video_hand_reconstruction_hawor_mapper.py29CODE
LOWdata_juicer/ops/mapper/pair_preference_mapper.py41CODE
LOW…/ops/mapper/video_whole_body_pose_estimation_mapper.py31CODE
LOWdata_juicer/ops/mapper/llm_extract_mapper.py40CODE
LOWdata_juicer/ops/mapper/image_diffusion_mapper.py43CODE
LOWdata_juicer/ops/mapper/agent_bad_case_signal_mapper.py126CODE
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py29CODE
LOWdata_juicer/ops/mapper/s3_upload_file_mapper.py33CODE
LOWdata_juicer/ops/mapper/nlpcda_zh_mapper.py32CODE
LOW…_juicer/ops/mapper/video_hand_reconstruction_mapper.py38CODE
LOWdata_juicer/ops/mapper/dialog_topic_detection_mapper.py69CODE
LOWdata_juicer/ops/mapper/vggt_mapper.py43CODE
LOW…s/mapper/video_camera_calibration_droidcalib_mapper.py35CODE
LOW…a_juicer/ops/mapper/extract_entity_attribute_mapper.py54CODE
LOWdata_juicer/ops/mapper/extract_nickname_mapper.py61CODE
LOWdata_juicer/ops/mapper/s3_download_file_mapper.py35CODE
LOW…_juicer/ops/mapper/video_captioning_from_vlm_mapper.py58CODE
LOWdata_juicer/ops/mapper/image_captioning_mapper.py46CODE
LOW…_juicer/ops/mapper/generate_qa_from_examples_mapper.py57CODE
LOW…uicer/ops/mapper/video_captioning_from_video_mapper.py54CODE
LOWdata_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py37CODE
LOW…/ops/mapper/video_captioning_from_summarizer_mapper.py26CODE
LOWdata_juicer/ops/mapper/calibrate_qa_mapper.py40CODE
LOWdata_juicer/ops/mapper/text_tagging_by_prompt_mapper.py67CODE
LOWdata_juicer/ops/mapper/nlpaug_en_mapper.py32CODE
LOWdata_juicer/ops/mapper/video_clip_reassembly_mapper.py45CODE
LOW…icer/ops/mapper/video_captioning_from_frames_mapper.py56CODE
42 more matches not shown…
Verbosity Indicators33 hits · 62 pts
SeverityFileLineSnippetContext
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py46 # Step 1: Extract frames from videosCOMMENT
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py58 # Step 2: Run camera calibrationCOMMENT
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py55 # Step 1: Extract frames from videosCOMMENT
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py68 # Step 2: Run camera calibrationCOMMENT
LOW…ts/ops/mapper/test_video_camera_pose_megasam_mapper.py33 # Step 1: Extract real frames from videosCOMMENT
LOW…ts/ops/mapper/test_video_camera_pose_megasam_mapper.py47 # Step 2: Add dummy camera calibration data matching real frame dimsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py543 # Step 1: Validate event logger availabilityCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py548 # Step 2: Load DAG events and execution planCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py554 # Step 3: Reconstruct node states from plan and eventsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py558 # Step 4: Calculate statisticsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py561 # Step 5: Determine ready nodesCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py564 # Step 6: Determine resumption strategyCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py651 # Step 2: Check if this fits in available memoryCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py660 # Step 3: Ensure sufficient parallelism for large datasetsCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py674 # Step 4: Adjust for data skewCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py681 # Step 5: Apply final boundsCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py635 # Step 1: Calculate dynamic target based on available memoryCOMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py158 # Step 1: Decode all frames and record their dimensionsCOMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py172 # Step 2: Check if all frames share the same resolution (typical for a single clip)COMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py241 # Step 3: Write results to tag_dictCOMMENT
LOW…ps/mapper/video_camera_calibration_deepcalib_mapper.py101 # Step 1: Decode and preprocess all frames, record original dimensionsCOMMENT
LOW…ps/mapper/video_camera_calibration_deepcalib_mapper.py119 # Step 2: Batch inferenceCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py286 # Step 1: Convert to row-oriented format (list of sample dictionaries)COMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py289 # Step 2: Retrieve or generate unique IDs for each sampleCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py296 # Step 3: Process samples in batchesCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py331 # Step 5: Convert back to column-oriented format efficientlyCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py317 # Step 4: Update samples with annotation resultsCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml37 # Step 1: Download videos from S3 to localCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml52 # Step 2: Video filtering - durationCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml59 # Step 3: Video filtering - resolutionCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml69 # Step 4: Split videos by durationCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml79 # Step 5: Adjust video aspect ratioCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml87 # Step 6: Upload processed videos to S3COMMENT
Structural Annotation Overuse33 hits · 62 pts
SeverityFileLineSnippetContext
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py46 # Step 1: Extract frames from videosCOMMENT
LOW…pper/test_video_camera_calibration_deepcalib_mapper.py58 # Step 2: Run camera calibrationCOMMENT
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py55 # Step 1: Extract frames from videosCOMMENT
LOW…ps/mapper/test_video_camera_calibration_moge_mapper.py68 # Step 2: Run camera calibrationCOMMENT
LOW…ts/ops/mapper/test_video_camera_pose_megasam_mapper.py33 # Step 1: Extract real frames from videosCOMMENT
LOW…ts/ops/mapper/test_video_camera_pose_megasam_mapper.py47 # Step 2: Add dummy camera calibration data matching real frame dimsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py543 # Step 1: Validate event logger availabilityCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py548 # Step 2: Load DAG events and execution planCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py554 # Step 3: Reconstruct node states from plan and eventsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py558 # Step 4: Calculate statisticsCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py561 # Step 5: Determine ready nodesCOMMENT
LOWdata_juicer/core/executor/dag_execution_mixin.py564 # Step 6: Determine resumption strategyCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py651 # Step 2: Check if this fits in available memoryCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py660 # Step 3: Ensure sufficient parallelism for large datasetsCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py674 # Step 4: Adjust for data skewCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py681 # Step 5: Apply final boundsCOMMENT
LOWdata_juicer/core/executor/partition_size_optimizer.py635 # Step 1: Calculate dynamic target based on available memoryCOMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py158 # Step 1: Decode all frames and record their dimensionsCOMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py172 # Step 2: Check if all frames share the same resolution (typical for a single clip)COMMENT
LOW…cer/ops/mapper/video_camera_calibration_moge_mapper.py241 # Step 3: Write results to tag_dictCOMMENT
LOW…ps/mapper/video_camera_calibration_deepcalib_mapper.py101 # Step 1: Decode and preprocess all frames, record original dimensionsCOMMENT
LOW…ps/mapper/video_camera_calibration_deepcalib_mapper.py119 # Step 2: Batch inferenceCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py286 # Step 1: Convert to row-oriented format (list of sample dictionaries)COMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py289 # Step 2: Retrieve or generate unique IDs for each sampleCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py296 # Step 3: Process samples in batchesCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py331 # Step 5: Convert back to column-oriented format efficientlyCOMMENT
LOWdata_juicer/ops/mapper/annotation/annotation_mapper.py317 # Step 4: Update samples with annotation resultsCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml37 # Step 1: Download videos from S3 to localCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml52 # Step 2: Video filtering - durationCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml59 # Step 3: Video filtering - resolutionCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml69 # Step 4: Split videos by durationCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml79 # Step 5: Adjust video aspect ratioCOMMENT
LOW…s_video_on_ray/configs/s3_video_processing_config.yaml87 # Step 6: Upload processed videos to S3COMMENT
Modern AI Meta-Vocabulary11 hits · 28 pts
SeverityFileLineSnippetContext
MEDIUMREADME.md167- **[data-juicer-agents](https://github.com/datajuicer/data-juicer-agents)** — DJ Copilot and agentic workflowsCODE
MEDIUMtests/utils/test_model_utils.py114 # Test with processor for embedding modelCOMMENT
MEDIUMtests/utils/test_model_utils.py196 # Test embedding modelCOMMENT
MEDIUMdata_juicer/config/config_all.yaml872 hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name.CODE
MEDIUMdata_juicer/config/config_all.yaml883 hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name.CODE
MEDIUMdata_juicer/config/config_all.yaml908 hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name.CODE
MEDIUMdata_juicer/config/config_all.yaml988 api_or_hf_model: text-embedding-v4 # API or huggingface embedding model nameCODE
MEDIUM…ta_juicer/ops/mapper/extract_entity_relation_mapper.py1# This OP is modified from light RAGCOMMENT
MEDIUMdata_juicer/ops/mapper/tool_success_tagger_mapper.py5# Configurable success/error patterns for multi-agent and multi-language.COMMENT
MEDIUMdata_juicer/ops/mapper/extract_keyword_mapper.py33 # This prompt is modified from light RAGCOMMENT
MEDIUMdata_juicer/ops/mapper/agent_dialog_normalize_mapper.py5# dialog/text ops. Supports multi-platform, multi-agent tool formats.COMMENT
Modern Structural Boilerplate27 hits · 26 pts
SeverityFileLineSnippetContext
LOWtools/evaluator/gpt_eval/gpt_evaluator.py16logger = logging.getLogger(__name__)CODE
LOWtools/mm_eval/inception_metrics/util.py120def set_cache_dir(path: str) -> None:CODE
LOWdata_juicer/core/__init__.py10__all__ = [CODE
LOWdata_juicer/core/executor/__init__.py7__all__ = ["ExecutorBase", "ExecutorFactory", "DefaultExecutor", "RayExecutor", "PartitionedRayExecutor"]CODE
LOWdata_juicer/core/executor/dag_execution_mixin.py641 def _update_node_states_from_events(self, node_states: Dict[str, Dict[str, Any]], dag_events: List[Any]) -> None:CODE
LOWdata_juicer/core/data/__init__.py8__all__ = ["DJDataset", "NestedDataset", "wrap_func_with_nested_access", "add_same_content_to_new_column"]CODE
LOWdata_juicer/analysis/__init__.py6__all__ = [CODE
LOWdata_juicer/config/__init__.py15__all__ = [CODE
LOWdata_juicer/utils/llm_semantic_ops.py18logger = logging.getLogger(__name__)CODE
LOWdata_juicer/utils/job/__init__.py18__all__ = [CODE
LOWdata_juicer/format/__init__.py9__all__ = [CODE
LOWdata_juicer/ops/__init__.py41__all__ = [CODE
LOWdata_juicer/ops/pipeline/__init__.py5__all__ = ["LLMRayVLLMEnginePipeline", "RayRepartitionPipeline", "VLMRayVLLMEnginePipeline"]CODE
LOWdata_juicer/ops/selector/__init__.py7__all__ = [CODE
LOWdata_juicer/ops/aggregator/__init__.py6__all__ = ["NestedAggregator", "MetaTagsAggregator", "EntityAttributeAggregator", "MostRelevantEntitiesAggregator"]CODE
LOWdata_juicer/ops/mapper/sentence_augmentation_mapper.py10logger = logging.getLogger(__name__)CODE
LOWdata_juicer/ops/mapper/agent_bad_case_signal_mapper.py20logger = logging.getLogger(__name__)CODE
LOWdata_juicer/ops/mapper/__init__.py143__all__ = [CODE
LOWdata_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py17logger = logging.getLogger(__name__)CODE
LOWdata_juicer/ops/grouper/__init__.py5__all__ = ["KeyValueGrouper", "NaiveGrouper", "NaiveReverseGrouper"]CODE
LOWdata_juicer/ops/common/__init__.py14__all__ = [CODE
LOWdata_juicer/ops/filter/text_pair_similarity_filter.py14logger = logging.getLogger(__name__)CODE
LOW…/ops/filter/instruction_following_difficulty_filter.py11logger = logging.getLogger(__name__)CODE
LOWdata_juicer/ops/filter/llm_perplexity_filter.py15logger = logging.getLogger(__name__)CODE
LOWdata_juicer/ops/filter/__init__.py61__all__ = [CODE
LOWdata_juicer/ops/deduplicator/__init__.py20__all__ = [CODE
LOWdemos/agent/scripts/generate_bad_case_report.py87def set_report_locale(lang: str) -> None:CODE
Docstring Block Structure5 hits · 25 pts
SeverityFileLineSnippetContext
HIGHdata_juicer/core/data/dj_dataset.py59Get values from a specific column/field, optionally limited to first k rows. Args: column: Name of STRING
HIGHdata_juicer/core/data/dj_dataset.py228Get column values from HuggingFace dataset. Args: column: Name of the column to retrieve STRING
HIGHdata_juicer/core/data/data_validator.py292 Convert field type specifications to actual Python type objects. Args: field_types: DictioSTRING
HIGHdata_juicer/core/data/ray_dataset.py132Get column values from Ray dataset. Args: column: Name of the column to retrieve k: OptSTRING
HIGHdata_juicer/core/data/schema.py41Map HuggingFace feature type to Python type. Recursively maps nested types (e.g., List[str], Dict[str, int]). STRING
Fake / Example Data15 hits · 21 pts
SeverityFileLineSnippetContext
LOWlabel_studio_localhost_connection.json4 "username": "admin@example.com",CODE
LOWtools/humanops/label_studio_service.py947 args.username = "admin@example.com"STRING
LOWtests/ops/test_mixins.py479 dingtalk={"access_token": "fake_token"},CODE
LOWtests/ops/test_mixins.py496 "access_token": "fake_token",CODE
LOWtests/ops/test_mixins.py516 dingtalk={"access_token": "fake_token"},CODE
LOWtests/ops/test_mixins.py531 dingtalk={"access_token": "fake_token"},CODE
LOWtests/ops/mapper/test_clean_html_mapper.py35 '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.'CODE
LOWtests/ops/mapper/test_clean_html_mapper.py35 '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.'CODE
LOWtests/ops/mapper/test_clean_html_mapper.py41 'Welcome to My WebsiteLorem ipsum dolor sit amet, 'CODE
LOWtests/ops/mapper/test_clean_html_mapper.py41 'Welcome to My WebsiteLorem ipsum dolor sit amet, 'CODE
LOWdocs/operators/mapper/clean_html_mapper.md28<div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafaCODE
LOWdocs/operators/mapper/clean_html_mapper.md28<div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafaCODE
LOWdocs/operators/mapper/clean_html_mapper.md35*ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All RiCOMMENT
LOWdocs/operators/mapper/clean_html_mapper.md35*ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All RiCOMMENT
LOWdata_juicer/config/config_all.yaml1278 recipients: ["user@example.com"]CODE
AI Slop Vocabulary8 hits · 12 pts
SeverityFileLineSnippetContext
LOWdata_juicer/core/executor/ray_executor_partitioned.py606 # No post-convergence operations, just return the merged resultCOMMENT
MEDIUMdata_juicer/core/executor/event_logging_mixin.py250 """Generate a comprehensive status report."""STRING
MEDIUMdata_juicer/core/executor/partition_size_optimizer.py450 # Calculate percentile-based memory estimates (p90 is more robust than mean)COMMENT
MEDIUMdata_juicer/core/executor/partition_size_optimizer.py776 """Get comprehensive partition recommendations."""STRING
MEDIUMdata_juicer/config/config.py1493 # If global_parser is available, use it for robust parsingCOMMENT
MEDIUMdata_juicer/utils/job/snapshot.py401 """Convert snapshot to JSON-serializable dictionary with comprehensive progress tracking."""STRING
LOWdata_juicer/ops/op_env.py505 # and if any of them is arbitrary equality, just use itCOMMENT
MEDIUMdemos/partition_and_checkpoint/run_demo.py264 """Run the comprehensive demo."""STRING
Cross-Language Confusion1 hit · 5 pts
SeverityFileLineSnippetContext
HIGHdata_juicer/core/data/ray_dataset.py475 elif not schema.equals(batch.schema):CODE
Overly Generic Function Names3 hits · 4 pts
SeverityFileLineSnippetContext
LOWtests/ops/mapper/test_python_file_mapper.py13 "def process_data(sample):\n"CODE
LOWtests/ops/mapper/test_python_file_mapper.py25 "def process_data(samples):\n"CODE
LOWtests/ops/mapper/test_python_file_mapper.py38 "def process_data(sample):\n"CODE
Slop Phrases1 hit · 3 pts
SeverityFileLineSnippetContext
MEDIUMdata_juicer/tools/hpo/objects.py45 # feel free to customize the quality scorer, via the following argsCOMMENT