Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
This report presents the forensic synthetic code analysis of datajuicer/data-juicer, a Python project with 6,717 GitHub stars. SynthScan v2.0 examined 211,400 lines of code across 1177 source files, recording 3507 pattern matches distributed across 20 syntactic categories. The overall adjusted score of 22.3 places this repository in the Moderate AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 3507 distinct pattern matches across 20 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 38 | def register_objects_from_init(directory: str): | CODE |
| LOW | tools/check_ray_cluster.py | 6 | def get_ray_cluster_resources(): | CODE |
| LOW | tools/evaluator/evaluator.py | 221 | def megatron_checkpoint_exists(self, iteration): | CODE |
| LOW | tools/converter/modeling_megatron_llama.py | 419 | def _set_gradient_checkpointing(self, module, value=False): | STRING |
| LOW | tools/converter/modeling_megatron_llama.py | 526 | def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length): | STRING |
| LOW | tools/converter/modeling_megatron_llama.py | 803 | def prepare_inputs_for_generation( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 60 | def add_transformers_checkpoint_args(parser): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 148 | def megatron_to_transformers_fix_query_key_value_ordering( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 186 | def transformers_to_megatron_fix_query_key_value_ordering( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 224 | def merge_transformers_sharded_states(path, num_checkpoints): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 240 | def get_megatron_sharded_states(args, tp_size, pp_size, pp_rank): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 262 | def get_element_from_dict_by_path(d, path): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 279 | def convert_checkpoint_from_megatron_to_transformers(args): | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 306 | def get_module_dir_by_obj_name(obj_name: str) -> str: | CODE |
| LOW⚡ | tools/mm_eval/inception_metrics/util.py | 317 | def get_top_level_function_name(obj: Any) -> str: | CODE |
| LOW⚡ | tools/mm_eval/inception_metrics/util.py | 329 | def list_dir_recursively_with_ignore(dir_path: str, ignores: List[str] = None, add_base_to_relative: bool = False) -> Li | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 362 | def copy_files_and_create_dirs(files: List[Tuple[str, str]]) -> None: | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 78 | def get_feature_detector_name(url): | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 269 | def compute_feature_stats_for_dataset( | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 373 | def compute_feature_stats_for_generator( | CODE |
| LOW⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 484 | def rewrite_opts_for_gen_dataset(opts): | CODE |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 76 | def convert_absolute_path_to_relative_path( | CODE |
| LOW | tools/fmt_conversion/multimodal/utils.py | 29 | def check_args_load_to_dj_data( | CODE |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 71 | def dj_to_llama_factory_sharegpt( | CODE |
| LOW | …_to_data_juicer_format/llama_factory_sharegpt_to_dj.py | 72 | def llama_factory_sharegpt_to_dj( | CODE |
| LOW | tools/humanops/label_studio_service.py | 257 | def start_label_studio_container( | CODE |
| LOW | tools/humanops/label_studio_service.py | 631 | def get_container_error_message(container_name): | CODE |
| LOW | tools/humanops/label_studio_service.py | 1063 | def start_label_studio_with_docker(args, api_token): | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 252 | def _compact_class_or_module_name(name: str) -> str: | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 316 | def get_op_list_from_code_for_formatter(): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 492 | def generate_op_table_section(op_type, op_record_list, reference_op_record_dict): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 540 | def get_op_desc_in_en_zh_batched(descs): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 576 | def parse_op_record_from_current_doc(): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 619 | def check_and_update_op_record(old_op_record_list, new_op_record_list): | CODE |
| LOW | tests/tools/test_mcp_server.py | 19 | def test_granular_ops_mode_default_transport(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 31 | def test_recipe_flow_mode_default_transport(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 65 | def test_custom_port_passed_to_create(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 85 | def test_invalid_transport_exits(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 107 | def test_server_transport_env_var_set(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 60 | def test_search_all_returns_nonempty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 64 | def test_search_by_op_type_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 70 | def test_search_by_op_type_mapper(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 82 | def test_search_match_all_tags(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 88 | def test_search_match_any_tag(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 94 | def test_search_by_type_and_tag(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 100 | def test_search_invalid_type_returns_empty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 104 | def test_search_result_is_list_of_dicts(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 119 | def test_regex_search_by_name(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 125 | def test_regex_search_case_insensitive(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 130 | def test_regex_search_with_op_type_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 136 | def test_regex_search_with_tag_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 141 | def test_regex_search_custom_fields(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 147 | def test_regex_search_invalid_pattern_returns_empty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 151 | def test_regex_search_no_match_returns_empty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 155 | def test_regex_search_result_is_list_of_dicts(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 233 | def test_specified_ops_only_scans_given_ops(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 241 | def test_all_ops_dict_populated(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 246 | def test_records_map_deprecated_still_returns_all_ops(self): | CODE |
| LOW | tests/tools/test_op_search.py | 16 | def test_record_has_required_fields(self): | CODE |
| LOW | tests/tools/test_op_search.py | 21 | def test_record_type_is_valid(self): | CODE |
| 1516 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | app.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 392 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 394 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 744 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 746 | # ============================================================================ | COMMENT |
| MEDIUM | tools/evaluator/gpt_eval/gpt_evaluator.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/distributed_deduplication/dedup_utils.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/modeling_megatron_llama.py | 5 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/convert_gpt_to_transformers.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/convert_gpt_to_transformers.py | 7 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/preprocess/raw_arxiv_to_jsonl.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/preprocess/raw_stackexchange_to_jsonl.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 37 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 116 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 136 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 234 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/util.py | 327 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 376 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 16 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 23 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 30 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 37 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 35 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 75 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …on_metrics/video_metrics/frechet_inception_distance.py | 22 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …on_metrics/video_metrics/frechet_inception_distance.py | 69 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 51 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 74 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 106 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 221 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 266 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 370 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 482 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 493 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/kernel_video_distance.py | 19 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/kernel_video_distance.py | 76 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 31 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 49 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 88 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 102 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/frechet_video_distance.py | 23 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/frechet_video_distance.py | 73 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 19 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 36 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 92 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/video_inception_score.py | 22 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/video_inception_score.py | 66 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …val/inception_metrics/video_metrics/inception_score.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …val/inception_metrics/video_metrics/inception_score.py | 52 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ion_metrics/video_metrics/kernel_inception_distance.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ion_metrics/video_metrics/kernel_inception_distance.py | 58 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/run.py | 6 | # -------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 206 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 208 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 255 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 257 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 296 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 298 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_ray_exporter.py | 346 | # ------------------------------------------------------------------ | COMMENT |
| 261 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 78 | except Exception as e: | CODE |
| LOW | service.py | 102 | except Exception as e: | CODE |
| LOW | app.py | 118 | except Exception as e: | CODE |
| LOW | app.py | 179 | except Exception as e: | CODE |
| LOW | app.py | 421 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 37 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 65 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 133 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 92 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 191 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 222 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 227 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 274 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 314 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 381 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 451 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 541 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 575 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 623 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 667 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 728 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 820 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 851 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 918 | except Exception as e: | CODE |
| LOW | tools/check_ray_cluster.py | 80 | except Exception as e: | CODE |
| LOW | tools/count_rows.py | 56 | except Exception as e: | CODE |
| LOW | tools/count_rows.py | 76 | except Exception as e: | CODE |
| MEDIUM | tools/count_rows.py | 77 | print(f"Error: Could not read file with extension {extension}: {e}") | CODE |
| LOW | tools/count_rows.py | 127 | except Exception as e: | CODE |
| MEDIUM | tools/count_rows.py | 160 | print(f"Error: Path not found: {args.path}") | CODE |
| MEDIUM | tools/count_rows.py | 185 | print(f"Error: Path is neither a file nor a directory: {args.path}") | CODE |
| LOW | tools/generate_uv_lock.py | 61 | except Exception as e: | CODE |
| MEDIUM | tools/generate_uv_lock.py | 62 | print(f"Error: {e}", file=sys.stderr) | CODE |
| LOW | tools/generate_uv_lock.py | 76 | except Exception as e: | CODE |
| MEDIUM | tools/generate_uv_lock.py | 77 | print(f"Error: {e}", file=sys.stderr) | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 188 | except Exception as e: | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 229 | except Exception: | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 233 | except Exception: | CODE |
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 74 | except Exception: | CODE |
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 169 | except Exception as e: | CODE |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 85 | except Exception as e: | CODE |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 184 | except Exception as e: | CODE |
| MEDIUM | tools/evaluator/gpt_eval/answer_generator.py | 187 | print(f"Error in megatron response: {response}, retry in 10s") | CODE |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 54 | except Exception as e: | CODE |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 65 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 285 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 296 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 512 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 616 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 665 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 679 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 697 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 810 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 851 | except Exception as e: | STRING |
| LOW⚡ | tools/humanops/label_studio_service.py | 857 | except Exception as cleanup_error: | STRING |
| LOW⚡ | tools/humanops/label_studio_service.py | 866 | except Exception as e: | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 553 | except Exception: | CODE |
| LOW | tests/run.py | 118 | except Exception as e: | CODE |
| LOW | tests/run.py | 123 | except Exception as e: | CODE |
| LOW | tests/tools/test_process_data.py | 52 | except Exception as e: | CODE |
| 277 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …on_metrics/video_metrics/frechet_inception_distance.py | 19 | CODE | |
| LOW | tests/core/test_exporter.py | 2 | CODE | |
| LOW | tests/core/test_exporter.py | 4 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 12 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 13 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 15 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 15 | CODE | |
| LOW | tests/core/executor/test_event_logging_mixin.py | 20 | CODE | |
| LOW | tests/core/executor/test_event_logging_mixin.py | 20 | CODE | |
| LOW | tests/core/executor/test_partitioned_integration.py | 19 | CODE | |
| LOW | tests/core/executor/test_dag_execution_strategies.py | 14 | CODE | |
| LOW | tests/core/executor/test_dag_execution_strategies.py | 16 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 3 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 298 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 298 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 299 | CODE | |
| LOW | tests/config/test_config.py | 2 | CODE | |
| LOW | tests/config/test_config.py | 3 | CODE | |
| LOW | tests/config/test_config.py | 14 | CODE | |
| LOW | tests/utils/test_model_utils.py | 6 | CODE | |
| LOW | tests/utils/test_datasets_json_compat.py | 7 | CODE | |
| LOW | tests/utils/test_logger_utils.py | 204 | CODE | |
| LOW | tests/utils/test_lazy_loader.py | 8 | CODE | |
| LOW | tests/utils/test_lazy_loader.py | 9 | CODE | |
| LOW | tests/utils/test_lazy_loader.py | 11 | CODE | |
| LOW | tests/utils/test_s3_utils.py | 6 | CODE | |
| LOW | tests/format/test_empty_formatter.py | 1 | CODE | |
| LOW | tests/format/test_text_formatter.py | 1 | CODE | |
| LOW | tests/format/test_text_formatter.py | 157 | CODE | |
| LOW | tests/ops/test_op_fusion.py | 4 | CODE | |
| LOW | tests/ops/test_op_fusion.py | 7 | CODE | |
| LOW | …mapper/test_video_whole_body_pose_estimation_mapper.py | 7 | CODE | |
| LOW | tests/ops/mapper/test_sdxl_prompt2prompt_mapper.py | 1 | CODE | |
| LOW | tests/ops/mapper/test_sdxl_prompt2prompt_mapper.py | 2 | CODE | |
| LOW | tests/ops/mapper/test_video_undistort_mapper.py | 3 | CODE | |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 8 | CODE | |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 11 | CODE | |
| LOW | …per/test_video_camera_calibration_droidcalib_mapper.py | 9 | CODE | |
| LOW | tests/ops/mapper/test_video_split_by_duration_mapper.py | 12 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 5 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 6 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 7 | CODE | |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 7 | CODE | |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 10 | CODE | |
| LOW | tests/ops/mapper/test_query_intent_detection_mapper.py | 2 | CODE | |
| LOW | …ts/ops/mapper/test_video_hand_action_compute_mapper.py | 1 | CODE | |
| LOW | tests/ops/mapper/test_vggt_mapper.py | 7 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 6 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 7 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 8 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 10 | CODE | |
| LOW | …/annotation/test_human_preference_annotation_mapper.py | 3 | CODE | |
| LOW | …/annotation/test_human_preference_annotation_mapper.py | 5 | CODE | |
| LOW | tests/ops/mapper/annotation/test_annotation_mapper.py | 2 | CODE | |
| LOW | tests/ops/mapper/annotation/test_annotation_mapper.py | 5 | CODE | |
| LOW | tests/ops/filter/test_image_pair_similarity_filter.py | 8 | CODE | |
| LOW | …filter/test_instruction_following_difficulty_filter.py | 6 | CODE | |
| LOW | tests/ops/filter/test_llm_perplexity_filter.py | 6 | CODE | |
| LOW | tests/ops/deduplicator/test_ray_video_deduplicator.py | 4 | CODE | |
| LOW | tests/ops/deduplicator/test_ray_image_deduplicator.py | 4 | CODE | |
| 353 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 38 | CODE | |
| LOW | hatch_build.py | 21 | CODE | |
| LOW | hatch_build.py | 31 | CODE | |
| LOW | app.py | 149 | CODE | |
| LOW | app.py | 220 | CODE | |
| LOW | app.py | 387 | CODE | |
| LOW | app.py | 243 | CODE | |
| LOW | tools/dj_install.py | 18 | CODE | |
| LOW | tools/dj_install.py | 42 | CODE | |
| LOW | tools/dj_install.py | 70 | CODE | |
| LOW | tools/check_s3_integration.py | 38 | CODE | |
| LOW | tools/check_s3_integration.py | 101 | CODE | |
| LOW | tools/check_s3_integration.py | 201 | CODE | |
| LOW | tools/check_s3_integration.py | 283 | CODE | |
| LOW | tools/check_ray_cluster.py | 6 | CODE | |
| LOW | tools/count_rows.py | 34 | CODE | |
| LOW | tools/data_resplit.py | 13 | CODE | |
| LOW | tools/process_data.py | 18 | CODE | |
| LOW | tools/evaluator/recorder/wandb_writer.py | 99 | CODE | |
| LOW | tools/evaluator/recorder/wandb_writer.py | 214 | CODE | |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 202 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 876 | CODE | |
| LOW | tools/converter/convert_gpt_to_transformers.py | 279 | CODE | |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 29 | CODE | |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 73 | CODE | |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 57 | CODE | |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 84 | CODE | |
| LOW | tools/preprocess/serialize_meta.py | 20 | CODE | |
| LOW | tools/preprocess/reformat_jsonl_nan_value.py | 30 | CODE | |
| LOW | tools/mm_eval/inception_metrics/util.py | 396 | CODE | |
| LOW | tools/mm_eval/inception_metrics/dataset.py | 24 | CODE | |
| LOW | tools/mm_eval/inception_metrics/dataset.py | 101 | CODE | |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 76 | CODE | |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 72 | CODE | |
| LOW | …data_juicer_format_to_target_format/dj_to_internvid.py | 41 | CODE | |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 98 | CODE | |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 44 | CODE | |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 65 | CODE | |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 50 | CODE | |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 54 | CODE | |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 73 | CODE | |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 92 | CODE | |
| LOW | …_to_data_juicer_format/llama_factory_sharegpt_to_dj.py | 72 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 257 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 631 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 710 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 815 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 170 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 316 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 576 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 619 | CODE | |
| LOW | tests/tools/test_process_data.py | 12 | CODE | |
| LOW | tests/tools/test_process_data.py | 189 | CODE | |
| LOW | tests/tools/test_process_data.py | 232 | CODE | |
| LOW | tests/tools/test_process_data.py | 275 | CODE | |
| LOW | tests/tools/test_process_data.py | 330 | CODE | |
| LOW | tests/utils/test_availability_utils.py | 86 | CODE | |
| LOW | tests/utils/test_availability_utils.py | 97 | CODE | |
| LOW | tests/utils/test_video_utils.py | 83 | CODE | |
| LOW | …/mapper/test_video_hand_reconstruction_hawor_mapper.py | 44 | CODE | |
| 266 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tools/preprocess/raw_arxiv_to_jsonl.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/serialize_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/reformat_jsonl_nan_value.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/postprocess/deserialize_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/raw_alpaca_cot_merge_add_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | …data_juicer_format_to_target_format/dj_to_internvid.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | …ce_format_to_data_juicer_format/video_chatgpt_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | tests/ops/mapper/test_extract_event_mapper.py | 0 | △芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到 | STRING |
| HIGH | tests/ops/mapper/test_extract_support_text_mapper.py | 0 | △芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到 | STRING |
| HIGH | tests/ops/mapper/test_extract_keyword_mapper.py | 0 | △芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到 | STRING |
| HIGH | tests/ops/mapper/test_extract_entity_relation_mapper.py | 0 | △芩婆走到中间,看着众人。 芩婆:当年,我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居,一日为了救人,得罪附近山匪,夜里便遭了山匪所袭,唯有二子生还,流落街头。 封磬震惊:二子?不是只有一个儿子吗? 芩婆:我和漆木山得知这个噩耗后,到 | STRING |
| HIGH | tests/ops/mapper/test_calibrate_response_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖 | STRING |
| HIGH | tests/ops/mapper/test_calibrate_query_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖 | STRING |
| HIGH | tests/ops/mapper/test_calibrate_qa_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例,你必须贴合他的语言风格: 问题:你是谁? 李莲花:在下李莲花,不才略有一点神医之名,有礼。 问题:你就是个假神医! 李莲花:此言差矣,我从未说过我是神医,又何来假神医之说。 问题:李相夷是江湖 | STRING |
| HIGH | docs/DeveloperGuide_ZH.md | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | docs/DeveloperGuide.md | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | data_juicer/ops/filter/text_length_filter.py | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | data_juicer/format/json_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/format/parquet_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/format/csv_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/ops/base_op.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/video_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …icer/ops/deduplicator/document_minhash_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/document_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/image_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …icer/ops/deduplicator/document_simhash_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …_juicer/ops/mapper/punctuation_normalization_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/remove_bibliography_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_copyright_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_html_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | …a_juicer/ops/mapper/whitespace_normalization_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/expand_macro_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/grouper/naive_grouper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_ip_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/clean_links_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/clean_email_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/query_intent_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | data_juicer/ops/mapper/query_topic_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | …_juicer/ops/mapper/query_sentiment_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | …_juicer/ops/mapper/video_captioning_from_vlm_mapper.py | 0 | :param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $n | STRING |
| HIGH | …uicer/ops/mapper/video_captioning_from_video_mapper.py | 0 | :param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $n | STRING |
| HIGH | …icer/ops/mapper/video_captioning_from_frames_mapper.py | 0 | :param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $n | STRING |
| HIGH | demos/process_code_data/app.py | 0 | # refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed data | STRING |
| HIGH | demos/process_sci_data/app.py | 0 | # refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed data | STRING |
| HIGH | demos/process_cft_zh_data/app.py | 0 | # refined open source dataset by data-juicer we found that there are still some "bad" samples in existing processed data | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tools/check_s3_integration.py | 759 | # Create a small test dataset | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 854 | # Create a small test dataset | COMMENT |
| MEDIUM⚡ | tools/humanops/label_studio_service.py | 280 | # Create the file if it doesn't exist | COMMENT |
| MEDIUM | tools/humanops/label_studio_service.py | 343 | # Create a named volume for Label Studio data | COMMENT |
| MEDIUM | tools/humanops/label_studio_service.py | 823 | # Create a test project | COMMENT |
| MEDIUM⚡ | tools/humanops/label_studio_service.py | 842 | # Create a sample task using import_tasks instead of create_task | STRING |
| MEDIUM | tests/tools/test_process_data.py | 15 | # Create a temporary file for logging | COMMENT |
| MEDIUM⚡ | tests/core/executor/test_event_logging_mixin.py | 532 | # Create a mock executor with disabled logging | COMMENT |
| MEDIUM | tests/core/executor/test_ray_executor_partitioned.py | 129 | # Create a simple config without loading from file | COMMENT |
| MEDIUM | tests/core/executor/test_ray_executor_partitioned.py | 301 | # Create a simple config without loading from file | COMMENT |
| MEDIUM | tests/core/data/test_ray_dataset.py | 80 | # Create a PyArrow table similar to what would be passed to the function | COMMENT |
| MEDIUM | tests/core/data/test_ray_dataset.py | 134 | _text = "I have a very long text. " * LONG_TEXT_MULTIPLIER # Create a long text to test large JSON objects | CODE |
| MEDIUM | tests/core/data/test_ray_dataset.py | 136 | # Create a temporary JSONL file | COMMENT |
| MEDIUM | tests/core/data/test_load_strategy.py | 525 | # Create a proper Dataset object for the mock to return | COMMENT |
| MEDIUM | tests/core/data/test_load_strategy.py | 566 | # Create a proper Dataset object for the mock to return | COMMENT |
| MEDIUM | tests/config/test_config.py | 941 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/config/test_config.py | 977 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/config/test_config.py | 1014 | # Create a temporary config file with invalid work_dir | COMMENT |
| MEDIUM | tests/config/test_config.py | 1043 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/utils/test_model_utils.py | 161 | # Create a mock vllm module | COMMENT |
| MEDIUM | tests/utils/test_datasets_json_compat.py | 113 | # Create a JSONL file with big integer | COMMENT |
| MEDIUM⚡ | tests/utils/test_fingerprint_utils.py | 222 | # Create an object that dill cannot serialize | COMMENT |
| MEDIUM⚡ | tests/utils/test_ckpt_utils.py | 416 | # Create a mock checkpoint file | COMMENT |
| MEDIUM⚡ | tests/utils/test_ckpt_utils.py | 495 | # Create a corrupted checkpoint file | COMMENT |
| MEDIUM | tests/utils/test_ckpt_utils.py | 602 | # Create a file that starts with checkpoint_op_ and ends with | COMMENT |
| MEDIUM | tests/utils/test_ckpt_utils.py | 679 | # Create a mock checkpoint file so path exists | COMMENT |
| MEDIUM | tests/utils/test_lazy_loader.py | 305 | # Create an empty uv.lock | COMMENT |
| MEDIUM | tests/utils/test_lazy_loader.py | 320 | # Create an invalid uv.lock | COMMENT |
| MEDIUM | tests/utils/test_constant.py | 22 | # Create a temporary jsonl file for testing | COMMENT |
| MEDIUM | tests/utils/test_constant.py | 35 | # Create a temporary config with the test dataset path | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 25 | # Create a test file with complex extension | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 29 | # Create a directory structure for relative path testing | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 33 | # Create a test JSONL file in the relative path directory | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 85 | # Create a temp directory with the name that matches the error | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 89 | # Create the file with the name from the error | COMMENT |
| MEDIUM⚡ | tests/format/test_text_formatter.py | 129 | # Create a minimal .docx in memory | COMMENT |
| MEDIUM | tests/ops/test_op_env.py | 780 | # Create a temporary Python file | STRING |
| MEDIUM | tests/ops/test_op_fusion_mapper_bug.py | 35 | # Create a NEW dict instead of mutating in-place | COMMENT |
| MEDIUM⚡ | tests/ops/mapper/test_python_lambda_mapper.py | 46 | PythonLambdaMapper("lambda x, y: {'sum': x + y}") # Creating a lambda accepts two arguments | CODE |
| MEDIUM | tests/ops/mapper/test_image_mmpose_mapper.py | 24 | # Create a temporary file for logging | COMMENT |
| MEDIUM⚡ | …/ops/mapper/test_video_atomic_action_segment_mapper.py | 76 | # Create a speed profile with clear minima | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 95 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 170 | # Create a sample | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 173 | # Create an annotation with preference for the left option (answer1) | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 189 | # Create a sample | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 192 | # Create an annotation with preference for the right option (answer2) | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 230 | # Create a sample with custom keys | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 238 | # Create a dictionary version of the sample | COMMENT |
| MEDIUM⚡ | tests/ops/mapper/annotation/test_annotation_mapper.py | 302 | # Create a mapper with a very short timeout | COMMENT |
| MEDIUM⚡ | tests/ops/mapper/annotation/test_annotation_mapper.py | 305 | # Create a task but don't add annotations | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 89 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 421 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 435 | self.mock_ls_client = MagicMock() # Create a mock client | CODE |
| MEDIUM | data_juicer/tools/__init__.py | 5 | # This file is used for package installation. Script of tools will be | COMMENT |
| MEDIUM | data_juicer/core/executor/ray_executor_partitioned.py | 529 | # Create a RayDataset wrapper for this partition | COMMENT |
| MEDIUM | data_juicer/core/data/load_strategy.py | 96 | # Create the lookup key | COMMENT |
| MEDIUM | data_juicer/utils/video_utils.py | 464 | # Create a named temporary file that will be automatically deleted when closed | COMMENT |
| MEDIUM | data_juicer/utils/lazy_loader.py | 303 | # Create a temporary directory for cloning | COMMENT |
| MEDIUM⚡ | data_juicer/utils/registry.py | 22 | """This class is used to register some modules to registry by a repo | STRING |
| MEDIUM | data_juicer/utils/constant.py | 270 | # Create a stream reader for the file and decode the | COMMENT |
| 8 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/check_s3_integration.py | 140 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 324 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 507 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 690 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 777 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 870 | # Check if credentials are available | COMMENT |
| LOW | tools/check_ray_cluster.py | 43 | # Check if head node participates in workload | COMMENT |
| LOW | tools/check_ray_cluster.py | 65 | # Check if this is the head node by comparing node IDs | COMMENT |
| LOW | tools/generate_smtp_cert.py | 157 | # Check if OpenSSL is installed | COMMENT |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 306 | # Check if the file exists (all processes must agree). | COMMENT |
| LOW⚡ | tools/humanops/label_studio_service.py | 294 | # Set permissions to 777 (rwxrwxrwx) to ensure the container can write to it | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 347 | # Check if volume exists, if not create it | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 385 | # Set user to root for ensuring write permissions inside container | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 414 | # Check if there's a specific error message we can extract | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 426 | # Check if the container was partially created and clean it up | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 461 | # Check if container is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 619 | # Check if container is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 764 | # Check if process is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 786 | # Check if the server is responding | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 985 | # Check if Docker is installed and running | STRING |
| LOW⚡ | tests/core/executor/test_pipeline_dag.py | 60 | # Assign nodes to DAG | COMMENT |
| LOW | tests/core/executor/test_partitioned_integration.py | 135 | # Check if deduplicator config exists | COMMENT |
| LOW | …/annotation/test_human_preference_annotation_mapper.py | 293 | # Check if each predefined ID is in the sample-to-task mapping | COMMENT |
| LOW | tests/ops/mapper/annotation/test_annotation_mapper.py | 368 | # Check if each predefined ID is in the sample-to-task mapping | COMMENT |
| LOW | …cs/op_doc_enhance_workflow/check_tools/check_render.py | 86 | # Check if it's just "not available" | COMMENT |
| LOW | data_juicer/core/exporter.py | 89 | # Check if export_path is S3 and create storage_options if needed | COMMENT |
| LOW | data_juicer/core/ray_exporter.py | 88 | # Check if export_path is S3 and create filesystem if needed | COMMENT |
| LOW | data_juicer/core/ray_exporter.py | 161 | # Check if dataset is empty by calling columns() regardless of columns parameter | COMMENT |
| LOW | data_juicer/core/tracer/tracer.py | 92 | # Check if sample has changed | COMMENT |
| LOW | data_juicer/core/tracer/ray_tracer.py | 87 | # Check if sample has changed | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 375 | # Check if user provided a job_id (indicating resumption attempt) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 746 | # Check if the current work_dir already contains the job_id | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 754 | # Check if this directory has events files (indicating it's a work directory) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 798 | # Check if config validation passed (done during config initialization) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 803 | # Check if job is already completed | COMMENT |
| LOW⚡ | data_juicer/core/executor/event_logging_mixin.py | 434 | # Check if DAGExecutionMixin is available and has the method to get DAG node | COMMENT |
| LOW | data_juicer/core/executor/event_logging_mixin.py | 1054 | # Check if job has any completion events | COMMENT |
| LOW | data_juicer/core/executor/event_logging_mixin.py | 1063 | # Check if all partitions are completed (success or failure) | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 63 | # Check if DAG execution is enabled (default: True for distributed executors, False for standalone) | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 412 | # Check if we have per-op metrics | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 750 | # Check if all dependencies are completed | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 877 | # Check if this node was already completed | COMMENT |
| LOW | data_juicer/core/data/data_validator.py | 336 | # Check if fields exist in dataset | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 169 | # Check if dataset is empty - Ray returns None for columns() on empty datasets | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 422 | # Check if open_json is available (PyArrow 20.0.0+) | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 511 | # Check if open_json is available (PyArrow 20.0.0+) | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 516 | js.open_json # Check if attribute exists | CODE |
| LOW | data_juicer/analysis/correlation_analysis.py | 115 | # Loop over the data and create a `Text` for each "pixel". | COMMENT |
| LOW | data_juicer/config/config.py | 881 | # Check if this is a resumption attempt by looking for existing job directory | COMMENT |
| LOW | data_juicer/config/config.py | 1878 | # Check if {job_id} is at the end of the path | COMMENT |
| LOW | data_juicer/utils/model_utils.py | 683 | # Check if we need accelerate for device_map | COMMENT |
| LOW | data_juicer/utils/model_utils.py | 983 | # Check if we need accelerate for device_map | COMMENT |
| LOW | data_juicer/utils/constant.py | 266 | # Open the file in binary mode and | COMMENT |
| LOW | data_juicer/utils/common_utils.py | 173 | # Check if version is provided when using as @deprecated (without parameters) | COMMENT |
| LOW | data_juicer/utils/job/common.py | 342 | # Check if processes are still running | COMMENT |
| LOW | data_juicer/ops/mixins.py | 278 | # Check if notifications are enabled for this operator | COMMENT |
| LOW | data_juicer/ops/mixins.py | 299 | # Check if notifications are enabled | COMMENT |
| LOW | data_juicer/ops/mixins.py | 305 | # Check if notification handlers are initialized | COMMENT |
| LOW | data_juicer/ops/mixins.py | 330 | # Check if this specific channel is enabled | COMMENT |
| LOW | data_juicer/ops/base_op.py | 125 | # Check if collection is already complete (early exit for performance) | COMMENT |
| 14 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/converter/modeling_megatron_llama.py | 1 | # Some codes are adapted from https://pypi.org/project/transformers | COMMENT |
| LOW | tools/converter/convert_gpt_to_transformers.py | 1 | # Some code here has been modified from: | COMMENT |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 1 | # Part of the code here has been modified from: | COMMENT |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 1 | # Part of the code here has been modified from: | COMMENT |
| LOW | tools/mm_eval/inception_metrics/util.py | 401 | # Doesn't look like an URL scheme so interpret it as a local filename. | COMMENT |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 1 | # This tool is used to convert the absolute paths or relative paths in | COMMENT |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 21 | # 'images': ['coco/train2017/000000033471.jpg'], | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 21 | # - in json | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 41 | # "value": "The back of the bus features an advertisement." | COMMENT |
| LOW | …data_juicer_format_to_target_format/dj_to_internvid.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 21 | # 'doors lock/ unlock at the same time. <|__dj__eoc|>', | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 41 | # 'time.', | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 61 | # {'face_detections': None, | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 21 | # 'tags': '' }} | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 41 | # { 'num_captions_per_audio': 1, | COMMENT |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 21 | # (exploring spatial, temporal, relationships, and reasoning concepts); | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 21 | # {'videos': ['videos/retrieval/14111B1211bA1F31-E-2YB57--C518FCEBC553abJYFa5541a31C8a57522AYJbYF4aTdfofa112.mp4'], # n | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 41 | # | COMMENT |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 21 | # - in jsonl | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 1 | # This tool is used to convert multimodal dataset in WavCaps format to a target | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 21 | # 'caption': 'Someone is ringing a bell.', | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 41 | # 'caption': 'An airplane is landing.', | COMMENT |
| LOW | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 1 | # This tool is used to convert multimodal dataset in MSR-VTT format to a | COMMENT |
| LOW | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 21 | # - two new fields are added: | COMMENT |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 1 | # This tool is used to convert multimodal dataset in InternVid format to a | COMMENT |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 21 | # - videos: video paths list, including cut videos according to their timestamps # noqa: E501 | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 1 | # This tool is used to convert multimodal dataset in LLaVA format to a target | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 21 | # "from": "human", | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 41 | # Corresponding Data-Juicer format: | COMMENT |
| LOW | …ce_format_to_data_juicer_format/video_chatgpt_to_dj.py | 1 | # This tool is used to convert multimodal dataset in Video_Chatgpt format to a | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 1 | # This tool is used to convert multimodal dataset in MMC4 format to a target | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 21 | # 0.3234919607639313, | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 41 | # {'text': 'When you lock the door using the lock tab on the driver’s door, ' | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 61 | # 0.27694183588027954], | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 1 | # This tool is used to convert multimodal dataset in Youku-mPLUG format to a | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 21 | # | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 41 | # | COMMENT |
| LOW | …cer_format_to_target_format/dj_to_ms_swift_sharegpt.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …cer_format_to_target_format/dj_to_ms_swift_sharegpt.py | 21 | # [ | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 21 | # ... | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 41 | # }, | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 21 | # ... | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 41 | # }, | COMMENT |
| LOW | …og/data_juicer_format_to_target_format/dj_to_alpaca.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …og/data_juicer_format_to_target_format/dj_to_alpaca.py | 21 | # { | COMMENT |
| LOW | …og/source_format_to_data_juicer_format/alpaca_to_dj.py | 1 | # This tool is used to convert dataset in Alpaca format to a | COMMENT |
| LOW | …og/source_format_to_data_juicer_format/alpaca_to_dj.py | 21 | # { | COMMENT |
| LOW | …ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py | 1 | # This tool is used to convert dataset in ModelScope-Swift ShareGPT format to a | COMMENT |
| LOW | …ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py | 21 | # | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 1 | # This tool is used to convert dataset in ModelScope-Swift Messages format to a | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 21 | # }, | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 41 | # { | COMMENT |
| 44 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/converter/modeling_megatron_llama.py | 58 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 550 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 716 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 876 | CODE | |
| LOW | …s/mm_eval/inception_metrics/calc_metrics_for_videos.py | 20 | CODE | |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 54 | CODE | |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 73 | CODE | |
| LOW | scripts/dlc/partition_data_dlc.py | 20 | CODE | |
| LOW | data_juicer/core/exporter.py | 16 | CODE | |
| LOW | data_juicer/core/data/ray_dataset.py | 493 | CODE | |
| LOW | data_juicer/utils/model_utils.py | 1642 | CODE | |
| LOW | data_juicer/utils/llm_semantic_ops.py | 309 | CODE | |
| LOW | data_juicer/download/wikipedia.py | 722 | CODE | |
| LOW | …r/ops/pipeline/llm_inference_with_ray_vllm_pipeline.py | 26 | CODE | |
| LOW | …a_juicer/ops/aggregator/entity_attribute_aggregator.py | 63 | CODE | |
| LOW | …er/ops/aggregator/most_relevant_entities_aggregator.py | 49 | CODE | |
| LOW | …er/ops/aggregator/most_relevant_entities_aggregator.py | 129 | CODE | |
| LOW | data_juicer/ops/aggregator/meta_tags_aggregator.py | 91 | CODE | |
| LOW | data_juicer/ops/aggregator/nested_aggregator.py | 58 | CODE | |
| LOW | data_juicer/ops/mapper/extract_support_text_mapper.py | 51 | CODE | |
| LOW | data_juicer/ops/mapper/pii_redaction_mapper.py | 107 | CODE | |
| LOW | data_juicer/ops/mapper/image_tagging_vlm_mapper.py | 61 | CODE | |
| LOW | …juicer/ops/mapper/dialog_sentiment_detection_mapper.py | 66 | CODE | |
| LOW | …a_juicer/ops/mapper/video_trajectory_overlay_mapper.py | 46 | CODE | |
| LOW | data_juicer/ops/mapper/sentence_augmentation_mapper.py | 33 | CODE | |
| LOW | data_juicer/ops/mapper/image_sam_3d_body_mapper.py | 30 | CODE | |
| LOW | data_juicer/ops/mapper/relation_identity_mapper.py | 48 | CODE | |
| LOW | …ta_juicer/ops/mapper/dialog_intent_detection_mapper.py | 71 | CODE | |
| LOW | …juicer/ops/mapper/dialog_sentiment_intensity_mapper.py | 79 | CODE | |
| LOW | data_juicer/ops/mapper/video_split_by_scene_mapper.py | 60 | CODE | |
| LOW | data_juicer/ops/mapper/agent_insight_llm_mapper.py | 198 | CODE | |
| LOW | data_juicer/ops/mapper/pii_llm_suspect_mapper.py | 346 | CODE | |
| LOW | …ta_juicer/ops/mapper/extract_entity_relation_mapper.py | 162 | CODE | |
| LOW | …r/ops/mapper/video_hand_reconstruction_hawor_mapper.py | 29 | CODE | |
| LOW | data_juicer/ops/mapper/pair_preference_mapper.py | 41 | CODE | |
| LOW | …/ops/mapper/video_whole_body_pose_estimation_mapper.py | 31 | CODE | |
| LOW | data_juicer/ops/mapper/llm_extract_mapper.py | 40 | CODE | |
| LOW | data_juicer/ops/mapper/image_diffusion_mapper.py | 43 | CODE | |
| LOW | data_juicer/ops/mapper/agent_bad_case_signal_mapper.py | 126 | CODE | |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 29 | CODE | |
| LOW | data_juicer/ops/mapper/s3_upload_file_mapper.py | 33 | CODE | |
| LOW | data_juicer/ops/mapper/nlpcda_zh_mapper.py | 32 | CODE | |
| LOW | …_juicer/ops/mapper/video_hand_reconstruction_mapper.py | 38 | CODE | |
| LOW | data_juicer/ops/mapper/dialog_topic_detection_mapper.py | 69 | CODE | |
| LOW | data_juicer/ops/mapper/vggt_mapper.py | 43 | CODE | |
| LOW | …s/mapper/video_camera_calibration_droidcalib_mapper.py | 35 | CODE | |
| LOW | …a_juicer/ops/mapper/extract_entity_attribute_mapper.py | 54 | CODE | |
| LOW | data_juicer/ops/mapper/extract_nickname_mapper.py | 61 | CODE | |
| LOW | data_juicer/ops/mapper/s3_download_file_mapper.py | 35 | CODE | |
| LOW | …_juicer/ops/mapper/video_captioning_from_vlm_mapper.py | 58 | CODE | |
| LOW | data_juicer/ops/mapper/image_captioning_mapper.py | 46 | CODE | |
| LOW | …_juicer/ops/mapper/generate_qa_from_examples_mapper.py | 57 | CODE | |
| LOW | …uicer/ops/mapper/video_captioning_from_video_mapper.py | 54 | CODE | |
| LOW | data_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py | 37 | CODE | |
| LOW | …/ops/mapper/video_captioning_from_summarizer_mapper.py | 26 | CODE | |
| LOW | data_juicer/ops/mapper/calibrate_qa_mapper.py | 40 | CODE | |
| LOW | data_juicer/ops/mapper/text_tagging_by_prompt_mapper.py | 67 | CODE | |
| LOW | data_juicer/ops/mapper/nlpaug_en_mapper.py | 32 | CODE | |
| LOW | data_juicer/ops/mapper/video_clip_reassembly_mapper.py | 45 | CODE | |
| LOW | …icer/ops/mapper/video_captioning_from_frames_mapper.py | 56 | CODE | |
| 42 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 46 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 58 | # Step 2: Run camera calibration | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 55 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 68 | # Step 2: Run camera calibration | COMMENT |
| LOW⚡ | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 33 | # Step 1: Extract real frames from videos | COMMENT |
| LOW | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 47 | # Step 2: Add dummy camera calibration data matching real frame dims | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 543 | # Step 1: Validate event logger availability | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 548 | # Step 2: Load DAG events and execution plan | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 554 | # Step 3: Reconstruct node states from plan and events | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 558 | # Step 4: Calculate statistics | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 561 | # Step 5: Determine ready nodes | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 564 | # Step 6: Determine resumption strategy | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 651 | # Step 2: Check if this fits in available memory | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 660 | # Step 3: Ensure sufficient parallelism for large datasets | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 674 | # Step 4: Adjust for data skew | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 681 | # Step 5: Apply final bounds | COMMENT |
| LOW | data_juicer/core/executor/partition_size_optimizer.py | 635 | # Step 1: Calculate dynamic target based on available memory | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 158 | # Step 1: Decode all frames and record their dimensions | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 172 | # Step 2: Check if all frames share the same resolution (typical for a single clip) | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 241 | # Step 3: Write results to tag_dict | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 101 | # Step 1: Decode and preprocess all frames, record original dimensions | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 119 | # Step 2: Batch inference | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 286 | # Step 1: Convert to row-oriented format (list of sample dictionaries) | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 289 | # Step 2: Retrieve or generate unique IDs for each sample | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 296 | # Step 3: Process samples in batches | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 331 | # Step 5: Convert back to column-oriented format efficiently | COMMENT |
| LOW | data_juicer/ops/mapper/annotation/annotation_mapper.py | 317 | # Step 4: Update samples with annotation results | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 37 | # Step 1: Download videos from S3 to local | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 52 | # Step 2: Video filtering - duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 59 | # Step 3: Video filtering - resolution | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 69 | # Step 4: Split videos by duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 79 | # Step 5: Adjust video aspect ratio | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 87 | # Step 6: Upload processed videos to S3 | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 46 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 58 | # Step 2: Run camera calibration | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 55 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 68 | # Step 2: Run camera calibration | COMMENT |
| LOW⚡ | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 33 | # Step 1: Extract real frames from videos | COMMENT |
| LOW | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 47 | # Step 2: Add dummy camera calibration data matching real frame dims | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 543 | # Step 1: Validate event logger availability | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 548 | # Step 2: Load DAG events and execution plan | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 554 | # Step 3: Reconstruct node states from plan and events | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 558 | # Step 4: Calculate statistics | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 561 | # Step 5: Determine ready nodes | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 564 | # Step 6: Determine resumption strategy | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 651 | # Step 2: Check if this fits in available memory | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 660 | # Step 3: Ensure sufficient parallelism for large datasets | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 674 | # Step 4: Adjust for data skew | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 681 | # Step 5: Apply final bounds | COMMENT |
| LOW | data_juicer/core/executor/partition_size_optimizer.py | 635 | # Step 1: Calculate dynamic target based on available memory | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 158 | # Step 1: Decode all frames and record their dimensions | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 172 | # Step 2: Check if all frames share the same resolution (typical for a single clip) | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 241 | # Step 3: Write results to tag_dict | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 101 | # Step 1: Decode and preprocess all frames, record original dimensions | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 119 | # Step 2: Batch inference | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 286 | # Step 1: Convert to row-oriented format (list of sample dictionaries) | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 289 | # Step 2: Retrieve or generate unique IDs for each sample | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 296 | # Step 3: Process samples in batches | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 331 | # Step 5: Convert back to column-oriented format efficiently | COMMENT |
| LOW | data_juicer/ops/mapper/annotation/annotation_mapper.py | 317 | # Step 4: Update samples with annotation results | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 37 | # Step 1: Download videos from S3 to local | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 52 | # Step 2: Video filtering - duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 59 | # Step 3: Video filtering - resolution | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 69 | # Step 4: Split videos by duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 79 | # Step 5: Adjust video aspect ratio | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 87 | # Step 6: Upload processed videos to S3 | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | README.md | 167 | - **[data-juicer-agents](https://github.com/datajuicer/data-juicer-agents)** — DJ Copilot and agentic workflows | CODE |
| MEDIUM | tests/utils/test_model_utils.py | 114 | # Test with processor for embedding model | COMMENT |
| MEDIUM | tests/utils/test_model_utils.py | 196 | # Test embedding model | COMMENT |
| MEDIUM | data_juicer/config/config_all.yaml | 872 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 883 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 908 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 988 | api_or_hf_model: text-embedding-v4 # API or huggingface embedding model name | CODE |
| MEDIUM | …ta_juicer/ops/mapper/extract_entity_relation_mapper.py | 1 | # This OP is modified from light RAG | COMMENT |
| MEDIUM | data_juicer/ops/mapper/tool_success_tagger_mapper.py | 5 | # Configurable success/error patterns for multi-agent and multi-language. | COMMENT |
| MEDIUM | data_juicer/ops/mapper/extract_keyword_mapper.py | 33 | # This prompt is modified from light RAG | COMMENT |
| MEDIUM | data_juicer/ops/mapper/agent_dialog_normalize_mapper.py | 5 | # dialog/text ops. Supports multi-platform, multi-agent tool formats. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 120 | def set_cache_dir(path: str) -> None: | CODE |
| LOW | data_juicer/core/__init__.py | 10 | __all__ = [ | CODE |
| LOW | data_juicer/core/executor/__init__.py | 7 | __all__ = ["ExecutorBase", "ExecutorFactory", "DefaultExecutor", "RayExecutor", "PartitionedRayExecutor"] | CODE |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 641 | def _update_node_states_from_events(self, node_states: Dict[str, Dict[str, Any]], dag_events: List[Any]) -> None: | CODE |
| LOW | data_juicer/core/data/__init__.py | 8 | __all__ = ["DJDataset", "NestedDataset", "wrap_func_with_nested_access", "add_same_content_to_new_column"] | CODE |
| LOW | data_juicer/analysis/__init__.py | 6 | __all__ = [ | CODE |
| LOW | data_juicer/config/__init__.py | 15 | __all__ = [ | CODE |
| LOW | data_juicer/utils/llm_semantic_ops.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/utils/job/__init__.py | 18 | __all__ = [ | CODE |
| LOW | data_juicer/format/__init__.py | 9 | __all__ = [ | CODE |
| LOW | data_juicer/ops/__init__.py | 41 | __all__ = [ | CODE |
| LOW | data_juicer/ops/pipeline/__init__.py | 5 | __all__ = ["LLMRayVLLMEnginePipeline", "RayRepartitionPipeline", "VLMRayVLLMEnginePipeline"] | CODE |
| LOW | data_juicer/ops/selector/__init__.py | 7 | __all__ = [ | CODE |
| LOW | data_juicer/ops/aggregator/__init__.py | 6 | __all__ = ["NestedAggregator", "MetaTagsAggregator", "EntityAttributeAggregator", "MostRelevantEntitiesAggregator"] | CODE |
| LOW | data_juicer/ops/mapper/sentence_augmentation_mapper.py | 10 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/mapper/agent_bad_case_signal_mapper.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/mapper/__init__.py | 143 | __all__ = [ | CODE |
| LOW | data_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/grouper/__init__.py | 5 | __all__ = ["KeyValueGrouper", "NaiveGrouper", "NaiveReverseGrouper"] | CODE |
| LOW | data_juicer/ops/common/__init__.py | 14 | __all__ = [ | CODE |
| LOW | data_juicer/ops/filter/text_pair_similarity_filter.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | …/ops/filter/instruction_following_difficulty_filter.py | 11 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/filter/llm_perplexity_filter.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/filter/__init__.py | 61 | __all__ = [ | CODE |
| LOW | data_juicer/ops/deduplicator/__init__.py | 20 | __all__ = [ | CODE |
| LOW | demos/agent/scripts/generate_bad_case_report.py | 87 | def set_report_locale(lang: str) -> None: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | data_juicer/core/data/dj_dataset.py | 59 | Get values from a specific column/field, optionally limited to first k rows. Args: column: Name of | STRING |
| HIGH | data_juicer/core/data/dj_dataset.py | 228 | Get column values from HuggingFace dataset. Args: column: Name of the column to retrieve | STRING |
| HIGH | data_juicer/core/data/data_validator.py | 292 | Convert field type specifications to actual Python type objects. Args: field_types: Dictio | STRING |
| HIGH | data_juicer/core/data/ray_dataset.py | 132 | Get column values from Ray dataset. Args: column: Name of the column to retrieve k: Opt | STRING |
| HIGH | data_juicer/core/data/schema.py | 41 | Map HuggingFace feature type to Python type. Recursively maps nested types (e.g., List[str], Dict[str, int]). | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | label_studio_localhost_connection.json | 4 | "username": "admin@example.com", | CODE |
| LOW | tools/humanops/label_studio_service.py | 947 | args.username = "admin@example.com" | STRING |
| LOW⚡ | tests/ops/test_mixins.py | 479 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 496 | "access_token": "fake_token", | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 516 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 531 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 35 | '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 35 | '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 41 | 'Welcome to My WebsiteLorem ipsum dolor sit amet, ' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 41 | 'Welcome to My WebsiteLorem ipsum dolor sit amet, ' | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 28 | <div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafa | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 28 | <div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafa | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 35 | *ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All Ri | COMMENT |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 35 | *ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All Ri | COMMENT |
| LOW | data_juicer/config/config_all.yaml | 1278 | recipients: ["user@example.com"] | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 606 | # No post-convergence operations, just return the merged result | COMMENT |
| MEDIUM | data_juicer/core/executor/event_logging_mixin.py | 250 | """Generate a comprehensive status report.""" | STRING |
| MEDIUM | data_juicer/core/executor/partition_size_optimizer.py | 450 | # Calculate percentile-based memory estimates (p90 is more robust than mean) | COMMENT |
| MEDIUM | data_juicer/core/executor/partition_size_optimizer.py | 776 | """Get comprehensive partition recommendations.""" | STRING |
| MEDIUM | data_juicer/config/config.py | 1493 | # If global_parser is available, use it for robust parsing | COMMENT |
| MEDIUM | data_juicer/utils/job/snapshot.py | 401 | """Convert snapshot to JSON-serializable dictionary with comprehensive progress tracking.""" | STRING |
| LOW | data_juicer/ops/op_env.py | 505 | # and if any of them is arbitrary equality, just use it | COMMENT |
| MEDIUM | demos/partition_and_checkpoint/run_demo.py | 264 | """Run the comprehensive demo.""" | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | data_juicer/core/data/ray_dataset.py | 475 | elif not schema.equals(batch.schema): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/ops/mapper/test_python_file_mapper.py | 13 | "def process_data(sample):\n" | CODE |
| LOW⚡ | tests/ops/mapper/test_python_file_mapper.py | 25 | "def process_data(samples):\n" | CODE |
| LOW⚡ | tests/ops/mapper/test_python_file_mapper.py | 38 | "def process_data(sample):\n" | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | data_juicer/tools/hpo/objects.py | 45 | # feel free to customize the quality scorer, via the following args | COMMENT |