Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
This report summarizes scanner evidence for datajuicer/data-juicer, a Python project with 7,133 GitHub stars. SynthScan examined 250,306 lines of code across 1315 source files, recording 4626 pattern matches distributed across 22 syntactic categories. The legacy adjusted pattern signal is 25.0 (Elevated pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 3238 finding(s).
Are imports declared and package/API names verified? 458 finding(s).
Are wrappers, guards, and repeated structures necessary? 3459 finding(s).
Which changes need focused human review? 4626 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 4626 distinct pattern matches across 22 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 38 | def register_objects_from_init(directory: str): | CODE |
| LOW | tools/check_ray_cluster.py | 6 | def get_ray_cluster_resources(): | CODE |
| LOW | tools/evaluator/evaluator.py | 221 | def megatron_checkpoint_exists(self, iteration): | CODE |
| LOW | tools/converter/modeling_megatron_llama.py | 419 | def _set_gradient_checkpointing(self, module, value=False): | STRING |
| LOW | tools/converter/modeling_megatron_llama.py | 526 | def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length): | STRING |
| LOW | tools/converter/modeling_megatron_llama.py | 803 | def prepare_inputs_for_generation( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 60 | def add_transformers_checkpoint_args(parser): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 148 | def megatron_to_transformers_fix_query_key_value_ordering( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 186 | def transformers_to_megatron_fix_query_key_value_ordering( | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 224 | def merge_transformers_sharded_states(path, num_checkpoints): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 240 | def get_megatron_sharded_states(args, tp_size, pp_size, pp_rank): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 262 | def get_element_from_dict_by_path(d, path): | CODE |
| LOW | tools/converter/convert_gpt_to_transformers.py | 279 | def convert_checkpoint_from_megatron_to_transformers(args): | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 306 | def get_module_dir_by_obj_name(obj_name: str) -> str: | CODE |
| LOW⚡ | tools/mm_eval/inception_metrics/util.py | 317 | def get_top_level_function_name(obj: Any) -> str: | CODE |
| LOW⚡ | tools/mm_eval/inception_metrics/util.py | 329 | def list_dir_recursively_with_ignore(dir_path: str, ignores: List[str] = None, add_base_to_relative: bool = False) -> Li | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 362 | def copy_files_and_create_dirs(files: List[Tuple[str, str]]) -> None: | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 78 | def get_feature_detector_name(url): | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 269 | def compute_feature_stats_for_dataset( | CODE |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 373 | def compute_feature_stats_for_generator( | CODE |
| LOW⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 484 | def rewrite_opts_for_gen_dataset(opts): | CODE |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 76 | def convert_absolute_path_to_relative_path( | CODE |
| LOW | tools/fmt_conversion/multimodal/utils.py | 29 | def check_args_load_to_dj_data( | CODE |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 71 | def dj_to_llama_factory_sharegpt( | CODE |
| LOW | …_to_data_juicer_format/llama_factory_sharegpt_to_dj.py | 72 | def llama_factory_sharegpt_to_dj( | CODE |
| LOW | tools/humanops/label_studio_service.py | 257 | def start_label_studio_container( | CODE |
| LOW | tools/humanops/label_studio_service.py | 631 | def get_container_error_message(container_name): | CODE |
| LOW | tools/humanops/label_studio_service.py | 1063 | def start_label_studio_with_docker(args, api_token): | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 250 | def _compact_class_or_module_name(name: str) -> str: | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 343 | def get_op_list_from_code_for_formatter(): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 520 | def generate_op_table_section(op_type, op_record_list, reference_op_record_dict): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 568 | def get_op_desc_in_en_zh_batched(descs): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 608 | def parse_op_record_from_current_doc(): | CODE |
| LOW | .pre-commit-hooks/build_op_doc.py | 651 | def check_and_update_op_record(old_op_record_list, new_op_record_list): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 74 | def test_scored_comparisons_are_averaged(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 78 | def test_failed_comparison_count_is_printed(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 82 | def test_genuine_zero_scores_are_still_counted(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 86 | def test_failed_judge_call_is_left_out_and_counted(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 91 | def test_unparseable_reply_is_left_out_and_counted(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 96 | def test_failed_comparison_keeps_a_null_score_in_the_rows(self): | CODE |
| LOW⚡ | tests/tools/test_gpt_evaluator.py | 102 | def test_all_comparisons_failed_reports_no_averages(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 19 | def test_granular_ops_mode_default_transport(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 31 | def test_recipe_flow_mode_default_transport(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 65 | def test_custom_port_passed_to_create(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 85 | def test_invalid_transport_exits(self): | CODE |
| LOW | tests/tools/test_mcp_server.py | 107 | def test_server_transport_env_var_set(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 60 | def test_search_all_returns_nonempty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 64 | def test_search_by_op_type_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 70 | def test_search_by_op_type_mapper(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 82 | def test_search_match_all_tags(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 88 | def test_search_match_any_tag(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 94 | def test_search_by_type_and_tag(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 100 | def test_search_invalid_type_returns_empty(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 104 | def test_search_result_is_list_of_dicts(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 119 | def test_regex_search_by_name(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 125 | def test_regex_search_case_insensitive(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 130 | def test_regex_search_with_op_type_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 136 | def test_regex_search_with_tag_filter(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 141 | def test_regex_search_custom_fields(self): | CODE |
| LOW⚡ | tests/tools/test_op_search.py | 147 | def test_regex_search_invalid_pattern_returns_empty(self): | CODE |
| 2361 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | app.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 392 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 394 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 744 | # ============================================================================ | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 746 | # ============================================================================ | COMMENT |
| MEDIUM | tools/evaluator/gpt_eval/gpt_evaluator.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/distributed_deduplication/dedup_utils.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/modeling_megatron_llama.py | 5 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/convert_gpt_to_transformers.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/converter/convert_gpt_to_transformers.py | 7 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/preprocess/raw_arxiv_to_jsonl.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/preprocess/raw_stackexchange_to_jsonl.py | 3 | # -------------------------------------------------------- | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 37 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 116 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 136 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 234 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/util.py | 327 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM | tools/mm_eval/inception_metrics/util.py | 376 | # ------------------------------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 16 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 23 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 30 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tools/mm_eval/inception_metrics/distributed.py | 37 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 35 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …al/inception_metrics/video_metrics/precision_recall.py | 75 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …on_metrics/video_metrics/frechet_inception_distance.py | 22 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …on_metrics/video_metrics/frechet_inception_distance.py | 69 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 51 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 74 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 106 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 221 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 266 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …m_eval/inception_metrics/video_metrics/metric_utils.py | 370 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 482 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …m_eval/inception_metrics/video_metrics/metric_utils.py | 493 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/kernel_video_distance.py | 19 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/kernel_video_distance.py | 76 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 31 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 49 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 88 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …mm_eval/inception_metrics/video_metrics/metric_main.py | 102 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/frechet_video_distance.py | 23 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/frechet_video_distance.py | 73 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 19 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 36 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …eption_metrics/video_metrics/video_precision_recall.py | 92 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/video_inception_score.py | 22 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ception_metrics/video_metrics/video_inception_score.py | 66 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …val/inception_metrics/video_metrics/inception_score.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …val/inception_metrics/video_metrics/inception_score.py | 52 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ion_metrics/video_metrics/kernel_inception_distance.py | 18 | #---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …ion_metrics/video_metrics/kernel_inception_distance.py | 58 | # ---------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/run.py | 6 | # -------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 215 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 217 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 293 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 295 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 334 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_exporter.py | 336 | # ------------------------------------------------------------------ | COMMENT |
| MEDIUM⚡ | tests/core/test_ray_exporter.py | 346 | # ------------------------------------------------------------------ | COMMENT |
| 307 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 78 | except Exception as e: | CODE |
| LOW | service.py | 102 | except Exception as e: | CODE |
| LOW | app.py | 118 | except Exception as e: | CODE |
| LOW | app.py | 179 | except Exception as e: | CODE |
| LOW | app.py | 439 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 37 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 65 | except Exception as e: | CODE |
| LOW | tools/dj_install.py | 133 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 92 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 191 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 222 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 227 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 274 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 314 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 381 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 451 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 541 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 575 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 623 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 667 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 728 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 820 | except Exception as e: | CODE |
| LOW | tools/check_s3_integration.py | 851 | except Exception: | CODE |
| LOW | tools/check_s3_integration.py | 918 | except Exception as e: | CODE |
| LOW | tools/check_ray_cluster.py | 80 | except Exception as e: | CODE |
| LOW | tools/count_rows.py | 56 | except Exception as e: | CODE |
| LOW | tools/count_rows.py | 76 | except Exception as e: | CODE |
| MEDIUM | tools/count_rows.py | 77 | print(f"Error: Could not read file with extension {extension}: {e}") | CODE |
| LOW | tools/count_rows.py | 127 | except Exception as e: | CODE |
| MEDIUM | tools/count_rows.py | 160 | print(f"Error: Path not found: {args.path}") | CODE |
| MEDIUM | tools/count_rows.py | 185 | print(f"Error: Path is neither a file nor a directory: {args.path}") | CODE |
| LOW | tools/generate_uv_lock.py | 61 | except Exception as e: | CODE |
| MEDIUM | tools/generate_uv_lock.py | 62 | print(f"Error: {e}", file=sys.stderr) | CODE |
| LOW | tools/generate_uv_lock.py | 76 | except Exception as e: | CODE |
| MEDIUM | tools/generate_uv_lock.py | 77 | print(f"Error: {e}", file=sys.stderr) | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 188 | except Exception as e: | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 229 | except Exception: | CODE |
| LOW | tools/evaluator/recorder/wandb_writer.py | 233 | except Exception: | CODE |
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 74 | except Exception: | CODE |
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 176 | except Exception as e: | CODE |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 85 | except Exception as e: | CODE |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 184 | except Exception as e: | CODE |
| MEDIUM | tools/evaluator/gpt_eval/answer_generator.py | 187 | print(f"Error in megatron response: {response}, retry in 10s") | CODE |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 54 | except Exception as e: | CODE |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 65 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 285 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 296 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 512 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 616 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 665 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 679 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 697 | except Exception as e: | CODE |
| LOW | tools/humanops/label_studio_service.py | 810 | except Exception as e: | CODE |
| LOW⚡ | tools/humanops/label_studio_service.py | 851 | except Exception as e: | STRING |
| LOW⚡ | tools/humanops/label_studio_service.py | 857 | except Exception as cleanup_error: | STRING |
| LOW⚡ | tools/humanops/label_studio_service.py | 866 | except Exception as e: | STRING |
| LOW | .pre-commit-hooks/build_op_doc.py | 585 | except Exception: | CODE |
| LOW | tests/run.py | 118 | except Exception as e: | CODE |
| LOW | tests/run.py | 123 | except Exception as e: | CODE |
| LOW | tests/tools/test_process_data.py | 52 | except Exception as e: | CODE |
| 334 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | hatch_build.py | 50 | except (subprocess.CalledProcessError, FileNotFoundError): | CODE |
| HIGH | tools/check_s3_integration.py | 222 | except Exception: | CODE |
| HIGH | tools/check_s3_integration.py | 314 | except Exception: | CODE |
| HIGH | tools/check_s3_integration.py | 575 | except Exception: | CODE |
| HIGH | tools/check_s3_integration.py | 667 | except Exception: | CODE |
| HIGH | tools/check_s3_integration.py | 851 | except Exception: | CODE |
| HIGH | tools/evaluator/recorder/wandb_writer.py | 229 | except Exception: | CODE |
| HIGH | tools/evaluator/recorder/wandb_writer.py | 233 | except Exception: | CODE |
| HIGH | tools/mm_eval/inception_metrics/util.py | 173 | except ValueError: | CODE |
| HIGH | tools/mm_eval/inception_metrics/util.py | 254 | except: | CODE |
| HIGH | tools/mm_eval/inception_metrics/util.py | 270 | except ImportError: | CODE |
| HIGH | tools/humanops/label_studio_service.py | 525 | except (requests.exceptions.ConnectionError, requests.exceptions.Timeout): | CODE |
| HIGH | tools/humanops/label_studio_service.py | 792 | except (requests.exceptions.ConnectionError, requests.exceptions.Timeout): | CODE |
| HIGH | tests/core/executor/test_ray_executor.py | 118 | except Exception: | CODE |
| HIGH | tests/utils/test_logger_utils.py | 222 | except ValueError: | CODE |
| HIGH | tests/utils/test_logger_utils.py | 301 | except ValueError: | CODE |
| HIGH | tests/utils/job/test_stopper.py | 72 | except OSError: | CODE |
| HIGH | tests/format/test_text_formatter.py | 309 | except Exception: | CODE |
| HIGH | docs/op_doc_enhance_workflow/utils/parse_class.py | 315 | except (ValueError, SyntaxError, TypeError): | CODE |
| HIGH | data_juicer/core/exporter.py | 215 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | data_juicer/core/elasticjuicer/adaptive_mapper.py | 162 | except Exception: # noqa: BLE001 - diagnostics must not fail the pipeline | CODE |
| HIGH | data_juicer/core/elasticjuicer/stage_profile.py | 285 | except Exception: | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 768 | except (AttributeError, TypeError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 1230 | except (AttributeError, TypeError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 1749 | except (AttributeError, TypeError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 1841 | except (AttributeError, TypeError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 788 | except (TypeError, ValueError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 1949 | except (AttributeError, RuntimeError): | CODE |
| HIGH | data_juicer/core/executor/ray_executor_partitioned.py | 802 | except (AttributeError, RuntimeError): | CODE |
| HIGH | data_juicer/core/executor/dag_execution_strategies.py | 462 | except ImportError: | CODE |
| HIGH | data_juicer/core/executor/gpu_memory_probe.py | 1376 | except Exception: | CODE |
| HIGH | data_juicer/core/executor/partition_size_optimizer.py | 106 | except ImportError: | CODE |
| HIGH | data_juicer/core/executor/partition_size_optimizer.py | 513 | except TypeError: | CODE |
| HIGH | data_juicer/config/config.py | 1770 | except ValueError: | CODE |
| HIGH | data_juicer/config/config.py | 1776 | except ValueError: | CODE |
| HIGH | data_juicer/utils/model_utils.py | 698 | except Exception: | CODE |
| HIGH | data_juicer/utils/model_utils.py | 703 | except Exception: | CODE |
| HIGH | data_juicer/utils/model_utils.py | 710 | except Exception: | CODE |
| HIGH | data_juicer/utils/model_utils.py | 718 | except Exception: | CODE |
| HIGH | data_juicer/utils/model_utils.py | 2344 | except Exception: | CODE |
| HIGH | data_juicer/utils/video_utils.py | 713 | except Empty: | CODE |
| HIGH | data_juicer/utils/unittest_utils.py | 141 | except Exception: | CODE |
| HIGH | data_juicer/utils/s3_utils.py | 20 | except ImportError: | CODE |
| HIGH | data_juicer/utils/compress.py | 29 | except OSError: | CODE |
| HIGH | data_juicer/utils/file_utils.py | 207 | except FileExistsError: | CODE |
| HIGH | data_juicer/utils/process_utils.py | 48 | except ImportError: | CODE |
| HIGH | data_juicer/utils/datasets_json_compat.py | 79 | except ImportError: | CODE |
| HIGH | data_juicer/utils/nltk_utils.py | 84 | except Exception: | CODE |
| HIGH | data_juicer/utils/llm_semantic_ops.py | 366 | except json.JSONDecodeError: | CODE |
| HIGH | data_juicer/utils/job/common.py | 355 | except Exception: | CODE |
| HIGH | data_juicer/utils/job/common.py | 364 | except (psutil.NoSuchProcess, psutil.AccessDenied): | CODE |
| HIGH | data_juicer/download/wikipedia.py | 684 | except ValueError: | CODE |
| HIGH | data_juicer/download/wikipedia.py | 691 | except ValueError: | CODE |
| HIGH | data_juicer/format/formatter.py | 107 | except OSError: | CODE |
| HIGH | data_juicer/format/text_formatter.py | 314 | except OSError: | CODE |
| HIGH | data_juicer/ops/mapper/pii_redaction_mapper.py | 203 | except re.error: | CODE |
| HIGH | data_juicer/ops/mapper/agent_insight_llm_mapper.py | 111 | except (json.JSONDecodeError, TypeError): | CODE |
| HIGH | data_juicer/ops/mapper/agent_insight_llm_mapper.py | 75 | except Exception: | CODE |
| HIGH | data_juicer/ops/mapper/agent_insight_llm_mapper.py | 80 | except Exception: | CODE |
| HIGH | data_juicer/ops/mapper/pii_llm_suspect_mapper.py | 53 | except Exception: | CODE |
| 17 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …on_metrics/video_metrics/frechet_inception_distance.py | 19 | CODE | |
| LOW | tests/core/test_exporter.py | 8 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 12 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 13 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 15 | CODE | |
| LOW | tests/core/executor/test_partition_size_optimizer.py | 15 | CODE | |
| LOW | tests/core/executor/test_event_logging_mixin.py | 21 | CODE | |
| LOW | tests/core/executor/test_event_logging_mixin.py | 21 | CODE | |
| LOW | tests/core/executor/test_auto_partition_cluster.py | 10 | CODE | |
| LOW | tests/core/executor/test_partitioned_integration.py | 19 | CODE | |
| LOW | tests/core/executor/test_dag_execution_strategies.py | 14 | CODE | |
| LOW | tests/core/executor/test_dag_execution_strategies.py | 16 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 4 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 334 | CODE | |
| LOW | tests/core/executor/test_ray_executor_partitioned.py | 334 | CODE | |
| LOW | tests/config/test_config.py | 2 | CODE | |
| LOW | tests/config/test_config.py | 3 | CODE | |
| LOW | tests/config/test_config.py | 14 | CODE | |
| LOW | tests/utils/test_model_utils.py | 11 | CODE | |
| LOW | tests/utils/test_model_utils.py | 13 | CODE | |
| LOW | tests/utils/test_datasets_json_compat.py | 7 | CODE | |
| LOW | tests/utils/test_logger_utils.py | 204 | CODE | |
| LOW | tests/utils/test_lazy_loader.py | 2 | CODE | |
| LOW | tests/utils/test_ASD_mapper_utils.py | 25 | CODE | |
| LOW | tests/utils/test_s3_utils.py | 6 | CODE | |
| LOW | tests/utils/job/test_stopper.py | 5 | CODE | |
| LOW | tests/utils/job/test_snapshot.py | 6 | CODE | |
| LOW | tests/download/test_download.py | 96 | CODE | |
| LOW | tests/format/test_empty_formatter.py | 1 | CODE | |
| LOW | tests/format/test_text_formatter.py | 1 | CODE | |
| LOW | tests/format/test_text_formatter.py | 157 | CODE | |
| LOW | tests/ops/test_base_op.py | 7 | CODE | |
| LOW | tests/ops/test_base_op.py | 7 | CODE | |
| LOW | tests/ops/test_op_fusion.py | 4 | CODE | |
| LOW | …mapper/test_video_whole_body_pose_estimation_mapper.py | 7 | CODE | |
| LOW | tests/ops/mapper/test_sdxl_prompt2prompt_mapper.py | 1 | CODE | |
| LOW | tests/ops/mapper/test_sdxl_prompt2prompt_mapper.py | 2 | CODE | |
| LOW | tests/ops/mapper/test_video_undistort_mapper.py | 3 | CODE | |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 8 | CODE | |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 11 | CODE | |
| LOW | …per/test_video_camera_calibration_droidcalib_mapper.py | 9 | CODE | |
| LOW | tests/ops/mapper/test_video_split_by_duration_mapper.py | 12 | CODE | |
| LOW | tests/ops/mapper/test_video_audio_ASR_mapper.py | 13 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 5 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 6 | CODE | |
| LOW | …apper/test_imgdiff_difference_area_generator_mapper.py | 7 | CODE | |
| LOW | …ps/mapper/test_video_human_tracks_extraction_mapper.py | 19 | CODE | |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 7 | CODE | |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 10 | CODE | |
| LOW | tests/ops/mapper/test_query_intent_detection_mapper.py | 2 | CODE | |
| LOW | …s/ops/mapper/test_video_audio_speech_emotion_mapper.py | 15 | CODE | |
| LOW | …ts/ops/mapper/test_video_hand_action_compute_mapper.py | 1 | CODE | |
| LOW | tests/ops/mapper/test_vggt_mapper.py | 7 | CODE | |
| LOW | …ps/mapper/test_video_audio_detect_age_gender_mapper.py | 15 | CODE | |
| LOW | …per/test_video_human_tracks_face_demographic_mapper.py | 15 | CODE | |
| LOW | tests/ops/mapper/test_extract_entity_relation_mapper.py | 10 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 6 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 7 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 8 | CODE | |
| LOW | …er/test_imgdiff_difference_caption_generator_mapper.py | 10 | CODE | |
| 381 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | service.py | 38 | CODE | |
| LOW | hatch_build.py | 21 | CODE | |
| LOW | hatch_build.py | 31 | CODE | |
| LOW | app.py | 149 | CODE | |
| LOW | app.py | 233 | CODE | |
| LOW | app.py | 405 | CODE | |
| LOW | app.py | 256 | CODE | |
| LOW | tools/dj_install.py | 18 | CODE | |
| LOW | tools/dj_install.py | 42 | CODE | |
| LOW | tools/dj_install.py | 70 | CODE | |
| LOW | tools/check_s3_integration.py | 38 | CODE | |
| LOW | tools/check_s3_integration.py | 101 | CODE | |
| LOW | tools/check_s3_integration.py | 201 | CODE | |
| LOW | tools/check_s3_integration.py | 283 | CODE | |
| LOW | tools/check_ray_cluster.py | 6 | CODE | |
| LOW | tools/count_rows.py | 34 | CODE | |
| LOW | tools/data_resplit.py | 13 | CODE | |
| LOW | tools/process_data.py | 18 | CODE | |
| LOW | tools/evaluator/recorder/wandb_writer.py | 99 | CODE | |
| LOW | tools/evaluator/recorder/wandb_writer.py | 214 | CODE | |
| LOW | tools/evaluator/gpt_eval/answer_generator.py | 202 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 876 | CODE | |
| LOW | tools/converter/convert_gpt_to_transformers.py | 279 | CODE | |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 29 | CODE | |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 73 | CODE | |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 57 | CODE | |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 84 | CODE | |
| LOW | tools/preprocess/serialize_meta.py | 20 | CODE | |
| LOW | tools/preprocess/reformat_jsonl_nan_value.py | 30 | CODE | |
| LOW | tools/mm_eval/inception_metrics/util.py | 396 | CODE | |
| LOW | tools/mm_eval/inception_metrics/dataset.py | 24 | CODE | |
| LOW | tools/mm_eval/inception_metrics/dataset.py | 101 | CODE | |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 76 | CODE | |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 72 | CODE | |
| LOW | …data_juicer_format_to_target_format/dj_to_internvid.py | 41 | CODE | |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 98 | CODE | |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 44 | CODE | |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 65 | CODE | |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 50 | CODE | |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 54 | CODE | |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 73 | CODE | |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 92 | CODE | |
| LOW | …_to_data_juicer_format/llama_factory_sharegpt_to_dj.py | 72 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 257 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 631 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 710 | CODE | |
| LOW | tools/humanops/label_studio_service.py | 815 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 168 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 281 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 343 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 608 | CODE | |
| LOW | .pre-commit-hooks/build_op_doc.py | 651 | CODE | |
| LOW | tests/tools/test_process_data.py | 12 | CODE | |
| LOW | tests/tools/test_process_data.py | 189 | CODE | |
| LOW | tests/tools/test_process_data.py | 232 | CODE | |
| LOW | tests/tools/test_process_data.py | 275 | CODE | |
| LOW | tests/tools/test_process_data.py | 330 | CODE | |
| LOW | tests/tools/test_hpo.py | 27 | CODE | |
| LOW | tests/core/elasticjuicer/test_stage_profile.py | 199 | CODE | |
| LOW | tests/core/elasticjuicer/test_stage_profile.py | 222 | CODE | |
| 308 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tools/preprocess/raw_arxiv_to_jsonl.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/serialize_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/reformat_jsonl_nan_value.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/postprocess/deserialize_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | tools/preprocess/raw_alpaca_cot_merge_add_meta.py | 0 | find all tar files in the source directory. :param src_dir: path to source dataset directory :return: iterator over json | STRING |
| HIGH | …data_juicer_format_to_target_format/dj_to_internvid.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 0 | convert a data-juicer-format dataset to a msr-vtt-like dataset. :param dj_ds_path: path to the input dataset in data-jui | STRING |
| HIGH | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | …ce_format_to_data_juicer_format/video_chatgpt_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 0 | convert a youku-mplug-like dataset to the data-juicer format. :param youku_ds_path: path to the input youku-mplug-like d | STRING |
| HIGH | …sts/ops/aggregator/test_entity_attribute_aggregator.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | tests/ops/aggregator/test_meta_tags_aggregator.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | tests/ops/aggregator/test_nested_aggregator.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | …s/aggregator/test_most_relevant_entities_aggregator.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | tests/ops/mapper/test_dialog_topic_detection_mapper.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | …s/ops/mapper/test_dialog_sentiment_intensity_mapper.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | tests/ops/mapper/test_dialog_intent_detection_mapper.py | 0 | pure logic tests that do not require an external api. | STRING |
| HIGH | tests/ops/mapper/test_optimize_qa_mapper.py | 0 | test parse_output method without requiring model initialization. | STRING |
| HIGH | tests/ops/mapper/test_optimize_prompt_mapper.py | 0 | test parse_output method without requiring model initialization. | STRING |
| HIGH | tests/ops/mapper/test_generate_qa_from_text_mapper.py | 0 | test parse_output method without requiring model initialization. | STRING |
| HIGH | tests/ops/mapper/test_extract_event_mapper.py | 0 | △芩婆走到中间，看着众人。 芩婆：当年，我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居，一日为了救人，得罪附近山匪，夜里便遭了山匪所袭，唯有二子生还，流落街头。 封磬震惊：二子？不是只有一个儿子吗？ 芩婆：我和漆木山得知这个噩耗后，到 | STRING |
| HIGH | tests/ops/mapper/test_extract_support_text_mapper.py | 0 | △芩婆走到中间，看着众人。 芩婆：当年，我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居，一日为了救人，得罪附近山匪，夜里便遭了山匪所袭，唯有二子生还，流落街头。 封磬震惊：二子？不是只有一个儿子吗？ 芩婆：我和漆木山得知这个噩耗后，到 | STRING |
| HIGH | tests/ops/mapper/test_extract_keyword_mapper.py | 0 | △芩婆走到中间，看着众人。 芩婆：当年，我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居，一日为了救人，得罪附近山匪，夜里便遭了山匪所袭，唯有二子生还，流落街头。 封磬震惊：二子？不是只有一个儿子吗？ 芩婆：我和漆木山得知这个噩耗后，到 | STRING |
| HIGH | tests/ops/mapper/test_extract_entity_relation_mapper.py | 0 | △芩婆走到中间，看着众人。 芩婆：当年，我那老鬼漆木山与李相夷之父乃是挚交。原本李家隐世而居，一日为了救人，得罪附近山匪，夜里便遭了山匪所袭，唯有二子生还，流落街头。 封磬震惊：二子？不是只有一个儿子吗？ 芩婆：我和漆木山得知这个噩耗后，到 | STRING |
| HIGH | …/ops/mapper/test_video_active_speaker_detect_mapper.py | 0 | create a real bbox pkl file for testing. the vlm model still runs on real cropped video frames — no mock is used. | STRING |
| HIGH | …test_video_captioning_face_attribute_emotion_mapper.py | 0 | create a real bbox pkl file for testing. the vlm model still runs on real cropped video frames — no mock is used. | STRING |
| HIGH | …pper/test_video_captioning_from_human_tracks_mapper.py | 0 | create a real bbox pkl file for testing. the vlm model still runs on real cropped video frames — no mock is used. | STRING |
| HIGH | tests/ops/mapper/test_calibrate_response_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例，你必须贴合他的语言风格： 问题：你是谁？ 李莲花：在下李莲花，不才略有一点神医之名，有礼。 问题：你就是个假神医！ 李莲花：此言差矣，我从未说过我是神医，又何来假神医之说。 问题：李相夷是江湖 | STRING |
| HIGH | tests/ops/mapper/test_calibrate_query_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例，你必须贴合他的语言风格： 问题：你是谁？ 李莲花：在下李莲花，不才略有一点神医之名，有礼。 问题：你就是个假神医！ 李莲花：此言差矣，我从未说过我是神医，又何来假神医之说。 问题：李相夷是江湖 | STRING |
| HIGH | tests/ops/mapper/test_calibrate_qa_mapper.py | 0 | # 角色语言风格 1. 下面是李莲花的问答样例，你必须贴合他的语言风格： 问题：你是谁？ 李莲花：在下李莲花，不才略有一点神医之名，有礼。 问题：你就是个假神医！ 李莲花：此言差矣，我从未说过我是神医，又何来假神医之说。 问题：李相夷是江湖 | STRING |
| HIGH | docs/DeveloperGuide_ZH.md | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | docs/DeveloperGuide.md | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | data_juicer/ops/filter/text_length_filter.py | 0 | initialization method. :param min_len: the min text length in the filtering. samples will be filtered if their text leng | STRING |
| HIGH | data_juicer/format/json_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/format/parquet_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/format/csv_formatter.py | 0 | initialization method. :param dataset_path: a dataset file or a dataset directory :param suffixes: files with specified | STRING |
| HIGH | data_juicer/ops/base_op.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/video_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …icer/ops/deduplicator/document_minhash_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/document_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | data_juicer/ops/deduplicator/image_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …icer/ops/deduplicator/document_simhash_deduplicator.py | 0 | for doc-level, dataset --> dataset. :param dataset: input dataset :param show_num: number of traced samples used when tr | STRING |
| HIGH | …_juicer/ops/mapper/punctuation_normalization_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/remove_bibliography_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_copyright_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_html_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | …a_juicer/ops/mapper/whitespace_normalization_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/expand_macro_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | …juicer/ops/mapper/video_audio_speech_emotion_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/video_audio_ASR_mapper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/grouper/naive_grouper.py | 0 | initialization method. :param args: extra args :param kwargs: extra args | STRING |
| HIGH | data_juicer/ops/mapper/clean_ip_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/clean_links_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/clean_email_mapper.py | 0 | initialization method. :param pattern: regular expression pattern to search for within text. :param repl: replacement st | STRING |
| HIGH | data_juicer/ops/mapper/query_intent_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | data_juicer/ops/mapper/query_topic_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | …_juicer/ops/mapper/query_sentiment_detection_mapper.py | 0 | initialization method. :param hf_model: huggingface model id to predict sentiment label. :param zh_to_en_hf_model: trans | STRING |
| HIGH | …_juicer/ops/mapper/video_captioning_from_vlm_mapper.py | 0 | :param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $n | STRING |
| HIGH | …uicer/ops/mapper/video_captioning_from_video_mapper.py | 0 | :param samples: :return: note: this is a batched_op, whose the input and output type are both list. suppose there are $n | STRING |
| 4 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tools/check_s3_integration.py | 759 | # Create a small test dataset | COMMENT |
| MEDIUM | tools/check_s3_integration.py | 854 | # Create a small test dataset | COMMENT |
| MEDIUM⚡ | tools/humanops/label_studio_service.py | 280 | # Create the file if it doesn't exist | COMMENT |
| MEDIUM | tools/humanops/label_studio_service.py | 343 | # Create a named volume for Label Studio data | COMMENT |
| MEDIUM | tools/humanops/label_studio_service.py | 823 | # Create a test project | COMMENT |
| MEDIUM⚡ | tools/humanops/label_studio_service.py | 842 | # Create a sample task using import_tasks instead of create_task | STRING |
| MEDIUM | tests/tools/test_process_data.py | 15 | # Create a temporary file for logging | COMMENT |
| MEDIUM⚡ | tests/core/executor/test_event_logging_mixin.py | 534 | # Create a mock executor with disabled logging | COMMENT |
| MEDIUM | tests/core/executor/test_ray_executor_partitioned.py | 165 | # Create a simple config without loading from file | COMMENT |
| MEDIUM | tests/core/executor/test_ray_executor_partitioned.py | 337 | # Create a simple config without loading from file | COMMENT |
| MEDIUM | tests/core/data/test_ray_dataset.py | 82 | # Create a PyArrow table similar to what would be passed to the function | COMMENT |
| MEDIUM | tests/core/data/test_ray_dataset.py | 136 | _text = "I have a very long text. " * LONG_TEXT_MULTIPLIER # Create a long text to test large JSON objects | CODE |
| MEDIUM | tests/core/data/test_ray_dataset.py | 138 | # Create a temporary JSONL file | COMMENT |
| MEDIUM | tests/core/data/test_load_strategy.py | 602 | # Create a proper Dataset object for the mock to return | COMMENT |
| MEDIUM | tests/core/data/test_load_strategy.py | 643 | # Create a proper Dataset object for the mock to return | COMMENT |
| MEDIUM | tests/config/test_config.py | 958 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/config/test_config.py | 994 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/config/test_config.py | 1031 | # Create a temporary config file with invalid work_dir | COMMENT |
| MEDIUM | tests/config/test_config.py | 1060 | # Create a temporary config file | COMMENT |
| MEDIUM | tests/utils/test_model_utils.py | 326 | # Create a mock vllm module | COMMENT |
| MEDIUM | tests/utils/test_datasets_json_compat.py | 113 | # Create a JSONL file with big integer | COMMENT |
| MEDIUM⚡ | tests/utils/test_fingerprint_utils.py | 222 | # Create an object that dill cannot serialize | COMMENT |
| MEDIUM | tests/utils/test_fingerprint_utils.py | 299 | # Create a wrapper chain simulating decorators | COMMENT |
| MEDIUM⚡ | tests/utils/test_ckpt_utils.py | 416 | # Create a mock checkpoint file | COMMENT |
| MEDIUM⚡ | tests/utils/test_ckpt_utils.py | 495 | # Create a corrupted checkpoint file | COMMENT |
| MEDIUM | tests/utils/test_ckpt_utils.py | 602 | # Create a file that starts with checkpoint_op_ and ends with | COMMENT |
| MEDIUM | tests/utils/test_ckpt_utils.py | 679 | # Create a mock checkpoint file so path exists | COMMENT |
| MEDIUM | tests/utils/test_lazy_loader.py | 305 | # Create an empty uv.lock | COMMENT |
| MEDIUM | tests/utils/test_lazy_loader.py | 320 | # Create an invalid uv.lock | COMMENT |
| MEDIUM | tests/utils/test_constant.py | 24 | # Create a temporary jsonl file for testing | COMMENT |
| MEDIUM | tests/utils/test_constant.py | 40 | # Create a temporary config with the test dataset path | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 25 | # Create a test file with complex extension | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 29 | # Create a directory structure for relative path testing | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 33 | # Create a test JSONL file in the relative path directory | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 85 | # Create a temp directory with the name that matches the error | COMMENT |
| MEDIUM⚡ | tests/format/test_load.py | 89 | # Create the file with the name from the error | COMMENT |
| MEDIUM⚡ | tests/format/test_text_formatter.py | 129 | # Create a minimal .docx in memory | COMMENT |
| MEDIUM | tests/ops/test_op_env.py | 780 | # Create a temporary Python file | STRING |
| MEDIUM⚡ | tests/ops/mapper/test_python_lambda_mapper.py | 46 | PythonLambdaMapper("lambda x, y: {'sum': x + y}") # Creating a lambda accepts two arguments | CODE |
| MEDIUM | tests/ops/mapper/test_image_mmpose_mapper.py | 24 | # Create a temporary file for logging | COMMENT |
| MEDIUM⚡ | …/ops/mapper/test_video_atomic_action_segment_mapper.py | 76 | # Create a speed profile with clear minima | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 95 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 170 | # Create a sample | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 173 | # Create an annotation with preference for the left option (answer1) | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 189 | # Create a sample | COMMENT |
| MEDIUM⚡ | …/annotation/test_human_preference_annotation_mapper.py | 192 | # Create an annotation with preference for the right option (answer2) | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 230 | # Create a sample with custom keys | COMMENT |
| MEDIUM | …/annotation/test_human_preference_annotation_mapper.py | 238 | # Create a dictionary version of the sample | COMMENT |
| MEDIUM⚡ | tests/ops/mapper/annotation/test_annotation_mapper.py | 302 | # Create a mapper with a very short timeout | COMMENT |
| MEDIUM⚡ | tests/ops/mapper/annotation/test_annotation_mapper.py | 305 | # Create a task but don't add annotations | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 89 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 421 | # Create a dictionary version of samples (column-oriented) | COMMENT |
| MEDIUM | tests/ops/mapper/annotation/test_annotation_mapper.py | 435 | self.mock_ls_client = MagicMock() # Create a mock client | CODE |
| MEDIUM | data_juicer/tools/__init__.py | 5 | # This file is used for package installation. Script of tools will be | COMMENT |
| MEDIUM | data_juicer/core/ray_exporter.py | 90 | # Create a PyArrow filesystem when export_path points to remote | COMMENT |
| MEDIUM | data_juicer/core/data/load_strategy.py | 101 | # Create the lookup key | COMMENT |
| MEDIUM | data_juicer/utils/video_utils.py | 464 | # Create a named temporary file that will be automatically deleted when closed | COMMENT |
| MEDIUM | data_juicer/utils/lazy_loader.py | 302 | # Create a temporary directory for cloning | COMMENT |
| MEDIUM⚡ | data_juicer/utils/registry.py | 22 | """This class is used to register some modules to registry by a repo | STRING |
| MEDIUM | data_juicer/utils/constant.py | 284 | # Create a stream reader for the file and decode the | COMMENT |
| 8 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/check_s3_integration.py | 140 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 324 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 507 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 690 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 777 | # Check if credentials are available | COMMENT |
| LOW | tools/check_s3_integration.py | 870 | # Check if credentials are available | COMMENT |
| LOW | tools/check_ray_cluster.py | 43 | # Check if head node participates in workload | COMMENT |
| LOW | tools/check_ray_cluster.py | 65 | # Check if this is the head node by comparing node IDs | COMMENT |
| LOW | tools/generate_smtp_cert.py | 157 | # Check if OpenSSL is installed | COMMENT |
| LOW | …m_eval/inception_metrics/video_metrics/metric_utils.py | 306 | # Check if the file exists (all processes must agree). | COMMENT |
| LOW⚡ | tools/humanops/label_studio_service.py | 294 | # Set permissions to 777 (rwxrwxrwx) to ensure the container can write to it | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 347 | # Check if volume exists, if not create it | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 385 | # Set user to root for ensuring write permissions inside container | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 414 | # Check if there's a specific error message we can extract | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 426 | # Check if the container was partially created and clean it up | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 461 | # Check if container is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 619 | # Check if container is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 764 | # Check if process is still running | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 786 | # Check if the server is responding | COMMENT |
| LOW | tools/humanops/label_studio_service.py | 985 | # Check if Docker is installed and running | STRING |
| LOW⚡ | tests/core/executor/test_pipeline_dag.py | 60 | # Assign nodes to DAG | COMMENT |
| LOW | tests/core/executor/test_partitioned_integration.py | 135 | # Check if deduplicator config exists | COMMENT |
| LOW | …/annotation/test_human_preference_annotation_mapper.py | 293 | # Check if each predefined ID is in the sample-to-task mapping | COMMENT |
| LOW | tests/ops/mapper/annotation/test_annotation_mapper.py | 368 | # Check if each predefined ID is in the sample-to-task mapping | COMMENT |
| LOW | …cs/op_doc_enhance_workflow/check_tools/check_render.py | 86 | # Check if it's just "not available" | COMMENT |
| LOW | data_juicer/core/exporter.py | 92 | # Check if export_path is S3 and create storage_options if needed | COMMENT |
| LOW | data_juicer/core/exporter.py | 143 | # Check if export_path is HDFS. Wrap PyArrow HadoopFileSystem via | COMMENT |
| LOW | data_juicer/core/ray_exporter.py | 156 | # Check if dataset is empty by calling columns() regardless of columns parameter | COMMENT |
| LOW | data_juicer/core/tracer/tracer.py | 92 | # Check if sample has changed | COMMENT |
| LOW | data_juicer/core/tracer/ray_tracer.py | 87 | # Check if sample has changed | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 866 | # Check if user provided a job_id (indicating resumption attempt) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 2291 | # Check if the current work_dir already contains the job_id | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 2299 | # Check if this directory has events files (indicating it's a work directory) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 2343 | # Check if config validation passed (done during config initialization) | COMMENT |
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 2348 | # Check if job is already completed | COMMENT |
| LOW⚡ | data_juicer/core/executor/event_logging_mixin.py | 434 | # Check if DAGExecutionMixin is available and has the method to get DAG node | COMMENT |
| LOW | data_juicer/core/executor/event_logging_mixin.py | 1056 | # Check if job has any completion events | COMMENT |
| LOW | data_juicer/core/executor/event_logging_mixin.py | 1065 | # Check if all partitions are completed (success or failure) | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 79 | # Check if DAG execution is enabled (default: True for distributed executors, False for standalone) | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 439 | # Check if we have per-op metrics | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 777 | # Check if all dependencies are completed | COMMENT |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 904 | # Check if this node was already completed | COMMENT |
| LOW | data_juicer/core/data/data_validator.py | 336 | # Check if fields exist in dataset | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 470 | # Check if open_json is available (PyArrow 20.0.0+) | COMMENT |
| LOW | data_juicer/core/data/ray_dataset.py | 581 | # Check if open_json is available (PyArrow 20.0.0+) | COMMENT |
| LOW | data_juicer/analysis/correlation_analysis.py | 115 | # Loop over the data and create a `Text` for each "pixel". | COMMENT |
| LOW | data_juicer/config/config.py | 1032 | # Check if this is a resumption attempt by looking for existing job directory | COMMENT |
| LOW | data_juicer/config/config.py | 2072 | # Check if {job_id} is at the end of the path | COMMENT |
| LOW | data_juicer/utils/model_utils.py | 904 | # Check if we need accelerate for device_map | COMMENT |
| LOW | data_juicer/utils/model_utils.py | 1204 | # Check if we need accelerate for device_map | COMMENT |
| LOW | data_juicer/utils/constant.py | 280 | # Open the file in binary mode and | COMMENT |
| LOW | data_juicer/utils/common_utils.py | 173 | # Check if version is provided when using as @deprecated (without parameters) | COMMENT |
| LOW | data_juicer/utils/job/common.py | 342 | # Check if processes are still running | COMMENT |
| LOW | data_juicer/ops/mixins.py | 278 | # Check if notifications are enabled for this operator | COMMENT |
| LOW | data_juicer/ops/mixins.py | 299 | # Check if notifications are enabled | COMMENT |
| LOW | data_juicer/ops/mixins.py | 305 | # Check if notification handlers are initialized | COMMENT |
| LOW | data_juicer/ops/mixins.py | 330 | # Check if this specific channel is enabled | COMMENT |
| LOW | data_juicer/ops/base_op.py | 125 | # Check if collection is already complete (early exit for performance) | COMMENT |
| LOW | data_juicer/ops/base_op.py | 198 | # Check if collection is already complete (early exit for performance) | COMMENT |
| LOW | data_juicer/ops/mapper/s3_upload_file_mapper.py | 156 | # Check if file exists locally | COMMENT |
| 11 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/converter/modeling_megatron_llama.py | 58 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 550 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 716 | CODE | |
| LOW | tools/converter/modeling_megatron_llama.py | 876 | CODE | |
| LOW | …s/mm_eval/inception_metrics/calc_metrics_for_videos.py | 20 | CODE | |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 54 | CODE | |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 73 | CODE | |
| LOW | …ps/mapper/test_video_human_tracks_extraction_mapper.py | 60 | CODE | |
| LOW | …pper/test_video_captioning_from_human_tracks_mapper.py | 75 | CODE | |
| LOW | scripts/dlc/partition_data_dlc.py | 20 | CODE | |
| LOW | data_juicer/core/exporter.py | 19 | CODE | |
| LOW | data_juicer/core/elasticjuicer/adaptive_mapper.py | 111 | CODE | |
| LOW | data_juicer/core/elasticjuicer/batch_controller.py | 50 | CODE | |
| LOW | data_juicer/core/executor/gpu_memory_probe.py | 825 | CODE | |
| LOW | data_juicer/core/data/ray_dataset.py | 541 | CODE | |
| LOW | data_juicer/utils/model_utils.py | 1863 | CODE | |
| LOW | data_juicer/utils/llm_semantic_ops.py | 309 | CODE | |
| LOW | data_juicer/download/wikipedia.py | 722 | CODE | |
| LOW | …r/ops/pipeline/llm_inference_with_ray_vllm_pipeline.py | 26 | CODE | |
| LOW | data_juicer/ops/selector/image_ohem_selector.py | 34 | CODE | |
| LOW | …a_juicer/ops/aggregator/entity_attribute_aggregator.py | 63 | CODE | |
| LOW | …er/ops/aggregator/most_relevant_entities_aggregator.py | 49 | CODE | |
| LOW | …er/ops/aggregator/most_relevant_entities_aggregator.py | 129 | CODE | |
| LOW | data_juicer/ops/aggregator/meta_tags_aggregator.py | 91 | CODE | |
| LOW | data_juicer/ops/aggregator/nested_aggregator.py | 58 | CODE | |
| LOW | data_juicer/ops/mapper/extract_support_text_mapper.py | 51 | CODE | |
| LOW | data_juicer/ops/mapper/pii_redaction_mapper.py | 107 | CODE | |
| LOW | data_juicer/ops/mapper/image_tagging_vlm_mapper.py | 61 | CODE | |
| LOW | …juicer/ops/mapper/dialog_sentiment_detection_mapper.py | 66 | CODE | |
| LOW | …a_juicer/ops/mapper/video_trajectory_overlay_mapper.py | 46 | CODE | |
| LOW | data_juicer/ops/mapper/sentence_augmentation_mapper.py | 33 | CODE | |
| LOW | data_juicer/ops/mapper/image_sam_3d_body_mapper.py | 30 | CODE | |
| LOW | data_juicer/ops/mapper/relation_identity_mapper.py | 48 | CODE | |
| LOW | …ta_juicer/ops/mapper/dialog_intent_detection_mapper.py | 71 | CODE | |
| LOW | …juicer/ops/mapper/dialog_sentiment_intensity_mapper.py | 79 | CODE | |
| LOW | data_juicer/ops/mapper/video_split_by_scene_mapper.py | 60 | CODE | |
| LOW | data_juicer/ops/mapper/agent_insight_llm_mapper.py | 198 | CODE | |
| LOW | data_juicer/ops/mapper/pii_llm_suspect_mapper.py | 346 | CODE | |
| LOW | …ta_juicer/ops/mapper/extract_entity_relation_mapper.py | 162 | CODE | |
| LOW | …r/ops/mapper/video_hand_reconstruction_hawor_mapper.py | 29 | CODE | |
| LOW | data_juicer/ops/mapper/pair_preference_mapper.py | 41 | CODE | |
| LOW | …/ops/mapper/video_whole_body_pose_estimation_mapper.py | 31 | CODE | |
| LOW | data_juicer/ops/mapper/llm_extract_mapper.py | 40 | CODE | |
| LOW | data_juicer/ops/mapper/image_diffusion_mapper.py | 43 | CODE | |
| LOW | data_juicer/ops/mapper/agent_bad_case_signal_mapper.py | 126 | CODE | |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 29 | CODE | |
| LOW | data_juicer/ops/mapper/s3_upload_file_mapper.py | 33 | CODE | |
| LOW | data_juicer/ops/mapper/nlpcda_zh_mapper.py | 32 | CODE | |
| LOW | …_juicer/ops/mapper/video_hand_reconstruction_mapper.py | 38 | CODE | |
| LOW | data_juicer/ops/mapper/dialog_topic_detection_mapper.py | 69 | CODE | |
| LOW | data_juicer/ops/mapper/vggt_mapper.py | 43 | CODE | |
| LOW | …s/mapper/video_camera_calibration_droidcalib_mapper.py | 35 | CODE | |
| LOW | …a_juicer/ops/mapper/extract_entity_attribute_mapper.py | 54 | CODE | |
| LOW | data_juicer/ops/mapper/extract_nickname_mapper.py | 61 | CODE | |
| LOW | data_juicer/ops/mapper/s3_download_file_mapper.py | 35 | CODE | |
| LOW | …_juicer/ops/mapper/video_captioning_from_vlm_mapper.py | 58 | CODE | |
| LOW | data_juicer/ops/mapper/image_captioning_mapper.py | 46 | CODE | |
| LOW | …_juicer/ops/mapper/generate_qa_from_examples_mapper.py | 57 | CODE | |
| LOW | …uicer/ops/mapper/video_captioning_from_video_mapper.py | 54 | CODE | |
| LOW | data_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py | 37 | CODE | |
| 48 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/converter/modeling_megatron_llama.py | 1 | # Some codes are adapted from https://pypi.org/project/transformers | COMMENT |
| LOW | tools/converter/convert_gpt_to_transformers.py | 1 | # Some code here has been modified from: | COMMENT |
| LOW | tools/preprocess/raw_arxiv_to_jsonl.py | 1 | # Part of the code here has been modified from: | COMMENT |
| LOW | tools/preprocess/raw_stackexchange_to_jsonl.py | 1 | # Part of the code here has been modified from: | COMMENT |
| LOW | tools/mm_eval/inception_metrics/util.py | 401 | # Doesn't look like an URL scheme so interpret it as a local filename. | COMMENT |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 1 | # This tool is used to convert the absolute paths or relative paths in | COMMENT |
| LOW | …onversion/multimodal/absolute_path_to_relative_path.py | 21 | # 'images': ['coco/train2017/000000033471.jpg'], | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 21 | # - in json | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_llava.py | 41 | # "value": "The back of the bus features an advertisement." | COMMENT |
| LOW | …data_juicer_format_to_target_format/dj_to_internvid.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 21 | # 'doors lock/ unlock at the same time. <|__dj__eoc|>', | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 41 | # 'time.', | COMMENT |
| LOW | …odal/data_juicer_format_to_target_format/dj_to_mmc4.py | 61 | # {'face_detections': None, | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 21 | # 'tags': '' }} | COMMENT |
| LOW | …l/data_juicer_format_to_target_format/dj_to_wavcaps.py | 41 | # { 'num_captions_per_audio': 1, | COMMENT |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …_juicer_format_to_target_format/dj_to_video_chatgpt.py | 21 | # (exploring spatial, temporal, relationships, and reasoning concepts); | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 21 | # {'videos': ['videos/retrieval/14111B1211bA1F31-E-2YB57--C518FCEBC553abJYFa5541a31C8a57522AYJbYF4aTdfofa112.mp4'], # n | COMMENT |
| LOW | …dal/data_juicer_format_to_target_format/dj_to_youku.py | 41 | # | COMMENT |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 1 | # This tool is used to convert multimodal dataset in Data-Juicer format to a | COMMENT |
| LOW | …al/data_juicer_format_to_target_format/dj_to_msrvtt.py | 21 | # - in jsonl | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 1 | # This tool is used to convert multimodal dataset in WavCaps format to a target | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 21 | # 'caption': 'Someone is ringing a bell.', | COMMENT |
| LOW | …l/source_format_to_data_juicer_format/wavcaps_to_dj.py | 41 | # 'caption': 'An airplane is landing.', | COMMENT |
| LOW | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 1 | # This tool is used to convert multimodal dataset in MSR-VTT format to a | COMMENT |
| LOW | …al/source_format_to_data_juicer_format/msrvtt_to_dj.py | 21 | # - two new fields are added: | COMMENT |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 1 | # This tool is used to convert multimodal dataset in InternVid format to a | COMMENT |
| LOW | …source_format_to_data_juicer_format/internvid_to_dj.py | 21 | # - videos: video paths list, including cut videos according to their timestamps # noqa: E501 | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 1 | # This tool is used to convert multimodal dataset in LLaVA format to a target | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 21 | # "from": "human", | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/llava_to_dj.py | 41 | # Corresponding Data-Juicer format: | COMMENT |
| LOW | …ce_format_to_data_juicer_format/video_chatgpt_to_dj.py | 1 | # This tool is used to convert multimodal dataset in Video_Chatgpt format to a | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 1 | # This tool is used to convert multimodal dataset in MMC4 format to a target | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 21 | # 0.3234919607639313, | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 41 | # {'text': 'When you lock the door using the lock tab on the driver’s door, ' | COMMENT |
| LOW | …odal/source_format_to_data_juicer_format/mmc4_to_dj.py | 61 | # 0.27694183588027954], | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 1 | # This tool is used to convert multimodal dataset in Youku-mPLUG format to a | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 21 | # | COMMENT |
| LOW | …dal/source_format_to_data_juicer_format/youku_to_dj.py | 41 | # | COMMENT |
| LOW | …cer_format_to_target_format/dj_to_ms_swift_sharegpt.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …cer_format_to_target_format/dj_to_ms_swift_sharegpt.py | 21 | # [ | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 21 | # ... | COMMENT |
| LOW | …/data_juicer_format_to_target_format/dj_to_messages.py | 41 | # }, | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 21 | # ... | COMMENT |
| LOW | …ormat_to_target_format/dj_to_llama_factory_sharegpt.py | 41 | # }, | COMMENT |
| LOW | …og/data_juicer_format_to_target_format/dj_to_alpaca.py | 1 | # This tool is used to convert dataset in Data-Juicer format to a | COMMENT |
| LOW | …og/data_juicer_format_to_target_format/dj_to_alpaca.py | 21 | # { | COMMENT |
| LOW | …og/source_format_to_data_juicer_format/alpaca_to_dj.py | 1 | # This tool is used to convert dataset in Alpaca format to a | COMMENT |
| LOW | …og/source_format_to_data_juicer_format/alpaca_to_dj.py | 21 | # { | COMMENT |
| LOW | …ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py | 1 | # This tool is used to convert dataset in ModelScope-Swift ShareGPT format to a | COMMENT |
| LOW | …ormat_to_data_juicer_format/ms_swift_sharegpt_to_dj.py | 21 | # | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 1 | # This tool is used to convert dataset in ModelScope-Swift Messages format to a | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 21 | # }, | COMMENT |
| LOW | …/source_format_to_data_juicer_format/messages_to_dj.py | 41 | # { | COMMENT |
| 44 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 46 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 58 | # Step 2: Run camera calibration | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 61 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 74 | # Step 2: Run camera calibration | COMMENT |
| LOW⚡ | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 33 | # Step 1: Extract real frames from videos | COMMENT |
| LOW | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 47 | # Step 2: Add dummy camera calibration data matching real frame dims | COMMENT |
| LOW⚡ | docs/ProcessData.md | 90 | # Step 1: Language filter | COMMENT |
| LOW⚡ | docs/ProcessData.md | 95 | # Step 2: Conditional dedup — only when dataset is large | COMMENT |
| LOW⚡ | docs/ProcessData.md | 101 | # Step 3: Length filter | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 570 | # Step 1: Validate event logger availability | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 575 | # Step 2: Load DAG events and execution plan | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 581 | # Step 3: Reconstruct node states from plan and events | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 585 | # Step 4: Calculate statistics | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 588 | # Step 5: Determine ready nodes | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 591 | # Step 6: Determine resumption strategy | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 626 | # Step 2: Check if this fits in available memory | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 635 | # Step 3: Ensure sufficient parallelism for large datasets | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 649 | # Step 4: Adjust for data skew | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 656 | # Step 5: Apply final bounds | COMMENT |
| LOW | data_juicer/core/executor/partition_size_optimizer.py | 605 | # Step 1: Calculate dynamic target based on available memory | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 158 | # Step 1: Decode all frames and record their dimensions | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 172 | # Step 2: Check if all frames share the same resolution (typical for a single clip) | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 241 | # Step 3: Write results to tag_dict | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 101 | # Step 1: Decode and preprocess all frames, record original dimensions | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 119 | # Step 2: Batch inference | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 286 | # Step 1: Convert to row-oriented format (list of sample dictionaries) | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 289 | # Step 2: Retrieve or generate unique IDs for each sample | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 296 | # Step 3: Process samples in batches | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 331 | # Step 5: Convert back to column-oriented format efficiently | COMMENT |
| LOW | data_juicer/ops/mapper/annotation/annotation_mapper.py | 317 | # Step 4: Update samples with annotation results | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 37 | # Step 1: Download videos from S3 to local | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 52 | # Step 2: Video filtering - duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 59 | # Step 3: Video filtering - resolution | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 69 | # Step 4: Split videos by duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 79 | # Step 5: Adjust video aspect ratio | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 87 | # Step 6: Upload processed videos to S3 | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 46 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …pper/test_video_camera_calibration_deepcalib_mapper.py | 58 | # Step 2: Run camera calibration | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 61 | # Step 1: Extract frames from videos | COMMENT |
| LOW | …ps/mapper/test_video_camera_calibration_moge_mapper.py | 74 | # Step 2: Run camera calibration | COMMENT |
| LOW⚡ | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 33 | # Step 1: Extract real frames from videos | COMMENT |
| LOW | …ts/ops/mapper/test_video_camera_pose_megasam_mapper.py | 47 | # Step 2: Add dummy camera calibration data matching real frame dims | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 570 | # Step 1: Validate event logger availability | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 575 | # Step 2: Load DAG events and execution plan | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 581 | # Step 3: Reconstruct node states from plan and events | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 585 | # Step 4: Calculate statistics | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 588 | # Step 5: Determine ready nodes | COMMENT |
| LOW⚡ | data_juicer/core/executor/dag_execution_mixin.py | 591 | # Step 6: Determine resumption strategy | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 626 | # Step 2: Check if this fits in available memory | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 635 | # Step 3: Ensure sufficient parallelism for large datasets | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 649 | # Step 4: Adjust for data skew | COMMENT |
| LOW⚡ | data_juicer/core/executor/partition_size_optimizer.py | 656 | # Step 5: Apply final bounds | COMMENT |
| LOW | data_juicer/core/executor/partition_size_optimizer.py | 605 | # Step 1: Calculate dynamic target based on available memory | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 158 | # Step 1: Decode all frames and record their dimensions | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 172 | # Step 2: Check if all frames share the same resolution (typical for a single clip) | COMMENT |
| LOW | …cer/ops/mapper/video_camera_calibration_moge_mapper.py | 241 | # Step 3: Write results to tag_dict | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 101 | # Step 1: Decode and preprocess all frames, record original dimensions | COMMENT |
| LOW | …ps/mapper/video_camera_calibration_deepcalib_mapper.py | 119 | # Step 2: Batch inference | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 286 | # Step 1: Convert to row-oriented format (list of sample dictionaries) | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 289 | # Step 2: Retrieve or generate unique IDs for each sample | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 296 | # Step 3: Process samples in batches | COMMENT |
| LOW⚡ | data_juicer/ops/mapper/annotation/annotation_mapper.py | 331 | # Step 5: Convert back to column-oriented format efficiently | COMMENT |
| LOW | data_juicer/ops/mapper/annotation/annotation_mapper.py | 317 | # Step 4: Update samples with annotation results | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 37 | # Step 1: Download videos from S3 to local | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 52 | # Step 2: Video filtering - duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 59 | # Step 3: Video filtering - resolution | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 69 | # Step 4: Split videos by duration | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 79 | # Step 5: Adjust video aspect ratio | COMMENT |
| LOW⚡ | …s_video_on_ray/configs/s3_video_processing_config.yaml | 87 | # Step 6: Upload processed videos to S3 | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | README.md | 192 | - **[data-juicer-agents](https://github.com/datajuicer/data-juicer-agents)** — DJ Copilot and agentic workflows | CODE |
| MEDIUM | tests/utils/test_model_utils.py | 216 | # Test with processor for embedding model | COMMENT |
| MEDIUM | tests/utils/test_model_utils.py | 365 | # Test embedding model | COMMENT |
| MEDIUM | data_juicer/config/config_all.yaml | 978 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 989 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 1014 | hf_model: 'Qwen/Qwen2.5-0.5B' # Huggingface embedding model name. | CODE |
| MEDIUM | data_juicer/config/config_all.yaml | 1094 | api_or_hf_model: text-embedding-v4 # API or huggingface embedding model name | CODE |
| MEDIUM | …ta_juicer/ops/mapper/extract_entity_relation_mapper.py | 1 | # This OP is modified from light RAG | COMMENT |
| MEDIUM | data_juicer/ops/mapper/tool_success_tagger_mapper.py | 5 | # Configurable success/error patterns for multi-agent and multi-language. | COMMENT |
| MEDIUM | data_juicer/ops/mapper/extract_keyword_mapper.py | 33 | # This prompt is modified from light RAG | COMMENT |
| MEDIUM | data_juicer/ops/mapper/agent_dialog_normalize_mapper.py | 5 | # dialog/text ops. Supports multi-platform, multi-agent tool formats. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/evaluator/gpt_eval/gpt_evaluator.py | 16 | logger = logging.getLogger(__name__) | CODE |
| LOW | tools/mm_eval/inception_metrics/util.py | 120 | def set_cache_dir(path: str) -> None: | CODE |
| LOW | data_juicer/core/__init__.py | 11 | __all__ = [ | CODE |
| LOW | data_juicer/core/executor/__init__.py | 7 | __all__ = ["ExecutorBase", "ExecutorFactory", "DefaultExecutor", "RayExecutor", "PartitionedRayExecutor"] | CODE |
| LOW | data_juicer/core/executor/dag_execution_mixin.py | 668 | def _update_node_states_from_events(self, node_states: Dict[str, Dict[str, Any]], dag_events: List[Any]) -> None: | CODE |
| LOW | data_juicer/core/data/__init__.py | 8 | __all__ = ["DJDataset", "NestedDataset", "wrap_func_with_nested_access", "add_same_content_to_new_column"] | CODE |
| LOW | data_juicer/analysis/__init__.py | 6 | __all__ = [ | CODE |
| LOW | data_juicer/config/__init__.py | 15 | __all__ = [ | CODE |
| LOW | data_juicer/utils/llm_semantic_ops.py | 18 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/utils/job/__init__.py | 18 | __all__ = [ | CODE |
| LOW | data_juicer/format/__init__.py | 9 | __all__ = [ | CODE |
| LOW | data_juicer/ops/__init__.py | 96 | __all__ = [ | CODE |
| LOW | data_juicer/ops/pipeline/__init__.py | 5 | __all__ = ["LLMRayVLLMEnginePipeline", "RayRepartitionPipeline", "VLMRayVLLMEnginePipeline"] | CODE |
| LOW | data_juicer/ops/selector/__init__.py | 8 | __all__ = [ | CODE |
| LOW | data_juicer/ops/selector/image_ohem_selector.py | 19 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/aggregator/__init__.py | 6 | __all__ = ["NestedAggregator", "MetaTagsAggregator", "EntityAttributeAggregator", "MostRelevantEntitiesAggregator"] | CODE |
| LOW | data_juicer/ops/mapper/sentence_augmentation_mapper.py | 10 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/mapper/agent_bad_case_signal_mapper.py | 20 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/mapper/__init__.py | 157 | __all__ = [ | CODE |
| LOW | data_juicer/ops/mapper/sdxl_prompt2prompt_mapper.py | 17 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/grouper/__init__.py | 5 | __all__ = ["KeyValueGrouper", "NaiveGrouper", "NaiveReverseGrouper"] | CODE |
| LOW | data_juicer/ops/common/__init__.py | 14 | __all__ = [ | CODE |
| LOW | data_juicer/ops/filter/text_pair_similarity_filter.py | 14 | logger = logging.getLogger(__name__) | CODE |
| LOW | …/ops/filter/instruction_following_difficulty_filter.py | 11 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/filter/llm_perplexity_filter.py | 15 | logger = logging.getLogger(__name__) | CODE |
| LOW | data_juicer/ops/filter/__init__.py | 62 | __all__ = [ | CODE |
| LOW | data_juicer/ops/deduplicator/__init__.py | 20 | __all__ = [ | CODE |
| LOW | demos/agent/scripts/generate_bad_case_report.py | 87 | def set_report_locale(lang: str) -> None: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | data_juicer/core/data/dj_dataset.py | 59 | Get values from a specific column/field, optionally limited to first k rows. Args: column: Name of | STRING |
| HIGH | data_juicer/core/data/dj_dataset.py | 228 | Get column values from HuggingFace dataset. Args: column: Name of the column to retrieve | STRING |
| HIGH | data_juicer/core/data/data_validator.py | 292 | Convert field type specifications to actual Python type objects. Args: field_types: Dictio | STRING |
| HIGH | data_juicer/core/data/ray_dataset.py | 153 | Get column values from Ray dataset. Args: column: Name of the column to retrieve k: Opt | STRING |
| HIGH | data_juicer/core/data/schema.py | 41 | Map HuggingFace feature type to Python type. Recursively maps nested types (e.g., List[str], Dict[str, int]). | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | label_studio_localhost_connection.json | 4 | "username": "admin@example.com", | CODE |
| LOW | tools/humanops/label_studio_service.py | 947 | args.username = "admin@example.com" | STRING |
| LOW⚡ | tests/ops/test_mixins.py | 479 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 496 | "access_token": "fake_token", | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 516 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/test_mixins.py | 531 | dingtalk={"access_token": "fake_token"}, | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 35 | '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 35 | '<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 41 | 'Welcome to My WebsiteLorem ipsum dolor sit amet, ' | CODE |
| LOW⚡ | tests/ops/mapper/test_clean_html_mapper.py | 41 | 'Welcome to My WebsiteLorem ipsum dolor sit amet, ' | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 28 | <div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafa | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 28 | <div class="sample-card" style="border:1px solid #ddd; padding:12px; margin:8px 0; border-radius:6px; background:#fafafa | CODE |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 35 | *ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All Ri | COMMENT |
| LOW⚡ | docs/operators/mapper/clean_html_mapper.md | 35 | *ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All Ri | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | data_juicer/core/executor/ray_executor_partitioned.py | 2150 | # No post-convergence operations, just return the merged result | COMMENT |
| MEDIUM | data_juicer/core/executor/event_logging_mixin.py | 250 | """Generate a comprehensive status report.""" | STRING |
| MEDIUM | data_juicer/core/executor/partition_size_optimizer.py | 420 | # Calculate percentile-based memory estimates (p90 is more robust than mean) | COMMENT |
| MEDIUM | data_juicer/core/executor/partition_size_optimizer.py | 762 | """Get comprehensive partition recommendations.""" | STRING |
| MEDIUM | data_juicer/config/config.py | 1673 | # If global_parser is available, use it for robust parsing | COMMENT |
| MEDIUM | data_juicer/utils/job/snapshot.py | 401 | """Convert snapshot to JSON-serializable dictionary with comprehensive progress tracking.""" | STRING |
| LOW | data_juicer/ops/op_env.py | 505 | # and if any of them is arbitrary equality, just use it | COMMENT |
| MEDIUM | demos/partition_and_checkpoint/run_demo.py | 262 | """Run the comprehensive demo.""" | STRING |
| MEDIUM | …ty/humanvbench_models/audio_code/wav2vec_age_gender.py | 62 | # model_name = '/mnt1/daoyuan_mm/wav2vec2-large-robust-24-ft-age-gender' | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | docs/ProcessData.md | 154 | export OPENAI_API_KEY="your-api-key" | CODE |
| HIGH | docs/ProcessData_ZH.md | 158 | export OPENAI_API_KEY="your-api-key" | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | data_juicer/core/data/ray_dataset.py | 523 | elif not schema.equals(batch.schema): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/ops/mapper/test_python_file_mapper.py | 13 | "def process_data(sample):\n" | CODE |
| LOW⚡ | tests/ops/mapper/test_python_file_mapper.py | 25 | "def process_data(samples):\n" | CODE |
| LOW⚡ | tests/ops/mapper/test_python_file_mapper.py | 38 | "def process_data(sample):\n" | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | data_juicer/tools/hpo/objects.py | 45 | # feel free to customize the quality scorer, via the following args | COMMENT |