huggingface/datasets

22.3

Adjusted Score

22.3

Raw Score

100%

Time Factor

2026-07-31

Last Push

21.8K

Stars

Python

Language

91.2K

Lines of Code

274

Files

1.9K

Pattern Hits

2026-08-02

Scan Date

0.18

HC Hit Rate

What These Metrics Mean

Adjusted Score: Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score: The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor: The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits: Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate: High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files: Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 50MEDIUM 79LOW 1746

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 1875 distinct pattern matches across 17 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers1201 hits · 1043 pts

Severity	File	Line	Snippet	Context
LOW	tests/test_streaming_download_manager.py	30	def test_streaming_dl_manager_download_dummy_path():	CODE
LOW	tests/test_streaming_download_manager.py	36	def test_streaming_dl_manager_download_dummy_url():	CODE
LOW⚡	tests/test_streaming_download_manager.py	51	def test_streaming_dl_manager_extract_throws(urlpath):	CODE
LOW⚡	tests/test_streaming_download_manager.py	56	def test_streaming_dl_manager_download(text_path):	CODE
LOW⚡	tests/test_streaming_download_manager.py	64	def test_streaming_dl_manager_download_and_extract_no_extraction_dummy_path():	CODE
LOW⚡	tests/test_streaming_download_manager.py	70	def test_streaming_dl_manager_download_and_extract_no_extraction_dummy_url():	CODE
LOW⚡	tests/test_streaming_download_manager.py	76	def test_streaming_dl_manager_extract(text_gz_path, text_path):	CODE
LOW	tests/test_streaming_download_manager.py	88	def test_iter_files_in_archive(archive_jsonl, request):	CODE
LOW	tests/test_streaming_download_manager.py	96	def test_streaming_dl_manager_download_and_extract_with_extraction(text_gz_path, text_path):	CODE
LOW	tests/test_streaming_download_manager.py	111	def test_streaming_dl_manager_download_and_extract_with_join(input_path, filename, expected_path):	CODE
LOW	tests/test_streaming_download_manager.py	119	def test_streaming_dl_manager_extract_all_supported_single_file_compression_types(	CODE
LOW⚡	tests/test_streaming_download_manager.py	143	def test_streaming_gg_drive_no_extract():	CODE
LOW⚡	tests/test_streaming_download_manager.py	151	def test_streaming_gg_drive_gzipped():	CODE
LOW⚡	tests/test_streaming_download_manager.py	159	def test_streaming_gg_drive_zipped():	CODE
LOW	tests/test_dataset_list.py	27	def test_list_dict_equivalent(self):	CODE
LOW	tests/test_dataset_list.py	39	def test_variable_list_records(self): # checks if the type can be inferred from the second record	CODE
LOW⚡	tests/test_fingerprint.py	162	def test_recurse_hash_for_class(self):	CODE
LOW⚡	tests/test_fingerprint.py	169	def test_recurse_hash_for_method(self):	CODE
LOW⚡	tests/test_fingerprint.py	176	def test_hash_ipython_function(self):	CODE
LOW	tests/test_fingerprint.py	113	def test_hash_tokenizer_with_cache(self):	CODE
LOW	tests/test_fingerprint.py	137	def test_recurse_hash_for_function(self):	CODE
LOW	tests/test_fingerprint.py	150	def test_hash_ignores_line_definition_of_function(self):	CODE
LOW	tests/test_fingerprint.py	204	def test_recurse_hash_for_function_with_shuffled_globals(self):	CODE
LOW	tests/test_fingerprint.py	242	def test_hash_arrow_table_is_independent_of_chunking(self):	CODE
LOW	tests/test_fingerprint.py	300	def test_set_doesnt_depend_on_order(self):	CODE
LOW	tests/test_fingerprint.py	311	def test_hash_tiktoken_encoding(self):	CODE
LOW	tests/test_fingerprint.py	339	def test_hash_torch_generator(self):	CODE
LOW	tests/test_fingerprint.py	367	def test_hash_torch_compiled_function(self):	CODE
LOW	tests/test_fingerprint.py	380	def test_hash_torch_compiled_module(self):	CODE
LOW	tests/test_fingerprint.py	396	def test_move_script_doesnt_change_hash(tmp_path: Path):	CODE
LOW	tests/test_fingerprint.py	421	def test_fingerprint_in_multiprocessing():	CODE
LOW	tests/test_fingerprint.py	432	def test_temp_cache_dir_with_tmpdir_nonexistent(tmp_path, caplog):	CODE
LOW	tests/test_fingerprint.py	480	def test_temp_cache_dir_with_tmpdir_existing(tmp_path, monkeypatch):	CODE
LOW	tests/test_fingerprint.py	500	def test_temp_cache_dir_without_tmpdir(monkeypatch):	CODE
LOW	tests/test_fingerprint.py	521	def test_temp_cache_dir_tmpdir_creation_failure(tmp_path, monkeypatch, caplog):	CODE
LOW	tests/test_fingerprint.py	543	def test_temp_cache_dir_tmpdir_not_directory(tmp_path, monkeypatch):	CODE
LOW	tests/test_fingerprint.py	560	def test_fingerprint_when_transform_version_changes():	CODE
LOW	tests/test_version.py	19	def test_version_equality_and_hash(other, expected_equality):	CODE
LOW	tests/conftest.py	11	def pytest_collection_modifyitems(config, items):	CODE
LOW⚡	tests/conftest.py	45	def set_update_download_counts_to_false(monkeypatch):	CODE
LOW⚡	tests/conftest.py	51	def set_sqlalchemy_silence_uber_warning(monkeypatch):	CODE
LOW⚡	tests/conftest.py	61	def zero_time_out_for_remote_code():	CODE
LOW	tests/test_sharding_utils.py	48	def test_number_of_shards_in_gen_kwargs(gen_kwargs, expected):	CODE
LOW	tests/test_patching.py	72	def test_patch_submodule_builtin():	CODE
LOW	tests/test_patching.py	86	def test_patch_submodule_missing():	CODE
LOW	tests/test_patching.py	93	def test_patch_submodule_missing_builtin():	CODE
LOW	tests/test_patching.py	104	def test_patch_submodule_start_and_stop():	CODE
LOW	tests/test_patching.py	114	def test_patch_submodule_successive():	CODE
LOW	tests/test_patching.py	146	def test_patch_submodule_doesnt_exist():	CODE
LOW⚡	tests/test_arrow_dataset.py	109	def picklable_map_function_with_indices(x, i):	CODE
LOW⚡	tests/test_arrow_dataset.py	113	def picklable_map_function_with_rank(x, r):	CODE
LOW⚡	tests/test_arrow_dataset.py	117	def picklable_map_function_with_indices_and_rank(x, i, r):	CODE
LOW⚡	tests/test_arrow_dataset.py	121	def picklable_filter_function(x):	CODE
LOW⚡	tests/test_arrow_dataset.py	125	def picklable_filter_function_with_rank(x, r):	CODE
LOW⚡	tests/test_arrow_dataset.py	129	def assert_arrow_metadata_are_synced_with_dataset_features(dataset: Dataset):	CODE
LOW⚡	tests/test_arrow_dataset.py	1716	def func_return_multi_row_pd_dataframe(x):	CODE
LOW⚡	tests/test_arrow_dataset.py	1724	def test_map_return_pl_dataframe(self, in_memory):	CODE
LOW⚡	tests/test_arrow_dataset.py	1727	def func_return_single_row_pl_dataframe(x):	CODE
LOW⚡	tests/test_arrow_dataset.py	5058	def test_dataset_from_dict_with_large_list():	CODE
LOW⚡	tests/test_arrow_dataset.py	5066	def test_dataset_save_to_disk_with_large_list(tmp_path):	CODE
1141 more matches not shown…

Decorative Section Separators52 hits · 177 pts

Severity	File	Line	Snippet	Context
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	266	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	268	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	307	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	309	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	357	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	359	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	486	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	488	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	576	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	578	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	615	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	617	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	633	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	635	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	648	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	650	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	663	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	665	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	678	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	tests/packaged_modules/test_tsfile.py	680	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	27	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	29	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	42	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	44	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	84	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	86	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	224	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	226	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	455	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	457	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	538	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	540	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	709	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	tests/packaged_modules/test_tsfile.py	711	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/hdf5/hdf5.py	93	# ┌───────────┐	COMMENT
MEDIUM	src/datasets/packaged_modules/hdf5/hdf5.py	95	# └───────────┘	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	139	# ┌────────────┐	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	141	# └────────────┘	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	184	# ┌───────────────────┐	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	186	# └───────────────────┘	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	207	# ┌───────────┐	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	209	# └───────────┘	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	294	# ┌─────────────┐	COMMENT
MEDIUM⚡	src/datasets/packaged_modules/hdf5/hdf5.py	296	# └─────────────┘	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	52	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	54	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	137	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	139	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	211	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	213	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	228	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	src/datasets/packaged_modules/tsfile/tsfile.py	230	# ---------------------------------------------------------------------------	COMMENT

Unused Imports178 hits · 164 pts

Severity	File	Line	Context
LOW	tests/utils.py	82	CODE
LOW	tests/utils.py	96	CODE
LOW	tests/utils.py	110	CODE
LOW	tests/utils.py	124	CODE
LOW	tests/utils.py	138	CODE
LOW	tests/utils.py	286	CODE
LOW	tests/utils.py	301	CODE
LOW	tests/utils.py	316	CODE
LOW	tests/utils.py	331	CODE
LOW	tests/utils.py	346	CODE
LOW	tests/utils.py	361	CODE
LOW	tests/utils.py	376	CODE
LOW	tests/_test_patching.py	3	CODE
LOW	tests/_test_patching.py	4	CODE
LOW	tests/_test_patching.py	5	CODE
LOW	tests/_test_patching.py	6	CODE
LOW	tests/_test_patching.py	7	CODE
LOW	tests/_test_patching.py	8	CODE
LOW	tests/packaged_modules/test_tsfile.py	3	CODE
LOW	src/datasets/arrow_dataset.py	156	CODE
LOW	src/datasets/arrow_dataset.py	159	CODE
LOW	src/datasets/arrow_dataset.py	160	CODE
LOW	src/datasets/iterable_dataset.py	91	CODE
LOW	src/datasets/iterable_dataset.py	94	CODE
LOW	src/datasets/__init__.py	17	CODE
LOW	src/datasets/__init__.py	17	CODE
LOW	src/datasets/__init__.py	18	CODE
LOW	src/datasets/__init__.py	19	CODE
LOW	src/datasets/__init__.py	19	CODE
LOW	src/datasets/__init__.py	19	CODE
LOW	src/datasets/__init__.py	19	CODE
LOW	src/datasets/__init__.py	20	CODE
LOW	src/datasets/__init__.py	20	CODE
LOW	src/datasets/__init__.py	21	CODE
LOW	src/datasets/__init__.py	21	CODE
LOW	src/datasets/__init__.py	22	CODE
LOW	src/datasets/__init__.py	23	CODE
LOW	src/datasets/__init__.py	24	CODE
LOW	src/datasets/__init__.py	24	CODE
LOW	src/datasets/__init__.py	24	CODE
LOW	src/datasets/__init__.py	25	CODE
LOW	src/datasets/__init__.py	26	CODE
LOW	src/datasets/__init__.py	26	CODE
LOW	src/datasets/__init__.py	26	CODE
LOW	src/datasets/__init__.py	26	CODE
LOW	src/datasets/__init__.py	26	CODE
LOW	src/datasets/__init__.py	33	CODE
LOW	src/datasets/__init__.py	33	CODE
LOW	src/datasets/__init__.py	34	CODE
LOW	src/datasets/__init__.py	34	CODE
LOW	src/datasets/__init__.py	34	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
LOW	src/datasets/__init__.py	35	CODE
118 more matches not shown…

Deep Nesting202 hits · 154 pts

Severity	File	Line	Context
LOW	tests/test_arrow_dataset.py	155	CODE
LOW	tests/test_arrow_dataset.py	591	CODE
LOW	tests/test_arrow_dataset.py	938	CODE
LOW	tests/test_arrow_dataset.py	1001	CODE
LOW	tests/test_arrow_dataset.py	1074	CODE
LOW	tests/test_arrow_dataset.py	1186	CODE
LOW	tests/test_arrow_dataset.py	1322	CODE
LOW	tests/test_arrow_dataset.py	1399	CODE
LOW	tests/test_arrow_dataset.py	1407	CODE
LOW	tests/test_arrow_dataset.py	1414	CODE
LOW	tests/test_arrow_dataset.py	1435	CODE
LOW	tests/test_arrow_dataset.py	1513	CODE
LOW	tests/test_arrow_dataset.py	1866	CODE
LOW	tests/test_arrow_dataset.py	1983	CODE
LOW	tests/test_arrow_dataset.py	1992	CODE
LOW	tests/test_arrow_dataset.py	2007	CODE
LOW	tests/test_arrow_dataset.py	2025	CODE
LOW	tests/test_arrow_dataset.py	2069	CODE
LOW	tests/test_arrow_dataset.py	2081	CODE
LOW	tests/test_arrow_dataset.py	2102	CODE
LOW	tests/test_arrow_dataset.py	2123	CODE
LOW	tests/test_arrow_dataset.py	2145	CODE
LOW	tests/test_arrow_dataset.py	2191	CODE
LOW	tests/test_arrow_dataset.py	2220	CODE
LOW	tests/test_arrow_dataset.py	2314	CODE
LOW	tests/test_arrow_dataset.py	2333	CODE
LOW	tests/test_arrow_dataset.py	2363	CODE
LOW	tests/test_arrow_dataset.py	2398	CODE
LOW	tests/test_arrow_dataset.py	2549	CODE
LOW	tests/test_arrow_dataset.py	2581	CODE
LOW	tests/test_arrow_dataset.py	2599	CODE
LOW	tests/test_arrow_dataset.py	2630	CODE
LOW	tests/test_arrow_dataset.py	2898	CODE
LOW	tests/test_arrow_dataset.py	4554	CODE
LOW	tests/test_arrow_reader.py	200	CODE
LOW	tests/test_extract.py	33	CODE
LOW	tests/test_extract.py	94	CODE
LOW	tests/test_dataset_dict.py	316	CODE
LOW	tests/features/test_features.py	521	CODE
LOW	tests/features/test_array_xd.py	34	CODE
LOW	tests/packaged_modules/test_tsfile.py	52	CODE
LOW	tests/commands/test_test.py	34	CODE
LOW	benchmarks/utils.py	22	CODE
LOW	src/datasets/arrow_dataset.py	6660	CODE
LOW	src/datasets/arrow_dataset.py	6897	CODE
LOW	src/datasets/arrow_dataset.py	7031	CODE
LOW	src/datasets/arrow_dataset.py	7145	CODE
LOW	src/datasets/arrow_dataset.py	7303	CODE
LOW	src/datasets/arrow_dataset.py	7361	CODE
LOW	src/datasets/arrow_dataset.py	248	CODE
LOW	src/datasets/arrow_dataset.py	1790	CODE
LOW	src/datasets/arrow_dataset.py	3214	CODE
LOW	src/datasets/arrow_dataset.py	3669	CODE
LOW	src/datasets/arrow_dataset.py	4514	CODE
LOW	src/datasets/arrow_dataset.py	4971	CODE
LOW	src/datasets/arrow_dataset.py	5720	CODE
LOW	src/datasets/arrow_dataset.py	5888	CODE
LOW	src/datasets/arrow_dataset.py	3927	CODE
LOW	src/datasets/data_files.py	121	CODE
LOW	src/datasets/data_files.py	257	CODE
142 more matches not shown…

Cross-File Repetition26 hits · 130 pts

Severity	File	Snippet	Context
HIGH	src/datasets/io/parquet.py	writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle	STRING
HIGH	src/datasets/io/csv.py	writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle	STRING
HIGH	src/datasets/io/json.py	writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle	STRING
HIGH	src/datasets/features/pdf.py	decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a	STRING
HIGH	src/datasets/features/nifti.py	decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a	STRING
HIGH	src/datasets/features/video.py	decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a	STRING
HIGH	src/datasets/features/image.py	decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a	STRING
HIGH	src/datasets/features/pdf.py	if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.	STRING
HIGH	src/datasets/features/mesh.py	if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.	STRING
HIGH	src/datasets/features/nifti.py	if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.	STRING
HIGH	src/datasets/features/video.py	if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.	STRING
HIGH	src/datasets/features/image.py	if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.	STRING
HIGH	src/datasets/features/pdf.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/mesh.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/audio.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/nifti.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/video.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/image.py	cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage	STRING
HIGH	src/datasets/features/pdf.py	embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di	STRING
HIGH	src/datasets/features/audio.py	embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di	STRING
HIGH	src/datasets/features/nifti.py	embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di	STRING
HIGH	src/datasets/features/video.py	embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di	STRING
HIGH	src/datasets/features/image.py	embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di	STRING
HIGH	src/datasets/packaged_modules/conll/conll.py	the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dict[str,list[str]]. if str or list[	STRING
HIGH	src/datasets/packaged_modules/xml/xml.py	the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dict[str,list[str]]. if str or list[	STRING
HIGH	src/datasets/packaged_modules/text/text.py	the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dict[str,list[str]]. if str or list[	STRING

Docstring Block Structure15 hits · 75 pts

Severity	File	Line	Snippet	Context
HIGH	src/datasets/arrow_dataset.py	1000	Convert `dict` to a `pyarrow.Table` to create a [`Dataset`]. Important: a dataset created with from_di	STRING
HIGH	src/datasets/arrow_dataset.py	1166	Convert a list of dicts to a `pyarrow.Table` to create a [`Dataset`]`. Note that the keys of the first	STRING
HIGH	src/datasets/data_files.py	307	Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to reso	STRING
HIGH	src/datasets/iterable_dataset.py	3061	Create an IterableDataset from a polars DataFrame or LazyFrame. Iterating over the dataset is mostly z	STRING
HIGH	src/datasets/iterable_dataset.py	4299	Enable or disable the dataset features decoding for audio, image, video. When enabled (default), media	STRING
HIGH	src/datasets/arrow_reader.py	538	Creates a `ReadInstruction` instance out of a string spec. Args: spec (`str`): Spli	STRING
HIGH	src/datasets/table.py	136	Return the position i of a sorted array so that arr[i] <= x < arr[i+1] Args: arr (`List[int]`): non-em	STRING
HIGH	src/datasets/table.py	654	Drop one or more columns and return a new table. Args: columns (`List[str]`):	STRING
HIGH	src/datasets/table.py	731	Convert pandas.DataFrame to an Arrow Table. The column types in the resulting Arrow Table are inferred	STRING
HIGH	src/datasets/table.py	1019	Drop one or more columns and return a new table. Args: columns (`List[str]`):	STRING
HIGH	src/datasets/table.py	1296	Drop one or more columns and return a new table. Args: columns (`List[str]`):	STRING
HIGH	src/datasets/table.py	1772	Drop one or more columns and return a new table. Args: columns (`List[str]`):	STRING
HIGH	src/datasets/table.py	1910	Improved version of `pa.Array.cast` It supports casting `pa.StructArray` objects to re-order the fields. It als	STRING
HIGH	src/datasets/table.py	2023	Cast an array to the arrow type that corresponds to the requested feature type. For custom features like [`Audio`] o	STRING
HIGH	src/datasets/table.py	2169	Embed data into an arrays's storage. For custom features like Audio or Image, it takes into account the "embed_stora	STRING

Self-Referential Comments20 hits · 62 pts

Severity	File	Line	Snippet	Context
MEDIUM	tests/test_fingerprint.py	547	# Create a regular file and point TMPDIR to it	COMMENT
MEDIUM	tests/test_arrow_dataset.py	4871	# Create a simple Dataset	COMMENT
MEDIUM	tests/test_arrow_dataset.py	4922	# Create a Dataset with a column to group by	COMMENT
MEDIUM	tests/test_arrow_dataset.py	5004	# Create a dataset where one category has more elements than batch_size	COMMENT
MEDIUM	tests/test_iterable_dataset.py	2990	# Create a simple IterableDataset	COMMENT
MEDIUM	tests/test_iterable_dataset.py	3073	# Create a Dataset with a column to group by	COMMENT
MEDIUM	tests/test_iterable_dataset.py	3155	# Create a dataset where one category has more elements than batch_size	COMMENT
MEDIUM⚡	tests/packaged_modules/test_hdf5.py	259	# Create a dataset with a zero dimension	COMMENT
MEDIUM⚡	tests/packaged_modules/test_hdf5.py	261	# Create a dataset with zero in the middle dimension	COMMENT
MEDIUM⚡	tests/packaged_modules/test_hdf5.py	263	# Create a dataset with zero in the last dimension	COMMENT
MEDIUM	.github/workflows/release-conda.yml	34	# Create the build environment	COMMENT
MEDIUM	src/datasets/arrow_dataset.py	2292	# Create the new feature	COMMENT
MEDIUM	src/datasets/arrow_writer.py	297	"""This function is called when calling pa.array(typed_sequence)"""	STRING
MEDIUM	src/datasets/arrow_reader.py	129	# Create the absolute instruction (per split)	COMMENT
MEDIUM	src/datasets/builder.py	891	# Create a tmp dir and rename to self._output_dir on successful exit.	COMMENT
MEDIUM	src/datasets/builder.py	1043	# Create a dataset for each of the given splits	COMMENT
MEDIUM	src/datasets/builder.py	1167	# Create a dataset for each of the given splits	COMMENT
MEDIUM	src/datasets/splits.py	240	# Create the slice for each of the elements	COMMENT
MEDIUM	src/datasets/load.py	1695	# Create a dataset builder	COMMENT
MEDIUM	src/datasets/formatting/formatting.py	371	# Create a copy and avoid triggering descriptors	COMMENT

Cross-Language Confusion9 hits · 48 pts

Severity	File	Line	Snippet	Context
HIGH	setup.py	22	{"run": {"name": "Install `datasets@main`", "command": 'pip uninstall datasets -y && pip install "datasets @ git+	STRING
HIGH	tests/test_arrow_dataset.py	3563	# Arrow null, not as the JSON string "null". Otherwise null_count is wrong and a missing	COMMENT
HIGH	tests/test_table.py	157	assert table.equals(in_memory_pa_table)	CODE
HIGH	tests/packaged_modules/test_json.py	428	"arguments": json.dumps({"command": "pwd && date -u +%Y-%m-%dT%H:%M:%SZ"}),	CODE
HIGH	tests/packaged_modules/test_json.py	163	"col_2": [null, 2, 20]	STRING
HIGH	src/datasets/table.py	1957	# Ensure each null value in the array translates to [null] * pa_type.list_size in the array's values	COMMENT
HIGH	src/datasets/table.py	2087	# Ensure each null value in the array translates to [null] * pa_type.list_size in the array's va	COMMENT
HIGH	src/datasets/table.py	266	return self.table.equals(args, *kwargs)	CODE
HIGH	src/datasets/table.py	484	return self.equals(other)	CODE

Over-Commented Block35 hits · 34 pts

Severity	File	Line	Snippet	Context
LOW	tests/test_data_files.py	81	path.relative_to(*other)	COMMENT
LOW	utils/release.py	1	# Copyright 2021 The HuggingFace Team. All rights reserved.	COMMENT
LOW	src/datasets/arrow_dataset.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT
LOW	src/datasets/arrow_dataset.py	7221	# Note that we only have 3 examples per dataset since the first dataset ran out of examples	COMMENT
LOW	src/datasets/naming.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/arrow_writer.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/__init__.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/arrow_reader.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/builder.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/inspect.py	1	# Copyright 2020 The HuggingFace Datasets Authors.	COMMENT
LOW	src/datasets/splits.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/load.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/info.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/features/features.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/features/features.py	2021	raise TypeError(f"Expected a dict but got a {type(feature)}: {feature}")	COMMENT
LOW	…d_modules/folder_based_builder/folder_based_builder.py	401	)	COMMENT
LOW	…c/datasets/packaged_modules/imagefolder/imagefolder.py	21	BASE_COLUMN_NAME = "image"	COMMENT
LOW	…c/datasets/packaged_modules/audiofolder/audiofolder.py	21	BASE_COLUMN_NAME = "audio"	COMMENT
LOW	src/datasets/utils/logging.py	1	# Copyright 2020 Optuna, Hugging Face	COMMENT
LOW	src/datasets/utils/version.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/utils/__init__.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/utils/py_utils.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/utils/_filelock.py	1	#!/usr/bin/env python	COMMENT
LOW	src/datasets/utils/_dill.py	1	# Copyright 2023 The HuggingFace Team. All rights reserved.	COMMENT
LOW	src/datasets/utils/_dill.py	301	"""	COMMENT
LOW	src/datasets/utils/_dill.py	401	# Filenames of functions created in ipykernel the filename	COMMENT
LOW	src/datasets/utils/tf_utils.py	1	# Copyright 2022 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/download/download_manager.py	1	# Copyright 2020 The TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/formatting/polars_formatter.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT
LOW	src/datasets/formatting/__init__.py	1	# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.	COMMENT
LOW	src/datasets/formatting/formatting.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT
LOW	src/datasets/formatting/torch_formatter.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT
LOW	src/datasets/formatting/jax_formatter.py	1	# Copyright 2021 The HuggingFace Authors.	COMMENT
LOW	src/datasets/formatting/tf_formatter.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT
LOW	src/datasets/formatting/np_formatter.py	1	# Copyright 2020 The HuggingFace Authors.	COMMENT

Fake / Example Data26 hits · 30 pts

Severity	File	Line	Snippet	Context
LOW	tests/test_builder.py	194	"dummy_kwarg_with_different_length": [f"dummy_data{i}.txt" for i in range(3)],	CODE
LOW	tests/test_builder.py	215	"dummy_kwarg_with_different_length": [f"dummy_data{i}.txt" for i in range(3)],	CODE
LOW	tests/test_data_files.py	27	_FILES_TO_IGNORE = {".dummy", "README.md", "dummy_data.zip", "dataset_infos.json"}	CODE
LOW	tests/features/test_array_xd.py	35	dummy_data = []	CODE
LOW	tests/features/test_array_xd.py	54	dummy_data.append((i, example))	CODE
LOW	tests/features/test_array_xd.py	56	return dummy_data	CODE
LOW⚡	tests/features/test_array_xd.py	360	@pytest.mark.parametrize("dtype, dummy_value", [("int32", 1), ("bool", True), ("float64", 1)])	CODE
LOW⚡	tests/features/test_array_xd.py	361	def test_table_to_pandas(dtype, dummy_value):	CODE
LOW⚡	tests/features/test_array_xd.py	363	dataset = datasets.Dataset.from_dict({"foo": [[[dummy_value] * 2] * 2]}, features=features)	CODE
LOW⚡	tests/features/test_array_xd.py	367	np.testing.assert_equal(arr, np.array([[[dummy_value] * 2] * 2], dtype=np.dtype(dtype)))	CODE
LOW⚡	tests/features/test_array_xd.py	370	@pytest.mark.parametrize("dtype, dummy_value", [("int32", 1), ("bool", True), ("float64", 1)])	CODE
LOW⚡	tests/features/test_array_xd.py	371	def test_array_xd_numpy_arrow_extractor(dtype, dummy_value):	CODE
LOW⚡	tests/features/test_array_xd.py	373	dataset = datasets.Dataset.from_dict({"foo": [[[dummy_value] * 2] * 2]}, features=features)	CODE
LOW⚡	tests/features/test_array_xd.py	376	np.testing.assert_equal(arr, np.array([[[dummy_value] * 2] * 2], dtype=np.dtype(dtype)))	CODE
LOW⚡	tests/packaged_modules/test_text.py	19	Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma	STRING
LOW⚡	tests/packaged_modules/test_text.py	19	Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma	STRING
LOW⚡	tests/packaged_modules/test_text.py	25	Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma	STRING
LOW⚡	tests/packaged_modules/test_text.py	25	Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma	STRING
LOW	benchmarks/utils.py	23	dummy_data = []	CODE
LOW⚡	benchmarks/utils.py	42	dummy_data.append((i, example))	CODE
LOW⚡	benchmarks/utils.py	44	return dummy_data	CODE
LOW⚡	benchmarks/utils.py	48	dummy_data = generate_examples(features, num_examples=num_examples, seq_shapes=seq_shapes)	CODE
LOW⚡	benchmarks/utils.py	51	for key, record in dummy_data:	CODE
LOW	benchmarks/benchmark_array_xd.py	25	def write(my_features, dummy_data, tmp_dir):	CODE
LOW	benchmarks/benchmark_array_xd.py	27	for key, record in dummy_data:	CODE
LOW	src/datasets/data_files.py	112	"dummy_data.zip",	CODE

AI Structural Patterns39 hits · 30 pts

Severity	File	Line	Context
LOW	tests/test_arrow_dataset.py	1170	CODE
LOW	tests/test_dataset_dict.py	341	CODE
LOW	src/datasets/arrow_dataset.py	6660	CODE
LOW	src/datasets/arrow_dataset.py	6825	CODE
LOW	src/datasets/arrow_dataset.py	345	CODE
LOW	src/datasets/arrow_dataset.py	1491	CODE
LOW	src/datasets/arrow_dataset.py	3214	CODE
LOW	src/datasets/arrow_dataset.py	3669	CODE
LOW	src/datasets/arrow_dataset.py	4143	CODE
LOW	src/datasets/arrow_dataset.py	4971	CODE
LOW	src/datasets/arrow_dataset.py	6072	CODE
LOW	src/datasets/arrow_dataset.py	6321	CODE
LOW	src/datasets/arrow_dataset.py	6401	CODE
LOW	src/datasets/iterable_dataset.py	1345	CODE
LOW	src/datasets/iterable_dataset.py	3464	CODE
LOW	src/datasets/iterable_dataset.py	3559	CODE
LOW	src/datasets/iterable_dataset.py	5000	CODE
LOW	src/datasets/dataset_dict.py	2521	CODE
LOW	src/datasets/dataset_dict.py	2696	CODE
LOW	src/datasets/dataset_dict.py	824	CODE
LOW	src/datasets/dataset_dict.py	996	CODE
LOW	src/datasets/dataset_dict.py	1633	CODE
LOW	src/datasets/dataset_dict.py	1923	CODE
LOW	src/datasets/dataset_dict.py	2331	CODE
LOW	src/datasets/arrow_writer.py	490	CODE
LOW	src/datasets/builder.py	308	CODE
LOW	src/datasets/builder.py	702	CODE
LOW	src/datasets/search.py	454	CODE
LOW	src/datasets/load.py	1220	CODE
LOW	src/datasets/load.py	1381	CODE
LOW	src/datasets/load.py	1404	CODE
LOW	src/datasets/load.py	1428	CODE
LOW	src/datasets/load.py	1452	CODE
LOW	src/datasets/load.py	1475	CODE
LOW	src/datasets/io/generator.py	9	CODE
LOW	src/datasets/io/spark.py	18	CODE
LOW	src/datasets/packaged_modules/cache/cache.py	100	CODE
LOW	src/datasets/utils/py_utils.py	416	CODE
LOW	src/datasets/utils/file_utils.py	333	CODE

Excessive Try-Catch Wrapping29 hits · 28 pts

Severity	File	Line	Snippet	Context
LOW	src/datasets/arrow_dataset.py	5213	except Exception as error:	CODE
LOW	src/datasets/fingerprint.py	87	except Exception as e:	CODE
LOW	src/datasets/arrow_writer.py	567	except Exception: # pyarrow.lib.ArrowInvalid, OSError	CODE
LOW	src/datasets/builder.py	1567	except Exception as e:	CODE
LOW	src/datasets/builder.py	1657	except Exception:	CODE
LOW	src/datasets/builder.py	1874	except Exception as e:	CODE
LOW	src/datasets/inspect.py	290	except Exception as err:	CODE
LOW	src/datasets/load.py	206	except Exception:	CODE
LOW	src/datasets/load.py	1200	except Exception as e1:	CODE
LOW	src/datasets/load.py	1204	except Exception:	CODE
LOW	src/datasets/features/features.py	1250	except Exception:	CODE
LOW	src/datasets/features/features.py	1279	except Exception:	CODE
LOW	src/datasets/packaged_modules/tsfile/tsfile.py	340	except Exception as e:	CODE
LOW	src/datasets/packaged_modules/tsfile/tsfile.py	406	except Exception as e:	CODE
LOW	src/datasets/packaged_modules/tsfile/tsfile.py	420	except Exception:	CODE
LOW	src/datasets/packaged_modules/tsfile/tsfile.py	494	except Exception as e:	CODE
LOW	src/datasets/packaged_modules/tsfile/tsfile.py	516	except Exception as e:	CODE
LOW	src/datasets/utils/_dataset_viewer.py	57	except Exception as e: # noqa catch any exception of the dataset viewer API and consider the parquet export doesn't	CODE
LOW	src/datasets/utils/_dataset_viewer.py	93	except Exception as e: # noqa catch any exception of the dataset viewer API and consider the dataset info doesn't e	CODE
LOW	src/datasets/utils/_dill.py	78	except Exception: # TypeError, decimal.InvalidOperation, etc.	CODE
LOW	src/datasets/utils/_dill.py	141	except Exception: # TypeError, decimal.InvalidOperation, etc.	CODE
LOW	src/datasets/utils/_dill.py	276	except Exception:	CODE
LOW	src/datasets/utils/_dill.py	288	except Exception:	CODE
LOW	src/datasets/utils/json.py	41	except Exception:	CODE
LOW	src/datasets/utils/json.py	61	except Exception:	CODE
LOW	src/datasets/utils/extract.py	186	except Exception: # catch all errors in case future python versions change the zipfile internals	CODE
LOW	src/datasets/download/download_manager.py	200	except Exception:	CODE
LOW	src/datasets/formatting/formatting.py	530	except Exception as exc:	CODE
LOW	src/datasets/formatting/formatting.py	549	except Exception as exc:	CODE

AI Slop Vocabulary17 hits · 23 pts

Severity	File	Line	Snippet	Context
LOW	tests/test_iterable_dataset.py	484	(3, lambda x: {"id+1": x["id"] + 1}, False, None), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	521	(3, lambda x: {"id+1": x["id"] + 1}, False, None), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	622	(3, lambda x: {"id+1": x["id"] + 1}, False, None, ["extra_column"]), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	845	(3, lambda id_: {"id+1": id_ + 1}, False, None, ["id"]), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	882	(3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	924	(3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	964	(3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	1078	), # just add 1 to the id	CODE
LOW	tests/test_iterable_dataset.py	1174	(3, lambda id_: pa.table({"id+1": pc.add(id_, 1)}), False, None, ["id"]), # just add 1 to the id	CODE
MEDIUM	src/datasets/arrow_dataset.py	601	"""To be used in dataset transforms that modify the features of the dataset, in order to update the features stored	STRING
LOW	src/datasets/features/audio.py	144	# If we already had PCM-byte, we don`t have to make "read file, make bytes" (just use it!)	COMMENT
MEDIUM	…d_modules/folder_based_builder/folder_based_builder.py	315	# Use block_size equal to the chunk size divided by 32 to leverage multithreading	COMMENT
MEDIUM	src/datasets/packaged_modules/json/json.py	227	# Use block_size equal to the chunk size divided by 32 to leverage multithreading	COMMENT
MEDIUM	src/datasets/utils/track.py	40	"""Utility class to create an iterable from a generator function, in order to reset the generator when needed."""	STRING
MEDIUM	src/datasets/utils/file_utils.py	771	# use xopen instead of fs.open to make data fetching more robust	COMMENT
MEDIUM	src/datasets/utils/_dill.py	316	# to be robust in case a script is moved for example.	COMMENT
MEDIUM	src/datasets/utils/_dill.py	411	# to be robust in case a script is moved for example.	COMMENT

Redundant / Tautological Comments13 hits · 20 pts

Severity	File	Line	Snippet	Context
LOW	tests/test_fingerprint.py	436	# Set TMPDIR to a non-existent directory	COMMENT
LOW	tests/test_fingerprint.py	484	# Set TMPDIR to an existing directory	COMMENT
LOW	tests/test_fingerprint.py	527	# Set TMPDIR to a path that will fail to create (e.g., invalid permissions)	COMMENT
LOW	tests/test_buckets.py	30	# Write files with full paths relative to the bucket path (forward slashes for MemoryFileSystem)	COMMENT
LOW	src/datasets/arrow_dataset.py	3464	# Check if we've already cached this computation (indexed by a hash)	COMMENT
LOW	src/datasets/arrow_dataset.py	4587	# Check if we need to convert indices	COMMENT
LOW	src/datasets/arrow_dataset.py	4801	# Check if we've already cached this computation (indexed by a hash)	COMMENT
LOW	src/datasets/arrow_dataset.py	4943	# Check if we've already cached this computation (indexed by a hash)	COMMENT
LOW	src/datasets/arrow_dataset.py	5166	# Check if we've already cached this computation (indexed by a hash)	COMMENT
LOW	src/datasets/fingerprint.py	52	# Check if TMPDIR is set and handle the case where it doesn't exist	COMMENT
LOW	src/datasets/builder.py	841	# Check if the data already exists	COMMENT
LOW	src/datasets/utils/file_utils.py	528	# Check if the file object is seekable even before reading the magic number (to avoid https://bugs.python.org/issue2	COMMENT
LOW	src/datasets/formatting/formatting.py	618	# Check if key is valid	COMMENT

Slop Phrases5 hits · 8 pts

Severity	File	Line	Snippet	Context
LOW	src/datasets/arrow_dataset.py	3368	# If the array is empty we do nothing (but we make sure to handle an empty indices mapping and remove the reques	COMMENT
LOW	src/datasets/arrow_dataset.py	4357	# don't forget to copy the objects	COMMENT
LOW	src/datasets/arrow_dataset.py	4357	# don't forget to copy the objects	COMMENT
LOW	src/datasets/arrow_dataset.py	6745	# We make sure to get info from this commit	COMMENT
LOW	src/datasets/dataset_dict.py	2617	# We make sure to get info from this commit	COMMENT

Modern Structural Boilerplate6 hits · 6 pts

Severity	File	Line	Snippet	Context
LOW	src/datasets/arrow_writer.py	85	def set_batch_size(feature: FeatureType) -> None:	CODE
LOW	src/datasets/arrow_writer.py	125	def set_batch_size(feature: FeatureType) -> None:	CODE
LOW	src/datasets/features/__init__.py	1	__all__ = [	CODE
LOW	src/datasets/utils/logging.py	110	def set_verbosity(verbosity: int) -> None:	CODE
LOW	src/datasets/utils/json.py	132	def set_json_types_in_feature(feature: "FeatureType", json_field_paths: list[list[str]]) -> None:	CODE
LOW	src/datasets/download/__init__.py	1	__all__ = [	CODE

Overly Generic Function Names2 hits · 2 pts

Severity	File	Line	Snippet	Context
LOW	tests/features/test_array_xd.py	461	def process_data(batch):	CODE
LOW	src/datasets/utils/experimental.py	26	... def my_function():	STRING

Analysis Overview

What These Metrics Mean

Score History

Severity Breakdown

Directory Score Breakdown

Pattern Findings