🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
This report summarizes scanner evidence for huggingface/datasets, a Python project with 22,045 GitHub stars. SynthScan examined 99,972 lines of code across 299 source files, recording 2232 pattern matches distributed across 18 syntactic categories. The legacy adjusted pattern signal is 25.6 (Elevated pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 1850 finding(s).
Are imports declared and package/API names verified? 217 finding(s).
Are wrappers, guards, and repeated structures necessary? 1687 finding(s).
Which changes need focused human review? 2232 finding(s).
This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 2232 distinct pattern matches across 18 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_streaming_download_manager.py | 30 | def test_streaming_dl_manager_download_dummy_path(): | CODE |
| LOW | tests/test_streaming_download_manager.py | 36 | def test_streaming_dl_manager_download_dummy_url(): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 51 | def test_streaming_dl_manager_extract_throws(urlpath): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 56 | def test_streaming_dl_manager_download(text_path): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 64 | def test_streaming_dl_manager_download_and_extract_no_extraction_dummy_path(): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 70 | def test_streaming_dl_manager_download_and_extract_no_extraction_dummy_url(): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 76 | def test_streaming_dl_manager_extract(text_gz_path, text_path): | CODE |
| LOW | tests/test_streaming_download_manager.py | 88 | def test_iter_files_in_archive(archive_jsonl, request): | CODE |
| LOW | tests/test_streaming_download_manager.py | 96 | def test_streaming_dl_manager_download_and_extract_with_extraction(text_gz_path, text_path): | CODE |
| LOW | tests/test_streaming_download_manager.py | 111 | def test_streaming_dl_manager_download_and_extract_with_join(input_path, filename, expected_path): | CODE |
| LOW | tests/test_streaming_download_manager.py | 119 | def test_streaming_dl_manager_extract_all_supported_single_file_compression_types( | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 143 | def test_streaming_gg_drive_no_extract(): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 151 | def test_streaming_gg_drive_gzipped(): | CODE |
| LOW⚡ | tests/test_streaming_download_manager.py | 159 | def test_streaming_gg_drive_zipped(): | CODE |
| LOW | tests/test_dataset_list.py | 27 | def test_list_dict_equivalent(self): | CODE |
| LOW | tests/test_dataset_list.py | 39 | def test_variable_list_records(self): # checks if the type can be inferred from the second record | CODE |
| LOW⚡ | tests/test_fingerprint.py | 162 | def test_recurse_hash_for_class(self): | CODE |
| LOW⚡ | tests/test_fingerprint.py | 169 | def test_recurse_hash_for_method(self): | CODE |
| LOW⚡ | tests/test_fingerprint.py | 176 | def test_hash_ipython_function(self): | CODE |
| LOW | tests/test_fingerprint.py | 113 | def test_hash_tokenizer_with_cache(self): | CODE |
| LOW | tests/test_fingerprint.py | 137 | def test_recurse_hash_for_function(self): | CODE |
| LOW | tests/test_fingerprint.py | 150 | def test_hash_ignores_line_definition_of_function(self): | CODE |
| LOW | tests/test_fingerprint.py | 204 | def test_recurse_hash_for_function_with_shuffled_globals(self): | CODE |
| LOW | tests/test_fingerprint.py | 242 | def test_hash_arrow_table_is_independent_of_chunking(self): | CODE |
| LOW | tests/test_fingerprint.py | 300 | def test_set_doesnt_depend_on_order(self): | CODE |
| LOW | tests/test_fingerprint.py | 311 | def test_hash_tiktoken_encoding(self): | CODE |
| LOW | tests/test_fingerprint.py | 339 | def test_hash_torch_generator(self): | CODE |
| LOW | tests/test_fingerprint.py | 367 | def test_hash_torch_compiled_function(self): | CODE |
| LOW | tests/test_fingerprint.py | 380 | def test_hash_torch_compiled_module(self): | CODE |
| LOW | tests/test_fingerprint.py | 396 | def test_move_script_doesnt_change_hash(tmp_path: Path): | CODE |
| LOW | tests/test_fingerprint.py | 421 | def test_fingerprint_in_multiprocessing(): | CODE |
| LOW | tests/test_fingerprint.py | 432 | def test_temp_cache_dir_with_tmpdir_nonexistent(tmp_path, caplog): | CODE |
| LOW | tests/test_fingerprint.py | 480 | def test_temp_cache_dir_with_tmpdir_existing(tmp_path, monkeypatch): | CODE |
| LOW | tests/test_fingerprint.py | 500 | def test_temp_cache_dir_without_tmpdir(monkeypatch): | CODE |
| LOW | tests/test_fingerprint.py | 521 | def test_temp_cache_dir_tmpdir_creation_failure(tmp_path, monkeypatch, caplog): | CODE |
| LOW | tests/test_fingerprint.py | 543 | def test_temp_cache_dir_tmpdir_not_directory(tmp_path, monkeypatch): | CODE |
| LOW | tests/test_fingerprint.py | 560 | def test_fingerprint_when_transform_version_changes(): | CODE |
| LOW | tests/test_version.py | 19 | def test_version_equality_and_hash(other, expected_equality): | CODE |
| LOW | tests/conftest.py | 11 | def pytest_collection_modifyitems(config, items): | CODE |
| LOW⚡ | tests/conftest.py | 45 | def set_update_download_counts_to_false(monkeypatch): | CODE |
| LOW⚡ | tests/conftest.py | 51 | def set_sqlalchemy_silence_uber_warning(monkeypatch): | CODE |
| LOW⚡ | tests/conftest.py | 61 | def zero_time_out_for_remote_code(): | CODE |
| LOW | tests/test_sharding_utils.py | 48 | def test_number_of_shards_in_gen_kwargs(gen_kwargs, expected): | CODE |
| LOW | tests/test_patching.py | 72 | def test_patch_submodule_builtin(): | CODE |
| LOW | tests/test_patching.py | 86 | def test_patch_submodule_missing(): | CODE |
| LOW | tests/test_patching.py | 93 | def test_patch_submodule_missing_builtin(): | CODE |
| LOW | tests/test_patching.py | 104 | def test_patch_submodule_start_and_stop(): | CODE |
| LOW | tests/test_patching.py | 114 | def test_patch_submodule_successive(): | CODE |
| LOW | tests/test_patching.py | 146 | def test_patch_submodule_doesnt_exist(): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 109 | def picklable_map_function_with_indices(x, i): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 113 | def picklable_map_function_with_rank(x, r): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 117 | def picklable_map_function_with_indices_and_rank(x, i, r): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 121 | def picklable_filter_function(x): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 125 | def picklable_filter_function_with_rank(x, r): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 129 | def assert_arrow_metadata_are_synced_with_dataset_features(dataset: Dataset): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 1716 | def func_return_multi_row_pd_dataframe(x): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 1724 | def test_map_return_pl_dataframe(self, in_memory): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 1727 | def func_return_single_row_pl_dataframe(x): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 5153 | def test_dataset_from_dict_with_large_list(): | CODE |
| LOW⚡ | tests/test_arrow_dataset.py | 5161 | def test_dataset_save_to_disk_with_large_list(tmp_path): | CODE |
| 1423 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/packaged_modules/test_fasta.py | 0 | build the chained fsspec uri datasets uses to read a single compressed file. the builder opens files with the streaming- | STRING |
| HIGH | tests/packaged_modules/test_fastq.py | 0 | build the chained fsspec uri datasets uses to read a single compressed file. the builder opens files with the streaming- | STRING |
| HIGH | tests/packaged_modules/test_genbank.py | 0 | build the chained fsspec uri datasets uses to read a single compressed file. the builder opens files with the streaming- | STRING |
| HIGH | tests/packaged_modules/test_fasta.py | 0 | columns= applies to a user-supplied features schema too. | STRING |
| HIGH | tests/packaged_modules/test_fastq.py | 0 | columns= applies to a user-supplied features schema too. | STRING |
| HIGH | tests/packaged_modules/test_genbank.py | 0 | columns= applies to a user-supplied features schema too. | STRING |
| HIGH | src/datasets/io/parquet.py | 0 | writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle | STRING |
| HIGH | src/datasets/io/csv.py | 0 | writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle | STRING |
| HIGH | src/datasets/io/json.py | 0 | writes the pyarrow table as json lines to a binary file handle. caller is responsible for opening and closing the handle | STRING |
| HIGH | src/datasets/features/bio_structure.py | 0 | encode example into a format for arrow. args: value (`str`, `bytes`, `bio.seqrecord.seqrecord` or `dict`): data passed a | STRING |
| HIGH | src/datasets/features/pdf.py | 0 | encode example into a format for arrow. args: value (`str`, `bytes`, `bio.seqrecord.seqrecord` or `dict`): data passed a | STRING |
| HIGH | src/datasets/features/nifti.py | 0 | encode example into a format for arrow. args: value (`str`, `bytes`, `bio.seqrecord.seqrecord` or `dict`): data passed a | STRING |
| HIGH | src/datasets/features/bio_sequence.py | 0 | encode example into a format for arrow. args: value (`str`, `bytes`, `bio.seqrecord.seqrecord` or `dict`): data passed a | STRING |
| HIGH | src/datasets/features/bio_structure.py | 0 | decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a | STRING |
| HIGH | src/datasets/features/pdf.py | 0 | decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a | STRING |
| HIGH | src/datasets/features/nifti.py | 0 | decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a | STRING |
| HIGH | src/datasets/features/video.py | 0 | decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a | STRING |
| HIGH | src/datasets/features/image.py | 0 | decode example image file into image data. args: value (`str` or `dict`): a string with the absolute image file path, a | STRING |
| HIGH | src/datasets/features/bio_structure.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/pdf.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/mesh.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/nifti.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/video.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/bio_sequence.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/image.py | 0 | if in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary. | STRING |
| HIGH | src/datasets/features/bio_structure.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/pdf.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/mesh.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/audio.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/nifti.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/video.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/bio_sequence.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/image.py | 0 | cast an arrow array to the image arrow storage type. the arrow types that can be converted to the image pyarrow storage | STRING |
| HIGH | src/datasets/features/pdf.py | 0 | embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di | STRING |
| HIGH | src/datasets/features/audio.py | 0 | embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di | STRING |
| HIGH | src/datasets/features/nifti.py | 0 | embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di | STRING |
| HIGH | src/datasets/features/video.py | 0 | embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di | STRING |
| HIGH | src/datasets/features/image.py | 0 | embed image files into the arrow array. args: storage (`pa.structarray`): pyarrow array to embed. token_per_repo_id (`di | STRING |
| HIGH | src/datasets/packaged_modules/conll/conll.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| HIGH | src/datasets/packaged_modules/xml/xml.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| HIGH | src/datasets/packaged_modules/text/text.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| HIGH | src/datasets/packaged_modules/fasta/fasta.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| HIGH | src/datasets/packaged_modules/genbank/genbank.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| HIGH | src/datasets/packaged_modules/fastq/fastq.py | 0 | generate splits from data files. the `data_files` kwarg in load_dataset() can be a str, list[str], dict[str,str], or dic | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM⚡ | tests/features/test_bio.py | 34 | # -------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/features/test_bio.py | 37 | # -------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/features/test_bio.py | 113 | # -------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/features/test_bio.py | 115 | # -------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 266 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 268 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 307 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 309 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 357 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 359 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 486 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 488 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 576 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 578 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 615 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 617 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 633 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 635 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 648 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 650 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 663 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 665 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 678 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_tsfile.py | 680 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 27 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 29 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 42 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 44 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 84 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 86 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 224 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 226 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 455 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 457 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 538 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 540 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 709 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_tsfile.py | 711 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_genbank.py | 844 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | tests/packaged_modules/test_genbank.py | 848 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/hdf5/hdf5.py | 93 | # ┌───────────┐ | COMMENT |
| MEDIUM | src/datasets/packaged_modules/hdf5/hdf5.py | 95 | # └───────────┘ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 139 | # ┌────────────┐ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 141 | # └────────────┘ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 184 | # ┌───────────────────┐ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 186 | # └───────────────────┘ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 207 | # ┌───────────┐ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 209 | # └───────────┘ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 292 | # ┌─────────────┐ | COMMENT |
| MEDIUM⚡ | src/datasets/packaged_modules/hdf5/hdf5.py | 294 | # └─────────────┘ | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 52 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 54 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 137 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 139 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 211 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 213 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 228 | # --------------------------------------------------------------------------- | COMMENT |
| MEDIUM | src/datasets/packaged_modules/tsfile/tsfile.py | 230 | # --------------------------------------------------------------------------- | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/utils.py | 79 | CODE | |
| LOW | tests/utils.py | 93 | CODE | |
| LOW | tests/utils.py | 107 | CODE | |
| LOW | tests/utils.py | 121 | CODE | |
| LOW | tests/utils.py | 269 | CODE | |
| LOW | tests/utils.py | 284 | CODE | |
| LOW | tests/utils.py | 299 | CODE | |
| LOW | tests/utils.py | 314 | CODE | |
| LOW | tests/utils.py | 329 | CODE | |
| LOW | tests/utils.py | 344 | CODE | |
| LOW | tests/utils.py | 359 | CODE | |
| LOW | tests/_test_patching.py | 3 | CODE | |
| LOW | tests/_test_patching.py | 4 | CODE | |
| LOW | tests/_test_patching.py | 5 | CODE | |
| LOW | tests/_test_patching.py | 6 | CODE | |
| LOW | tests/_test_patching.py | 7 | CODE | |
| LOW | tests/_test_patching.py | 8 | CODE | |
| LOW | tests/packaged_modules/test_tsfile.py | 3 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 147 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 150 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 151 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 83 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 86 | CODE | |
| LOW | src/datasets/__init__.py | 17 | CODE | |
| LOW | src/datasets/__init__.py | 17 | CODE | |
| LOW | src/datasets/__init__.py | 18 | CODE | |
| LOW | src/datasets/__init__.py | 19 | CODE | |
| LOW | src/datasets/__init__.py | 19 | CODE | |
| LOW | src/datasets/__init__.py | 19 | CODE | |
| LOW | src/datasets/__init__.py | 19 | CODE | |
| LOW | src/datasets/__init__.py | 20 | CODE | |
| LOW | src/datasets/__init__.py | 20 | CODE | |
| LOW | src/datasets/__init__.py | 21 | CODE | |
| LOW | src/datasets/__init__.py | 21 | CODE | |
| LOW | src/datasets/__init__.py | 22 | CODE | |
| LOW | src/datasets/__init__.py | 23 | CODE | |
| LOW | src/datasets/__init__.py | 24 | CODE | |
| LOW | src/datasets/__init__.py | 24 | CODE | |
| LOW | src/datasets/__init__.py | 24 | CODE | |
| LOW | src/datasets/__init__.py | 25 | CODE | |
| LOW | src/datasets/__init__.py | 26 | CODE | |
| LOW | src/datasets/__init__.py | 26 | CODE | |
| LOW | src/datasets/__init__.py | 26 | CODE | |
| LOW | src/datasets/__init__.py | 26 | CODE | |
| LOW | src/datasets/__init__.py | 26 | CODE | |
| LOW | src/datasets/__init__.py | 33 | CODE | |
| LOW | src/datasets/__init__.py | 33 | CODE | |
| LOW | src/datasets/__init__.py | 34 | CODE | |
| LOW | src/datasets/__init__.py | 34 | CODE | |
| LOW | src/datasets/__init__.py | 34 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 35 | CODE | |
| LOW | src/datasets/__init__.py | 46 | CODE | |
| 121 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_arrow_dataset.py | 155 | CODE | |
| LOW | tests/test_arrow_dataset.py | 591 | CODE | |
| LOW | tests/test_arrow_dataset.py | 938 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1001 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1074 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1186 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1322 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1399 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1407 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1414 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1435 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1513 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1866 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1983 | CODE | |
| LOW | tests/test_arrow_dataset.py | 1992 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2007 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2025 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2069 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2081 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2102 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2123 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2145 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2191 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2220 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2314 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2333 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2363 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2414 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2565 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2597 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2615 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2646 | CODE | |
| LOW | tests/test_arrow_dataset.py | 2914 | CODE | |
| LOW | tests/test_arrow_dataset.py | 4649 | CODE | |
| LOW | tests/test_arrow_reader.py | 200 | CODE | |
| LOW | tests/test_extract.py | 33 | CODE | |
| LOW | tests/test_extract.py | 94 | CODE | |
| LOW | tests/test_dataset_dict.py | 316 | CODE | |
| LOW | tests/features/test_features.py | 550 | CODE | |
| LOW | tests/features/test_array_xd.py | 39 | CODE | |
| LOW | tests/packaged_modules/test_tsfile.py | 52 | CODE | |
| LOW | tests/commands/test_test.py | 34 | CODE | |
| LOW | benchmarks/utils.py | 22 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6654 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6885 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 7019 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 7133 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 7368 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 7426 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 239 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 1781 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 3205 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 3660 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 4505 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 4967 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 5716 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 5884 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 3918 | CODE | |
| LOW | src/datasets/data_files.py | 123 | CODE | |
| LOW | src/datasets/data_files.py | 259 | CODE | |
| 156 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | tests/conftest.py | 56 | except (ModuleNotFoundError, AttributeError): | CODE |
| HIGH | tests/test_arrow_writer.py | 305 | except pa.lib.ArrowInvalid: | CODE |
| HIGH | src/datasets/arrow_dataset.py | 3573 | except NonExistentDatasetError: | CODE |
| HIGH | src/datasets/iterable_dataset.py | 126 | except (TypeError, pa.ArrowInvalid, pa.ArrowNotImplementedError): | CODE |
| HIGH | src/datasets/iterable_dataset.py | 2966 | except DataSourcesShufflingDisallowed: | CODE |
| HIGH | src/datasets/config.py | 67 | except importlib.metadata.PackageNotFoundError: | CODE |
| HIGH | src/datasets/config.py | 78 | except importlib.metadata.PackageNotFoundError: | CODE |
| HIGH | src/datasets/config.py | 55 | except importlib.metadata.PackageNotFoundError: | CODE |
| HIGH | src/datasets/config.py | 126 | except importlib.metadata.PackageNotFoundError: | CODE |
| HIGH | src/datasets/arrow_writer.py | 594 | except Exception: # pyarrow.lib.ArrowInvalid, OSError | CODE |
| HIGH | src/datasets/builder.py | 662 | except ValueError: # Invalid version (ex: incomplete data dir) | CODE |
| HIGH | src/datasets/search.py | 21 | except ImportError: | CODE |
| HIGH | src/datasets/search.py | 26 | except ImportError: | CODE |
| HIGH | src/datasets/load.py | 198 | except Exception: | CODE |
| HIGH | src/datasets/load.py | 610 | except FileNotFoundError: | CODE |
| HIGH | src/datasets/load.py | 702 | except FileNotFoundError: | CODE |
| HIGH | src/datasets/io/json.py | 140 | except (pa.ArrowInvalid, pa.ArrowNotImplementedError, pa.ArrowCapacityError, pa.ArrowMemoryError): | CODE |
| HIGH | src/datasets/packaged_modules/vortex/vortex.py | 321 | except RuntimeError: | CODE |
| HIGH | src/datasets/packaged_modules/tsfile/tsfile.py | 420 | except Exception: | CODE |
| HIGH | src/datasets/packaged_modules/json/json.py | 105 | except FullReadDisallowed: | CODE |
| HIGH | src/datasets/packaged_modules/json/json.py | 273 | except ValueError: | CODE |
| HIGH | src/datasets/packaged_modules/genbank/genbank.py | 585 | except ValueError: | CODE |
| HIGH | src/datasets/utils/_dill.py | 288 | except Exception: | CODE |
| HIGH | src/datasets/download/download_manager.py | 200 | except Exception: | CODE |
| HIGH | src/datasets/formatting/torch_formatter.py | 85 | except ImportError: | CODE |
| HIGH | src/datasets/formatting/jax_formatter.py | 115 | except ImportError: | CODE |
| HIGH | src/datasets/formatting/tf_formatter.py | 79 | except ImportError: | CODE |
| HIGH | src/datasets/formatting/np_formatter.py | 75 | except ImportError: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | src/datasets/arrow_dataset.py | 991 | Convert `dict` to a `pyarrow.Table` to create a [`Dataset`]. Important: a dataset created with from_di | STRING |
| HIGH | src/datasets/arrow_dataset.py | 1157 | Convert a list of dicts to a `pyarrow.Table` to create a [`Dataset`]`. Note that the keys of the first | STRING |
| HIGH | src/datasets/data_files.py | 309 | Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to reso | STRING |
| HIGH | src/datasets/iterable_dataset.py | 3273 | Create an IterableDataset from a polars DataFrame or LazyFrame. Iterating over the dataset is mostly z | STRING |
| HIGH | src/datasets/iterable_dataset.py | 4521 | Enable or disable the dataset features decoding for audio, image, video. When enabled (default), media | STRING |
| HIGH | src/datasets/arrow_reader.py | 538 | Creates a `ReadInstruction` instance out of a string spec. Args: spec (`str`): Spli | STRING |
| HIGH | src/datasets/table.py | 136 | Return the position i of a sorted array so that arr[i] <= x < arr[i+1] Args: arr (`List[int]`): non-em | STRING |
| HIGH | src/datasets/table.py | 654 | Drop one or more columns and return a new table. Args: columns (`List[str]`): | STRING |
| HIGH | src/datasets/table.py | 731 | Convert pandas.DataFrame to an Arrow Table. The column types in the resulting Arrow Table are inferred | STRING |
| HIGH | src/datasets/table.py | 1019 | Drop one or more columns and return a new table. Args: columns (`List[str]`): | STRING |
| HIGH | src/datasets/table.py | 1296 | Drop one or more columns and return a new table. Args: columns (`List[str]`): | STRING |
| HIGH | src/datasets/table.py | 1772 | Drop one or more columns and return a new table. Args: columns (`List[str]`): | STRING |
| HIGH | src/datasets/table.py | 1910 | Improved version of `pa.Array.cast` It supports casting `pa.StructArray` objects to re-order the fields. It als | STRING |
| HIGH | src/datasets/table.py | 2023 | Cast an array to the arrow type that corresponds to the requested feature type. For custom features like [`Audio`] o | STRING |
| HIGH | src/datasets/table.py | 2172 | Embed data into an arrays's storage. For custom features like Audio or Image, it takes into account the "embed_stora | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | tests/test_fingerprint.py | 547 | # Create a regular file and point TMPDIR to it | COMMENT |
| MEDIUM | tests/test_arrow_dataset.py | 4966 | # Create a simple Dataset | COMMENT |
| MEDIUM | tests/test_arrow_dataset.py | 5017 | # Create a Dataset with a column to group by | COMMENT |
| MEDIUM | tests/test_arrow_dataset.py | 5099 | # Create a dataset where one category has more elements than batch_size | COMMENT |
| MEDIUM | tests/test_iterable_dataset.py | 3785 | # Create a simple IterableDataset | COMMENT |
| MEDIUM | tests/test_iterable_dataset.py | 3868 | # Create a Dataset with a column to group by | COMMENT |
| MEDIUM | tests/test_iterable_dataset.py | 3950 | # Create a dataset where one category has more elements than batch_size | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_hdf5.py | 259 | # Create a dataset with a zero dimension | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_hdf5.py | 261 | # Create a dataset with zero in the middle dimension | COMMENT |
| MEDIUM⚡ | tests/packaged_modules/test_hdf5.py | 263 | # Create a dataset with zero in the last dimension | COMMENT |
| MEDIUM | .github/workflows/release-conda.yml | 38 | # Create the build environment | COMMENT |
| MEDIUM | src/datasets/arrow_dataset.py | 2283 | # Create the new feature | COMMENT |
| MEDIUM | src/datasets/arrow_writer.py | 324 | """This function is called when calling pa.array(typed_sequence)""" | STRING |
| MEDIUM | src/datasets/arrow_reader.py | 129 | # Create the absolute instruction (per split) | COMMENT |
| MEDIUM | src/datasets/builder.py | 891 | # Create a tmp dir and rename to self._output_dir on successful exit. | COMMENT |
| MEDIUM | src/datasets/builder.py | 1043 | # Create a dataset for each of the given splits | COMMENT |
| MEDIUM | src/datasets/builder.py | 1167 | # Create a dataset for each of the given splits | COMMENT |
| MEDIUM | src/datasets/splits.py | 240 | # Create the slice for each of the elements | COMMENT |
| MEDIUM | src/datasets/load.py | 1666 | # Create a dataset builder | COMMENT |
| MEDIUM | src/datasets/formatting/formatting.py | 371 | # Create a copy and avoid triggering descriptors | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | setup.py | 22 | {"run": {"name": "Install `datasets@main`", "command": 'pip uninstall datasets -y && pip install "datasets @ git+ | STRING |
| HIGH | tests/test_arrow_dataset.py | 3579 | # Arrow null, not as the JSON string "null". Otherwise null_count is wrong and a missing | COMMENT |
| HIGH | tests/test_table.py | 157 | assert table.equals(in_memory_pa_table) | CODE |
| HIGH | tests/packaged_modules/test_json.py | 428 | "arguments": json.dumps({"command": "pwd && date -u +%Y-%m-%dT%H:%M:%SZ"}), | CODE |
| HIGH | tests/packaged_modules/test_json.py | 163 | "col_2": [null, 2, 20] | STRING |
| HIGH | src/datasets/arrow_writer.py | 754 | f"Couldn't write column '{col}': its type in this file is null, because the first " | CODE |
| HIGH | src/datasets/table.py | 1957 | # Ensure each null value in the array translates to [null] * pa_type.list_size in the array's values | COMMENT |
| HIGH | src/datasets/table.py | 2090 | # Ensure each null value in the array translates to [null] * pa_type.list_size in the array's va | COMMENT |
| HIGH | src/datasets/table.py | 266 | return self.table.equals(*args, **kwargs) | CODE |
| HIGH | src/datasets/table.py | 484 | return self.equals(other) | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_data_files.py | 81 | path.relative_to(*other) | COMMENT |
| LOW | utils/release.py | 1 | # Copyright 2021 The HuggingFace Team. All rights reserved. | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 7201 | lengths = np.delete(lengths, exhausted_indices).tolist() | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 7221 | # For example, if the max_length is 5 and the i-th dataset has 3 samples, the i-th column will be [0,1,2,0,1] | COMMENT |
| LOW | src/datasets/naming.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/arrow_writer.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/__init__.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/arrow_reader.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/builder.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/inspect.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors. | COMMENT |
| LOW | src/datasets/splits.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/load.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/load.py | 1021 | filename = filename + ".py" | COMMENT |
| LOW | src/datasets/info.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/features/features.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/features/features.py | 2081 | raise TypeError(f"Expected a dict but got a {type(feature)}: {feature}") | COMMENT |
| LOW | …d_modules/folder_based_builder/folder_based_builder.py | 401 | ) | COMMENT |
| LOW | …c/datasets/packaged_modules/imagefolder/imagefolder.py | 21 | BASE_COLUMN_NAME = "image" | COMMENT |
| LOW | …c/datasets/packaged_modules/audiofolder/audiofolder.py | 21 | BASE_COLUMN_NAME = "audio" | COMMENT |
| LOW | src/datasets/utils/logging.py | 1 | # Copyright 2020 Optuna, Hugging Face | COMMENT |
| LOW | src/datasets/utils/version.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/utils/__init__.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/utils/py_utils.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/utils/_filelock.py | 1 | #!/usr/bin/env python | COMMENT |
| LOW | src/datasets/utils/_dill.py | 1 | # Copyright 2023 The HuggingFace Team. All rights reserved. | COMMENT |
| LOW | src/datasets/utils/_dill.py | 301 | """ | COMMENT |
| LOW | src/datasets/utils/_dill.py | 401 | # Filenames of functions created in ipykernel the filename | COMMENT |
| LOW | src/datasets/utils/tf_utils.py | 1 | # Copyright 2022 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/download/download_manager.py | 1 | # Copyright 2020 The TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/formatting/polars_formatter.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/formatting/__init__.py | 1 | # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors. | COMMENT |
| LOW | src/datasets/formatting/formatting.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/formatting/torch_formatter.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/formatting/jax_formatter.py | 1 | # Copyright 2021 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/formatting/tf_formatter.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| LOW | src/datasets/formatting/np_formatter.py | 1 | # Copyright 2020 The HuggingFace Authors. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_builder.py | 194 | "dummy_kwarg_with_different_length": [f"dummy_data{i}.txt" for i in range(3)], | CODE |
| LOW | tests/test_builder.py | 215 | "dummy_kwarg_with_different_length": [f"dummy_data{i}.txt" for i in range(3)], | CODE |
| LOW | tests/test_data_files.py | 27 | _FILES_TO_IGNORE = {".dummy", "README.md", "dummy_data.zip", "dataset_infos.json"} | CODE |
| LOW | tests/features/test_array_xd.py | 40 | dummy_data = [] | CODE |
| LOW | tests/features/test_array_xd.py | 59 | dummy_data.append((i, example)) | CODE |
| LOW | tests/features/test_array_xd.py | 61 | return dummy_data | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 365 | @pytest.mark.parametrize("dtype, dummy_value", [("int32", 1), ("bool", True), ("float64", 1)]) | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 366 | def test_table_to_pandas(dtype, dummy_value): | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 368 | dataset = datasets.Dataset.from_dict({"foo": [[[dummy_value] * 2] * 2]}, features=features) | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 372 | np.testing.assert_equal(arr, np.array([[[dummy_value] * 2] * 2], dtype=np.dtype(dtype))) | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 375 | @pytest.mark.parametrize("dtype, dummy_value", [("int32", 1), ("bool", True), ("float64", 1)]) | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 376 | def test_array_xd_numpy_arrow_extractor(dtype, dummy_value): | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 378 | dataset = datasets.Dataset.from_dict({"foo": [[[dummy_value] * 2] * 2]}, features=features) | CODE |
| LOW⚡ | tests/features/test_array_xd.py | 381 | np.testing.assert_equal(arr, np.array([[[dummy_value] * 2] * 2], dtype=np.dtype(dtype))) | CODE |
| LOW⚡ | tests/packaged_modules/test_text.py | 19 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma | STRING |
| LOW⚡ | tests/packaged_modules/test_text.py | 19 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma | STRING |
| LOW⚡ | tests/packaged_modules/test_text.py | 25 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma | STRING |
| LOW⚡ | tests/packaged_modules/test_text.py | 25 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore ma | STRING |
| LOW | benchmarks/utils.py | 23 | dummy_data = [] | CODE |
| LOW⚡ | benchmarks/utils.py | 42 | dummy_data.append((i, example)) | CODE |
| LOW⚡ | benchmarks/utils.py | 44 | return dummy_data | CODE |
| LOW⚡ | benchmarks/utils.py | 48 | dummy_data = generate_examples(features, num_examples=num_examples, seq_shapes=seq_shapes) | CODE |
| LOW⚡ | benchmarks/utils.py | 51 | for key, record in dummy_data: | CODE |
| LOW | benchmarks/benchmark_array_xd.py | 25 | def write(my_features, dummy_data, tmp_dir): | CODE |
| LOW | benchmarks/benchmark_array_xd.py | 27 | for key, record in dummy_data: | CODE |
| LOW | src/datasets/data_files.py | 114 | "dummy_data.zip", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_arrow_dataset.py | 1170 | CODE | |
| LOW | tests/test_dataset_dict.py | 341 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6654 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6816 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 336 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 1482 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 3205 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 3660 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 4134 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 4967 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6068 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6315 | CODE | |
| LOW | src/datasets/arrow_dataset.py | 6395 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 1378 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 3676 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 3771 | CODE | |
| LOW | src/datasets/iterable_dataset.py | 5305 | CODE | |
| LOW | src/datasets/dataset_dict.py | 2515 | CODE | |
| LOW | src/datasets/dataset_dict.py | 2687 | CODE | |
| LOW | src/datasets/dataset_dict.py | 814 | CODE | |
| LOW | src/datasets/dataset_dict.py | 986 | CODE | |
| LOW | src/datasets/dataset_dict.py | 1623 | CODE | |
| LOW | src/datasets/dataset_dict.py | 1911 | CODE | |
| LOW | src/datasets/dataset_dict.py | 2327 | CODE | |
| LOW | src/datasets/arrow_writer.py | 517 | CODE | |
| LOW | src/datasets/builder.py | 308 | CODE | |
| LOW | src/datasets/builder.py | 702 | CODE | |
| LOW | src/datasets/search.py | 454 | CODE | |
| LOW | src/datasets/load.py | 1187 | CODE | |
| LOW | src/datasets/load.py | 1352 | CODE | |
| LOW | src/datasets/load.py | 1375 | CODE | |
| LOW | src/datasets/load.py | 1399 | CODE | |
| LOW | src/datasets/load.py | 1423 | CODE | |
| LOW | src/datasets/load.py | 1446 | CODE | |
| LOW | src/datasets/io/generator.py | 9 | CODE | |
| LOW | src/datasets/io/spark.py | 18 | CODE | |
| LOW | src/datasets/packaged_modules/cache/cache.py | 100 | CODE | |
| LOW | src/datasets/utils/py_utils.py | 416 | CODE | |
| LOW | src/datasets/utils/file_utils.py | 327 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | src/datasets/arrow_dataset.py | 5209 | except Exception as error: | CODE |
| LOW | src/datasets/fingerprint.py | 87 | except Exception as e: | CODE |
| LOW | src/datasets/arrow_writer.py | 594 | except Exception: # pyarrow.lib.ArrowInvalid, OSError | CODE |
| LOW | src/datasets/builder.py | 1567 | except Exception as e: | CODE |
| LOW | src/datasets/builder.py | 1657 | except Exception: | CODE |
| LOW | src/datasets/builder.py | 1873 | except Exception as e: | CODE |
| LOW | src/datasets/inspect.py | 290 | except Exception as err: | CODE |
| LOW | src/datasets/load.py | 198 | except Exception: | CODE |
| LOW | src/datasets/load.py | 1167 | except Exception as e1: | CODE |
| LOW | src/datasets/load.py | 1171 | except Exception: | CODE |
| LOW | src/datasets/features/features.py | 1302 | except Exception: | CODE |
| LOW | src/datasets/features/features.py | 1335 | except Exception: | CODE |
| LOW | src/datasets/packaged_modules/tsfile/tsfile.py | 340 | except Exception as e: | CODE |
| LOW | src/datasets/packaged_modules/tsfile/tsfile.py | 406 | except Exception as e: | CODE |
| LOW | src/datasets/packaged_modules/tsfile/tsfile.py | 420 | except Exception: | CODE |
| LOW | src/datasets/packaged_modules/tsfile/tsfile.py | 494 | except Exception as e: | CODE |
| LOW | src/datasets/packaged_modules/tsfile/tsfile.py | 516 | except Exception as e: | CODE |
| LOW | src/datasets/utils/_dataset_viewer.py | 57 | except Exception as e: # noqa catch any exception of the dataset viewer API and consider the parquet export doesn't | CODE |
| LOW | src/datasets/utils/_dataset_viewer.py | 93 | except Exception as e: # noqa catch any exception of the dataset viewer API and consider the dataset info doesn't e | CODE |
| LOW | src/datasets/utils/_dill.py | 78 | except Exception: # TypeError, decimal.InvalidOperation, etc. | CODE |
| LOW | src/datasets/utils/_dill.py | 141 | except Exception: # TypeError, decimal.InvalidOperation, etc. | CODE |
| LOW | src/datasets/utils/_dill.py | 276 | except Exception: | CODE |
| LOW | src/datasets/utils/_dill.py | 288 | except Exception: | CODE |
| LOW | src/datasets/utils/json.py | 41 | except Exception: | CODE |
| LOW | src/datasets/utils/json.py | 61 | except Exception: | CODE |
| LOW | src/datasets/utils/extract.py | 190 | except Exception: # catch all errors in case future python versions change the zipfile internals | CODE |
| LOW | src/datasets/download/download_manager.py | 200 | except Exception: | CODE |
| LOW | src/datasets/formatting/formatting.py | 530 | except Exception as exc: | CODE |
| LOW | src/datasets/formatting/formatting.py | 549 | except Exception as exc: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_fingerprint.py | 436 | # Set TMPDIR to a non-existent directory | COMMENT |
| LOW | tests/test_fingerprint.py | 484 | # Set TMPDIR to an existing directory | COMMENT |
| LOW | tests/test_fingerprint.py | 527 | # Set TMPDIR to a path that will fail to create (e.g., invalid permissions) | COMMENT |
| LOW | tests/test_buckets.py | 30 | # Write files with full paths relative to the bucket path (forward slashes for MemoryFileSystem) | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 3455 | # Check if we've already cached this computation (indexed by a hash) | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 4578 | # Check if we need to convert indices | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 4792 | # Check if we've already cached this computation (indexed by a hash) | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 4938 | # Check if we've already cached this computation (indexed by a hash) | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 5162 | # Check if we've already cached this computation (indexed by a hash) | COMMENT |
| LOW | src/datasets/fingerprint.py | 52 | # Check if TMPDIR is set and handle the case where it doesn't exist | COMMENT |
| LOW | src/datasets/builder.py | 841 | # Check if the data already exists | COMMENT |
| LOW | src/datasets/packaged_modules/fasta/fasta.py | 254 | # Check if adding this record would exceed byte limit | COMMENT |
| LOW | src/datasets/packaged_modules/genbank/genbank.py | 693 | # Check if adding this record would exceed byte limit | COMMENT |
| LOW | src/datasets/packaged_modules/fastq/fastq.py | 276 | # Check if adding this record would exceed byte limit | COMMENT |
| LOW | src/datasets/utils/file_utils.py | 522 | # Check if the file object is seekable even before reading the magic number (to avoid https://bugs.python.org/issue2 | COMMENT |
| LOW | src/datasets/formatting/formatting.py | 618 | # Check if key is valid | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/test_iterable_dataset.py | 652 | (3, lambda x: {"id+1": x["id"] + 1}, False, None), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 689 | (3, lambda x: {"id+1": x["id"] + 1}, False, None), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 790 | (3, lambda x: {"id+1": x["id"] + 1}, False, None, ["extra_column"]), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1013 | (3, lambda id_: {"id+1": id_ + 1}, False, None, ["id"]), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1050 | (3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1092 | (3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1132 | (3, lambda t: t.append_column("id+1", pc.add(t["id"], 1)), False, None), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1246 | ), # just add 1 to the id | CODE |
| LOW | tests/test_iterable_dataset.py | 1342 | (3, lambda id_: pa.table({"id+1": pc.add(id_, 1)}), False, None, ["id"]), # just add 1 to the id | CODE |
| MEDIUM | src/datasets/arrow_dataset.py | 592 | """To be used in dataset transforms that modify the features of the dataset, in order to update the features stored | STRING |
| LOW | src/datasets/features/audio.py | 144 | # If we already had PCM-byte, we don`t have to make "read file, make bytes" (just use it!) | COMMENT |
| MEDIUM | …d_modules/folder_based_builder/folder_based_builder.py | 319 | # Use block_size equal to the chunk size divided by 32 to leverage multithreading | COMMENT |
| MEDIUM | src/datasets/packaged_modules/json/json.py | 227 | # Use block_size equal to the chunk size divided by 32 to leverage multithreading | COMMENT |
| MEDIUM | src/datasets/utils/track.py | 40 | """Utility class to create an iterable from a generator function, in order to reset the generator when needed.""" | STRING |
| MEDIUM | src/datasets/utils/file_utils.py | 765 | # use xopen instead of fs.open to make data fetching more robust | COMMENT |
| MEDIUM | src/datasets/utils/_dill.py | 316 | # to be robust in case a script is moved for example. | COMMENT |
| MEDIUM | src/datasets/utils/_dill.py | 411 | # to be robust in case a script is moved for example. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | src/datasets/arrow_dataset.py | 3359 | # If the array is empty we do nothing (but we make sure to handle an empty indices mapping and remove the reques | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 4348 | # don't forget to copy the objects | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 4348 | # don't forget to copy the objects | COMMENT |
| LOW | src/datasets/arrow_dataset.py | 6739 | # We make sure to get info from this commit | COMMENT |
| LOW | src/datasets/dataset_dict.py | 2611 | # We make sure to get info from this commit | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | src/datasets/arrow_writer.py | 85 | def set_batch_size(feature: FeatureType) -> None: | CODE |
| LOW | src/datasets/arrow_writer.py | 125 | def set_batch_size(feature: FeatureType) -> None: | CODE |
| LOW | src/datasets/features/__init__.py | 1 | __all__ = [ | CODE |
| LOW | src/datasets/utils/logging.py | 110 | def set_verbosity(verbosity: int) -> None: | CODE |
| LOW | src/datasets/utils/json.py | 132 | def set_json_types_in_feature(feature: "FeatureType", json_field_paths: list[list[str]]) -> None: | CODE |
| LOW | src/datasets/download/__init__.py | 1 | __all__ = [ | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tests/features/test_array_xd.py | 466 | def process_data(batch): | CODE |
| LOW | src/datasets/utils/experimental.py | 26 | ... def my_function(): | STRING |