huggingface/tokenizers

6.5

Adjusted Score

6.5

Raw Score

100%

Time Factor

2026-07-14

Last Push

10.9K

Stars

Rust

Language

66.4K

Lines of Code

291

Files

379

Pattern Hits

2026-07-14

Scan Date

0.00

HC Hit Rate

What These Metrics Mean

Adjusted Score: Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score: The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor: The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits: Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate: High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files: Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 1MEDIUM 25LOW 353

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 379 distinct pattern matches across 10 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Over-Commented Block217 hits · 212 pts

Severity	File	Line	Snippet	Context
LOW	tokenizers/examples/unstable_wasm/src/utils.rs	1	pub fn set_panic_hook() {	COMMENT
LOW	tokenizers/benches/unigram_benchmark.rs	21	// let mut model = Unigram::default();	COMMENT
LOW	tokenizers/benches/ci_benchmark.rs	1	//! Consolidated benchmark suite for CI regression detection.	COMMENT
LOW	tokenizers/src/lib.rs	1	#![cfg_attr(docsrs, feature(doc_cfg))]	COMMENT
LOW	tokenizers/src/lib.rs	21	//! splitting text is simply on whitespace.	COMMENT
LOW	tokenizers/src/lib.rs	41	//! }	COMMENT
LOW	tokenizers/src/lib.rs	61	//!	COMMENT
LOW	tokenizers/src/lib.rs	81	//!	COMMENT
LOW	tokenizers/src/lib.rs	101	//! .with_pre_tokenizer(Some(ByteLevel::default()))	COMMENT
LOW	tokenizers/src/lib.rs	121	//! by the total number of core/threads your CPU provides but this can be tuned by setting the `RAYON_RS_NUM_THREADS`	COMMENT
LOW	tokenizers/src/normalizers/bert.rs	21	// Cc, Cf, Cn or Co	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	81	/// Retrieve the size of the vocabulary	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	801	.map(\|(i, sequence)\| encode(true, i, sequence))	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	841	let pair_encoding = pair	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	861	/// // Pre-tokenized sequences:	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	881	// Encode each sequence	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	901	/// tokenizer.encode("Single sequence", false);	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	961		COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	981	/// decode_stream.step(1246).unwrap(),	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	1001	/// ]);	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	1021	/// // Valid utf-8 again, this corresponds to both tokens: [1, 2]	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	1041	/// .unwrap();	COMMENT
LOW	tokenizers/src/tokenizer/mod.rs	1061	pub struct DecodeStream<'tok, M, N, PT, PP, D> {	COMMENT
LOW	tokenizers/src/tokenizer/added_vocabulary.rs	121	match_.end()	COMMENT
LOW	tokenizers/src/tokenizer/added_vocabulary.rs	141	#[derive(Clone)]	COMMENT
LOW	tokenizers/src/tokenizer/normalizer.rs	101	/// It is possible to retrieve a part of the original string, by indexing it with	COMMENT
LOW	tokenizers/src/tokenizer/normalizer.rs	301	.collect(),	COMMENT
LOW	tokenizers/src/tokenizer/normalizer.rs	421	self.normalized	COMMENT
LOW	tokenizers/src/tokenizer/normalizer.rs	681	}	COMMENT
LOW	tokenizers/src/tokenizer/pre_tokenizer.rs	61	/// Split the `PreTokenizedString` by providing a `split_fn` in charge of splitting	COMMENT
LOW	tokenizers/src/utils/parallelism.rs	61	PARALLELISM.store(if val { 2 } else { 1 }, Ordering::SeqCst);	COMMENT
LOW	tokenizers/src/utils/mod.rs	61	///	COMMENT
LOW	tokenizers/src/utils/mod.rs	81	/// ```should_panic	COMMENT
LOW	tokenizers/src/utils/mod.rs	101	/// # #[macro_use] extern crate tokenizers;	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	41	/// Parity selection variant.	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	261	/// this trainer accepts multiple corpora (one per language) and selects	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	281	///	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	301	/// and accumulates word counts into a single internal map. Parity-aware	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	1001	LengthState {	COMMENT
LOW	tokenizers/src/models/bpe/parity_trainer.rs	1881	// lang 1: "abab" x5, "xyxy" x10	COMMENT
LOW	tokenizers/src/models/bpe/mod.rs	21	JsonError(#[from] serde_json::Error),	COMMENT
LOW	tokenizers/src/models/bpe/trainer.rs	181	/// # Examples	COMMENT
LOW	tokenizers/src/models/bpe/model.rs	21	type VocabR = AHashMap<u32, String>;	COMMENT
LOW	tokenizers/src/models/bpe/model.rs	301	pub(crate) vocab_r: VocabR,	COMMENT
LOW	tokenizers/src/processors/template.rs	1	//! # Template Processing	COMMENT
LOW	tokenizers/src/processors/template.rs	21	//! So, we can define a [`TemplateProcessing`] that will achieve this result:	COMMENT
LOW	tokenizers/src/processors/template.rs	41	//! ```	COMMENT
LOW	tokenizers/src/processors/template.rs	81	///	COMMENT
LOW	tokenizers/src/processors/template.rs	181	/// // Simple cases, where a single id/token is necessary:	COMMENT
LOW	tokenizers/src/processors/template.rs	241	/// # use tokenizers::processors::template::Template;	COMMENT
LOW	tokenizers/src/processors/template.rs	321	/// A `Template` is actually a sequence of `Piece` that will be	COMMENT
LOW	tokenizers/src/processors/template.rs	601	// .take_overflowing()	COMMENT
LOW	tokenizers/src/processors/template.rs	621	// )?);	COMMENT
LOW	tokenizers/src/processors/template.rs	661	// 1 => (	COMMENT
LOW	tokenizers/src/decoders/ctc.rs	1	use crate::decoders::wordpiece;	COMMENT
LOW	docs/source/conf.py	1	# Configuration file for the Sphinx documentation builder.	COMMENT
LOW	docs/source/conf.py	41	# ones.	COMMENT
LOW	bindings/python/tests/documentation/test_quicktour.py	81	print(output.ids)	COMMENT
LOW	bindings/python/tests/documentation/test_quicktour.py	161	# END encode_batch	COMMENT
LOW	bindings/python/src/normalizers.rs	141	})?;	COMMENT
157 more matches not shown…

Unused Imports59 hits · 58 pts

Severity	File	Line	Context
LOW	docs/source/_ext/rust_doc.py	4	CODE
LOW	docs/source/_ext/entities.py	5	CODE
LOW	docs/source/_ext/entities.py	10	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	89	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	105	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	105	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	105	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	105	CODE
LOW	bindings/python/py_src/tokenizers/__init__.py	105	CODE
LOW	bindings/python/py_src/tokenizers/tools/__init__.py	1	CODE
LOW	bindings/python/py_src/tokenizers/tools/__init__.py	1	CODE
LOW	bindings/python/py_src/tokenizers/tools/visualizer.py	2	CODE
LOW	bindings/python/py_src/tokenizers/tools/visualizer.py	5	CODE
LOW	bindings/python/py_src/tokenizers/tools/visualizer.py	6	CODE
LOW	…rc/tokenizers/implementations/sentencepiece_unigram.py	1	CODE
LOW	…rc/tokenizers/implementations/sentencepiece_unigram.py	2	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	1	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	2	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	3	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	4	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	5	CODE
LOW	…s/python/py_src/tokenizers/implementations/__init__.py	6	CODE
LOW	bindings/python/tests/test_freethreaded.py	159	CODE
LOW	bindings/python/tests/test_benchmarks.py	21	CODE
LOW	bindings/python/tests/test_serialization.py	11	CODE
LOW	…ts/documentation/test_tutorial_train_from_iterators.py	8	CODE
LOW	bindings/python/tests/documentation/test_quicktour.py	3	CODE
LOW	bindings/python/tests/documentation/test_pipeline.py	4	CODE
LOW	…gs/python/tests/implementations/test_bert_wordpiece.py	4	CODE
LOW	bindings/python/tests/implementations/test_char_bpe.py	4	CODE
LOW	…gs/python/tests/implementations/test_byte_level_bpe.py	4	CODE
LOW	bindings/python/tests/bindings/test_processors.py	18	CODE
LOW	bindings/python/tests/bindings/test_normalizers.py	6	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	17	CODE
LOW	bindings/python/tests/bindings/test_encoding.py	5	CODE
LOW	bindings/python/tests/bindings/test_models.py	6	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	17	CODE
LOW	bindings/python/scripts/spm_parity_check.py	5	CODE
LOW	bindings/python/scripts/convert.py	2	CODE
LOW	bindings/python/scripts/convert.py	7	CODE
LOW	bindings/python/scripts/convert.py	16	CODE
LOW	bindings/python/scripts/convert.py	16	CODE
LOW	bindings/python/scripts/convert.py	16	CODE
LOW	bindings/python/scripts/convert.py	22	CODE
LOW	bindings/python/scripts/convert.py	23	CODE
LOW	bindings/python/scripts/convert.py	25	CODE

Hyper-Verbose Identifiers55 hits · 56 pts

Severity	File	Line	Snippet	Context
LOW	bindings/python/conftest.py	12	def pytest_collection_modifyitems(config, items):	CODE
LOW	…dings/python/py_src/tokenizers/normalizers/__init__.py	23	def unicode_normalizer_from_str(normalizer: str) -> Normalizer:	CODE
LOW	bindings/python/py_src/tokenizers/tools/visualizer.py	192	def consecutive_chars_to_html(	CODE
LOW	…on/py_src/tokenizers/implementations/base_tokenizer.py	25	def num_special_tokens_to_add(self, is_pair: bool) -> int:	CODE
LOW	bindings/python/tests/test_freethreaded.py	49	def test_encode_while_swapping_post_processor(self):	CODE
LOW	bindings/python/tests/test_freethreaded.py	81	def test_encode_while_mutating_trainer_fields(self):	CODE
LOW	bindings/python/tests/test_freethreaded.py	114	def test_concurrent_setters_no_lock_poisoning(self):	CODE
LOW	bindings/python/tests/test_freethreaded.py	156	def test_gil_actually_disabled_on_import(self):	CODE
LOW	bindings/python/tests/test_benchmarks.py	90	def test_encode_batch_multithreaded(self, benchmark, gpt2_tokenizer, lines):	CODE
LOW	bindings/python/tests/test_benchmarks.py	127	def test_encode_batch_multithreaded(self, benchmark, llama3_tokenizer, lines):	CODE
LOW⚡	bindings/python/tests/test_benchmarks.py	161	def test_async_encode_batch_fast(self, benchmark, llama3_tokenizer, lines):	CODE
LOW	bindings/python/tests/utils.py	70	def doc_pipeline_bert_tokenizer(data_dir):	CODE
LOW	bindings/python/tests/utils.py	78	def multiprocessing_with_parallelism(tokenizer, enabled: bool):	CODE
LOW	bindings/python/tests/test_serialization.py	16	def test_full_serialization_albert(self, albert_base):	CODE
LOW	bindings/python/tests/test_serialization.py	120	def test_full_deserialization_hub(self):	CODE
LOW	…gs/python/tests/implementations/test_bert_wordpiece.py	45	def test_multiprocessing_with_parallelism(self, bert_files):	CODE
LOW	bindings/python/tests/implementations/test_char_bpe.py	52	def test_multiprocessing_with_parallelism(self, openai_files):	CODE
LOW	…ngs/python/tests/implementations/test_sentencepiece.py	31	def test_train_with_unk_token(self, tmpdir):	CODE
LOW	…ngs/python/tests/implementations/test_sentencepiece.py	53	def test_train_from_iterator_with_unk_token(self):	CODE
LOW	…gs/python/tests/implementations/test_byte_level_bpe.py	92	def test_multiprocessing_with_parallelism(self, roberta_files):	CODE
LOW	bindings/python/tests/bindings/test_normalizers.py	30	def test_handle_chinese_chars(self):	CODE
LOW	bindings/python/tests/bindings/test_normalizers.py	93	def test_item_getters_and_setters(self):	CODE
LOW	bindings/python/tests/bindings/test_normalizers.py	230	def test_normalizer_interface(self):	CODE
LOW⚡	bindings/python/tests/bindings/test_tokenizer.py	617	def test_multiprocessing_with_parallelism(self):	CODE
LOW⚡	bindings/python/tests/bindings/test_tokenizer.py	622	def test_multithreaded_concurrency(self):	CODE
LOW⚡	bindings/python/tests/bindings/test_tokenizer.py	1039	def test_async_methods_existence(self):	CODE
LOW⚡	bindings/python/tests/bindings/test_tokenizer.py	1047	async def test_performance_comparison(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	21	def test_instantiate_with_content_only(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	57	def test_can_set_single_world(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	73	def test_has_expected_type_and_methods(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	114	def test_add_tokens_with_normalizer(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	130	def test_normalizer_change_refreshes_added_tokens(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	329	def test_encode_add_special_tokens(self, roberta_files):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	418	def test_decode_stream_copy_and_prefix_ids(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	440	def test_decode_stream_fallback(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	473	def test_decode_skip_special_tokens(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	667	def test_from_pretrained_revision(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	676	def test_unigram_byte_fallback(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	703	def test_encode_special_tokens(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	807	def test_weakref_with_multiple_references(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	937	async def test_with_truncation_padding(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	949	async def test_various_input_formats(self):	CODE
LOW	bindings/python/tests/bindings/test_tokenizer.py	1068	async def encode_sync_with_executor(_):	CODE
LOW	bindings/python/tests/bindings/test_pre_tokenizers.py	187	def test_item_getters_and_setters(self):	CODE
LOW	bindings/python/tests/bindings/test_encoding.py	118	def test_invalid_truncate_direction(self, encodings):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	163	def test_train_parallelism_with_custom_pretokenizer(self, train_files):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	187	def test_train_with_special_tokens(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	260	def test_cannot_train_different_model(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	294	def test_continuing_prefix_trainer_mismatch(self, train_files):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	338	def test_instantiate_defaults(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	348	def test_instantiate_variants(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	367	def test_train_from_iterator_with_dev(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	379	def test_train_from_iterator_with_ratio(self):	CODE
LOW	bindings/python/tests/bindings/test_trainers.py	397	def test_train_iterators_dev_iterators_length_mismatch(self):	CODE
LOW	bindings/node/index.d.ts	326	export declare function whitespaceSplitPreTokenizer(): PreTokenizer	CODE

Decorative Section Separators12 hits · 42 pts

Severity	File	Line	Snippet	Context
MEDIUM	bindings/python/tests/test_benchmarks.py	26	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	28	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	74	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	76	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	107	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	109	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	bindings/python/tests/test_benchmarks.py	149	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	bindings/python/tests/test_benchmarks.py	151	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	bindings/python/tests/test_benchmarks.py	168	# ---------------------------------------------------------------------------	COMMENT
MEDIUM⚡	bindings/python/tests/test_benchmarks.py	170	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	191	# ---------------------------------------------------------------------------	COMMENT
MEDIUM	bindings/python/tests/test_benchmarks.py	193	# ---------------------------------------------------------------------------	COMMENT

Self-Referential Comments8 hits · 25 pts

Severity	File	Line	Snippet	Context
MEDIUM	.github/workflows/CI.yml	1	# This file is autogenerated by maturin v1.7.4	COMMENT
MEDIUM	bindings/python/py_src/tokenizers/tools/visualizer.py	282	# Create a span from the current consecutive characters	COMMENT
MEDIUM	…on/py_src/tokenizers/implementations/byte_level_bpe.py	50	# Create the normalizer structure	COMMENT
MEDIUM	…on/py_src/tokenizers/implementations/char_level_bpe.py	65	# Create the normalizer structure	COMMENT
MEDIUM	bindings/python/tests/documentation/test_quicktour.py	16	# This method contains everything we don't want to run	COMMENT
MEDIUM⚡	bindings/python/tests/bindings/test_tokenizer.py	623	# Create a single shared tokenizer instance (thread-safe)	COMMENT
MEDIUM⚡	bindings/python/tests/bindings/test_tokenizer.py	1049	# Create a large batch for performance comparison	COMMENT
MEDIUM	bindings/python/examples/example.py	59	# Create a Tokenizer using BPE	STRING

Excessive Try-Catch Wrapping12 hits · 14 pts

Severity	File	Line	Snippet	Context
LOW	docs/source/_ext/entities.py	194	except Exception as err:	CODE
LOW	docs/source/_ext/entities.py	217	except Exception:	CODE
LOW	…rc/tokenizers/implementations/sentencepiece_unigram.py	154	except Exception:	CODE
LOW	bindings/python/tests/test_serialization.py	145	except Exception as e:	CODE
LOW	bindings/python/tests/documentation/test_quicktour.py	58	except Exception:	CODE
LOW	bindings/python/tests/documentation/test_pipeline.py	24	except Exception:	CODE
LOW	bindings/python/tests/documentation/test_pipeline.py	152	except Exception:	CODE
MEDIUM	bindings/python/tests/documentation/test_pipeline.py	17	def test_pipeline(self, doc_wiki_tokenizer):	CODE
MEDIUM	bindings/python/tests/documentation/test_pipeline.py	149	def test_bert_example(self, doc_pipeline_bert_tokenizer):	CODE
LOW	bindings/python/scripts/spm_parity_check.py	14	except Exception:	CODE
LOW	bindings/python/scripts/spm_parity_check.py	195	except Exception:	CODE
LOW	bindings/python/scripts/convert.py	46	except Exception:	CODE

AI Slop Vocabulary3 hits · 9 pts

Severity	File	Line	Snippet	Context
MEDIUM	tokenizers/src/lib.rs	120	//! - tokenizers is designed to leverage CPU parallelism when possible. The level of parallelism is determined	COMMENT
MEDIUM	tokenizers/src/models/bpe/trainer.rs	760	// this is the more robust version that only tests max length of learned tokens	COMMENT
MEDIUM	bindings/python/src/pre_tokenizers.rs	153	/// keep track of the pre-tokenization, and leverage the capabilities of the	COMMENT

Deep Nesting6 hits · 6 pts

Severity	File	Line	Context
LOW	docs/source/_ext/entities.py	113	CODE
LOW	docs/source/_ext/entities.py	136	CODE
LOW	bindings/python/scripts/spm_parity_check.py	150	CODE
LOW	bindings/python/scripts/spm_parity_check.py	209	CODE
LOW	bindings/python/scripts/convert.py	333	CODE
LOW	bindings/python/benches/test_tiktoken.py	81	CODE

AI Structural Patterns6 hits · 6 pts

Severity	File	Line	Context
LOW	bindings/python/py_src/tokenizers/tools/visualizer.py	331	CODE
LOW	…on/py_src/tokenizers/implementations/byte_level_bpe.py	16	CODE
LOW	…on/py_src/tokenizers/implementations/char_level_bpe.py	25	CODE
LOW	…on/py_src/tokenizers/implementations/char_level_bpe.py	123	CODE
LOW	…on/py_src/tokenizers/implementations/bert_wordpiece.py	15	CODE
LOW	…on/py_src/tokenizers/implementations/bert_wordpiece.py	118	CODE

Cross-Language Confusion1 hit · 5 pts

Severity	File	Line	Snippet	Context
HIGH	bindings/python/tests/bindings/test_trainers.py	66	== b"""{"BpeTrainer":{"min_frequency":12,"vocab_size":30000,"show_progress":true,"progress_format":"Indicati	CODE

Analysis Overview

What These Metrics Mean

Score History

Severity Breakdown

Directory Score Breakdown

Pattern Findings