AI Slop Code RIsk Monitor

google/sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

16.7 Elevated pattern signal View on GitHub

Maintenance and Governance Review

This report summarizes scanner evidence for google/sentencepiece, a C++ project with 12,125 GitHub stars. SynthScan examined 40,095 lines of code across 96 source files, recording 367 pattern matches distributed across 13 syntactic categories. The legacy adjusted pattern signal is 16.7 (Elevated pattern signal); it is not an authorship estimate.

Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.

16.7
Adjusted Signal
16.7
Raw Pattern Score
100%
Time Factor
2026-10-10
Last Push
12.1K
Stars
C++
Language
40.1K
Lines of Code
96
Files
367
Pattern Hits
2026-10-12
Scan Date
0.03
HC Hit Rate

Risk Review · v3

Use each vector to guide manual verification of the underlying findings.

Cognitive Ownership Risk 18.6/100 · LOW

Can maintainers explain intent, assumptions, and trade-offs? 158 finding(s).

Hallucinated Dependency & Supply Chain Risk 2.2/100 · LOW

Are imports declared and package/API names verified? 6 finding(s).

Agent Over-Engineering Ratio 47.9/100 · MEDIUM

Are wrappers, guards, and repeated structures necessary? 243 finding(s).

PR Review Velocity Impact 61.6/100 · HIGH

Which changes need focused human review? 367 finding(s).

What These Metrics Mean

Adjusted Signal
Legacy pattern score normalized per 1,000 scanned lines and multiplied by the repository-level time factor. It is not an estimate of AI authorship.
Raw Pattern Score
Sum of severity- and context-weighted pattern findings before applying the time factor.
Time Factor
A coarse multiplier based on repository last-push metadata. It does not identify when individual code was written.
Pattern Hits
Count of configured pattern findings. Review each location and surrounding code before acting.
HC Hit Rate
High- and critical-severity findings per scanned file. Severity is a configured detector weight, not proof of a vulnerability or authorship.
Lines / Files
Scan coverage denominators used by the legacy normalized score.

Pattern-Signal History

This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.

Severity Breakdown

Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.

CRITICAL 0HIGH 3MEDIUM 193LOW 171

Directory Score Breakdown

This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.

Pattern Findings

The scanner identified 367 distinct pattern matches across 13 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.

Decorative Section Separators186 hits · 457 pts
SeverityFileLineSnippetContext
MEDIUMCMakeLists.txt15# ==============================================================================COMMENT
MEDIUMCMakeLists.txt17# ==============================================================================COMMENT
MEDIUMCMakeLists.txt38# ==============================================================================COMMENT
MEDIUMCMakeLists.txt40# ==============================================================================COMMENT
MEDIUMCMakeLists.txt62# ==============================================================================COMMENT
MEDIUMCMakeLists.txt64# ==============================================================================COMMENT
MEDIUMCMakeLists.txt129# ==============================================================================COMMENT
MEDIUMCMakeLists.txt131# ==============================================================================COMMENT
MEDIUMCMakeLists.txt319# ==============================================================================COMMENT
MEDIUMCMakeLists.txt321# ==============================================================================COMMENT
MEDIUM⚡CMakeLists.txt377# ==============================================================================COMMENT
MEDIUM⚡CMakeLists.txt379# ==============================================================================COMMENT
MEDIUM⚡CMakeLists.txt385# ==============================================================================COMMENT
MEDIUM⚡CMakeLists.txt387# ==============================================================================COMMENT
MEDIUMlite/CMakeLists.txt41# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt45# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt116# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt118# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt183# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt186# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt219# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt222# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt238# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt241# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt266# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt268# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt286# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt288# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt379# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt381# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt415# ------------------------------------------------------------------------------COMMENT
MEDIUMlite/CMakeLists.txt417# ------------------------------------------------------------------------------COMMENT
MEDIUM⚡data/Scripts.txt1383# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1391# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1401# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1410# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1418# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1515# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1524# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1531# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1540# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1546# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1583# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1591# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1598# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1605# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1613# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1649# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1656# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1662# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1669# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1680# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1686# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1696# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1804# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1813# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1821# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1828# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1902# ================================================COMMENT
MEDIUM⚡data/Scripts.txt1911# ================================================COMMENT
126 more matches not shown…
Over-Commented Block95 hits · 94 pts
SeverityFileLineSnippetContext
LOWCMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWpython/README.md61# Encode text to pieces as bytesCOMMENT
LOWpython/README.md81# Output:COMMENT
LOWpython/README.md101# ['▁', 'This', '▁', 'is', '▁a', '▁', 't', 'e', 'st']COMMENT
LOWpython/README.md121ids_np = np.array([284, 47, 11, 4, 15, 400], dtype=np.int32)COMMENT
LOWpython/setup.py1#!/usr/bin/env pythonCOMMENT
LOWpython/tools/gen_stubs.py1#!/usr/bin/env pythonCOMMENT
LOWpython/test/gen_stubs_test.py1# Copyright 2024 Google Sepm.COMMENT
LOWpython/test/sentencepiece_test.py1#!/usr/bin/pythonCOMMENT
LOWlite/CMakeLists.txt1# Copyright 2026 Google Inc.COMMENT
LOWlite/amalgamate.py1#!/usr/bin/env python3COMMENT
LOWlite/cached_sentencepiece_lite.h1// Copyright 2026 Google LLCCOMMENT
LOWlite/cached_sentencepiece_lite.h21#include <memory>COMMENT
LOWlite/cached_sentencepiece_lite.h81 // - kTiny (64 KB, 2,048 slots): Minimal memory footprint. Recommended forCOMMENT
LOWlite/cached_sentencepiece_lite.h201// instance into the effective hashing seed (`cache.seed() ^COMMENT
LOWlite/cached_sentencepiece_lite.h221// std::string_view text) {COMMENT
LOWlite/sentencepiece_lite.h1// Copyright 2026 Google LLCCOMMENT
LOWlite/sentencepiece_lite.h21// normalization.COMMENT
LOWlite/sentencepiece_lite.h41// std::map).COMMENT
LOWlite/sentencepiece_lite.h201 // - Unigram (--model_type=unigram):COMMENT
LOWlite/sentencepiece_model_converters.h1// Copyright 2026 Google LLCCOMMENT
LOWlite/sentencepiece_model_converters.h21#include <string>COMMENT
LOWlite/sentencepiece_model_converters.h41 // InvalidArgument error.COMMENT
LOWlite/sentencepiece_model_converters.h61// 1. Initialization / Load Time:COMMENT
LOWlite/sentencepiece_model_converters.h81// Raw Proto: 4.5 MB | Lite FB Heap: 9.6 MB | OSS Runtime Heap: 27.7 MBCOMMENT
LOWlite/sentencepiece_model_converters.h101// // 3. Initialize processor.COMMENT
LOWthird_party/rapidhash/rapidhash.h41 */COMMENT
LOWthird_party/rapidhash/rapidhash.h61#endifCOMMENT
LOWthird_party/darts_clone/darts.h1#ifndef DARTS_H_COMMENT
LOWthird_party/darts_clone/darts.h101COMMENT
LOWthird_party/darts_clone/darts.h141 // member variables with 0 and NULLs.COMMENT
LOWthird_party/darts_clone/darts.h221COMMENT
LOWthird_party/darts_clone/darts.h241 Details::progress_func_type progress_func = NULL);COMMENT
LOWthird_party/darts_clone/darts.h261 // if it exists, its value and length are set to `result'. Otherwise, theCOMMENT
LOWthird_party/darts_clone/darts.h281COMMENT
LOWthird_party/libsais/libsais.c21COMMENT
LOWthird_party/libsais/libsais.c41typedef size_t fast_uint_t;COMMENT
LOWthird_party/libsais/libsais.c101#error Your compiler, configuration or platform is not supported.COMMENT
LOWthird_party/libsais/libsais.c121COMMENT
LOWthird_party/libsais/libsais.c161 __BYTE_ORDER == __BIG_ENDIAN) || \COMMENT
LOWthird_party/libsais/libsais.h21COMMENT
LOW.github/workflows/requirements/cibuildwheel.txt1#COMMENT
LOW.github/scripts/update_dependencies.py1#!/usr/bin/env python3COMMENT
LOWdoc/special_symbols.md141# Output:COMMENT
LOWdata/Scripts.txt1# Scripts-9.0.0.txtCOMMENT
LOWdata/gen_unicode_scripts_code.pl1#!/usr/bin/perlCOMMENT
LOWsrc/filesystem.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/freelist.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/unicode_script.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h21#include <utility>COMMENT
LOWsrc/CMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWsrc/ret_check.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/ret_check.h21#ifndef RET_CHECKCOMMENT
LOWsrc/bpe_model.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/word_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_trainer.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/char_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h21#include <string>COMMENT
35 more matches not shown…
Hyper-Verbose Identifiers36 hits · 34 pts
SeverityFileLineSnippetContext
LOWpython/test/numpy_test.py57 def test_return_type_validation_and_compat(self):CODE
LOWpython/test/numpy_test.py101 def test_decode_numpy_batch_list(self):CODE
LOWpython/test/numpy_test.py127 def test_decode_numpy_non_contiguous(self):CODE
LOWpython/test/numpy_test.py146 def test_decode_numpy_invalid_types(self):CODE
LOWpython/test/numpy_test.py157 def test_parallel_encode_numpy(self):CODE
LOWpython/test/numpy_test.py240 def test_decode_invalid_ids_numpy(self):CODE
LOWpython/test/numpy_test.py249 def test_decode_numpy_batch_mixed(self):CODE
LOWpython/test/numpy_test.py261 def test_native_batch_id_to_piece_numpy(self):CODE
LOWpython/test/numpy_test.py277 def test_native_batch_other_id_methods_numpy(self):CODE
LOWpython/test/gil_release_test.py31def run_heavy_op_with_gil_check(op_func, name):CODE
LOWpython/test/gen_stubs_test.py45 def test_committed_stub_in_sync(self):CODE
LOWpython/test/gen_stubs_test.py101 def test_leaked_module_level_import_ignored(self):CODE
LOWpython/test/sentencepiece_test.py134 def test_decode_batch_with_leading_empty_sequence(self):CODE
LOWpython/test/sentencepiece_test.py308 def test_special_tokens_combinations(self):CODE
LOWpython/test/sentencepiece_test.py734 def test_classmethod_factories(self):CODE
LOWpython/test/sentencepiece_test.py806 def test_word_model_user_defined_symbol(self):CODE
LOWpython/test/sentencepiece_test.py926 def test_train_with_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py940 def test_inconsistent_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py965 def test_pretokenizer_exception(self):CODE
LOWpython/test/sentencepiece_test.py991 def test_pretokenizer_and_delimiter_mutually_exclusive(self):CODE
LOWpython/test/sentencepiece_test.py1004 def test_complex_regex_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py1267 def test_normalizer_default_flags_follow_spec(self):CODE
LOWpython/test/sentencepiece_test.py1374 def test_trainer_with_normalizer(self):CODE
LOWpython/test/sentencepiece_test.py1426 def test_offset_mapping_empty_batch(self):CODE
LOWpython/test/sentencepiece_test.py1527 def test_decode_offset_mapping(self):CODE
LOWpython/test/sentencepiece_test.py1642 def test_decode_return_type_bytes(self):CODE
LOWpython/test/sentencepiece_test.py1688 def test_legacy_out_type_compat(self):CODE
LOWpython/test/sentencepiece_test.py1727 def test_normalizer_model_proto(self):CODE
LOWpython/test/sentencepiece_test.py1739 def test_normalizer_normalizer_spec(self):CODE
LOWpython/test/sentencepiece_test.py1752 def test_encode_return_type_explicit(self):CODE
LOWpython/test/sentencepiece_test.py1859 def test_native_batch_piece_to_id(self):CODE
LOWpython/test/sentencepiece_test.py1889 def test_native_batch_id_to_piece(self):CODE
LOWpython/test/sentencepiece_test.py1923 def test_native_batch_other_id_methods(self):CODE
LOWpython/test/sentencepiece_test.py1947 def test_convert_to_flatbuffer(self):CODE
LOWpython/src/sentencepiece/__init__.py841 def serialized_normalizer_spec(self):CODE
LOW.github/scripts/update_dependencies.py201def update_module_bazel_content(content: str):CODE
Excessive Try-Catch Wrapping16 hits · 23 pts
SeverityFileLineSnippetContext
MEDIUMbenchmark/analyze_results.py7 print("Error: 'results.json' not found.")CODE
MEDIUMbenchmark/analyze_results.py15 print("Error: 'results.json' is empty.")CODE
LOW⚡benchmark/download_models.py11except Exception as e:CODE
LOW⚡benchmark/download_models.py18except Exception as e:CODE
LOW⚡benchmark/download_models.py27except Exception as e:CODE
LOW⚡benchmark/download_models.py35except Exception as e:CODE
MEDIUMbenchmark/run_bench.py139 print(f"Error: Dataset file '{path}' not found.")CODE
MEDIUMbenchmark/run_bench.py163 print("Error: No tasks available to run. Please run 'python download_models.py' first.")CODE
LOWpython/test/sentencepiece_test.py35except Exception:CODE
LOWpython/src/sentencepiece/__init__.py725 except Exception:CODE
LOWpython/src/sentencepiece/__init__.py886except Exception:CODE
LOW.github/scripts/update_dependencies.py87 except Exception as e:CODE
LOW.github/scripts/update_dependencies.py103 except Exception as e:CODE
MEDIUM.github/scripts/update_dependencies.py104 print(f"Error fetching tags for {repo_path}: {e}")CODE
LOW.github/scripts/update_dependencies.py140 except Exception as e:CODE
LOW.github/scripts/update_dependencies.py333 except Exception as e:CODE
Cross-Language Confusion2 hits · 15 pts
SeverityFileLineSnippetContext
HIGHpython/test/sentencepiece_test.py391 # suppress logging (redirect to /dev/null)COMMENT
HIGHpython/test/sentencepiece_test.py432 # suppress logging (redirect to /dev/null)COMMENT
Structural Annotation Overuse8 hits · 14 pts
SeverityFileLineSnippetContext
LOW⚡doc/auto_character_coverage.md54#### Step 1: Extract Candidate Merge Sequence Over All Unicode CharactersCOMMENT
LOW⚡doc/auto_character_coverage.md61#### Step 2: Select Top $k$ Merge Tokens and Merge GainCOMMENT
LOW⚡doc/auto_character_coverage.md67#### Step 3: Character Gain and Full Character Closure ConstraintCOMMENT
LOWdoc/auto_character_coverage.md79#### Step 4: Remaining Budget and Standalone CharactersCOMMENT
LOWdoc/auto_character_coverage.md93#### Step 5: Global Objective Maximization (Equivalence to Total Token Minimization)COMMENT
LOWdoc/model_proto.md56### Step 1: Define your extensionCOMMENT
LOWdoc/model_proto.md73### Step 2: Compile your protoCOMMENT
LOWdoc/model_proto.md81### Step 3: Read/Write Custom Metadata in PythonCOMMENT
Deep Nesting12 hits · 12 pts
SeverityFileLineSnippetContext
LOWbenchmark/run_bench.py37CODE
LOWbenchmark/run_bench.py62CODE
LOWbenchmark/run_bench.py122CODE
LOWbenchmark/run_bench.py135CODE
LOWpython/setup.py36CODE
LOWpython/setup.py164CODE
LOWpython/tools/gen_stubs.py345CODE
LOWpython/src/sentencepiece/__init__.py739CODE
LOWpython/src/sentencepiece/__init__.py794CODE
LOWlite/amalgamate.py62CODE
LOW.github/scripts/update_dependencies.py61CODE
LOW.github/scripts/update_dependencies.py111CODE
Self-Referential Comments2 hits · 6 pts
SeverityFileLineSnippetContext
MEDIUMpython/tools/gen_stubs.py49# This file is auto-generated by python/tools/gen_stubs.py.COMMENT
MEDIUMpython/test/numpy_test.py206 # Create a processor local to this scopeCOMMENT
Empty Error Handlers1 hit · 5 pts
SeverityFileLineSnippetContext
HIGHpython/src/sentencepiece/__init__.py886except Exception:CODE
AI Structural Patterns4 hits · 4 pts
SeverityFileLineSnippetContext
LOWpython/src/sentencepiece/__init__.py204CODE
LOWpython/src/sentencepiece/__init__.py281CODE
LOWpython/src/sentencepiece/__init__.py481CODE
LOWpython/src/sentencepiece/__init__.py794CODE
Unused Imports3 hits · 2 pts
SeverityFileLineSnippetContext
LOWpython/test/sentencepiece_test.py23CODE
LOWpython/test/sentencepiece_test.py25CODE
LOWpython/src/sentencepiece/__init__.py882CODE
Example Usage Blocks1 hit · 2 pts
SeverityFileLineSnippetContext
LOWlite/cached_sentencepiece_lite.h215// Example usage:COMMENT
Fake / Example Data1 hit · 1 pts
SeverityFileLineSnippetContext
LOWdoc/model_proto.md95model_proto.Extensions[my_metadata_pb2.model_author] = "John Doe"CODE