Repository Analysis

google/sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

13.8 Low AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of google/sentencepiece, a C++ project with 12,049 GitHub stars. SynthScan v2.0 examined 42,048 lines of code across 89 source files, recording 327 pattern matches distributed across 12 syntactic categories. The overall adjusted score of 13.8 places this repository in the Low AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

13.8
Adjusted Score
13.8
Raw Score
100%
Time Factor
2026-08-28
Last Push
12.0K
Stars
C++
Language
42.0K
Lines of Code
89
Files
327
Pattern Hits
2026-08-29
Scan Date
0.02
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 2MEDIUM 175LOW 150

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 327 distinct pattern matches across 12 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Decorative Section Separators168 hits · 397 pts
SeverityFileLineSnippetContext
MEDIUMCMakeLists.txt15# ==============================================================================COMMENT
MEDIUMCMakeLists.txt17# ==============================================================================COMMENT
MEDIUMCMakeLists.txt38# ==============================================================================COMMENT
MEDIUMCMakeLists.txt40# ==============================================================================COMMENT
MEDIUMCMakeLists.txt61# ==============================================================================COMMENT
MEDIUMCMakeLists.txt63# ==============================================================================COMMENT
MEDIUMCMakeLists.txt135# ==============================================================================COMMENT
MEDIUMCMakeLists.txt137# ==============================================================================COMMENT
MEDIUMCMakeLists.txt294# ==============================================================================COMMENT
MEDIUMCMakeLists.txt296# ==============================================================================COMMENT
MEDIUMCMakeLists.txt363# ==============================================================================COMMENT
MEDIUMCMakeLists.txt365# ==============================================================================COMMENT
MEDIUMCMakeLists.txt369# ==============================================================================COMMENT
MEDIUMCMakeLists.txt371# ==============================================================================COMMENT
MEDIUMdata/Scripts.txt1383# ================================================COMMENT
MEDIUMdata/Scripts.txt1391# ================================================COMMENT
MEDIUMdata/Scripts.txt1401# ================================================COMMENT
MEDIUMdata/Scripts.txt1410# ================================================COMMENT
MEDIUMdata/Scripts.txt1418# ================================================COMMENT
MEDIUMdata/Scripts.txt1515# ================================================COMMENT
MEDIUMdata/Scripts.txt1524# ================================================COMMENT
MEDIUMdata/Scripts.txt1531# ================================================COMMENT
MEDIUMdata/Scripts.txt1540# ================================================COMMENT
MEDIUMdata/Scripts.txt1546# ================================================COMMENT
MEDIUMdata/Scripts.txt1583# ================================================COMMENT
MEDIUMdata/Scripts.txt1591# ================================================COMMENT
MEDIUMdata/Scripts.txt1598# ================================================COMMENT
MEDIUMdata/Scripts.txt1605# ================================================COMMENT
MEDIUMdata/Scripts.txt1613# ================================================COMMENT
MEDIUMdata/Scripts.txt1649# ================================================COMMENT
MEDIUMdata/Scripts.txt1656# ================================================COMMENT
MEDIUMdata/Scripts.txt1662# ================================================COMMENT
MEDIUMdata/Scripts.txt1669# ================================================COMMENT
MEDIUMdata/Scripts.txt1680# ================================================COMMENT
MEDIUMdata/Scripts.txt1686# ================================================COMMENT
MEDIUMdata/Scripts.txt1696# ================================================COMMENT
MEDIUMdata/Scripts.txt1804# ================================================COMMENT
MEDIUMdata/Scripts.txt1813# ================================================COMMENT
MEDIUMdata/Scripts.txt1821# ================================================COMMENT
MEDIUMdata/Scripts.txt1828# ================================================COMMENT
MEDIUMdata/Scripts.txt1902# ================================================COMMENT
MEDIUMdata/Scripts.txt1911# ================================================COMMENT
MEDIUMdata/Scripts.txt1920# ================================================COMMENT
MEDIUMdata/Scripts.txt1926# ================================================COMMENT
MEDIUMdata/Scripts.txt1932# ================================================COMMENT
MEDIUMdata/Scripts.txt1939# ================================================COMMENT
MEDIUMdata/Scripts.txt1999# ================================================COMMENT
MEDIUMdata/Scripts.txt2006# ================================================COMMENT
MEDIUMdata/Scripts.txt2012# ================================================COMMENT
MEDIUMdata/Scripts.txt2026# ================================================COMMENT
MEDIUMdata/Scripts.txt2034# ================================================COMMENT
MEDIUMdata/Scripts.txt2044# ================================================COMMENT
MEDIUMdata/Scripts.txt2085# ================================================COMMENT
MEDIUMdata/Scripts.txt2093# ================================================COMMENT
MEDIUMdata/Scripts.txt2101# ================================================COMMENT
MEDIUMdata/Scripts.txt2108# ================================================COMMENT
MEDIUMdata/Scripts.txt2115# ================================================COMMENT
MEDIUMdata/Scripts.txt2121# ================================================COMMENT
MEDIUMdata/Scripts.txt2185# ================================================COMMENT
MEDIUMdata/Scripts.txt2195# ================================================COMMENT
108 more matches not shown…
Over-Commented Block72 hits · 72 pts
SeverityFileLineSnippetContext
LOWCMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWpython/README.md61# Encode text to pieces as bytesCOMMENT
LOWpython/README.md81# Output:COMMENT
LOWpython/README.md101# ['▁', 'This', '▁', 'is', '▁a', '▁', 't', 'e', 'st']COMMENT
LOWpython/README.md121ids_np = np.array([284, 47, 11, 4, 15, 400], dtype=np.int32)COMMENT
LOWpython/setup.py1#!/usr/bin/env pythonCOMMENT
LOWpython/tools/gen_stubs.py1#!/usr/bin/env pythonCOMMENT
LOWpython/test/gen_stubs_test.py1# Copyright 2024 Google Sepm.COMMENT
LOWpython/test/sentencepiece_test.py1#!/usr/bin/pythonCOMMENT
LOWthird_party/darts_clone/darts.h1#ifndef DARTS_H_COMMENT
LOWthird_party/darts_clone/darts.h101COMMENT
LOWthird_party/darts_clone/darts.h141 // member variables with 0 and NULLs.COMMENT
LOWthird_party/darts_clone/darts.h221COMMENT
LOWthird_party/darts_clone/darts.h241 Details::progress_func_type progress_func = NULL);COMMENT
LOWthird_party/darts_clone/darts.h261 // if it exists, its value and length are set to `result'. Otherwise, theCOMMENT
LOWthird_party/darts_clone/darts.h281COMMENT
LOWthird_party/libsais/libsais.c21COMMENT
LOWthird_party/libsais/libsais.c41typedef size_t fast_uint_t;COMMENT
LOWthird_party/libsais/libsais.c101#error Your compiler, configuration or platform is not supported.COMMENT
LOWthird_party/libsais/libsais.c121COMMENT
LOWthird_party/libsais/libsais.c161 __BYTE_ORDER == __BIG_ENDIAN) || \COMMENT
LOWthird_party/libsais/libsais.h21COMMENT
LOW.github/workflows/requirements/cibuildwheel.txt1#COMMENT
LOW.github/scripts/update_dependencies.py1#!/usr/bin/env python3COMMENT
LOWdoc/special_symbols.md141# Output:COMMENT
LOWdata/Scripts.txt1# Scripts-9.0.0.txtCOMMENT
LOWdata/gen_unicode_scripts_code.pl1#!/usr/bin/perlCOMMENT
LOWsrc/filesystem.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/freelist.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/unicode_script.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h21#include <string>COMMENT
LOWsrc/model_interface.h101COMMENT
LOWsrc/CMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWsrc/ret_check.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/ret_check.h21#ifndef RET_CHECKCOMMENT
LOWsrc/bpe_model.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/word_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_trainer.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/char_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h21#include <string>COMMENT
LOWsrc/gtest_prod.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/unigram_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/unigram_model.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_processor.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_processor.h21#include <string>COMMENT
LOWsrc/sentencepiece_processor.h41//COMMENT
LOWsrc/sentencepiece_processor.h61//COMMENT
LOWsrc/sentencepiece_processor.h121// `total_tasks`: Number of tasks to execute (= batch size)COMMENT
LOWsrc/sentencepiece_processor.h201COMMENT
LOWsrc/sentencepiece_processor.h221 // value, samples one segmentation from the hypotheses (Lattice) according toCOMMENT
LOWsrc/sentencepiece_processor.h401COMMENT
LOWsrc/sentencepiece_processor.h501// std::random_device.COMMENT
LOWsrc/spec_parser.h1// Copyright 2016 Google LLC.COMMENT
LOWsrc/spec_parser.h21#include <string>COMMENT
LOWsrc/unicode_script_map.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/util.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/util.h21#include <memory>COMMENT
LOWsrc/builder.h1// Copyright 2016 Google Inc.COMMENT
12 more matches not shown…
Hyper-Verbose Identifiers42 hits · 42 pts
SeverityFileLineSnippetContext
LOWpython/test/numpy_test.py57 def test_return_type_validation_and_compat(self):CODE
LOWpython/test/numpy_test.py101 def test_decode_numpy_batch_list(self):CODE
LOWpython/test/numpy_test.py127 def test_decode_numpy_non_contiguous(self):CODE
LOWpython/test/numpy_test.py146 def test_decode_numpy_invalid_types(self):CODE
LOWpython/test/numpy_test.py157 def test_parallel_encode_numpy(self):CODE
LOWpython/test/numpy_test.py240 def test_decode_invalid_ids_numpy(self):CODE
LOWpython/test/numpy_test.py249 def test_decode_numpy_batch_mixed(self):CODE
LOWpython/test/numpy_test.py261 def test_native_batch_id_to_piece_numpy(self):CODE
LOWpython/test/numpy_test.py277 def test_native_batch_other_id_methods_numpy(self):CODE
LOWpython/test/gil_release_test.py31def run_heavy_op_with_gil_check(op_func, name):CODE
LOWpython/test/clean_sentencepiece_test_manual.py115 def test_encode_decode_types_single(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py294 def test_encode_extra_options(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py439 def test_normalization_and_entropy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py571 def run_heavy_op_with_gil_check(op_func, name):CODE
LOWpython/test/clean_sentencepiece_test_manual.py630 def test_gil_release_encode_with_list_modification(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py651 def test_gil_release_decode_pieces_with_list_modification(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py671 def test_encode_bytes_is_zero_copy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py707 def test_encode_str_is_zero_copy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py890 def test_train_with_writer_only(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py901 def test_train_with_logstream(self):CODE
LOWpython/test/gen_stubs_test.py45 def test_committed_stub_in_sync(self):CODE
LOWpython/test/gen_stubs_test.py101 def test_leaked_module_level_import_ignored(self):CODE
LOWpython/test/sentencepiece_test.py257 def test_special_tokens_combinations(self):CODE
LOWpython/test/sentencepiece_test.py683 def test_classmethod_factories(self):CODE
LOWpython/test/sentencepiece_test.py755 def test_word_model_user_defined_symbol(self):CODE
LOWpython/test/sentencepiece_test.py877 def test_train_with_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py891 def test_inconsistent_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py916 def test_pretokenizer_exception(self):CODE
LOWpython/test/sentencepiece_test.py932 def test_pretokenizer_and_delimiter_mutually_exclusive(self):CODE
LOWpython/test/sentencepiece_test.py945 def test_complex_regex_pretokenizer(self):CODE
LOWpython/test/sentencepiece_test.py1253 def test_trainer_with_normalizer(self):CODE
LOWpython/test/sentencepiece_test.py1367 def test_decode_offset_mapping(self):CODE
LOWpython/test/sentencepiece_test.py1459 def test_decode_return_type_bytes(self):CODE
LOWpython/test/sentencepiece_test.py1505 def test_legacy_out_type_compat(self):CODE
LOWpython/test/sentencepiece_test.py1537 def test_normalizer_model_proto(self):CODE
LOWpython/test/sentencepiece_test.py1545 def test_encode_return_type_explicit(self):CODE
LOWpython/test/sentencepiece_test.py1652 def test_native_batch_piece_to_id(self):CODE
LOWpython/test/sentencepiece_test.py1682 def test_native_batch_id_to_piece(self):CODE
LOWpython/test/sentencepiece_test.py1716 def test_native_batch_other_id_methods(self):CODE
LOWpython/src/sentencepiece/__init__.py165 def _resolve_and_validate_return_type(self, return_type):CODE
LOWpython/src/sentencepiece/__init__.py787 def serialized_normalizer_spec(self):CODE
LOW.github/scripts/update_dependencies.py180def update_module_bazel_content(content: str):CODE
Excessive Try-Catch Wrapping16 hits · 23 pts
SeverityFileLineSnippetContext
MEDIUMbenchmark/analyze_results.py7 print("Error: 'results.json' not found.")CODE
MEDIUMbenchmark/analyze_results.py15 print("Error: 'results.json' is empty.")CODE
LOWbenchmark/download_models.py11except Exception as e:CODE
LOWbenchmark/download_models.py18except Exception as e:CODE
LOWbenchmark/download_models.py27except Exception as e:CODE
LOWbenchmark/download_models.py35except Exception as e:CODE
MEDIUMbenchmark/run_bench.py139 print(f"Error: Dataset file '{path}' not found.")CODE
MEDIUMbenchmark/run_bench.py163 print("Error: No tasks available to run. Please run 'python download_models.py' first.")CODE
LOWpython/test/clean_sentencepiece_test_manual.py682 except Exception as e:CODE
LOWpython/test/clean_sentencepiece_test_manual.py761 except Exception as e:CODE
LOWpython/src/sentencepiece/__init__.py837except Exception:CODE
LOW.github/scripts/update_dependencies.py74 except Exception as e:CODE
LOW.github/scripts/update_dependencies.py88 except Exception as e:CODE
MEDIUM.github/scripts/update_dependencies.py89 print(f"Error fetching tags for {repo_path}: {e}")CODE
LOW.github/scripts/update_dependencies.py122 except Exception as e:CODE
LOW.github/scripts/update_dependencies.py252 except Exception as e:CODE
Cross-Language Confusion2 hits · 15 pts
SeverityFileLineSnippetContext
HIGHpython/test/sentencepiece_test.py340 # suppress logging (redirect to /dev/null)COMMENT
HIGHpython/test/sentencepiece_test.py381 # suppress logging (redirect to /dev/null)COMMENT
Deep Nesting11 hits · 11 pts
SeverityFileLineSnippetContext
LOWbenchmark/run_bench.py37CODE
LOWbenchmark/run_bench.py62CODE
LOWbenchmark/run_bench.py122CODE
LOWbenchmark/run_bench.py135CODE
LOWpython/setup.py41CODE
LOWpython/setup.py166CODE
LOWpython/tools/gen_stubs.py395CODE
LOWpython/src/sentencepiece/__init__.py691CODE
LOWpython/src/sentencepiece/__init__.py746CODE
LOW.github/scripts/update_dependencies.py52CODE
LOW.github/scripts/update_dependencies.py94CODE
Self-Referential Comments2 hits · 6 pts
SeverityFileLineSnippetContext
MEDIUMpython/tools/gen_stubs.py48# This file is auto-generated by python/tools/gen_stubs.py.COMMENT
MEDIUMpython/test/numpy_test.py206 # Create a processor local to this scopeCOMMENT
Structural Annotation Overuse3 hits · 4 pts
SeverityFileLineSnippetContext
LOWdoc/model_proto.md56### Step 1: Define your extensionCOMMENT
LOWdoc/model_proto.md73### Step 2: Compile your protoCOMMENT
LOWdoc/model_proto.md81### Step 3: Read/Write Custom Metadata in PythonCOMMENT
AI Structural Patterns4 hits · 4 pts
SeverityFileLineSnippetContext
LOWpython/src/sentencepiece/__init__.py137CODE
LOWpython/src/sentencepiece/__init__.py220CODE
LOWpython/src/sentencepiece/__init__.py432CODE
LOWpython/src/sentencepiece/__init__.py746CODE
Unused Imports5 hits · 4 pts
SeverityFileLineSnippetContext
LOWpython/test/sentencepiece_test.py19CODE
LOWpython/test/sentencepiece_test.py24CODE
LOWpython/test/sentencepiece_test.py26CODE
LOWpython/src/sentencepiece/__init__.py833CODE
LOW.github/scripts/update_dependencies.py19CODE
Redundant / Tautological Comments1 hit · 2 pts
SeverityFileLineSnippetContext
LOWpython/test/clean_sentencepiece_test_manual.py913 # Check if log file is not emptyCOMMENT
Fake / Example Data1 hit · 1 pts
SeverityFileLineSnippetContext
LOWdoc/model_proto.md95model_proto.Extensions[my_metadata_pb2.model_author] = "John Doe"CODE