Repository Analysis

google/sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

14.1 Low AI signal View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of google/sentencepiece, a C++ project with 11,961 GitHub stars. SynthScan v2.0 examined 35,516 lines of code across 88 source files, recording 299 pattern matches distributed across 12 syntactic categories. The overall adjusted score of 14.1 places this repository in the Low AI signal band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

14.1
Adjusted Score
14.1
Raw Score
100%
Time Factor
2026-07-14
Last Push
12.0K
Stars
C++
Language
35.5K
Lines of Code
88
Files
299
Pattern Hits
2026-07-14
Scan Date
0.02
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 0HIGH 2MEDIUM 151LOW 146

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 299 distinct pattern matches across 12 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Decorative Section Separators145 hits · 322 pts
SeverityFileLineSnippetContext
MEDIUMcontrib/nlcodec/bpe_model_trainer_nlcodec.h31// ─── Data Structures ─────────────────────────────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/bpe_model_trainer_nlcodec.h115// ─── Fast BPE Training Function ─────────────────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/benchmark.sh15# ─── Defaults ─────────────────────────────────────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/benchmark.sh47# ─── Parse arguments ─────────────────────────────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/benchmark.sh60# ─── Step 1: Build SentencePiece if needed ────────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/benchmark.sh75# ─── Step 2: Download CC-100 data if needed ───────────────────────────────────COMMENT
MEDIUMcontrib/nlcodec/benchmark.sh119# ─── Step 3: Run benchmark ───────────────────────────────────────────────────COMMENT
MEDIUMdata/Scripts.txt1383# ================================================COMMENT
MEDIUMdata/Scripts.txt1391# ================================================COMMENT
MEDIUMdata/Scripts.txt1401# ================================================COMMENT
MEDIUMdata/Scripts.txt1410# ================================================COMMENT
MEDIUMdata/Scripts.txt1418# ================================================COMMENT
MEDIUMdata/Scripts.txt1515# ================================================COMMENT
MEDIUMdata/Scripts.txt1524# ================================================COMMENT
MEDIUMdata/Scripts.txt1531# ================================================COMMENT
MEDIUMdata/Scripts.txt1540# ================================================COMMENT
MEDIUMdata/Scripts.txt1546# ================================================COMMENT
MEDIUMdata/Scripts.txt1583# ================================================COMMENT
MEDIUMdata/Scripts.txt1591# ================================================COMMENT
MEDIUMdata/Scripts.txt1598# ================================================COMMENT
MEDIUMdata/Scripts.txt1605# ================================================COMMENT
MEDIUMdata/Scripts.txt1613# ================================================COMMENT
MEDIUMdata/Scripts.txt1649# ================================================COMMENT
MEDIUMdata/Scripts.txt1656# ================================================COMMENT
MEDIUMdata/Scripts.txt1662# ================================================COMMENT
MEDIUMdata/Scripts.txt1669# ================================================COMMENT
MEDIUMdata/Scripts.txt1680# ================================================COMMENT
MEDIUMdata/Scripts.txt1686# ================================================COMMENT
MEDIUMdata/Scripts.txt1696# ================================================COMMENT
MEDIUMdata/Scripts.txt1804# ================================================COMMENT
MEDIUMdata/Scripts.txt1813# ================================================COMMENT
MEDIUMdata/Scripts.txt1821# ================================================COMMENT
MEDIUMdata/Scripts.txt1828# ================================================COMMENT
MEDIUMdata/Scripts.txt1902# ================================================COMMENT
MEDIUMdata/Scripts.txt1911# ================================================COMMENT
MEDIUMdata/Scripts.txt1920# ================================================COMMENT
MEDIUMdata/Scripts.txt1926# ================================================COMMENT
MEDIUMdata/Scripts.txt1932# ================================================COMMENT
MEDIUMdata/Scripts.txt1939# ================================================COMMENT
MEDIUMdata/Scripts.txt1999# ================================================COMMENT
MEDIUMdata/Scripts.txt2006# ================================================COMMENT
MEDIUMdata/Scripts.txt2012# ================================================COMMENT
MEDIUMdata/Scripts.txt2026# ================================================COMMENT
MEDIUMdata/Scripts.txt2034# ================================================COMMENT
MEDIUMdata/Scripts.txt2044# ================================================COMMENT
MEDIUMdata/Scripts.txt2085# ================================================COMMENT
MEDIUMdata/Scripts.txt2093# ================================================COMMENT
MEDIUMdata/Scripts.txt2101# ================================================COMMENT
MEDIUMdata/Scripts.txt2108# ================================================COMMENT
MEDIUMdata/Scripts.txt2115# ================================================COMMENT
MEDIUMdata/Scripts.txt2121# ================================================COMMENT
MEDIUMdata/Scripts.txt2185# ================================================COMMENT
MEDIUMdata/Scripts.txt2195# ================================================COMMENT
MEDIUMdata/Scripts.txt2201# ================================================COMMENT
MEDIUMdata/Scripts.txt2211# ================================================COMMENT
MEDIUMdata/Scripts.txt2252# ================================================COMMENT
MEDIUMdata/Scripts.txt2259# ================================================COMMENT
MEDIUMdata/Scripts.txt2267# ================================================COMMENT
MEDIUMdata/Scripts.txt2342# ================================================COMMENT
MEDIUMdata/Scripts.txt2350# ================================================COMMENT
85 more matches not shown…
Over-Commented Block80 hits · 80 pts
SeverityFileLineSnippetContext
LOWCMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWpython/README.md61# Encode text to pieces as bytesCOMMENT
LOWpython/README.md81# Output:COMMENT
LOWpython/README.md101# ['▁', 'This', '▁', 'is', '▁a', '▁', 't', 'e', 'st']COMMENT
LOWpython/README.md121ids_np = np.array([284, 47, 11, 4, 15, 400], dtype=np.int32)COMMENT
LOWpython/setup.py1#!/usr/bin/env pythonCOMMENT
LOWpython/tools/gen_stubs.py1#!/usr/bin/env pythonCOMMENT
LOWpython/test/gen_stubs_test.py1# Copyright 2024 Google Sepm.COMMENT
LOWpython/test/sentencepiece_test.py1#!/usr/bin/pythonCOMMENT
LOWcontrib/nlcodec/bpe_model_trainer_nlcodec.h1// Copyright 2024 nlcodec / Thamme GowdaCOMMENT
LOWthird_party/darts_clone/darts.h1#ifndef DARTS_H_COMMENT
LOWthird_party/darts_clone/darts.h101COMMENT
LOWthird_party/darts_clone/darts.h141 // member variables with 0 and NULLs.COMMENT
LOWthird_party/darts_clone/darts.h221COMMENT
LOWthird_party/darts_clone/darts.h241 Details::progress_func_type progress_func = NULL);COMMENT
LOWthird_party/darts_clone/darts.h261 // if it exists, its value and length are set to `result'. Otherwise, theCOMMENT
LOWthird_party/darts_clone/darts.h281COMMENT
LOW.github/workflows/requirements/cibuildwheel.txt1#COMMENT
LOW.github/workflows/requirements/cibuildwheel.txt81 # -c base.txtCOMMENT
LOWdoc/special_symbols.md141# Output:COMMENT
LOWdata/Scripts.txt1# Scripts-9.0.0.txtCOMMENT
LOWdata/gen_unicode_scripts_code.pl1#!/usr/bin/perlCOMMENT
LOWsrc/filesystem.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/freelist.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/unicode_script.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/model_interface.h21#include <utility>COMMENT
LOWsrc/CMakeLists.txt1# Copyright 2018 Google Inc.COMMENT
LOWsrc/bpe_model.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/testharness.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/testharness.h121 .IsEq(std::string(a), std::string(b), #a, #b)COMMENT
LOWsrc/word_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_trainer.h1// Copyright 2018 Google Inc.COMMENT
LOWsrc/sentencepiece_trainer.h41COMMENT
LOWsrc/char_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/trainer_interface.h21#include <string>COMMENT
LOWsrc/common.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/common.h21#include "third_party/absl/log/check.h"COMMENT
LOWsrc/common.h41#define FRIEND_TEST(a, b) friend class a##_Test_##b;COMMENT
LOWsrc/unigram_model_trainer.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/unigram_model_trainer.h81 // node_int_type should be of integer type (int32_t or int64_t),COMMENT
LOWsrc/unigram_model.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/unigram_model.h101 // generation probability (Product of piece probabilities).COMMENT
LOWsrc/sentencepiece_processor.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/sentencepiece_processor.h21#include <memory>COMMENT
LOWsrc/sentencepiece_processor.h41// SentencePieceProcessor provides Encode() and Decode() methods,COMMENT
LOWsrc/sentencepiece_processor.h61//COMMENT
LOWsrc/sentencepiece_processor.h121// Currently, the C++ API does not include a dedicated batch processing API.COMMENT
LOWsrc/sentencepiece_processor.h361 // - Unigram (--model_type=unigram):COMMENT
LOWsrc/sentencepiece_processor.h381 // Same as above, but returns a sequence of ids.COMMENT
LOWsrc/sentencepiece_processor.h421 virtual absl::Status CalculateEntropy(absl::string_view input, float alpha,COMMENT
LOWsrc/sentencepiece_processor.h781 // Returns -1 if not defined.COMMENT
LOWsrc/sentencepiece_processor.h901// IO related functions to absorb model formats.COMMENT
LOWsrc/spec_parser.h1// Copyright 2016 Google LLC.COMMENT
LOWsrc/spec_parser.h281#undef PARSE_DUOBLECOMMENT
LOWsrc/unicode_script_map.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/util.h1// Copyright 2016 Google Inc.COMMENT
LOWsrc/util.h21#include <algorithm>COMMENT
LOWsrc/util.h281std::wstring Utf8ToWide(const absl::string_view input);COMMENT
20 more matches not shown…
Hyper-Verbose Identifiers38 hits · 38 pts
SeverityFileLineSnippetContext
LOWpython/test/numpy_test.py57 def test_return_type_validation_and_compat(self):CODE
LOWpython/test/numpy_test.py101 def test_decode_numpy_batch_list(self):CODE
LOWpython/test/numpy_test.py127 def test_decode_numpy_non_contiguous(self):CODE
LOWpython/test/numpy_test.py146 def test_decode_numpy_invalid_types(self):CODE
LOWpython/test/numpy_test.py157 def test_parallel_encode_numpy(self):CODE
LOWpython/test/numpy_test.py240 def test_decode_invalid_ids_numpy(self):CODE
LOWpython/test/numpy_test.py249 def test_decode_numpy_batch_mixed(self):CODE
LOWpython/test/numpy_test.py261 def test_native_batch_id_to_piece_numpy(self):CODE
LOWpython/test/numpy_test.py277 def test_native_batch_other_id_methods_numpy(self):CODE
LOWpython/test/gil_release_test.py31def run_heavy_op_with_gil_check(op_func, name):CODE
LOWpython/test/clean_sentencepiece_test_manual.py115 def test_encode_decode_types_single(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py300 def test_encode_extra_options(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py402 def test_sample_encode_and_score(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py503 def test_normalization_and_entropy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py642 def run_heavy_op_with_gil_check(op_func, name):CODE
LOWpython/test/clean_sentencepiece_test_manual.py701 def test_gil_release_encode_with_list_modification(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py722 def test_gil_release_decode_pieces_with_list_modification(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py742 def test_encode_bytes_is_zero_copy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py778 def test_encode_str_is_zero_copy(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py961 def test_train_with_writer_only(self):CODE
LOWpython/test/clean_sentencepiece_test_manual.py972 def test_train_with_logstream(self):CODE
LOWpython/test/gen_stubs_test.py45 def test_committed_stub_in_sync(self):CODE
LOWpython/test/gen_stubs_test.py101 def test_leaked_module_level_import_ignored(self):CODE
LOWpython/test/sentencepiece_test.py250 def test_special_tokens_combinations(self):CODE
LOWpython/test/sentencepiece_test.py701 def test_classmethod_factories(self):CODE
LOWpython/test/sentencepiece_test.py771 def test_word_model_user_defined_symbol(self):CODE
LOWpython/test/sentencepiece_test.py1164 def test_trainer_with_normalizer(self):CODE
LOWpython/test/sentencepiece_test.py1199 def test_override_normalize_spec(self):CODE
LOWpython/test/sentencepiece_test.py1288 def test_decode_offset_mapping(self):CODE
LOWpython/test/sentencepiece_test.py1364 def test_decode_return_type_bytes(self):CODE
LOWpython/test/sentencepiece_test.py1401 def test_legacy_out_type_compat(self):CODE
LOWpython/test/sentencepiece_test.py1430 def test_normalizer_model_proto(self):CODE
LOWpython/test/sentencepiece_test.py1438 def test_encode_return_type_explicit(self):CODE
LOWpython/test/sentencepiece_test.py1510 def test_native_batch_piece_to_id(self):CODE
LOWpython/test/sentencepiece_test.py1540 def test_native_batch_id_to_piece(self):CODE
LOWpython/test/sentencepiece_test.py1574 def test_native_batch_other_id_methods(self):CODE
LOWpython/src/sentencepiece/__init__.py165 def _resolve_and_validate_return_type(self, return_type):CODE
LOWpython/src/sentencepiece/__init__.py875 def serialized_normalizer_spec(self):CODE
Excessive Try-Catch Wrapping11 hits · 17 pts
SeverityFileLineSnippetContext
MEDIUMbenchmark/analyze_results.py7 print("Error: 'results.json' not found.")CODE
MEDIUMbenchmark/analyze_results.py15 print("Error: 'results.json' is empty.")CODE
LOWbenchmark/download_models.py11except Exception as e:CODE
LOWbenchmark/download_models.py18except Exception as e:CODE
LOWbenchmark/download_models.py27except Exception as e:CODE
LOWbenchmark/download_models.py35except Exception as e:CODE
MEDIUMbenchmark/run_bench.py139 print(f"Error: Dataset file '{path}' not found.")CODE
MEDIUMbenchmark/run_bench.py163 print("Error: No tasks available to run. Please run 'python download_models.py' first.")CODE
LOWpython/test/clean_sentencepiece_test_manual.py753 except Exception as e:CODE
LOWpython/test/clean_sentencepiece_test_manual.py832 except Exception as e:CODE
LOWpython/src/sentencepiece/__init__.py925except Exception:CODE
Cross-Language Confusion2 hits · 15 pts
SeverityFileLineSnippetContext
HIGHpython/test/sentencepiece_test.py320 # suppress logging (redirect to /dev/null)COMMENT
HIGHpython/test/sentencepiece_test.py360 # suppress logging (redirect to /dev/null)COMMENT
Deep Nesting8 hits · 8 pts
SeverityFileLineSnippetContext
LOWbenchmark/run_bench.py37CODE
LOWbenchmark/run_bench.py62CODE
LOWbenchmark/run_bench.py122CODE
LOWbenchmark/run_bench.py135CODE
LOWpython/setup.py162CODE
LOWpython/tools/gen_stubs.py414CODE
LOWpython/src/sentencepiece/__init__.py784CODE
LOWpython/src/sentencepiece/__init__.py834CODE
Self-Referential Comments2 hits · 6 pts
SeverityFileLineSnippetContext
MEDIUMpython/tools/gen_stubs.py48# This file is auto-generated by python/tools/gen_stubs.py.COMMENT
MEDIUMpython/test/numpy_test.py206 # Create a processor local to this scopeCOMMENT
AI Structural Patterns5 hits · 5 pts
SeverityFileLineSnippetContext
LOWpython/src/sentencepiece/__init__.py137CODE
LOWpython/src/sentencepiece/__init__.py220CODE
LOWpython/src/sentencepiece/__init__.py437CODE
LOWpython/src/sentencepiece/__init__.py511CODE
LOWpython/src/sentencepiece/__init__.py834CODE
Structural Annotation Overuse3 hits · 4 pts
SeverityFileLineSnippetContext
LOWdoc/model_proto.md56### Step 1: Define your extensionCOMMENT
LOWdoc/model_proto.md73### Step 2: Compile your protoCOMMENT
LOWdoc/model_proto.md81### Step 3: Read/Write Custom Metadata in PythonCOMMENT
Unused Imports3 hits · 3 pts
SeverityFileLineSnippetContext
LOWpython/setup.py17CODE
LOWpython/test/sentencepiece_test.py23CODE
LOWpython/src/sentencepiece/__init__.py921CODE
Redundant / Tautological Comments1 hit · 2 pts
SeverityFileLineSnippetContext
LOWpython/test/clean_sentencepiece_test_manual.py984 # Check if log file is not emptyCOMMENT
Fake / Example Data1 hit · 1 pts
SeverityFileLineSnippetContext
LOWdoc/model_proto.md95model_proto.Extensions[my_metadata_pb2.model_author] = "John Doe"CODE