Repository Analysis

lakehq/sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

3.6 Likely human-written View on GitHub

Analysis Overview

This report presents the forensic synthetic code analysis of lakehq/sail, a Rust project with 3,278 GitHub stars. SynthScan v2.0 examined 545,418 lines of code across 2002 source files, recording 1561 pattern matches distributed across 17 syntactic categories. The overall adjusted score of 3.6 places this repository in the Likely human-written band.

The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).

3.6
Adjusted Score
3.6
Raw Score
100%
Time Factor
2026-08-09
Last Push
3.3K
Stars
Rust
Language
545.4K
Lines of Code
2.0K
Files
1.6K
Pattern Hits
2026-08-09
Scan Date
0.01
HC Hit Rate

What These Metrics Mean

Adjusted Score
Primary synthetic code indicator. Raw score normalised per 1,000 lines of code and multiplied by the temporal discount factor. This is the definitive comparative metric — use it to rank repositories by AI authorship density.
Raw Score
The unmodified sum of all severity-weighted, context-multiplied pattern match scores before temporal discounting. Reflects the absolute signal strength independent of when the repository was last active.
Time Factor
The temporal discount multiplier (0–100%) applied to the raw score. Repositories last updated before ChatGPT's launch (Nov 2022) receive a 5% factor. Full signal is only assigned to repositories active in the post-adoption era (Jan 2024+).
Pattern Hits
Total count of individual pattern matches across all files and categories. A high hit count with a low score may indicate a very large codebase with isolated AI snippets; a low count with a high score indicates dense, concentrated AI signatures.
HC Hit Rate
High+Critical pattern hits per file, averaged across the repository. This orthogonal signal catches repositories where a few files are densely packed with high-severity AI tells — a strong indicator even when the normalised score appears moderate due to codebase size.
Lines of Code / Files
Total lines and files analysed. The scanner examines 94 file extensions. These denominators are used to normalise the score, enabling fair comparison between repositories of vastly different sizes.

Score History

Longitudinal tracking requires multiple scan runs. Once this repository is re-scanned after new commits land, this chart will visualise how the synthetic code signal evolves over time — enabling you to detect whether AI authorship is growing, stabilising, or being actively corrected by human engineers.

No multi-scan history yet — run the scanner again to build trend data.

Severity Breakdown

Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.

CRITICAL 11HIGH 4MEDIUM 161LOW 1385

Directory Score Breakdown

This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.

Pattern Findings

The scanner identified 1561 distinct pattern matches across 17 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.

Hyper-Verbose Identifiers947 hits · 920 pts
SeverityFileLineSnippetContext
LOWcrates/sail-python-udf/src/python/spark.py73def _pandas_serializer_kwargs(config) -> dict[str, bool]:CODE
LOWcrates/sail-python-udf/src/python/spark.py600def _python_values_to_arrow_array(CODE
LOWcrates/sail-python-udf/src/python/spark.py625def _arrow_array_to_output_type(data, data_type: pa.DataType) -> pa.Array:CODE
LOWcrates/sail-cli/src/python/spark_mcp_server.py48def override_default_logging_config():CODE
LOWcrates/sail-cli/src/python/spark_mcp_server.py58def configure_fastmcp_log_level():CODE
LOWpython/pysail/tests/conftest.py56def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:CODE
LOWpython/pysail/tests/conftest.py80def configure_sail_environment():CODE
LOWpython/pysail/tests/flight/test_flight.py31def test_flight_server_connectivity(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py37def test_flight_execute_simple_query(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py50def test_flight_multiple_columns(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py61def test_flight_error_handling(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py118def test_flight_string_operations(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py135def test_flight_numeric_operations(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py153def test_flight_create_temp_view(flight_connection):CODE
LOWpython/pysail/tests/flight/test_flight.py197def test_flight_multiple_cursors(flight_connection):CODE
LOWpython/pysail/tests/celeborn/test_lifecycle.py36def test_lifecycle_manager_registers_requests_slots_and_unregisters(CODE
LOWpython/pysail/tests/celeborn/test_lifecycle.py46def test_lifecycle_manager_returns_registration_failure() -> None:CODE
LOWpython/pysail/tests/celeborn/conftest.py186def replication_endpoint_resolver(CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py48def test_shuffle_client_registers_and_unregisters(CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py58def test_shuffle_client_pushes_and_reads_partition(CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py69def test_shuffle_client_commits_after_all_mappers_end(CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py82def test_shuffle_client_replicates_data(CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py105def test_shuffle_client_returns_registration_failure() -> None:CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py114def test_shuffle_client_stop_does_not_stop_lifecycle_manager(CODE
LOWpython/pysail/tests/spark/test_basic.py95def test_schema_simple_string(spark):CODE
LOWpython/pysail/tests/spark/test_basic.py100def test_data_frame_operations(df):CODE
LOWpython/pysail/tests/spark/test_basic.py252def test_write_partitioned_parquet(spark, df, tmpdir):CODE
LOWpython/pysail/tests/spark/test_basic.py409def test_udf_binary_view_input(spark, tmp_path):CODE
LOWpython/pysail/tests/spark/test_basic.py425def test_sql_named_window_clause(spark, df_view):CODE
LOWpython/pysail/tests/spark/test_basic.py454def test_sql_dataframe_argument(spark):CODE
LOWpython/pysail/tests/spark/test_basic.py471def _assert_coalesce_string_date_ansi_error(df):CODE
LOWpython/pysail/tests/spark/test_basic.py478def test_coalesce_string_literal_and_date_default_ansi_enabled(spark):CODE
LOWpython/pysail/tests/spark/test_basic.py485def test_coalesce_string_literal_and_date_ansi_disabled(spark):CODE
LOWpython/pysail/tests/spark/test_basic.py501def test_coalesce_string_literal_and_date_ansi_enabled(spark):CODE
LOWpython/pysail/tests/spark/test_basic.py525def test_large_plan_no_proto_recursion_limit(spark):CODE
LOWpython/pysail/tests/spark/conftest.py191def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:CODE
LOWpython/pysail/tests/spark/test_clickbench.py154def test_clickbench_query_plan(spark, q, queries, snapshot):CODE
LOWpython/pysail/tests/spark/test_tpcds.py36def test_derived_tpcds_query_result(spark, query, snapshot):CODE
LOWpython/pysail/tests/spark/test_tpcds.py49def test_derived_tpcds_query_plan(spark, query, snapshot):CODE
LOWpython/pysail/tests/spark/test_tpch.py32def test_derived_tpch_query_result(spark, query, snapshot):CODE
LOWpython/pysail/tests/spark/test_tpch.py42def test_derived_tpch_query_plan(spark, query, snapshot):CODE
LOWpython/pysail/tests/spark/test_config.py15def test_warning_on_static_config_addition(monkeypatch):CODE
LOWpython/pysail/tests/spark/test_config.py23def test_warning_on_static_config_deletion(monkeypatch):CODE
LOWpython/pysail/tests/spark/test_config.py30def test_warning_on_static_config_modification(monkeypatch):CODE
LOW…n/pysail/tests/spark/delta/test_delta_write_options.py8def test_delta_write_option_write_batch_size(spark, tmp_path):CODE
LOW…n/pysail/tests/spark/delta/test_delta_write_options.py38def test_delta_write_option_target_file_size(spark, tmp_path):CODE
LOW…n/pysail/tests/spark/delta/test_delta_write_options.py70def test_delta_write_options_combined(spark, tmp_path):CODE
LOW…n/pysail/tests/spark/delta/test_delta_write_options.py103def test_delta_write_options_with_partitioning(spark, tmp_path):CODE
LOW…l/tests/spark/delta/test_delta_merge_into_dataframe.py14def test_dataframe_merge_into_basic(spark, tmp_path):CODE
LOW…l/tests/spark/delta/test_delta_merge_into_dataframe.py43def test_dataframe_merge_into_insert_all_and_delete(spark, tmp_path):CODE
LOW…l/tests/spark/delta/test_delta_merge_into_dataframe.py79def test_merge_into_path_target_with_dataframe_source(spark, tmp_path):CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py9def _create_empty_generated_column_table(table_path: Path) -> None:CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py61def test_dataframewriter_path_append_computes_omitted_generated_column(spark, tmp_path):CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py83def test_dataframewriter_path_append_rejects_generated_column_mismatch(spark, tmp_path):CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py96def test_dataframewriter_path_overwrite_computes_omitted_generated_column(spark, tmp_path):CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py113def test_dataframewriter_path_overwrite_rejects_generated_column_mismatch(spark, tmp_path):CODE
LOW…sail/tests/spark/delta/test_delta_generated_columns.py126def test_dataframewriter_path_overwrite_schema_does_not_rewrite_old_generated_column(spark, tmp_path):CODE
LOW…n/pysail/tests/spark/delta/test_delta_metadata_only.py56def test_read_metadata_only_table_without_add_actions(spark, tmp_path: Path, read_mode: str):CODE
LOW…s/spark/delta/test_delta_column_mapping_rename_drop.py33def _mapping_physical_metadata(field_meta: dict) -> dict:CODE
LOW…s/spark/delta/test_delta_column_mapping_rename_drop.py43def test_column_mapping_rename_preserves_physical(spark, tmp_path: Path):CODE
887 more matches not shown…
Decorative Section Separators150 hits · 465 pts
SeverityFileLineSnippetContext
MEDIUMcrates/sail-delta-lake/src/snapshot/mod.rs728 // ── Capability token verification ─────────────────────────────────────────COMMENT
MEDIUM…sail-delta-lake/src/physical_plan/scan_by_adds_exec.rs343 // ── Fast path: bulk scan for files without deletion vectors ──────────COMMENT
MEDIUM…sail-delta-lake/src/physical_plan/scan_by_adds_exec.rs355 // ── DV path: per-file scan with row-index filtering ─────────────────COMMENT
MEDIUMcrates/sail-delta-lake/src/deletion_vector/resolve.rs79// ── Z85 (base-85) encoding for UUIDs ─────────────────────────────────────────COMMENT
MEDIUMcrates/sail-delta-lake/src/schema/arrow_conversions.rs20// ── Delta → Arrow ────────────────────────────────────────────────────────────COMMENT
MEDIUMcrates/sail-delta-lake/src/schema/arrow_conversions.rs139// ── Arrow → Delta ────────────────────────────────────────────────────────────COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs218/// ┌─────────┐ ┌─────────┐ ┌──────────────┐COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs219/// │ ┌─────┐ │ │ ┌─────┐ │ │ ┌──────────┐ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs221/// │ ├─────┤ │ │ ├─────┤ │ │ ├──────────┤ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs222/// │ │NULL │ │ │ │ Y │ │──────────▶│ │[NULL, Y] │ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs223/// │ ├─────┤ │ │ ├─────┤ │ │ ├──────────┤ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs225/// │ └─────┘ │ │ └─────┘ │ │ └──────────┘ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs226/// └─────────┘ └─────────┘ └──────────────┘COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs237/// ┌──────────────┐ ┌──────────────┐ ┌─────────────────────────────┐COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs238/// │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌────────────────────────┐ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs240/// │ ├──────────┤ │ │ ├──────────┤ │ │ ├────────────────────────┤ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs241/// │ │[NULL, Y] │ │ │ │[Q, R, S] │ │───────▶│ │ [[NULL, Y], [Q, R, S]] │ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs242/// │ ├──────────┤ │ │ ├──────────┤ │ │ ├────────────────────────│ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs244/// │ └──────────┘ │ │ └──────────┘ │ │ └────────────────────────┘ │COMMENT
MEDIUMcrates/sail-function/src/scalar/array/spark_array.rs245/// └──────────────┘ └──────────────┘ └─────────────────────────────┘COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_parquet.py291# -----------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_parquet.py297# -----------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py53# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py55# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py190# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py192# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py219# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py221# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py638# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py640# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py918# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py920# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py1449# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_python.py1451# ============================================================================COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py92# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py94# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py103# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py105# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py121# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py123# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py132# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py134# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py170# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py172# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py189# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py191# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py201# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py203# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py216# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py218# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py226# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py228# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py242# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py244# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py253# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py255# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py264# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py266# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py78# ---------------------------------------------------------------------------COMMENT
MEDIUMpython/pysail/tests/spark/datasource/test_jdbc.py80# ---------------------------------------------------------------------------COMMENT
90 more matches not shown…
Over-Commented Block305 hits · 305 pts
SeverityFileLineSnippetContext
LOWcrates/sail-execution/src/driver/output.rs101 break;COMMENT
LOWcrates/sail-execution/src/driver/job_scheduler/core.rs101 .and_then(|task| task.attempts.get(key.attempt));COMMENT
LOWcrates/sail-execution/src/driver/task_assigner/state.rs81 ///COMMENT
LOWcrates/sail-execution/src/driver/task_assigner/state.rs161COMMENT
LOWcrates/sail-execution/src/driver/actor/handler.rs241 .get_task_state(&key)COMMENT
LOWcrates/sail-execution/src/job_graph/mod.rs141 /// For each partition in the current stage, execute the same partition to fetch the inputCOMMENT
LOWcrates/sail-execution/src/task/scheduling.rs1use crate::id::{TaskKey, WorkerId};COMMENT
LOWcrates/sail-physical-plan/src/barrier.rs1use std::sync::Arc;COMMENT
LOWcrates/sail-plan/src/function/scalar/json.rs21/// `JsonPath::Index` respectively).COMMENT
LOWcrates/sail-plan/src/function/scalar/math.rs41/// - right: If left is a numeric right must be numeric expression, or an INTERVAL otherwise.COMMENT
LOWcrates/sail-plan/src/function/scalar/math.rs81 })COMMENT
LOWcrates/sail-plan/src/function/scalar/math.rs261 expr_fn::nullif(divisor, lit(0))COMMENT
LOWcrates/sail-plan/src/resolver/data_type.rs21COMMENT
LOWcrates/sail-plan/src/resolver/command/write.rs41use sail_function::scalar::misc::raise_error::RaiseError;COMMENT
LOWcrates/sail-plan/src/resolver/command/write.rs61 /// If the target exists, return an error.COMMENT
LOWcrates/sail-plan/src/resolver/command/write.rs81 ///COMMENT
LOWcrates/sail-plan/src/resolver/command/write.rs881 .map(|(column, name)| col(column).alias(name))COMMENT
LOWcrates/sail-plan/src/resolver/command/write.rs1141 ///COMMENT
LOWcrates/sail-plan/src/resolver/query/pivoting.rs201 /// Infer pivot values by collecting the distinct values of the pivot columnCOMMENT
LOW…/sail-data-source/src/formats/python/table_provider.rs21///COMMENT
LOWcrates/sail-data-source/src/formats/python/stream.rs21/// # Timeout LimitationsCOMMENT
LOWcrates/sail-data-source/src/formats/python/stream.rs101/// Stream that reads RecordBatches from a Python datasource.COMMENT
LOWcrates/sail-data-source/src/formats/python/exec.rs1//! Execution plan for Python data source batch reads.COMMENT
LOWcrates/sail-data-source/src/formats/python/exec.rs41COMMENT
LOWcrates/sail-data-source/src/formats/python/error.rs1//! Error types for Python data source operations.COMMENT
LOW…tes/sail-data-source/src/formats/python/arrow_utils.rs1//! Arrow conversion utilities for Python data sources.COMMENT
LOW…es/sail-data-source/src/formats/python/table_format.rs21use sail_common_datafusion::utils::items::ItemTaker;COMMENT
LOW…ates/sail-data-source/src/formats/python/write_exec.rs1//! Distributed write execution plan for Python data sources.COMMENT
LOWcrates/sail-data-source/src/formats/python/mod.rs1//! Python data source support for Sail.COMMENT
LOWcrates/sail-data-source/src/formats/python/executor.rs41 }COMMENT
LOWcrates/sail-data-source/src/formats/python/executor.rs61pub struct WriterPlan {COMMENT
LOWcrates/sail-data-source/src/formats/python/executor.rs81///COMMENT
LOWcrates/sail-data-source/src/formats/python/executor.rs161}COMMENT
LOWcrates/sail-data-source/src/formats/python/filter.rs1//! Filter pushdown system for Python data sources.COMMENT
LOW…ates/sail-data-source/src/formats/python/datasource.rs21/// # Caching StrategyCOMMENT
LOW…ates/sail-data-source/src/formats/python/datasource.rs121 *cache = Some(ds.clone().unbind());COMMENT
LOWcrates/sail-data-source/src/formats/text/reader.rs121// [Credit]: https://github.com/apache/arrow-rs/blob/ebb6ede98b2b4d96a1a4f501a28ab42a3b937f73/arrow-csv/src/reader/mod.rCOMMENT
LOWcrates/sail-spark-connect/build.rs81 version: String,COMMENT
LOWcrates/sail-spark-connect/src/error.rs281 }COMMENT
LOWcrates/sail-pyarrow/src/lib.rs1// Licensed to the Apache Software Foundation (ASF) under oneCOMMENT
LOWcrates/sail-pyarrow/src/lib.rs21//!COMMENT
LOWcrates/sail-pyarrow/src/lib.rs41//! | `pyarrow.DataType` | [DataType] |COMMENT
LOWcrates/sail-pyarrow/src/lib.rs481/// This could be used in circumstances where you either want to consume a `pyarrow.Table` directlyCOMMENT
LOWcrates/sail-delta-lake/Cargo.toml1# Licensed under the Apache License, Version 2.0 (the "License");COMMENT
LOWcrates/sail-delta-lake/src/lib.rs1// Licensed under the Apache License, Version 2.0 (the "License");COMMENT
LOWcrates/sail-delta-lake/src/options.rs1// Licensed under the Apache License, Version 2.0 (the "License");COMMENT
LOWcrates/sail-delta-lake/src/datasource/actions.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/datasource/pruning.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/datasource/expressions.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/datasource/mod.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/datasource/mod.rs81pub struct DeltaScanConfigBuilder {COMMENT
LOWcrates/sail-delta-lake/src/datasource/schema.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/datasource/scan.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/transaction/protocol.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/transaction/mod.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
LOWcrates/sail-delta-lake/src/transaction/mod.rs981}COMMENT
LOWcrates/sail-delta-lake/src/transaction/mod.rs1001// self.create_checkpoint = create_checkpoint;COMMENT
LOWcrates/sail-delta-lake/src/transaction/mod.rs1081 // pub fn with_app_metadata(mut self, app_metadata: HashMap<String, Value>) -> Self {COMMENT
LOWcrates/sail-delta-lake/src/transaction/mod.rs1101 // selfCOMMENT
LOW…es/sail-delta-lake/src/transaction/conflict_checker.rs1// https://github.com/delta-io/delta-rs/blob/5575ad16bf641420404611d65f4ad7626e9acb16/LICENSE.txtCOMMENT
245 more matches not shown…
Hallucination Indicators11 hits · 110 pts
SeverityFileLineSnippetContext
CRITICALcrates/sail-spark-connect/data/config/spark-3.5.json1081 "doc": "When set to true, user is allowed to use org.apache.spark.sql.functions.udf(f: AnyRef, dataType: DataType)CODE
CRITICALcrates/sail-session/src/runtime.rs74 let paths = self.config.runtime.temporary_files.paths.as_slice();CODE
CRITICALcrates/sail-session/src/session_factory/server.rs121 self.config.execution.checkpoint.path.clone(),CODE
CRITICAL…spark/catalog/hms/test_spark_to_sail_dsv2_roundtrip.py145 hive_conf = jvm.org.apache.hadoop.hive.conf.HiveConf()CODE
CRITICAL…spark/catalog/hms/test_spark_to_sail_dsv2_roundtrip.py147 client = jvm.org.apache.hadoop.hive.metastore.HiveMetaStoreClient(hive_conf)CODE
CRITICAL…tests/spark/catalog/hms/test_iceberg_catalog_commit.py39 hive_conf = jvm.org.apache.hadoop.hive.conf.HiveConf()CODE
CRITICAL…tests/spark/catalog/hms/test_iceberg_catalog_commit.py41 client = jvm.org.apache.hadoop.hive.metastore.HiveMetaStoreClient(hive_conf)CODE
CRITICALscripts/spark-config/generate.py58 clazz = spark._jvm.java.lang.Class.forName("org.apache.spark.sql.internal.SQLConf$")CODE
CRITICALscripts/spark-config/generate.py64 spark._jvm.org.apache.spark.util.Utils.classForName("org.apache.spark.sql.connect.config.Connect$", True, False)CODE
CRITICAL.github/actions/commit-workflow/action.yml35 const workflows = await github.paginate(github.rest.actions.listWorkflowRunsForRepo.endpoint.merge({CODE
CRITICAL.github/actions/commit-workflow/action.yml43 const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts.endpoint.merge({CODE
Unused Imports56 hits · 53 pts
SeverityFileLineSnippetContext
LOWcrates/sail-python-udf/src/python/spark.py1CODE
LOWpython/pysail/flight/__init__.py1CODE
LOWpython/pysail/tests/celeborn/test_lifecycle.py3CODE
LOWpython/pysail/tests/celeborn/conftest.py3CODE
LOWpython/pysail/tests/celeborn/test_shuffle.py3CODE
LOWpython/pysail/tests/spark/conftest.py1CODE
LOWpython/pysail/tests/spark/delta/conftest.py1CODE
LOW…n/pysail/tests/spark/delta/test_delta_metadata_only.py1CODE
LOW…s/spark/delta/test_delta_column_mapping_rename_drop.py1CODE
LOW…s/spark/delta/test_delta_column_mapping_delta_spark.py3CODE
LOWpython/pysail/tests/spark/delta/test_delta_variant.py1CODE
LOW…n/pysail/tests/spark/delta/test_delta_user_metadata.py1CODE
LOW…/pysail/tests/spark/delta/test_delta_column_mapping.py1CODE
LOW…thon/pysail/tests/spark/delta/test_delta_checkpoint.py1CODE
LOWpython/pysail/tests/spark/datasource/test_jdbc.py7CODE
LOWpython/pysail/tests/spark/datasource/test_vortex.py7CODE
LOWpython/pysail/tests/spark/datasource/test_vortex.py13CODE
LOWpython/pysail/tests/spark/optimizer/test_features.py1CODE
LOW…ts/spark/catalog/unity/test_managed_delta_dataframe.py4CODE
LOWpython/pysail/tests/spark/catalog/unity/conftest.py6CODE
LOWpython/pysail/tests/spark/catalog/glue/conftest.py3CODE
LOW…pysail/tests/spark/catalog/glue/test_iceberg_commit.py4CODE
LOW…n/pysail/tests/spark/catalog/glue/test_delta_commit.py4CODE
LOW…on/pysail/tests/spark/catalog/iceberg_rest/conftest.py7CODE
LOW…log/iceberg_rest/test_namespace_separator_incorrect.py1CODE
LOW…tests/spark/catalog/iceberg_rest/test_view_metadata.py1CODE
LOW…il/tests/spark/catalog/iceberg_rest/test_sort_order.py1CODE
LOW…pysail/tests/spark/catalog/iceberg_rest/test_nessie.py3CODE
LOW…atalog/iceberg_rest/test_namespace_separator_custom.py1CODE
LOW…il/tests/spark/catalog/iceberg_rest/test_lakekeeper.py1CODE
LOW…pysail/tests/spark/catalog/iceberg_rest/test_commit.py1CODE
LOWpython/pysail/tests/spark/catalog/hms/conftest.py4CODE
LOW…ests/spark/catalog/hms/test_spark_to_sail_roundtrip.py8CODE
LOW…spark/catalog/hms/test_spark_to_sail_dsv2_roundtrip.py12CODE
LOWpython/pysail/tests/spark/catalog/hms/test_hms_smoke.py4CODE
LOW…ysail/tests/spark/catalog/hms/test_hms_s3_roundtrip.py4CODE
LOW…ests/spark/catalog/hms/test_sail_to_spark_roundtrip.py8CODE
LOW…tests/spark/catalog/hms/test_iceberg_catalog_commit.py4CODE
LOW…n/pysail/tests/spark/execution/test_shuffle_storage.py1CODE
LOWpython/pysail/docs/conf.py2CODE
LOWpython/pysail/testing/snapshot/yaml.py1CODE
LOWpython/pysail/testing/spark/session.py1CODE
LOWpython/pysail/testing/spark/utils/jvm.py1CODE
LOWpython/pysail/testing/spark/steps/plan.py1CODE
LOWpython/pysail/testing/spark/steps/files.py1CODE
LOWpython/pysail/testing/spark/steps/iceberg.py3CODE
LOWpython/pysail/testing/spark/steps/delta.py1CODE
LOWpython/pysail/testing/spark/steps/sql.py1CODE
LOWpython/pysail/examples/spark/convert_to_delta.py1CODE
LOWpython/pysail/examples/spark/tpch.py1CODE
LOWpython/pysail/spark/__init__.py1CODE
LOWpython/pysail/spark/datasource/vortex.py18CODE
LOWpython/pysail/spark/datasource/jdbc.py11CODE
LOWpython/pysail/spark/utils/_function_scanner.py7CODE
LOWscripts/spark-tests/plugins/ibis.py1CODE
LOWscripts/spark-tests/plugins/spark.py3CODE
Deep Nesting36 hits · 30 pts
SeverityFileLineSnippetContext
LOWcrates/sail-python-udf/src/python/spark.py938CODE
LOWcrates/sail-python-udf/src/python/spark.py1270CODE
LOWpython/pysail/tests/spark/conftest.py191CODE
LOWpython/pysail/tests/spark/delta/test_delta_variant.py131CODE
LOW…sail/tests/spark/delta/test_delta_partition_pruning.py205CODE
LOW…sail/tests/spark/delta/test_delta_partition_pruning.py378CODE
LOW…sail/tests/spark/delta/test_delta_partition_pruning.py483CODE
LOW…/pysail/tests/spark/delta/test_delta_column_mapping.py19CODE
LOW…/pysail/tests/spark/delta/test_delta_column_mapping.py39CODE
LOW…/pysail/tests/spark/delta/test_delta_column_mapping.py572CODE
LOW…pysail/tests/spark/delta/test_delta_schema_handling.py38CODE
LOW…thon/pysail/tests/spark/delta/test_delta_concurrent.py10CODE
LOWpython/pysail/tests/spark/datasource/test_python.py643CODE
LOWpython/pysail/tests/spark/datasource/test_python.py118CODE
LOWpython/pysail/tests/spark/datasource/test_python.py687CODE
LOWpython/pysail/tests/spark/datasource/test_socket.py51CODE
LOW…pysail/tests/spark/catalog/iceberg_rest/test_commit.py629CODE
LOW…hon/pysail/tests/spark/iceberg/test_iceberg_pruning.py134CODE
LOW…hon/pysail/tests/spark/iceberg/test_iceberg_pruning.py171CODE
LOWpython/pysail/testing/snapshot/yaml.py170CODE
LOWpython/pysail/testing/spark/utils/files.py33CODE
LOWpython/pysail/testing/spark/utils/files.py37CODE
LOWpython/pysail/testing/spark/steps/plan.py13CODE
LOWpython/pysail/testing/spark/steps/plan.py150CODE
LOWpython/pysail/testing/spark/steps/iceberg.py503CODE
LOWpython/pysail/testing/spark/steps/delta.py233CODE
LOWpython/pysail/testing/spark/steps/delta.py504CODE
LOWpython/pysail/testing/spark/steps/delta.py544CODE
LOWpython/pysail/testing/spark/steps/delta.py602CODE
LOWpython/pysail/examples/spark/tpch.py106CODE
LOWpython/pysail/examples/spark/tpch.py29CODE
LOWpython/pysail/examples/spark/tpch.py64CODE
LOWpython/pysail/examples/spark/tpch.py86CODE
LOWpython/pysail/spark/datasource/jdbc.py495CODE
LOWpython/pysail/spark/utils/_function_scanner.py72CODE
LOWscripts/common-gold-data/count_missing_functions.py8CODE
AI Slop Vocabulary6 hits · 18 pts
SeverityFileLineSnippetContext
MEDIUMcrates/sail-catalog-hms/tests/failover_tests.rs10#[ignore = "requires the HMS testcontainers harness"]COMMENT
MEDIUM…s/sail-catalog-hms/tests/kerberos_integration_tests.rs15#[ignore = "requires the Kerberos HMS testcontainers harness"]COMMENT
MEDIUM…s/sail-catalog-hms/tests/kerberos_integration_tests.rs25#[ignore = "subprocess helper for the Kerberos HMS testcontainers harness"]COMMENT
MEDIUM…s/sail-catalog-hms/tests/kerberos_integration_tests.rs87#[ignore = "requires the Kerberos HMS testcontainers harness"]COMMENT
MEDIUM…s/sail-catalog-hms/tests/kerberos_integration_tests.rs97#[ignore = "subprocess helper for the Kerberos HMS testcontainers harness"]COMMENT
MEDIUMcrates/sail-iceberg/src/datasource/pruning.rs33// TODO: Implement robust logical expression parsing for summary pruningCOMMENT
Excessive Try-Catch Wrapping16 hits · 16 pts
SeverityFileLineSnippetContext
LOWcrates/sail-data-source/src/formats/python/discovery.py24 except Exception: # noqa: BLE001CODE
LOW…thon/pysail/tests/spark/delta/test_delta_concurrent.py28 except Exception as exc: # noqa: BLE001CODE
LOW…thon/pysail/tests/spark/delta/test_delta_concurrent.py80 except Exception as exc: # noqa: BLE001CODE
LOW…thon/pysail/tests/spark/delta/test_delta_concurrent.py114 except Exception as exc: # noqa: BLE001CODE
LOWpython/pysail/tests/spark/datasource/test_python.py539 except Exception as e: # noqa: BLE001CODE
LOWpython/pysail/tests/spark/catalog/unity/conftest.py99 except Exception:CODE
LOW…on/pysail/tests/spark/catalog/iceberg_rest/conftest.py102 except Exception:CODE
LOWpython/pysail/tests/spark/catalog/hms/conftest.py144 except Exception as exc: # noqa: BLE001CODE
LOWpython/pysail/tests/spark/catalog/hms/conftest.py476 except Exception: # noqa: BLE001CODE
LOWpython/pysail/tests/spark/iceberg/utils.py51 except Exception: # noqa: S110, BLE001CODE
LOWpython/pysail/testing/spark/steps/delta.py218 except Exception as e:CODE
LOWpython/pysail/testing/spark/steps/sql.py82 except Exception: # noqa: BLE001CODE
LOWpython/pysail/spark/datasource/vortex.py199 except Exception as e:CODE
LOWpython/pysail/spark/datasource/vortex.py256 except Exception as e:CODE
LOWpython/pysail/spark/datasource/jdbc.py245 except Exception as e:CODE
LOWpython/pysail/spark/datasource/jdbc.py445 except Exception as e:CODE
Self-Referential Comments5 hits · 15 pts
SeverityFileLineSnippetContext
MEDIUMpython/pysail/tests/flight/test_flight.py157 # Create a temp viewCOMMENT
MEDIUMpython/pysail/tests/spark/test_clickbench.py124 # Create an empty dataset with the ClickBench schema.COMMENT
MEDIUM…thon/pysail/tests/spark/delta/test_delta_concurrent.py97 # Create the table first with a single write (version 0).COMMENT
MEDIUMscripts/spark-tests/run-tests.sh21 # Define a few environment variables if they are not set.COMMENT
MEDIUMscripts/spark-tests/run-tests.sh40# Define the directory for test logs. The `tmp/` directory is in `.gitignore`.COMMENT
Modern Structural Boilerplate9 hits · 10 pts
SeverityFileLineSnippetContext
LOWcrates/sail-data-source/src/formats/python/discovery.py6logger = logging.getLogger(__name__)CODE
LOWcrates/sail-cli/src/python/spark_mcp_server.py12logger = logging.getLogger(__name__)CODE
LOWpython/pysail/__init__.py7__all__ = ["__version__"]CODE
LOWpython/pysail/flight/__init__.py5__all__ = [CODE
LOWpython/pysail/tests/spark/datasource/test_python.py47def _update_test_state(path: str, **updates: Any) -> None:CODE
LOW…pysail/tests/spark/catalog/glue/test_iceberg_commit.py37def _update_glue_parameters(moto_endpoint: str, database: str, table: str, parameters: dict[str, str]) -> None:CODE
LOWpython/pysail/examples/spark/compatibility_check.py10logger = logging.getLogger(__name__)CODE
LOWpython/pysail/spark/__init__.py5__all__ = [CODE
LOWpython/pysail/spark/utils/_function_scanner.py20logger = logging.getLogger(__name__)CODE
Cross-Language Confusion2 hits · 8 pts
SeverityFileLineSnippetContext
HIGHcrates/sail-python-udf/src/python/spark.py251 """Converts data to string where the behavior is similar to the Java `Object.toString()` method."""STRING
HIGHpython/pysail/tests/spark/catalog/hms/conftest.py562 return value.toString() if hasattr(value, "toString") else str(value)CODE
Verbosity Indicators5 hits · 8 pts
SeverityFileLineSnippetContext
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs708 // Step 1: Write add/remove actions into a sidecar file using batched writes.COMMENT
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs751 // Step 2: Build the main V2 checkpoint with header actions + sidecar refs.COMMENT
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs811 // Step 3: Write _last_checkpoint hint pointing to the UUID-named checkpoint.COMMENT
LOWcrates/sail-delta-lake/src/delta_log/cleanup.rs289 // Step 1: Collect all sidecar paths referenced by remaining checkpoints.COMMENT
LOWcrates/sail-delta-lake/src/delta_log/cleanup.rs331 // Step 2: List all files in _sidecars/ and delete orphans older than 24 hours.COMMENT
Structural Annotation Overuse5 hits · 8 pts
SeverityFileLineSnippetContext
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs708 // Step 1: Write add/remove actions into a sidecar file using batched writes.COMMENT
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs751 // Step 2: Build the main V2 checkpoint with header actions + sidecar refs.COMMENT
LOWcrates/sail-delta-lake/src/checkpoint/mod.rs811 // Step 3: Write _last_checkpoint hint pointing to the UUID-named checkpoint.COMMENT
LOWcrates/sail-delta-lake/src/delta_log/cleanup.rs289 // Step 1: Collect all sidecar paths referenced by remaining checkpoints.COMMENT
LOWcrates/sail-delta-lake/src/delta_log/cleanup.rs331 // Step 2: List all files in _sidecars/ and delete orphans older than 24 hours.COMMENT
TODO Padding4 hits · 6 pts
SeverityFileLineSnippetContext
LOWcrates/sail-sql-analyzer/src/statement.rs2168// TODO: implement the conversion properly for column-level ALTER TABLE operationsCOMMENT
LOWcrates/sail-sql-analyzer/src/statement.rs2170 // TODO: implement the conversion properlyCOMMENT
LOWcrates/sail-python/src/globals.rs21 // TODO: implement a systematic way to manage static configCOMMENT
LOWcrates/sail-function/src/scalar/map/utils.rs195 // TODO: implement configuration and logic for spark.sql.mapKeyDedupPolicy=EXCEPTION (this is deCOMMENT
Magic Placeholder Names1 hit · 5 pts
SeverityFileLineSnippetContext
HIGHdocs/guide/integrations/langchain.md120export OPENAI_API_KEY="your-api-key"CODE
Synthetic Comment Markers1 hit · 5 pts
SeverityFileLineSnippetContext
HIGHdocs/guide/integrations/mcp-server.md58Please note that there is no guardrail right now to prevent Claude from generating SQL statements that will alter your dCODE
Fake / Example Data2 hits · 2 pts
SeverityFileLineSnippetContext
LOWcrates/sail-plan/data/functions/scalar/misc.yaml284 > SELECT from_protobuf(s, 'Person', '/path/to/descriptor.desc', map()) IS NULL AS result FROM (SELECT NAMED_STCODE
LOW…l-spark-connect/tests/gold_data/function/protobuf.json5 "query": "SELECT from_protobuf(s, 'Person', '/path/to/descriptor.desc', map()) IS NULL AS result FROM (SELECT NACODE