Home / apache / spark
AI Slop Code RIsk Monitor

apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

5.2 Moderate pattern signal View on GitHub

Maintenance and Governance Review

This report summarizes scanner evidence for apache/spark, a Scala project with 44,156 GitHub stars. SynthScan examined 3,311,757 lines of code across 12827 source files, recording 10043 pattern matches distributed across 26 syntactic categories. The legacy adjusted pattern signal is 5.2 (Moderate pattern signal); it is not an authorship estimate.

Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.

5.2
Adjusted Signal
5.2
Raw Pattern Score
100%
Time Factor
2026-10-09
Last Push
44.2K
Stars
Scala
Language
3.3M
Lines of Code
12.8K
Files
10.0K
Pattern Hits
2026-10-12
Scan Date
0.08
HC Hit Rate

Risk Review · v3

Use each vector to guide manual verification of the underlying findings.

Cognitive Ownership Risk 11.0/100 · LOW

Can maintainers explain intent, assumptions, and trade-offs? 7780 finding(s).

Hallucinated Dependency & Supply Chain Risk 9.7/100 · LOW

Are imports declared and package/API names verified? 1391 finding(s).

Agent Over-Engineering Ratio 8.8/100 · LOW

Are wrappers, guards, and repeated structures necessary? 4633 finding(s).

PR Review Velocity Impact 23.4/100 · LOW

Which changes need focused human review? 10043 finding(s).

What These Metrics Mean

Adjusted Signal
Legacy pattern score normalized per 1,000 scanned lines and multiplied by the repository-level time factor. It is not an estimate of AI authorship.
Raw Pattern Score
Sum of severity- and context-weighted pattern findings before applying the time factor.
Time Factor
A coarse multiplier based on repository last-push metadata. It does not identify when individual code was written.
Pattern Hits
Count of configured pattern findings. Review each location and surrounding code before acting.
HC Hit Rate
High- and critical-severity findings per scanned file. Severity is a configured detector weight, not proof of a vulnerability or authorship.
Lines / Files
Scan coverage denominators used by the legacy normalized score.

Pattern-Signal History

This chart tracks the legacy adjusted pattern score across scan runs. Changes can reflect source changes, scan coverage, or detector/scoring versions; they do not measure AI adoption or prove that a team refactored code. Compare runs only when their scoring versions are compatible. The dashed line tracks the raw pattern-hit count.

Severity Breakdown

Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.

CRITICAL 490HIGH 478MEDIUM 417LOW 8658

Directory Score Breakdown

This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.

Pattern Findings

The scanner identified 10043 distinct pattern matches across 26 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.

Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.

Hallucination Indicators490 hits · 5288 pts
SeverityFileLineSnippetContext
CRITICAL…rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala137 case e: Throwable if org.apache.commons.lang3.exception.ExceptionUtils.indexOfThrowable(STRING
CRITICAL…cheduler/cluster/YarnClientSchedulerBackendSuite.scala55 Some(org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(0L, 1)))CODE
CRITICAL⚡…org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala49 assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("first"))CODE
CRITICAL⚡…org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala56 assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("second"))CODE
CRITICAL⚡…org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala63 assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("second"))CODE
CRITICAL…k/scheduler/cluster/k8s/DeploymentAllocatorSuite.scala149 assert(deployment.getSpec.getTemplate.getSpec.getContainers.asScala.exists(CODE
CRITICAL…heduler/cluster/k8s/ExecutorPVCResizePluginSuite.scala93 captor.getValue.getSpec.getResources.getRequests.get("storage")).longValue()CODE
CRITICAL…heduler/cluster/k8s/ExecutorPVCResizePluginSuite.scala197 captor.getValue.getSpec.getResources.getRequests.get("storage")).longValue()CODE
CRITICAL…k/scheduler/cluster/k8s/StatefulSetPodsAllocator.scala178 val statefulSet = new io.fabric8.kubernetes.api.model.apps.StatefulSetBuilder()CODE
CRITICAL…rverExpectations/stage_with_summaries_expectation.json5 "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<CODE
CRITICAL…ectations/stage_with_accumulable_json_expectation.json9 "details" : "org.apache.spark.rdd.RDD.foreach(RDD.scala:765)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$CODE
CRITICAL…ions/stage_list_with_accumulable_json_expectation.json9 "details" : "org.apache.spark.rdd.RDD.foreach(RDD.scala:765)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$CODE
CRITICAL…oryServerExpectations/stage_list_json_expectation.json5 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$lCODE
CRITICAL…oryServerExpectations/stage_list_json_expectation.json87 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$lCODE
CRITICAL…oryServerExpectations/stage_list_json_expectation.json93 "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost tasCODE
CRITICAL…oryServerExpectations/stage_list_json_expectation.json170 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…oryServerExpectations/stage_list_json_expectation.json252 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$liCODE
CRITICAL…ctations/stage_list_with_peak_metrics_expectation.json5 "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<CODE
CRITICAL…erExpectations/one_stage_attempt_json_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…attempt_json_details_with_failed_task_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…toryServerExpectations/one_stage_json_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…Expectations/complete_stage_list_json_expectation.json5 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$lCODE
CRITICAL…Expectations/complete_stage_list_json_expectation.json87 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…Expectations/complete_stage_list_json_expectation.json169 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$liCODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json5 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json71 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json137 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json203 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json269 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json335 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json401 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json467 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json533 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…xpectations/stage_task_list_w__status_expectation.json599 "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.CODE
CRITICAL…pectations/excludeOnFailure_for_stage_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<consolCODE
CRITICAL…pectations/excludeOnFailure_for_stage_expectation.json176 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…pectations/excludeOnFailure_for_stage_expectation.json441 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…ations/stage_with_speculation_summary_expectation.json5 "details" : "org.apache.spark.rdd.RDD.collect(RDD.scala:1029)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<cCODE
CRITICAL…rExpectations/stage_with_peak_metrics_expectation.json5 "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<CODE
CRITICAL…ectations/one_stage_json_with_details_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$linCODE
CRITICAL…tions/one_stage_json_with_partitionId_expectation.json5 "details" : "org.apache.spark.sql.Dataset.count(Dataset.scala:3130)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<inCODE
CRITICAL…erExpectations/failed_stage_list_json_expectation.json5 "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$lCODE
CRITICAL…erExpectations/failed_stage_list_json_expectation.json11 "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost tasCODE
CRITICAL…tions/excludeOnFailure_node_for_stage_expectation.json5 "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<consolCODE
CRITICAL…tions/excludeOnFailure_node_for_stage_expectation.json371 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…tions/excludeOnFailure_node_for_stage_expectation.json834 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…tions/excludeOnFailure_node_for_stage_expectation.json901 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…tions/excludeOnFailure_node_for_stage_expectation.json968 "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anCODE
CRITICAL…/src/test/scala/org/apache/spark/ui/UIUtilsSuite.scala220 val e1 = "Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 iCODE
CRITICAL…he/spark/shuffle/sort/ShuffleExternalSorterSuite.scala109 // at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:384)COMMENT
CRITICAL…/src/test/scala/org/apache/spark/util/UtilsSuite.scala721 val rootLogger = org.apache.logging.log4j.LogManager.getRootLogger()CODE
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1764 // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1769 // ----> at org.apache.spark.util.UtilsSuite.callGetTryFromNested(UtilsSuite.scala:1626) <---- STITCHED.STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1777 // at org.apache.spark.util.UtilsSuite.callDoTryNested(UtilsSuite.scala:1630)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1781 // at org.apache.spark.util.UtilsSuite.callDoTryNestedNested(UtilsSuite.scala:1654)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1814 // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1818 // at org.apache.spark.util.UtilsSuite.callDoTry(UtilsSuite.scala:1534)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1823 // ----> at org.apache.spark.util.UtilsSuite.callGetTryFromNestedNested(UtilsSuite.scala:1650) <---- STITCHED.STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1831 // at org.apache.spark.util.UtilsSuite.callDoTryNestedNested(UtilsSuite.scala:1654)STRING
CRITICAL⚡…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1859 // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529)STRING
430 more matches not shown…
Hyper-Verbose Identifiers3633 hits · 3320 pts
SeverityFileLineSnippetContext
LOW…la/org/apache/spark/deploy/yarn/YarnClusterSuite.scala67 private def getOrCreatePyConnectDepChecker(CODE
LOW…apache/spark/network/shuffle/ShuffleTestAccessor.scala136 def getOrCreateAppShufflePartitionInfo(CODE
LOW⚡…scala/org/apache/spark/deploy/yarn/YarnAllocator.scala277 private def getOrUpdateAllocatedHostToContainersMapForRPId(CODE
LOW⚡…scala/org/apache/spark/deploy/yarn/YarnAllocator.scala283 private def getOrUpdateRunningExecutorForRPId(rpId: Int): mutable.Set[String] = synchronized {CODE
LOW⚡…scala/org/apache/spark/deploy/yarn/YarnAllocator.scala287 private def getOrUpdateNumExecutorsStartingForRPId(rpId: Int): AtomicInteger = synchronized {CODE
LOW⚡…scala/org/apache/spark/deploy/yarn/YarnAllocator.scala291 private def getOrUpdateTargetNumExecutorsForRPId(rpId: Int): Int = synchronized {CODE
LOW…a/org/apache/spark/storage/DiskBlockManagerSuite.scala144 private def getAndSetUmask(posix: POSIX, mask: String): String = {CODE
LOW…/resources/org/apache/spark/ui/static/executorspage.js332function reselectCheckboxesBasedOnTaskTableState() {CODE
LOW…resources/org/apache/spark/ui/static/streaming-page.js62function getMaxMarginLeftForTimeline() {CODE
LOW…resources/org/apache/spark/ui/static/streaming-page.js69function getOnClickTimelineFunction() {CODE
LOW…/resources/org/apache/spark/ui/static/timeline-view.js171 function getStageIdAndAttemptForStageEntry(baseElem) {CODE
LOW…/resources/org/apache/spark/ui/static/timeline-view.js239function drawTaskAssignmentTimeline(groupArray, eventObjArray, minLaunchTime, maxFinishTime, offset) {CODE
LOW…main/resources/org/apache/spark/ui/static/stagepage.js120function getColumnNameForTaskMetricSummary(columnKey) {CODE
LOW…main/resources/org/apache/spark/ui/static/stagepage.js175function displayRowsForSummaryMetricsTable(row, type, columnIndex) {CODE
LOW…main/resources/org/apache/spark/ui/static/stagepage.js218function createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTable) {CODE
LOW…main/resources/org/apache/spark/ui/static/stagepage.js277function createRowMetadataForColumn(colKey, data, checkboxId) {CODE
LOW…main/resources/org/apache/spark/ui/static/stagepage.js287function reselectCheckboxesBasedOnTaskTableState() {CODE
LOW…esources/org/apache/spark/ui/static/environmentpage.js47function createRESTEndPointForEnvironmentPage(appId) {CODE
LOW…/resources/org/apache/spark/ui/static/spark-dag-viz.js232function getMaxChildWidthAndPaddingTop(g, v, svg) {CODE
LOW…src/main/resources/org/apache/spark/ui/static/table.js52function expandAllThreadStackTrace(toggleButton) {CODE
LOW…src/main/resources/org/apache/spark/ui/static/table.js66function collapseAllThreadStackTrace(toggleButton) {CODE
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js1081 */function injectEdgeLabelProxies(g){_.forEach(g.edges(),function(e){var edge=g.edge(e);if(edge.width&&edge.height){varCOMMENT
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js1325 */function findSmallestWidthAlignment(g,xss){return _.minBy(_.values(xss),function(xs){var max=Number.NEGATIVE_INFINITYCOMMENT
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js360function cartesianNormalizeInPlace(d){var l=sqrt(d[0]*d[0]+d[1]*d[1]+d[2]*d[2]);d[0]/=l,d[1]/=l,d[2]/=l}var lambda0$1,phCODE
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js439}}}function clipAntimeridianIntersect(lambda0,phi0,lambda1,phi1){var cosPhi0,cosPhi1,sinLambda0Lambda1=sin(lambda0-lambdCODE
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js891percentRe=/^%/,requoteRe=/[\\^$*+?|[\]().{}]/g;function pad(value,fill,width){var sign=value<0?"-":"",string=(sign?-valuCODE
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js1298scanPos=0,prevLayerLength=prevLayer.length,lastNode=_.last(layer);_.forEach(layer,function(v,i){var w=findOtherInnerSegmCODE
LOW…src/main/resources/org/apache/spark/ui/static/utils.js204function createRESTEndPointForExecutorsPage(appId) {CODE
LOW…src/main/resources/org/apache/spark/ui/static/utils.js227function createRESTEndPointForMiscellaneousProcess(appId) {CODE
LOWcore/src/main/scala/org/apache/spark/SparkContext.scala3420 def getOrCreate(config: SparkConf): SparkContext = {STRING
LOWcore/src/main/scala/org/apache/spark/SparkContext.scala3444 def getOrCreate(): SparkContext = {STRING
LOWcore/src/main/scala/org/apache/spark/util/Utils.scala750 private[spark] def getOrCreateLocalRootDirs(conf: ReadOnlySparkConf): Array[String] = {CODE
LOWcore/src/main/scala/org/apache/spark/util/Utils.scala785 private def getOrCreateLocalRootDirsImpl(conf: ReadOnlySparkConf): Array[String] = {CODE
LOW…cala/org/apache/spark/util/UninterruptibleThread.scala61 def getAndSetUninterruptible(value: Boolean): Boolean = synchronized {CODE
LOW…c/main/scala/org/apache/spark/util/AccumulatorV2.scala493 private def getOrCreate = {CODE
LOW…a/org/apache/spark/deploy/master/ApplicationInfo.scala89 private[deploy] def getOrUpdateExecutorsForRPId(rpId: Int): mutable.Set[Int] = {CODE
LOW…in/scala/org/apache/spark/scheduler/DAGScheduler.scala661 private def getOrCreateShuffleMapStage(CODE
LOW…in/scala/org/apache/spark/scheduler/DAGScheduler.scala999 private def getOrCreateParentStages(shuffleDeps: HashSet[ShuffleDependency[_, _, _]],CODE
LOW…/scala/org/apache/spark/status/AppStatusListener.scala1172 private def getOrCreateExecutor(executorId: String, addTime: Long): LiveExecutor = {CODE
LOW…/scala/org/apache/spark/status/AppStatusListener.scala1179 private def getOrCreateOtherProcess(processId: String,CODE
LOW…/scala/org/apache/spark/status/AppStatusListener.scala1239 private def getOrCreateStage(info: StageInfo): LiveStage = {CODE
LOWcore/src/main/scala/org/apache/spark/rdd/RDD.scala369 private[spark] def computeOrReadCheckpoint(split: Partition, context: TaskContext): Iterator[T] =CODE
LOWcore/src/main/scala/org/apache/spark/rdd/RDD.scala430 private[spark] def getOrCompute(partition: Partition, context: TaskContext): Iterator[T] = {CODE
LOW…main/scala/org/apache/spark/storage/BlockManager.scala1585 def getOrElseUpdateRDDBlock[T](CODE
LOW…main/scala/org/apache/spark/storage/BlockManager.scala1610 private def getOrElseUpdate[T](CODE
LOW…g/apache/spark/api/python/PythonWorkerLogCapture.scala97 private def getOrCreateLogWriter(workerId: String): (RollingLogWriter, AtomicLong) = {CODE
LOW…org/apache/spark/resource/ResourceProfileManager.scala141 def getOrAddEquivalentProfile(rp: ResourceProfile): ResourceProfile = {CODE
LOW…in/scala/org/apache/spark/resource/ResourceUtils.scala323 def getOrDiscoverAllResources(CODE
LOW…in/scala/org/apache/spark/resource/ResourceUtils.scala356 def getOrDiscoverAllResourcesForResourceProfile(CODE
LOW…/scala/org/apache/spark/resource/ResourceProfile.scala385 private[spark] def getOrCreateDefaultProfile(conf: SparkConf): ResourceProfile = {CODE
LOWpython/run-tests.py234def run_individual_python_test(target_dir, test_name, pyspark_python, keep_test_output):CODE
LOWpython/run-tests.py396def get_default_python_executables():CODE
LOWpython/pyspark/worker.py564 def _check_partition_boundaries(self, arguments: list) -> bool:CODE
LOWpython/pyspark/worker.py581 def _remove_partition_by_exprs(self, arg: Any) -> Any:CODE
LOWpython/pyspark/worker.py687 def _process_arrow_batch_by_partitions(CODE
LOWpython/pyspark/worker.py829 def _detect_partition_boundaries(self, batch) -> list:CODE
LOWpython/pyspark/worker.py864 def _remove_partition_by_exprs(self, arg: Any) -> Any:CODE
LOWpython/pyspark/worker.py1472 def verify_and_convert_result(result):CODE
LOWpython/pyspark/worker.py1568def _elementwise_udf_input_type(data_type: DataType, depth: int) -> DataType:CODE
LOWpython/pyspark/worker.py1581def _elementwise_flatten_inputs(CODE
3573 more matches not shown…
Over-Commented Block3171 hits · 2974 pts
SeverityFileLineSnippetContext
LOW.asf.yaml1# Licensed to the Apache Software Foundation (ASF) under one or moreCOMMENT
LOW.pre-commit-config.yaml1#COMMENT
LOWpyproject.toml1#COMMENT
LOW…rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala101 yarnConf.setInt("yarn.scheduler.capacity.root.default.maximum-capacity", 100)COMMENT
LOW…rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala141 }COMMENT
LOW…scala/org/apache/spark/deploy/yarn/YarnAllocator.scala321 ResourceProfile.getResourcesForClusterManager(rp.id, rp.executorResources,COMMENT
LOW…c/main/scala/org/apache/spark/deploy/yarn/Client.scala801 // Update the configuration with all the distributed files, minus the conf archive. TheCOMMENT
LOW…/kubernetes/docker/src/main/dockerfiles/spark/decom.sh1#!/usr/bin/env bashCOMMENT
LOW…rnetes/docker/src/main/dockerfiles/spark/entrypoint.sh1#!/usr/bin/env bashCOMMENT
LOW…s/core/src/test/resources/driver-podgroup-template.yml1#COMMENT
LOW…er/cluster/k8s/KubernetesClusterSchedulerBackend.scala321 log"(out of ${MDC(TOTAL, executorIds.size)}) that are still running after graceful " +COMMENT
LOW…/kubernetes/integration-tests/tests/pyfiles_connect.py1#COMMENT
LOW…/kubernetes/integration-tests/tests/decommissioning.py1#COMMENT
LOW…managers/kubernetes/integration-tests/tests/pyfiles.py1#COMMENT
LOW…ernetes/integration-tests/tests/worker_memory_check.py1#COMMENT
LOW…ernetes/integration-tests/tests/py_container_checks.py1#COMMENT
LOW…tes/integration-tests/tests/decommissioning_cleanup.py1#COMMENT
LOW…tes/integration-tests/tests/python_executable_check.py1#COMMENT
LOW…nagers/kubernetes/integration-tests/tests/autoscale.py1#COMMENT
LOW…ntegration-tests/scripts/setup-integration-test-env.sh1#!/usr/bin/env bashCOMMENT
LOW…agers/kubernetes/integration-tests/dev/spark-rbac.yaml1#COMMENT
LOW…tes/integration-tests/dev/dev-run-integration-tests.sh1#!/usr/bin/env bashCOMMENT
LOW…tegration-tests/src/test/resources/driver-template.yml1#COMMENT
LOW…gration-tests/src/test/resources/executor-template.yml1#COMMENT
LOW…-tests/src/test/resources/driver-schedule-template.yml1#COMMENT
LOW…st/resources/volcano/high-priority-driver-template.yml1#COMMENT
LOW…rces/volcano/low-priority-driver-podgroup-template.yml1#COMMENT
LOW…sources/volcano/driver-podgroup-template-memory-3g.yml1#COMMENT
LOW…/resources/volcano/queue0-driver-podgroup-template.yml1#COMMENT
LOW…n-tests/src/test/resources/volcano/priorityClasses.yml1#COMMENT
LOW…/resources/volcano/queue1-driver-podgroup-template.yml1#COMMENT
LOW…/resources/volcano/medium-priority-driver-template.yml1#COMMENT
LOW…t/resources/volcano/queue-driver-podgroup-template.yml1#COMMENT
LOW…s/volcano/medium-priority-driver-podgroup-template.yml1#COMMENT
LOW…est/resources/volcano/low-priority-driver-template.yml1#COMMENT
LOW…ces/volcano/high-priority-driver-podgroup-template.yml1#COMMENT
LOW…/org/apache/spark/launcher/AbstractCommandBuilder.java201 if (isBeeLine && "1".equals(getenv("SPARK_CONNECT_BEELINE")) &&COMMENT
LOW…/org/apache/spark/launcher/AbstractCommandBuilder.java321 return scala;COMMENT
LOW…/test/scala/org/apache/spark/ContextCleanerSuite.scala221 postGCTester.assertCleanup()COMMENT
LOW…he/spark/shuffle/sort/ShuffleExternalSorterSuite.scala101 // same memory page. When a task reads memory written by another task, many types of failuresCOMMENT
LOW…t/scala/org/apache/spark/util/SizeEstimatorSuite.scala301 // objectSize=8, fields=12 => shellSize=20, aligned to 24COMMENT
LOW…t/scala/org/apache/spark/util/SizeEstimatorSuite.scala361 // DummyString has: pointer(arr,8) + Int(hashCode,4) + Int(hash32,4) = 16 bytes of fieldsCOMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1701 // Example:COMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1721 // Second access (isFirstAccess = false) - suppressed exception should be added.COMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1761 //COMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1801 Utils.doTryWithCallerStacktrace {COMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1821 // at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1377)COMMENT
LOW…/src/test/scala/org/apache/spark/util/UtilsSuite.scala1861 // at scala.util.Try$.apply(Try.scala:217)COMMENT
LOW…ache/spark/deploy/history/FsHistoryProviderSuite.scala1041 // Test for the following race condition:COMMENT
LOW…ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala2261 makeMapStatus("hostC", reduceRdd.partitions.length)))COMMENT
LOW…ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala7621 // Fan-out/diamond: ONE PipelinedShuffleDependency (one producer, one shuffle id) is read by TWOCOMMENT
LOW…ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala7981 // Exactly 4 TaskEnd events (2 producer + 2 consumer). A double-post from buffer+replay wouldCOMMENT
LOW…ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala8541 // needs a live canCommit RPC, which the mock backend does not drive.)COMMENT
LOW…la/org/apache/spark/scheduler/HealthTrackerSuite.scala361 // This ensures that we don't trigger spurious excluding for long tasksets, when the tasksetCOMMENT
LOW…rg/apache/spark/scheduler/TaskSchedulerImplSuite.scala2601 // Secondly, try to offer the available resources to the lower priority task set, where eachCOMMENT
LOW…a/org/apache/spark/scheduler/TaskSetManagerSuite.scala3141 // it looks gone whenever MapOutputTracker.getMapOutputLocation returns None. A pipelinedCOMMENT
LOW…spark/scheduler/local/LocalSchedulerBackendSuite.scala121 val endpointAfterStart = backend.localEndpointForTestingCOMMENT
LOW…st/scala/org/apache/spark/executor/ExecutorSuite.scala101 }COMMENT
LOW…sources/org/apache/spark/ui/static/graphlib-dot.min.js141// Label for the graph itselfCOMMENT
LOW…n/resources/org/apache/spark/ui/static/dagre-d3.min.js201h=s?Math.atan2(k,bl)*rad2deg-120:NaN;return new Cubehelix(h<0?h+360:h,s,l,o.opacity)}function cubehelix(h,s,l,opacity){rCOMMENT
3111 more matches not shown…
Cross-File Repetition192 hits · 960 pts
SeverityFileLineSnippetContext
HIGH…c/main/scala/org/apache/spark/deploy/SparkSubmit.scala0welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/STRING
HIGH…/src/main/scala/org/apache/spark/repl/SparkILoop.scala0welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/STRING
HIGHpython/pyspark/shell.py0welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/STRING
HIGHpython/pyspark/core/rdd.py0return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rdSTRING
HIGHpython/pyspark/mllib/common.py0return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rdSTRING
HIGHpython/pyspark/ml/common.py0return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rdSTRING
HIGHpython/pyspark/tests/test_rdd.py0executes a job with the group ``job_group``. each job waits for 3 seconds and then exits.STRING
HIGHpython/pyspark/tests/test_pin_thread.py0executes a job with the group ``job_group``. each job waits for 3 seconds and then exits.STRING
HIGHpython/pyspark/sql/tests/test_job_cancellation.py0executes a job with the group ``job_group``. each job waits for 3 seconds and then exits.STRING
HIGHpython/pyspark/ml/tree.py0trees in this ensemble. warning: these have null parent estimators.STRING
HIGHpython/pyspark/ml/regression.py0trees in this ensemble. warning: these have null parent estimators.STRING
HIGHpython/pyspark/ml/classification.py0trees in this ensemble. warning: these have null parent estimators.STRING
HIGHpython/pyspark/ml/wrapper.py0returns the number of features the model was trained on. if unknown, returns -1STRING
HIGHpython/pyspark/ml/regression.py0returns the number of features the model was trained on. if unknown, returns -1STRING
HIGHpython/pyspark/ml/base.py0returns the number of features the model was trained on. if unknown, returns -1STRING
HIGHpython/pyspark/ml/connect/base.py0returns the number of features the model was trained on. if unknown, returns -1STRING
HIGHpython/pyspark/ml/classification.py0creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of STRING
HIGHpython/pyspark/ml/tuning.py0creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of STRING
HIGHpython/pyspark/ml/connect/tuning.py0creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of STRING
HIGHpython/pyspark/ml/classification.py0given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence.STRING
HIGHpython/pyspark/ml/pipeline.py0given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence.STRING
HIGHpython/pyspark/ml/tuning.py0given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence.STRING
HIGHpython/pyspark/ml/classification.py0transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.jSTRING
HIGHpython/pyspark/ml/pipeline.py0transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.jSTRING
HIGHpython/pyspark/ml/tuning.py0transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.jSTRING
HIGH…thon/pyspark/ml/tests/connect/test_connect_function.py0these test cases exercise the interface to the proto plan generation but do not call spark.STRING
HIGH…hon/pyspark/sql/tests/connect/test_connect_function.py0these test cases exercise the interface to the proto plan generation but do not call spark.STRING
HIGHpython/pyspark/sql/tests/connect/test_connect_plan.py0these test cases exercise the interface to the proto plan generation but do not call spark.STRING
HIGHpython/pyspark/pandas/series.py0same as `to_pandas()`, without issuing the advice log for internal usage.STRING
HIGHpython/pyspark/pandas/frame.py0same as `to_pandas()`, without issuing the advice log for internal usage.STRING
HIGHpython/pyspark/pandas/indexes/multi.py0same as `to_pandas()`, without issuing the advice log for internal usage.STRING
HIGHpython/pyspark/pandas/indexes/base.py0same as `to_pandas()`, without issuing the advice log for internal usage.STRING
HIGHpython/pyspark/pandas/series.py0return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only.STRING
HIGHpython/pyspark/pandas/frame.py0return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only.STRING
HIGHpython/pyspark/pandas/indexes/base.py0return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only.STRING
HIGHpython/pyspark/pandas/sql_formatter.py0a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object hSTRING
HIGHpython/pyspark/sql/sql_formatter.py0a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object hSTRING
HIGHpython/pyspark/sql/connect/sql_formatter.py0a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object hSTRING
HIGH…pyspark/pandas/tests/data_type_ops/test_num_reverse.py0unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark returnSTRING
HIGH…hon/pyspark/pandas/tests/data_type_ops/test_num_ops.py0unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark returnSTRING
HIGH…park/pandas/tests/data_type_ops/test_num_arithmetic.py0unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark returnSTRING
HIGHpython/pyspark/sql/catalog.py0creates an external table based on the dataset in a data source. it returns the dataframe associated with the external tSTRING
HIGHpython/pyspark/sql/context.py0creates an external table based on the dataset in a data source. it returns the dataframe associated with the external tSTRING
HIGHpython/pyspark/sql/connect/context.py0creates an external table based on the dataset in a data source. it returns the dataframe associated with the external tSTRING
HIGHpython/pyspark/sql/dataframe.py0returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"),STRING
HIGHpython/pyspark/sql/classic/dataframe.py0returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"),STRING
HIGHpython/pyspark/sql/connect/dataframe.py0returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"),STRING
HIGHpython/pyspark/sql/udtf.py0user-defined function related classes and functionsSTRING
HIGHpython/pyspark/sql/connect/udtf.py0user-defined function related classes and functionsSTRING
HIGHpython/pyspark/sql/udf.py0user-defined function related classes and functionsSTRING
HIGHpython/pyspark/sql/connect/udf.py0user-defined function related classes and functionsSTRING
HIGHpython/pyspark/sql/tests/test_tvf.py0select * from variant_explode(parse_json('["hello", "world"]'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode(parse_json('["hello", "world"]'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode(parse_json('["hello", "world"]'))STRING
HIGHpython/pyspark/sql/tests/test_tvf.py0select * from variant_explode(parse_json('{"a": true, "b": 3.14}'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode(parse_json('{"a": true, "b": 3.14}'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode(parse_json('{"a": true, "b": 3.14}'))STRING
HIGHpython/pyspark/sql/tests/test_tvf.py0select * from variant_explode_outer(parse_json('["hello", "world"]'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode_outer(parse_json('["hello", "world"]'))STRING
HIGH…che/spark/sql/DataFrameTableValuedFunctionsSuite.scala0select * from variant_explode_outer(parse_json('["hello", "world"]'))STRING
132 more matches not shown…
Cross-Language Confusion158 hits · 790 pts
SeverityFileLineSnippetContext
HIGHpython/pyspark/core/rdd.py242 return self._jrdd.toString()CODE
HIGHpython/pyspark/mllib/tree.py90 return self._java_model.toString()CODE
HIGHpython/pyspark/mllib/tree.py150 return self._java_model.toString()CODE
HIGHpython/pyspark/mllib/stat/test.py64 return self._java_model.toString()CODE
HIGH⚡python/pyspark/tests/test_util.py73 # This attempts java.lang.String(null) which throws an NPE.COMMENT
HIGH⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py142 # A numpy integer Series cannot hold a null, so pandas widens it to float64.COMMENT
HIGH…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py450 decimal, date, timestamp, duration, time, null, and nested types.STRING
HIGH…am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py526 # Every value is null, so there is no integer left to convert.COMMENT
HIGH…am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py97 a few non-temporal control rows. Each type is tested without nulls, with a null,STRING
HIGH…am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py477 numpy integers have no null, so by default PyArrow widens a null-bearing integerSTRING
HIGH…park/tests/upstream/pyarrow/test_pyarrow_array_cast.py31- Success: [0, 1, null]@int16 - element values via scalar.as_py() and Arrow type after castSTRING
HIGH…park/tests/upstream/pyarrow/test_pyarrow_array_cast.py108 as "[val1, val2, null]@arrow_type" using each scalar's as_py() value.STRING
HIGH⚡…park/tests/upstream/pyarrow/test_pyarrow_array_cast.py123 On success: "[val1, val2, null]@arrow_type"STRING
HIGH⚡…park/tests/upstream/pyarrow/test_pyarrow_array_cast.py124 e.g. "[0, 1, -1, 127, -128, null]@int16"STRING
HIGHpython/pyspark/testing/utils.py423 script = "$(test $(tput colors)) && $(test $(tput colors) -ge 8) && echo true || echo false"CODE
HIGH⚡python/pyspark/ml/tests/test_wrapper.py54 self.assertIn("LinearRegression_", model._java_obj.toString())CODE
HIGH⚡python/pyspark/ml/tests/test_wrapper.py55 self.assertIn("LinearRegressionTrainingSummary", summary._java_obj.toString())CODE
HIGH⚡python/pyspark/ml/tests/test_wrapper.py61 model._java_obj.toString()CODE
HIGH⚡python/pyspark/ml/tests/test_wrapper.py62 self.assertIn("LinearRegressionTrainingSummary", summary._java_obj.toString())CODE
HIGHpython/pyspark/ml/tests/test_wrapper.py74 model._java_obj.toString()CODE
HIGHpython/pyspark/ml/tests/test_wrapper.py76 summary._java_obj.toString()CODE
HIGHpython/pyspark/ml/tests/test_functions.py253 self.assertTrue(df1.equals(df2))STRING
HIGHpython/pyspark/ml/tests/test_functions.py259 self.assertFalse(df1.equals(df3))STRING
HIGHpython/pyspark/ml/tests/test_param.py261 "inputCol: input column name. (undefined)",CODE
HIGHpython/pyspark/errors/exceptions/captured.py284 desc=e.toString(),CODE
HIGHpython/pyspark/resource/requests.py321 that the cluster manager doesn't support the result is undefined, it may error or may justSTRING
HIGHpython/pyspark/pandas/numpy_compat.py123 # also a floating null, indistinguishable from a NaN after from_pandas, so it reads FalseCOMMENT
HIGHpython/pyspark/pandas/numpy_compat.py333 # and pairs them with a zero exponent. from_pandas delivers a NaN as a null; the sign of aCOMMENT
HIGHpython/pyspark/pandas/series.py6764 if get_option("compute.eager_check") and not self.index.equals(other.index):CODE
HIGHpython/pyspark/pandas/utils.py998 return left._jc.equals(right._jc)CODE
HIGHpython/pyspark/pandas/frame.py1715 # | 2|[{0, null}, {1, n...|COMMENT
HIGHpython/pyspark/pandas/indexing.py557 cast(ClassicColumn, col)._jc.toString() for col in data_spark_columnsCODE
HIGHpython/pyspark/pandas/groupby.py1307 Flag to ignore NA(nan/null) values during truth testing.STRING
HIGHpython/pyspark/pandas/base.py1466 # If even one StructField is null, that row should be dropped.COMMENT
HIGH⚡python/pyspark/pandas/tests/computation/test_combine.py682 # Only update where new value > 150 (and old is null)COMMENT
HIGHpython/pyspark/pandas/tests/window/test_rolling_adv.py42 # pandas 3 returns 0.0 (not null); pandas < 3 returns nan. Both are matched here.COMMENT
HIGH…thon/pyspark/pandas/tests/window/test_expanding_adv.py43 # pandas 3 returns 0.0 (not null); pandas < 3 returns nan. Both are matched here.COMMENT
HIGH…hon/pyspark/pandas/tests/diff_frames_ops/test_error.py198 psidx1.equals(psidx2)CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py201 self.assert_eq(pidx.equals(pidx), psidx.equals(psidx))CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py205 pidx.equals(pd.Index(["a", "b", "c"])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py206 psidx.equals(ps.Index(["a", "b", "c"])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py209 pidx.equals(pd.Index(["b", "b", "a"])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py210 psidx.equals(ps.Index(["b", "b", "a"])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py221 self.assert_eq(pmidx.equals(pmidx), psmidx.equals(psmidx))CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py225 pmidx.equals(pd.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py226 psmidx.equals(ps.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py229 pmidx.equals(pd.MultiIndex.from_tuples([("c", "z"), ("b", "y"), ("a", "x")])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py230 psmidx.equals(ps.MultiIndex.from_tuples([("c", "z"), ("b", "y"), ("a", "x")])),CODE
HIGH⚡python/pyspark/pandas/tests/indexes/test_basic.py234 self.assert_eq(pidx.equals(pmidx), psidx.equals(psmidx))CODE
HIGHpython/pyspark/pandas/indexes/base.py388 and self.equals(other)CODE
HIGH⚡python/pyspark/pandas/indexes/base.py412 >>> idx.equals(idx)STRING
HIGH⚡python/pyspark/pandas/indexes/base.py415 ... idx.equals(ps.Index(['a', 'b', 'c']))STRING
HIGH⚡python/pyspark/pandas/indexes/base.py418 ... idx.equals(ps.Index(['b', 'b', 'a']))STRING
HIGH⚡python/pyspark/pandas/indexes/base.py420 >>> idx.equals(midx)STRING
HIGH⚡python/pyspark/pandas/indexes/base.py425 >>> midx.equals(midx)STRING
HIGH⚡python/pyspark/pandas/indexes/base.py428 ... midx.equals(ps.MultiIndex.from_tuples([('a', 'x'), ('b', 'y'), ('c', 'z')]))STRING
HIGH⚡python/pyspark/pandas/indexes/base.py431 ... midx.equals(ps.MultiIndex.from_tuples([('c', 'z'), ('b', 'y'), ('a', 'x')]))STRING
HIGH⚡python/pyspark/pandas/indexes/base.py433 >>> midx.equals(idx)STRING
HIGH⚡python/pyspark/sql/conversion.py283 if batch.schema.equals(arrow_schema, check_metadata=False):CODE
HIGHpython/pyspark/sql/types.py2077 return stringConcat.toString()CODE
98 more matches not shown…
Decorative Section Separators186 hits · 675 pts
SeverityFileLineSnippetContext
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py662# -------------------------------------------------COMMENT
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py698# ------------------------------------COMMENT
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py704# -----------------------------------COMMENT
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py816# -------------------------------COMMENT
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py1125# ------------------------------------COMMENT
MEDIUMpython/pyspark/cloudpickle/cloudpickle.py1207# ---------------------------------COMMENT
MEDIUM…pyspark/tests/upstream/pandas/test_pandas_api_types.py157 # =====================================================================COMMENT
MEDIUM…pyspark/tests/upstream/pandas/test_pandas_api_types.py161 # =====================================================================COMMENT
MEDIUM⚡…pyspark/tests/upstream/pandas/test_pandas_api_types.py180 # =====================================================================COMMENT
MEDIUM⚡…pyspark/tests/upstream/pandas/test_pandas_api_types.py182 # =====================================================================COMMENT
MEDIUM⚡…pyspark/tests/upstream/pandas/test_pandas_api_types.py187 # =====================================================================COMMENT
MEDIUM⚡…pyspark/tests/upstream/pandas/test_pandas_api_types.py190 # =====================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py92 # =========================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py94 # =========================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py96 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py288 # =========================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py290 # =========================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py292 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py361 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py363 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py403 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py405 # -------------------------------------------------------------------------COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py563 # =========================================================================COMMENT
MEDIUM⚡…/upstream/pyarrow/test_pyarrow_array_type_inference.py565 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py50 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py52 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py67 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py69 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py209 # -------------------------------------------------------------------------COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py211 # -------------------------------------------------------------------------COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py239 # -------------------------------------------------------------------------COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py241 # -------------------------------------------------------------------------COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py441 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py443 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py531 # =========================================================================COMMENT
MEDIUM…/upstream/pyarrow/test_pyarrow_array_type_inference.py533 # =========================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py135 # =====================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py137 # =====================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py148 # =====================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py150 # =====================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py159 # =====================================================================COMMENT
MEDIUM⚡…ream/pyarrow/test_pyarrow_array_from_pandas_default.py161 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py174 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py176 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py199 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py201 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py248 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py250 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py272 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py274 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py294 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py296 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py341 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py343 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py354 # =====================================================================COMMENT
MEDIUM…ream/pyarrow/test_pyarrow_array_from_pandas_default.py356 # =====================================================================COMMENT
MEDIUM⚡…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py104 # =====================================================================COMMENT
MEDIUM⚡…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py106 # =====================================================================COMMENT
MEDIUM⚡…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py156 # =====================================================================COMMENT
MEDIUM⚡…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py158 # =====================================================================COMMENT
126 more matches not shown…
Empty Error Handlers124 hits · 620 pts
SeverityFileLineSnippetContext
HIGHpython/run-tests.py131except asyncio.CancelledError:CODE
HIGHpython/run-tests.py181except asyncio.CancelledError:CODE
HIGHpython/conf_vscode/sitecustomize.py51except ImportError:CODE
HIGHpython/conf_vscode/sitecustomize.py39except Exception:CODE
HIGHpython/pyspark/worker.py3952except Exception:CODE
HIGHpython/pyspark/threaddump.py54except Exception:CODE
HIGHpython/pyspark/util.py556except IOError:CODE
HIGHpython/pyspark/util.py815except Exception:CODE
HIGHpython/pyspark/loose_version.py23except ValueError:CODE
HIGHpython/pyspark/memory_profiler_ext.py32except Exception:CODE
HIGHpython/pyspark/install.py311except Exception:CODE
HIGHpython/pyspark/daemon.py205except OSError:CODE
HIGHpython/pyspark/daemon.py272except Exception:CODE
HIGHpython/pyspark/core/rdd.py5371except (ModuleNotFoundError, TypeError):CODE
HIGHpython/pyspark/core/rdd.py2333except TypeError:CODE
HIGHpython/pyspark/logger/worker_io.py252except Exception:CODE
HIGHpython/pyspark/cloudpickle/cloudpickle.py232except Exception:CODE
HIGHpython/pyspark/cloudpickle/cloudpickle.py437except KeyError:CODE
HIGHpython/pyspark/mllib/clustering.py1297except TypeError:CODE
HIGHpython/pyspark/mllib/evaluation.py680except TypeError:CODE
HIGHpython/pyspark/mllib/common.py118except Py4JJavaError:CODE
HIGHpython/pyspark/mllib/fpm.py226except OSError:CODE
HIGHpython/pyspark/mllib/linalg/__init__.py1633except TypeError:CODE
HIGHpython/pyspark/mllib/linalg/distributed.py1646except TypeError:CODE
HIGHpython/pyspark/mllib/tests/test_algorithms.py230except OSError:CODE
HIGHpython/pyspark/mllib/stat/_statistics.py392except TypeError:CODE
HIGHpython/pyspark/tests/test_rdd.py354except Exception:CODE
HIGHpython/pyspark/tests/test_worker.py55except Exception:CODE
HIGHpython/pyspark/tests/test_worker.py175except Exception:CODE
HIGHpython/pyspark/tests/test_context.py215except BaseException:CODE
HIGHpython/pyspark/tests/test_context.py237except Exception:CODE
HIGHpython/pyspark/testing/streamingutils.py77except BaseException:CODE
HIGHpython/pyspark/testing/streamingutils.py91except BaseException:CODE
HIGHpython/pyspark/testing/pandasutils.py31except ImportError:CODE
HIGHpython/pyspark/testing/pandasutils.py39except ImportError:CODE
HIGHpython/pyspark/testing/pandasutils.py47except ImportError:CODE
HIGHpython/pyspark/testing/utils.py1031except ImportError:CODE
HIGHpython/pyspark/testing/utils.py1040except ImportError:CODE
HIGHpython/pyspark/ml/functions.py27except ImportError:CODE
HIGHpython/pyspark/ml/wrapper.py73except Exception:CODE
HIGHpython/pyspark/ml/regression.py3338except OSError:CODE
HIGHpython/pyspark/ml/clustering.py2193except TypeError:CODE
HIGHpython/pyspark/ml/clustering.py2214except OSError:CODE
HIGHpython/pyspark/ml/classification.py4380except OSError:CODE
HIGHpython/pyspark/ml/evaluation.py1193except OSError:CODE
HIGHpython/pyspark/ml/feature.py7831except OSError:CODE
HIGHpython/pyspark/ml/recommendation.py746except OSError:CODE
HIGHpython/pyspark/ml/common.py132except Py4JJavaError:CODE
HIGHpython/pyspark/ml/stat.py571except TypeError:CODE
HIGHpython/pyspark/ml/fpm.py559except OSError:CODE
HIGHpython/pyspark/ml/linalg/__init__.py1361except TypeError:CODE
HIGHpython/pyspark/ml/tests/test_wrapper.py69except BaseException:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py134except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py176except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py400except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py423except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py285except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py314except OSError:CODE
HIGHpython/pyspark/ml/tests/test_persistence.py342except OSError:CODE
HIGHpython/pyspark/ml/tests/test_classification.py390except OSError:CODE
64 more matches not shown…
Unused Imports638 hits · 582 pts
SeverityFileLineSnippetContext
LOWpython/run-tests.py43CODE
LOWpython/packaging/connect/pyspark_connect/__init__.py22CODE
LOWpython/pyspark/worker.py47CODE
LOWpython/pyspark/util.py60CODE
LOWpython/pyspark/util.py62CODE
LOWpython/pyspark/util.py63CODE
LOWpython/pyspark/util.py65CODE
LOWpython/pyspark/util.py66CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py68CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py80CODE
LOWpython/pyspark/util.py956CODE
LOWpython/pyspark/util.py976CODE
LOWpython/pyspark/conf.py27CODE
LOWpython/pyspark/shell.py34CODE
LOWpython/pyspark/__init__.py68CODE
LOWpython/pyspark/__init__.py69CODE
LOWpython/pyspark/__init__.py69CODE
LOWpython/pyspark/__init__.py70CODE
LOWpython/pyspark/__init__.py71CODE
LOWpython/pyspark/__init__.py71CODE
LOWpython/pyspark/__init__.py72CODE
LOWpython/pyspark/__init__.py72CODE
LOWpython/pyspark/__init__.py73CODE
LOWpython/pyspark/__init__.py74CODE
LOWpython/pyspark/__init__.py74CODE
LOWpython/pyspark/__init__.py74CODE
578 more matches not shown…
Self-Referential Comments144 hits · 403 pts
SeverityFileLineSnippetContext
MEDIUMbin/docker-image-tool.sh80# Create a smaller build context for docker in dev builds to make the build faster. DockerCOMMENT
MEDIUMpython/run-tests.py266 # Create a unique temp directory under 'target/' for each run. The TMPDIR variable isCOMMENT
MEDIUMpython/run-tests.py554 # Create the target directory before starting tasks to avoid races.COMMENT
MEDIUMpython/pyspark/java_gateway.py77 # Create a temporary directory where the gateway server should write the connectionCOMMENT
MEDIUMpython/pyspark/statcounter.py18# This file is ported from spark/util/StatCounter.scalaCOMMENT
MEDIUMpython/pyspark/daemon.py116 # Create a new process group to corral our childrenCOMMENT
MEDIUMpython/pyspark/daemon.py122 # Create a listening socket on the loopback interfaceCOMMENT
MEDIUMpython/pyspark/core/rdd.py245 # This method is called when attempting to pickle an RDD, which is always an error:COMMENT
MEDIUMpython/pyspark/core/rdd.py2827 ... # Create the conf for writingSTRING
MEDIUMpython/pyspark/core/rdd.py2839 ... # Create the conf for readingSTRING
MEDIUMpython/pyspark/core/rdd.py2986 ... # Create the conf for writingSTRING
MEDIUMpython/pyspark/core/rdd.py2998 ... # Create the conf for readingSTRING
MEDIUMpython/pyspark/core/context.py298 # Create the Java SparkContext through Py4JCOMMENT
MEDIUMpython/pyspark/core/context.py303 # Create a single Accumulator in Java that we'll send all our updates through;COMMENT
MEDIUMpython/pyspark/core/context.py381 # Create a temporary directory inside spark.local.dir:COMMENT
MEDIUMpython/pyspark/core/context.py491 # This method is called when attempting to pickle SparkContext, which is always an error:STRING
MEDIUMpython/pyspark/core/context.py1492 ... # Create the conf for writingCODE
MEDIUMpython/pyspark/core/context.py1504 ... # Create the conf for readingCODE
MEDIUMpython/pyspark/core/context.py1689 ... # Create the conf for writingCODE
MEDIUMpython/pyspark/core/context.py1701 ... # Create the conf for readingCODE
MEDIUMpython/pyspark/mllib/tests/test_linalg.py542 # Create a CSC matrix with non-sorted indicesCOMMENT
MEDIUMpython/pyspark/mllib/tests/test_streaming_algorithms.py101 # Create a toy dataset by setting a tiny offset for each point.COMMENT
MEDIUMpython/pyspark/mllib/tests/test_streaming_algorithms.py396 # Create a model with initial Weights equal to coefsCOMMENT
MEDIUMpython/pyspark/tests/test_rdd.py736 # Create a DataFrame with many columns, call a Python function on each row, and take onlyCOMMENT
MEDIUMpython/pyspark/pipelines/init_cli.py52 # Create the storage directoryCOMMENT
MEDIUM⚡python/pyspark/pipelines/init_cli.py65 # Create the transformations directoryCOMMENT
MEDIUM⚡python/pyspark/pipelines/init_cli.py69 # Create the Python example fileCOMMENT
MEDIUM⚡python/pyspark/pipelines/init_cli.py74 # Create the SQL example fileCOMMENT
MEDIUMpython/pyspark/pipelines/tests/test_cli.py376 # Create a minimal pipeline specSTRING
MEDIUMpython/pyspark/pipelines/tests/test_cli.py400 # Create a minimal pipeline specSTRING
MEDIUMpython/pyspark/pipelines/tests/test_cli.py425 # Create a minimal pipeline specSTRING
MEDIUMpython/pyspark/ml/pipeline.py188 # Create a new instance of this stage.COMMENT
MEDIUMpython/pyspark/ml/pipeline.py346 # Create a new instance of this stage.COMMENT
MEDIUMpython/pyspark/ml/tuning.py982 # Create a new instance of this stage.COMMENT
MEDIUMpython/pyspark/ml/tuning.py1560 # Create a new instance of this stage.COMMENT
MEDIUMpython/pyspark/ml/tuning.py1685 # Create a new instance of this stage.COMMENT
MEDIUMpython/pyspark/ml/tests/test_feature.py361 # Create a DataFrameCOMMENT
MEDIUMpython/pyspark/pandas/tests/io/test_io.py34# This file contains test cases for 'Serialization / IO / Conversion'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_time_series.py26# This file contains test cases for 'Time series-related'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_spark.py34# This file contains test cases for 'Spark-related'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_attrs.py26# This file contains test cases for 'Attributes and underlying data'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_constructor.py33# This file contains test cases for 'Constructor'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_conversion.py25# This file contains test cases for 'Conversion'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_reindexing.py31# This file contains test cases for 'Reindexing / Selection / Label manipulation'COMMENT
MEDIUMpython/pyspark/pandas/tests/frame/test_reshaping.py27# This file contains test cases for 'Reshaping, sorting, transposing'COMMENT
MEDIUMpython/pyspark/pandas/tests/computation/test_combine.py25# This file contains test cases for 'Combining / joining / merging'COMMENT
MEDIUM…on/pyspark/pandas/tests/computation/test_apply_func.py29# This file contains test cases for 'Function application, GroupBy & Window'COMMENT
MEDIUM…/pyspark/pandas/tests/computation/test_missing_data.py27# This file contains test cases for 'Missing data handling'COMMENT
MEDIUM…on/pyspark/pandas/tests/computation/test_binary_ops.py26# This file contains test cases for 'Binary operator functions'COMMENT
MEDIUMpython/pyspark/pandas/tests/computation/test_compute.py26# This file contains test cases for 'Computations / Descriptive Stats'COMMENT
MEDIUM…thon/pyspark/pandas/tests/indexes/test_indexing_adv.py56 # Create the equivalent of pdf.loc[3] as a Koalas SeriesCOMMENT
MEDIUM…thon/pyspark/pandas/tests/indexes/test_indexing_adv.py142 # Create the equivalent of pdf.loc[3] as a Koalas SeriesCOMMENT
MEDIUMpython/pyspark/pandas/tests/indexes/test_indexing.py26# This file contains test cases for 'Indexing, Iteration'COMMENT
MEDIUMpython/pyspark/pandas/indexes/base.py264 # This method is used via `DataFrame.info` API internally.COMMENT
MEDIUMpython/pyspark/sql/dataframe.py566 ... # Create a table with Rate source.STRING
MEDIUMpython/pyspark/sql/dataframe.py6834 >>> # Create a simple UDTF that processes table dataSTRING
MEDIUMpython/pyspark/sql/dataframe.py6840 >>> # Create a DataFrameSTRING
MEDIUMpython/pyspark/sql/session.py645 # Create a new SparkSession in the JVMCOMMENT
MEDIUMpython/pyspark/sql/session.py1669 # Create a DataFrame from pandas DataFrame.COMMENT
MEDIUMpython/pyspark/sql/session.py1674 # Create a DataFrame from PyArrow Table.COMMENT
84 more matches not shown…
Deep Nesting416 hits · 334 pts
SeverityFileLineSnippetContext
LOWpython/run-tests.py234CODE
LOWpython/run-tests.py478CODE
LOWpython/pyspark/worker.py241CODE
LOWpython/pyspark/worker.py273CODE
LOWpython/pyspark/worker.py316CODE
LOWpython/pyspark/worker.py428CODE
LOWpython/pyspark/worker.py1746CODE
LOWpython/pyspark/worker.py3804CODE
LOWpython/pyspark/worker.py949CODE
LOWpython/pyspark/worker.py525CODE
LOWpython/pyspark/worker.py687CODE
LOWpython/pyspark/worker.py963CODE
LOWpython/pyspark/worker.py1043CODE
LOWpython/pyspark/worker.py1099CODE
LOWpython/pyspark/worker.py1900CODE
LOWpython/pyspark/worker.py2451CODE
LOWpython/pyspark/worker.py2575CODE
LOWpython/pyspark/worker.py2988CODE
LOWpython/pyspark/worker.py3236CODE
LOWpython/pyspark/worker.py3438CODE
LOWpython/pyspark/worker.py3631CODE
LOWpython/pyspark/worker.py3879CODE
LOWpython/pyspark/worker.py1238CODE
LOWpython/pyspark/worker.py3028CODE
LOWpython/pyspark/worker.py3681CODE
LOWpython/pyspark/worker.py3897CODE
LOWpython/pyspark/worker.py971CODE
LOWpython/pyspark/worker.py989CODE
LOWpython/pyspark/worker.py999CODE
LOWpython/pyspark/worker.py1021CODE
LOWpython/pyspark/worker_message.py140CODE
LOWpython/pyspark/util.py587CODE
LOWpython/pyspark/conf.py180CODE
LOWpython/pyspark/shuffle.py63CODE
LOWpython/pyspark/shuffle.py802CODE
LOWpython/pyspark/statcounter.py60CODE
LOWpython/pyspark/accumulators.py268CODE
LOWpython/pyspark/accumulators.py273CODE
LOWpython/pyspark/profiler.py189CODE
LOWpython/pyspark/daemon.py46CODE
LOWpython/pyspark/daemon.py115CODE
LOWpython/pyspark/core/rdd.py2210CODE
LOWpython/pyspark/core/rdd.py3672CODE
LOWpython/pyspark/core/rdd.py3724CODE
LOWpython/pyspark/core/context.py225CODE
LOWpython/pyspark/core/context.py1816CODE
LOWpython/pyspark/logger/worker_io.py214CODE
LOWpython/pyspark/cloudpickle/cloudpickle.py313CODE
LOWpython/pyspark/cloudpickle/cloudpickle.py338CODE
LOWpython/pyspark/cloudpickle/cloudpickle.py1069CODE
LOWpython/pyspark/cloudpickle/cloudpickle.py1441CODE
LOWpython/pyspark/mllib/classification.py235CODE
LOWpython/pyspark/mllib/common.py75CODE
LOWpython/pyspark/mllib/common.py96CODE
LOWpython/pyspark/mllib/common.py160CODE
LOWpython/pyspark/mllib/linalg/__init__.py98CODE
LOWpython/pyspark/mllib/linalg/__init__.py116CODE
LOWpython/pyspark/mllib/linalg/__init__.py417CODE
LOWpython/pyspark/mllib/linalg/__init__.py826CODE
LOWpython/pyspark/tests/test_serializers.py223CODE
356 more matches not shown…
Excessive Try-Catch Wrapping249 hits · 277 pts
SeverityFileLineSnippetContext
LOWpython/conf_vscode/sitecustomize.py39 except Exception:CODE
MEDIUMpython/pyspark/worker.py1507def evaluate(*a) -> tuple:CODE
MEDIUMpython/pyspark/worker.py3897def _reader_thread():CODE
LOWpython/pyspark/worker.py925 except Exception as e:CODE
LOWpython/pyspark/worker.py1146 except Exception as e:CODE
LOWpython/pyspark/worker.py1290 except Exception as e:CODE
LOWpython/pyspark/worker.py1305 except Exception as e:CODE
LOWpython/pyspark/worker.py1416 except Exception as e:CODE
LOWpython/pyspark/worker.py1512 except Exception as e:CODE
LOWpython/pyspark/worker.py3915 except Exception as e:CODE
LOWpython/pyspark/worker.py3952 except Exception:CODE
LOWpython/pyspark/worker.py4011 except Exception:CODE
LOW⚡python/pyspark/threaddump.py45 except Exception as e:CODE
MEDIUM⚡python/pyspark/threaddump.py46 print(f"Error getting children of process {args.pid}: {e}")CODE
LOW⚡python/pyspark/threaddump.py54 except Exception:CODE
MEDIUMpython/pyspark/threaddump.py28def main() -> int:CODE
LOWpython/pyspark/util.py815 except Exception:CODE
LOWpython/pyspark/serializers.py466 except Exception as e:CODE
LOWpython/pyspark/shell.py73 except Exception:CODE
LOWpython/pyspark/shell.py90 except Exception:CODE
LOWpython/pyspark/memory_profiler_ext.py32 except Exception:CODE
LOWpython/pyspark/memory_profiler_ext.py69 except Exception:CODE
LOWpython/pyspark/install.py168 except Exception:CODE
LOWpython/pyspark/install.py199 except Exception as e:CODE
LOWpython/pyspark/install.py278 except Exception as e:CODE
LOWpython/pyspark/install.py311 except Exception:CODE
LOWpython/pyspark/install.py343 except Exception as e:CODE
LOWpython/pyspark/instrumentation_utils.py48 except Exception as ex:CODE
LOWpython/pyspark/instrumentation_utils.py72 except Exception as ex:CODE
LOWpython/pyspark/daemon.py96 except Exception:CODE
LOWpython/pyspark/daemon.py272 except Exception:CODE
LOWpython/pyspark/core/context.py374 except Exception:CODE
LOWpython/pyspark/core/broadcast.py181 except Exception as e:CODE
LOWpython/pyspark/logger/worker_io.py252 except Exception:CODE
LOWpython/pyspark/cloudpickle/cloudpickle.py232 except Exception:CODE
LOWpython/pyspark/tests/test_rdd.py354 except Exception:CODE
LOWpython/pyspark/tests/test_rdd.py919 except Exception:CODE
LOWpython/pyspark/tests/test_taskcontext.py207 except Exception:CODE
LOWpython/pyspark/tests/test_taskcontext.py278 except Exception:CODE
MEDIUMpython/pyspark/tests/test_taskcontext.py204def f(iterator):CODE
LOWpython/pyspark/tests/test_util.py183 except Exception as e:CODE
LOWpython/pyspark/tests/test_pin_thread.py68 except Exception as e:CODE
LOWpython/pyspark/tests/test_pin_thread.py123 except Exception:CODE
LOWpython/pyspark/tests/test_worker.py55 except Exception:CODE
LOW⚡python/pyspark/tests/test_worker.py175 except Exception:CODE
MEDIUMpython/pyspark/tests/test_worker.py52def run():CODE
MEDIUMpython/pyspark/tests/test_worker.py172def count():CODE
LOWpython/pyspark/tests/test_context.py237 except Exception:CODE
LOWpython/pyspark/tests/test_context.py333 except Exception as e:CODE
MEDIUMpython/pyspark/tests/test_context.py330def run_job():CODE
LOWpython/pyspark/tests/test_install_spark.py56 except Exception:CODE
LOW…tests/upstream/numpy/test_numpy_ufunc_type_coercion.py187 except Exception as e:CODE
LOW…pyspark/tests/upstream/pandas/test_pandas_api_types.py112 except Exception as e:CODE
LOW…ark/tests/upstream/pandas/test_pandas_series_astype.py200 except Exception as e:CODE
MEDIUM…ark/tests/upstream/pandas/test_pandas_series_astype.py194def _astype_cell(self, series, dtype, **kwargs):CODE
LOW…ream/pyarrow/test_pyarrow_array_from_pandas_default.py124 except Exception as e:CODE
LOW⚡…stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py96 except Exception as e:CODE
LOW⚡…tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py82 except Exception as e:CODE
LOW…park/tests/upstream/pyarrow/test_pyarrow_table_cast.py84 except Exception as e:CODE
LOW⚡…am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py727 except Exception as e:CODE
189 more matches not shown…
AI Slop Vocabulary60 hits · 156 pts
SeverityFileLineSnippetContext
MEDIUM…/org/apache/spark/launcher/AbstractCommandBuilder.java244 // Place slf4j-api-* jar first to be robustCOMMENT
MEDIUM…c/test/scala/org/apache/spark/ui/UISeleniumSuite.scala468 // Essentially, we want to check that none of the stage rows showCOMMENT
MEDIUM…c/test/scala/org/apache/spark/ui/UISeleniumSuite.scala524 // Essentially, we want to check that none of the stage rows showCOMMENT
MEDIUM…ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala2767 // For a robust test assertion, limit number of job tasks to 1; that is,COMMENT
MEDIUM…apache/spark/scheduler/SchedulerIntegrationSuite.scala431 // it really can only be "best-effort" in any case, and the scheduler should be robust to that.STRING
MEDIUM…rg/apache/spark/scheduler/TaskSchedulerImplSuite.scala548 // Even though we launched a local task above, we still utilize non-local exec2.COMMENT
MEDIUM…he/spark/scheduler/HealthTrackerIntegrationSuite.scala80 // robust to one bad node.COMMENT
MEDIUM…main/scala/org/apache/spark/storage/BlockManager.scala1400 // BlockTransferService, which will leverage it to spill the block; if not, then passed-inCOMMENT
MEDIUMpython/packaging/classic/setup.py183 # TODO(SPARK-32837) leverage pip's custom optionsCOMMENT
LOW…spark/messages/socket/spark_socket_message_receiver.py54 # For socket communication, we just pass along the underlying socketCOMMENT
LOWpython/pyspark/tests/test_install_spark.py63 # we just use a hard-coded version.COMMENT
MEDIUM…am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py324 # is robust where raw address equality is not.COMMENT
LOWpython/pyspark/ml/tests/test_functions.py208 # just return the batch size as the "prediction"STRING
MEDIUMpython/pyspark/errors/utils.py373 # Excluding Python magic methods that do not utilize JVM functions.COMMENT
LOWpython/pyspark/pandas/resample.py360 # here just use Pandas' resample on a 1-length series to get it.COMMENT
LOWpython/pyspark/pandas/generic.py3102 # If Series has only a single value, just return it as a scalar.STRING
MEDIUMpython/pyspark/pandas/series.py6098 # If `where` has duplicate items, leverage the pandas directlyCOMMENT
LOWpython/pyspark/pandas/utils.py796 # '+' is meaningless for writing methods, but pandas just pass it as 'w'.COMMENT
LOWpython/pyspark/pandas/utils.py800 # '+' is meaningless for writing methods, but pandas just pass it as 'a'.COMMENT
LOWpython/pyspark/pandas/frame.py10262 # In this case, we can simply use `summary` to calculate the stats.COMMENT
MEDIUMpython/pyspark/pandas/tests/groupby/test_stat.py30 # TODO: All statistical functions should leverage this utilityCOMMENT
MEDIUMpython/pyspark/sql/session.py622 # used in conjunction with Spark Connect mode.COMMENT
MEDIUM⚡python/pyspark/sql/tests/test_functions.py3161 """Test tuple_sketch_agg + operations + estimate comprehensive test - double"""STRING
MEDIUMpython/pyspark/sql/tests/test_functions.py3216 """Test tuple_sketch_agg + operations + estimate comprehensive test - integer"""STRING
MEDIUM⚡python/pyspark/sql/tests/df_golden/test_df_golden.py83 # ``SPARK_LOCAL_REMOTE`` (the test harness) ``stop()`` terminates theCOMMENT
MEDIUM…ing/test_pandas_transform_with_state_state_variable.py351 # TODO SPARK-50908 holistic fix for TTL suiteCOMMENT
MEDIUMpython/pyspark/sql/connect/client/core.py560 # Rewrite the URL to use http as the scheme so that we can leverageCOMMENT
LOWR/pkg/R/sparkR.R664#' To remove/unset property simply set `value` to NULL e.g. setLocalProperty("key", NULL)COMMENT
MEDIUMR/pkg/R/column.R296#' Can be used in conjunction with \code{when} to specify a default value for expressions.COMMENT
MEDIUM…apache/spark/streaming/ReceivedBlockTrackerSuite.scala320 // deletion more robust rather than a parallelized operation where we fire and forgetCOMMENT
MEDIUM…cala/org/apache/spark/streaming/ui/StreamingPage.scala163 // We leverage timeFormat as the value would be same as timeFormat. This means it isCOMMENT
MEDIUM…rg/apache/spark/network/crypto/CtrTransportCipher.java229 // to utilize two helper ByteArrayWritableChannel for streaming. One is used to receive raw dataCOMMENT
MEDIUM…network/shuffle/streaming/StreamingShuffleMessage.java68 // Essentially, other message types from reader to writer won't have a valid sequence number.COMMENT
MEDIUM…scala/org/apache/spark/examples/mllib/LDAExample.scala139 // add (1.0 / actualCorpusSize) to MiniBatchFraction be more robust on tiny datasets.COMMENT
MEDIUM…a/org/apache/spark/sql/StatisticsCollectionSuite.scala978 // We can't leverage LogicalRDD.fromDataset here, since it triggers physical planning andCOMMENT
MEDIUM…c/test/scala/org/apache/spark/sql/DataFrameSuite.scala1649 // We can't leverage LogicalRDD.fromDataset here, since it triggers physical planning andCOMMENT
MEDIUM…apache/spark/sql/streaming/FileStreamSourceSuite.scala2342 // file stream source will not leverage unread files - next batch will also triggerCOMMENT
MEDIUM…org/apache/spark/sql/execution/UnionCodegenSuite.scala616 // Explicit cap so the assertion is robust to future default changes.STRING
MEDIUM…l/execution/datasources/PushVariantIntoScanSuite.scala1119 // Project/Filter nodes wrap it. This keeps scan-content assertions robust against optimizerCOMMENT
MEDIUM…ion/datasources/v2/state/StateDataSourceTestBase.scala108 // check with more data - leverage full partitionsCOMMENT
MEDIUM…k/sql/execution/exchange/EnsureRequirementsSuite.scala1026 // (spark.sql.optimizer.excludedRules), and EnsureRequirements must also stay robustCOMMENT
MEDIUM…park/sql/catalyst/analysis/ResolveSessionCatalog.scala251 // resolution was skipped) so the rewrite stays robust across analyzer ordering changes.COMMENT
MEDIUM…in/scala/org/apache/spark/sql/jdbc/OracleDialect.scala169 // Not sure if there is a more robust way to identify the field as a float (or otherCOMMENT
MEDIUM…icpruning/RowLevelOperationRuntimeGroupFiltering.scala84 // in order to leverage a regular batch scan in the group filter queryCOMMENT
MEDIUM…on/python/streaming/ApplyInPandasWithStateWriter.scala107 // from the entire data part of Arrow RecordBatch. We leverage the state metadata to alsoCOMMENT
MEDIUM…erators/stateful/StatefulStreamlineAggregateExec.scala426 // The writes which did not go into state store yet. We also leverage this dirty writes to theCOMMENT
MEDIUM…ors/stateful/join/StreamingSymmetricHashJoinExec.scala1098 // to let users leverage both sides of event time column for output of join, so the watermarkCOMMENT
MEDIUM…/execution/streaming/runtime/FileStreamSourceLog.scala130 // be started. We leverage the fact to skip calculation if possible.COMMENT
MEDIUM…sql/execution/streaming/runtime/ProgressReporter.scala572 // by itself, so leverage it.COMMENT
MEDIUM…ark/sql/catalyst/expressions/CodeGenerationSuite.scala775 | // to make the test more robust, in case the compiler can eliminate the else branch.STRING
MEDIUM…e/spark/sql/catalyst/analysis/RelationResolution.scala450 // To utilize this code path to execute V1 commands, e.g. INSERT,COMMENT
MEDIUM…ql/catalyst/expressions/SubExprEvaluationRuntime.scala100 // We leverage `IdentityHashMap` so we compare expression keys by reference here.COMMENT
MEDIUM…k/sql/catalyst/expressions/codegen/CodeFormatter.scala119 // examines the number of parenthesis and braces in that line. This isn't the most robustCOMMENT
MEDIUM…/pipelines/graph/AutoCdcScd2ColumnEvolutionSuite.scala78 // is column-schema narrowing rather than a tracked-set change (robust once SPARK-58391 lands).COMMENT
MEDIUM…/pipelines/graph/AutoCdcScd2ColumnEvolutionSuite.scala170 // tracked set (robust once SPARK-58391 lands).COMMENT
MEDIUM…/spark/sql/hive/execution/HiveCompatibilitySuite.scala287 // The isolated classloader seemed to make some of our test reset mechanisms less robust.COMMENT
MEDIUM…n/scala/org/apache/spark/sql/hive/HiveInspectors.scala1182 // TODO: hard-coding a list here is not very robust. A better idea is to have some kind of queryCOMMENT
MEDIUM…/main/java/org/apache/spark/sql/streaming/Trigger.java97 * @deprecated This is deprecated as of Spark 3.4.0. Use {@link #AvailableNow()} to leverageCOMMENT
MEDIUM…k/sql/connect/pipelines/PipelineEventSenderSuite.scala284 // total is logged at shutdown. The assertions match on substrings so they stay robust toCOMMENT
MEDIUM…e/spark/sql/hive/thriftserver/SharedThriftServer.scala134 // It's much more robust than set a random port generated by ourselves aheadCOMMENT
Structural Annotation Overuse75 hits · 150 pts
SeverityFileLineSnippetContext
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1059 // Step 1: Write an in-progress log containing only ApplicationStart (no job).COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1066 // Step 2: Load the app UI; this builds the disk store from the in-progress snapshot.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1072 // Step 3: Simulate ApplicationCache LRU eviction BEFORE the app completes.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1082 // Step 4: Complete the app. Write a new log file (without .inprogress suffix) thatCOMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1092 // Step 5: checkForLogs() detects the completed log.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1097 // Step 6: Load the UI again.COMMENT
LOW⚡python/pyspark/sql/conversion.py288 # Step 1: pick source columns from batch to align with target schemaCOMMENT
LOWpython/pyspark/sql/conversion.py317 # Step 2: check types / cast, collect all mismatchesCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py109 # Step 1: Write initial data and run streaming queryCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py116 # Step 2: Repartition to more partitionsCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py121 # Step 3: Add more data and restart queryCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py129 # Step 4: Repartition to fewer partitionsCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py134 # Step 5: Add more data and restart queryCOMMENT
LOW⚡R/pkg/inst/worker/worker.R247 # Step 1: hash the data to an environmentCOMMENT
LOWR/pkg/inst/worker/worker.R264 # Step 2: write out all of the environment as key-value pairs.COMMENT
LOWcommon/config/README.md22### Step 1: Choose the Right FileCOMMENT
LOWcommon/config/README.md29### Step 2: Add the Config EntryCOMMENT
LOWcommon/config/README.md54### Step 3: Register the Config File (if new)COMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1814 // Step 1: Run the stateful query to create the full checkpoint structureCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1817 // Step 2: Delete the state directoryCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1823 // Step 3: Attempt to read state - expected to fail since state is deletedCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1829 // Step 4: Verify the state directory was NOT recreated by the readerCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1866 // Step 1: Write data with correct schema and commitCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1877 // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType)COMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1906 // Step 1: Write data with correct schema and commitCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1918 // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType)COMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala302 // Step 2: Scheduler (via triggerNow) submits first op asCOMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala307 // Step 3: Release snapshot. Post-work queues remaining op (Cleanup)COMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala317 // Step 4: Scheduler (via triggerNow) picks up Cleanup asCOMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala326 // Step 5: Release cleanup. FromUnloadedProvidersQueue calls closeProvider.COMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala235 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala277 // Step 1: Create state by running a composite key streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala311 // Step 1: Create state by running stream-stream joinCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala323 // Step 2: Test all 4 state stores created by stream-stream joinCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala350 // Step 1: Create state by running flatMapGroupsWithStateCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala820 // Step 1: Create state by running dropDuplicatesWithinWatermarkCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala845 // Step 1: Create state by running dropDuplicates with columnCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala870 // Step 1: Create state by running session window aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala899 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala972 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala128 // Step 1: Run initial query to create stateCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala131 // Step 2: Read state data before repartitionCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala150 // Step 3: Run repartitionCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala159 // Step 4: Verify offset and commit logsCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala164 // Step 5: Validate state for each store and column family after repartitionCOMMENT
LOW…ng/state/OfflineStateRepartitionIntegrationSuite.scala192 // Step 6: Resume query with new input and verifyCOMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala776 // Step 1: subquery1 (cp) and subquery2 (np) merge:COMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala783 // Step 2: subquery3 (np) merges with merged(1,2) (cp). The cp Filter is tagged, so only aCOMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala832 // Step 1: subquery1 (cp) and subquery2 (np) merge as usual:COMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala838 // Step 2: subquery3 (np, condition a > 1) merges with merged(1,2) (cp). The cp Filter isCOMMENT
LOW…scala/org/apache/spark/sql/avro/AvroOutputWriter.scala38// NOTE: This class is instantiated and used on executor side only, no need to be serializable.COMMENT
LOW⚡…la/org/apache/spark/sql/execution/SparkSqlParser.scala146 // Step 1: Apply variable substitution to expand any variable references.COMMENT
LOW⚡…la/org/apache/spark/sql/execution/SparkSqlParser.scala149 // Step 2: Apply parameter substitution if a parameter context is provided.COMMENT
LOW…la/org/apache/spark/sql/execution/SparkSqlParser.scala174 // Step 3: Set up the origin with SQL text and position mapper to enableCOMMENT
LOW…/parquet/ParquetOutputWriterWithVariantShredding.scala39// NOTE: This class is instantiated and used on executor side only, no need to be serializable.COMMENT
LOW…xecution/datasources/parquet/ParquetOutputWriter.scala27// NOTE: This class is instantiated and used on executor side only, no need to be serializable.COMMENT
LOW⚡…/execution/aggregate/TungstenAggregationIterator.scala326 // Step 5: Get the sorted iterator from the externalSorter.COMMENT
LOW⚡…/execution/aggregate/TungstenAggregationIterator.scala329 // Step 6: Pre-load the first key-value pair from the sorted iterator to makeCOMMENT
LOW…/execution/aggregate/TungstenAggregationIterator.scala342 // Step 7: set sortBased to true.COMMENT
LOW⚡…t/analysis/SequentialStreamingUnionAnalysisSuite.scala227 // Step 1: Flatten the nested unionsCOMMENT
15 more matches not shown…
Verbosity Indicators66 hits · 127 pts
SeverityFileLineSnippetContext
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1059 // Step 1: Write an in-progress log containing only ApplicationStart (no job).COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1066 // Step 2: Load the app UI; this builds the disk store from the in-progress snapshot.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1072 // Step 3: Simulate ApplicationCache LRU eviction BEFORE the app completes.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1082 // Step 4: Complete the app. Write a new log file (without .inprogress suffix) thatCOMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1092 // Step 5: checkForLogs() detects the completed log.COMMENT
LOW⚡…ache/spark/deploy/history/FsHistoryProviderSuite.scala1097 // Step 6: Load the UI again.COMMENT
LOW⚡…/util/collection/unsafe/sort/UnsafeExternalSorter.java474 // Step 1:COMMENT
LOW⚡…/util/collection/unsafe/sort/UnsafeExternalSorter.java477 // Step 2:COMMENT
LOW⚡…/util/collection/unsafe/sort/UnsafeExternalSorter.java480 // Step 3:COMMENT
LOW…/scala/org/apache/spark/storage/BlockInfoManager.scala457 // reader counts. We need to check if the readLocksByTask per tasks are present, if theyCOMMENT
LOW⚡python/pyspark/sql/conversion.py288 # Step 1: pick source columns from batch to align with target schemaCOMMENT
LOWpython/pyspark/sql/conversion.py317 # Step 2: check types / cast, collect all mismatchesCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py109 # Step 1: Write initial data and run streaming queryCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py116 # Step 2: Repartition to more partitionsCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py121 # Step 3: Add more data and restart queryCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py129 # Step 4: Repartition to fewer partitionsCOMMENT
LOW⚡…/streaming/test_streaming_offline_state_repartition.py134 # Step 5: Add more data and restart queryCOMMENT
LOW…rk/sql/streaming/transform_with_state_driver_worker.py70 # and the following code block should be only run once for each query runCOMMENT
LOWR/pkg/inst/worker/daemon.R98 # Forking succeeded and we need to check if they finished their jobs every second.COMMENT
LOW⚡R/pkg/inst/worker/worker.R247 # Step 1: hash the data to an environmentCOMMENT
LOWR/pkg/inst/worker/worker.R264 # Step 2: write out all of the environment as key-value pairs.COMMENT
LOW…ming/FlatMapGroupsWithStateWithInitialStateSuite.scala57 // We need to check if not explicitly calling update will still save the init state or notCOMMENT
LOW…ming/FlatMapGroupsWithStateWithInitialStateSuite.scala124 // We need to check if not explicitly calling update will still save the state or notCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1814 // Step 1: Run the stateful query to create the full checkpoint structureCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1817 // Step 2: Delete the state directoryCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1823 // Step 3: Attempt to read state - expected to fail since state is deletedCOMMENT
LOW⚡…on/datasources/v2/state/StateDataSourceReadSuite.scala1829 // Step 4: Verify the state directory was NOT recreated by the readerCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1866 // Step 1: Write data with correct schema and commitCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1877 // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType)COMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1906 // Step 1: Write data with correct schema and commitCOMMENT
LOW…execution/streaming/state/RocksDBStateStoreSuite.scala1918 // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType)COMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala302 // Step 2: Scheduler (via triggerNow) submits first op asCOMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala307 // Step 3: Release snapshot. Post-work queues remaining op (Cleanup)COMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala317 // Step 4: Scheduler (via triggerNow) picks up Cleanup asCOMMENT
LOW⚡…eaming/state/StateStoreDecoupledMaintenanceSuite.scala326 // Step 5: Release cleanup. FromUnloadedProvidersQueue calls closeProvider.COMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala235 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala277 // Step 1: Create state by running a composite key streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala311 // Step 1: Create state by running stream-stream joinCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala323 // Step 2: Test all 4 state stores created by stream-stream joinCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala350 // Step 1: Create state by running flatMapGroupsWithStateCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala820 // Step 1: Create state by running dropDuplicatesWithinWatermarkCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala845 // Step 1: Create state by running dropDuplicates with columnCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala870 // Step 1: Create state by running session window aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala899 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW…state/StatePartitionAllColumnFamiliesWriterSuite.scala972 // Step 1: Create state by running a streaming aggregationCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala128 // Step 1: Run initial query to create stateCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala131 // Step 2: Read state data before repartitionCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala150 // Step 3: Run repartitionCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala159 // Step 4: Verify offset and commit logsCOMMENT
LOW⚡…ng/state/OfflineStateRepartitionIntegrationSuite.scala164 // Step 5: Validate state for each store and column family after repartitionCOMMENT
LOW…ng/state/OfflineStateRepartitionIntegrationSuite.scala192 // Step 6: Resume query with new input and verifyCOMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala776 // Step 1: subquery1 (cp) and subquery2 (np) merge:COMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala783 // Step 2: subquery3 (np) merges with merged(1,2) (cp). The cp Filter is tagged, so only aCOMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala832 // Step 1: subquery1 (cp) and subquery2 (np) merge as usual:COMMENT
LOW⚡…ark/sql/execution/planmerging/MergeSubplansSuite.scala838 // Step 2: subquery3 (np, condition a > 1) merges with merged(1,2) (cp). The cp Filter isCOMMENT
LOW…g/apache/spark/sql/classic/StreamingQueryManager.scala429 // The following code block checks if a stream with the same name or id is running. Then itCOMMENT
LOW⚡…la/org/apache/spark/sql/execution/SparkSqlParser.scala146 // Step 1: Apply variable substitution to expand any variable references.COMMENT
LOW⚡…la/org/apache/spark/sql/execution/SparkSqlParser.scala149 // Step 2: Apply parameter substitution if a parameter context is provided.COMMENT
LOW…la/org/apache/spark/sql/execution/SparkSqlParser.scala174 // Step 3: Set up the origin with SQL text and position mapper to enableCOMMENT
LOW…ql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala133 // Also, we need to check if join is done on 2 tables from 2 different databases within sameCOMMENT
6 more matches not shown…
AI Structural Patterns128 hits · 120 pts
SeverityFileLineSnippetContext
LOWpython/pyspark/core/context.py171CODE
LOWpython/pyspark/mllib/regression.py291CODE
LOWpython/pyspark/mllib/regression.py475CODE
LOWpython/pyspark/mllib/regression.py655CODE
LOWpython/pyspark/mllib/classification.py330CODE
LOWpython/pyspark/mllib/classification.py422CODE
LOWpython/pyspark/mllib/classification.py645CODE
LOWpython/pyspark/pipelines/api.py113CODE
LOWpython/pyspark/pipelines/api.py126CODE
LOWpython/pyspark/pipelines/api.py217CODE
LOWpython/pyspark/pipelines/api.py230CODE
LOWpython/pyspark/pipelines/api.py531CODE
LOWpython/pyspark/streaming/kinesis.py69CODE
LOWpython/pyspark/streaming/kinesis.py88CODE
LOWpython/pyspark/testing/utils.py745CODE
LOWpython/pyspark/ml/regression.py301CODE
LOWpython/pyspark/ml/regression.py335CODE
LOWpython/pyspark/ml/regression.py1110CODE
LOWpython/pyspark/ml/regression.py1146CODE
LOWpython/pyspark/ml/regression.py1410CODE
LOWpython/pyspark/ml/regression.py1450CODE
LOWpython/pyspark/ml/regression.py1760CODE
LOWpython/pyspark/ml/regression.py1802CODE
LOWpython/pyspark/ml/regression.py2161CODE
LOWpython/pyspark/ml/regression.py2201CODE
LOWpython/pyspark/ml/regression.py2566CODE
LOWpython/pyspark/ml/regression.py2602CODE
LOWpython/pyspark/ml/regression.py3136CODE
LOWpython/pyspark/ml/regression.py3167CODE
LOWpython/pyspark/ml/clustering.py409CODE
LOWpython/pyspark/ml/clustering.py439CODE
LOWpython/pyspark/ml/clustering.py784CODE
LOWpython/pyspark/ml/clustering.py816CODE
LOWpython/pyspark/ml/clustering.py1142CODE
LOWpython/pyspark/ml/clustering.py1168CODE
LOWpython/pyspark/ml/clustering.py1700CODE
LOWpython/pyspark/ml/clustering.py1738CODE
LOWpython/pyspark/ml/classification.py725CODE
LOWpython/pyspark/ml/classification.py757CODE
LOWpython/pyspark/ml/classification.py1243CODE
LOWpython/pyspark/ml/classification.py1269CODE
LOWpython/pyspark/ml/classification.py1295CODE
LOWpython/pyspark/ml/classification.py1340CODE
LOWpython/pyspark/ml/classification.py1366CODE
LOWpython/pyspark/ml/classification.py1393CODE
LOWpython/pyspark/ml/classification.py1783CODE
LOWpython/pyspark/ml/classification.py1821CODE
LOWpython/pyspark/ml/classification.py2085CODE
LOWpython/pyspark/ml/classification.py2128CODE
LOWpython/pyspark/ml/classification.py2563CODE
LOWpython/pyspark/ml/classification.py2607CODE
LOWpython/pyspark/ml/classification.py2967CODE
LOWpython/pyspark/ml/classification.py2994CODE
LOWpython/pyspark/ml/classification.py3217CODE
LOWpython/pyspark/ml/classification.py3249CODE
LOWpython/pyspark/ml/classification.py4112CODE
LOWpython/pyspark/ml/classification.py4149CODE
LOWpython/pyspark/ml/evaluation.py579CODE
LOWpython/pyspark/ml/evaluation.py690CODE
LOWpython/pyspark/ml/feature.py3625CODE
68 more matches not shown…
Modern Structural Boilerplate121 hits · 120 pts
SeverityFileLineSnippetContext
LOWpython/packaging/connect/pyspark_connect/__init__.py24__all__ = [CODE
LOWpython/pyspark/taskcontext.py166 def _setTaskContext(cls: Type["TaskContext"], taskContext: Optional["TaskContext"]) -> None:CODE
LOWpython/pyspark/conf.py18__all__ = ["SparkConf"]CODE
LOWpython/pyspark/serializers.py79__all__ = [CODE
LOWpython/pyspark/__init__.py133__all__ = [CODE
LOWpython/pyspark/memory_profiler_ext.py36__all__ = [CODE
LOWpython/pyspark/storagelevel.py18__all__ = ["StorageLevel"]CODE
LOWpython/pyspark/accumulators.py36__all__ = ["Accumulator", "AccumulatorParam"]CODE
LOWpython/pyspark/resultiterable.py24__all__ = ["ResultIterable"]CODE
LOWpython/pyspark/messages/__init__.py22__all__ = [CODE
LOWpython/pyspark/core/files.py20__all__ = ["SparkFiles"]CODE
LOWpython/pyspark/core/rdd.py119__all__ = ["RDD"]CODE
LOWpython/pyspark/core/context.py81__all__ = ["SparkContext"]CODE
LOWpython/pyspark/core/broadcast.py48__all__ = ["Broadcast"]CODE
LOWpython/pyspark/core/status.py18__all__ = ["SparkJobInfo", "SparkStageInfo", "SparkExecutorInfo", "StatusTracker"]CODE
LOWpython/pyspark/logger/__init__.py24__all__ = ["PySparkLogger", "SPARK_LOG_SCHEMA"]CODE
LOWpython/pyspark/cloudpickle/__init__.py8__all__ = [ # noqaCODE
LOWpython/pyspark/mllib/tree.py33__all__ = [CODE
LOWpython/pyspark/mllib/regression.py50__all__ = [CODE
LOWpython/pyspark/mllib/clustering.py42__all__ = [CODE
LOWpython/pyspark/mllib/classification.py41__all__ = [CODE
LOWpython/pyspark/mllib/evaluation.py28__all__ = [CODE
LOWpython/pyspark/mllib/__init__.py33__all__ = [CODE
LOWpython/pyspark/mllib/feature.py40__all__ = [CODE
LOWpython/pyspark/mllib/random.py33__all__ = [CODE
LOWpython/pyspark/mllib/recommendation.py28__all__ = ["MatrixFactorizationModel", "ALS", "Rating"]CODE
LOWpython/pyspark/mllib/fpm.py26__all__ = ["FPGrowth", "FPGrowthModel", "PrefixSpan", "PrefixSpanModel"]CODE
LOWpython/pyspark/mllib/linalg/__init__.py73__all__ = [CODE
LOWpython/pyspark/mllib/linalg/distributed.py40__all__ = [CODE
LOWpython/pyspark/mllib/stat/__init__.py27__all__ = [CODE
LOWpython/pyspark/mllib/stat/test.py22__all__ = ["ChiSqTestResult", "KolmogorovSmirnovTestResult"]CODE
LOWpython/pyspark/mllib/stat/distribution.py18__all__ = ["MultivariateGaussian"]CODE
LOWpython/pyspark/mllib/stat/_statistics.py33__all__ = ["MultivariateStatisticalSummary", "Statistics"]CODE
LOWpython/pyspark/pipelines/__init__.py27__all__ = [CODE
LOWpython/pyspark/streaming/dstream.py50__all__ = ["DStream"]CODE
LOWpython/pyspark/streaming/kinesis.py25__all__ = ["KinesisUtils", "InitialPositionInStream", "MetricsLevel", "utf8_decoder"]CODE
LOWpython/pyspark/streaming/__init__.py22__all__ = ["StreamingContext", "DStream", "StreamingListener"]CODE
LOWpython/pyspark/streaming/context.py30__all__ = ["StreamingContext"]CODE
LOWpython/pyspark/streaming/listener.py20__all__ = ["StreamingListener"]CODE
LOWpython/pyspark/testing/__init__.py21__all__ = ["assertDataFrameEqual", "assertSchemaEqual", "main"]CODE
LOWpython/pyspark/testing/utils.py46__all__ = ["assertDataFrameEqual", "assertSchemaEqual"]CODE
LOWpython/pyspark/testing/connectutils.py106 def _set_relation_in_plan(self, plan: pb2.Plan, relation: pb2.Relation) -> None:CODE
LOWpython/pyspark/testing/connectutils.py111 def _set_command_in_plan(self, plan: pb2.Plan, command: pb2.Command) -> None:CODE
LOWpython/pyspark/ml/regression.py83__all__ = [CODE
LOWpython/pyspark/ml/clustering.py65__all__ = [CODE
LOWpython/pyspark/ml/classification.py113__all__ = [CODE
LOWpython/pyspark/ml/evaluation.py50__all__ = [CODE
LOWpython/pyspark/ml/__init__.py49__all__ = [CODE
LOWpython/pyspark/ml/feature.py77__all__ = [CODE
LOWpython/pyspark/ml/recommendation.py41__all__ = ["ALS", "ALSModel"]CODE
LOWpython/pyspark/ml/tuning.py71__all__ = [CODE
LOWpython/pyspark/ml/image.py36__all__ = ["ImageSchema"]CODE
LOWpython/pyspark/ml/fpm.py35__all__ = ["FPGrowth", "FPGrowthModel", "PrefixSpan"]CODE
LOWpython/pyspark/ml/linalg/__init__.py59__all__ = [CODE
LOWpython/pyspark/ml/torch/distributor.py704 def set_torch_config(context: "BarrierTaskContext") -> None:STRING
LOWpython/pyspark/ml/torch/distributor.py730 def set_gpus(context: "BarrierTaskContext") -> None:STRING
LOWpython/pyspark/ml/torch/distributor.py739 def set_gpus(context: "BarrierTaskContext") -> None:STRING
LOWpython/pyspark/ml/param/__init__.py41__all__ = ["Param", "Params", "TypeConverters"]CODE
LOWpython/pyspark/ml/connect/__init__.py37__all__ = [CODE
LOWpython/pyspark/errors/__init__.py57__all__ = [CODE
61 more matches not shown…
Fake / Example Data105 hits · 89 pts
SeverityFileLineSnippetContext
LOW…/org/apache/spark/security/FileTokenIngestorSuite.java274 String token = createUnsignedJwt("user@example.com", "https://accounts.google.com");CODE
LOW…/org/apache/spark/security/FileTokenIngestorSuite.java280 assertEquals("user@example.com", result.get().getPrincipal());CODE
LOW…he/spark/shuffle/streaming/StreamingShuffleSuite.scala62 private val queryId = "dummy-id"CODE
LOW…spark/deploy/security/UserCredentialManagerSuite.scala40 .set(SECURITY_OIDC_IDENTITY_TOKEN_FILE, "/tmp/fake-token")CODE
LOW…es/org/apache/spark/ui/static/jquery.dataTables.min.js4!function(n){"use strict";var a;"function"==typeof define&&define.amd?define(["jquery"],function(t){return n(t,window,doCODE
LOWpython/pyspark/sql/tests/test_stat.py317 dummy_value = 1CODE
LOWpython/pyspark/sql/tests/test_stat.py320 .replace({"Alice": "Bob"}, dummy_value)CODE
LOW⚡…hon/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py1122 .withColumn("name", lit("John Doe"))CODE
LOWpython/pyspark/sql/pandas/functions.py109 >>> df = spark.createDataFrame([("John Doe",)], ("name",))STRING
LOWpython/pyspark/sql/pandas/functions.py124 >>> df = spark.createDataFrame([("John Doe",)], ("name",))STRING
LOWpython/pyspark/sql/pandas/functions.py506 >>> df = spark.createDataFrame([("John Doe",)], ("name",))STRING
LOWpython/pyspark/sql/pandas/functions.py518 >>> df = spark.createDataFrame([("John Doe",)], ("name",))STRING
LOW…apache/spark/graphx/lib/ConnectedComponentsSuite.scala119 val defaultUser = ("John Doe", "Missing")CODE
LOWdocs/graphx-programming-guide.md193val defaultUser = ("John Doe", "Missing")CODE
LOWdocs/graphx-programming-guide.md432val defaultUser = ("John Doe", "Missing")CODE
LOWexamples/src/main/python/sql/arrow.py325 df = spark.createDataFrame([(1, "John Doe", 21)], ("id", "name", "age"))CODE
LOW…/spark/security/aws/AwsStsCredentialProviderSuite.java97 private static final String TEST_PRINCIPAL = "user@example.com";CODE
LOW…s/test-data/xml-resources/mixed_children_as_string.xml4 Lorem ipsum dolor sit amet. Ut <i>voluptas</i> distinctio et impedit deserunt aut quam fugit et quaerat oditCODE
LOW…s/test-data/xml-resources/mixed_children_as_string.xml4 Lorem ipsum dolor sit amet. Ut <i>voluptas</i> distinctio et impedit deserunt aut quam fugit et quaerat oditCODE
LOW…/test/resources/test-data/xml-resources/processing.xml4 lorem ipsumCODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql150INSERT INTO products VALUES (1, 'Super Widget', 'Electronics', 155.99, 99.99, 1, 'Acme Inc', 'John D.', '123 Main St', 2CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql150INSERT INTO products VALUES (1, 'Super Widget', 'Electronics', 155.99, 99.99, 1, 'Acme Inc', 'John D.', '123 Main St', 2CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql154INSERT INTO customers VALUES (1, 'Alice Johnson', 'alice@example.com', '555-1000', '101 Maple Ave', NULL, 'Springfield',CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql155INSERT INTO customers VALUES (2, 'Bob Smith', 'bob@example.com', '555-1002', '202 Oak St', 'Apt 3', 'Oakville', 'CA', '6CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql156INSERT INTO customers VALUES (3, 'Cathy Lee', 'cathy@example.com', '555-1003', '303 Pine Ln', NULL, 'Pineville', 'TX', 'CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql166INSERT INTO employees VALUES (1, 'Dan Miller', 'dan@example.com', '555-2001', 'Manager', 'Sales', TIMESTAMP '2018-01-01'CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql167INSERT INTO employees VALUES (2, 'Eva Perez', 'eva@example.com', '555-2002', 'Salesperson', 'Sales', TIMESTAMP '2019-03-CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql168INSERT INTO employees VALUES (3, 'Frank Wong', 'frank@example.com', '555-2003', 'Warehouse', 'Operations', TIMESTAMP '20CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql170INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 MCODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql170INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 MCODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql170INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 MCODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql171INSERT INTO suppliers VALUES (2, 'Widgets Co', 'Mary K.', 'Customer Success', 'mary@widgets.com', '555-4001', NULL, '456CODE
LOW⚡…-tests/inputs/scripting/randomly_generated_scripts.sql172INSERT INTO suppliers VALUES (3, 'Toy Supply', 'Ann T.', 'Director', 'ann@toysupply.com', '555-5001', NULL, '789 Oak St'CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql212 INSERT INTO suppliers VALUES (v_temp_id, 'Temp Supplier', 'Temp Contact', 'Temp Role', 'tempCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql260 INSERT INTO customers VALUES (v_new_customer_id, 'New Customer', 'new@customer.com', '555-1111', '55CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql393 VALUES (sub_emp.employee_id + 9999, v_name_part, CONCAT(v_name_part, '@company.com')CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql405 VALUES (emp.employee_id + 10000, CONCAT('Emp_', emp.employee_id), emp.employee_name, 'EmployCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql474 INSERT INTO products VALUES ((SELECT COALESCE(MAX(product_id), 0) + 1 FROM products), 'Rare ' || v_mCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql519 '555-1212',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql552 '555-1111',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql553 '123 Main St',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql727 '555-7777',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql918 INSERT INTO employees VALUES (v_new_id, 'New Emp ' || v_new_id, 'new' || v_new_id || '@cCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql930 INSERT INTO employees VALUES (v_temp_id, 'Manager ' || v_temp_id, 'manager' || v_temp_id || '@companCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql941 INSERT INTO employees VALUES (v_low_level_emp + 10, 'Temp Emp ' || v_low_level_emp, 'temp' || v_low_level_emCODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql982 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql983 '123 Main St',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1097 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1169 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1417 '123 Main St',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1444 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1491 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1528 '555-0001',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1556 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql1763 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql2072 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql2275 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql2582 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql2615 '555-0000',CODE
LOW…-tests/inputs/scripting/randomly_generated_scripts.sql2788 '555-0000',CODE
45 more matches not shown…
Redundant / Tautological Comments60 hits · 87 pts
SeverityFileLineSnippetContext
LOWpython/run-tests.py516 # Check if the python executable has coverage installed when 'COVERAGE_PROCESS_START'COMMENT
LOWpython/pyspark/worker.py742 # Check if this is a continuation of the previous batch's partitionCOMMENT
LOWpython/pyspark/worker.py851 # Check if any partition column changed from previous rowCOMMENT
LOWpython/pyspark/worker.py3009 # Check if the entire column is null: an empty table (no columns)COMMENT
LOWpython/pyspark/worker.py3653 # Check if the entire column is null.COMMENT
LOWpython/pyspark/shell.py56 # Check if th eprogress bar needs to be disabled.COMMENT
LOW⚡python/pyspark/pipelines/cli.py74 # Check if it's a simple file path (no wildcards at all)COMMENT
LOW⚡python/pyspark/pipelines/cli.py78 # Check if it's a folder path ending with /**COMMENT
LOWpython/pyspark/pandas/frame.py12699 # Check if DataFrame has rows - if yes, raise error; if no, return empty SeriesCOMMENT
LOWpython/pyspark/pandas/frame.py12834 # Check if DataFrame has rows - if yes, raise error; if no, return empty SeriesCOMMENT
LOWpython/pyspark/pandas/data_type_ops/categorical_ops.py116 # Check if categoricals have the same dtype, same categories, and same orderedCOMMENT
LOWpython/pyspark/pandas/typedef/typehints.py649 # Check if the name is Tuple.COMMENT
LOWpython/pyspark/pandas/indexes/base.py2069 # Check if the `self` and `other` have different index types.COMMENT
LOWpython/pyspark/sql/metrics.py188 # Add yourself to the list if you have to.COMMENT
LOWpython/pyspark/sql/dataframe.py412 >>> # Check if the DataFrames are equalSTRING
LOWpython/pyspark/sql/session.py2301 # Check if the target path already existsCOMMENT
LOWpython/pyspark/sql/types.py3537 >>> # Check if numeric values are within the allowed range.STRING
LOW⚡python/pyspark/sql/tests/test_utils.py1734 # Check if the error message contains information about 2 mismatches only.COMMENT
LOWpython/pyspark/sql/tests/arrow/test_arrow_map.py394 # Set it to a small odd value to exercise batching logic for all test casesCOMMENT
LOWpython/pyspark/sql/tests/pandas/bench_pipelined_udf.py92# Output results as JSON to stdoutCOMMENT
LOW…s/pandas/streaming/test_pandas_transform_with_state.py1433 # Set it to a very small number so that every row would be a separate pandas dfCOMMENT
LOW…s/pandas/streaming/test_pandas_transform_with_state.py1460 # Set it to a very large number so that every row would be in the same pandas dfCOMMENT
LOW…s/pandas/streaming/test_pandas_transform_with_state.py1526 # Set it to a very small number so that every row would be a separate pandas dfCOMMENT
LOW⚡…/pyspark/sql/tests/pandas/streaming/test_tws_tester.py752 # Set watermark to 15000 - key1's timer should fire.COMMENT
LOW⚡…/pyspark/sql/tests/pandas/streaming/test_tws_tester.py757 # Set watermark to 16000 - key2's timer should fire.COMMENT
LOW…/pyspark/sql/tests/pandas/streaming/test_tws_tester.py791 # Set watermark to 6000.COMMENT
LOW…/pyspark/sql/tests/pandas/streaming/test_tws_tester.py822 # Set watermark to 20 seconds.COMMENT
LOW…/pyspark/sql/tests/pandas/streaming/test_tws_tester.py924 # Set watermark to 10000.COMMENT
LOWpython/pyspark/sql/streaming/readwriter.py1551 # Check if the data should be processedSTRING
LOWpython/pyspark/sql/worker/plan_data_source_read.py155 # Check if the names are the same as the schema.COMMENT
LOWpython/pyspark/sql/worker/create_data_source.py81 # Check if the provider name matches the data source's name.COMMENT
LOWpython/pyspark/sql/worker/write_into_data_source.py97 # Check if the provider name matches the data source's name.COMMENT
LOWpython/pyspark/sql/connect/session.py1219 # Check if total size exceeds the limitCOMMENT
LOWpython/pyspark/sql/connect/session.py1229 # Check if adding this chunk would exceed batch sizeCOMMENT
LOWpython/pyspark/sql/connect/client/artifact.py198 # Check if it is a file from the schemeCOMMENT
LOWpython/pyspark/sql/pandas/conversion.py866 # Check if any columns need to be fixed for Spark to infer properlyCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py68 # Check if all arguments have type hintsCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py78 # Check if the return has a type hintCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py227 # Check if all arguments have type hintsCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py237 # Check if the return has a type hintCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py420 # Check if all arguments have type hintsCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py430 # Check if the return has a type hintCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py513 # Check if all arguments have type hintsCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py523 # Check if the return has a type hintCOMMENT
LOWpython/pyspark/sql/pandas/typehints.py599 # Check if the name is Tuple first. After that, check the generic types.COMMENT
LOWsbin/spark-daemon.sh50# Check if --config is passed as an argument. It is an optional parameter.COMMENT
LOWsbin/spark-daemon.sh154 # Check if the process has died; in that case we'll tail the log so the user can seeCOMMENT
LOWsbin/decommission-worker.sh48# Check if --block-until-exit is set.COMMENT
LOWsbin/workers.sh57# Check if --config is passed as an argument. It is an optional parameter.COMMENT
LOW…l/src/test/scala/org/apache/spark/repl/ReplSuite.scala254 |# Set everything to be logged to the consoleSTRING
LOWR/pkg/tests/fulltests/test_jvm_api.R26 # Check if get returns the same elementCOMMENT
LOWR/pkg/R/sparkR.R456 # Check if version number of SparkSession matches version number of SparkR packageCOMMENT
LOWR/pkg/R/serialize.R45 # Check if all elements are of same typeCOMMENT
LOWR/pkg/R/jobj.R31# Check if jobj was created with the current SparkContextCOMMENT
LOWR/pkg/R/DataFrame.R386 # Check if the column names have . in itCOMMENT
LOWR/pkg/R/DataFrame.R2282 # Check if there is any duplicated column name in the DataFrameCOMMENT
LOWR/pkg/inst/worker/worker.R97# Set libPaths to include SparkR package as loadNamespace needs thisCOMMENT
LOWdev/create-release/generate-llms-txt.py188 # Check if docs directory existsCOMMENT
LOWdev/create-release/release-build.sh1045 # Check if any failures were recordedCOMMENT
LOWdev/create-release/release-util.sh159 # Check if the RC already exists, and if re-creating the RC, skip tag creation.COMMENT
Slop Phrases14 hits · 22 pts
SeverityFileLineSnippetContext
LOW…/main/java/org/apache/spark/SparkFirehoseListener.java27 * This is a concrete Java class in order to ensure that we don't forget to update it when addingCOMMENT
MEDIUM…/org/apache/spark/storage/BlockReplicationPolicy.scala101 * Method to prioritize a bunch of candidate peers of a block. This is a basic implementation,COMMENT
LOWpython/packaging/classic/setup.py170# Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/classic/setup.py170# Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/classic/setup.py363 # Don't forget to update python/docs/source/getting_started/install.rstCOMMENT
LOWpython/packaging/connect/setup.py84 # Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/connect/setup.py84 # Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/connect/setup.py115 # Don't forget to update python/docs/source/getting_started/install.rstCOMMENT
LOWpython/packaging/client/setup.py132 # Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/client/setup.py132 # Also don't forget to update python/docs/source/getting_started/install.rst,COMMENT
LOWpython/packaging/client/setup.py213 # Don't forget to update python/docs/source/getting_started/install.rstCOMMENT
LOWpython/pyspark/pandas/config.py114# NOTE: if you are fixing or adding an option here, make sure you execute `show_options()` andCOMMENT
LOWdev/create-release/release-build.sh832 # NOTE: Don't forget to update the valid combinations of distributions atCOMMENT
LOW…/main/scala/org/apache/spark/sql/connect/Dataset.scala146 // Make sure we don't forget to set plan id.COMMENT
AI Response Leakage2 hits · 15 pts
SeverityFileLineSnippetContext
HIGHpython/pyspark/sql/tests/connect/test_connect_basic.py1803 # In this example, the max chunk size is set to a small value, so each ArrowCOMMENT
HIGH…cala/org/apache/spark/sql/connector/catalog/txns.scala192 // This is where the table pinning logic should occur. In this implementation, a tables is loadedCOMMENT
TODO Padding5 hits · 6 pts
SeverityFileLineSnippetContext
LOWpython/pyspark/pandas/groupby.py2194 # TODO: implement 'dropna' parameterCOMMENT
LOW…/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala66 // TODO: implement weighted bootstrappingCOMMENT
LOW…rk/sql/execution/datasources/v2/FileDataSourceV2.scala101 // TODO: implement a light-weight partition inference which only looks at the path of one leafCOMMENT
LOW…yst/expressions/aggregate/datasketchesAggregates.scala166 // TODO: implement support for decimal/date/timestamp/interval typesSTRING
LOW…che/spark/sql/hive/execution/InsertIntoHiveTable.scala121 // TODO: implement hive compatibility as rules.COMMENT
Modern AI Meta-Vocabulary2 hits · 6 pts
SeverityFileLineSnippetContext
MEDIUM…la/org/apache/spark/sql/FileBasedDataSourceSuite.scala1472 // Embed the nanos leaf inside a struct, an array, and a map value. The guardrailsCOMMENT
MEDIUM…pache/spark/sql/pipelines/graph/SqlPipelineSuite.scala1411 // MERGE API, which requires a row-level-operation-capable v2 catalog, so these tests scaffoldCOMMENT
Docstring Block Structure1 hit · 5 pts
SeverityFileLineSnippetContext
HIGHpython/pyspark/testing/sqlutils.py115 Read the classpath file for a project and return it as a comma-separated string. The classpath file is typicalSTRING
Synthetic Comment Markers1 hit · 2 pts
SeverityFileLineSnippetContext
HIGHpython/pyspark/ml/dl_util.py103 the empty string, nothing will be written after the auto-generated code.STRING
Example Usage Blocks1 hit · 2 pts
SeverityFileLineSnippetContext
LOWpython/pyspark/sql/tests/df_golden/regenerate.sh21# Usage:COMMENT
Overly Generic Function Names1 hit · 0 pts
SeverityFileLineSnippetContext
LOWpython/pyspark/sql/tests/test_udf_transpile_unit.py1698 def helper(v):CODE