Apache Spark - A unified analytics engine for large-scale data processing
This report presents the forensic synthetic code analysis of apache/spark, a Scala project with 43,898 GitHub stars. SynthScan v2.0 examined 3,202,834 lines of code across 12658 source files, recording 9508 pattern matches distributed across 24 syntactic categories. The overall adjusted score of 5.0 places this repository in the Low AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 9508 distinct pattern matches across 24 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| CRITICAL | …rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 137 | case e: Throwable if org.apache.commons.lang3.exception.ExceptionUtils.indexOfThrowable( | STRING |
| CRITICAL | …cheduler/cluster/YarnClientSchedulerBackendSuite.scala | 55 | Some(org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(0L, 1))) | CODE |
| CRITICAL⚡ | …org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala | 49 | assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("first")) | CODE |
| CRITICAL⚡ | …org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala | 56 | assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("second")) | CODE |
| CRITICAL⚡ | …org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala | 63 | assert(sparkPod.pod.getSpec.getContainers.asScala.toList.map(_.getName) == List("second")) | CODE |
| CRITICAL | …k/scheduler/cluster/k8s/DeploymentAllocatorSuite.scala | 150 | assert(deployment.getSpec.getTemplate.getSpec.getContainers.asScala.exists( | CODE |
| CRITICAL | …heduler/cluster/k8s/ExecutorPVCResizePluginSuite.scala | 187 | captor.getValue.getSpec.getResources.getRequests.get("storage")).longValue() | CODE |
| CRITICAL | …k/scheduler/cluster/k8s/StatefulSetPodsAllocator.scala | 177 | val statefulSet = new io.fabric8.kubernetes.api.model.apps.StatefulSetBuilder() | CODE |
| CRITICAL | …rverExpectations/stage_with_summaries_expectation.json | 5 | "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.< | CODE |
| CRITICAL | …ectations/stage_with_accumulable_json_expectation.json | 9 | "details" : "org.apache.spark.rdd.RDD.foreach(RDD.scala:765)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$ | CODE |
| CRITICAL | …ions/stage_list_with_accumulable_json_expectation.json | 9 | "details" : "org.apache.spark.rdd.RDD.foreach(RDD.scala:765)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$ | CODE |
| CRITICAL | …oryServerExpectations/stage_list_json_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$l | CODE |
| CRITICAL | …oryServerExpectations/stage_list_json_expectation.json | 87 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$l | CODE |
| CRITICAL | …oryServerExpectations/stage_list_json_expectation.json | 93 | "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost tas | CODE |
| CRITICAL | …oryServerExpectations/stage_list_json_expectation.json | 170 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …oryServerExpectations/stage_list_json_expectation.json | 252 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$li | CODE |
| CRITICAL | …ctations/stage_list_with_peak_metrics_expectation.json | 5 | "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.< | CODE |
| CRITICAL | …erExpectations/one_stage_attempt_json_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …attempt_json_details_with_failed_task_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …toryServerExpectations/one_stage_json_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …Expectations/complete_stage_list_json_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$l | CODE |
| CRITICAL | …Expectations/complete_stage_list_json_expectation.json | 87 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …Expectations/complete_stage_list_json_expectation.json | 169 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)\n$li | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 5 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 71 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 137 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 203 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 269 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 335 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 401 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 467 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 533 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …xpectations/stage_task_list_w__status_expectation.json | 599 | "errorMessage" : "java.lang.RuntimeException: bad exec\n\tat $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1. | CODE |
| CRITICAL | …pectations/excludeOnFailure_for_stage_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<consol | CODE |
| CRITICAL | …pectations/excludeOnFailure_for_stage_expectation.json | 176 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …pectations/excludeOnFailure_for_stage_expectation.json | 441 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …ations/stage_with_speculation_summary_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.collect(RDD.scala:1029)\n$line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<c | CODE |
| CRITICAL | …rExpectations/stage_with_peak_metrics_expectation.json | 5 | "details" : "org.apache.spark.sql.Dataset.foreach(Dataset.scala:2862)\n$line19.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.< | CODE |
| CRITICAL | …ectations/one_stage_json_with_details_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:271)\n$line10.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)\n$lin | CODE |
| CRITICAL | …tions/one_stage_json_with_partitionId_expectation.json | 5 | "details" : "org.apache.spark.sql.Dataset.count(Dataset.scala:3130)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<in | CODE |
| CRITICAL | …erExpectations/failed_stage_list_json_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line11.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)\n$l | CODE |
| CRITICAL | …erExpectations/failed_stage_list_json_expectation.json | 11 | "failureReason" : "Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost tas | CODE |
| CRITICAL | …tions/excludeOnFailure_node_for_stage_expectation.json | 5 | "details" : "org.apache.spark.rdd.RDD.map(RDD.scala:370)\n$line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<consol | CODE |
| CRITICAL | …tions/excludeOnFailure_node_for_stage_expectation.json | 371 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …tions/excludeOnFailure_node_for_stage_expectation.json | 834 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …tions/excludeOnFailure_node_for_stage_expectation.json | 901 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …tions/excludeOnFailure_node_for_stage_expectation.json | 968 | "errorMessage" : "java.lang.RuntimeException: Bad executor\n\tat $line15.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$an | CODE |
| CRITICAL | …/src/test/scala/org/apache/spark/ui/UIUtilsSuite.scala | 220 | val e1 = "Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 i | CODE |
| CRITICAL | …he/spark/shuffle/sort/ShuffleExternalSorterSuite.scala | 109 | // at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:384) | COMMENT |
| CRITICAL | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 721 | val rootLogger = org.apache.logging.log4j.LogManager.getRootLogger() | CODE |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1763 | // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1768 | // ----> at org.apache.spark.util.UtilsSuite.callGetTryFromNested(UtilsSuite.scala:1626) <---- STITCHED. | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1776 | // at org.apache.spark.util.UtilsSuite.callDoTryNested(UtilsSuite.scala:1630) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1780 | // at org.apache.spark.util.UtilsSuite.callDoTryNestedNested(UtilsSuite.scala:1654) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1813 | // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1817 | // at org.apache.spark.util.UtilsSuite.callDoTry(UtilsSuite.scala:1534) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1822 | // ----> at org.apache.spark.util.UtilsSuite.callGetTryFromNestedNested(UtilsSuite.scala:1650) <---- STITCHED. | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1830 | // at org.apache.spark.util.UtilsSuite.callDoTryNestedNested(UtilsSuite.scala:1654) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1858 | // at org.apache.spark.util.UtilsSuite.throwException(UtilsSuite.scala:1529) | STRING |
| CRITICAL⚡ | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1862 | // at org.apache.spark.util.UtilsSuite.callDoTry(UtilsSuite.scala:1534) | STRING |
| 416 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …la/org/apache/spark/deploy/yarn/YarnClusterSuite.scala | 67 | private def getOrCreatePyConnectDepChecker( | CODE |
| LOW | …apache/spark/network/shuffle/ShuffleTestAccessor.scala | 136 | def getOrCreateAppShufflePartitionInfo( | CODE |
| LOW⚡ | …scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 277 | private def getOrUpdateAllocatedHostToContainersMapForRPId( | CODE |
| LOW⚡ | …scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 283 | private def getOrUpdateRunningExecutorForRPId(rpId: Int): mutable.Set[String] = synchronized { | CODE |
| LOW⚡ | …scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 287 | private def getOrUpdateNumExecutorsStartingForRPId(rpId: Int): AtomicInteger = synchronized { | CODE |
| LOW⚡ | …scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 291 | private def getOrUpdateTargetNumExecutorsForRPId(rpId: Int): Int = synchronized { | CODE |
| LOW | …a/org/apache/spark/storage/DiskBlockManagerSuite.scala | 144 | private def getAndSetUmask(posix: POSIX, mask: String): String = { | CODE |
| LOW | …/resources/org/apache/spark/ui/static/executorspage.js | 331 | function reselectCheckboxesBasedOnTaskTableState() { | CODE |
| LOW | …resources/org/apache/spark/ui/static/streaming-page.js | 62 | function getMaxMarginLeftForTimeline() { | CODE |
| LOW | …resources/org/apache/spark/ui/static/streaming-page.js | 69 | function getOnClickTimelineFunction() { | CODE |
| LOW | …/resources/org/apache/spark/ui/static/timeline-view.js | 171 | function getStageIdAndAttemptForStageEntry(baseElem) { | CODE |
| LOW | …/resources/org/apache/spark/ui/static/timeline-view.js | 239 | function drawTaskAssignmentTimeline(groupArray, eventObjArray, minLaunchTime, maxFinishTime, offset) { | CODE |
| LOW | …main/resources/org/apache/spark/ui/static/stagepage.js | 120 | function getColumnNameForTaskMetricSummary(columnKey) { | CODE |
| LOW | …main/resources/org/apache/spark/ui/static/stagepage.js | 175 | function displayRowsForSummaryMetricsTable(row, type, columnIndex) { | CODE |
| LOW | …main/resources/org/apache/spark/ui/static/stagepage.js | 218 | function createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTable) { | CODE |
| LOW | …main/resources/org/apache/spark/ui/static/stagepage.js | 277 | function createRowMetadataForColumn(colKey, data, checkboxId) { | CODE |
| LOW | …main/resources/org/apache/spark/ui/static/stagepage.js | 287 | function reselectCheckboxesBasedOnTaskTableState() { | CODE |
| LOW | …esources/org/apache/spark/ui/static/environmentpage.js | 47 | function createRESTEndPointForEnvironmentPage(appId) { | CODE |
| LOW | …/resources/org/apache/spark/ui/static/spark-dag-viz.js | 232 | function getMaxChildWidthAndPaddingTop(g, v, svg) { | CODE |
| LOW | …src/main/resources/org/apache/spark/ui/static/table.js | 52 | function expandAllThreadStackTrace(toggleButton) { | CODE |
| LOW | …src/main/resources/org/apache/spark/ui/static/table.js | 66 | function collapseAllThreadStackTrace(toggleButton) { | CODE |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 1081 | */function injectEdgeLabelProxies(g){_.forEach(g.edges(),function(e){var edge=g.edge(e);if(edge.width&&edge.height){var | COMMENT |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 1325 | */function findSmallestWidthAlignment(g,xss){return _.minBy(_.values(xss),function(xs){var max=Number.NEGATIVE_INFINITY | COMMENT |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 360 | function cartesianNormalizeInPlace(d){var l=sqrt(d[0]*d[0]+d[1]*d[1]+d[2]*d[2]);d[0]/=l,d[1]/=l,d[2]/=l}var lambda0$1,ph | CODE |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 439 | }}}function clipAntimeridianIntersect(lambda0,phi0,lambda1,phi1){var cosPhi0,cosPhi1,sinLambda0Lambda1=sin(lambda0-lambd | CODE |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 891 | percentRe=/^%/,requoteRe=/[\\^$*+?|[\]().{}]/g;function pad(value,fill,width){var sign=value<0?"-":"",string=(sign?-valu | CODE |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 1298 | scanPos=0,prevLayerLength=prevLayer.length,lastNode=_.last(layer);_.forEach(layer,function(v,i){var w=findOtherInnerSegm | CODE |
| LOW | …src/main/resources/org/apache/spark/ui/static/utils.js | 188 | function createRESTEndPointForExecutorsPage(appId) { | CODE |
| LOW | …src/main/resources/org/apache/spark/ui/static/utils.js | 211 | function createRESTEndPointForMiscellaneousProcess(appId) { | CODE |
| LOW | core/src/main/scala/org/apache/spark/SparkContext.scala | 3381 | def getOrCreate(config: SparkConf): SparkContext = { | STRING |
| LOW | core/src/main/scala/org/apache/spark/SparkContext.scala | 3405 | def getOrCreate(): SparkContext = { | STRING |
| LOW | core/src/main/scala/org/apache/spark/util/Utils.scala | 750 | private[spark] def getOrCreateLocalRootDirs(conf: ReadOnlySparkConf): Array[String] = { | CODE |
| LOW | core/src/main/scala/org/apache/spark/util/Utils.scala | 785 | private def getOrCreateLocalRootDirsImpl(conf: ReadOnlySparkConf): Array[String] = { | CODE |
| LOW | …cala/org/apache/spark/util/UninterruptibleThread.scala | 61 | def getAndSetUninterruptible(value: Boolean): Boolean = synchronized { | CODE |
| LOW | …c/main/scala/org/apache/spark/util/AccumulatorV2.scala | 486 | private def getOrCreate = { | CODE |
| LOW | …a/org/apache/spark/deploy/master/ApplicationInfo.scala | 89 | private[deploy] def getOrUpdateExecutorsForRPId(rpId: Int): mutable.Set[Int] = { | CODE |
| LOW | …in/scala/org/apache/spark/scheduler/DAGScheduler.scala | 661 | private def getOrCreateShuffleMapStage( | CODE |
| LOW | …in/scala/org/apache/spark/scheduler/DAGScheduler.scala | 972 | private def getOrCreateParentStages(shuffleDeps: HashSet[ShuffleDependency[_, _, _]], | CODE |
| LOW | …/scala/org/apache/spark/status/AppStatusListener.scala | 1169 | private def getOrCreateExecutor(executorId: String, addTime: Long): LiveExecutor = { | CODE |
| LOW | …/scala/org/apache/spark/status/AppStatusListener.scala | 1176 | private def getOrCreateOtherProcess(processId: String, | CODE |
| LOW | …/scala/org/apache/spark/status/AppStatusListener.scala | 1236 | private def getOrCreateStage(info: StageInfo): LiveStage = { | CODE |
| LOW | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 369 | private[spark] def computeOrReadCheckpoint(split: Partition, context: TaskContext): Iterator[T] = | CODE |
| LOW | core/src/main/scala/org/apache/spark/rdd/RDD.scala | 430 | private[spark] def getOrCompute(partition: Partition, context: TaskContext): Iterator[T] = { | CODE |
| LOW | …main/scala/org/apache/spark/storage/BlockManager.scala | 1585 | def getOrElseUpdateRDDBlock[T]( | CODE |
| LOW | …main/scala/org/apache/spark/storage/BlockManager.scala | 1610 | private def getOrElseUpdate[T]( | CODE |
| LOW | …g/apache/spark/api/python/PythonWorkerLogCapture.scala | 97 | private def getOrCreateLogWriter(workerId: String): (RollingLogWriter, AtomicLong) = { | CODE |
| LOW | …in/scala/org/apache/spark/resource/ResourceUtils.scala | 323 | def getOrDiscoverAllResources( | CODE |
| LOW | …in/scala/org/apache/spark/resource/ResourceUtils.scala | 356 | def getOrDiscoverAllResourcesForResourceProfile( | CODE |
| LOW | …/scala/org/apache/spark/resource/ResourceProfile.scala | 385 | private[spark] def getOrCreateDefaultProfile(conf: SparkConf): ResourceProfile = { | CODE |
| LOW | python/run-tests.py | 234 | def run_individual_python_test(target_dir, test_name, pyspark_python, keep_test_output): | CODE |
| LOW | python/run-tests.py | 396 | def get_default_python_executables(): | CODE |
| LOW⚡ | python/pyspark/worker.py | 137 | def use_legacy_pandas_udf_conversion(self) -> bool: | CODE |
| LOW⚡ | python/pyspark/worker.py | 144 | def use_legacy_pandas_udtf_conversion(self) -> bool: | CODE |
| LOW⚡ | python/pyspark/worker.py | 151 | def map_in_batch_legacy_accept_any_iterable(self) -> bool: | CODE |
| LOW | python/pyspark/worker.py | 169 | def int_to_decimal_coercion_enabled(self) -> bool: | CODE |
| LOW | python/pyspark/worker.py | 187 | def arrow_max_records_per_batch(self) -> int: | CODE |
| LOW | python/pyspark/worker.py | 191 | def arrow_max_bytes_per_batch(self) -> int: | CODE |
| LOW | python/pyspark/worker.py | 389 | def verify_iterator_exhausted(iterator: Iterator) -> None: | CODE |
| LOW | python/pyspark/worker.py | 412 | def verify_iter_result_row_count( | CODE |
| LOW | python/pyspark/worker.py | 894 | def _check_partition_boundaries(self, arguments: list) -> bool: | CODE |
| 3333 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | .asf.yaml | 1 | # Licensed to the Apache Software Foundation (ASF) under one or more | COMMENT |
| LOW | .pre-commit-config.yaml | 1 | # | COMMENT |
| LOW | pyproject.toml | 1 | # | COMMENT |
| LOW | …rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 101 | yarnConf.setInt("yarn.scheduler.capacity.root.default.maximum-capacity", 100) | COMMENT |
| LOW | …rg/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 141 | } | COMMENT |
| LOW | …scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 321 | ResourceProfile.getResourcesForClusterManager(rp.id, rp.executorResources, | COMMENT |
| LOW | …c/main/scala/org/apache/spark/deploy/yarn/Client.scala | 801 | // Update the configuration with all the distributed files, minus the conf archive. The | COMMENT |
| LOW | …/kubernetes/docker/src/main/dockerfiles/spark/decom.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | …rnetes/docker/src/main/dockerfiles/spark/entrypoint.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | …s/core/src/test/resources/driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …er/cluster/k8s/KubernetesClusterSchedulerBackend.scala | 321 | // waits on this future to complete, blocking further allocations / deallocations. | COMMENT |
| LOW | …/kubernetes/integration-tests/tests/pyfiles_connect.py | 1 | # | COMMENT |
| LOW | …/kubernetes/integration-tests/tests/decommissioning.py | 1 | # | COMMENT |
| LOW | …managers/kubernetes/integration-tests/tests/pyfiles.py | 1 | # | COMMENT |
| LOW | …ernetes/integration-tests/tests/worker_memory_check.py | 1 | # | COMMENT |
| LOW | …ernetes/integration-tests/tests/py_container_checks.py | 1 | # | COMMENT |
| LOW | …tes/integration-tests/tests/decommissioning_cleanup.py | 1 | # | COMMENT |
| LOW | …tes/integration-tests/tests/python_executable_check.py | 1 | # | COMMENT |
| LOW | …nagers/kubernetes/integration-tests/tests/autoscale.py | 1 | # | COMMENT |
| LOW | …ntegration-tests/scripts/setup-integration-test-env.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | …agers/kubernetes/integration-tests/dev/spark-rbac.yaml | 1 | # | COMMENT |
| LOW | …tes/integration-tests/dev/dev-run-integration-tests.sh | 1 | #!/usr/bin/env bash | COMMENT |
| LOW | …tegration-tests/src/test/resources/driver-template.yml | 1 | # | COMMENT |
| LOW | …gration-tests/src/test/resources/executor-template.yml | 1 | # | COMMENT |
| LOW | …-tests/src/test/resources/driver-schedule-template.yml | 1 | # | COMMENT |
| LOW | …st/resources/volcano/high-priority-driver-template.yml | 1 | # | COMMENT |
| LOW | …rces/volcano/low-priority-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …sources/volcano/driver-podgroup-template-memory-3g.yml | 1 | # | COMMENT |
| LOW | …/resources/volcano/queue0-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …n-tests/src/test/resources/volcano/priorityClasses.yml | 1 | # | COMMENT |
| LOW | …/resources/volcano/queue1-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …/resources/volcano/medium-priority-driver-template.yml | 1 | # | COMMENT |
| LOW | …t/resources/volcano/queue-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …s/volcano/medium-priority-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …est/resources/volcano/low-priority-driver-template.yml | 1 | # | COMMENT |
| LOW | …ces/volcano/high-priority-driver-podgroup-template.yml | 1 | # | COMMENT |
| LOW | …/org/apache/spark/launcher/AbstractCommandBuilder.java | 201 | if (isBeeLine && "1".equals(getenv("SPARK_CONNECT_BEELINE")) && | COMMENT |
| LOW | …/org/apache/spark/launcher/AbstractCommandBuilder.java | 321 | return scala; | COMMENT |
| LOW | …/test/scala/org/apache/spark/ContextCleanerSuite.scala | 221 | postGCTester.assertCleanup() | COMMENT |
| LOW | …he/spark/shuffle/sort/ShuffleExternalSorterSuite.scala | 101 | // same memory page. When a task reads memory written by another task, many types of failures | COMMENT |
| LOW | …t/scala/org/apache/spark/util/SizeEstimatorSuite.scala | 301 | // objectSize=8, fields=12 => shellSize=20, aligned to 24 | COMMENT |
| LOW | …t/scala/org/apache/spark/util/SizeEstimatorSuite.scala | 361 | // DummyString has: pointer(arr,8) + Int(hashCode,4) + Int(hash32,4) = 16 bytes of fields | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1701 | // java.lang.Exception: test | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1721 | // The original stack trace with callDoTry should be in the suppressed exceptions. | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1761 | // scalastyle:off line.size.limit | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1801 | val t = callDoTryNested() | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1821 | // at org.apache.spark.util.Utils$.getTryWithCallerStacktrace(Utils.scala:1438) | COMMENT |
| LOW | …/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 1861 | // at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1377) | COMMENT |
| LOW | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1041 | // Test for the following race condition: | COMMENT |
| LOW | …ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 2061 | makeMapStatus("hostC", reduceRdd.partitions.length))) | COMMENT |
| LOW | …ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 7021 | assertDataStructuresEmpty() | COMMENT |
| LOW | …la/org/apache/spark/scheduler/HealthTrackerSuite.scala | 361 | // This ensures that we don't trigger spurious excluding for long tasksets, when the taskset | COMMENT |
| LOW | …rg/apache/spark/scheduler/TaskSchedulerImplSuite.scala | 2601 | // Secondly, try to offer the available resources to the lower priority task set, where each | COMMENT |
| LOW | …/test/scala/org/apache/spark/scheduler/PoolSuite.scala | 121 | scheduleTaskAndVerifyId(0, rootPool, 0) | COMMENT |
| LOW | …a/org/apache/spark/scheduler/TaskSetManagerSuite.scala | 3141 | // it looks gone whenever MapOutputTracker.getMapOutputLocation returns None. A pipelined | COMMENT |
| LOW | …st/scala/org/apache/spark/executor/ExecutorSuite.scala | 101 | } | COMMENT |
| LOW | …sources/org/apache/spark/ui/static/graphlib-dot.min.js | 141 | // Label for the graph itself | COMMENT |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 201 | h=s?Math.atan2(k,bl)*rad2deg-120:NaN;return new Cubehelix(h<0?h+360:h,s,l,o.opacity)}function cubehelix(h,s,l,opacity){r | COMMENT |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 281 | // Limit forces for very close nodes; randomize direction if coincident. | COMMENT |
| LOW | …n/resources/org/apache/spark/ui/static/dagre-d3.min.js | 301 | function formatTrim(s){out:for(var n=s.length,i=1,i0=-1,i1;i<n;++i){switch(s[i]){case".":i0=i1=i;break;case"0":if(i0===0 | COMMENT |
| 3004 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | …c/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 0 | welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/ | STRING |
| HIGH | …/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 0 | welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/ | STRING |
| HIGH | python/pyspark/shell.py | 0 | welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version %s /_/ | STRING |
| HIGH | python/pyspark/core/rdd.py | 0 | return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rd | STRING |
| HIGH | python/pyspark/mllib/common.py | 0 | return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rd | STRING |
| HIGH | python/pyspark/ml/common.py | 0 | return an javardd of object by unpickling it will convert each python object into java object by pickle, whenever the rd | STRING |
| HIGH | python/pyspark/tests/test_rdd.py | 0 | executes a job with the group ``job_group``. each job waits for 3 seconds and then exits. | STRING |
| HIGH | python/pyspark/tests/test_pin_thread.py | 0 | executes a job with the group ``job_group``. each job waits for 3 seconds and then exits. | STRING |
| HIGH | python/pyspark/sql/tests/test_job_cancellation.py | 0 | executes a job with the group ``job_group``. each job waits for 3 seconds and then exits. | STRING |
| HIGH | python/pyspark/ml/tree.py | 0 | trees in this ensemble. warning: these have null parent estimators. | STRING |
| HIGH | python/pyspark/ml/regression.py | 0 | trees in this ensemble. warning: these have null parent estimators. | STRING |
| HIGH | python/pyspark/ml/classification.py | 0 | trees in this ensemble. warning: these have null parent estimators. | STRING |
| HIGH | python/pyspark/ml/wrapper.py | 0 | returns the number of features the model was trained on. if unknown, returns -1 | STRING |
| HIGH | python/pyspark/ml/regression.py | 0 | returns the number of features the model was trained on. if unknown, returns -1 | STRING |
| HIGH | python/pyspark/ml/base.py | 0 | returns the number of features the model was trained on. if unknown, returns -1 | STRING |
| HIGH | python/pyspark/ml/connect/base.py | 0 | returns the number of features the model was trained on. if unknown, returns -1 | STRING |
| HIGH | python/pyspark/ml/classification.py | 0 | creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of | STRING |
| HIGH | python/pyspark/ml/tuning.py | 0 | creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of | STRING |
| HIGH | python/pyspark/ml/connect/tuning.py | 0 | creates a copy of this instance with a randomly generated uid and some extra params. this copies creates a deep copy of | STRING |
| HIGH | python/pyspark/ml/classification.py | 0 | given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence. | STRING |
| HIGH | python/pyspark/ml/pipeline.py | 0 | given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence. | STRING |
| HIGH | python/pyspark/ml/tuning.py | 0 | given a java trainvalidationsplitmodel, create and return a python wrapper of it. used for ml persistence. | STRING |
| HIGH | python/pyspark/ml/classification.py | 0 | transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.j | STRING |
| HIGH | python/pyspark/ml/pipeline.py | 0 | transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.j | STRING |
| HIGH | python/pyspark/ml/tuning.py | 0 | transfer this instance to a java trainvalidationsplitmodel. used for ml persistence. returns ------- py4j.java_gateway.j | STRING |
| HIGH | …thon/pyspark/ml/tests/connect/test_connect_function.py | 0 | these test cases exercise the interface to the proto plan generation but do not call spark. | STRING |
| HIGH | …hon/pyspark/sql/tests/connect/test_connect_function.py | 0 | these test cases exercise the interface to the proto plan generation but do not call spark. | STRING |
| HIGH | python/pyspark/sql/tests/connect/test_connect_plan.py | 0 | these test cases exercise the interface to the proto plan generation but do not call spark. | STRING |
| HIGH | python/pyspark/pandas/series.py | 0 | same as `to_pandas()`, without issuing the advice log for internal usage. | STRING |
| HIGH | python/pyspark/pandas/frame.py | 0 | same as `to_pandas()`, without issuing the advice log for internal usage. | STRING |
| HIGH | python/pyspark/pandas/indexes/multi.py | 0 | same as `to_pandas()`, without issuing the advice log for internal usage. | STRING |
| HIGH | python/pyspark/pandas/indexes/base.py | 0 | same as `to_pandas()`, without issuing the advice log for internal usage. | STRING |
| HIGH | python/pyspark/pandas/series.py | 0 | return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only. | STRING |
| HIGH | python/pyspark/pandas/frame.py | 0 | return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only. | STRING |
| HIGH | python/pyspark/pandas/indexes/base.py | 0 | return a pandas index directly from _internal to avoid overhead of copy. this method is for internal use only. | STRING |
| HIGH | python/pyspark/pandas/sql_formatter.py | 0 | a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object h | STRING |
| HIGH | python/pyspark/sql/sql_formatter.py | 0 | a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object h | STRING |
| HIGH | python/pyspark/sql/connect/sql_formatter.py | 0 | a standard ``string.formatter`` in python that can understand pyspark instances with basic python objects. this object h | STRING |
| HIGH | …pyspark/pandas/tests/data_type_ops/test_num_reverse.py | 0 | unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark return | STRING |
| HIGH | …hon/pyspark/pandas/tests/data_type_ops/test_num_ops.py | 0 | unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark return | STRING |
| HIGH | …park/pandas/tests/data_type_ops/test_num_arithmetic.py | 0 | unit tests for arithmetic operations of numeric data types. a few test cases are disabled because pandas-on-spark return | STRING |
| HIGH | python/pyspark/sql/catalog.py | 0 | creates an external table based on the dataset in a data source. it returns the dataframe associated with the external t | STRING |
| HIGH | python/pyspark/sql/context.py | 0 | creates an external table based on the dataset in a data source. it returns the dataframe associated with the external t | STRING |
| HIGH | python/pyspark/sql/connect/context.py | 0 | creates an external table based on the dataset in a data source. it returns the dataframe associated with the external t | STRING |
| HIGH | python/pyspark/sql/dataframe.py | 0 | returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"), | STRING |
| HIGH | python/pyspark/sql/classic/dataframe.py | 0 | returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"), | STRING |
| HIGH | python/pyspark/sql/connect/dataframe.py | 0 | returns the names of columns in this :class:`dataframe`. examples -------- >>> df = spark.createdataframe([(2, "alice"), | STRING |
| HIGH | python/pyspark/sql/udtf.py | 0 | user-defined function related classes and functions | STRING |
| HIGH | python/pyspark/sql/connect/udtf.py | 0 | user-defined function related classes and functions | STRING |
| HIGH | python/pyspark/sql/udf.py | 0 | user-defined function related classes and functions | STRING |
| HIGH | python/pyspark/sql/connect/udf.py | 0 | user-defined function related classes and functions | STRING |
| HIGH | python/pyspark/sql/tests/test_tvf.py | 0 | select * from variant_explode(parse_json('["hello", "world"]')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode(parse_json('["hello", "world"]')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode(parse_json('["hello", "world"]')) | STRING |
| HIGH | python/pyspark/sql/tests/test_tvf.py | 0 | select * from variant_explode(parse_json('{"a": true, "b": 3.14}')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode(parse_json('{"a": true, "b": 3.14}')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode(parse_json('{"a": true, "b": 3.14}')) | STRING |
| HIGH | python/pyspark/sql/tests/test_tvf.py | 0 | select * from variant_explode_outer(parse_json('["hello", "world"]')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode_outer(parse_json('["hello", "world"]')) | STRING |
| HIGH | …che/spark/sql/DataFrameTableValuedFunctionsSuite.scala | 0 | select * from variant_explode_outer(parse_json('["hello", "world"]')) | STRING |
| 128 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | python/pyspark/core/rdd.py | 242 | return self._jrdd.toString() | CODE |
| HIGH | python/pyspark/mllib/tree.py | 90 | return self._java_model.toString() | CODE |
| HIGH | python/pyspark/mllib/tree.py | 150 | return self._java_model.toString() | CODE |
| HIGH | python/pyspark/mllib/stat/test.py | 64 | return self._java_model.toString() | CODE |
| HIGH⚡ | python/pyspark/tests/test_util.py | 73 | # This attempts java.lang.String(null) which throws an NPE. | COMMENT |
| HIGH⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 142 | # A numpy integer Series cannot hold a null, so pandas widens it to float64. | COMMENT |
| HIGH | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 450 | decimal, date, timestamp, duration, time, null, and nested types. | STRING |
| HIGH | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 526 | # Every value is null, so there is no integer left to convert. | COMMENT |
| HIGH | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 97 | a few non-temporal control rows. Each type is tested without nulls, with a null, | STRING |
| HIGH | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 477 | numpy integers have no null, so by default PyArrow widens a null-bearing integer | STRING |
| HIGH | …park/tests/upstream/pyarrow/test_pyarrow_array_cast.py | 31 | - Success: [0, 1, null]@int16 - element values via scalar.as_py() and Arrow type after cast | STRING |
| HIGH | …park/tests/upstream/pyarrow/test_pyarrow_array_cast.py | 108 | as "[val1, val2, null]@arrow_type" using each scalar's as_py() value. | STRING |
| HIGH⚡ | …park/tests/upstream/pyarrow/test_pyarrow_array_cast.py | 123 | On success: "[val1, val2, null]@arrow_type" | STRING |
| HIGH⚡ | …park/tests/upstream/pyarrow/test_pyarrow_array_cast.py | 124 | e.g. "[0, 1, -1, 127, -128, null]@int16" | STRING |
| HIGH | python/pyspark/testing/utils.py | 423 | script = "$(test $(tput colors)) && $(test $(tput colors) -ge 8) && echo true || echo false" | CODE |
| HIGH⚡ | python/pyspark/ml/tests/test_wrapper.py | 54 | self.assertIn("LinearRegression_", model._java_obj.toString()) | CODE |
| HIGH⚡ | python/pyspark/ml/tests/test_wrapper.py | 55 | self.assertIn("LinearRegressionTrainingSummary", summary._java_obj.toString()) | CODE |
| HIGH⚡ | python/pyspark/ml/tests/test_wrapper.py | 61 | model._java_obj.toString() | CODE |
| HIGH⚡ | python/pyspark/ml/tests/test_wrapper.py | 62 | self.assertIn("LinearRegressionTrainingSummary", summary._java_obj.toString()) | CODE |
| HIGH | python/pyspark/ml/tests/test_wrapper.py | 74 | model._java_obj.toString() | CODE |
| HIGH | python/pyspark/ml/tests/test_wrapper.py | 76 | summary._java_obj.toString() | CODE |
| HIGH | python/pyspark/ml/tests/test_functions.py | 253 | self.assertTrue(df1.equals(df2)) | STRING |
| HIGH | python/pyspark/ml/tests/test_functions.py | 259 | self.assertFalse(df1.equals(df3)) | STRING |
| HIGH | python/pyspark/ml/tests/test_param.py | 261 | "inputCol: input column name. (undefined)", | CODE |
| HIGH | python/pyspark/errors/exceptions/captured.py | 284 | desc=e.toString(), | CODE |
| HIGH | python/pyspark/resource/requests.py | 321 | that the cluster manager doesn't support the result is undefined, it may error or may just | STRING |
| HIGH | python/pyspark/pandas/numpy_compat.py | 107 | # also a floating null, indistinguishable from a NaN after from_pandas, so it reads False | COMMENT |
| HIGH | python/pyspark/pandas/series.py | 6764 | if get_option("compute.eager_check") and not self.index.equals(other.index): | CODE |
| HIGH | python/pyspark/pandas/utils.py | 998 | return left._jc.equals(right._jc) | CODE |
| HIGH | python/pyspark/pandas/frame.py | 1715 | # | 2|[{0, null}, {1, n...| | COMMENT |
| HIGH | python/pyspark/pandas/indexing.py | 557 | cast(ClassicColumn, col)._jc.toString() for col in data_spark_columns | CODE |
| HIGH | python/pyspark/pandas/groupby.py | 1307 | Flag to ignore NA(nan/null) values during truth testing. | STRING |
| HIGH | python/pyspark/pandas/base.py | 1466 | # If even one StructField is null, that row should be dropped. | COMMENT |
| HIGH⚡ | python/pyspark/pandas/tests/computation/test_combine.py | 682 | # Only update where new value > 150 (and old is null) | COMMENT |
| HIGH | python/pyspark/pandas/tests/window/test_rolling_adv.py | 42 | # pandas 3 returns 0.0 (not null); pandas < 3 returns nan. Both are matched here. | COMMENT |
| HIGH | …thon/pyspark/pandas/tests/window/test_expanding_adv.py | 43 | # pandas 3 returns 0.0 (not null); pandas < 3 returns nan. Both are matched here. | COMMENT |
| HIGH | …hon/pyspark/pandas/tests/diff_frames_ops/test_error.py | 198 | psidx1.equals(psidx2) | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 201 | self.assert_eq(pidx.equals(pidx), psidx.equals(psidx)) | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 205 | pidx.equals(pd.Index(["a", "b", "c"])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 206 | psidx.equals(ps.Index(["a", "b", "c"])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 209 | pidx.equals(pd.Index(["b", "b", "a"])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 210 | psidx.equals(ps.Index(["b", "b", "a"])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 221 | self.assert_eq(pmidx.equals(pmidx), psmidx.equals(psmidx)) | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 225 | pmidx.equals(pd.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 226 | psmidx.equals(ps.MultiIndex.from_tuples([("a", "x"), ("b", "y"), ("c", "z")])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 229 | pmidx.equals(pd.MultiIndex.from_tuples([("c", "z"), ("b", "y"), ("a", "x")])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 230 | psmidx.equals(ps.MultiIndex.from_tuples([("c", "z"), ("b", "y"), ("a", "x")])), | CODE |
| HIGH⚡ | python/pyspark/pandas/tests/indexes/test_basic.py | 234 | self.assert_eq(pidx.equals(pmidx), psidx.equals(psmidx)) | CODE |
| HIGH | python/pyspark/pandas/indexes/base.py | 388 | and self.equals(other) | CODE |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 412 | >>> idx.equals(idx) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 415 | ... idx.equals(ps.Index(['a', 'b', 'c'])) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 418 | ... idx.equals(ps.Index(['b', 'b', 'a'])) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 420 | >>> idx.equals(midx) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 425 | >>> midx.equals(midx) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 428 | ... midx.equals(ps.MultiIndex.from_tuples([('a', 'x'), ('b', 'y'), ('c', 'z')])) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 431 | ... midx.equals(ps.MultiIndex.from_tuples([('c', 'z'), ('b', 'y'), ('a', 'x')])) | STRING |
| HIGH⚡ | python/pyspark/pandas/indexes/base.py | 433 | >>> midx.equals(idx) | STRING |
| HIGH⚡ | python/pyspark/sql/conversion.py | 180 | if batch.schema.equals(arrow_schema, check_metadata=False): | CODE |
| HIGH | python/pyspark/sql/types.py | 1847 | return stringConcat.toString() | CODE |
| HIGH | python/pyspark/sql/types.py | 264 | null, UDTs, arrays, structs, and maps.""" | STRING |
| 97 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 662 | # ------------------------------------------------- | COMMENT |
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 698 | # ------------------------------------ | COMMENT |
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 704 | # ----------------------------------- | COMMENT |
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 816 | # ------------------------------- | COMMENT |
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 1125 | # ------------------------------------ | COMMENT |
| MEDIUM | python/pyspark/cloudpickle/cloudpickle.py | 1207 | # --------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 92 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 94 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 96 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 288 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 290 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 292 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 361 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 363 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 403 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 405 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 563 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 565 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 50 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 52 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 67 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 69 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 209 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 211 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 239 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 241 | # ------------------------------------------------------------------------- | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 441 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 443 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 531 | # ========================================================================= | COMMENT |
| MEDIUM | …/upstream/pyarrow/test_pyarrow_array_type_inference.py | 533 | # ========================================================================= | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 135 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 137 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 148 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 150 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 159 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 161 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 174 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 176 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 199 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 201 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 248 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 250 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 272 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 274 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 294 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 296 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 341 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 343 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 354 | # ===================================================================== | COMMENT |
| MEDIUM | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 356 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 104 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 106 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 156 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 158 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 166 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 168 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 176 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 178 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 194 | # ===================================================================== | COMMENT |
| MEDIUM⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 196 | # ===================================================================== | COMMENT |
| 110 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/run-tests.py | 43 | CODE | |
| LOW | python/packaging/connect/pyspark_connect/__init__.py | 22 | CODE | |
| LOW | python/pyspark/worker.py | 52 | CODE | |
| LOW | python/pyspark/util.py | 60 | CODE | |
| LOW | python/pyspark/util.py | 62 | CODE | |
| LOW | python/pyspark/util.py | 63 | CODE | |
| LOW | python/pyspark/util.py | 65 | CODE | |
| LOW | python/pyspark/util.py | 66 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 68 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 80 | CODE | |
| LOW | python/pyspark/util.py | 956 | CODE | |
| LOW | python/pyspark/util.py | 976 | CODE | |
| LOW | python/pyspark/conf.py | 27 | CODE | |
| LOW | python/pyspark/shell.py | 34 | CODE | |
| LOW | python/pyspark/__init__.py | 68 | CODE | |
| LOW | python/pyspark/__init__.py | 69 | CODE | |
| LOW | python/pyspark/__init__.py | 69 | CODE | |
| LOW | python/pyspark/__init__.py | 70 | CODE | |
| LOW | python/pyspark/__init__.py | 71 | CODE | |
| LOW | python/pyspark/__init__.py | 71 | CODE | |
| LOW | python/pyspark/__init__.py | 72 | CODE | |
| LOW | python/pyspark/__init__.py | 72 | CODE | |
| LOW | python/pyspark/__init__.py | 73 | CODE | |
| LOW | python/pyspark/__init__.py | 74 | CODE | |
| LOW | python/pyspark/__init__.py | 74 | CODE | |
| LOW | python/pyspark/__init__.py | 74 | CODE | |
| 565 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | bin/docker-image-tool.sh | 80 | # Create a smaller build context for docker in dev builds to make the build faster. Docker | COMMENT |
| MEDIUM | python/run-tests.py | 266 | # Create a unique temp directory under 'target/' for each run. The TMPDIR variable is | COMMENT |
| MEDIUM | python/run-tests.py | 554 | # Create the target directory before starting tasks to avoid races. | COMMENT |
| MEDIUM | python/pyspark/java_gateway.py | 77 | # Create a temporary directory where the gateway server should write the connection | COMMENT |
| MEDIUM | python/pyspark/statcounter.py | 18 | # This file is ported from spark/util/StatCounter.scala | COMMENT |
| MEDIUM | python/pyspark/daemon.py | 116 | # Create a new process group to corral our children | COMMENT |
| MEDIUM | python/pyspark/daemon.py | 122 | # Create a listening socket on the loopback interface | COMMENT |
| MEDIUM | python/pyspark/core/rdd.py | 245 | # This method is called when attempting to pickle an RDD, which is always an error: | COMMENT |
| MEDIUM | python/pyspark/core/rdd.py | 2827 | ... # Create the conf for writing | STRING |
| MEDIUM | python/pyspark/core/rdd.py | 2839 | ... # Create the conf for reading | STRING |
| MEDIUM | python/pyspark/core/rdd.py | 2986 | ... # Create the conf for writing | STRING |
| MEDIUM | python/pyspark/core/rdd.py | 2998 | ... # Create the conf for reading | STRING |
| MEDIUM | python/pyspark/core/context.py | 298 | # Create the Java SparkContext through Py4J | COMMENT |
| MEDIUM | python/pyspark/core/context.py | 303 | # Create a single Accumulator in Java that we'll send all our updates through; | COMMENT |
| MEDIUM | python/pyspark/core/context.py | 381 | # Create a temporary directory inside spark.local.dir: | COMMENT |
| MEDIUM | python/pyspark/core/context.py | 491 | # This method is called when attempting to pickle SparkContext, which is always an error: | STRING |
| MEDIUM | python/pyspark/core/context.py | 1492 | ... # Create the conf for writing | CODE |
| MEDIUM | python/pyspark/core/context.py | 1504 | ... # Create the conf for reading | CODE |
| MEDIUM | python/pyspark/core/context.py | 1689 | ... # Create the conf for writing | CODE |
| MEDIUM | python/pyspark/core/context.py | 1701 | ... # Create the conf for reading | CODE |
| MEDIUM | python/pyspark/mllib/tests/test_linalg.py | 542 | # Create a CSC matrix with non-sorted indices | COMMENT |
| MEDIUM | python/pyspark/mllib/tests/test_streaming_algorithms.py | 101 | # Create a toy dataset by setting a tiny offset for each point. | COMMENT |
| MEDIUM | python/pyspark/mllib/tests/test_streaming_algorithms.py | 396 | # Create a model with initial Weights equal to coefs | COMMENT |
| MEDIUM | python/pyspark/tests/test_rdd.py | 736 | # Create a DataFrame with many columns, call a Python function on each row, and take only | COMMENT |
| MEDIUM | python/pyspark/pipelines/init_cli.py | 52 | # Create the storage directory | COMMENT |
| MEDIUM⚡ | python/pyspark/pipelines/init_cli.py | 65 | # Create the transformations directory | COMMENT |
| MEDIUM⚡ | python/pyspark/pipelines/init_cli.py | 69 | # Create the Python example file | COMMENT |
| MEDIUM⚡ | python/pyspark/pipelines/init_cli.py | 74 | # Create the SQL example file | COMMENT |
| MEDIUM | python/pyspark/pipelines/tests/test_cli.py | 376 | # Create a minimal pipeline spec | STRING |
| MEDIUM | python/pyspark/pipelines/tests/test_cli.py | 400 | # Create a minimal pipeline spec | STRING |
| MEDIUM | python/pyspark/pipelines/tests/test_cli.py | 425 | # Create a minimal pipeline spec | STRING |
| MEDIUM | python/pyspark/ml/pipeline.py | 188 | # Create a new instance of this stage. | COMMENT |
| MEDIUM | python/pyspark/ml/pipeline.py | 346 | # Create a new instance of this stage. | COMMENT |
| MEDIUM | python/pyspark/ml/tuning.py | 982 | # Create a new instance of this stage. | COMMENT |
| MEDIUM | python/pyspark/ml/tuning.py | 1560 | # Create a new instance of this stage. | COMMENT |
| MEDIUM | python/pyspark/ml/tuning.py | 1685 | # Create a new instance of this stage. | COMMENT |
| MEDIUM | python/pyspark/ml/tests/test_feature.py | 361 | # Create a DataFrame | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/io/test_io.py | 34 | # This file contains test cases for 'Serialization / IO / Conversion' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_time_series.py | 26 | # This file contains test cases for 'Time series-related' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_spark.py | 34 | # This file contains test cases for 'Spark-related' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_attrs.py | 26 | # This file contains test cases for 'Attributes and underlying data' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_constructor.py | 33 | # This file contains test cases for 'Constructor' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_conversion.py | 25 | # This file contains test cases for 'Conversion' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_reindexing.py | 31 | # This file contains test cases for 'Reindexing / Selection / Label manipulation' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/frame/test_reshaping.py | 27 | # This file contains test cases for 'Reshaping, sorting, transposing' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/computation/test_combine.py | 25 | # This file contains test cases for 'Combining / joining / merging' | COMMENT |
| MEDIUM | …on/pyspark/pandas/tests/computation/test_apply_func.py | 29 | # This file contains test cases for 'Function application, GroupBy & Window' | COMMENT |
| MEDIUM | …/pyspark/pandas/tests/computation/test_missing_data.py | 27 | # This file contains test cases for 'Missing data handling' | COMMENT |
| MEDIUM | …on/pyspark/pandas/tests/computation/test_binary_ops.py | 26 | # This file contains test cases for 'Binary operator functions' | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/computation/test_compute.py | 26 | # This file contains test cases for 'Computations / Descriptive Stats' | COMMENT |
| MEDIUM | …thon/pyspark/pandas/tests/indexes/test_indexing_adv.py | 56 | # Create the equivalent of pdf.loc[3] as a Koalas Series | COMMENT |
| MEDIUM | …thon/pyspark/pandas/tests/indexes/test_indexing_adv.py | 142 | # Create the equivalent of pdf.loc[3] as a Koalas Series | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/indexes/test_indexing.py | 26 | # This file contains test cases for 'Indexing, Iteration' | COMMENT |
| MEDIUM | python/pyspark/pandas/indexes/base.py | 264 | # This method is used via `DataFrame.info` API internally. | COMMENT |
| MEDIUM | python/pyspark/sql/dataframe.py | 566 | ... # Create a table with Rate source. | STRING |
| MEDIUM | python/pyspark/sql/dataframe.py | 6834 | >>> # Create a simple UDTF that processes table data | STRING |
| MEDIUM | python/pyspark/sql/dataframe.py | 6840 | >>> # Create a DataFrame | STRING |
| MEDIUM | python/pyspark/sql/session.py | 645 | # Create a new SparkSession in the JVM | COMMENT |
| MEDIUM | python/pyspark/sql/session.py | 1669 | # Create a DataFrame from pandas DataFrame. | COMMENT |
| MEDIUM | python/pyspark/sql/session.py | 1674 | # Create a DataFrame from PyArrow Table. | COMMENT |
| 83 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/run-tests.py | 234 | CODE | |
| LOW | python/run-tests.py | 478 | CODE | |
| LOW | python/pyspark/worker.py | 571 | CODE | |
| LOW | python/pyspark/worker.py | 603 | CODE | |
| LOW | python/pyspark/worker.py | 646 | CODE | |
| LOW | python/pyspark/worker.py | 758 | CODE | |
| LOW | python/pyspark/worker.py | 2046 | CODE | |
| LOW | python/pyspark/worker.py | 5070 | CODE | |
| LOW | python/pyspark/worker.py | 1279 | CODE | |
| LOW | python/pyspark/worker.py | 855 | CODE | |
| LOW | python/pyspark/worker.py | 1017 | CODE | |
| LOW | python/pyspark/worker.py | 1293 | CODE | |
| LOW | python/pyspark/worker.py | 1373 | CODE | |
| LOW | python/pyspark/worker.py | 1429 | CODE | |
| LOW | python/pyspark/worker.py | 2488 | CODE | |
| LOW | python/pyspark/worker.py | 2621 | CODE | |
| LOW | python/pyspark/worker.py | 2702 | CODE | |
| LOW | python/pyspark/worker.py | 2783 | CODE | |
| LOW | python/pyspark/worker.py | 3581 | CODE | |
| LOW | python/pyspark/worker.py | 3695 | CODE | |
| LOW | python/pyspark/worker.py | 4253 | CODE | |
| LOW | python/pyspark/worker.py | 4502 | CODE | |
| LOW | python/pyspark/worker.py | 4704 | CODE | |
| LOW | python/pyspark/worker.py | 4895 | CODE | |
| LOW | python/pyspark/worker.py | 5138 | CODE | |
| LOW | python/pyspark/worker.py | 1568 | CODE | |
| LOW | python/pyspark/worker.py | 4293 | CODE | |
| LOW | python/pyspark/worker.py | 4945 | CODE | |
| LOW | python/pyspark/worker.py | 5156 | CODE | |
| LOW | python/pyspark/worker.py | 1301 | CODE | |
| LOW | python/pyspark/worker.py | 1319 | CODE | |
| LOW | python/pyspark/worker.py | 1329 | CODE | |
| LOW | python/pyspark/worker.py | 1351 | CODE | |
| LOW | python/pyspark/worker_message.py | 140 | CODE | |
| LOW | python/pyspark/util.py | 587 | CODE | |
| LOW | python/pyspark/conf.py | 180 | CODE | |
| LOW | python/pyspark/shuffle.py | 62 | CODE | |
| LOW | python/pyspark/shuffle.py | 779 | CODE | |
| LOW | python/pyspark/statcounter.py | 60 | CODE | |
| LOW | python/pyspark/accumulators.py | 264 | CODE | |
| LOW | python/pyspark/accumulators.py | 269 | CODE | |
| LOW | python/pyspark/profiler.py | 189 | CODE | |
| LOW | python/pyspark/daemon.py | 46 | CODE | |
| LOW | python/pyspark/daemon.py | 115 | CODE | |
| LOW | python/pyspark/core/rdd.py | 2210 | CODE | |
| LOW | python/pyspark/core/rdd.py | 3672 | CODE | |
| LOW | python/pyspark/core/rdd.py | 3724 | CODE | |
| LOW | python/pyspark/core/context.py | 225 | CODE | |
| LOW | python/pyspark/core/context.py | 1816 | CODE | |
| LOW | python/pyspark/logger/worker_io.py | 214 | CODE | |
| LOW | python/pyspark/cloudpickle/cloudpickle.py | 313 | CODE | |
| LOW | python/pyspark/cloudpickle/cloudpickle.py | 338 | CODE | |
| LOW | python/pyspark/cloudpickle/cloudpickle.py | 1069 | CODE | |
| LOW | python/pyspark/cloudpickle/cloudpickle.py | 1441 | CODE | |
| LOW | python/pyspark/mllib/classification.py | 235 | CODE | |
| LOW | python/pyspark/mllib/common.py | 75 | CODE | |
| LOW | python/pyspark/mllib/common.py | 96 | CODE | |
| LOW | python/pyspark/mllib/common.py | 160 | CODE | |
| LOW | python/pyspark/mllib/linalg/__init__.py | 98 | CODE | |
| LOW | python/pyspark/mllib/linalg/__init__.py | 116 | CODE | |
| 348 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/conf_vscode/sitecustomize.py | 39 | except Exception: | CODE |
| MEDIUM | python/pyspark/worker.py | 1837 | def evaluate(*a) -> tuple: | CODE |
| MEDIUM | python/pyspark/worker.py | 5156 | def _reader_thread(): | CODE |
| LOW | python/pyspark/worker.py | 1255 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 1476 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 1620 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 1635 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 1746 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 1842 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 5174 | except Exception as e: | CODE |
| LOW | python/pyspark/worker.py | 5211 | except Exception: | CODE |
| LOW | python/pyspark/worker.py | 5264 | except Exception: | CODE |
| LOW⚡ | python/pyspark/threaddump.py | 45 | except Exception as e: | CODE |
| MEDIUM⚡ | python/pyspark/threaddump.py | 46 | print(f"Error getting children of process {args.pid}: {e}") | CODE |
| LOW⚡ | python/pyspark/threaddump.py | 54 | except Exception: | CODE |
| MEDIUM | python/pyspark/threaddump.py | 28 | def main() -> int: | CODE |
| LOW | python/pyspark/util.py | 815 | except Exception: | CODE |
| LOW | python/pyspark/serializers.py | 435 | except Exception as e: | CODE |
| LOW | python/pyspark/shell.py | 73 | except Exception: | CODE |
| LOW | python/pyspark/shell.py | 90 | except Exception: | CODE |
| LOW | python/pyspark/memory_profiler_ext.py | 32 | except Exception: | CODE |
| LOW | python/pyspark/memory_profiler_ext.py | 69 | except Exception: | CODE |
| LOW | python/pyspark/install.py | 163 | except Exception: | CODE |
| LOW | python/pyspark/install.py | 213 | except Exception: | CODE |
| LOW | python/pyspark/install.py | 245 | except Exception as e: | CODE |
| LOW | python/pyspark/instrumentation_utils.py | 48 | except Exception as ex: | CODE |
| LOW | python/pyspark/instrumentation_utils.py | 72 | except Exception as ex: | CODE |
| LOW | python/pyspark/daemon.py | 96 | except Exception: | CODE |
| LOW | python/pyspark/daemon.py | 272 | except Exception: | CODE |
| LOW | python/pyspark/core/context.py | 374 | except Exception: | CODE |
| LOW | python/pyspark/core/broadcast.py | 181 | except Exception as e: | CODE |
| LOW | python/pyspark/logger/worker_io.py | 252 | except Exception: | CODE |
| LOW | python/pyspark/cloudpickle/cloudpickle.py | 232 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_rdd.py | 354 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_rdd.py | 919 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_taskcontext.py | 207 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_taskcontext.py | 278 | except Exception: | CODE |
| MEDIUM | python/pyspark/tests/test_taskcontext.py | 204 | def f(iterator): | CODE |
| LOW | python/pyspark/tests/test_util.py | 183 | except Exception as e: | CODE |
| LOW | python/pyspark/tests/test_pin_thread.py | 68 | except Exception as e: | CODE |
| LOW | python/pyspark/tests/test_pin_thread.py | 123 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_worker.py | 55 | except Exception: | CODE |
| LOW⚡ | python/pyspark/tests/test_worker.py | 155 | except Exception: | CODE |
| MEDIUM | python/pyspark/tests/test_worker.py | 52 | def run(): | CODE |
| MEDIUM | python/pyspark/tests/test_worker.py | 152 | def count(): | CODE |
| LOW | python/pyspark/tests/test_context.py | 237 | except Exception: | CODE |
| LOW | python/pyspark/tests/test_context.py | 333 | except Exception as e: | CODE |
| MEDIUM | python/pyspark/tests/test_context.py | 330 | def run_job(): | CODE |
| LOW | python/pyspark/tests/test_install_spark.py | 53 | except Exception: | CODE |
| LOW | …ream/pyarrow/test_pyarrow_array_from_pandas_default.py | 124 | except Exception as e: | CODE |
| LOW⚡ | …stream/pyarrow/test_pyarrow_arrow_to_pandas_default.py | 96 | except Exception as e: | CODE |
| LOW⚡ | …tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py | 82 | except Exception as e: | CODE |
| LOW | …park/tests/upstream/pyarrow/test_pyarrow_table_cast.py | 84 | except Exception as e: | CODE |
| LOW⚡ | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 727 | except Exception as e: | CODE |
| LOW⚡ | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 732 | except Exception as e: | CODE |
| LOW | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 297 | except Exception as e: | CODE |
| LOW⚡ | …park/tests/upstream/pyarrow/test_pyarrow_array_cast.py | 130 | except Exception as e: | CODE |
| LOW | …upstream/pyarrow/test_pyarrow_dataframe_from_pandas.py | 152 | except Exception as e: | CODE |
| LOW | …upstream/pyarrow/test_pyarrow_dataframe_from_pandas.py | 216 | except Exception as e: | CODE |
| LOW | python/pyspark/testing/sqlutils.py | 110 | except Exception: | CODE |
| 178 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …/org/apache/spark/launcher/AbstractCommandBuilder.java | 244 | // Place slf4j-api-* jar first to be robust | COMMENT |
| MEDIUM | …c/test/scala/org/apache/spark/ui/UISeleniumSuite.scala | 467 | // Essentially, we want to check that none of the stage rows show | COMMENT |
| MEDIUM | …c/test/scala/org/apache/spark/ui/UISeleniumSuite.scala | 523 | // Essentially, we want to check that none of the stage rows show | COMMENT |
| MEDIUM | …ala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 2567 | // For a robust test assertion, limit number of job tasks to 1; that is, | COMMENT |
| MEDIUM | …apache/spark/scheduler/SchedulerIntegrationSuite.scala | 431 | // it really can only be "best-effort" in any case, and the scheduler should be robust to that. | STRING |
| MEDIUM | …rg/apache/spark/scheduler/TaskSchedulerImplSuite.scala | 548 | // Even though we launched a local task above, we still utilize non-local exec2. | COMMENT |
| MEDIUM | …he/spark/scheduler/HealthTrackerIntegrationSuite.scala | 80 | // robust to one bad node. | COMMENT |
| MEDIUM | …main/scala/org/apache/spark/storage/BlockManager.scala | 1400 | // BlockTransferService, which will leverage it to spill the block; if not, then passed-in | COMMENT |
| MEDIUM | python/packaging/classic/setup.py | 183 | # TODO(SPARK-32837) leverage pip's custom options | COMMENT |
| LOW | …spark/messages/socket/spark_socket_message_receiver.py | 54 | # For socket communication, we just pass along the underlying socket | COMMENT |
| LOW | python/pyspark/tests/test_install_spark.py | 60 | # we just use a hard-coded version. | COMMENT |
| MEDIUM | …am/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py | 324 | # is robust where raw address equality is not. | COMMENT |
| LOW | python/pyspark/ml/tests/test_functions.py | 208 | # just return the batch size as the "prediction" | STRING |
| MEDIUM | python/pyspark/errors/utils.py | 373 | # Excluding Python magic methods that do not utilize JVM functions. | COMMENT |
| LOW | python/pyspark/pandas/resample.py | 360 | # here just use Pandas' resample on a 1-length series to get it. | COMMENT |
| LOW | python/pyspark/pandas/generic.py | 3102 | # If Series has only a single value, just return it as a scalar. | STRING |
| MEDIUM | python/pyspark/pandas/series.py | 6098 | # If `where` has duplicate items, leverage the pandas directly | COMMENT |
| LOW | python/pyspark/pandas/utils.py | 796 | # '+' is meaningless for writing methods, but pandas just pass it as 'w'. | COMMENT |
| LOW | python/pyspark/pandas/utils.py | 800 | # '+' is meaningless for writing methods, but pandas just pass it as 'a'. | COMMENT |
| LOW | python/pyspark/pandas/frame.py | 10262 | # In this case, we can simply use `summary` to calculate the stats. | COMMENT |
| MEDIUM | python/pyspark/pandas/tests/groupby/test_stat.py | 30 | # TODO: All statistical functions should leverage this utility | COMMENT |
| MEDIUM | python/pyspark/sql/session.py | 622 | # used in conjunction with Spark Connect mode. | COMMENT |
| MEDIUM⚡ | python/pyspark/sql/tests/test_functions.py | 3161 | """Test tuple_sketch_agg + operations + estimate comprehensive test - double""" | STRING |
| MEDIUM | python/pyspark/sql/tests/test_functions.py | 3216 | """Test tuple_sketch_agg + operations + estimate comprehensive test - integer""" | STRING |
| MEDIUM⚡ | python/pyspark/sql/tests/df_golden/test_df_golden.py | 83 | # ``SPARK_LOCAL_REMOTE`` (the test harness) ``stop()`` terminates the | COMMENT |
| MEDIUM | …ing/test_pandas_transform_with_state_state_variable.py | 351 | # TODO SPARK-50908 holistic fix for TTL suite | COMMENT |
| MEDIUM | python/pyspark/sql/connect/client/core.py | 559 | # Rewrite the URL to use http as the scheme so that we can leverage | COMMENT |
| LOW | R/pkg/R/sparkR.R | 664 | #' To remove/unset property simply set `value` to NULL e.g. setLocalProperty("key", NULL) | COMMENT |
| MEDIUM | R/pkg/R/column.R | 296 | #' Can be used in conjunction with \code{when} to specify a default value for expressions. | COMMENT |
| MEDIUM | …apache/spark/streaming/ReceivedBlockTrackerSuite.scala | 320 | // deletion more robust rather than a parallelized operation where we fire and forget | COMMENT |
| MEDIUM | …cala/org/apache/spark/streaming/ui/StreamingPage.scala | 163 | // We leverage timeFormat as the value would be same as timeFormat. This means it is | COMMENT |
| MEDIUM | …rg/apache/spark/network/crypto/CtrTransportCipher.java | 229 | // to utilize two helper ByteArrayWritableChannel for streaming. One is used to receive raw data | COMMENT |
| MEDIUM | …network/shuffle/streaming/StreamingShuffleMessage.java | 68 | // Essentially, other message types from reader to writer won't have a valid sequence number. | COMMENT |
| MEDIUM | …scala/org/apache/spark/examples/mllib/LDAExample.scala | 139 | // add (1.0 / actualCorpusSize) to MiniBatchFraction be more robust on tiny datasets. | COMMENT |
| MEDIUM | …a/org/apache/spark/sql/StatisticsCollectionSuite.scala | 934 | // We can't leverage LogicalRDD.fromDataset here, since it triggers physical planning and | COMMENT |
| MEDIUM | …c/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 1649 | // We can't leverage LogicalRDD.fromDataset here, since it triggers physical planning and | COMMENT |
| MEDIUM | …apache/spark/sql/streaming/FileStreamSourceSuite.scala | 2342 | // file stream source will not leverage unread files - next batch will also trigger | COMMENT |
| MEDIUM | …org/apache/spark/sql/execution/UnionCodegenSuite.scala | 535 | // Explicit cap so the assertion is robust to future default changes. | STRING |
| MEDIUM | …l/execution/datasources/PushVariantIntoScanSuite.scala | 1119 | // Project/Filter nodes wrap it. This keeps scan-content assertions robust against optimizer | COMMENT |
| MEDIUM | …ion/datasources/v2/state/StateDataSourceTestBase.scala | 108 | // check with more data - leverage full partitions | COMMENT |
| MEDIUM | …k/sql/execution/exchange/EnsureRequirementsSuite.scala | 1020 | // (spark.sql.optimizer.excludedRules), and EnsureRequirements must also stay robust | COMMENT |
| MEDIUM | …park/sql/catalyst/analysis/ResolveSessionCatalog.scala | 251 | // resolution was skipped) so the rewrite stays robust across analyzer ordering changes. | COMMENT |
| MEDIUM | …in/scala/org/apache/spark/sql/jdbc/OracleDialect.scala | 166 | // Not sure if there is a more robust way to identify the field as a float (or other | COMMENT |
| MEDIUM | …icpruning/RowLevelOperationRuntimeGroupFiltering.scala | 95 | // in order to leverage a regular batch scan in the group filter query | COMMENT |
| MEDIUM | …on/python/streaming/ApplyInPandasWithStateWriter.scala | 107 | // from the entire data part of Arrow RecordBatch. We leverage the state metadata to also | COMMENT |
| MEDIUM | …erators/stateful/StatefulStreamlineAggregateExec.scala | 412 | // The writes which did not go into state store yet. We also leverage this dirty writes to the | COMMENT |
| MEDIUM | …ors/stateful/join/StreamingSymmetricHashJoinExec.scala | 1098 | // to let users leverage both sides of event time column for output of join, so the watermark | COMMENT |
| MEDIUM | …/execution/streaming/runtime/FileStreamSourceLog.scala | 130 | // be started. We leverage the fact to skip calculation if possible. | COMMENT |
| MEDIUM | …sql/execution/streaming/runtime/ProgressReporter.scala | 572 | // by itself, so leverage it. | COMMENT |
| MEDIUM | …ark/sql/catalyst/expressions/CodeGenerationSuite.scala | 645 | | // to make the test more robust, in case the compiler can eliminate the else branch. | STRING |
| MEDIUM | …e/spark/sql/catalyst/analysis/RelationResolution.scala | 452 | // To utilize this code path to execute V1 commands, e.g. INSERT, | COMMENT |
| MEDIUM | …ql/catalyst/expressions/SubExprEvaluationRuntime.scala | 100 | // We leverage `IdentityHashMap` so we compare expression keys by reference here. | COMMENT |
| MEDIUM | …k/sql/catalyst/expressions/codegen/CodeFormatter.scala | 119 | // examines the number of parenthesis and braces in that line. This isn't the most robust | COMMENT |
| MEDIUM | …/pipelines/graph/AutoCdcScd2ColumnEvolutionSuite.scala | 78 | // is column-schema narrowing rather than a tracked-set change (robust once SPARK-58391 lands). | COMMENT |
| MEDIUM | …/pipelines/graph/AutoCdcScd2ColumnEvolutionSuite.scala | 170 | // tracked set (robust once SPARK-58391 lands). | COMMENT |
| MEDIUM | …/spark/sql/hive/execution/HiveCompatibilitySuite.scala | 287 | // The isolated classloader seemed to make some of our test reset mechanisms less robust. | COMMENT |
| MEDIUM | …n/scala/org/apache/spark/sql/hive/HiveInspectors.scala | 1065 | // TODO: hard-coding a list here is not very robust. A better idea is to have some kind of query | COMMENT |
| MEDIUM | …/main/java/org/apache/spark/sql/streaming/Trigger.java | 97 | * @deprecated This is deprecated as of Spark 3.4.0. Use {@link #AvailableNow()} to leverage | COMMENT |
| MEDIUM | …k/sql/connect/pipelines/PipelineEventSenderSuite.scala | 284 | // total is logged at shutdown. The assertions match on substrings so they stay robust to | COMMENT |
| MEDIUM | …e/spark/sql/hive/thriftserver/SharedThriftServer.scala | 134 | // It's much more robust than set a random port generated by ourselves ahead | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1059 | // Step 1: Write an in-progress log containing only ApplicationStart (no job). | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1066 | // Step 2: Load the app UI; this builds the disk store from the in-progress snapshot. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1072 | // Step 3: Simulate ApplicationCache LRU eviction BEFORE the app completes. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1082 | // Step 4: Complete the app. Write a new log file (without .inprogress suffix) that | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1092 | // Step 5: checkForLogs() detects the completed log. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1097 | // Step 6: Load the UI again. | COMMENT |
| LOW⚡ | python/pyspark/sql/conversion.py | 185 | # Step 1: pick source columns from batch to align with target schema | COMMENT |
| LOW | python/pyspark/sql/conversion.py | 214 | # Step 2: check types / cast, collect all mismatches | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 109 | # Step 1: Write initial data and run streaming query | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 116 | # Step 2: Repartition to more partitions | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 121 | # Step 3: Add more data and restart query | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 129 | # Step 4: Repartition to fewer partitions | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 134 | # Step 5: Add more data and restart query | COMMENT |
| LOW⚡ | R/pkg/inst/worker/worker.R | 247 | # Step 1: hash the data to an environment | COMMENT |
| LOW | R/pkg/inst/worker/worker.R | 264 | # Step 2: write out all of the environment as key-value pairs. | COMMENT |
| LOW | common/config/README.md | 22 | ### Step 1: Choose the Right File | COMMENT |
| LOW | common/config/README.md | 29 | ### Step 2: Add the Config Entry | COMMENT |
| LOW | common/config/README.md | 54 | ### Step 3: Register the Config File (if new) | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1814 | // Step 1: Run the stateful query to create the full checkpoint structure | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1817 | // Step 2: Delete the state directory | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1823 | // Step 3: Attempt to read state - expected to fail since state is deleted | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1829 | // Step 4: Verify the state directory was NOT recreated by the reader | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1866 | // Step 1: Write data with correct schema and commit | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1877 | // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType) | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1906 | // Step 1: Write data with correct schema and commit | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1918 | // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType) | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 302 | // Step 2: Scheduler (via triggerNow) submits first op as | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 307 | // Step 3: Release snapshot. Post-work queues remaining op (Cleanup) | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 317 | // Step 4: Scheduler (via triggerNow) picks up Cleanup as | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 326 | // Step 5: Release cleanup. FromUnloadedProvidersQueue calls closeProvider. | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 235 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 277 | // Step 1: Create state by running a composite key streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 311 | // Step 1: Create state by running stream-stream join | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 323 | // Step 2: Test all 4 state stores created by stream-stream join | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 350 | // Step 1: Create state by running flatMapGroupsWithState | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 820 | // Step 1: Create state by running dropDuplicatesWithinWatermark | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 845 | // Step 1: Create state by running dropDuplicates with column | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 870 | // Step 1: Create state by running session window aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 899 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 972 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 128 | // Step 1: Run initial query to create state | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 131 | // Step 2: Read state data before repartition | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 150 | // Step 3: Run repartition | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 159 | // Step 4: Verify offset and commit logs | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 164 | // Step 5: Validate state for each store and column family after repartition | COMMENT |
| LOW | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 192 | // Step 6: Resume query with new input and verify | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 772 | // Step 1: subquery1 (cp) and subquery2 (np) merge: | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 779 | // Step 2: subquery3 (np) merges with merged(1,2) (cp). The cp Filter is tagged, so only a | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 828 | // Step 1: subquery1 (cp) and subquery2 (np) merge as usual: | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 834 | // Step 2: subquery3 (np, condition a > 1) merges with merged(1,2) (cp). The cp Filter is | COMMENT |
| LOW | …scala/org/apache/spark/sql/avro/AvroOutputWriter.scala | 38 | // NOTE: This class is instantiated and used on executor side only, no need to be serializable. | COMMENT |
| LOW⚡ | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 139 | // Step 1: Apply variable substitution to expand any variable references. | COMMENT |
| LOW⚡ | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 142 | // Step 2: Apply parameter substitution if a parameter context is provided. | COMMENT |
| LOW | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 167 | // Step 3: Set up the origin with SQL text and position mapper to enable | COMMENT |
| LOW | …/parquet/ParquetOutputWriterWithVariantShredding.scala | 39 | // NOTE: This class is instantiated and used on executor side only, no need to be serializable. | COMMENT |
| LOW | …xecution/datasources/parquet/ParquetOutputWriter.scala | 27 | // NOTE: This class is instantiated and used on executor side only, no need to be serializable. | COMMENT |
| LOW⚡ | …/execution/aggregate/TungstenAggregationIterator.scala | 326 | // Step 5: Get the sorted iterator from the externalSorter. | COMMENT |
| LOW⚡ | …/execution/aggregate/TungstenAggregationIterator.scala | 329 | // Step 6: Pre-load the first key-value pair from the sorted iterator to make | COMMENT |
| LOW | …/execution/aggregate/TungstenAggregationIterator.scala | 342 | // Step 7: set sortBased to true. | COMMENT |
| LOW⚡ | …t/analysis/SequentialStreamingUnionAnalysisSuite.scala | 227 | // Step 1: Flatten the nested unions | COMMENT |
| 15 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1059 | // Step 1: Write an in-progress log containing only ApplicationStart (no job). | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1066 | // Step 2: Load the app UI; this builds the disk store from the in-progress snapshot. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1072 | // Step 3: Simulate ApplicationCache LRU eviction BEFORE the app completes. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1082 | // Step 4: Complete the app. Write a new log file (without .inprogress suffix) that | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1092 | // Step 5: checkForLogs() detects the completed log. | COMMENT |
| LOW⚡ | …ache/spark/deploy/history/FsHistoryProviderSuite.scala | 1097 | // Step 6: Load the UI again. | COMMENT |
| LOW⚡ | …/util/collection/unsafe/sort/UnsafeExternalSorter.java | 474 | // Step 1: | COMMENT |
| LOW⚡ | …/util/collection/unsafe/sort/UnsafeExternalSorter.java | 477 | // Step 2: | COMMENT |
| LOW⚡ | …/util/collection/unsafe/sort/UnsafeExternalSorter.java | 480 | // Step 3: | COMMENT |
| LOW | …/scala/org/apache/spark/storage/BlockInfoManager.scala | 457 | // reader counts. We need to check if the readLocksByTask per tasks are present, if they | COMMENT |
| LOW⚡ | python/pyspark/sql/conversion.py | 185 | # Step 1: pick source columns from batch to align with target schema | COMMENT |
| LOW | python/pyspark/sql/conversion.py | 214 | # Step 2: check types / cast, collect all mismatches | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 109 | # Step 1: Write initial data and run streaming query | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 116 | # Step 2: Repartition to more partitions | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 121 | # Step 3: Add more data and restart query | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 129 | # Step 4: Repartition to fewer partitions | COMMENT |
| LOW⚡ | …/streaming/test_streaming_offline_state_repartition.py | 134 | # Step 5: Add more data and restart query | COMMENT |
| LOW | …rk/sql/streaming/transform_with_state_driver_worker.py | 70 | # and the following code block should be only run once for each query run | COMMENT |
| LOW | R/pkg/inst/worker/daemon.R | 98 | # Forking succeeded and we need to check if they finished their jobs every second. | COMMENT |
| LOW⚡ | R/pkg/inst/worker/worker.R | 247 | # Step 1: hash the data to an environment | COMMENT |
| LOW | R/pkg/inst/worker/worker.R | 264 | # Step 2: write out all of the environment as key-value pairs. | COMMENT |
| LOW | …ming/FlatMapGroupsWithStateWithInitialStateSuite.scala | 57 | // We need to check if not explicitly calling update will still save the init state or not | COMMENT |
| LOW | …ming/FlatMapGroupsWithStateWithInitialStateSuite.scala | 124 | // We need to check if not explicitly calling update will still save the state or not | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1814 | // Step 1: Run the stateful query to create the full checkpoint structure | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1817 | // Step 2: Delete the state directory | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1823 | // Step 3: Attempt to read state - expected to fail since state is deleted | COMMENT |
| LOW⚡ | …on/datasources/v2/state/StateDataSourceReadSuite.scala | 1829 | // Step 4: Verify the state directory was NOT recreated by the reader | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1866 | // Step 1: Write data with correct schema and commit | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1877 | // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType) | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1906 | // Step 1: Write data with correct schema and commit | COMMENT |
| LOW | …execution/streaming/state/RocksDBStateStoreSuite.scala | 1918 | // Step 2: Reopen with a wrong valueSchema (StringType instead of IntegerType) | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 302 | // Step 2: Scheduler (via triggerNow) submits first op as | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 307 | // Step 3: Release snapshot. Post-work queues remaining op (Cleanup) | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 317 | // Step 4: Scheduler (via triggerNow) picks up Cleanup as | COMMENT |
| LOW⚡ | …eaming/state/StateStoreDecoupledMaintenanceSuite.scala | 326 | // Step 5: Release cleanup. FromUnloadedProvidersQueue calls closeProvider. | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 235 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 277 | // Step 1: Create state by running a composite key streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 311 | // Step 1: Create state by running stream-stream join | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 323 | // Step 2: Test all 4 state stores created by stream-stream join | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 350 | // Step 1: Create state by running flatMapGroupsWithState | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 820 | // Step 1: Create state by running dropDuplicatesWithinWatermark | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 845 | // Step 1: Create state by running dropDuplicates with column | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 870 | // Step 1: Create state by running session window aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 899 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW | …state/StatePartitionAllColumnFamiliesWriterSuite.scala | 972 | // Step 1: Create state by running a streaming aggregation | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 128 | // Step 1: Run initial query to create state | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 131 | // Step 2: Read state data before repartition | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 150 | // Step 3: Run repartition | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 159 | // Step 4: Verify offset and commit logs | COMMENT |
| LOW⚡ | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 164 | // Step 5: Validate state for each store and column family after repartition | COMMENT |
| LOW | …ng/state/OfflineStateRepartitionIntegrationSuite.scala | 192 | // Step 6: Resume query with new input and verify | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 772 | // Step 1: subquery1 (cp) and subquery2 (np) merge: | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 779 | // Step 2: subquery3 (np) merges with merged(1,2) (cp). The cp Filter is tagged, so only a | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 828 | // Step 1: subquery1 (cp) and subquery2 (np) merge as usual: | COMMENT |
| LOW⚡ | …ark/sql/execution/planmerging/MergeSubplansSuite.scala | 834 | // Step 2: subquery3 (np, condition a > 1) merges with merged(1,2) (cp). The cp Filter is | COMMENT |
| LOW | …g/apache/spark/sql/classic/StreamingQueryManager.scala | 388 | // The following code block checks if a stream with the same name or id is running. Then it | COMMENT |
| LOW⚡ | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 139 | // Step 1: Apply variable substitution to expand any variable references. | COMMENT |
| LOW⚡ | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 142 | // Step 2: Apply parameter substitution if a parameter context is provided. | COMMENT |
| LOW | …la/org/apache/spark/sql/execution/SparkSqlParser.scala | 167 | // Step 3: Set up the origin with SQL text and position mapper to enable | COMMENT |
| LOW | …ql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala | 133 | // Also, we need to check if join is done on 2 tables from 2 different databases within same | COMMENT |
| 6 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/pyspark/core/context.py | 171 | CODE | |
| LOW | python/pyspark/mllib/regression.py | 291 | CODE | |
| LOW | python/pyspark/mllib/regression.py | 475 | CODE | |
| LOW | python/pyspark/mllib/regression.py | 655 | CODE | |
| LOW | python/pyspark/mllib/classification.py | 330 | CODE | |
| LOW | python/pyspark/mllib/classification.py | 422 | CODE | |
| LOW | python/pyspark/mllib/classification.py | 645 | CODE | |
| LOW | python/pyspark/pipelines/api.py | 113 | CODE | |
| LOW | python/pyspark/pipelines/api.py | 126 | CODE | |
| LOW | python/pyspark/pipelines/api.py | 217 | CODE | |
| LOW | python/pyspark/pipelines/api.py | 230 | CODE | |
| LOW | python/pyspark/pipelines/api.py | 531 | CODE | |
| LOW | python/pyspark/streaming/kinesis.py | 69 | CODE | |
| LOW | python/pyspark/streaming/kinesis.py | 88 | CODE | |
| LOW | python/pyspark/testing/utils.py | 745 | CODE | |
| LOW | python/pyspark/ml/regression.py | 301 | CODE | |
| LOW | python/pyspark/ml/regression.py | 335 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1110 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1146 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1410 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1450 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1760 | CODE | |
| LOW | python/pyspark/ml/regression.py | 1802 | CODE | |
| LOW | python/pyspark/ml/regression.py | 2161 | CODE | |
| LOW | python/pyspark/ml/regression.py | 2201 | CODE | |
| LOW | python/pyspark/ml/regression.py | 2566 | CODE | |
| LOW | python/pyspark/ml/regression.py | 2602 | CODE | |
| LOW | python/pyspark/ml/regression.py | 3136 | CODE | |
| LOW | python/pyspark/ml/regression.py | 3167 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 409 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 439 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 784 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 816 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 1142 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 1168 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 1700 | CODE | |
| LOW | python/pyspark/ml/clustering.py | 1738 | CODE | |
| LOW | python/pyspark/ml/classification.py | 725 | CODE | |
| LOW | python/pyspark/ml/classification.py | 757 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1243 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1269 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1295 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1340 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1366 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1393 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1783 | CODE | |
| LOW | python/pyspark/ml/classification.py | 1821 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2085 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2128 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2563 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2607 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2967 | CODE | |
| LOW | python/pyspark/ml/classification.py | 2994 | CODE | |
| LOW | python/pyspark/ml/classification.py | 3217 | CODE | |
| LOW | python/pyspark/ml/classification.py | 3249 | CODE | |
| LOW | python/pyspark/ml/classification.py | 4112 | CODE | |
| LOW | python/pyspark/ml/classification.py | 4149 | CODE | |
| LOW | python/pyspark/ml/evaluation.py | 579 | CODE | |
| LOW | python/pyspark/ml/evaluation.py | 690 | CODE | |
| LOW | python/pyspark/ml/feature.py | 3625 | CODE | |
| 68 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/packaging/connect/pyspark_connect/__init__.py | 24 | __all__ = [ | CODE |
| LOW | python/pyspark/taskcontext.py | 166 | def _setTaskContext(cls: Type["TaskContext"], taskContext: Optional["TaskContext"]) -> None: | CODE |
| LOW | python/pyspark/conf.py | 18 | __all__ = ["SparkConf"] | CODE |
| LOW | python/pyspark/serializers.py | 71 | __all__ = [ | CODE |
| LOW | python/pyspark/__init__.py | 133 | __all__ = [ | CODE |
| LOW | python/pyspark/memory_profiler_ext.py | 36 | __all__ = [ | CODE |
| LOW | python/pyspark/storagelevel.py | 18 | __all__ = ["StorageLevel"] | CODE |
| LOW | python/pyspark/accumulators.py | 36 | __all__ = ["Accumulator", "AccumulatorParam"] | CODE |
| LOW | python/pyspark/resultiterable.py | 24 | __all__ = ["ResultIterable"] | CODE |
| LOW | python/pyspark/messages/__init__.py | 22 | __all__ = [ | CODE |
| LOW | python/pyspark/core/files.py | 20 | __all__ = ["SparkFiles"] | CODE |
| LOW | python/pyspark/core/rdd.py | 119 | __all__ = ["RDD"] | CODE |
| LOW | python/pyspark/core/context.py | 81 | __all__ = ["SparkContext"] | CODE |
| LOW | python/pyspark/core/broadcast.py | 48 | __all__ = ["Broadcast"] | CODE |
| LOW | python/pyspark/core/status.py | 18 | __all__ = ["SparkJobInfo", "SparkStageInfo", "SparkExecutorInfo", "StatusTracker"] | CODE |
| LOW | python/pyspark/logger/__init__.py | 24 | __all__ = ["PySparkLogger", "SPARK_LOG_SCHEMA"] | CODE |
| LOW | python/pyspark/cloudpickle/__init__.py | 8 | __all__ = [ # noqa | CODE |
| LOW | python/pyspark/mllib/tree.py | 33 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/regression.py | 50 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/clustering.py | 42 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/classification.py | 41 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/evaluation.py | 28 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/__init__.py | 33 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/feature.py | 40 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/random.py | 33 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/recommendation.py | 28 | __all__ = ["MatrixFactorizationModel", "ALS", "Rating"] | CODE |
| LOW | python/pyspark/mllib/fpm.py | 26 | __all__ = ["FPGrowth", "FPGrowthModel", "PrefixSpan", "PrefixSpanModel"] | CODE |
| LOW | python/pyspark/mllib/linalg/__init__.py | 73 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/linalg/distributed.py | 40 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/stat/__init__.py | 27 | __all__ = [ | CODE |
| LOW | python/pyspark/mllib/stat/test.py | 22 | __all__ = ["ChiSqTestResult", "KolmogorovSmirnovTestResult"] | CODE |
| LOW | python/pyspark/mllib/stat/distribution.py | 18 | __all__ = ["MultivariateGaussian"] | CODE |
| LOW | python/pyspark/mllib/stat/_statistics.py | 33 | __all__ = ["MultivariateStatisticalSummary", "Statistics"] | CODE |
| LOW | python/pyspark/pipelines/__init__.py | 27 | __all__ = [ | CODE |
| LOW | python/pyspark/streaming/dstream.py | 50 | __all__ = ["DStream"] | CODE |
| LOW | python/pyspark/streaming/kinesis.py | 25 | __all__ = ["KinesisUtils", "InitialPositionInStream", "MetricsLevel", "utf8_decoder"] | CODE |
| LOW | python/pyspark/streaming/__init__.py | 22 | __all__ = ["StreamingContext", "DStream", "StreamingListener"] | CODE |
| LOW | python/pyspark/streaming/context.py | 30 | __all__ = ["StreamingContext"] | CODE |
| LOW | python/pyspark/streaming/listener.py | 20 | __all__ = ["StreamingListener"] | CODE |
| LOW | python/pyspark/testing/__init__.py | 21 | __all__ = ["assertDataFrameEqual", "assertSchemaEqual", "main"] | CODE |
| LOW | python/pyspark/testing/utils.py | 46 | __all__ = ["assertDataFrameEqual", "assertSchemaEqual"] | CODE |
| LOW | python/pyspark/testing/connectutils.py | 106 | def _set_relation_in_plan(self, plan: pb2.Plan, relation: pb2.Relation) -> None: | CODE |
| LOW | python/pyspark/testing/connectutils.py | 111 | def _set_command_in_plan(self, plan: pb2.Plan, command: pb2.Command) -> None: | CODE |
| LOW | python/pyspark/ml/regression.py | 83 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/clustering.py | 65 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/classification.py | 113 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/evaluation.py | 50 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/__init__.py | 49 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/feature.py | 77 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/recommendation.py | 41 | __all__ = ["ALS", "ALSModel"] | CODE |
| LOW | python/pyspark/ml/tuning.py | 71 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/image.py | 36 | __all__ = ["ImageSchema"] | CODE |
| LOW | python/pyspark/ml/fpm.py | 35 | __all__ = ["FPGrowth", "FPGrowthModel", "PrefixSpan"] | CODE |
| LOW | python/pyspark/ml/linalg/__init__.py | 59 | __all__ = [ | CODE |
| LOW | python/pyspark/ml/torch/distributor.py | 687 | def set_torch_config(context: "BarrierTaskContext") -> None: | STRING |
| LOW | python/pyspark/ml/torch/distributor.py | 704 | def set_gpus(context: "BarrierTaskContext") -> None: | STRING |
| LOW | python/pyspark/ml/torch/distributor.py | 713 | def set_gpus(context: "BarrierTaskContext") -> None: | STRING |
| LOW | python/pyspark/ml/param/__init__.py | 41 | __all__ = ["Param", "Params", "TypeConverters"] | CODE |
| LOW | python/pyspark/ml/connect/__init__.py | 37 | __all__ = [ | CODE |
| LOW | python/pyspark/errors/__init__.py | 57 | __all__ = [ | CODE |
| 59 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …/org/apache/spark/security/FileTokenIngestorSuite.java | 274 | String token = createUnsignedJwt("user@example.com", "https://accounts.google.com"); | CODE |
| LOW | …/org/apache/spark/security/FileTokenIngestorSuite.java | 280 | assertEquals("user@example.com", result.get().getPrincipal()); | CODE |
| LOW | …he/spark/shuffle/streaming/StreamingShuffleSuite.scala | 62 | private val queryId = "dummy-id" | CODE |
| LOW | …spark/deploy/security/UserCredentialManagerSuite.scala | 39 | .set(SECURITY_OIDC_IDENTITY_TOKEN_FILE, "/tmp/fake-token") | CODE |
| LOW | …es/org/apache/spark/ui/static/jquery.dataTables.min.js | 4 | !function(n){"use strict";var a;"function"==typeof define&&define.amd?define(["jquery"],function(t){return n(t,window,do | CODE |
| LOW | python/pyspark/sql/tests/test_stat.py | 317 | dummy_value = 1 | CODE |
| LOW | python/pyspark/sql/tests/test_stat.py | 320 | .replace({"Alice": "Bob"}, dummy_value) | CODE |
| LOW⚡ | …hon/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py | 1122 | .withColumn("name", lit("John Doe")) | CODE |
| LOW | python/pyspark/sql/pandas/functions.py | 109 | >>> df = spark.createDataFrame([("John Doe",)], ("name",)) | STRING |
| LOW | python/pyspark/sql/pandas/functions.py | 124 | >>> df = spark.createDataFrame([("John Doe",)], ("name",)) | STRING |
| LOW | python/pyspark/sql/pandas/functions.py | 506 | >>> df = spark.createDataFrame([("John Doe",)], ("name",)) | STRING |
| LOW | python/pyspark/sql/pandas/functions.py | 518 | >>> df = spark.createDataFrame([("John Doe",)], ("name",)) | STRING |
| LOW | …apache/spark/graphx/lib/ConnectedComponentsSuite.scala | 119 | val defaultUser = ("John Doe", "Missing") | CODE |
| LOW | docs/graphx-programming-guide.md | 193 | val defaultUser = ("John Doe", "Missing") | CODE |
| LOW | docs/graphx-programming-guide.md | 432 | val defaultUser = ("John Doe", "Missing") | CODE |
| LOW | examples/src/main/python/sql/arrow.py | 325 | df = spark.createDataFrame([(1, "John Doe", 21)], ("id", "name", "age")) | CODE |
| LOW | …/spark/security/aws/AwsStsCredentialProviderSuite.java | 97 | private static final String TEST_PRINCIPAL = "user@example.com"; | CODE |
| LOW | …s/test-data/xml-resources/mixed_children_as_string.xml | 4 | Lorem ipsum dolor sit amet. Ut <i>voluptas</i> distinctio et impedit deserunt aut quam fugit et quaerat odit | CODE |
| LOW | …s/test-data/xml-resources/mixed_children_as_string.xml | 4 | Lorem ipsum dolor sit amet. Ut <i>voluptas</i> distinctio et impedit deserunt aut quam fugit et quaerat odit | CODE |
| LOW | …/test/resources/test-data/xml-resources/processing.xml | 4 | lorem ipsum | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 150 | INSERT INTO products VALUES (1, 'Super Widget', 'Electronics', 155.99, 99.99, 1, 'Acme Inc', 'John D.', '123 Main St', 2 | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 150 | INSERT INTO products VALUES (1, 'Super Widget', 'Electronics', 155.99, 99.99, 1, 'Acme Inc', 'John D.', '123 Main St', 2 | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 154 | INSERT INTO customers VALUES (1, 'Alice Johnson', 'alice@example.com', '555-1000', '101 Maple Ave', NULL, 'Springfield', | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 155 | INSERT INTO customers VALUES (2, 'Bob Smith', 'bob@example.com', '555-1002', '202 Oak St', 'Apt 3', 'Oakville', 'CA', '6 | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 156 | INSERT INTO customers VALUES (3, 'Cathy Lee', 'cathy@example.com', '555-1003', '303 Pine Ln', NULL, 'Pineville', 'TX', ' | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 166 | INSERT INTO employees VALUES (1, 'Dan Miller', 'dan@example.com', '555-2001', 'Manager', 'Sales', TIMESTAMP '2018-01-01' | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 167 | INSERT INTO employees VALUES (2, 'Eva Perez', 'eva@example.com', '555-2002', 'Salesperson', 'Sales', TIMESTAMP '2019-03- | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 168 | INSERT INTO employees VALUES (3, 'Frank Wong', 'frank@example.com', '555-2003', 'Warehouse', 'Operations', TIMESTAMP '20 | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 170 | INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 M | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 170 | INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 M | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 170 | INSERT INTO suppliers VALUES (1, 'Acme Inc', 'John D.', 'Sales Manager', 'john@acme.com', '555-3001', '555-3002', '123 M | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 171 | INSERT INTO suppliers VALUES (2, 'Widgets Co', 'Mary K.', 'Customer Success', 'mary@widgets.com', '555-4001', NULL, '456 | CODE |
| LOW⚡ | …-tests/inputs/scripting/randomly_generated_scripts.sql | 172 | INSERT INTO suppliers VALUES (3, 'Toy Supply', 'Ann T.', 'Director', 'ann@toysupply.com', '555-5001', NULL, '789 Oak St' | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 212 | INSERT INTO suppliers VALUES (v_temp_id, 'Temp Supplier', 'Temp Contact', 'Temp Role', 'temp | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 260 | INSERT INTO customers VALUES (v_new_customer_id, 'New Customer', 'new@customer.com', '555-1111', '55 | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 393 | VALUES (sub_emp.employee_id + 9999, v_name_part, CONCAT(v_name_part, '@company.com') | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 405 | VALUES (emp.employee_id + 10000, CONCAT('Emp_', emp.employee_id), emp.employee_name, 'Employ | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 474 | INSERT INTO products VALUES ((SELECT COALESCE(MAX(product_id), 0) + 1 FROM products), 'Rare ' || v_m | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 519 | '555-1212', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 552 | '555-1111', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 553 | '123 Main St', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 727 | '555-7777', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 918 | INSERT INTO employees VALUES (v_new_id, 'New Emp ' || v_new_id, 'new' || v_new_id || '@c | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 930 | INSERT INTO employees VALUES (v_temp_id, 'Manager ' || v_temp_id, 'manager' || v_temp_id || '@compan | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 941 | INSERT INTO employees VALUES (v_low_level_emp + 10, 'Temp Emp ' || v_low_level_emp, 'temp' || v_low_level_em | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 982 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 983 | '123 Main St', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1097 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1169 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1417 | '123 Main St', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1444 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1491 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1528 | '555-0001', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1556 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 1763 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 2072 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 2275 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 2582 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 2615 | '555-0000', | CODE |
| LOW | …-tests/inputs/scripting/randomly_generated_scripts.sql | 2788 | '555-0000', | CODE |
| 45 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/run-tests.py | 516 | # Check if the python executable has coverage installed when 'COVERAGE_PROCESS_START' | COMMENT |
| LOW | python/pyspark/worker.py | 1072 | # Check if this is a continuation of the previous batch's partition | COMMENT |
| LOW | python/pyspark/worker.py | 1181 | # Check if any partition column changed from previous row | COMMENT |
| LOW | python/pyspark/worker.py | 4274 | # Check if the entire column is null: an empty table (no columns) | COMMENT |
| LOW | python/pyspark/worker.py | 4917 | # Check if the entire column is null. | COMMENT |
| LOW | python/pyspark/shell.py | 56 | # Check if th eprogress bar needs to be disabled. | COMMENT |
| LOW⚡ | python/pyspark/pipelines/cli.py | 74 | # Check if it's a simple file path (no wildcards at all) | COMMENT |
| LOW⚡ | python/pyspark/pipelines/cli.py | 78 | # Check if it's a folder path ending with /** | COMMENT |
| LOW | python/pyspark/pandas/frame.py | 12699 | # Check if DataFrame has rows - if yes, raise error; if no, return empty Series | COMMENT |
| LOW | python/pyspark/pandas/frame.py | 12834 | # Check if DataFrame has rows - if yes, raise error; if no, return empty Series | COMMENT |
| LOW | python/pyspark/pandas/data_type_ops/categorical_ops.py | 116 | # Check if categoricals have the same dtype, same categories, and same ordered | COMMENT |
| LOW | python/pyspark/pandas/typedef/typehints.py | 649 | # Check if the name is Tuple. | COMMENT |
| LOW | python/pyspark/pandas/indexes/base.py | 2069 | # Check if the `self` and `other` have different index types. | COMMENT |
| LOW | python/pyspark/sql/metrics.py | 188 | # Add yourself to the list if you have to. | COMMENT |
| LOW | python/pyspark/sql/dataframe.py | 412 | >>> # Check if the DataFrames are equal | STRING |
| LOW | python/pyspark/sql/session.py | 2301 | # Check if the target path already exists | COMMENT |
| LOW | python/pyspark/sql/types.py | 3141 | >>> # Check if numeric values are within the allowed range. | STRING |
| LOW⚡ | python/pyspark/sql/tests/test_utils.py | 1733 | # Check if the error message contains information about 2 mismatches only. | COMMENT |
| LOW | python/pyspark/sql/tests/arrow/test_arrow_map.py | 376 | # Set it to a small odd value to exercise batching logic for all test cases | COMMENT |
| LOW | python/pyspark/sql/tests/pandas/bench_pipelined_udf.py | 92 | # Output results as JSON to stdout | COMMENT |
| LOW | …s/pandas/streaming/test_pandas_transform_with_state.py | 1433 | # Set it to a very small number so that every row would be a separate pandas df | COMMENT |
| LOW | …s/pandas/streaming/test_pandas_transform_with_state.py | 1460 | # Set it to a very large number so that every row would be in the same pandas df | COMMENT |
| LOW | …s/pandas/streaming/test_pandas_transform_with_state.py | 1526 | # Set it to a very small number so that every row would be a separate pandas df | COMMENT |
| LOW⚡ | …/pyspark/sql/tests/pandas/streaming/test_tws_tester.py | 752 | # Set watermark to 15000 - key1's timer should fire. | COMMENT |
| LOW⚡ | …/pyspark/sql/tests/pandas/streaming/test_tws_tester.py | 757 | # Set watermark to 16000 - key2's timer should fire. | COMMENT |
| LOW | …/pyspark/sql/tests/pandas/streaming/test_tws_tester.py | 791 | # Set watermark to 6000. | COMMENT |
| LOW | …/pyspark/sql/tests/pandas/streaming/test_tws_tester.py | 822 | # Set watermark to 20 seconds. | COMMENT |
| LOW | …/pyspark/sql/tests/pandas/streaming/test_tws_tester.py | 924 | # Set watermark to 10000. | COMMENT |
| LOW | python/pyspark/sql/streaming/readwriter.py | 1551 | # Check if the data should be processed | STRING |
| LOW | python/pyspark/sql/worker/plan_data_source_read.py | 155 | # Check if the names are the same as the schema. | COMMENT |
| LOW | python/pyspark/sql/worker/create_data_source.py | 81 | # Check if the provider name matches the data source's name. | COMMENT |
| LOW | python/pyspark/sql/worker/write_into_data_source.py | 97 | # Check if the provider name matches the data source's name. | COMMENT |
| LOW | python/pyspark/sql/connect/session.py | 1172 | # Check if total size exceeds the limit | COMMENT |
| LOW | python/pyspark/sql/connect/session.py | 1182 | # Check if adding this chunk would exceed batch size | COMMENT |
| LOW | python/pyspark/sql/connect/client/artifact.py | 198 | # Check if it is a file from the scheme | COMMENT |
| LOW | python/pyspark/sql/pandas/conversion.py | 866 | # Check if any columns need to be fixed for Spark to infer properly | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 68 | # Check if all arguments have type hints | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 78 | # Check if the return has a type hint | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 227 | # Check if all arguments have type hints | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 237 | # Check if the return has a type hint | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 420 | # Check if all arguments have type hints | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 430 | # Check if the return has a type hint | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 513 | # Check if all arguments have type hints | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 523 | # Check if the return has a type hint | COMMENT |
| LOW | python/pyspark/sql/pandas/typehints.py | 599 | # Check if the name is Tuple first. After that, check the generic types. | COMMENT |
| LOW | sbin/spark-daemon.sh | 50 | # Check if --config is passed as an argument. It is an optional parameter. | COMMENT |
| LOW | sbin/spark-daemon.sh | 154 | # Check if the process has died; in that case we'll tail the log so the user can see | COMMENT |
| LOW | sbin/decommission-worker.sh | 48 | # Check if --block-until-exit is set. | COMMENT |
| LOW | sbin/workers.sh | 57 | # Check if --config is passed as an argument. It is an optional parameter. | COMMENT |
| LOW | …l/src/test/scala/org/apache/spark/repl/ReplSuite.scala | 254 | |# Set everything to be logged to the console | STRING |
| LOW | R/pkg/tests/fulltests/test_jvm_api.R | 26 | # Check if get returns the same element | COMMENT |
| LOW | R/pkg/R/sparkR.R | 456 | # Check if version number of SparkSession matches version number of SparkR package | COMMENT |
| LOW | R/pkg/R/serialize.R | 45 | # Check if all elements are of same type | COMMENT |
| LOW | R/pkg/R/jobj.R | 31 | # Check if jobj was created with the current SparkContext | COMMENT |
| LOW | R/pkg/R/DataFrame.R | 386 | # Check if the column names have . in it | COMMENT |
| LOW | R/pkg/R/DataFrame.R | 2282 | # Check if there is any duplicated column name in the DataFrame | COMMENT |
| LOW | R/pkg/inst/worker/worker.R | 97 | # Set libPaths to include SparkR package as loadNamespace needs this | COMMENT |
| LOW | .github/workflows/build_and_test.yml | 1150 | # Print the values of environment variables `SKIP_ERRORDOC`, `SKIP_SCALADOC`, `SKIP_PYTHONDOC`, `SKIP_RDOC` and | COMMENT |
| LOW | dev/create-release/generate-llms-txt.py | 188 | # Check if docs directory exists | COMMENT |
| LOW | dev/create-release/release-build.sh | 1045 | # Check if any failures were recorded | COMMENT |
| 1 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …/main/java/org/apache/spark/SparkFirehoseListener.java | 27 | * This is a concrete Java class in order to ensure that we don't forget to update it when adding | COMMENT |
| MEDIUM | …/org/apache/spark/storage/BlockReplicationPolicy.scala | 101 | * Method to prioritize a bunch of candidate peers of a block. This is a basic implementation, | COMMENT |
| LOW | python/packaging/classic/setup.py | 170 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/classic/setup.py | 170 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/classic/setup.py | 362 | # Don't forget to update python/docs/source/getting_started/install.rst | COMMENT |
| LOW | python/packaging/connect/setup.py | 84 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/connect/setup.py | 84 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/connect/setup.py | 115 | # Don't forget to update python/docs/source/getting_started/install.rst | COMMENT |
| LOW | python/packaging/client/setup.py | 132 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/client/setup.py | 132 | # Also don't forget to update python/docs/source/getting_started/install.rst, | COMMENT |
| LOW | python/packaging/client/setup.py | 212 | # Don't forget to update python/docs/source/getting_started/install.rst | COMMENT |
| LOW | python/pyspark/pandas/config.py | 114 | # NOTE: if you are fixing or adding an option here, make sure you execute `show_options()` and | COMMENT |
| LOW | dev/create-release/release-build.sh | 832 | # NOTE: Don't forget to update the valid combinations of distributions at | COMMENT |
| LOW | …/main/scala/org/apache/spark/sql/connect/Dataset.scala | 146 | // Make sure we don't forget to set plan id. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/pyspark/pandas/groupby.py | 2194 | # TODO: implement 'dropna' parameter | COMMENT |
| LOW | …he/spark/ml/classification/JavaGBTClassifierSuite.java | 71 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …ml/classification/JavaDecisionTreeClassifierSuite.java | 66 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …ml/classification/JavaRandomForestClassifierSuite.java | 90 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …park/ml/regression/JavaDecisionTreeRegressorSuite.java | 68 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …park/ml/regression/JavaRandomForestRegressorSuite.java | 92 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …/apache/spark/ml/regression/JavaGBTRegressorSuite.java | 72 | // TODO: Add test once save/load are implemented. SPARK-6725 | COMMENT |
| LOW | …/scala/org/apache/spark/ml/tree/impl/BaggedPoint.scala | 66 | // TODO: implement weighted bootstrapping | COMMENT |
| LOW | …rk/sql/execution/datasources/v2/FileDataSourceV2.scala | 101 | // TODO: implement a light-weight partition inference which only looks at the path of one leaf | COMMENT |
| LOW | …yst/expressions/aggregate/datasketchesAggregates.scala | 159 | // TODO: implement support for decimal/datetime/interval types | STRING |
| LOW | …che/spark/sql/hive/execution/InsertIntoHiveTable.scala | 121 | // TODO: implement hive compatibility as rules. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | python/pyspark/sql/tests/connect/test_connect_basic.py | 1663 | # In this example, the max chunk size is set to a small value, so each Arrow | COMMENT |
| HIGH | …cala/org/apache/spark/sql/connector/catalog/txns.scala | 191 | // This is where the table pinning logic should occur. In this implementation, a tables is loaded | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | …la/org/apache/spark/sql/FileBasedDataSourceSuite.scala | 1472 | // Embed the nanos leaf inside a struct, an array, and a map value. The guardrails | COMMENT |
| MEDIUM | …pache/spark/sql/pipelines/graph/SqlPipelineSuite.scala | 1353 | // MERGE API, which requires a row-level-operation-capable v2 catalog, so these tests scaffold | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | python/pyspark/testing/sqlutils.py | 115 | Read the classpath file for a project and return it as a comma-separated string. The classpath file is typical | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | python/pyspark/ml/dl_util.py | 103 | the empty string, nothing will be written after the auto-generated code. | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | python/pyspark/sql/tests/df_golden/regenerate.sh | 21 | # Usage: | COMMENT |