Spark Connect server - Details for Query 58

Submitted Time: 2026/09/03 05:48:02
Duration: 0.9 s
Succeeded Jobs: 85 86

Download

Plan Visualization

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [id="node0" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; subgraph cluster1 { isCluster="true"; id="cluster1"; label="WholeStageCodegen (2)\n \nduration: 10 ms"; tooltip="WholeStageCodegen (2)"; 2 [id="node2" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 1 ms peak memory: 256.0 KiB number of output rows: 10 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[passenger_count#5977L], functions=[])"]; } 3 [id="node3" labelType="html" label="AQEShuffleRead number of partitions: 1 number of empty partitions: 190 partition data size: 600.0 B number of coalesced partitions: 1" tooltip="AQEShuffleRead coalesced"]; 4 [id="node4" labelType="html" label="Exchange shuffle records written: 10 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 10 local bytes read: 587.0 B merged fetch fallback count: 0 local blocks read: 10 remote merged chunks fetched: 0 remote blocks read: 0 data size: 160.0 B local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time: 5 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time: 0 ms remote bytes read: 0.0 B number of partitions: 200 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 587.0 B" tooltip="Exchange hashpartitioning(passenger_count#5977L, 200), ENSURE_REQUIREMENTS, [plan_id=1512]"]; subgraph cluster5 { isCluster="true"; id="cluster5"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n747 ms (122 ms, 625 ms, 625 ms (stage 119.0: task 675))"; tooltip="WholeStageCodegen (1)"; 6 [id="node6" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 586 ms (46 ms, 540 ms, 540 ms (stage 119.0: task 675)) peak memory total (min, med, max (stageId: taskId)) 512.0 KiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 119.0: task 676)) number of output rows: 10 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[passenger_count#5977L], functions=[])"]; 7 [id="node7" labelType="html" label="ColumnarToRow number of output rows: 8,714,667 number of input batches: 2,128" tooltip="ColumnarToRow"]; } 8 [id="node8" labelType="html" label="Scan parquet number of files read: 1 scan time total (min, med, max (stageId: taskId)) 136 ms (46 ms, 90 ms, 90 ms (stage 119.0: task 675)) metadata time: 0 ms size of files read: 118.4 MiB number of output rows: 8,714,667" tooltip="FileScan parquet [passenger_count#5977L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/jovyan/data/yellow-taxi/yellow_tripdata_2018-06.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<passenger_count:bigint>"]; 2->0; 3->2; 4->3; 6->4; 7->6; 8->7; }

Plan Details

== Physical Plan ==
AdaptiveSparkPlan (13)
+- == Final Plan ==
   ResultQueryStage (8), Statistics(sizeInBytes=8.0 EiB)
   +- * HashAggregate (7)
      +- AQEShuffleRead (6)
         +- ShuffleQueryStage (5), Statistics(sizeInBytes=160.0 B, rowCount=10)
            +- Exchange (4)
               +- * HashAggregate (3)
                  +- * ColumnarToRow (2)
                     +- Scan parquet  (1)
+- == Initial Plan ==
   CollectLimit (12)
   +- HashAggregate (11)
      +- Exchange (10)
         +- HashAggregate (9)
            +- Scan parquet  (1)


(1) Scan parquet 
Output [1]: [passenger_count#5977L]
Batched: true
Location: InMemoryFileIndex [file:/home/jovyan/data/yellow-taxi/yellow_tripdata_2018-06.parquet]
ReadSchema: struct<passenger_count:bigint>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [passenger_count#5977L]

(3) HashAggregate [codegen id : 1]
Input [1]: [passenger_count#5977L]
Keys [1]: [passenger_count#5977L]
Functions: []
Aggregate Attributes: []
Results [1]: [passenger_count#5977L]

(4) Exchange
Input [1]: [passenger_count#5977L]
Arguments: hashpartitioning(passenger_count#5977L, 200), ENSURE_REQUIREMENTS, [plan_id=1512]

(5) ShuffleQueryStage
Output [1]: [passenger_count#5977L]
Arguments: 0

(6) AQEShuffleRead
Input [1]: [passenger_count#5977L]
Arguments: coalesced

(7) HashAggregate [codegen id : 2]
Input [1]: [passenger_count#5977L]
Keys [1]: [passenger_count#5977L]
Functions: []
Aggregate Attributes: []
Results [1]: [passenger_count#5977L]

(8) ResultQueryStage
Output [1]: [passenger_count#5977L]
Arguments: 1

(9) HashAggregate
Input [1]: [passenger_count#5977L]
Keys [1]: [passenger_count#5977L]
Functions: []
Aggregate Attributes: []
Results [1]: [passenger_count#5977L]

(10) Exchange
Input [1]: [passenger_count#5977L]
Arguments: hashpartitioning(passenger_count#5977L, 200), ENSURE_REQUIREMENTS, [plan_id=1497]

(11) HashAggregate
Input [1]: [passenger_count#5977L]
Keys [1]: [passenger_count#5977L]
Functions: []
Aggregate Attributes: []
Results [1]: [passenger_count#5977L]

(12) CollectLimit
Input [1]: [passenger_count#5977L]
Arguments: 1000

(13) AdaptiveSparkPlan
Output [1]: [passenger_count#5977L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true