Spark Connect server - Details for Job 53

Status: SUCCEEDED
Submitted: 2026/09/01 08:56:18
Duration: 0.9 s
Completed Stages: 1

Enable zooming

digraph G { id="graph_66"; subgraph graph_stage_66 { id="graph_stage_66"; isCluster="true"; label="Stage 66"; 194 [id="node_194" labelType="html" label="JsonFile: file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_054922.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_060422.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_083426.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_004914.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_023417.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_021917.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_013415.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_011915.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_030418.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_041919.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_221911.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_081926.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_071924.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_051921.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_073424.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_223411.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_024917.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_033418.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_074925.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_040419.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_061922.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_231912.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_063423.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_224911.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_233412.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_070424.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_084926.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_044920.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_020416.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_064923.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_214910.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_230412.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_000413.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_053421.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_050421.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_014916.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_003414.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_010415.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_034919.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_031918.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_001914.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_220410.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260831_234913.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_043420.json,file:/home/jovyan/data/openaq-api-data/bidhannagar_20260901_080425.json [194]<br>Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd"\nplan {\n root {\n common {\n plan_id: 1\n }\n show_string {\n ...}"]; subgraph cluster_268 { id="cluster_268"; isCluster="true"; label="map"; 195 [id="node_195" labelType="html" label="MapPartitionsRDD [195]<br>Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd"\nplan {\n root {\n common {\n plan_id: 1\n }\n show_string {\n ...}"]; } subgraph cluster_269 { id="cluster_269"; isCluster="true"; label="mapPartitions"; 196 [id="node_196" labelType="html" label="MapPartitionsRDD [196]<br>Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd"\nplan {\n root {\n common {\n plan_id: 1\n }\n show_string {\n ...}"]; } } 194->195; 195->196; }

Completed Stages (1)

Page:

Stage Id ▾	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Output	Shuffle Read	Shuffle Write
66	Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd" plan { root { common { plan_id: 1 } show_string { ... Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd" plan { root { common { plan_id: 1 } show_string { ... +details session_id: "20940173-f487-4e25-8f98-6f39178312cd" plan { root { common { plan_id: 1 } show_string { input { common { plan_id: 0 } read { data_source { format: "json" schema: "" options { key: "multiLine" value: "true" } paths: "/home/jovyan/data/openaq-api-data/" } } } num_rows: 20 truncate: 20 } } } client_type: "_SPARK_CONNECT_PYTHON spark/4.1.0 os/linux python/3.13.11" request_options { result_chunking_options { allow_arrow_batch_chunking: true } } request_options { reattach_options { reattachable: true } } operation_id: "2eb0b6b5-8833-4258-8ae2-10a9e7e4e610" client_observed_server_side_session_id: "874f7c59-4413-4e4f-b2ce-16cea30ebf91"	2026/09/01 08:56:18	0.8 s	1/1	347.9 KiB

Stage Id ▾

Description

Submitted

Duration

Tasks: Succeeded/Total

Input

Output

Shuffle Read

Shuffle Write

Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd" plan { root { common { plan_id: 1 } show_string { ... Spark Connect - session_id: "20940173-f487-4e25-8f98-6f39178312cd" plan { root { common { plan_id: 1 } show_string { ... +details

session_id: "20940173-f487-4e25-8f98-6f39178312cd"
plan {
  root {
    common {
      plan_id: 1
    }
    show_string {
      input {
        common {
          plan_id: 0
        }
        read {
          data_source {
            format: "json"
            schema: ""
            options {
              key: "multiLine"
              value: "true"
            }
            paths: "/home/jovyan/data/openaq-api-data/"
          }
        }
      }
      num_rows: 20
      truncate: 20
    }
  }
}
client_type: "_SPARK_CONNECT_PYTHON spark/4.1.0 os/linux python/3.13.11"
request_options {
  result_chunking_options {
    allow_arrow_batch_chunking: true
  }
}
request_options {
  reattach_options {
    reattachable: true
  }
}
operation_id: "2eb0b6b5-8833-4258-8ae2-10a9e7e4e610"
client_observed_server_side_session_id: "874f7c59-4413-4e4f-b2ce-16cea30ebf91"

2026/09/01 08:56:18

0.8 s

1/1

347.9 KiB

Page: