not sure if it is your case, but if the source data is heavy and deeply nested I'd recommend explicitly providing the schema when reading the json.

df = spark.read.schema(schema).json(updated_dataset)

On Thu, 21 Jan 2021 at 04:15, srinivasarao daruna <sree.srinu38@gmail.com> wrote:
I am running a spark job on a huge dataset. I have allocated 10 R5.16xlarge machines. (each consists 64cores, 512G).

The source data is json and i need to do some json transformations. So, i read them as text and then convert to a dataframe.

ds = spark.read.textFile()
updated_dataset = ds.withColumn(applying my transformations).as[String]
df = spark.read.json(updated_dataset)


Some notes:
The source data is heavy and deeply nested. The printSchema contains a lot of nested structs.

in the spark ui, json stage is first and after that is completed, it is not showing any jobs in the UI and it's just hanging there.

All executors were dead and only the driver was active.

Thank You,