spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From purna pradeep <purna2prad...@gmail.com>
Subject spark partitionBy with partitioned column in json output
Date Mon, 04 Jun 2018 23:59:32 GMT
im reading below json in spark

    {"bucket": "B01", "actionType": "A1", "preaction": "NULL",
"postaction": "NULL"}
    {"bucket": "B02", "actionType": "A2", "preaction": "NULL",
"postaction": "NULL"}
    {"bucket": "B03", "actionType": "A3", "preaction": "NULL",
"postaction": "NULL"}

    val df=spark.read.json("actions.json").toDF()

Now im writing the same to a json output as below

    df.write. format("json"). mode("append").
partitionBy("bucket","actionType"). save("output.json")


and the output.json is as below

    {"preaction":"NULL","postaction":"NULL"}

bucket,actionType columns are missing in the json output, i need
partitionby columns as well in the output

Mime
View raw message