spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serega Sheypak <serega.shey...@gmail.com>
Subject Append more files to existing partitioned data
Date Sat, 17 Mar 2018 12:18:52 GMT
Hi, I', using spark-sql to process my data and store result as parquet
partitioned by several columns

ds.write
  .partitionBy("year", "month", "day", "hour", "workflowId")
  .parquet("/here/is/my/dir")


I want to run more jobs that will produce new partitions or add more files
to existing partitions.
What is the right way to do it?

Mime
View raw message