spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Bolshakov <bolshakov.de...@gmail.com>
Subject Re: Append more files to existing partitioned data
Date Sat, 17 Mar 2018 13:15:58 GMT
Hello Serega,

https://spark.apache.org/docs/latest/sql-programming-guide.html

Please try SaveMode.Append option. Does it work for you?


сб, 17 мар. 2018 г., 15:19 Serega Sheypak <serega.sheypak@gmail.com>:

> Hi, I', using spark-sql to process my data and store result as parquet
> partitioned by several columns
>
> ds.write
>   .partitionBy("year", "month", "day", "hour", "workflowId")
>   .parquet("/here/is/my/dir")
>
>
> I want to run more jobs that will produce new partitions or add more files
> to existing partitions.
> What is the right way to do it?
>

Mime
View raw message