spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: SparkSQL API to insert DataFrame into a static partition?
Date Wed, 02 Dec 2015 03:57:30 GMT
I don't think there's api for that, but think it is reasonable and helpful
for ETL.

As a workaround you can first register your dataframe as temp table, and
use sql to insert to the static partition.

On Wed, Dec 2, 2015 at 10:50 AM, Isabelle Phan <nliphan@gmail.com> wrote:

> Hello,
>
> Is there any API to insert data into a single partition of a table?
>
> Let's say I have a table with 2 columns (col_a, col_b) and a partition by
> date.
> After doing some computation for a specific date, I have a DataFrame with
> 2 columns (col_a, col_b) which I would like to insert into a specific date
> partition. What is the best way to achieve this?
>
> It seems that if I add a date column to my DataFrame, and turn on dynamic
> partitioning, I can do:
> df.write.partitionBy("date").insertInto("my_table")
> But it seems overkill to use dynamic partitioning function for such a case.
>
>
> Thanks for any pointers!
>
> Isabelle
>
>
>


-- 
Best Regards

Jeff Zhang

Mime
View raw message