spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: commit DB Transaction for each partition
Date Sun, 30 Aug 2015 03:35:43 GMT
What problem are you having? you will have to trigger an action at the end
to execute this piece of code. Like:

rdd.mapPartitions(partitionOfRecords => {

DBConnectionInit()

val results = partitionOfRecords.map(......)

DBConnection.commit()

results

})*.count()*



Thanks
Best Regards

On Thu, Aug 27, 2015 at 7:32 PM, Ahmed Nawar <ahmed.nawar@gmail.com> wrote:

> Dears,
>
>     I needs to commit DB Transaction for each partition,Not for each row.
> below didn't work for me.
>
>
> rdd.mapPartitions(partitionOfRecords => {
>
> DBConnectionInit()
>
> val results = partitionOfRecords.map(......)
>
> DBConnection.commit()
>
> results
>
> })
>
>
>
> Best regards,
>
> Ahmed Atef Nawwar
>
> Data Management & Big Data Consultant
>

Mime
View raw message