spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmed Nawar <ahmed.na...@gmail.com>
Subject Re: commit DB Transaction for each partition
Date Sun, 30 Aug 2015 06:49:15 GMT
Dear Akhil,

   Yes, But any thing like count() will make me lose the RDD return
from mapPartitions.
and i need it.

   Cody supported me by a hint, adding empty foreach {} for the returned
RDD.

   If you have better idea please share with me.


   Thanks for your support.




On Sun, Aug 30, 2015 at 6:35 AM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> What problem are you having? you will have to trigger an action at the end
> to execute this piece of code. Like:
>
> rdd.mapPartitions(partitionOfRecords => {
>
> DBConnectionInit()
>
> val results = partitionOfRecords.map(......)
>
> DBConnection.commit()
>
> results
>
> })*.count()*
>
>
>
> Thanks
> Best Regards
>
> On Thu, Aug 27, 2015 at 7:32 PM, Ahmed Nawar <ahmed.nawar@gmail.com>
> wrote:
>
>> Dears,
>>
>>     I needs to commit DB Transaction for each partition,Not for each row.
>> below didn't work for me.
>>
>>
>> rdd.mapPartitions(partitionOfRecords => {
>>
>> DBConnectionInit()
>>
>> val results = partitionOfRecords.map(......)
>>
>> DBConnection.commit()
>>
>> results
>>
>> })
>>
>>
>>
>> Best regards,
>>
>> Ahmed Atef Nawwar
>>
>> Data Management & Big Data Consultant
>>
>
>

Mime
View raw message