spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshat Aranya <aara...@gmail.com>
Subject Re: Stateful mapPartitions
Date Fri, 05 Dec 2014 00:31:03 GMT
I want to have a database connection per partition of the RDD, and then
reuse that connection whenever mapPartitions is called, which results in
compute being called on the partition.

On Thu, Dec 4, 2014 at 11:07 AM, Paolo Platter <paolo.platter@agilelab.it>
wrote:

>  Could you provide some further details ?
> What do you nerd to do with db cpnnection?
>
> Paolo
>
> Inviata dal mio Windows Phone
>  ------------------------------
> Da: Akshat Aranya <aaranya@gmail.com>
> Inviato: ‎04/‎12/‎2014 18:57
> A: user@spark.apache.org
> Oggetto: Stateful mapPartitions
>
>  Is it possible to have some state across multiple calls to mapPartitions
> on each partition, for instance, if I want to keep a database connection
> open?
>

Mime
View raw message