spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "NN Murthy" <>
Subject RE: update of RDDs
Date Thu, 01 May 2014 13:01:31 GMT
Thanks a lot for very prompt response. Then next questions are the following.

1.       Can we conclude that Spark is NOT the solution for our requirement? Or

2.       Is there a design approach to meet such requirements using Spark? 


From: Mayur Rustagi [] 
Sent: 01 May 2014 18:22
Subject: Re: update of RDDs


RDD are immutable so cannot be updated. You can create new RDD containing updated entries(often
not what you want to do). 



Mayur Rustagi
Ph: +1 (760) 203 3257

@mayur_rustagi <> 



On Thu, May 1, 2014 at 4:42 AM, narayanabhatla NarasimhaMurthy <>

In our application, we need distributed RDDs containing key-value maps. We
have operations that update RDDs by way of adding entries to the map, delete
entries from the map as well as update value part of maps.
We also have map reduce functions that operate on the RDDs.The questions are
the following.
1. Can RDDs be updated? if Yes, what rae the methods?
2. If we update RDDs, will it happen in place or does it create new RDDs
with almost double the original RDD size (original+newly created RDD)?
Thank you very much.

View this message in context:
Sent from the Apache Spark User List mailing list archive at




The information contained in this e-mail message and/or attachments to it may
contain confidential or privileged information. If you are not the intended
recipient, any dissemination, use, review, distribution, printing or copying
of the information contained in this e-mail message and/or attachments to it
are strictly prohibited. If you have received this communication in error,
please notify us by reply e-mail or directly to or
telephone and immediately and permanently delete the message and any
attachments. Thank you.


This email has been scrubbed for your protection by SecureMX.
For more information visit

View raw message