spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rok Roskar <rokros...@gmail.com>
Subject Re: java.util.NoSuchElementException: key not found:
Date Mon, 02 Mar 2015 15:34:42 GMT
aha ok, thanks.

If I create different RDDs from a parent RDD and force evaluation
thread-by-thread, then it should presumably be fine, correct? Or do I need
to checkpoint the child RDDs as a precaution in case it needs to be removed
from memory and recomputed?

On Sat, Feb 28, 2015 at 4:28 AM, Shixiong Zhu <zsxwing@gmail.com> wrote:

> RDD is not thread-safe. You should not use it in multiple threads.
>
> Best Regards,
> Shixiong Zhu
>
> 2015-02-27 23:14 GMT+08:00 rok <rokroskar@gmail.com>:
>
>> I'm seeing this java.util.NoSuchElementException: key not found: exception
>> pop up sometimes when I run operations on an RDD from multiple threads in
>> a
>> python application. It ends up shutting down the SparkContext so I'm
>> assuming this is a bug -- from what I understand, I should be able to run
>> operations on the same RDD from multiple threads or is this not
>> recommended?
>>
>> I can't reproduce it all the time and I've tried eliminating caching
>> wherever possible to see if that would have an effect, but it doesn't seem
>> to. Each thread first splits the base RDD and then runs the
>> LogisticRegressionWithSGD on the subset.
>>
>> Is there a workaround to this exception?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/java-util-NoSuchElementException-key-not-found-tp21848.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message