spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan <delta1...@gmail.com>
Subject Re: SocketException when reading from S3 (s3n format)
Date Wed, 04 Jun 2014 16:46:16 GMT
Thanks Nicholas.

The obvious fix for this issue, in my case, was to cache the input since
it's only 35 megabytes.

Dan


On Wed, Jun 4, 2014 at 7:34 AM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> I think by default a thread can die up to 4 times before Spark considers
> it a failure. Are you seeing that happen? I believe that is a configurable
> thing, but don't know off the top of my head how to change it.
>
> I've seen this error before when reading data from a large amount of files
> on S3, and it is typically harmless. Spark just retries the operation and
> proceeds normally.
>
>
> On Wed, Jun 4, 2014 at 4:05 AM, yuzeh <delta1080@gmail.com> wrote:
>
>> I should add that I'm using spark 0.9.1.
>>
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/SocketException-when-reading-from-S3-s3n-format-tp6889p6890.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>

Mime
View raw message