spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avishek Saha <avishek.s...@gmail.com>
Subject Re: PySpark + executor lost
Date Fri, 08 Aug 2014 20:28:26 GMT
You mean YARN cluster, right?

Also, my jobs runs thru all their stages just fine. But the entire
code crashes when I do a "saveAsTextFile".

On 8 August 2014 13:24, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
> Hi Avishek,
>
> As of Spark 1.0, PySpark does in fact run on YARN.
>
> -Sandy
>
>
> On Fri, Aug 8, 2014 at 12:47 PM, Avishek Saha <avishek.saha@gmail.com>
> wrote:
>>
>> So I think I have a better idea of the problem now.
>>
>> The environment is YARN client and IIRC PySpark doesn't run on YARN
>> cluster.
>>
>> So my client is heavily loaded which causes iy loose a lot of e executors
>> which might be part of the problem.
>>
>> Btw any plans in supporting PySpark in YARN clusters mode?
>>
>> On Aug 7, 2014 3:04 PM, "Davies Liu" <davies@databricks.com> wrote:
>>>
>>> What is the environment ? YARN or Mesos or Standalone?
>>>
>>> It will be more helpful if you could show more loggings.
>>>
>>> On Wed, Aug 6, 2014 at 7:25 PM, Avishek Saha <avishek.saha@gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > I get a lot of executor lost error for "saveAsTextFile" with PySpark
>>> > and Hadoop 2.4.
>>> >
>>> > For small datasets this error occurs but since the dataset is small it
>>> > gets eventually written to the file.
>>> > For large datasets, it takes forever to write the final output.
>>> >
>>> > Any help is appreciated.
>>> > Avishek
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: user-help@spark.apache.org
>>> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message