spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: PySpark + executor lost
Date Fri, 08 Aug 2014 20:24:44 GMT
Hi Avishek,

As of Spark 1.0, PySpark does in fact run on YARN.

-Sandy


On Fri, Aug 8, 2014 at 12:47 PM, Avishek Saha <avishek.saha@gmail.com>
wrote:

> So I think I have a better idea of the problem now.
>
> The environment is YARN client and IIRC PySpark doesn't run on YARN
> cluster.
>
> So my client is heavily loaded which causes iy loose a lot of e executors
> which might be part of the problem.
>
> Btw any plans in supporting PySpark in YARN clusters mode?
> On Aug 7, 2014 3:04 PM, "Davies Liu" <davies@databricks.com> wrote:
>
>> What is the environment ? YARN or Mesos or Standalone?
>>
>> It will be more helpful if you could show more loggings.
>>
>> On Wed, Aug 6, 2014 at 7:25 PM, Avishek Saha <avishek.saha@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I get a lot of executor lost error for "saveAsTextFile" with PySpark
>> > and Hadoop 2.4.
>> >
>> > For small datasets this error occurs but since the dataset is small it
>> > gets eventually written to the file.
>> > For large datasets, it takes forever to write the final output.
>> >
>> > Any help is appreciated.
>> > Avishek
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>

Mime
View raw message