spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: PySpark + executor lost
Date Fri, 08 Aug 2014 22:48:44 GMT
What exactly do you mean by "YARN cluster".  Do you mean running Spark
against a YARN cluster in general, or particularly in "yarn-cluster" mode,
where the driver runs inside a Spark application master?

Also, what error are you seeing in your executors?

-Sandy


On Fri, Aug 8, 2014 at 2:00 PM, Avishek Saha <avishek.saha@gmail.com> wrote:

> Btw, I get this for Spark-1.0.2
> I guess YARN cluster is still not supported for PySpark.
>
>
> -------------------------------------------------------------------------------------
> Error: Cluster deploy mode is currently not supported for python.
> Run with --help for usage help or --verbose for debug output
>
> On 8 August 2014 13:28, Avishek Saha <avishek.saha@gmail.com> wrote:
> > You mean YARN cluster, right?
> >
> > Also, my jobs runs thru all their stages just fine. But the entire
> > code crashes when I do a "saveAsTextFile".
> >
> > On 8 August 2014 13:24, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
> >> Hi Avishek,
> >>
> >> As of Spark 1.0, PySpark does in fact run on YARN.
> >>
> >> -Sandy
> >>
> >>
> >> On Fri, Aug 8, 2014 at 12:47 PM, Avishek Saha <avishek.saha@gmail.com>
> >> wrote:
> >>>
> >>> So I think I have a better idea of the problem now.
> >>>
> >>> The environment is YARN client and IIRC PySpark doesn't run on YARN
> >>> cluster.
> >>>
> >>> So my client is heavily loaded which causes iy loose a lot of e
> executors
> >>> which might be part of the problem.
> >>>
> >>> Btw any plans in supporting PySpark in YARN clusters mode?
> >>>
> >>> On Aug 7, 2014 3:04 PM, "Davies Liu" <davies@databricks.com> wrote:
> >>>>
> >>>> What is the environment ? YARN or Mesos or Standalone?
> >>>>
> >>>> It will be more helpful if you could show more loggings.
> >>>>
> >>>> On Wed, Aug 6, 2014 at 7:25 PM, Avishek Saha <avishek.saha@gmail.com>
> >>>> wrote:
> >>>> > Hi,
> >>>> >
> >>>> > I get a lot of executor lost error for "saveAsTextFile" with PySpark
> >>>> > and Hadoop 2.4.
> >>>> >
> >>>> > For small datasets this error occurs but since the dataset is small
> it
> >>>> > gets eventually written to the file.
> >>>> > For large datasets, it takes forever to write the final output.
> >>>> >
> >>>> > Any help is appreciated.
> >>>> > Avishek
> >>>> >
> >>>> >
> ---------------------------------------------------------------------
> >>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >>>> > For additional commands, e-mail: user-help@spark.apache.org
> >>>> >
> >>
> >>
>

Mime
View raw message