spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: spark-shell working in scala-2.11 (breaking change?)
Date Sat, 31 Jan 2015 16:18:49 GMT
Looking at https://github.com/apache/spark/pull/1222/files , the following
change may have caused what Stephen described:

+ if (!fileSystem.isDirectory(new Path(logBaseDir))) {

When there is no schema associated with logBaseDir, local path should be
assumed.

On Fri, Jan 30, 2015 at 8:37 AM, Stephen Haberman <
stephen.haberman@gmail.com> wrote:

> Hi Krishna/all,
>
> I think I found it, and it wasn't related to Scala-2.11...
>
> I had "spark.eventLog.dir=/mnt/spark/work/history", which worked
> in Spark 1.2, but now am running Spark master, and it wants a
> Hadoop URI, e.g. file:///mnt/spark/work/history (I believe due to
> commit 45645191).
>
> This looks like a breaking change to the spark.eventLog.dir
> config property.
>
> Perhaps it should be patched to convert the previously supported
> "just a file path" values to HDFS-compatible "file://..." URIs
> for backwards compatibility?
>
> - Stephen
>
>
> On Wed, 28 Jan 2015 12:27:17 -0800
> Krishna Sankar <ksankar42@gmail.com> wrote:
>
> > Stephen,
> >    Scala 2.11 worked fine for me. Did the dev change and then
> > compile. Not using in production, but I go back and forth
> > between 2.10 & 2.11. Cheers
> > <k/>
> >
> > On Wed, Jan 28, 2015 at 12:18 PM, Stephen Haberman <
> > stephen.haberman@gmail.com> wrote:
> >
> > > Hey,
> > >
> > > I recently compiled Spark master against scala-2.11 (by
> > > running the dev/change-versions script), but when I run
> > > spark-shell, it looks like the "sc" variable is missing.
> > >
> > > Is this a known/unknown issue? Are others successfully using
> > > Spark with scala-2.11, and specifically spark-shell?
> > >
> > > It is possible I did something dumb while compiling master,
> > > but I'm not sure what it would be.
> > >
> > > Thanks,
> > > Stephen
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > > For additional commands, e-mail: user-help@spark.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message