spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Lee <>
Subject Re: Yay for 1.0.0! EC2 Still has problems.
Date Wed, 04 Jun 2014 13:35:45 GMT
On Wed, Jun 4, 2014 at 12:31 PM, Matei Zaharia <>

> Ah, sorry to hear you had more problems. Some thoughts on them:

There will always be more problems, 'tis the nature of coding. :-) I try
not to bother the list until I've smacked my head against them for a few
hours, so it's only the "most confusing" stuff I pour out here. I'm
actually progressing pretty well.

> (the streaming.Twitter ones especially) depend on there being a
> "/mnt/spark" and "/mnt2/spark" directory (I think for java tempfiles?) and
> those don't seem to exist out-of-the-box.
> I think this is a side-effect of the r3 instances not having those drives
> mounted. Our setup script would normally create these directories. What was
> the error?
Oh, I went back to m1.large while those issues get sorted out. I decided I
had enough problems without messing with that too. (seriously, why does
Amazon do these things? It's like they _try_ to make the instances

I forget the exact error, but it traced through createTempFile and it was
fairly clear about the directory being missing. Things like
"bin/run-example SparkPi" worked fine, but I'll bet twitter4j creates temp
files, so "bin/run-example streaming.TwitterPopularTags" broke.

What did you change to? It should be changed to say
> log4j.rootCategory=WARN, console but maybe another is
> somehow arriving on the classpath. This is definitely a common problem so
> we need to add some explicit docs on it.

I seem to have this sorted out, don't ask me how. Once again I was probably
editing things on the cluster master when I should have been editing the
cluster controller, or vice versa. But, yeah, many of the examples just get
lost in a sea of DAG INFO messages.

> Are you going through
> You should be able to do just sbt package. Once you do that you don’t need
> to deploy your application’s JAR to the cluster, just pass it to
> spark-submit and it will automatically be sent over.

Ah, that answers another question I just asked elsewhere... Yup, I re-read
pretty much every documentation page daily. And I'm making my way through
every video.

> > Meanwhile I'm learning scala... Great Turing's Ghost, it's the dream
> language we've theorized about for years! I hadn't realized!
> Indeed, glad you’re enjoying it.

"Enjoying", not yet alas, I'm sure I'll get there. But I do understand the
implications of a mixed functional-imperative language with closures and
lambdas. That is serious voodoo.

Jeremy Lee  BCompSci(Hons)
  The Unorthodox Engineers

View raw message