spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Lee <unorthodox.engine...@gmail.com>
Subject Re: Yay for 1.0.0! EC2 Still has problems.
Date Wed, 04 Jun 2014 13:35:45 GMT
On Wed, Jun 4, 2014 at 12:31 PM, Matei Zaharia <matei.zaharia@gmail.com>
wrote:

> Ah, sorry to hear you had more problems. Some thoughts on them:
>

There will always be more problems, 'tis the nature of coding. :-) I try
not to bother the list until I've smacked my head against them for a few
hours, so it's only the "most confusing" stuff I pour out here. I'm
actually progressing pretty well.


> (the streaming.Twitter ones especially) depend on there being a
> "/mnt/spark" and "/mnt2/spark" directory (I think for java tempfiles?) and
> those don't seem to exist out-of-the-box.
>
> I think this is a side-effect of the r3 instances not having those drives
> mounted. Our setup script would normally create these directories. What was
> the error?
>
>
Oh, I went back to m1.large while those issues get sorted out. I decided I
had enough problems without messing with that too. (seriously, why does
Amazon do these things? It's like they _try_ to make the instances
incompatible.)

I forget the exact error, but it traced through createTempFile and it was
fairly clear about the directory being missing. Things like
"bin/run-example SparkPi" worked fine, but I'll bet twitter4j creates temp
files, so "bin/run-example streaming.TwitterPopularTags" broke.

What did you change log4j.properties to? It should be changed to say
> log4j.rootCategory=WARN, console but maybe another log4j.properties is
> somehow arriving on the classpath. This is definitely a common problem so
> we need to add some explicit docs on it.
>

I seem to have this sorted out, don't ask me how. Once again I was probably
editing things on the cluster master when I should have been editing the
cluster controller, or vice versa. But, yeah, many of the examples just get
lost in a sea of DAG INFO messages.


> Are you going through http://spark.apache.org/docs/latest/quick-start.html?
> You should be able to do just sbt package. Once you do that you don’t need
> to deploy your application’s JAR to the cluster, just pass it to
> spark-submit and it will automatically be sent over.
>

Ah, that answers another question I just asked elsewhere... Yup, I re-read
pretty much every documentation page daily. And I'm making my way through
every video.


> > Meanwhile I'm learning scala... Great Turing's Ghost, it's the dream
> language we've theorized about for years! I hadn't realized!
>
> Indeed, glad you’re enjoying it.
>

"Enjoying", not yet alas, I'm sure I'll get there. But I do understand the
implications of a mixed functional-imperative language with closures and
lambdas. That is serious voodoo.

-- 
Jeremy Lee  BCompSci(Hons)
  The Unorthodox Engineers

Mime
View raw message