spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Semenov <>
Subject Re: Looking at EMR Logs
Date Fri, 31 Mar 2017 14:46:23 GMT
You can provide your own log directory, where Spark log will be saved, and
that you could replay afterwards.

Set in your job this: `spark.eventLog.dir=s3://bucket/some/directory` and
run it.
Note! The path `s3://bucket/some/directory` must exist before you run your
job, it'll not be created automatically.

The Spark HistoryServer on EMR won't show you anything because it's looking
for logs in `hdfs:///var/log/spark/apps` by default.

After that you can either copy the log files from s3 to the hdfs path
above, or you can copy them locally to `/tmp/spark-events` (the default
directory for spark logs) and run the history server like:
cd /usr/local/src/spark-1.6.1-bin-hadoop2.6
and then open http://localhost:18080

On Thu, Mar 30, 2017 at 8:45 PM, Paul Tremblay <>

> I am looking for tips on evaluating my Spark job after it has run.
> I know that right now I can look at the history of jobs through the web
> ui. I also know how to look at the current resources being used by a
> similar web ui.
> However, I would like to look at the logs after the job is finished to
> evaluate such things as how many tasks were completed, how many executors
> were used, etc. I currently save my logs to S3.
> Thanks!
> Henry
> --
> Paul Henry Tremblay
> Robert Half Technology

View raw message