spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Where are logs for Spark Kafka Yarn on Cloudera
Date Tue, 29 Sep 2015 14:03:13 GMT
(-dev@)

Try using the "yarn logs" command to read logs for finished
applications. You can also browse the RM UI to find more information
about the applications you ran.

On Mon, Sep 28, 2015 at 11:37 PM, Rachana Srivastava
<Rachana.Srivastava@markmonitor.com> wrote:
> Hello all,
>
>
>
> I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working
> fine I am saving the output to hdfs.  The example works fine in local mode
> but not on yarn mode.  I cannot see any output logged when I changed the
> mode to yarn-client or yarn-cluster or cannot find any errors logged.  For
> my application id I was looking for logs under
> /var/log/hadoop-yarn/containers (e.g
> /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr)
> but I cannot find anything useful information.   Is it the only location
> where application logs are logged.
>
>
>
> Also tried setting log output to spark.yarn.app.container.log.dir but got
> access denied error.
>
>
>
> Question:  Do we need to have some special setup to run spark streaming on
> Yarn?  How do we debug?  Where to find more details to test streaming on
> Yarn.
>
>
>
> Thanks,
>
>
>
> Rachana



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message