spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <>
Subject Where are logs for Spark Kafka Yarn on Cloudera
Date Tue, 29 Sep 2015 06:37:48 GMT
Hello all,

I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working fine I am saving
the output to hdfs.  The example works fine in local mode but not on yarn mode.  I cannot
see any output logged when I changed the mode to yarn-client or yarn-cluster or cannot find
any errors logged.  For my application id I was looking for logs under /var/log/hadoop-yarn/containers
(e.g /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr)
but I cannot find anything useful information.   Is it the only location where application
logs are logged.

Also tried setting log output to but got access denied error.

Question:  Do we need to have some special setup to run spark streaming on Yarn?  How do we
debug?  Where to find more details to test streaming on Yarn.



View raw message