spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <Rachana.Srivast...@markmonitor.com>
Subject Where are logs for Spark Kafka Yarn on Cloudera
Date Tue, 29 Sep 2015 06:37:48 GMT
Hello all,

I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working fine I am saving
the output to hdfs.  The example works fine in local mode but not on yarn mode.  I cannot
see any output logged when I changed the mode to yarn-client or yarn-cluster or cannot find
any errors logged.  For my application id I was looking for logs under /var/log/hadoop-yarn/containers
(e.g /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr)
but I cannot find anything useful information.   Is it the only location where application
logs are logged.

Also tried setting log output to spark.yarn.app.container.log.dir but got access denied error.

Question:  Do we need to have some special setup to run spark streaming on Yarn?  How do we
debug?  Where to find more details to test streaming on Yarn.

Thanks,

Rachana

Mime
View raw message