spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhailau, Alex" <Alex.Mikhai...@mlb.com>
Subject Re: Referencing YARN application id, YARN container hostname, Executor ID and YARN attempt for jobs running on Spark EMR 5.7.0 in log statements?
Date Mon, 28 Aug 2017 21:41:50 GMT
Thanks, Vadim. The issue is not access to logs. I am able to view them.

I have cloudwatch logs agent push logs to elasticsearch and then into Kibana using json-event-layout
for log4j output. I would like to also log applicationId, executorId, etc in those log statements
for clarity. Is there an MDC way with spark or something other than to achieve this?

Alex

From: Vadim Semenov <vadim.semenov@datadoghq.com>
Date: Monday, August 28, 2017 at 5:18 PM
To: "Mikhailau, Alex" <Alex.Mikhailau@mlb.com>
Cc: "user@spark.apache.org" <user@spark.apache.org>
Subject: Re: Referencing YARN application id, YARN container hostname, Executor ID and YARN
attempt for jobs running on Spark EMR 5.7.0 in log statements?

When you create a EMR cluster you can specify a S3 path where logs will be saved after cluster,
something like this:

s3://bucket/j-18ASDKLJLAKSD/containers/application_1494074597524_0001/container_1494074597524_0001_01_000001/stderr.gz

http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-web-log-files.html

On Mon, Aug 28, 2017 at 4:43 PM, Mikhailau, Alex <Alex.Mikhailau@mlb.com<mailto:Alex.Mikhailau@mlb.com>>
wrote:
Does anyone have a working solution for logging YARN application id, YARN container hostname,
Executor ID and YARN attempt for jobs running on Spark EMR 5.7.0 in log statements? Are there
specific ENV variables available or other workflow for doing that?

Thank you

Alex

Mime
View raw message