spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (Jira)" <>
Subject [jira] [Commented] (SPARK-25065) Driver and executors pick the wrong logging configuration file.
Date Fri, 01 May 2020 16:52:02 GMT


Apache Spark commented on SPARK-25065:

User 'ScrapCodes' has created a pull request for this issue:

> Driver and executors pick the wrong logging configuration file.
> ---------------------------------------------------------------
>                 Key: SPARK-25065
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.4.4, 3.0.0
>            Reporter: Prashant Sharma
>            Priority: Major
> Currently, when running in kubernetes mode, it sets necessary configuration properties
by creating a file and mounting a conf dir.
> The shipped Dockerfile, do not copy conf to the image, and this is on purpose and that
is well understood. However, one would like to have his custom logging configuration file
in the image conf directory.
> In order to achieve this, it is not enough to copy it in the spark's conf dir of the
resultant image, as it is reset during kubernetes mount conf volume step.
> In order to reproduce, please add {code}-Dlog4j.debug{code} to {code:java}spark.(executor|driver).extraJavaOptions{code}.
This way, it was found the provided log4j file is not picked and the one coming from kubernetes
client jar was picked up by the driver process.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message