spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config
Date Mon, 25 Feb 2019 10:59:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-25590.
-------------------------------
       Resolution: Fixed
         Assignee: Jiaxin Shan
    Fix Version/s: 3.0.0

Resolved by https://github.com/apache/spark/pull/23814 , per Marcelo

> kubernetes-model-2.0.0.jar masks default Spark logging config
> -------------------------------------------------------------
>
>                 Key: SPARK-25590
>                 URL: https://issues.apache.org/jira/browse/SPARK-25590
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Marcelo Vanzin
>            Assignee: Jiaxin Shan
>            Priority: Major
>             Fix For: 3.0.0
>
>
> That jar file, which is packaged when the k8s profile is enabled, has a log4j configuration
embedded in it:
> {noformat}
> $ jar tf /path/to/kubernetes-model-2.0.0.jar | grep log4j
> log4j.properties
> {noformat}
> What this causes is that Spark will always use that log4j configuration instead of its
own default (log4j-defaults.properties), unless the user overrides it by somehow adding their
own in the classpath before the kubernetes one.
> You can see that by running spark-shell. With the k8s jar in:
> {noformat}
> $ ./bin/spark-shell 
> ...
> Setting default log level to "WARN"
> {noformat}
> Removing the k8s jar:
> {noformat}
> $ ./bin/spark-shell 
> ...
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> {noformat}
> The proper fix would be for the k8s jar to not ship that file, and then just upgrade
the dependency in Spark, but if there's something easy we can do in the meantime...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message