spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Narcis Andrei Moga (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-16784) Configurable log4j settings
Date Mon, 25 Feb 2019 08:34:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776633#comment-16776633
] 

Narcis Andrei Moga edited comment on SPARK-16784 at 2/25/19 8:33 AM:
---------------------------------------------------------------------

I have the same problem after migration from Spark 2.2.1 to 2.4.0 and deploy mode  cluster
& standalone namager (it not happens in client mode deploy)

I test in docker and all required files are present in all containers (1 master & 2 workers
- Spark have no config for this test - it is just untar)

*1) Executor command observed in the stderr file*

Spark Executor Command: "/srv/java/jdk/bin/java" "-cp" "/usr/lib/spark/conf/:/usr/lib/spark/jars/*"
"-Xmx1024M" "-Dspark.driver.port=45431" "-Dspark.cassandra.connection.port=9042" 
 "-Dspark.rpc.askTimeout=10s" "-Dspark.application.ldap.port=55389" _*"-Duser.timezone=UTC"*_
_*"-Dlog4j.configuration=[file:///log4j.properties.executor]"*_

"-Dcom.sun.management.jmxremote" 
 "-Dcom.sun.management.jmxremote.authenticate=false"

"-Dcom.sun.management.jmxremote.local.only=false"

"-Dcom.sun.management.jmxremote.ssl=false" "-Djava.net.preferIPv4Stack=true" 
 "-Dcom.sun.management.jmxremote.port=0" "-Djava.util.logging.config.file=/jmx-logging.properties"
"org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" 
 "spark://CoarseGrainedScheduler@c1-spark-executor2:45431" "--executor-id" "1" "--hostname"
"172.18.0.22" "--cores" "1" "--app-id" "app-20190224171936-0010" "--worker-url" 
 "spark://Worker@172.18.0.22:36555"

*2) Partial command of the Driver observed in the stderr file*

Launch Command: "/srv/java/jdk/bin/java" "-cp" "/usr/lib/spark/conf/:/usr/lib/spark/jars/*"
"-Xmx1024M" 
 _*"-Dspark.driver.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=[file:///log4j.properties.driver*_]
 "-Dspark.kafka.ppu.topic.name=..." 
 ....

*3) Submit command*

spark-submit \
 --deploy-mode cluster \
 --master spark://172.18.0.20:7077 \
 --properties-file /application.properties \
 --class com....... \
 /logs-correlation-2.4.1-1.noarch.jar

*4) application.properties contains*

spark.driver.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=[file:///log4j.properties.driver]

spark.executor.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=[file:///log4j.properties.executor]

 

 


was (Author: andreim):
I have the same problem after migration from Spark 2.2.1 to 2.4.0 and deploy mode  cluster
& standalone namager (it not happens in client mode deploy)

I test in docker and all required files are present in all containers (1 master & 2 workers
- Spark have no config for this test - it is just untar)

*1) Executor command observed in the stderr file*

Spark Executor Command: "/srv/java/jdk/bin/java" "-cp" "/usr/lib/spark/conf/:/usr/lib/spark/jars/*"
"-Xmx1024M" "-Dspark.driver.port=45431" "-Dspark.cassandra.connection.port=9042" 
"-Dspark.rpc.askTimeout=10s" "-Dspark.application.ldap.port=55389" _*"-Duser.timezone=UTC"*_
_*"-Dlog4j.configuration=file:///log4j.properties.executor"*_ "-Dcom.sun.management.jmxremote"

"-Dcom.sun.management.jmxremote.authenticate=false" "-Dcom.sun.management.jmxremote.local.only=false"
"-Dcom.sun.management.jmxremote.ssl=false" "-Djava.net.preferIPv4Stack=true" 
"-Dcom.sun.management.jmxremote.port=0" "-Djava.util.logging.config.file=/jmx-logging.properties"
"org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" 
"spark://CoarseGrainedScheduler@c1-spark-executor2:45431" "--executor-id" "1" "--hostname"
"172.18.0.22" "--cores" "1" "--app-id" "app-20190224171936-0010" "--worker-url" 
"spark://Worker@172.18.0.22:36555"

*2) Partial command of the Driver observed in the stderr file*

Launch Command: "/srv/java/jdk/bin/java" "-cp" "/usr/lib/spark/conf/:/usr/lib/spark/jars/*"
"-Xmx1024M" 
_*"-Dspark.driver.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=file:///log4j.properties.driver*_
"-Dspark.kafka.ppu.topic.name=..." 
....

*3) Submit command*

spark-submit \
--deploy-mode cluster \
--master spark://172.18.0.20:7077 \
--properties-file /application.properties \
--class com....... \
/logs-correlation-2.4.1-1.noarch.jar

*4) application.properties contains*

spark.driver.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=file:///log4j.properties.driver

spark.executor.extraJavaOptions=-Duser.timezone=UTC -Dlog4j.configuration=file:///log4j.properties.executor

 

 

> Configurable log4j settings
> ---------------------------
>
>                 Key: SPARK-16784
>                 URL: https://issues.apache.org/jira/browse/SPARK-16784
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Michael Gummelt
>            Priority: Major
>
> I often want to change the logging configuration on a single spark job.  This is easy
in client mode.  I just modify log4j.properties.  It's difficult in cluster mode, because
I need to modify the log4j.properties in the distribution in which the driver runs.  I'd like
a way of setting this dynamically, such as a java system property.  Some brief searching showed
that log4j doesn't seem to accept such a property, but I'd like to open up this idea for further
comment.  Maybe we can find a solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message