hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Commented] (HIVE-17704) Hive QA isn't collecting spark.log anymore
Date Sat, 07 Oct 2017 00:35:00 GMT


Sahil Takiar commented on HIVE-17704:

Ok had some time to work on this today, here is what I found. The {{spark.log}} file isn't
generated locally anymore. I went back to branch-1.1 and couldn't even it generate it locally
there either. On branch-1.1, the file will only be generated if Spark picks up {{data/conf/spark/}}
as the log4j config file for the Driver and Executors. Hive tries to ensure this happens by
copying the config file into {{SPARK_HOME/conf/}}, but as far as I can tell that doesn't always
work. I used the {{-Dlog4j.debug}} and found out that Spark will usually pick up the log4j
file from some jar on its classpath. Most blog posts online discussing how to configure log4j
for Spark require setting {{-Dlog4j.configuration=}} in the {{spark.driver.extraJavaOptions}}
config. I tried that locally at it seems to work. I think there are a few other fixes that
require this to work for the master branch. For one, Spark is still on log4j1.x, but the properties
file we are giving it is for log4j2.x. It would be nice if each executor logged to a different

Alternatively, we could just archive everything under {{itests/qtest-spark/target/sparkOnYarn/sparkOnYarn-logDir}}
which contains all the container logs. Which is probably the easiest approach

> Hive QA isn't collecting spark.log anymore
> ------------------------------------------
>                 Key: HIVE-17704
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
> Checked a recent Hive QA run, and the {{spark.log}} file isn't being collected by ptest
(maybe isn't not even being generated anymore, haven't checked).
> We should re-enable it, its hard to debug Hive QA runs without the {{spark.log}} file.

This message was sent by Atlassian JIRA

View raw message