spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <>
Subject [jira] [Reopened] (SPARK-31170) Spark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
Date Thu, 19 Mar 2020 23:20:00 GMT


Dongjoon Hyun reopened SPARK-31170:

This is reverted because this broke all `hive-1.2` profile Jenkins jobs (2 SBT/2 Maven).

> Spark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
> --------------------------------------------------------------------
>                 Key: SPARK-31170
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Kent Yao
>            Assignee: Kent Yao
>            Priority: Major
>             Fix For: 3.0.0
> In Spark CLI, we create a hive CliSessionState and it does not load the hive-site.xml.
So the configurations in hive-site.xml will not take effects like other spark-hive integration
> Also, the warehouse directory is not correctly picked. If the `default` database does
not exist, the CliSessionState will create one during the first time it talks to the metastore.
The `Location` of the default DB will be neither the value of spark.sql.warehousr.dir nor
the user-specified value of hive.metastore.warehourse.dir, but the default value of hive.metastore.warehourse.dir
which will always be `/user/hive/warehouse`.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message