spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Lee <>
Subject Spark 2.0 preview - How to configure warehouse for Catalyst? always pointing to /user/hive/warehouse
Date Fri, 17 Jun 2016 23:34:11 GMT
>From branch-2.0, Spark 2.0.0 preview,

I found it interesting, no matter what you do by configuring


it will always pull up the default path which is /user/hive/warehouse

In the code, I notice that at LOC45


object SimpleAnalyzer extends Analyzer(

    new SessionCatalog(

      new InMemoryCatalog,


      new SimpleCatalystConf(caseSensitiveAnalysis = true)),

    new SimpleCatalystConf(caseSensitiveAnalysis = true))

It will always initialize with the SimpleCatalystConf which is applying the hardcoded default

defined in LOC58


case class SimpleCatalystConf(

    caseSensitiveAnalysis: Boolean,

    orderByOrdinal: Boolean = true,

    groupByOrdinal: Boolean = true,

    optimizerMaxIterations: Int = 100,

    optimizerInSetConversionThreshold: Int = 10,

    maxCaseBranchesForCodegen: Int = 20,

    runSQLonFile: Boolean = true,

    warehousePath: String = "/user/hive/warehouse")

  extends CatalystConf

I couldn't find any other way to get around this.

It looks like this was fixed (in SPARK-15387) after


[SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to ... ยท apache/spark@9c817d0<>
...make database directory. ## What changes were proposed in this pull request? After #12871
is fixed, we are forced to make `/user/hive/warehouse` when SimpleAnalyzer is used but SimpleAnalyzer

Just want to confirm this was the root cause and the PR that fixed it. Thanks.

View raw message