spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Lee <alee...@hotmail.com>
Subject Spark 2.0 preview - How to configure warehouse for Catalyst? always pointing to /user/hive/warehouse
Date Fri, 17 Jun 2016 23:34:11 GMT
>From branch-2.0, Spark 2.0.0 preview,

I found it interesting, no matter what you do by configuring


spark.sql.warehouse.dir


it will always pull up the default path which is /user/hive/warehouse


In the code, I notice that at LOC45

./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala


object SimpleAnalyzer extends Analyzer(

    new SessionCatalog(

      new InMemoryCatalog,

      EmptyFunctionRegistry,

      new SimpleCatalystConf(caseSensitiveAnalysis = true)),

    new SimpleCatalystConf(caseSensitiveAnalysis = true))


It will always initialize with the SimpleCatalystConf which is applying the hardcoded default
value

defined in LOC58


./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystConf.scala


case class SimpleCatalystConf(

    caseSensitiveAnalysis: Boolean,

    orderByOrdinal: Boolean = true,

    groupByOrdinal: Boolean = true,

    optimizerMaxIterations: Int = 100,

    optimizerInSetConversionThreshold: Int = 10,

    maxCaseBranchesForCodegen: Int = 20,

    runSQLonFile: Boolean = true,

    warehousePath: String = "/user/hive/warehouse")

  extends CatalystConf


I couldn't find any other way to get around this.


It looks like this was fixed (in SPARK-15387) after


https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93

[https://avatars0.githubusercontent.com/u/4736016?v=3&s=200]<https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93>

[SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to ... ยท apache/spark@9c817d0<https://github.com/apache/spark/commit/9c817d027713859cac483b4baaaf8b53c040ad93>
github.com
...make database directory. ## What changes were proposed in this pull request? After #12871
is fixed, we are forced to make `/user/hive/warehouse` when SimpleAnalyzer is used but SimpleAnalyzer
ma...


Just want to confirm this was the root cause and the PR that fixed it. Thanks.





Mime
View raw message