spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-17767) Spark SQL ExternalCatalog API custom implementation support
Date Mon, 03 Oct 2016 23:22:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543733#comment-15543733
] 

Reynold Xin edited comment on SPARK-17767 at 10/3/16 11:22 PM:
---------------------------------------------------------------

Guys - while I think this is very useful to do, I'm going to mark this as "later" for now.
The reason is that there are a lot of things to consider before making this switch, including:

- The ExternalCatalog API is currently internal, and we can't just make it public without
thinking about the consequences and whether this API is maintainable in the long run.
- SPARK-15777 We need to design this in the context of catalog federation and persistence.
- SPARK-15691 Refactoring of how we integrate with Hive.

This is not as simple as just submitting a PR to make it pluggable.


was (Author: rxin):
Guys - while I think this is very useful to do, I'm going to mark this as "later" for now.
The reason is that there are a lot of things to consider before making this switch, including:

- The ExternalCatalog API is currently internal, and we can't just make it public without
thinking about the consequences.
- SPARK-15777 We need to design this in the context of catalog federation and persistence.
- SPARK-15691 Refactoring of how we integrate with Hive.


> Spark SQL ExternalCatalog API custom implementation support
> -----------------------------------------------------------
>
>                 Key: SPARK-17767
>                 URL: https://issues.apache.org/jira/browse/SPARK-17767
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Alex Liu
>
> There is no way/easy way to configure Spark to use customized ExternalCatalog.  Internal
source code is hardcoded to use either hive or in-memory metastore. Spark SQL thriftserver
is hardcoded to use HiveExternalCatalog. We should be able to create a custom external catalog
and thriftserver should be able to use it. Potentially Spark SQL thriftserver shouldn't depend
on Hive thriftserer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message