spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "angerszhu (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-29018) Build spark thrift server on it's own code based on protocol v11
Date Wed, 18 Dec 2019 12:19:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

angerszhu updated SPARK-29018:
------------------------------
    Summary: Build spark thrift server on it's own code based on protocol v11  (was: Spark
ThriftServer change to it's own API)

> Build spark thrift server on it's own code based on protocol v11
> ----------------------------------------------------------------
>
>                 Key: SPARK-29018
>                 URL: https://issues.apache.org/jira/browse/SPARK-29018
>             Project: Spark
>          Issue Type: Umbrella
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: angerszhu
>            Priority: Major
>
> With the development of Spark and Hive,in current `sql/hive-thriftserver`, we need
to do a lot of work to solve code conflicts between different hive versions. It's an annoying
and unending work in current ways. And these issues are troubling us when we develop new features
for the SparkThriftServer2. We suppose to implement a new thrift server based on latest v11
`TCLService.thrift` thrift protocol. Implement all API in spark's own code to get rid of hive
code .
>  Finally, the new thrift server have below feature:
>  # support all functions current `hive-thriftserver` support
>  # use all code maintained by spark itself
>  # realize origin function fit to spark’s own feature, won't limited by hive's code
>  # support running without hive metastore or with hive metastore
>  # support user impersonation by Multi-tenant authority separation, hive authentication
and DFS authentication
>  # add a new module `spark-jdbc`, with connection url `jdbc:spark:<host>:<port>/<db>`,
all `hive-jdbc` support we will all support
>  # support both `hive-jdbc` and `spark-jdbc` client for compatibility with most clients
> We have done all these works in our repo, now we plan merge our code into the master
step by step.
>  # *phase1* pr about build new module `spark-service` on folder `sql/service`
>  2. *phase2* pr thrift protocol and generated thrift protocol java code 
>  3. *phase3* pr with all `spark-service` module code and description about design, also
UT
>  4. *phase4* pr about build new module `spark-jdbc` on folder `sql/jdbc`
>  5. *phase5* pr with all `spark-jdbc` module code and UT
>  6. *phase6* pr about support thriftserver Impersonation
>  7. *phase7* pr about build spark's own beeline client `spark-beeline`
>  8. *phase8* pr about spark's own Cli client `Spark SQL CLI` `spark-cli`
>  
>  
> Google Doc : [https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.im79wwkzycsr]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message