spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto
Date Mon, 16 Mar 2020 22:54:07 GMT

     [ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun updated SPARK-30022:
----------------------------------
    Affects Version/s:     (was: 3.0.0)
                       3.1.0

> Supporting Parsing of Simple Hive Virtual View created from Presto
> ------------------------------------------------------------------
>
>                 Key: SPARK-30022
>                 URL: https://issues.apache.org/jira/browse/SPARK-30022
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Arun Ravi M V
>            Priority: Major
>
>  
> We have an environment were we use Apache Spark and Presto (both backed by Apache Hive
Metastore). Currently, Views created from Presto fail to get parsed in Apache Spark. This
is because Presto stores the View definition and View schema in a base64 encoded fashion and
Spark is unable to process it. I would like to propose a minor change that will allow us to
read these encoded definitions created by Presto in a Spark Program.
> Assuming that the UDFs are made available, the user should be able to read presto views
after the fix.
>  
> I would like to propose a change to [https://github.com/apache/spark/blob/9459833eae7fae887af560f3127997e023c51d00/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L440]
> to have support creation of CatlogTable for Views created from Presto.
>  
> Hive Metastore DB, the table definition for Presto views  (*select* * *from* TBLS *where*
`TBL_TYPE` *like* '%VIRTUAL_VIEW%') shows that the VIEW_EXPANDED_TEXT is hardcoded as `/*
Presto View **/` and VIEW_ORIGINAL_TEXT is `/** Presto View: base64({ "originalSql": "" "catalog":
"", "schema": "", "columns": [
> { "name": "", "type": "" }
> ], "owner": ""}) */`  
> Refer: [https://github.com/prestodb/presto/blob/3242715959a169dbcdd88946c28488d2365c8886/presto-hive/src/main/java/com/facebook/presto/hive/HiveUtil.java#L614]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message