spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Zhuge <>
Subject [DISCUSS] ViewCatalog interface for DSv2
Date Tue, 13 Aug 2019 15:56:31 GMT
Catalog support has been added to DSv2 along with a table catalog
interface. Here I'd like to propose a view catalog interface, for the
following benefit:

   - Abstraction for view management thus allowing different view backends
   - Disassociation of view definition storage from Hive Metastore

A catalog plugin can be both TableCatalog and ViewCatalog. Resolve an
identifier as view first then table.

More details in SPIP and PR if we decide to proceed. Here is a quick glance
at the API:

ViewCatalog interface:

   - loadView
   - listViews
   - createView
   - deleteView

View interface:

   - name
   - originalSql
   - defaultCatalog
   - defaultNamespace
   - viewColumns
   - owner
   - createTime
   - softwareVersion
   - options (map)

ViewColumn interface:

   - name
   - type

John Zhuge

View raw message