spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhu, Luke" <>
Subject Re: Use our own metastore with Spark SQL
Date Tue, 15 Oct 2019 02:05:58 GMT
I had a similar issue this summer while prototyping Spark on K8s. I ended
up sticking with Hive Metastore 2 to meet time goals. Not sure if I was
using it correctly, but I only needed Hadoop + Hive JARs; I did not need to
run HDFS, YARN, etc. Using the metastore with an s3a warehouse.dir path
seemed to work fine.

When Spark supports Metastore 3.0, things should be a bit easier as HMS 3
will have clearer instructions for standalone deployments.

If you have more time and truly need to move away from everything Hadoop,
you can also implement ExternalCatalog:

See for ongoing progress on
a Glue ExternalCatalog implementation. If you are using EMR, you can also
check  out

On Mon, Oct 14, 2019 at 12:24 PM xweb <> wrote:

> Is it possible to use our own metastore instead of Hive Metastore with
> Spark
> SQL?
> Can you please point me to some docs or code I can look at to get it done?
> We are moving away from everything Hadoop.
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message