spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Everett Anderson <ever...@nuna.com.INVALID>
Subject Slow metastore list tables in Spark 2.x
Date Wed, 04 Jan 2017 19:26:11 GMT
Hi,

In Spark 1.6.2, we were able to very quickly -- nearly instantly -- search
through the list of (many) table names in our Hive metastore with

sqlContext.tableNames().filter(_.matches("some regex")).foreach { println }

In Spark 2.0.2, however, this takes forever. Similarly, queries with
Catalog that should return a Dataset like

spark.catalog.listTables("default")

take forever.  Setting the log level to DEBUG in the spark-shell, I can see
the above command is scrolling through every table name in the metastore.

Does anyone have a better way to quickly search through the metastore for a
table names matching a regexp in Spark 2?

Thanks!

- Everett

Mime
View raw message