spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <>
Subject Re: HiveContext: cache table not supported for partitioned table?
Date Fri, 03 Oct 2014 03:01:57 GMT
Cache table works with partitioned table.

I guess you’re experimenting with a default local metastore and the 
metastore_db directory doesn’t exist at the first place. In this case, 
all metastore tables/views don’t exist at first and will throw the error 
message you saw when the |PARTITIONS| metastore table is accessed for 
the first time by Hive client. However, you should also see this line 
before this error:

    14/10/03 10:51:30 ERROR ObjectStore: Direct SQL failed, falling back
    to ORM

And then the table is created on the fly. The cache operation is also 
performed normally. You can verify this by selecting it and check the 
Spark UI for cached RDDs. If you try to uncache the table and cache it 
again, you won’t see this error any more.

Normally, in production environment you won’t see this error because 
metastore database is usually setup ahead of time.

On 10/3/14 3:39 AM, Du Li wrote:

> Hi,
> In Spark 1.1 HiveContext, I ran a create partitioned table command 
> followed by a cache table command and got 
> a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not 
> exist. But cache table worked fine if the table is not a partitioned 
> table.
> Can anybody confirm that cache of partitioned table is not supported 
> yet in current version?
> Thanks,
> Du


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message