spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "@Sanjiv Singh" <>
Subject Spark SQL is not returning records for HIVE transactional tables on HDP
Date Sat, 12 Mar 2016 08:24:23 GMT
Hi All,

I am facing this issue on HDP setup on which COMPACTION is required only
once for transactional tables to fetch records with Spark SQL.
On the other hand, Apache setup doesn't required compaction even once.

May be something got triggered on meta-store after compaction, Spark SQL
start recognizing delta files.

Let know me if needed other details to get root cause.

Try this,

*See complete scenario :*

hive> create table int) clustered by (id) into 2 buckets
STORED AS ORC TBLPROPERTIES ('transactional'='true');
hive> insert into values(10);

scala> sqlContext.table("").count // Gives 0, which is wrong
because data is still in delta files

Now run major compaction:


scala> sqlContext.table("").count // Gives 1

hive> insert into foo values(20);

scala> sqlContext.table("").count* // Gives 2 , no compaction

Sanjiv Singh
Mob :  +091 9990-447-339

View raw message