spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bijay Kumar Pathak <bkpat...@mtu.edu>
Subject Spark SQL insert overwrite table not showing all the partition.
Date Thu, 21 Apr 2016 22:20:13 GMT
Hi,

I have a job which writes to the Hive table with dynamic partition. Inside
the job,  I am writing into the table two-time but I am only seeing the
partition with last write although I can see in the Spark UI it is
processing data fro both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
              SELECT * from temp_table
                """.format(date_val)
                 )



Thanks,
Bijay

Mime
View raw message