spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "feiwang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-29295) Duplicate result when dropping partition of an external table and then overwriting
Date Tue, 01 Oct 2019 02:28:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941475#comment-16941475
] 

feiwang commented on SPARK-29295:
---------------------------------

Yes, how can we resolve this issue?
Only by upgrading the hive version?

> Duplicate result when dropping partition of an external table and then overwriting
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-29295
>                 URL: https://issues.apache.org/jira/browse/SPARK-29295
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.4
>            Reporter: feiwang
>            Priority: Major
>
> When we drop a partition of a external table and then overwrite it, if we set CONVERT_METASTORE_PARQUET=true(default
value), it will overwrite this partition.
> But when we set CONVERT_METASTORE_PARQUET=false, it will give duplicate result.
> Here is a reproduce code below(you can add it into SQLQuerySuite in hive module):
> {code:java}
>   test("spark gives duplicate result when dropping a partition of an external partitioned
table" +
>     " firstly and they overwrite it") {
>     withTable("test") {
>       withTempDir { f =>
>         sql("create external table test(id int) partitioned by (name string) stored as
" +
>           s"parquet location '${f.getAbsolutePath}'")
>         withSQLConf(HiveUtils.CONVERT_METASTORE_PARQUET.key -> false.toString) {
>           sql("insert overwrite table test partition(name='n1') select 1")
>           sql("ALTER TABLE test DROP PARTITION(name='n1')")
>           sql("insert overwrite table test partition(name='n1') select 2")
>           checkAnswer( sql("select id from test where name = 'n1' order by id"),
>             Array(Row(1), Row(2)))
>         }
>         withSQLConf(HiveUtils.CONVERT_METASTORE_PARQUET.key -> true.toString) {
>           sql("insert overwrite table test partition(name='n1') select 1")
>           sql("ALTER TABLE test DROP PARTITION(name='n1')")
>           sql("insert overwrite table test partition(name='n1') select 2")
>           checkAnswer( sql("select id from test where name = 'n1' order by id"),
>             Array(Row(2)))
>         }
>       }
>     }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message