hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20183) Inserting from bucketed table can cause data loss, if the source table contains empty buckets
Date Wed, 18 Jul 2018 07:11:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547471#comment-16547471
] 

Peter Vary commented on HIVE-20183:
-----------------------------------

[~kgyrtkirk]: Could you please review?
 The thoughts behind the patch are the following:
 * FileSinkOperator takes care bucketing here: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L692-L705]
 * Essentially it changes the output buckets based on the source *fileId* instead of *taskId*
 * The fileId is set here: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java#L154-L167]
 * This is only called when we read the first row from the table, so when no data is in the
files then it is not called at all
 * We change this, by the patch to call this in the *closeOp*, if it is not set yet.

The tests are all green. Added new tests for TestCliDriver, TestSparkCliDriver:

* [https://builds.apache.org/job/PreCommit-HIVE-Build/12661/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_bucket7_/]

* [https://builds.apache.org/job/PreCommit-HIVE-Build/12661/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_bucket7_/]

I think Tez do not have to be tested, since it uses different ways to handle 0 length files.

Thanks,

Peter

> Inserting from bucketed table can cause data loss, if the source table contains empty
buckets
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-20183
>                 URL: https://issues.apache.org/jira/browse/HIVE-20183
>             Project: Hive
>          Issue Type: Bug
>          Components: Operators
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>            Priority: Major
>         Attachments: HIVE-20183.2.patch, HIVE-20183.patch
>
>
> Could be reproduced by the following:
> {code}
> set hive.enforce.bucketing=true;
> set hive.enforce.sorting=true;
> set hive.optimize.bucketingsorting=true;
> create table bucket1 (id int, val string) clustered by (id) sorted by (id ASC) INTO 4
BUCKETS;
> insert into bucket1 values (1, 'abc'), (3, 'abc');
> select * from bucket1;
> +-------------+--------------+
> | bucket1.id  | bucket1.val  |
> +-------------+--------------+
> | 3           | abc          |
> | 1           | abc          |
> +-------------+--------------+
> create table bucket2 like bucket1;
> insert overwrite table bucket2 select * from bucket1;
> select * from bucket2;
> +-------------+--------------+
> | bucket2.id  | bucket2.val  |
> +-------------+--------------+
> | 1           | abc          |
> +-------------+--------------+
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message