hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-23954) count(*) with count(distinct) gives wrong results with hive.optimize.countdistinct=true
Date Tue, 25 Aug 2020 06:02:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23954?focusedWorklogId=474149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474149
]

ASF GitHub Bot logged work on HIVE-23954:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 25/Aug/20 06:01
            Start Date: 25/Aug/20 06:01
    Worklog Time Spent: 10m 
      Work Description: EugeneChung commented on pull request #1414:
URL: https://github.com/apache/hive/pull/1414#issuecomment-679795129


   It seems the error is not related with my patch.
   
   ```
   [2020-08-20T11:29:42.268Z] Status: Downloaded newer image for postgres:latest
   
   [2020-08-20T11:31:48.820Z] 3a1dc3a0b3a75eaf727731e23f9967bbc6007831481878432cad7c8354e0c922
   
   [2020-08-20T11:31:48.821Z] waiting for postgres to be available...
   
   [2020-08-20T11:31:48.821Z] psql: FATAL:  the database system is starting up
   
   [2020-08-20T11:31:48.821Z] ok
   
   [2020-08-20T11:31:48.821Z] NOTICE:  database "ms_hive_precommit_pr_1414_1_wqrcz_rrtk5_5s4x9"
does not exist, skipping
   
   [2020-08-20T11:31:48.821Z] DROP DATABASE
   
   [2020-08-20T11:31:48.821Z] ERROR:  role "hive" does not exist
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 474149)
    Time Spent: 20m  (was: 10m)

> count(*) with count(distinct) gives wrong results with hive.optimize.countdistinct=true
> ---------------------------------------------------------------------------------------
>
>                 Key: HIVE-23954
>                 URL: https://issues.apache.org/jira/browse/HIVE-23954
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer
>    Affects Versions: 3.0.0, 3.1.0
>            Reporter: Eugene Chung
>            Assignee: Eugene Chung
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-23954.01.patch, HIVE-23954.01.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> {code:java}
> select count(*), count(distinct mid) from db1.table1 where partitioned_column = '...'{code}
>  
> is not working properly when hive.optimize.countdistinct is true. By default, it's true
for all 3.x versions.
> In the two plans below, the aggregations part in the Output of Group By Operator of Map
1 are different.
>  
> - hive.optimize.countdistinct=false
> {code:java}
> +----------------------------------------------------+
> |                      Explain                       |
> +----------------------------------------------------+
> | Plan optimized by CBO.                             |
> |                                                    |
> | Vertex dependency in root stage                    |
> | Reducer 2 <- Map 1 (SIMPLE_EDGE)                   |
> |                                                    |
> | Stage-0                                            |
> |   Fetch Operator                                   |
> |     limit:-1                                       |
> |     Stage-1                                        |
> |       Reducer 2                                    |
> |       File Output Operator [FS_7]                  |
> |         Group By Operator [GBY_5] (rows=1 width=24) |
> |           Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(DISTINCT
KEY._col0:0._col0)"] |
> |         <-Map 1 [SIMPLE_EDGE]                      |
> |           SHUFFLE [RS_4]                           |
> |             Group By Operator [GBY_3] (rows=343640771 width=4160) |
> |               Output:["_col0","_col1","_col2"],aggregations:["count()","count(DISTINCT
mid)"],keys:mid |
> |               Select Operator [SEL_2] (rows=343640771 width=4160) |
> |                 Output:["mid"]                     |
> |                 TableScan [TS_0] (rows=343640771 width=4160) |
> |                   db1@table1,table1,Tbl:COMPLETE,Col:NONE,Output:["mid"] |
> |                                                    |
> +----------------------------------------------------+{code}
>  
> - hive.optimize.countdistinct=true
> {code:java}
> +----------------------------------------------------+
> |                      Explain                       |
> +----------------------------------------------------+
> | Plan optimized by CBO.                             |
> |                                                    |
> | Vertex dependency in root stage                    |
> | Reducer 2 <- Map 1 (SIMPLE_EDGE)                   |
> |                                                    |
> | Stage-0                                            |
> |   Fetch Operator                                   |
> |     limit:-1                                       |
> |     Stage-1                                        |
> |       Reducer 2                                    |
> |       File Output Operator [FS_7]                  |
> |         Group By Operator [GBY_14] (rows=1 width=16) |
> |           Output:["_col0","_col1"],aggregations:["count(_col1)","count(_col0)"] |
> |           Group By Operator [GBY_11] (rows=343640771 width=4160) |
> |             Output:["_col0","_col1"],aggregations:["count(VALUE._col0)"],keys:KEY._col0
|
> |           <-Map 1 [SIMPLE_EDGE]                    |
> |             SHUFFLE [RS_10]                        |
> |               PartitionCols:_col0                  |
> |               Group By Operator [GBY_9] (rows=343640771 width=4160) |
> |                 Output:["_col0","_col1"],aggregations:["count()"],keys:mid |
> |                 Select Operator [SEL_2] (rows=343640771 width=4160) |
> |                   Output:["mid"]                   |
> |                   TableScan [TS_0] (rows=343640771 width=4160) |
> |                     db1@table1,table1,Tbl:COMPLETE,Col:NONE,Output:["mid"] |
> |                                                    |
> +----------------------------------------------------+
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message