hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <>
Subject [jira] [Commented] (HIVE-14324) ORC PPD for floats is broken
Date Mon, 25 Jul 2016 22:40:20 GMT


Gopal V commented on HIVE-14324:

LGTM  - +1.

The narrowing to Float is necessary before widening it back to Double.

> ORC PPD for floats is broken
> ----------------------------
>                 Key: HIVE-14324
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Critical
>         Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
> ORC stores min/max stats, bloom filters by passing floats as doubles using java's widening
conversion. So if we write a float value of 0.22 to ORC file, the min/max stats and bloom
filter will use 0.2199999988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and then to double
which compares 0.22 to 0.2199999988079071 and fails PPD evaluation. 
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type check when
SARGs are created and also PPD evaluation does not convert to column type. But now predicate
leaf creation in SARG enforces strict type check for boxed literals and predicate type and
PPD evaluation converts stats and constants to column type (predicate).

This message was sent by Atlassian JIRA

View raw message