hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
Date Wed, 18 Jul 2018 03:20:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gopal V updated HIVE-20165:
---------------------------
    Attachment: HIVE-20165.1.branch-3.patch

> Enable ZLIB for streaming ingest
> --------------------------------
>
>                 Key: HIVE-20165
>                 URL: https://issues.apache.org/jira/browse/HIVE-20165
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 4.0.0, 3.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Gopal V
>            Priority: Major
>         Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, HIVE-20165.2.patch,
HIVE-20165.3.patch
>
>
> Per [~gopalv]'s recommendation tried running streaming ingest with and without zlib.
Following are the numbers
>  
>  *Compression: NONE*
>  Total rows committed: 93800000
>  Throughput: *1563333* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
>  *14.1 G*  /apps/hive/warehouse/prasanth.db/culvert
>   
>  *Compression: ZLIB*
>  Total rows committed: 92100000
>  Throughput: *1535000* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
>  *7.4 G*  /apps/hive/warehouse/prasanth.db/culvert
>   
>  ZLIB is getting us 2x compression and only 2% lesser throughput. We should enable ZLIB
by default for streaming ingest. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message