hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-20165) Enable ZLIB for streaming ingest
Date Tue, 17 Jul 2018 18:15:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prasanth Jayachandran updated HIVE-20165:
-----------------------------------------
    Attachment: HIVE-20165.3.patch

> Enable ZLIB for streaming ingest
> --------------------------------
>
>                 Key: HIVE-20165
>                 URL: https://issues.apache.org/jira/browse/HIVE-20165
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 4.0.0, 3.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>         Attachments: HIVE-20165.1.patch, HIVE-20165.2.patch, HIVE-20165.3.patch
>
>
> Per [~gopalv]'s recommendation tried running streaming ingest with and without zlib.
Following are the numbers
>  
>  *Compression: NONE*
>  Total rows committed: 93800000
>  Throughput: *1563333* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
>  *14.1 G*  /apps/hive/warehouse/prasanth.db/culvert
>   
>  *Compression: ZLIB*
>  Total rows committed: 92100000
>  Throughput: *1535000* rows/second
> $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert
>  *7.4 G*  /apps/hive/warehouse/prasanth.db/culvert
>   
>  ZLIB is getting us 2x compression and only 2% lesser throughput. We should enable ZLIB
by default for streaming ingest. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message