incubator-hcatalog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan" <maso...@syncsort.com>
Subject Review Request 26536: Compress serialized bytes for all objects before encoding.
Date Fri, 10 Oct 2014 01:52:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26536/
-----------------------------------------------------------

Review request for hcatalog and hive.


Repository: hive-git


Description
-------

Currently all objects shipped via Hadoop Configuration are not compressed.  Compression is
limited to the data member "partitions" in InputJobInfo.  This patch will compress all serialized
objects.  This will cut down the traffic in the cluster when Configuration object is shipped
to all compute nodes.  Also, there is a 5 MB size limit on the Configuration object.  Compression
will make this limiit hardly reachable.


Diffs
-----

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 4fdb5c9 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InputJobInfo.java 360e77b


Diff: https://reviews.apache.org/r/26536/diff/


Testing
-------

No new test required.  All existing tests should pass.


Thanks,

Mariappan Asokan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message