chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-664) network compression between agent and collector
Date Sun, 16 Sep 2012 04:17:07 GMT


Eric Yang commented on CHUKWA-664:

When compression is enabled, flush is called on every chunk.  When it is not compressed, flush
is not called.  Flush is the cause of increased TCP fragments.  When agent to collector subscribing
ratio is too high, increased TCP fragments can cause excessive retransmission under high load
conditions and leading to tcp incast problem.  A chunk is typically very small, and we don't
need to flush immediately.  This would save number of TCP headers to send.  Collector would
provide HTTP response code to agent if re-transmit of the last set of chunks is necessary.
 Therefore, it is best to let TCP buffer fill up then send data.  This will help the throughput
rate for compressed data stream for the current patch.
> network compression between agent and collector
> -----------------------------------------------
>                 Key: CHUKWA-664
>                 URL:
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: Data Collection
>    Affects Versions: 0.5.0, 0.6.0
>            Reporter: Sourygna Luangsay
>            Priority: Trivial
>             Fix For: 0.6.0
>         Attachments: chukwa-664.patch
> As suggested in$13d9c100$3b8d4300$@com%3E
, Chukwa should be able to compress network communications between agent and collector.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message