chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-707) Replace Chukwa collector with Apache Kafka
Date Wed, 23 Jul 2014 23:30:39 GMT


Eric Yang commented on CHUKWA-707:

Kafka topic is designed to have one consumer per topic.  Kafka topics should be partitioned
by host +  data type.  Otherwise, n consumers are running inside n Chukwa agents, and causes
the same message to be consumed n times and send to writer n times.

> Replace Chukwa collector with Apache Kafka
> ------------------------------------------
>                 Key: CHUKWA-707
>                 URL:
>             Project: Chukwa
>          Issue Type: New Feature
>            Reporter: Eric Yang
>            Assignee: shreyas subramanya
>         Attachments: CHUKWA-707.patch1
> Chukwa collector has stopped evolving since 2010.  Newer framework has offer better features
of message queues, and Apache Kafka looks like a good replacement for Chukwa collector.
> Chukwa agent can implement a connector to Apache Kafka to replace Chukwa collector, and
HBase consumer to write data to HBase.  HICC REST API change to new HBase storage format.

This message was sent by Atlassian JIRA

View raw message