chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shreyas subramanya (JIRA)" <>
Subject [jira] [Updated] (CHUKWA-707) Replace Chukwa collector with Apache Kafka
Date Tue, 29 Jul 2014 19:24:39 GMT


shreyas subramanya updated CHUKWA-707:

    Attachment: CHUKWA-707.patch2

I have modified the patch to create topics as CHUKWA-datatype-host. Each chukwa agent consumer
thread will only subscribe to topics matching its own host according to the regex CHUKWA-(.*)-currentHost.

The messages will have timestamp as their key. This design will also ensure that any streaming
software can perform computation on a datatype stream by subscribing to topics CHUKWA-datatype-.*

Currently the topic name and key format is defined by KafkaKeyFormatter class. This can be
made configurable in future when there is a need for it.

> Replace Chukwa collector with Apache Kafka
> ------------------------------------------
>                 Key: CHUKWA-707
>                 URL:
>             Project: Chukwa
>          Issue Type: New Feature
>            Reporter: Eric Yang
>            Assignee: shreyas subramanya
>         Attachments: CHUKWA-707.patch1, CHUKWA-707.patch2
> Chukwa collector has stopped evolving since 2010.  Newer framework has offer better features
of message queues, and Apache Kafka looks like a good replacement for Chukwa collector.
> Chukwa agent can implement a connector to Apache Kafka to replace Chukwa collector, and
HBase consumer to write data to HBase.  HICC REST API change to new HBase storage format.

This message was sent by Atlassian JIRA

View raw message