kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Wang <jamie.w...@actuate.com>
Subject Design questions and your opinion and suggestions are appreciated
Date Wed, 12 Dec 2012 22:25:54 GMT
Hi,

We are incorporating Kafka as part of our centralized log aggregation service for a clustered
server system. Our current thinking on the design as follows:  Each of our clustered server
will produce 8 different logs. A typical cluster will have about 4 or 5 nodes right now. Our
design is to create a topic per each type of log and the number of partitions within each
topic will equal to the number of nodes in the cluster.  Therefore with a cluster of 5 nodes,
we will have 8 topics and 5 partitions within each topic.

On each of the node, we have a producer that has 8 threads. Each thread will always write
to the same topic and partition.

On the consumer side, we plan to have an aggregation client that has 8 threads. Each thread
will pull from the same topic using round robin fashion to cycle through each partition within
the topic.

Couple of design questions we would like to hear your opinion and any suggestions are appreciated!

1) If we dynamically adds another node into the cluster, base on our design above, we would
want to create a new partition and add to each of the 8 topics. Is this doable and how do
I do that?

2) Can a producer send message to a specific partition? Base on our limited understanding
so far, in a clustered kafka system, a producer sends message to the cluster, how does the
messages being distributed among different brokers/partitions?

3) In our system, we also have a c++ process manager that's monitoring over kafka broker process.
is there a heartbeat in the broker that we can ping? Or where can I find the C++ API?

4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a command port that I
can send a shutdown command or how to trigger this shutdown method from an external process
such as our process monitor.

5) We are thinking if in the future our cluster grow much larger. We possibly have to add
additional kafka broker.  What are the parameters we should look for to determine at that
point we should add another broker?

Thanks in advance for your time and help.
Jamie


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message