kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Wang <jamie.w...@actuate.com>
Subject RE: Design questions and your opinion and suggestions are appreciated
Date Thu, 13 Dec 2012 18:39:23 GMT
Thanks for the pointers. 

To clarify: my question 1) actually is referring to if I can dynamically create/add new partition
to an existing topic on an existing broker?

For pointer 2) you seem to suggest a newer version of the broker will have a different behavior.
what would this new behavior like and what version will have this new behavior?

Thanks again.
Jamie


-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Wednesday, December 12, 2012 10:10 PM
To: users@kafka.apache.org
Subject: Re: Design questions and your opinion and suggestions are appreciated

1) In 0.7, a topic exists on every broker. So new partitions will be
automatically added on the new broker.
2) It's possible, if you use the partitioner. However, be aware in 0.7, if
a broker goes down, all partitions on it are down and you won't be able to
write to them.
3) You can try to connect to the Kafka port.
4) No. Just do kill -15
5) It depends. You may want to add a broker if you don't have enough
storage space, don't have enough I/Os, or don't have enough network
bandwidth.

Thanks,

Jun


On Wed, Dec 12, 2012 at 2:25 PM, Jamie Wang <jamie.wang@actuate.com> wrote:

> Hi,
>
> We are incorporating Kafka as part of our centralized log aggregation
> service for a clustered server system. Our current thinking on the design
> as follows:  Each of our clustered server will produce 8 different logs. A
> typical cluster will have about 4 or 5 nodes right now. Our design is to
> create a topic per each type of log and the number of partitions within
> each topic will equal to the number of nodes in the cluster.  Therefore
> with a cluster of 5 nodes, we will have 8 topics and 5 partitions within
> each topic.
>
> On each of the node, we have a producer that has 8 threads. Each thread
> will always write to the same topic and partition.
>
> On the consumer side, we plan to have an aggregation client that has 8
> threads. Each thread will pull from the same topic using round robin
> fashion to cycle through each partition within the topic.
>
> Couple of design questions we would like to hear your opinion and any
> suggestions are appreciated!
>
> 1) If we dynamically adds another node into the cluster, base on our
> design above, we would want to create a new partition and add to each of
> the 8 topics. Is this doable and how do I do that?
>
> 2) Can a producer send message to a specific partition? Base on our
> limited understanding so far, in a clustered kafka system, a producer sends
> message to the cluster, how does the messages being distributed among
> different brokers/partitions?
>
> 3) In our system, we also have a c++ process manager that's monitoring
> over kafka broker process. is there a heartbeat in the broker that we can
> ping? Or where can I find the C++ API?
>
> 4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a
> command port that I can send a shutdown command or how to trigger this
> shutdown method from an external process such as our process monitor.
>
> 5) We are thinking if in the future our cluster grow much larger. We
> possibly have to add additional kafka broker.  What are the parameters we
> should look for to determine at that point we should add another broker?
>
> Thanks in advance for your time and help.
> Jamie
>
>

Mime
View raw message