kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Design questions and your opinion and suggestions are appreciated
Date Thu, 13 Dec 2012 06:11:25 GMT
Right, in 0.7, partitioner is only supported for zookeeper based producer.

Thanks,

Jun

On Wed, Dec 12, 2012 at 4:44 PM, Subhash Agrawal <sagrawal@actuate.com>wrote:

> Hi,
>
> I have one more question related to this setup. If we don't use zookeeper
> in single Kafka broker setup, what will be the loss in
> terms of functionality? I noticed in ProducerConfig.scala file, that if
> brokerlist is configured, then it ignore partitioner class
> configuration.
>
> Thanks
> Subhash Agrawal
>
>
> -----Original Message-----
> From: Jamie Wang [mailto:jamie.wang@actuate.com]
> Sent: Wednesday, December 12, 2012 2:26 PM
> To: users@kafka.apache.org
> Subject: Design questions and your opinion and suggestions are appreciated
>
> Hi,
>
> We are incorporating Kafka as part of our centralized log aggregation
> service for a clustered server system. Our current thinking on the design
> as follows:  Each of our clustered server will produce 8 different logs. A
> typical cluster will have about 4 or 5 nodes right now. Our design is to
> create a topic per each type of log and the number of partitions within
> each topic will equal to the number of nodes in the cluster.  Therefore
> with a cluster of 5 nodes, we will have 8 topics and 5 partitions within
> each topic.
>
> On each of the node, we have a producer that has 8 threads. Each thread
> will always write to the same topic and partition.
>
> On the consumer side, we plan to have an aggregation client that has 8
> threads. Each thread will pull from the same topic using round robin
> fashion to cycle through each partition within the topic.
>
> Couple of design questions we would like to hear your opinion and any
> suggestions are appreciated!
>
> 1) If we dynamically adds another node into the cluster, base on our
> design above, we would want to create a new partition and add to each of
> the 8 topics. Is this doable and how do I do that?
>
> 2) Can a producer send message to a specific partition? Base on our
> limited understanding so far, in a clustered kafka system, a producer sends
> message to the cluster, how does the messages being distributed among
> different brokers/partitions?
>
> 3) In our system, we also have a c++ process manager that's monitoring
> over kafka broker process. is there a heartbeat in the broker that we can
> ping? Or where can I find the C++ API?
>
> 4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a
> command port that I can send a shutdown command or how to trigger this
> shutdown method from an external process such as our process monitor.
>
> 5) We are thinking if in the future our cluster grow much larger. We
> possibly have to add additional kafka broker.  What are the parameters we
> should look for to determine at that point we should add another broker?
>
> Thanks in advance for your time and help.
> Jamie
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message