For 2), the next Kafka release will support replication. So a partition is
still available when a single broker is down.

For 1), in 0.7, partitions are automatically added to new brokers. In 0.8,
you will need to run a command to change # partitions.

Thanks,

Jun

On Thu, Dec 13, 2012 at 10:39 AM, Jamie Wang <jamie.wang@actuate.com> wrote:

> Thanks for the pointers.
>
> To clarify: my question 1) actually is referring to if I can dynamically
> create/add new partition to an existing topic on an existing broker?
>
> For pointer 2) you seem to suggest a newer version of the broker will have
> a different behavior. what would this new behavior like and what version
> will have this new behavior?
>
> Thanks again.
> Jamie
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, December 12, 2012 10:10 PM
> To: users@kafka.apache.org
> Subject: Re: Design questions and your opinion and suggestions are
> appreciated
>
> 1) In 0.7, a topic exists on every broker. So new partitions will be
> automatically added on the new broker.
> 2) It's possible, if you use the partitioner. However, be aware in 0.7, if
> a broker goes down, all partitions on it are down and you won't be able to
> write to them.
> 3) You can try to connect to the Kafka port.
> 4) No. Just do kill -15
> 5) It depends. You may want to add a broker if you don't have enough
> storage space, don't have enough I/Os, or don't have enough network
> bandwidth.
>
> Thanks,
>
> Jun
>
>
> On Wed, Dec 12, 2012 at 2:25 PM, Jamie Wang <jamie.wang@actuate.com>
> wrote:
>
> > Hi,
> >
> > We are incorporating Kafka as part of our centralized log aggregation
> > service for a clustered server system. Our current thinking on the design
> > as follows:  Each of our clustered server will produce 8 different logs.
> A
> > typical cluster will have about 4 or 5 nodes right now. Our design is to
> > create a topic per each type of log and the number of partitions within
> > each topic will equal to the number of nodes in the cluster.  Therefore
> > with a cluster of 5 nodes, we will have 8 topics and 5 partitions within
> > each topic.
> >
> > On each of the node, we have a producer that has 8 threads. Each thread
> > will always write to the same topic and partition.
> >
> > On the consumer side, we plan to have an aggregation client that has 8
> > threads. Each thread will pull from the same topic using round robin
> > fashion to cycle through each partition within the topic.
> >
> > Couple of design questions we would like to hear your opinion and any
> > suggestions are appreciated!
> >
> > 1) If we dynamically adds another node into the cluster, base on our
> > design above, we would want to create a new partition and add to each of
> > the 8 topics. Is this doable and how do I do that?
> >
> > 2) Can a producer send message to a specific partition? Base on our
> > limited understanding so far, in a clustered kafka system, a producer
> sends
> > message to the cluster, how does the messages being distributed among
> > different brokers/partitions?
> >
> > 3) In our system, we also have a c++ process manager that's monitoring
> > over kafka broker process. is there a heartbeat in the broker that we can
> > ping? Or where can I find the C++ API?
> >
> > 4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a
> > command port that I can send a shutdown command or how to trigger this
> > shutdown method from an external process such as our process monitor.
> >
> > 5) We are thinking if in the future our cluster grow much larger. We
> > possibly have to add additional kafka broker.  What are the parameters we
> > should look for to determine at that point we should add another broker?
> >
> > Thanks in advance for your time and help.
> > Jamie
> >
> >
>