kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Topic messages with partitions=1 stored on multiple brokers
Date Mon, 23 Sep 2013 00:28:15 GMT
This could be related to
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-IamusingtheZKbasedproducerin0.7andIseedataonlyproducedonsomeofthebrokers%2Cbutnotall%2Cwhy%3F

Thanks,

Jun


On Fri, Sep 20, 2013 at 11:06 AM, Suchi Amalapurapu <suchi@bloomreach.com>wrote:

> Thx for your replies. Can u please clarify this further for me?
>
> In 0.7, if the partition count is per broker, shouldn't each broker have a
> log directory for every topic?
> I don't see that though... Only some topics have logs in different brokers.
>
> How is the consumer expected to handle this? My start sequence is as
> follows
> - start zookeeper
> - start producer
> - start broker1
> - start broker2
> - send message to topic1
> - send message to topic 2
>
> log directory for topic1 is created on broker1 and broker2
> log directory for topic2 is created only on broker1
>
> The consumer(kafka-storm spout) with a parallelism of one should ideally
> discover the brokers via zookeeper and in this case should read the
> messages in the correct sequence from both?
> Suchi
>
>
> On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <suchi@bloomreach.com
> >wrote:
>
> > Yes I understand that. I am letting the producer/consumer use zookeeper
> to
> > discover brokers.
> > I can clearly see in the logs(brokers) that both the brokers create a new
> > topic log for the same topic.
> >
> > The brokers are in different availability zones. Does that matter?
> > Suchi
> >
> >
> > On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <philip@loggly.com>
> wrote:
> >
> >> Seems to me you are confusing partitions and brokers. Partition count
> has
> >> nothing to do with the number of brokers to which a message a sent --
> just
> >> the number of partitions into which that message will be split when it
> >> gets
> >> to a broker.
> >>
> >> You need to explicitly set the destination brokers in the Producer, not
> >> via
> >> setting "partition" count.
> >>
> >> Philip
> >>
> >>
> >> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <
> suchi@bloomreach.com
> >> >wrote:
> >>
> >> > My current setup has 2 brokers, multiple topics, each topic with 1
> >> > partition, kafka 0.7.1
> >> > Some topics are logged onto multiple brokers inspite of partitions set
> >> to
> >> > 1. Kafka 0.7.1 does not have replication turned on.
> >> > Why are the messages being logged onto multiple brokers when the
> number
> >> of
> >> > partitions is set to 1?
> >> > I have a consumer with parallelism set to 1 and its missing some of
> >> these
> >> > messages.
> >> > Suchi
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message