kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Topic messages with partitions=1 stored on multiple brokers
Date Sun, 22 Sep 2013 15:42:16 GMT
Hi Suchi,

In 0.7 a producer would send to one broker within a session, hence you may
want to send enough messages to see them shown up on both brokers.

Guozhang


On Sat, Sep 21, 2013 at 11:27 AM, Philip O'Toole <philip@loggly.com> wrote:

> It may something with your configuration. Try it with a simpler setup of
> one broker, see that both topics appear on the broker, and build on your
> understanding and experimentation with that setup.
>
> Philip
>
> On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <suchi@bloomreach.com>
> wrote:
>
> > Thx for your replies. Can u please clarify this further for me?
> >
> > In 0.7, if the partition count is per broker, shouldn't each broker have
> a
> > log directory for every topic?
> > I don't see that though... Only some topics have logs in different
> brokers.
> >
> > How is the consumer expected to handle this? My start sequence is as
> follows
> > - start zookeeper
> > - start producer
> > - start broker1
> > - start broker2
> > - send message to topic1
> > - send message to topic 2
> >
> > log directory for topic1 is created on broker1 and broker2
> > log directory for topic2 is created only on broker1
> >
> > The consumer(kafka-storm spout) with a parallelism of one should ideally
> > discover the brokers via zookeeper and in this case should read the
> > messages in the correct sequence from both?
> > Suchi
> >
> >
> > On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <
> suchi@bloomreach.com>wrote:
> >
> >> Yes I understand that. I am letting the producer/consumer use zookeeper
> to
> >> discover brokers.
> >> I can clearly see in the logs(brokers) that both the brokers create a
> new
> >> topic log for the same topic.
> >>
> >> The brokers are in different availability zones. Does that matter?
> >> Suchi
> >>
> >>
> >> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <philip@loggly.com>
> wrote:
> >>
> >>> Seems to me you are confusing partitions and brokers. Partition count
> has
> >>> nothing to do with the number of brokers to which a message a sent --
> just
> >>> the number of partitions into which that message will be split when it
> >>> gets
> >>> to a broker.
> >>>
> >>> You need to explicitly set the destination brokers in the Producer, not
> >>> via
> >>> setting "partition" count.
> >>>
> >>> Philip
> >>>
> >>>
> >>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <
> suchi@bloomreach.com
> >>>> wrote:
> >>>
> >>>> My current setup has 2 brokers, multiple topics, each topic with 1
> >>>> partition, kafka 0.7.1
> >>>> Some topics are logged onto multiple brokers inspite of partitions set
> >>> to
> >>>> 1. Kafka 0.7.1 does not have replication turned on.
> >>>> Why are the messages being logged onto multiple brokers when the
> number
> >>> of
> >>>> partitions is set to 1?
> >>>> I have a consumer with parallelism set to 1 and its missing some of
> >>> these
> >>>> messages.
> >>>> Suchi
> >>
> >>
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message