kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip O'Toole <phi...@loggly.com>
Subject Re: Topic messages with partitions=1 stored on multiple brokers
Date Sat, 21 Sep 2013 18:27:15 GMT
It may something with your configuration. Try it with a simpler setup of one broker, see that
both topics appear on the broker, and build on your understanding and experimentation with
that setup. 

Philip 

On Sep 20, 2013, at 11:06 AM, Suchi Amalapurapu <suchi@bloomreach.com> wrote:

> Thx for your replies. Can u please clarify this further for me?
> 
> In 0.7, if the partition count is per broker, shouldn't each broker have a
> log directory for every topic?
> I don't see that though... Only some topics have logs in different brokers.
> 
> How is the consumer expected to handle this? My start sequence is as follows
> - start zookeeper
> - start producer
> - start broker1
> - start broker2
> - send message to topic1
> - send message to topic 2
> 
> log directory for topic1 is created on broker1 and broker2
> log directory for topic2 is created only on broker1
> 
> The consumer(kafka-storm spout) with a parallelism of one should ideally
> discover the brokers via zookeeper and in this case should read the
> messages in the correct sequence from both?
> Suchi
> 
> 
> On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <suchi@bloomreach.com>wrote:
> 
>> Yes I understand that. I am letting the producer/consumer use zookeeper to
>> discover brokers.
>> I can clearly see in the logs(brokers) that both the brokers create a new
>> topic log for the same topic.
>> 
>> The brokers are in different availability zones. Does that matter?
>> Suchi
>> 
>> 
>> On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <philip@loggly.com> wrote:
>> 
>>> Seems to me you are confusing partitions and brokers. Partition count has
>>> nothing to do with the number of brokers to which a message a sent -- just
>>> the number of partitions into which that message will be split when it
>>> gets
>>> to a broker.
>>> 
>>> You need to explicitly set the destination brokers in the Producer, not
>>> via
>>> setting "partition" count.
>>> 
>>> Philip
>>> 
>>> 
>>> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu <suchi@bloomreach.com
>>>> wrote:
>>> 
>>>> My current setup has 2 brokers, multiple topics, each topic with 1
>>>> partition, kafka 0.7.1
>>>> Some topics are logged onto multiple brokers inspite of partitions set
>>> to
>>>> 1. Kafka 0.7.1 does not have replication turned on.
>>>> Why are the messages being logged onto multiple brokers when the number
>>> of
>>>> partitions is set to 1?
>>>> I have a consumer with parallelism set to 1 and its missing some of
>>> these
>>>> messages.
>>>> Suchi
>> 
>> 

Mime
View raw message