kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Skip Montanaro <skip.montan...@gmail.com>
Subject Re: Multiple brokers - do they share the load?
Date Mon, 04 Dec 2017 18:59:55 GMT
Ben wrote:

> If you set these configs at broker start they will automatically replicate
> newly created topics:
>
> default.replication.factor
> num.partitions=3

Thanks. (I assume the first config is missing a trailing "=3" bit).
Final question (I think). If I restart my brokers with those
configuration changes, will existing topics acquire the desired
replication factor, or will I need to do something else to tweak
existing topics?

Skip

> Ben
>
> On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro <skip.montanaro@gmail.com>
> wrote:
>
>> Are dynamically created topics thus never replicated?
>>
>> On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <Dave.Tauzell@surescripts.com>
>> wrote:
>>
>> > You then also need to set this up for each topic you create:
>> >
>> > > bin/kafka-topics.sh --create --zookeeper localhost:2181
>> > --replication-factor 3 --partitions 3 --topic my-replicated-topic
>> >
>> > -Dave
>> >
>> > -----Original Message-----
>> > From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
>> > Sent: Thursday, November 30, 2017 4:20 PM
>> > To: users@kafka.apache.org
>> > Subject: Re: Multiple brokers - do they share the load?
>> >
>> > > If you create a partitioned topic with at least 3 partitions then you
>> > will see your client connect to all of the brokers.  The client decides
>> > which partition a message should go to and then sends it directly to the
>> > broker that is the leader for that partition.  If you have replicated
>> > topics, then the brokers themselves will also be connected to one-another
>> > in order to replication messages.
>> >
>> > Thanks. So if I create three brokers all pointing to the same Zookeeper
>> > instance, then define
>> >
>> > offsets.topic.replication.factor=3
>> > transaction.state.log.replication.factor=3
>> >
>> > the brokers will find each other via Zookeeper and dynamically created
>> > topics (which is all I've used so far) will be properly replicated?
>> > Should I also bump transaction.state.log.min.isr above 1?
>> >
>> > Apologies for such basic questions. I'm still trying to absorb the
>> > documentation related to overall architecture and broker specifics.
>> >
>> > Skip
>> >
>> > This e-mail and any files transmitted with it are confidential, may
>> > contain sensitive information, and are intended solely for the use of the
>> > individual or entity to whom they are addressed. If you have received
>> this
>> > e-mail in error, please notify the sender by reply e-mail immediately and
>> > destroy all copies of the e-mail and any attachments.
>> >
>>
>
> --
>
>
> This email, including attachments, is private and confidential. If you have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
> 7digital Group plc. Registered office: 69 Wilson Street, London EC2A 2BB.
> Registered in England and Wales. Registered No. 04843573.

Mime
View raw message