kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinivas Rapolu <cnu.t...@gmail.com>
Subject Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed
Date Fri, 16 Nov 2018 18:27:30 GMT
 By looking at the responses: It looks like there is *NO *standard
solution, we will have to come up with some custom/manual script..... any
one has any other suggestions?

On Fri, Nov 16, 2018 at 11:18 AM Srinivas Rapolu <cnu.text@gmail.com> wrote:

> Yes, that is the whole point of discussion, replication factor is more
> than 1. So keeping it in instance storage, we will lose the one copy of
> data, but retaining broker-id will auto-replicate the topic/partitions to
> new instance as zookeeper has brokerid-topic-partition assignment mapping.
>
> By looking at the responses: It looks like there is standard solution, we
> will have to come up with some manual script..... any one has any other
> suggestions?
>
> On Fri, Nov 16, 2018 at 10:08 AM Kaufman Ng <kaufman@confluent.io> wrote:
>
>> Yes data will be lost when you use instance storage. But keeping the
>> *same*
>> broker id will allow the new broker instance to easily replicate data from
>> other brokers, therefore restoring back to the previous fully replicated
>> state.
>>
>> Srinivas, do your topics have replicator factor > 1? If yes, data can be
>> recovered from other brokers.
>>
>> On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ryannedolan@gmail.com>
>> wrote:
>>
>> > > if we are not using EBS
>> >
>> > In that case, what's the point of keeping the broker id? The data will
>> be
>> > lost anyway right?
>> >
>> >
>> > On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cnu.text@gmail.com>
>> wrote:
>> >
>> > > Yes, I understand we need to specify the required broker.id in
>> > > server.properties/meta.properties file in-order to retain the
>> broker.id
>> > > which was failed.
>> > >
>> > > We can write custom script to query the zookeeper on launch process to
>> > get
>> > > the broker.id needs to be set.
>> > >
>> > > Do we have any standards/script/way defined/preferred for doing this
>> or
>> > > suggested by Kafka experts if we are not using EBS?
>> > >
>> > > Thanks and Regards,
>> > > Srinivas
>> > >
>> > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <kaufman@confluent.io>
>> wrote:
>> > >
>> > > > Srinivas,
>> > > >
>> > > > You might want to check out the AWS best practices from this blog
>> post:
>> > > >
>> > > >
>> > >
>> >
>> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
>> > > >
>> > > > Kafka broker ids by default are auto-generated, but you can also
>> > specify
>> > > > values (in server.properties file). By having static broker ids,
>> it's
>> > > > easier to recover in general. The blog post above mentions it in
>> more
>> > > > details.
>> > > >
>> > > > Good luck.
>> > > >
>> > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <amit5624@gmail.com>
>> wrote:
>> > > >
>> > > > > If at all you were hell bent on doing this, you could use
>> zookeeper
>> > to
>> > > > find
>> > > > > out the health of current brokers along with their broker id.
That
>> > > should
>> > > > > help re-spin/start the unhealthy instance with same instance
id.
>> > > > >
>> > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <
>> cnu.text@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > having all this stored in DB is getting too complicated,
>> especially
>> > > > with
>> > > > > > instance level storage and not EBS.
>> > > > > >
>> > > > > > I am sure there should be easy way to retain the old broker.id
>> for
>> > > new
>> > > > > AWS
>> > > > > > instance spun-up for auto replication.
>> > > > > >
>> > > > > > Any other ideas/help is appreciated.
>> > > > > >
>> > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <
>> > eno.thereska@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > The general answer depends on what control plane software
is
>> > taking
>> > > > > care
>> > > > > > of
>> > > > > > > your Kafka deployment. You probably have a layer that
launches
>> > > Kafka
>> > > > > > > instances and monitors their health, right? If so,
that layer
>> > > should
>> > > > > take
>> > > > > > > care of the mapping between instances and broker IDs
and keep
>> > that
>> > > > in a
>> > > > > > > table persisted somewhere (e.g., DynamoDB).
>> > > > > > >
>> > > > > > > Eno
>> > > > > > >
>> > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
>> > > cnu.text@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > EBS is one of the option. But we use instance
level storage
>> > where
>> > > > we
>> > > > > > > loose
>> > > > > > > > all data as soon as we have a broker failed in
AWS.
>> > > > > > > >
>> > > > > > > > In such scenario, anyone has better launch script
or
>> > cofiguration
>> > > > can
>> > > > > > be
>> > > > > > > > executed on new broker to retain the old id not
conflicting
>> > with
>> > > > > > existing
>> > > > > > > > broker ids.
>> > > > > > > >
>> > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov
<
>> > > > > > > andrey.dyachkov@gmail.com
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > You can attach EBS volume, which will store
data and
>> > > > metadata(e.g.
>> > > > > > > broker
>> > > > > > > > > id), and then attach it to the new AWS instance
and start
>> > > Kafka,
>> > > > it
>> > > > > > > will
>> > > > > > > > > pick the broker id plus you won’t need
to rebalance the
>> > > cluster.
>> > > > > > > > >
>> > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud
<
>> > > > > > nareshgoud.dulam@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Static IP. Buying static IP may help.
I am not aws
>> expert
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas
Rapolu <
>> > > > > > cnu.text@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hello Kafka experts,
>> > > > > > > > > > >
>> > > > > > > > > > > We are running Kafka on AWS, main
question is what is
>> the
>> > > > best
>> > > > > > way
>> > > > > > > to
>> > > > > > > > > > > retain broker.id on new instance
spun-up in-place of
>> > > > > > > instance/broker
>> > > > > > > > > > > failed.
>> > > > > > > > > > >
>> > > > > > > > > > > We are currently running Kafka
in AWS with broker.id
>> > gets
>> > > > auto
>> > > > > > > > > > generated.
>> > > > > > > > > > > But we are having issues when a
broker is failed, new
>> > > > > > > broker/instance
>> > > > > > > > > > > spun-up in AWS get assigned with
new broker.id. The
>> > issue
>> > > > is,
>> > > > > > with
>> > > > > > > > > this
>> > > > > > > > > > > approach, we need to re-assign
the
>> topics/replications on
>> > > to
>> > > > > the
>> > > > > > > new
>> > > > > > > > > > broker
>> > > > > > > > > > > manually.
>> > > > > > > > > > >
>> > > > > > > > > > > We learned that, replication can
be auto resolved by
>> > Kafka,
>> > > > if
>> > > > > we
>> > > > > > > can
>> > > > > > > > > > > manage to get the same broker.id
on the new AWS
>> instance
>> > > > > spun-up
>> > > > > > > > > > in-place
>> > > > > > > > > > > of failed broker/instance.
>> > > > > > > > > > >
>> > > > > > > > > > > I have read, we can set broker.id.generation.enable=
>> > false,
>> > > > but
>> > > > > > > what
>> > > > > > > > is
>> > > > > > > > > > the
>> > > > > > > > > > > best way to identify and retain
the broker.id? Any
>> > > > links/help
>> > > > > is
>> > > > > > > > > > > appreciated.
>> > > > > > > > > > > Thanks and Regards,
>> > > > > > > > > > > Cnu
>> > > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Naresh
>> > > > > > > > > > www.linkedin.com/in/naresh-dulam
>> > > > > > > > > > http://hadoopandspark.blogspot.com/
>> > > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Thanks,
>> > > > > > > > > Andrey
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Thanks and Regards,
>> > > > > Amit Pal
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Kaufman Ng
>> > > > +1 646 961 8063
>> > > > Solutions Architect | Confluent | www.confluent.io
>> > > >
>> > >
>> >
>>
>>
>> --
>> Kaufman Ng
>> +1 646 961 8063
>> Solutions Architect | Confluent | www.confluent.io
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message