kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed
Date Fri, 16 Nov 2018 15:53:32 GMT
> if we are not using EBS

In that case, what's the point of keeping the broker id? The data will be
lost anyway right?


On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cnu.text@gmail.com> wrote:

> Yes, I understand we need to specify the required broker.id in
> server.properties/meta.properties file in-order to retain the broker.id
> which was failed.
>
> We can write custom script to query the zookeeper on launch process to get
> the broker.id needs to be set.
>
> Do we have any standards/script/way defined/preferred for doing this or
> suggested by Kafka experts if we are not using EBS?
>
> Thanks and Regards,
> Srinivas
>
> On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <kaufman@confluent.io> wrote:
>
> > Srinivas,
> >
> > You might want to check out the AWS best practices from this blog post:
> >
> >
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
> >
> > Kafka broker ids by default are auto-generated, but you can also specify
> > values (in server.properties file). By having static broker ids, it's
> > easier to recover in general. The blog post above mentions it in more
> > details.
> >
> > Good luck.
> >
> > On Thu, Nov 15, 2018 at 8:09 AM amit pal <amit5624@gmail.com> wrote:
> >
> > > If at all you were hell bent on doing this, you could use zookeeper to
> > find
> > > out the health of current brokers along with their broker id. That
> should
> > > help re-spin/start the unhealthy instance with same instance id.
> > >
> > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cnu.text@gmail.com>
> > > wrote:
> > >
> > > > having all this stored in DB is getting too complicated, especially
> > with
> > > > instance level storage and not EBS.
> > > >
> > > > I am sure there should be easy way to retain the old broker.id for
> new
> > > AWS
> > > > instance spun-up for auto replication.
> > > >
> > > > Any other ideas/help is appreciated.
> > > >
> > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <eno.thereska@gmail.com
> >
> > > > wrote:
> > > >
> > > > > The general answer depends on what control plane software is taking
> > > care
> > > > of
> > > > > your Kafka deployment. You probably have a layer that launches
> Kafka
> > > > > instances and monitors their health, right? If so, that layer
> should
> > > take
> > > > > care of the mapping between instances and broker IDs and keep that
> > in a
> > > > > table persisted somewhere (e.g., DynamoDB).
> > > > >
> > > > > Eno
> > > > >
> > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
> cnu.text@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > EBS is one of the option. But we use instance level storage
where
> > we
> > > > > loose
> > > > > > all data as soon as we have a broker failed in AWS.
> > > > > >
> > > > > > In such scenario, anyone has better launch script or cofiguration
> > can
> > > > be
> > > > > > executed on new broker to retain the old id not conflicting
with
> > > > existing
> > > > > > broker ids.
> > > > > >
> > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > > andrey.dyachkov@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > You can attach EBS volume, which will store data and
> > metadata(e.g.
> > > > > broker
> > > > > > > id), and then attach it to the new AWS instance and start
> Kafka,
> > it
> > > > > will
> > > > > > > pick the broker id plus you won’t need to rebalance the
> cluster.
> > > > > > >
> > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > > nareshgoud.dulam@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Static IP. Buying static IP may help. I am not aws
expert
> > > > > > > >
> > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > > cnu.text@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Kafka experts,
> > > > > > > > >
> > > > > > > > > We are running Kafka on AWS, main question is
what is the
> > best
> > > > way
> > > > > to
> > > > > > > > > retain broker.id on new instance spun-up in-place
of
> > > > > instance/broker
> > > > > > > > > failed.
> > > > > > > > >
> > > > > > > > > We are currently running Kafka in AWS with broker.id
gets
> > auto
> > > > > > > > generated.
> > > > > > > > > But we are having issues when a broker is failed,
new
> > > > > broker/instance
> > > > > > > > > spun-up in AWS get assigned with new broker.id.
The issue
> > is,
> > > > with
> > > > > > > this
> > > > > > > > > approach, we need to re-assign the topics/replications
on
> to
> > > the
> > > > > new
> > > > > > > > broker
> > > > > > > > > manually.
> > > > > > > > >
> > > > > > > > > We learned that, replication can be auto resolved
by Kafka,
> > if
> > > we
> > > > > can
> > > > > > > > > manage to get the same broker.id on the new AWS
instance
> > > spun-up
> > > > > > > > in-place
> > > > > > > > > of failed broker/instance.
> > > > > > > > >
> > > > > > > > > I have read, we can set broker.id.generation.enable=
false,
> > but
> > > > > what
> > > > > > is
> > > > > > > > the
> > > > > > > > > best way to identify and retain the broker.id?
Any
> > links/help
> > > is
> > > > > > > > > appreciated.
> > > > > > > > > Thanks and Regards,
> > > > > > > > > Cnu
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks,
> > > > > > > > Naresh
> > > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Andrey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Amit Pal
> > >
> >
> >
> > --
> > Kaufman Ng
> > +1 646 961 8063
> > Solutions Architect | Confluent | www.confluent.io
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message