kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinivas Rapolu <cnu.t...@gmail.com>
Subject Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed
Date Thu, 15 Nov 2018 12:37:50 GMT
having all this stored in DB is getting too complicated, especially with
instance level storage and not EBS.

I am sure there should be easy way to retain the old broker.id for new AWS
instance spun-up for auto replication.

Any other ideas/help is appreciated.

On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <eno.thereska@gmail.com> wrote:

> The general answer depends on what control plane software is taking care of
> your Kafka deployment. You probably have a layer that launches Kafka
> instances and monitors their health, right? If so, that layer should take
> care of the mapping between instances and broker IDs and keep that in a
> table persisted somewhere (e.g., DynamoDB).
>
> Eno
>
> On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cnu.text@gmail.com>
> wrote:
>
> > EBS is one of the option. But we use instance level storage where we
> loose
> > all data as soon as we have a broker failed in AWS.
> >
> > In such scenario, anyone has better launch script or cofiguration can be
> > executed on new broker to retain the old id not conflicting with existing
> > broker ids.
> >
> > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> andrey.dyachkov@gmail.com
> > wrote:
> >
> > > You can attach EBS volume, which will store data and metadata(e.g.
> broker
> > > id), and then attach it to the new AWS instance and start Kafka, it
> will
> > > pick the broker id plus you won’t need to rebalance the cluster.
> > >
> > > On Wed 14. Nov 2018 at 19:48, naresh Goud <nareshgoud.dulam@gmail.com>
> > > wrote:
> > >
> > > > Static IP. Buying static IP may help. I am not aws expert
> > > >
> > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cnu.text@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hello Kafka experts,
> > > > >
> > > > > We are running Kafka on AWS, main question is what is the best way
> to
> > > > > retain broker.id on new instance spun-up in-place of
> instance/broker
> > > > > failed.
> > > > >
> > > > > We are currently running Kafka in AWS with broker.id gets auto
> > > > generated.
> > > > > But we are having issues when a broker is failed, new
> broker/instance
> > > > > spun-up in AWS get assigned with new broker.id. The issue is, with
> > > this
> > > > > approach, we need to re-assign the topics/replications on to the
> new
> > > > broker
> > > > > manually.
> > > > >
> > > > > We learned that, replication can be auto resolved by Kafka, if we
> can
> > > > > manage to get the same broker.id on the new AWS instance spun-up
> > > > in-place
> > > > > of failed broker/instance.
> > > > >
> > > > > I have read, we can set broker.id.generation.enable= false, but
> what
> > is
> > > > the
> > > > > best way to identify and retain the broker.id? Any links/help is
> > > > > appreciated.
> > > > > Thanks and Regards,
> > > > > Cnu
> > > > >
> > > > --
> > > > Thanks,
> > > > Naresh
> > > > www.linkedin.com/in/naresh-dulam
> > > > http://hadoopandspark.blogspot.com/
> > > >
> > > --
> > > Thanks,
> > > Andrey
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message