kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinivas Rapolu <cnu.t...@gmail.com>
Subject Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed
Date Fri, 16 Nov 2018 18:18:57 GMT
Yes, that is the whole point of discussion, replication factor is more than
1. So keeping it in instance storage, we will lose the one copy of data,
but retaining broker-id will auto-replicate the topic/partitions to new
instance as zookeeper has brokerid-topic-partition assignment mapping.

By looking at the responses: It looks like there is standard solution, we
will have to come up with some manual script..... any one has any other
suggestions?

On Fri, Nov 16, 2018 at 10:08 AM Kaufman Ng <kaufman@confluent.io> wrote:

> Yes data will be lost when you use instance storage. But keeping the *same*
> broker id will allow the new broker instance to easily replicate data from
> other brokers, therefore restoring back to the previous fully replicated
> state.
>
> Srinivas, do your topics have replicator factor > 1? If yes, data can be
> recovered from other brokers.
>
> On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
>
> > > if we are not using EBS
> >
> > In that case, what's the point of keeping the broker id? The data will be
> > lost anyway right?
> >
> >
> > On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cnu.text@gmail.com>
> wrote:
> >
> > > Yes, I understand we need to specify the required broker.id in
> > > server.properties/meta.properties file in-order to retain the
> broker.id
> > > which was failed.
> > >
> > > We can write custom script to query the zookeeper on launch process to
> > get
> > > the broker.id needs to be set.
> > >
> > > Do we have any standards/script/way defined/preferred for doing this or
> > > suggested by Kafka experts if we are not using EBS?
> > >
> > > Thanks and Regards,
> > > Srinivas
> > >
> > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <kaufman@confluent.io>
> wrote:
> > >
> > > > Srinivas,
> > > >
> > > > You might want to check out the AWS best practices from this blog
> post:
> > > >
> > > >
> > >
> >
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
> > > >
> > > > Kafka broker ids by default are auto-generated, but you can also
> > specify
> > > > values (in server.properties file). By having static broker ids, it's
> > > > easier to recover in general. The blog post above mentions it in more
> > > > details.
> > > >
> > > > Good luck.
> > > >
> > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <amit5624@gmail.com> wrote:
> > > >
> > > > > If at all you were hell bent on doing this, you could use zookeeper
> > to
> > > > find
> > > > > out the health of current brokers along with their broker id. That
> > > should
> > > > > help re-spin/start the unhealthy instance with same instance id.
> > > > >
> > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <
> cnu.text@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > having all this stored in DB is getting too complicated,
> especially
> > > > with
> > > > > > instance level storage and not EBS.
> > > > > >
> > > > > > I am sure there should be easy way to retain the old broker.id
> for
> > > new
> > > > > AWS
> > > > > > instance spun-up for auto replication.
> > > > > >
> > > > > > Any other ideas/help is appreciated.
> > > > > >
> > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <
> > eno.thereska@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > The general answer depends on what control plane software
is
> > taking
> > > > > care
> > > > > > of
> > > > > > > your Kafka deployment. You probably have a layer that launches
> > > Kafka
> > > > > > > instances and monitors their health, right? If so, that
layer
> > > should
> > > > > take
> > > > > > > care of the mapping between instances and broker IDs and
keep
> > that
> > > > in a
> > > > > > > table persisted somewhere (e.g., DynamoDB).
> > > > > > >
> > > > > > > Eno
> > > > > > >
> > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
> > > cnu.text@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > EBS is one of the option. But we use instance level
storage
> > where
> > > > we
> > > > > > > loose
> > > > > > > > all data as soon as we have a broker failed in AWS.
> > > > > > > >
> > > > > > > > In such scenario, anyone has better launch script
or
> > cofiguration
> > > > can
> > > > > > be
> > > > > > > > executed on new broker to retain the old id not conflicting
> > with
> > > > > > existing
> > > > > > > > broker ids.
> > > > > > > >
> > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > > > > andrey.dyachkov@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > You can attach EBS volume, which will store data
and
> > > > metadata(e.g.
> > > > > > > broker
> > > > > > > > > id), and then attach it to the new AWS instance
and start
> > > Kafka,
> > > > it
> > > > > > > will
> > > > > > > > > pick the broker id plus you won’t need to rebalance
the
> > > cluster.
> > > > > > > > >
> > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > > > > nareshgoud.dulam@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Static IP. Buying static IP may help. I
am not aws expert
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas
Rapolu <
> > > > > > cnu.text@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello Kafka experts,
> > > > > > > > > > >
> > > > > > > > > > > We are running Kafka on AWS, main question
is what is
> the
> > > > best
> > > > > > way
> > > > > > > to
> > > > > > > > > > > retain broker.id on new instance spun-up
in-place of
> > > > > > > instance/broker
> > > > > > > > > > > failed.
> > > > > > > > > > >
> > > > > > > > > > > We are currently running Kafka in AWS
with broker.id
> > gets
> > > > auto
> > > > > > > > > > generated.
> > > > > > > > > > > But we are having issues when a broker
is failed, new
> > > > > > > broker/instance
> > > > > > > > > > > spun-up in AWS get assigned with new
broker.id. The
> > issue
> > > > is,
> > > > > > with
> > > > > > > > > this
> > > > > > > > > > > approach, we need to re-assign the
topics/replications
> on
> > > to
> > > > > the
> > > > > > > new
> > > > > > > > > > broker
> > > > > > > > > > > manually.
> > > > > > > > > > >
> > > > > > > > > > > We learned that, replication can be
auto resolved by
> > Kafka,
> > > > if
> > > > > we
> > > > > > > can
> > > > > > > > > > > manage to get the same broker.id on
the new AWS
> instance
> > > > > spun-up
> > > > > > > > > > in-place
> > > > > > > > > > > of failed broker/instance.
> > > > > > > > > > >
> > > > > > > > > > > I have read, we can set broker.id.generation.enable=
> > false,
> > > > but
> > > > > > > what
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > best way to identify and retain the
broker.id? Any
> > > > links/help
> > > > > is
> > > > > > > > > > > appreciated.
> > > > > > > > > > > Thanks and Regards,
> > > > > > > > > > > Cnu
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Thanks,
> > > > > > > > > > Naresh
> > > > > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks,
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > > Amit Pal
> > > > >
> > > >
> > > >
> > > > --
> > > > Kaufman Ng
> > > > +1 646 961 8063
> > > > Solutions Architect | Confluent | www.confluent.io
> > > >
> > >
> >
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message