kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dillian Murphey <crackshotm...@gmail.com>
Subject Re: kafka cluster on aws
Date Wed, 14 Jan 2015 23:32:21 GMT
Trying to understand the docs.  Can I just use the docker image and run the
minotaur command from there?  I don't understand the Basion SSH stuff. Do I
need that? I just want a quick start for right now. Also, not sure where I
get the ENVIRONMENT.key.

Any extra help is greatly appreciated. You can email directly. Thanks!

On Wed, Jan 14, 2015 at 12:09 PM, Joe Stein <joe.stein@stealth.ly> wrote:

> We have an open source framework you can use to spin up Kafka (any version
> or even any build you want) clusters (and Zookeeper) with CloudFormation on
> AWS https://github.com/stealthly/minotaur
>
> It is very nice/handy you basically specify your instance types, counts,
> versions of code, etc and hit a <enter>
> https://github.com/stealthly/minotaur/tree/master/labs/kafka e.g.
>
> ./minotaur.py lab deploy kafka -e bdoss-dev -d testing -r us-east-1 -z
> us-east-1a -k http://example.com/kafka.tar.gz -n 3 -i m1.small
>
> There is some setup for the bastion host (
>
> https://github.com/stealthly/minotaur/tree/master/infrastructure/aws/bastion
> )
> and supervisor (
> https://github.com/stealthly/minotaur/tree/master/supervisor)
> and after that it is really nice and easy.
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Wed, Jan 14, 2015 at 2:54 PM, Joseph Lawson <jlawson@roomkey.com>
> wrote:
>
> > We have a separate daemon process that assigns EIPs to servers when they
> > startup in an autoscaling group based off of an autoscaling message.  So
> > for a cluster of 3 we have 3 EIPs. Then we inject the EIPs into startup
> > script for Kafka which checks to see if it has one of the EIPs and
> assigns
> > itself the index of that IP so in the list:
> > 10.0.0.1 10.0.0.2 10.0.0.3
> >
> > 1 is broker 0, 2 is broker 1 and 3 is broker 2.  All this is injected via
> > cloudformation and then we have a mod value so if we want to spin brokers
> > in the same group we do mod 1,2 and get brokers mod * 3 + index to
> > determine which is in the group. (the EIPs are different as it is a
> > different cloudformation)
> >
> > For redundancy make sure you run at least two that have full replicas of
> > all other partitions.  We run replication factor of 3 with three
> instances
> > so if any goes down the other two bring it back in sync once a fresh
> server
> > spins in the autoscaling group.
> >
> > ________________________________________
> > From: Dillian Murphey <crackshotmule@gmail.com>
> > Sent: Wednesday, January 14, 2015 2:42 PM
> > To: users@kafka.apache.org
> > Subject: kafka cluster on aws
> >
> > I can't seem to find much information to help me (being green to kafka)
> on
> > setting up a cluster on aws. Does anyone have any sources?
> >
> > The question I have off the bat is, what methods have already been
> explored
> > to generate a unique broker id? If I spin up a new server, do I just need
> > to maintain my own broker-id list somewhere so I don't re-use an already
> > allocated broker id?
> >
> > Also, I read an article about a broker going down and requiring a new
> > broker be spun up with the same id. Is this also something I need to
> > implement?
> >
> > I want to setup a kafka auto-scaling group on AWS, so I can add brokers
> at
> > well or based on load. It doesn't seem too complicated, or maybe I'm too
> > green to see it, but I don't want to re-invent everything myself.
> >
> > I know Loggly uses AWS/Kafka, so I'm hunting for more details on that
> too.
> >
> > Thanks for any help
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message