kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: startup scripts and packaging
Date Fri, 05 Aug 2011 01:17:18 GMT
One thing I think would be really helpful would be to make it so you could
easily spawn up a cluster. Hadoop does this and it is nice. For companies
that don't have standardized deployment infrastructure this is really nice,
and also for testing of all kinds. Basically it would be nice if you could
make a list containing one host:port per line and there were scripts like
   bin/kafka-cluster-deploy.sh machine-list.txt    # rsync the kafka
directory and config around to the given list of machines
   bin/kafka-cluster-start.sh machine-list.txt       # ssh to each machine
in the list and start it
   bin/kafka-cluster-stop.sh machine-list.txt       # ssh to each machine
and kill the kafka process
   bin/kafka-cluster-delete.sh machine-list.txt     # ssh around to each
machine in the list and delete the code and log directory

To do this I think you would need a way to override properties on the
command line. This would make it so you could rsync out the kafka code to a
bunch of machines but give a different node id and (if needed) port. I think
something like "--property-name value" to give properties that override what
is in the properties file would work.

One other minor and unrelated issue we have is that our stop script actually
kills all the kafka processes on the machine, which means if you are trying
to run multiple nodes on the same machine it is a little dangerous.

-Jay

On Thu, Aug 4, 2011 at 4:59 PM, Chris Burroughs
<chris.burroughs@gmail.com>wrote:

> I've been looking at how kafka is packaged and the provided scripts for
> running it.  There are a few things I want to improve: it should be
> easier to run in the foreground or background, convincing java to keep
> the executable bits set, and reasonable log4j defaults.  There are also
> a few plain old bugs detailed in KAFKA-81 that need to be fixed.
>
> If there are other things that would make the results of ./sbt package
> or release-zip more useful, or the bash scripts easier to work with,
> please let me know.
>
>
> Thanks,
> Chris Burroughs
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message