kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Burroughs <chris.burrou...@gmail.com>
Subject Re: startup scripts and packaging
Date Fri, 05 Aug 2011 13:14:51 GMT
All sounds cool and doable.  But didn't we delete contrib/deploy in
favour of https://issues.apache.org/jira/browse/KAFKA-10 ?

I agree that something like https://github.com/pcmanus/ccm would be
useful for bringing up local clusters for local testing.

(And yeah, we should create a pid file instead of killing every running
kafka)

On 08/04/2011 09:17 PM, Jay Kreps wrote:
> One thing I think would be really helpful would be to make it so you could
> easily spawn up a cluster. Hadoop does this and it is nice. For companies
> that don't have standardized deployment infrastructure this is really nice,
> and also for testing of all kinds. Basically it would be nice if you could
> make a list containing one host:port per line and there were scripts like
>    bin/kafka-cluster-deploy.sh machine-list.txt    # rsync the kafka
> directory and config around to the given list of machines
>    bin/kafka-cluster-start.sh machine-list.txt       # ssh to each machine
> in the list and start it
>    bin/kafka-cluster-stop.sh machine-list.txt       # ssh to each machine
> and kill the kafka process
>    bin/kafka-cluster-delete.sh machine-list.txt     # ssh around to each
> machine in the list and delete the code and log directory
> 
> To do this I think you would need a way to override properties on the
> command line. This would make it so you could rsync out the kafka code to a
> bunch of machines but give a different node id and (if needed) port. I think
> something like "--property-name value" to give properties that override what
> is in the properties file would work.
> 
> One other minor and unrelated issue we have is that our stop script actually
> kills all the kafka processes on the machine, which means if you are trying
> to run multiple nodes on the same machine it is a little dangerous.
> 
> -Jay
> 
> On Thu, Aug 4, 2011 at 4:59 PM, Chris Burroughs
> <chris.burroughs@gmail.com>wrote:
> 
>> I've been looking at how kafka is packaged and the provided scripts for
>> running it.  There are a few things I want to improve: it should be
>> easier to run in the foreground or background, convincing java to keep
>> the executable bits set, and reasonable log4j defaults.  There are also
>> a few plain old bugs detailed in KAFKA-81 that need to be fixed.
>>
>> If there are other things that would make the results of ./sbt package
>> or release-zip more useful, or the bash scripts easier to work with,
>> please let me know.
>>
>>
>> Thanks,
>> Chris Burroughs
>>
> 


Mime
View raw message