kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Hauser <ewhau...@gmail.com>
Subject Re: startup scripts and packaging
Date Fri, 05 Aug 2011 16:17:25 GMT
Based on the discussion, I just pushed our Chef cookbooks for Kafka
and ZooKeeper:


We use Cloudera ZK on Ubuntu, so a little work would need to be done
changing the Cloudera cookbook to enable Cloudera's yum repo for RH

We have an additional contribution to make as well.  Are pull requests
still acceptable?  I see that there is an ASF Jira now.

On Fri, Aug 5, 2011 at 9:14 AM, Chris Burroughs
<chris.burroughs@gmail.com> wrote:
> All sounds cool and doable.  But didn't we delete contrib/deploy in
> favour of https://issues.apache.org/jira/browse/KAFKA-10 ?
> I agree that something like https://github.com/pcmanus/ccm would be
> useful for bringing up local clusters for local testing.
> (And yeah, we should create a pid file instead of killing every running
> kafka)
> On 08/04/2011 09:17 PM, Jay Kreps wrote:
>> One thing I think would be really helpful would be to make it so you could
>> easily spawn up a cluster. Hadoop does this and it is nice. For companies
>> that don't have standardized deployment infrastructure this is really nice,
>> and also for testing of all kinds. Basically it would be nice if you could
>> make a list containing one host:port per line and there were scripts like
>>    bin/kafka-cluster-deploy.sh machine-list.txt    # rsync the kafka
>> directory and config around to the given list of machines
>>    bin/kafka-cluster-start.sh machine-list.txt       # ssh to each machine
>> in the list and start it
>>    bin/kafka-cluster-stop.sh machine-list.txt       # ssh to each machine
>> and kill the kafka process
>>    bin/kafka-cluster-delete.sh machine-list.txt     # ssh around to each
>> machine in the list and delete the code and log directory
>> To do this I think you would need a way to override properties on the
>> command line. This would make it so you could rsync out the kafka code to a
>> bunch of machines but give a different node id and (if needed) port. I think
>> something like "--property-name value" to give properties that override what
>> is in the properties file would work.
>> One other minor and unrelated issue we have is that our stop script actually
>> kills all the kafka processes on the machine, which means if you are trying
>> to run multiple nodes on the same machine it is a little dangerous.
>> -Jay
>> On Thu, Aug 4, 2011 at 4:59 PM, Chris Burroughs
>> <chris.burroughs@gmail.com>wrote:
>>> I've been looking at how kafka is packaged and the provided scripts for
>>> running it.  There are a few things I want to improve: it should be
>>> easier to run in the foreground or background, convincing java to keep
>>> the executable bits set, and reasonable log4j defaults.  There are also
>>> a few plain old bugs detailed in KAFKA-81 that need to be fixed.
>>> If there are other things that would make the results of ./sbt package
>>> or release-zip more useful, or the bash scripts easier to work with,
>>> please let me know.
>>> Thanks,
>>> Chris Burroughs

View raw message