storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Powis <>
Subject Re: Submitting & Managing Topologies
Date Tue, 05 Jan 2016 18:33:52 GMT
So we use Bamboo for our CI.  Basically a commit comes in, Bamboo picks up
the change and runs our automated test suite.  If the tests all pass, we
have bamboo package up a jar.  As part of this process we write the version
information and bamboo build number into a properties file in the jar.
Bamboo then publishes the jar to Artifactory.

As of now, when we want to redeploy a topology we have a simple shell
script (manual process, tho it could be automated if we wanted) that polls
Artifactory for the available builds that can be deployed, you select which
one you want to ship, and it handles deploying it out to our cluster.  When
the topology is deployed, it reads the version properties file and writes
that into the topologies config.  This way we can inspect a running
topology via the webui and easily see what version/build is currently out
in production.  Additionally our deployment script records when and what
builds are deployed.

I'd love to hear what other people are doing!

On Wed, Jan 6, 2016 at 12:55 AM, Tom Raney <>

> Hi Zack,
> I'm also interested in hearing more about how folks generally handle the
> deployment process.  In particular, versioning the topology in such a way
> that makes continuous integration possible without manual deployment steps.
> This project ( seems like a
> step forward to integrating with Jenkins nicely, but it appears abandoned
> or perhaps replaced by something else?  I'm not against using the Maven
> exec plugin to build something, but, I'd hate to do that when someone else
> has already built a solution.
> As far as handling the repo itself, I think it depends on how many spouts
> and bolts you're planning on re-using across different topologies.  And,
> unless the upload time becomes burdensome, it doesn't seem that terrible to
> just maintain a single large super jar.  If the jar size is an issue, you
> could break your project into Maven sub-modules within a single repo and
> have custom shaded builders for particular topologies to reduce the jar
> size.
> Regards,
> Tom
> On Mon, Jan 4, 2016 at 3:50 PM, Zack Allen <> wrote:
>> Hi all,
>> I would like to know the best practices in terms of the submission and
>> management of storm topologies.
>> Do you manage 1 repo per topology, or have 1 repo with many topologies in
>> it? For example, I have a topology that counts words, a topology that sees
>> if an input is a prime number, and a topology that adds exclamation points
>> to a piece of text.
>> What would be the best option in terms of updating the prime number
>> topology? I know you can remove one from the UI, but do you generally ship
>> a .jar with all of the topologies or do you have one strictly for that
>> piece of processing? I am leaning towards 1 repo for 1 topology to make
>> deployments more intuitive (push a change via github, assemble the .jar,
>> push to storm topology) but I would like to hear what everyone here does.
>> Thanks!

View raw message