storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Raney <tom.ra...@urbanairship.com>
Subject Re: Submitting & Managing Topologies
Date Tue, 05 Jan 2016 17:55:12 GMT
Hi Zack,

I'm also interested in hearing more about how folks generally handle the
deployment process.  In particular, versioning the topology in such a way
that makes continuous integration possible without manual deployment steps.

This project (https://github.com/ptgoetz/storm-maven-plugin) seems like a
step forward to integrating with Jenkins nicely, but it appears abandoned
or perhaps replaced by something else?  I'm not against using the Maven
exec plugin to build something, but, I'd hate to do that when someone else
has already built a solution.

As far as handling the repo itself, I think it depends on how many spouts
and bolts you're planning on re-using across different topologies.  And,
unless the upload time becomes burdensome, it doesn't seem that terrible to
just maintain a single large super jar.  If the jar size is an issue, you
could break your project into Maven sub-modules within a single repo and
have custom shaded builders for particular topologies to reduce the jar
size.

Regards,

Tom

On Mon, Jan 4, 2016 at 3:50 PM, Zack Allen <zallen@fastly.com> wrote:

> Hi all,
>
> I would like to know the best practices in terms of the submission and
> management of storm topologies.
>
> Do you manage 1 repo per topology, or have 1 repo with many topologies in
> it? For example, I have a topology that counts words, a topology that sees
> if an input is a prime number, and a topology that adds exclamation points
> to a piece of text.
>
> What would be the best option in terms of updating the prime number
> topology? I know you can remove one from the UI, but do you generally ship
> a .jar with all of the topologies or do you have one strictly for that
> piece of processing? I am leaning towards 1 repo for 1 topology to make
> deployments more intuitive (push a change via github, assemble the .jar,
> push to storm topology) but I would like to hear what everyone here does.
>
> Thanks!
>

Mime
View raw message