metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <>
Subject Re: [DISCUSS] Storm topology sideloading jars
Date Mon, 19 Sep 2016 14:17:27 GMT
I really would like this functionality.  Just a brief aside on what this
buys us.  As it stands now, we have a few main extension points:

   - Custom Java parsers are found via fully qualified classname
   - Custom Stellar functions are found via annotation of the class and
   being dropped on the classpath
   - Custom Enrichment adapters (e.g. the geo enrichment) are found via
   incorporating changes into the enrichment topology flux file

Right now the only way to add such things is to compile your code along
with ours and have them placed in our uber jars that we submit to Storm.
What would be better is for us to expose the interfaces and have developers
use them to create their custom functionality, build just their extension
points and the dependencies and drop the jar file in a directory to be
picked up the next time the topology starts.

I like the HDFS idea because it means that we do not have to ensure a 3rd
party jar directory is sync'd across the storm supervisors.  My question is
whether Storm supports pulling external dependencies via HDFS.  Does anyone

Thanks for bringing this up, Mike.



On Mon, Sep 19, 2016 at 10:08 AM, Michael Miklavcic <> wrote:

> As part of it is now
> possible to add hbase and hadoop conf to the Storm topology classpath. It
> is also desirable to expand this functionality to sideloading jars for
> Storm topologies. That way, users can add additional dependencies without
> having to recompile/repackage existing jars. One suggestion is to leverage
> HDFS to store custom jars and add them to the topology.classpath. I want to
> open this discussion to the community.
> Best,
> Mike

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message