storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Petr Janeček <>
Subject Running a local cluster under a nondefault classloader
Date Mon, 23 Jan 2017 12:52:45 GMT

The text below is an edited question from Stack Overflow (http://
(, I'm aware that markdown is 
not really supported.

A Local Cluster from a web classloader

I'm trying to run a local cluster from a web container (yes, it's only for 
dev & testing purposes) and am having difficulty with classloaders.

Direct approach

When I do it the easy and recommended (
current/Local-mode.html) way,

    ILocalCluster localCluster = new LocalCluster();
    localCluster.submitTopology(topologyName, stormConf, topology);

I get rewarded with

    Async loop died!: java.lang.ClassCastException:
SomeFilteringBolt cannot be cast to org.apache.storm.task.IBolt
        at org.apache.storm.daemon.executor$fn__7953$fn__7966.invoke
        at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:482)

This is because the classloader used to load and instantiate the `
StormTopology` is an instance of a Jetty `WebAppClassLoader`, but the (sub)
process spawned by `LocalCluster.submitTopology()` apparently uses the 
system classloader. I confirmed this by logging the classloader in the 
static block of the `SomeFilteringBolt` - the class is indeed loaded twice 
and the bolt from WebAppCL obviously cannot be cast to a bolt on the system 
classloader later on.

Expected behaviour

Now, this is surprising to me as I thought Storm would serialize the `
StormTopology` instance, "send" it locally, deserialize it and run it. If it
did that, though, it definitely would work. Rather it seems that it's 
directly using the provided `StormTopology` instance which is problematic 
under a different classloader.

What I have tried since

I tried setting these to `true` to force Storm to serialize my topology 
locally. No change.


I tried running the LocalCluster under the system classloader:

    ClassLoader originalClassloader = Thread.currentThread().
    try {
        Config topologyConf = createTopologyConfig();
        Map<String, Object> stormConf = createStormConfig(topologyConf);
        StormTopology topology = createTopology(topologyConf);
        ILocalCluster localCluster = new LocalCluster();
        localCluster.submitTopology(topologyName, stormConf, topology);
    } finally {

This actually got me a bit further:

    Thread  died: java.lang.ExceptionInInitializerError
        at clojure.core__init.__init0(Unknown Source)
        at clojure.core__init.<clinit>(Unknown Source)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(
        at clojure.lang.RT.classForName(
        at clojure.lang.RT.classForName(
        at clojure.lang.RT.loadClassForName(
        at clojure.lang.RT.load(
        at clojure.lang.RT.load(
        at clojure.lang.RT.doInit(
        at clojure.lang.RT.<clinit>(
        at clojure.lang.Namespace.<init>(
        at clojure.lang.Namespace.findOrCreate(
        at clojure.lang.Var.internPrivate(
        at org.apache.storm.LocalCluster.<clinit>(Unknown Source)
        ... 10 more
    Caused by: java.lang.IllegalStateException: Attempting to call unbound 
fn: #'clojure.core/refer
        at clojure.lang.Var$Unbound.throwArity(
        at clojure.lang.AFn.invoke(
        at clojure.lang.Var.invoke(
        at clojure.lang.RT.doInit(
        at clojure.lang.RT.<clinit>(
        ... 18 more


The question

How can I run a Storm topology in local mode safely from a classloader other
than the system classloader?

I'm running on Apache Storm 1.0.1, Jetty 8.1, Java 8u112 x64, Windows 7 x64.
View raw message