That may be the case.  

I have confirmed that supervisor is getting either the Client classpath or Server classpath sent to zookeeper to start worker processes.  If I edit the config file on the supervisor, the workers still start with the classpath that is on the nimbus server, with the javaagent included in the classpath.  The only way that seems possible is that supervisor is starting with a classpath it has gotten from zookeeper.

Yes, the weird part is that the agent is sneaking onto the class path when the worker starts up.

storm classpath returns: 
/usr/local/Cellar/storm/1.0.1/libexec/lib/asm-5.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/clojure-1.7.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/disruptor-3.3.2.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/kryo-3.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-api-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-core-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-slf4j-impl-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/minlog-1.3.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/objenesis-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/reflectasm-1.10.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/servlet-api-2.5.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/slf4j-api-1.7.7.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-core-1.0.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-rename-hack-1.0.1.jar

thanks for all of your help.
brad


Subject: Re: Nimbus setting java.class.path to include javaagent
From: ptgoetz@gmail.com
Date: Wed, 5 Oct 2016 15:33:47 -0400
To: user@storm.apache.org

What your seeing is the ZooKeeper client running within Nimbus/Supervisor just printing out itís class path, which it inherits from Stormís class path. Itís not really sending that to ZooKeeper.

Whatís weird is that the agent jar sneaking onto the class path.

On your supervisor node, can you run `storm classpath` and see what it prints?

-Taylor


On Oct 5, 2016, at 1:17 PM, Brad Rhodes <bradrhod@hotmail.com> wrote:

Storm 0.9.6.  We are working on moving to 1.0.2 as we speak.
 
nimbus.childopts: "-Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=
y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -
Dapplication.name=storm.test.lodgingamenity.nimbus"


supervisor.childopts: "-Xmx256m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad2.jar -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring
.properties -
Dapplication.name=storm.test.lodgingamenity.supervisor"


worker.childopts: "-Xmx1024m -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -
Dapplication.name=storm.test.lodgingamenity.worker.%ID%"
 
 
From: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
Sent: Tuesday, October 4, 2016 9:46 AM
To: user@storm.apache.org
Subject: Re: Nimbus setting java.class.path to include javaagent
 
Can you post your values for nimbus.childopts, supervisor.childopts, and worker.childopts? Also what version of Storm are you on?
 
-Taylor
 
On Sep 28, 2016, at 6:06 PM, Brad Rhodes <bradrhod@hotmail.com> wrote:
 
I have a situation where we have a javaagent jar file.  Someone has added a number of classes to the corporate standard javaagent jar file.  Nimbus appears to append -javaagent jar files to the -cp (claspath) for the supervisor process.
 
When Nimbus starts up java is started with a command like: 
 
/usr/java/latest/bin/java -server -Dstorm.options= -Dstorm.home=/opt/storm -Dstorm.log.dir=/opt/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf -Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar -agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -Dapplication.name=storm.test.lodgingamenity.nimbus -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/opt/storm/logback/cluster.xml backtype.storm.daemon.nimbus
 
Nimbus at some point sends the following to Zookeeper:
016-09-28T18:50:45.627+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar
 
and 
Server environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar
 
When the supervisor starts up you will see bobbrad.jar in the classpath.
 
I see the code in supervisor.clj 
(defn resources-jar []
  (->> (.split (current-classpath) File/pathSeparator)
       (filter #(.endsWith  % ".jar"))
       (filter #(zip-contains-dir? % RESOURCES-SUBDIR))
       first ))
It does not seem this code is the source of this running with zookeeper cluster.  Suprivisor is applying what comes down from Zookeeper java.classpath.
Can someone point me to where that is happening in a cluster environment.?
Thanks for any help.