storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brad Rhodes <bradr...@hotmail.com>
Subject RE: Nimbus setting java.class.path to include javaagent
Date Fri, 07 Oct 2016 22:41:37 GMT
I do not see any difference from 1.01 to 0.9.6 in this regard.  But I do not have a cluster
of 1.01 setup yet, so that is development environment and running local, which is different.

 

 

 

From: P. Taylor Goetz [mailto:ptgoetz@gmail.com] 
Sent: Friday, October 7, 2016 3:26 PM
To: user@storm.apache.org
Subject: Re: Nimbus setting java.class.path to include javaagent

 

Okay, in your development environment you are using 1.0.1, but your deployment environment
is 0.9.6. There's a lot of distance between those versions, and I've been looking at the wrong
code.

 

Before I go back to the 0.9.6 code, are you saying you're seeing this behavior with both versions?

 

-Taylor


On Oct 7, 2016, at 4:00 PM, Brad Rhodes <bradrhod@hotmail.com <mailto:bradrhod@hotmail.com>
> wrote:

I am asking again for a bit of help.  I would not re-ask, but I have spent a ton of time on
this, debugging the startup code in nimbus and I am somewhat stuck.

 

I can provide logs and traces showing that supervisor is getting Zookeeper java.class.path
(server or client) and using that to start the worker.  Let me know if you want to see them.

 

I know these values are getting sent to zookeeper in the org.apache.curator.framework : CuratorFrameworkFactory
class on line 61:

Public static Builder  builder() {return new Builder();}

 

I know the config processing is largely happening in storm-core/src/clj/backtype.storm/util.clj
and config.clj.   With some happening in storm-core/src/jvm/utils/Utils.java

 

I know that the initial set up of the config structure is done in Utils.java line 70.

 

Static{

   Map conf = readStormConfig()

   serializationDelegate = getSerializationDelegate(conf);

}

 

I just cannot find where in these files someone is taking the javaagent line and appending
it to the classpath.  It seems  somewhere between the initial config read, and when this is
sent to zookeeper by the Builder(); call. 

 

I could use any help, like if anyone is some pointers on how to debug Clojure startup code
with java static init <cinit>.  I am using the Cursive Debugger add-in for Intellij.
 But I think the issue here is more the static{} functions that are running in java.  This
all makes it super slow debugging. Sometimes you end up stepping through bytecode, and not
getting variables.   So if anyone has pointers on how they work in these areas, that would
be a huge help.

 

Thanks again for any help.

brad

 

 

 

 

 

From: Brad Rhodes [mailto:bradrhod@hotmail.com] 
Sent: Wednesday, October 5, 2016 2:15 PM
To: user@storm.apache.org <mailto:user@storm.apache.org> 
Subject: RE: Nimbus setting java.class.path to include javaagent

 

Hey Taylor,

Sorry, that path was from my development machine.

 

On the nimbus server in question storm.classpath returns:

/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar

 


  _____  


From: bradrhod@hotmail.com <mailto:bradrhod@hotmail.com> 
To: user@storm.apache.org <mailto:user@storm.apache.org> 
Subject: RE: Nimbus setting java.class.path to include javaagent
Date: Wed, 5 Oct 2016 12:52:20 -0700

That may be the case.  

 

I have confirmed that supervisor is getting either the Client classpath or Server classpath
sent to zookeeper to start worker processes.  If I edit the config file on the supervisor,
the workers still start with the classpath that is on the nimbus server, with the javaagent
included in the classpath.  The only way that seems possible is that supervisor is starting
with a classpath it has gotten from zookeeper.

 

Yes, the weird part is that the agent is sneaking onto the class path when the worker starts
up.

 

storm classpath returns: 

/usr/local/Cellar/storm/1.0.1/libexec/lib/asm-5.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/clojure-1.7.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/disruptor-3.3.2.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/kryo-3.0.3.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-api-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-core-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/log4j-slf4j-impl-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/minlog-1.3.0.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/objenesis-2.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/reflectasm-1.10.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/servlet-api-2.5.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/slf4j-api-1.7.7.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-core-1.0.1.jar:/usr/local/Cellar/storm/1.0.1/libexec/lib/storm-rename-hack-1.0.1.jar

 

thanks for all of your help.

brad


  _____  


Subject: Re: Nimbus setting java.class.path to include javaagent
From: ptgoetz@gmail.com <mailto:ptgoetz@gmail.com> 
Date: Wed, 5 Oct 2016 15:33:47 -0400
To: user@storm.apache.org <mailto:user@storm.apache.org> 

What your seeing is the ZooKeeper client running within Nimbus/Supervisor just printing out
it’s class path, which it inherits from Storm’s class path. It’s not really sending
that to ZooKeeper.

 

What’s weird is that the agent jar sneaking onto the class path.

 

On your supervisor node, can you run `storm classpath` and see what it prints?

 

-Taylor

 

 

On Oct 5, 2016, at 1:17 PM, Brad Rhodes <bradrhod@hotmail.com <mailto:bradrhod@hotmail.com>
> wrote:

 

Storm 0.9.6.  We are working on moving to 1.0.2 as we speak.

 

nimbus.childopts: "-Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar
-agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=
y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties -Dapplication.name=storm.test.lodgingamenity.nimbus"


supervisor.childopts: "-Xmx256m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad2.jar
-Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring
.properties -Dapplication.name=storm.test.lodgingamenity.supervisor"


worker.childopts: "-Xmx1024m -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties
-Dapplication.name=storm.test.lodgingamenity.worker.%ID%"

 

 

From: P. Taylor Goetz [mailto:ptgoetz@gmail.com] 
Sent: Tuesday, October 4, 2016 9:46 AM
To: user@storm.apache.org <mailto:user@storm.apache.org> 
Subject: Re: Nimbus setting java.class.path to include javaagent

 

Can you post your values for nimbus.childopts, supervisor.childopts, and worker.childopts?
Also what version of Storm are you on?

 

-Taylor

 

On Sep 28, 2016, at 6:06 PM, Brad Rhodes <bradrhod@hotmail.com <mailto:bradrhod@hotmail.com>
> wrote:

 

I have a situation where we have a javaagent jar file.  Someone has added a number of classes
to the corporate standard javaagent jar file.  Nimbus appears to append -javaagent jar files
to the -cp (claspath) for the supervisor process.

 

When Nimbus starts up java is started with a command like: 

 

/usr/java/latest/bin/java -server -Dstorm.options= -Dstorm.home=/opt/storm -Dstorm.log.dir=/opt/storm/logs
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf
-Xmx1024m -DjmxRegistryPort=1098 -DjmxServerPort=1099 -javaagent:/opt/monitoring_agent/bobbrad.jar
-agentlib:jdwp=transport=dt_socket,server=y,address=8100,suspend=y -Dmonitoring.agent.properties=/opt/monitoring_agent/monitoring.properties
-Dapplication.name=storm.test.lodgingamenity.nimbus -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/opt/storm/logback/cluster.xml
backtype.storm.daemon.nimbus

 

Nimbus at some point sends the following to Zookeeper:

016-09-28T18:50:45.627+0000 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar

 

and 

Server environment:java.class.path=/opt/storm/lib/hiccup-0.3.6.jar:/opt/storm/lib/clj-stacktrace-0.2.2.jar:/opt/storm/lib/clj-time-0.4.1.jar:/opt/storm/lib/tools.macro-0.1.0.jar:/opt/storm/lib/commons-exec-1.1.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/joda-time-2.0.jar:/opt/storm/lib/ring-devel-0.3.11.jar:/opt/storm/lib/tools.cli-0.2.4.jar:/opt/storm/lib/logback-core-1.0.13.jar:/opt/storm/lib/tools.logging-0.2.3.jar:/opt/storm/lib/jline-2.11.jar:/opt/storm/lib/math.numeric-tower-0.0.1.jar:/opt/storm/lib/chill-java-0.3.5.jar:/opt/storm/lib/objenesis-1.2.jar:/opt/storm/lib/compojure-1.1.3.jar:/opt/storm/lib/clojure-1.5.1.jar:/opt/storm/lib/commons-codec-1.6.jar:/opt/storm/lib/commons-io-2.4.jar:/opt/storm/lib/clout-1.0.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/commons-fileupload-1.2.1.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/storm-core-0.9.6.jar:/opt/storm/lib/jgrapht-core-0.9.0.jar:/opt/storm/lib/ring-servlet-0.3.11.jar:/opt/storm/lib/core.incubator-0.1.0.jar:/opt/storm/lib/snakeyaml-1.11.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/commons-lang-2.5.jar:/opt/storm/lib/carbonite-1.4.0.jar:/opt/storm/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/jetty-6.1.26.jar:/opt/storm/lib/commons-logging-1.1.3.jar:/opt/storm/lib/json-simple-1.1.jar:/opt/storm/lib/slf4j-api-1.7.5.jar:/opt/storm/lib/logback-classic-1.0.13.jar:/opt/storm/lib/jetty-util-6.1.26.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/ring-core-1.1.5.jar:/opt/storm/conf:/opt/monitoring_agent/bobbrad.jar

 

When the supervisor starts up you will see bobbrad.jar in the classpath.

 

I see the code in supervisor.clj 

(defn resources-jar []
  (->> (.split (current-classpath) File/pathSeparator)
       (filter #(.endsWith  % ".jar"))
       (filter #(zip-contains-dir? % RESOURCES-SUBDIR))
       first ))
It does not seem this code is the source of this running with zookeeper cluster.  Suprivisor
is applying what comes down from Zookeeper java.classpath.
Can someone point me to where that is happening in a cluster environment.?
Thanks for any help.

 

 


Mime
View raw message