metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1916) Stellar Classpath Function Resolver Should Handle ClassNotFoundException
Date Mon, 03 Dec 2018 14:30:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707272#comment-16707272
] 

ASF GitHub Bot commented on METRON-1916:
----------------------------------------

Github user nickwallen commented on the issue:

    https://github.com/apache/metron/pull/1288
  
    > In my mind we don't have a current state where Stellar is running but not all the
functions in the class path are loaded. 
    
    The same thing would happen today if an `Exception` is thrown in any of the functions
being loaded.  If an Exception is thrown, instead of N functions ready-to-rock, we would have
N-1 functions.  This just expands that behavior to `ClassNotFoundException` which gets thrown
when there is a missing dependency..
    
    > Before we would have crashed starting up. Now we will run and crash get an error
later.
    
    I don't see it that way.  Let's look at both scenarios.
    * If I am NOT using the REST function, then things just work as they should.  Yay!   I
don't want to worry about the missing dependency for a function that I have no knowledge about.
    * If I am using the REST function, an exception would still be thrown when the function
definition could not be found.  So the user still gets an exception when there is a problem.
    
    (Q) Any thoughts on alternative approaches?
    
    I prefer not to have to worry about classpath dependencies (like those required by the
REST function) that I am not using.  In this example, I was only trying to use the `STATS_*`
functions, but it was the REST function that blew things up for me.  It is difficult for a
user to track this down, because they have no knowledge of the REST function and its dependencies.
    
    (Alternative 1) An alternative approach would be to be more selective about what functions
we add to `stellar-common`.  Anything added to `stellar-common` forces a required dependency
on all Stellar users.  If the REST function had been a separate project, then a user could
choose to use or not use that projects and not be burdened by the additional dependency
    
    (Alternative 2) Could the `stellar.function.resolver.includes` and `stellar.function.resolver.excludes`
be enhanced to allow users to exclude functions they don't want to load?
    



> Stellar Classpath Function Resolver Should Handle ClassNotFoundException
> ------------------------------------------------------------------------
>
>                 Key: METRON-1916
>                 URL: https://issues.apache.org/jira/browse/METRON-1916
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>            Priority: Major
>
> Stellar can be executed in an environment where not all of the dependencies are available
for each of the functions provided in Stellar Common. When this happens, the classpath resolver
blows-up even if you do not use the function that is missing dependencies.
> For example this exception is thrown when not all of the dependencies are available for
the new REST function.  I was only attempting to use STATS_INIT, not the REST function.
> {code:java}
> 2018-11-30 15:47:00 DEBUG BaseFunctionResolver:165 - Resolving Stellar function in class
RestFunctions.RestGet
> 2018-11-30 15:47:00 ERROR DefaultProfileBuilder:302 - Bad 'init' expression: error='Unable
to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager', expr='STATS_INIT()',
profile='profile-with-stats', entity='global', variables-available='[adapter.threatinteladapter.end.ts,
bro_timestamp, ip_dst_port, enrichmentsplitterbolt.splitter.end.ts, enrichmentsplitterbolt.splitter.begin.ts,
adapter.hostfromjsonlistadapter.end.ts, adapter.geoadapter.begin.ts, uid, trans_depth, protocol,
original_string, ip_dst_addr, threatinteljoinbolt.joiner.ts, host, enrichmentjoinbolt.joiner.ts,
adapter.hostfromjsonlistadapter.begin.ts, threatintelsplitterbolt.splitter.begin.ts, ip_src_addr,
user_agent, timestamp, method, request_body_len, uri, tags, source.type, adapter.geoadapter.end.ts,
referrer, threatintelsplitterbolt.splitter.end.ts, adapter.threatinteladapter.begin.ts, ip_src_port,
guid, response_body_len]'
> org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager
> at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
> at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
> at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160)
> at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95)
> at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291)
> at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:136)
> at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:177)
> at org.apache.metron.profiler.storm.ProfileBuilderBolt.handleMessage(ProfileBuilderBolt.java:411)
> at org.apache.metron.profiler.storm.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:337)
> at org.apache.storm.topology.WindowedBoltExecutor$1.onActivation(WindowedBoltExecutor.java:307)
> at org.apache.storm.windowing.WindowManager.onTrigger(WindowManager.java:145)
> at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.handleWaterMarkEvent(WatermarkTimeTriggerPolicy.java:80)
> at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.track(WatermarkTimeTriggerPolicy.java:48)
> at org.apache.storm.windowing.WindowManager.track(WindowManager.java:177)
> at org.apache.storm.windowing.WindowManager.add(WindowManager.java:110)
> at org.apache.storm.windowing.WaterMarkEventGenerator.run(WaterMarkEventGenerator.java:80)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getConstructor(Class.java:1825)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.createFunction(BaseFunctionResolver.java:240)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunction(BaseFunctionResolver.java:194)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:166)
> at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:149)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:49)
> at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:688)
> at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:656)
> at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
> at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
> ... 21 more{code}
> When this happens the Classpath Function Resolver should allow me to use whichever functions
can be loaded correctly.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message