metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: Custom Storm Topologies
Date Tue, 03 Jan 2017 18:47:12 GMT
Well, yes :-)  
And clearly it should always be more efficient to write a custom bolt in Java than to invoke
a script and manage it.

--Matt

From: Otto Fowler <ottobackwards@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>, Matt Foley
<mattf@apache.org>
Subject: Re: Custom Storm Topologies

Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming script and keep
it alive and fed, long-term, rather than launching the script anew every time the Stellar
function is invoked? I’m thinking two basic rules: Write a line, read a line; and always
have a timeout. Prob need a UID of some sort for a cache of running process objects. 

--Matt 

On 1/2/17, 8:50 AM, "Carolyn Duby" <cduby@hortonworks.com> wrote: 


Inserting a script inline is ok for low throughput and prototyping but once you get higher
throughput (millions of events per second), it’s probably going to be a bottleneck. 


For Metron-571 you might want to consider a java based extension plugin similar to Eclipse
plugins. 

Thanks 
Carolyn 

On 12/31/16, 5:22 PM, "Tyler Moore" <tmoore@goflyball.com> wrote: 

>Thanks Jon, 
> 
>I'll look over the tutorial and put something together for the SHELL_EXEC 
>stellar function. 
>I don't believe I have permissions to assign in Jira if you want to assign 
>to me my username is devopsec. 
>I'll post back details and we can review security issues 
> 
>Regards, 
> 
>Tyler Moore 
>Software Engineer 
>Phone: 248-909-2769 
>Email: moore.tyler@goflyball.com 
> 
> 
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <zeolla@gmail.com> wrote: 
> 
>> Casey did a tutorial on how to add your own Stellar function here 
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing 
>> function that does this (current functions are listed here 
>> <https://github.com/apache/incubator-metron/tree/master/ 
>> metron-platform/metron-common#stellar-core-functions>). 
>> I noticed that some of the Stellar function documentation was a bit dated 
>> so I've opened a PR to update it here 
>> <https://github.com/apache/incubator-metron/pull/407>. 
>> 
>> As this is something I need as well, I'd be happy to assist you where I 
>> can. Perhaps you want to self-assign METRON-571 
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some 
>> security concerns with a SHELL_EXEC function because it could result in RCE 
>> - if that's the route you go I could probably help with a thorough secure 
>> code review. 
>> 
>> Jon 
>> 
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmoore@goflyball.com> wrote:

>> 
>> Thank you everyone for your suggestions, 
>> 
>> I believe that kicking off the function via stellar would be the optimal 
>> solution. If anyone has an example of calling external code via stellar 
>> that would be very helpful. Thanks! 
>> 
>> Regards, 
>> 
>> Tyler Moore 
>> IT Specialist 
>> tyler.mathieu@yahoo.com 
>> 248-909-2769 <(248)%20909-2769> 
>> 
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwards@gmail.com> wrote:

>> > 
>> > They are all extension points. 
>> > 
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com) 
>> wrote: 
>> >> 
>> >> Right but unless I'm missing something, both of those options are more 
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to

>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise 
>> that 
>> >> I'm missing, I would think doing this in Stellar gives you a more timely

>> >> and (re)configurable end result. 
>> >> 
>> >> Jon 
>> >> 
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com>

>> wrote: 
>> >>> 
>> >>> I think there are a couple of things you can do here. There way to get

>> >>> something else into the split is to have another adapter to split to,

>> which 
>> >>> is what I think you mean. You can also integrate with MaaS and create

>> a 
>> >>> service that you can call via STELLAR. 
>> >>> 
>> >>> 
>> >>> 
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com

>> ) 
>> >>> wrote: 
>> >>> 
>> >>> Or a Maas service? 
>> >>> 
>> >>> 
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)

>> >>> wrote: 
>> >>> 
>> >>> Depending on the details it sounds like a much simpler solution would

>> be 
>> >>> to 
>> >>> handle this in a Stellar function. 
>> >>> 
>> >>> Jon 
>> >>> 
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmoore@goflyball.com>
wrote: 
>> >>>> 
>> >>>> Happy Holidays Metron Devs! 
>> >>>> 
>> >>>> Could anyone lend me some guidance on customizing the storm topologies

>> >>> in 
>> >>>> metron? What I am am trying to accomplish: 
>> >>>> 
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http

>> post 
>> >>>> with the score of the threat to a remote rest api. This will 
>> >>> conditionally 
>> >>>> trigger notifications based on user settings in another database
(the 
>> >>>> backend processing logic is on another platform). 
>> >>>> The score should be available within the JSONObject but I am not
an 
>> >>> expert 
>> >>>> with storm and I am not completely understanding what conditions

>> >>> constitute 
>> >>>> when the threat feed is considered an "alert" in metron. Please

>> clarify. 
>> >>>> 
>> >>>> 2) How would I add an external dependency, my http rest java class,
to 
>> >>> the 
>> >>>> metron maven build process? More specifically, if I was adding a

>> custom 
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in 
>> >>>> maven as a dependency. I have limited experience with maven but,
my 
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.

>> >>>> Although, the metron quick dev platform is built on a vm, would
I need 
>> >>> to 
>> >>>> account for this? Please advise. 
>> >>>> 
>> >>>> ​Regards,​ 
>> >>>> 
>> >>>> Tyler Moore 
>> >>>> ​​ 
>> >>>> 
>> >>>> Software Engineer 
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> 
>> >>>> Email: moore.tyler@goflyball.com 
>> >>>> 
>> >>> -- 
>> >>> 
>> >>> Jon 
>> >>> 
>> >>> Sent from my mobile device 
>> >>> 
>> >>> -- 
>> >> 
>> >> Jon 
>> >> 
>> >> Sent from my mobile device 
>> >> 
>> 
>> -- 
>> 
>> Jon 
>> 
>> Sent from my mobile device 
>> 






Mime
View raw message