metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carolyn Duby <cd...@hortonworks.com>
Subject Re: Custom Storm Topologies
Date Mon, 02 Jan 2017 16:50:42 GMT

Inserting a script inline is ok for low throughput and prototyping but once you get higher
throughput (millions of events per second), it’s probably going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin similar to Eclipse
plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tmoore@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <zeolla@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can.  Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>?  I do have some
>> security concerns with a SHELL_EXEC function because it could result in RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmoore@goflyball.com> wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwards@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to
>> >> doing it entirely in Stellar.  Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here.  There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean.  You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmoore@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not
an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but,
my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would
I need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Mime
View raw message