metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carolyn Duby <cd...@hortonworks.com>
Subject Re: Custom Storm Topologies
Date Tue, 03 Jan 2017 20:47:22 GMT
Also please consider the security of the scripts and script injection attacks.  For example,
we should probably restrict file access.

Thanks
Carolyn



On 1/3/17, 3:25 PM, "Otto Fowler" <ottobackwards@gmail.com> wrote:

>A script bolt would still allow them to write the script the way they want
>to, but would avoid having to write all the scaffolding.
>The matter then would be how to integrate that script bolt into the
>topologies.
>
>
>On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>
>Right, that definitely is more efficient, but part of the point here is to
>lower the barrier of entry to using Metron.
>
>It makes Metron's triage abilities more flexible and allows a user to reuse
>existing code quickly and easily.  Having this available for PoC,
>prototyping, and low volume environments or situations (only when threat
>score is 100, for instance) is important, as it lowers the barrier to entry
>of migrating a company to a Metron environment.
>
>I see this as a tradeoff where I would prioritize ease of use over
>efficiency.  There's nothing wrong with making both options available, at
>some point, and making their different use cases clear.
>
>Jon
>
>On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <mattf@apache.org> wrote:
>
>Well, yes :-)
>And clearly it should always be more efficient to write a custom bolt in
>Java than to invoke a script and manage it.
>
>--Matt
>
>From: Otto Fowler <ottobackwards@gmail.com>
>Date: Tuesday, January 3, 2017 at 7:08 AM
>To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>,
>Matt Foley <mattf@apache.org>
>Subject: Re: Custom Storm Topologies
>
>Wouldn’t that be a bolt?
>
>
>On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
>Should we consider a script calling capability that can launch a streaming
>script and keep it alive and fed, long-term, rather than launching the
>script anew every time the Stellar function is invoked? I’m thinking two
>basic rules: Write a line, read a line; and always have a timeout. Prob
>need a UID of some sort for a cache of running process objects.
>
>--Matt
>
>On 1/2/17, 8:50 AM, "Carolyn Duby" <cduby@hortonworks.com> wrote:
>
>
>Inserting a script inline is ok for low throughput and prototyping but once
>you get higher throughput (millions of events per second), it’s probably
>going to be a bottleneck.
>
>
>For Metron-571 you might want to consider a java based extension plugin
>similar to Eclipse plugins.
>
>Thanks
>Carolyn
>
>On 12/31/16, 5:22 PM, "Tyler Moore" <tmoore@goflyball.com> wrote:
>
>>Thanks Jon,
>>
>>I'll look over the tutorial and put something together for the SHELL_EXEC
>>stellar function.
>>I don't believe I have permissions to assign in Jira if you want to assign
>>to me my username is devopsec.
>>I'll post back details and we can review security issues
>>
>>Regards,
>>
>>Tyler Moore
>>Software Engineer
>>Phone: 248-909-2769 <(248)%20909-2769>
>>Email: moore.tyler@goflyball.com
>>
>>
>>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <zeolla@gmail.com> wrote:
>>
>>> Casey did a tutorial on how to add your own Stellar function here
>>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>>> function that does this (current functions are listed here
>>> <https://github.com/apache/incubator-metron/tree/master/
>>> metron-platform/metron-common#stellar-core-functions>).
>>> I noticed that some of the Stellar function documentation was a bit dated
>>> so I've opened a PR to update it here
>>> <https://github.com/apache/incubator-metron/pull/407>.
>>>
>>> As this is something I need as well, I'd be happy to assist you where I
>>> can. Perhaps you want to self-assign METRON-571
>>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>>> security concerns with a SHELL_EXEC function because it could result in
>RCE
>>> - if that's the route you go I could probably help with a thorough secure
>>> code review.
>>>
>>> Jon
>>>
>>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmoore@goflyball.com>
>wrote:
>>>
>>> Thank you everyone for your suggestions,
>>>
>>> I believe that kicking off the function via stellar would be the optimal
>>> solution. If anyone has an example of calling external code via stellar
>>> that would be very helpful. Thanks!
>>>
>>> Regards,
>>>
>>> Tyler Moore
>>> IT Specialist
>>> tyler.mathieu@yahoo.com
>>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>>
>>> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwards@gmail.com> wrote:
>>> >
>>> > They are all extension points.
>>> >
>>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>>> wrote:
>>> >>
>>> >> Right but unless I'm missing something, both of those options are more
>>> >> rigid and the MaaS service would have an unnecessary delay as opposed
>to
>>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>>> that
>>> >> I'm missing, I would think doing this in Stellar gives you a more
>timely
>>> >> and (re)configurable end result.
>>> >>
>>> >> Jon
>>> >>
>>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com>
>>> wrote:
>>> >>>
>>> >>> I think there are a couple of things you can do here. There way
to
>get
>>> >>> something else into the split is to have another adapter to split
to,
>>> which
>>> >>> is what I think you mean. You can also integrate with MaaS and create
>>> a
>>> >>> service that you can call via STELLAR.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
>ottobackwards@gmail.com
>>> )
>>> >>> wrote:
>>> >>>
>>> >>> Or a Maas service?
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>>> >>> wrote:
>>> >>>
>>> >>> Depending on the details it sounds like a much simpler solution
would
>>> be
>>> >>> to
>>> >>> handle this in a Stellar function.
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmoore@goflyball.com>
>wrote:
>>> >>>>
>>> >>>> Happy Holidays Metron Devs!
>>> >>>>
>>> >>>> Could anyone lend me some guidance on customizing the storm
>topologies
>>> >>> in
>>> >>>> metron? What I am am trying to accomplish:
>>> >>>>
>>> >>>> 1) Add a method to the threat intel joiner bolt that sends an
http
>>> post
>>> >>>> with the score of the threat to a remote rest api. This will
>>> >>> conditionally
>>> >>>> trigger notifications based on user settings in another database
>(the
>>> >>>> backend processing logic is on another platform).
>>> >>>> The score should be available within the JSONObject but I am
not an
>>> >>> expert
>>> >>>> with storm and I am not completely understanding what conditions
>>> >>> constitute
>>> >>>> when the threat feed is considered an "alert" in metron. Please
>>> clarify.
>>> >>>>
>>> >>>> 2) How would I add an external dependency, my http rest java
class,
>to
>>> >>> the
>>> >>>> metron maven build process? More specifically, if I was adding
a
>>> custom
>>> >>>> class that needed accessed by a bolt in storm, how would I add
this
>in
>>> >>>> maven as a dependency. I have limited experience with maven
but, my
>>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>>> >>>> Although, the metron quick dev platform is built on a vm, would
I
>need
>>> >>> to
>>> >>>> account for this? Please advise.
>>> >>>>
>>> >>>> ​Regards,​
>>> >>>>
>>> >>>> Tyler Moore
>>> >>>> ​​
>>> >>>>
>>> >>>> Software Engineer
>>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>> >>>> Email: moore.tyler@goflyball.com
>>> >>>>
>>> >>> --
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>> Sent from my mobile device
>>> >>>
>>> >>> --
>>> >>
>>> >> Jon
>>> >>
>>> >> Sent from my mobile device
>>> >>
>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>
>
>
>
>
>--
>
>Jon
>
>Sent from my mobile device
Mime
View raw message