metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: Custom Storm Topologies
Date Tue, 03 Jan 2017 20:17:48 GMT
Right, that definitely is more efficient, but part of the point here is to
lower the barrier of entry to using Metron.

It makes Metron's triage abilities more flexible and allows a user to reuse
existing code quickly and easily.  Having this available for PoC,
prototyping, and low volume environments or situations (only when threat
score is 100, for instance) is important, as it lowers the barrier to entry
of migrating a company to a Metron environment.

I see this as a tradeoff where I would prioritize ease of use over
efficiency.  There's nothing wrong with making both options available, at
some point, and making their different use cases clear.

Jon

On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <mattf@apache.org> wrote:

Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in
Java than to invoke a script and manage it.

--Matt

From: Otto Fowler <ottobackwards@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>,
Matt Foley <mattf@apache.org>
Subject: Re: Custom Storm Topologies

Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cduby@hortonworks.com> wrote:


Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tmoore@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769 <(248)%20909-2769>
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <zeolla@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmoore@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwards@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmoore@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not
an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but,
my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would
I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>





-- 

Jon

Sent from my mobile device

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message