metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: Custom Storm Topologies
Date Mon, 02 Jan 2017 19:39:30 GMT
Should we consider a script calling capability that can launch a streaming script and keep
it alive and fed, long-term, rather than launching the script anew every time the Stellar
function is invoked?  I’m thinking two basic rules:  Write a line, read a line; and always
have a timeout.  Prob need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cduby@hortonworks.com> wrote:

    
    Inserting a script inline is ok for low throughput and prototyping but once you get higher
throughput (millions of events per second), it’s probably going to be a bottleneck.
    
    
    For Metron-571 you might want to consider a java based extension plugin similar to Eclipse
plugins.
    
    Thanks
    Carolyn
    
    On 12/31/16, 5:22 PM, "Tyler Moore" <tmoore@goflyball.com> wrote:
    
    >Thanks Jon,
    >
    >I'll look over the tutorial and put something together for the SHELL_EXEC
    >stellar function.
    >I don't believe I have permissions to assign in Jira if you want to assign
    >to me my username is devopsec.
    >I'll post back details and we can review security issues
    >
    >Regards,
    >
    >Tyler Moore
    >Software Engineer
    >Phone: 248-909-2769
    >Email: moore.tyler@goflyball.com
    >
    >
    >On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <zeolla@gmail.com> wrote:
    >
    >> Casey did a tutorial on how to add your own Stellar function here
    >> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
    >> function that does this (current functions are listed here
    >> <https://github.com/apache/incubator-metron/tree/master/
    >> metron-platform/metron-common#stellar-core-functions>).
    >> I noticed that some of the Stellar function documentation was a bit dated
    >> so I've opened a PR to update it here
    >> <https://github.com/apache/incubator-metron/pull/407>.
    >>
    >> As this is something I need as well, I'd be happy to assist you where I
    >> can.  Perhaps you want to self-assign METRON-571
    >> <https://issues.apache.org/jira/browse/METRON-571>?  I do have some
    >> security concerns with a SHELL_EXEC function because it could result in RCE
    >> - if that's the route you go I could probably help with a thorough secure
    >> code review.
    >>
    >> Jon
    >>
    >> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmoore@goflyball.com> wrote:
    >>
    >> Thank you everyone for your suggestions,
    >>
    >> I believe that kicking off the function via stellar would be the optimal
    >> solution. If anyone has an example of calling external code via stellar
    >> that would be very helpful. Thanks!
    >>
    >> Regards,
    >>
    >> Tyler Moore
    >> IT Specialist
    >> tyler.mathieu@yahoo.com
    >> 248-909-2769 <(248)%20909-2769>
    >>
    >> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwards@gmail.com> wrote:
    >> >
    >> > They are all extension points.
    >> >
    >> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
    >> wrote:
    >> >>
    >> >> Right but unless I'm missing something, both of those options are more
    >> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
    >> >> doing it entirely in Stellar.  Unless there's a reason to do otherwise
    >> that
    >> >> I'm missing, I would think doing this in Stellar gives you a more timely
    >> >> and (re)configurable end result.
    >> >>
    >> >> Jon
    >> >>
    >> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com>
    >> wrote:
    >> >>>
    >> >>> I think there are a couple of things you can do here.  There way
to get
    >> >>> something else into the split is to have another adapter to split
to,
    >> which
    >> >>> is what I think you mean.  You can also integrate with MaaS and
create
    >> a
    >> >>> service that you can call via STELLAR.
    >> >>>
    >> >>>
    >> >>>
    >> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
    >> )
    >> >>> wrote:
    >> >>>
    >> >>> Or a Maas service?
    >> >>>
    >> >>>
    >> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
    >> >>> wrote:
    >> >>>
    >> >>> Depending on the details it sounds like a much simpler solution
would
    >> be
    >> >>> to
    >> >>> handle this in a Stellar function.
    >> >>>
    >> >>> Jon
    >> >>>
    >> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmoore@goflyball.com>
wrote:
    >> >>>>
    >> >>>> Happy Holidays Metron Devs!
    >> >>>>
    >> >>>> Could anyone lend me some guidance on customizing the storm
topologies
    >> >>> in
    >> >>>> metron? What I am am trying to accomplish:
    >> >>>>
    >> >>>> 1) Add a method to the threat intel joiner bolt that sends an
http
    >> post
    >> >>>> with the score of the threat to a remote rest api. This will
    >> >>> conditionally
    >> >>>> trigger notifications based on user settings in another database
(the
    >> >>>> backend processing logic is on another platform).
    >> >>>> The score should be available within the JSONObject but I am
not an
    >> >>> expert
    >> >>>> with storm and I am not completely understanding what conditions
    >> >>> constitute
    >> >>>> when the threat feed is considered an "alert" in metron. Please
    >> clarify.
    >> >>>>
    >> >>>> 2) How would I add an external dependency, my http rest java
class, to
    >> >>> the
    >> >>>> metron maven build process? More specifically, if I was adding
a
    >> custom
    >> >>>> class that needed accessed by a bolt in storm, how would I add
this in
    >> >>>> maven as a dependency. I have limited experience with maven
but, my
    >> >>>> understanding is that I would add it to the pom.xml ​and recompile.
    >> >>>> Although, the metron quick dev platform is built on a vm, would
I need
    >> >>> to
    >> >>>> account for this? Please advise.
    >> >>>>
    >> >>>> ​Regards,​
    >> >>>>
    >> >>>> Tyler Moore
    >> >>>> ​​
    >> >>>>
    >> >>>> Software Engineer
    >> >>>> Phone: 248-909-2769 <(248)%20909-2769>
    >> >>>> Email: moore.tyler@goflyball.com
    >> >>>>
    >> >>> --
    >> >>>
    >> >>> Jon
    >> >>>
    >> >>> Sent from my mobile device
    >> >>>
    >> >>> --
    >> >>
    >> >> Jon
    >> >>
    >> >> Sent from my mobile device
    >> >>
    >>
    >> --
    >>
    >> Jon
    >>
    >> Sent from my mobile device
    >>
    



Mime
View raw message