metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS] Pcap panel architecture
Date Fri, 04 May 2018 02:42:16 GMT
Mike,

I never said there was anything problematic in metron-api, just that is was
inconsistent with the rest of Metron.  There is work involved in making it
consistent which is why I listed it as a downside.  I'm less concerned with
whether we use tabs or spaces but that we use one or the other.

I apologize for not making this clearer in my original message, but I did
not lead the POC development.  My involvement was helping troubleshoot
issues they ran into and answering questions about Metron in general.  I've
shared with you the information that I have which is my observations about
the types of issues they ran into.  I don't have a branch or pom file you
can experiment with.  I will reach out to that person and see if they are
able to share the exact errors they hit.  Also, the "trade-offs that you
seem to have already decided on" is not based on a specific issue or
challenge they faced in the POC.  It's based off of the past couple years
of working on our REST module and the reoccurring challenges and patterns I
see over a period of time.

Otto,

Makes sense to me.  I will start the other threads.

On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwards@gmail.com> wrote:

> I think my point is that maybe we should have a discuss about:
>
> * PCAP UI, goals etc
> * Where it would live and why, what that would mean etc
> * Backend ( this original mail )
>
>
>
> On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklavcic@gmail.com)
> wrote:
>
> Otto, what are you and your customers finding useful and/or difficult from
> a split management/alerts UI perspective? It might help us to restate the
> original scope and intent around maintaining separate management and alert
> UI's, to your point about "contrary to previous direction." I personally
> don't have a strong position on this other than 1) management is a
> different feature set from drilling into threat intel, yet many apps still
> have their management UI combined with the end user experience and 2) we
> should probably consider pcap in context of a workflow with alerts.
>
> On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
> > If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the
> > alerts ui anymore.
> >
> > It is becoming more of a “composite” app, with multiple feature ui’s
> > together. I didn’t think that
> > was what we were going for, thus the config ui and the alert ui.
> >
> > Just adding disparate thing as ‘new tabs’ to a ui may be expedient but
> it
> > seems contrary to
> > our previous direction.
> >
> > There are a few things to consider if we are going to start moving
> > everything into Alerts Ui aren’t there?
> >
> > It may be a better road to bring it in on it’s own like the alerts ui
> > effort, so it can be released with ‘qualifiers’ and tested with
> > the right expectations without effecting the Alerts UI.
> >
> >
> >
> > On May 3, 2018 at 17:25:54, Ryan Merriman (merrimanr@gmail.com) wrote:
> >
> > Otto,
> >
> > I'm assuming just adding it to the Alerts UI is less work but I wouldn't
> be
> > strongly opposed to it being it's own UI. What are the reasons for doing
> > that?
> >
> > Mike,
> >
> > On using metron-api:
> >
> > 1. I'm making an assumption about it not being used much. Maybe it
> > still works without issue. I agree, we'll have to test anything we build
> > so this is a minor issue.
> > 2. Updating metron-api to be asynchronous is a requirement in my opinion
> > 3. The MPack work is the major drawback for me. We're essentially
> > creating a brand new Metron component. There are a lot of examples we
> can
> > draw from but it's going to be a large chunk of new MPack code to
> maintain
> > and MPack development has been painful in the past. I think it will
> > include:
> > 1. Creating a start script
> > 2. Creating master.py and commands.py scripts for managing the
> > application lifecycle, service checks, etc
> > 3. Creating an -env.xml file for exposing properties in Ambari
> > 4. Adding the component to the various MPack files
> > (metron_theme.json, metainfo.xml, service_advisor.py, etc.)
> > 4. Our Storm topologies are completely different use cases and much more
> > complex so I don't understand the comparison. But if you prefer this
> > coding style then I think this is a minor issue as well.
> >
> > On micro-services:
> >
> > 1. Our REST service already includes a lot of dependencies and is
> > difficult to manage in it's current state. I just went through this on
> > https://github.com/apache/metron/pull/1008. It was painful. When we
> > tried to include mapreduce and yarn dependencies it became what seemed
> like
> > an endless NoSuchMethod, NoClassDef and similar errors. Even if we can
> get
> > it to work it's going to make managing our REST service that much harder
> > than it already is. I think the shaded jars are the source of all this
> > trouble and I agree it would be nice to improve our architecture in this
> > area. However I don't think it's a simple fix and now we're getting into
> > the "will likely take a long time to plan and implement" concern. If
> > anyone has ideas on how to solve our shaded jar challenge I would be all
> > for it.
> > 2. All the MPack work listed above would also be required here. A
> > micro-services pattern is a significant shift and can't even give you
> > concrete examples of what exactly we would have to do. We would need to
> go
> > through extensive design and planning to even get to that point.
> > 3. It would be a branch new component. See above plus any new
> > infrastructure we would need (web server/proxy, service discovery, etc)
> >
> > On pcap-query:
> >
> > 1. I don't recall any users or customers directly using metron-api but
> > if you say so I believe you :)
> > 2. As I understand it the pcap topology and pcap query are somewhat
> > decoupled. Maybe location of pcap files would be shared? MPack work here
> > is likely to include adding a couple properties and moving some around
> so
> > they can be shared. Deciding between Ambari and global config would be
> > similar to properties we add to any component.
> >
> > I think you may be underestimating how difficult it's going to be to
> solve
> > our dependency problem. Or maybe it's me that is overestimating it :) It
> > could be something we experiment with before we start on the pcap work.
> > There is major upside and it would benefit the whole project. But until
> > then we can't fit anymore more screwdrivers in the toolbox. For me the
> > only reasonable options are to use the existing metron-api as it's own
> > separate service or call out to the pcap_query.sh script from our
> existing
> > REST app. I could go either way really. I'm just not excited about all
> > the MPack code we have to write for a new component. Maybe it won't be
> > that bad.
> >
> > On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwards@gmail.com>
> > wrote:
> >
> > > First thought is why the Alerts-UI and Not a dedicated Query UI?
> > >
> > >
> > > On May 3, 2018 at 14:36:04, Ryan Merriman (merrimanr@gmail.com)
> wrote:
> > >
> > > We are planning on adding the pcap query feature to the Alerts UI.
> Before
> > > we start this work, I think it is important to get community buy in on
> > the
> > > architectural approach. There are a couple different options.
> > >
> > > One option is to leverage the existing metron-api module that exposes
> > pcap
> > > queries through a REST service. The upsides are:
> > >
> > > - some work has already been done
> > > - it's part of our build so we know unit and integration tests pass
> > >
> > > The downsides are:
> > >
> > > - It hasn't been used in a while and will need some end to end testing
> > > to make sure it still functions properly
> > > - It is synchronous and will block the UI, using up the limited number
> > > of concurrent connections available in a browser
> > > - It will require significant MPack work to properly set it up on
> install
> > > - It is a legacy module from OpenSOC and coding style is significantly
> > > different
> > >
> > > Another option would be moving to a micro-services architecture. We
> have
> > > experimented with a proof of concept and found it was too hard to add
> > this
> > > feature into our existing REST services because of all the
> dependencies
> > > that must coexist in the same application. The upsides are:
> > >
> > > - Would provide a platform for future Batch/MR/YARN type features
> > > - There would be fewer technical compromises since we are building it
> > > from the ground up
> > >
> > > The downsides are:
> > >
> > > - Will require the most effort and will likely take a long time to
> plan
> > > and implement
> > > - Like the previous option, will require significant MPack work
> > >
> > > A third option would be to add an endpoint to our existing REST
> service
> > > that delegates to the pcap_query.sh script through the Java Process
> > class.
> > > The upsides to this approach are:
> > >
> > > - We know the pcap_query.sh script works and would require minimal
> > > changes
> > > - Minimal MPack work is required since our REST service is already
> > > included
> > >
> > > The downsides are:
> > >
> > > - Does not set us up to easily add other batch-oriented features in
> the
> > > future
> > > - OS-level security becomes a concern since we are delegating to a
> > > script in a separate process
> > >
> > > I feel like ultimately we want to transition to a micro-services
> > > architecture because it will provide more flexibility and make it
> easier
> > > to
> > > grow our set of features. But in the meantime, wrapping the
> pcap_query.sh
> > > script would allow us to add this feature with less work and fewer
> lines
> > > of
> > > code. If and when we decide to deploy a separate REST application for
> > > batch features, the UI portion would require minimal changes.
> > >
> > > What does everyone think?
> > >
> > >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message