metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [DISCUSS] Pcap panel architecture
Date Thu, 03 May 2018 22:19:31 GMT
If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the
alerts ui anymore.

It is becoming more of a “composite” app, with multiple feature ui’s
together.  I didn’t think that
was what we were going for, thus the config ui and the alert ui.

Just adding disparate thing as ‘new tabs’ to a ui may be expedient but it
seems contrary to
our previous direction.

There are a few things to consider if we are going to start moving
everything into Alerts Ui aren’t there?

It may be a better road to bring it in on it’s own like the alerts ui
effort, so it can be released with ‘qualifiers’ and tested with
the right expectations without effecting the Alerts UI.



On May 3, 2018 at 17:25:54, Ryan Merriman (merrimanr@gmail.com) wrote:

Otto,

I'm assuming just adding it to the Alerts UI is less work but I wouldn't be
strongly opposed to it being it's own UI. What are the reasons for doing
that?

Mike,

On using metron-api:

1. I'm making an assumption about it not being used much. Maybe it
still works without issue. I agree, we'll have to test anything we build
so this is a minor issue.
2. Updating metron-api to be asynchronous is a requirement in my opinion
3. The MPack work is the major drawback for me. We're essentially
creating a brand new Metron component. There are a lot of examples we can
draw from but it's going to be a large chunk of new MPack code to maintain
and MPack development has been painful in the past. I think it will
include:
1. Creating a start script
2. Creating master.py and commands.py scripts for managing the
application lifecycle, service checks, etc
3. Creating an -env.xml file for exposing properties in Ambari
4. Adding the component to the various MPack files
(metron_theme.json, metainfo.xml, service_advisor.py, etc.)
4. Our Storm topologies are completely different use cases and much more
complex so I don't understand the comparison. But if you prefer this
coding style then I think this is a minor issue as well.

On micro-services:

1. Our REST service already includes a lot of dependencies and is
difficult to manage in it's current state. I just went through this on
https://github.com/apache/metron/pull/1008. It was painful. When we
tried to include mapreduce and yarn dependencies it became what seemed like
an endless NoSuchMethod, NoClassDef and similar errors. Even if we can get
it to work it's going to make managing our REST service that much harder
than it already is. I think the shaded jars are the source of all this
trouble and I agree it would be nice to improve our architecture in this
area. However I don't think it's a simple fix and now we're getting into
the "will likely take a long time to plan and implement" concern. If
anyone has ideas on how to solve our shaded jar challenge I would be all
for it.
2. All the MPack work listed above would also be required here. A
micro-services pattern is a significant shift and can't even give you
concrete examples of what exactly we would have to do. We would need to go
through extensive design and planning to even get to that point.
3. It would be a branch new component. See above plus any new
infrastructure we would need (web server/proxy, service discovery, etc)

On pcap-query:

1. I don't recall any users or customers directly using metron-api but
if you say so I believe you :)
2. As I understand it the pcap topology and pcap query are somewhat
decoupled. Maybe location of pcap files would be shared? MPack work here
is likely to include adding a couple properties and moving some around so
they can be shared. Deciding between Ambari and global config would be
similar to properties we add to any component.

I think you may be underestimating how difficult it's going to be to solve
our dependency problem. Or maybe it's me that is overestimating it :) It
could be something we experiment with before we start on the pcap work.
There is major upside and it would benefit the whole project. But until
then we can't fit anymore more screwdrivers in the toolbox. For me the
only reasonable options are to use the existing metron-api as it's own
separate service or call out to the pcap_query.sh script from our existing
REST app. I could go either way really. I'm just not excited about all
the MPack code we have to write for a new component. Maybe it won't be
that bad.

On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> First thought is why the Alerts-UI and Not a dedicated Query UI?
>
>
> On May 3, 2018 at 14:36:04, Ryan Merriman (merrimanr@gmail.com) wrote:
>
> We are planning on adding the pcap query feature to the Alerts UI. Before
> we start this work, I think it is important to get community buy in on
the
> architectural approach. There are a couple different options.
>
> One option is to leverage the existing metron-api module that exposes
pcap
> queries through a REST service. The upsides are:
>
> - some work has already been done
> - it's part of our build so we know unit and integration tests pass
>
> The downsides are:
>
> - It hasn't been used in a while and will need some end to end testing
> to make sure it still functions properly
> - It is synchronous and will block the UI, using up the limited number
> of concurrent connections available in a browser
> - It will require significant MPack work to properly set it up on install
> - It is a legacy module from OpenSOC and coding style is significantly
> different
>
> Another option would be moving to a micro-services architecture. We have
> experimented with a proof of concept and found it was too hard to add
this
> feature into our existing REST services because of all the dependencies
> that must coexist in the same application. The upsides are:
>
> - Would provide a platform for future Batch/MR/YARN type features
> - There would be fewer technical compromises since we are building it
> from the ground up
>
> The downsides are:
>
> - Will require the most effort and will likely take a long time to plan
> and implement
> - Like the previous option, will require significant MPack work
>
> A third option would be to add an endpoint to our existing REST service
> that delegates to the pcap_query.sh script through the Java Process
class.
> The upsides to this approach are:
>
> - We know the pcap_query.sh script works and would require minimal
> changes
> - Minimal MPack work is required since our REST service is already
> included
>
> The downsides are:
>
> - Does not set us up to easily add other batch-oriented features in the
> future
> - OS-level security becomes a concern since we are delegating to a
> script in a separate process
>
> I feel like ultimately we want to transition to a micro-services
> architecture because it will provide more flexibility and make it easier
> to
> grow our set of features. But in the meantime, wrapping the pcap_query.sh
> script would allow us to add this feature with less work and fewer lines
> of
> code. If and when we decide to deploy a separate REST application for
> batch features, the UI portion would require minimal changes.
>
> What does everyone think?
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message