metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS] Pcap panel architecture
Date Fri, 04 May 2018 03:27:19 GMT
I know, I was running with it :)

> On May 3, 2018, at 10:21 PM, Michael Miklavcic <michael.miklavcic@gmail.com> wrote:
> 
> Tabs vs spaces was a Silicon Valley joke, man :-)
> 
>> On Thu, May 3, 2018, 8:42 PM Ryan Merriman <merrimanr@gmail.com> wrote:
>> 
>> Mike,
>> 
>> I never said there was anything problematic in metron-api, just that is was
>> inconsistent with the rest of Metron.  There is work involved in making it
>> consistent which is why I listed it as a downside.  I'm less concerned with
>> whether we use tabs or spaces but that we use one or the other.
>> 
>> I apologize for not making this clearer in my original message, but I did
>> not lead the POC development.  My involvement was helping troubleshoot
>> issues they ran into and answering questions about Metron in general.  I've
>> shared with you the information that I have which is my observations about
>> the types of issues they ran into.  I don't have a branch or pom file you
>> can experiment with.  I will reach out to that person and see if they are
>> able to share the exact errors they hit.  Also, the "trade-offs that you
>> seem to have already decided on" is not based on a specific issue or
>> challenge they faced in the POC.  It's based off of the past couple years
>> of working on our REST module and the reoccurring challenges and patterns I
>> see over a period of time.
>> 
>> Otto,
>> 
>> Makes sense to me.  I will start the other threads.
>> 
>> On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwards@gmail.com>
>> wrote:
>> 
>>> I think my point is that maybe we should have a discuss about:
>>> 
>>> * PCAP UI, goals etc
>>> * Where it would live and why, what that would mean etc
>>> * Backend ( this original mail )
>>> 
>>> 
>>> 
>>> On May 3, 2018 at 18:34:00, Michael Miklavcic (
>> michael.miklavcic@gmail.com)
>>> wrote:
>>> 
>>> Otto, what are you and your customers finding useful and/or difficult
>> from
>>> a split management/alerts UI perspective? It might help us to restate the
>>> original scope and intent around maintaining separate management and
>> alert
>>> UI's, to your point about "contrary to previous direction." I personally
>>> don't have a strong position on this other than 1) management is a
>>> different feature set from drilling into threat intel, yet many apps
>> still
>>> have their management UI combined with the end user experience and 2) we
>>> should probably consider pcap in context of a workflow with alerts.
>>> 
>>> On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwards@gmail.com>
>>> wrote:
>>> 
>>>> If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t
>> the
>>>> alerts ui anymore.
>>>> 
>>>> It is becoming more of a “composite” app, with multiple feature ui’s
>>>> together. I didn’t think that
>>>> was what we were going for, thus the config ui and the alert ui.
>>>> 
>>>> Just adding disparate thing as ‘new tabs’ to a ui may be expedient but
>>> it
>>>> seems contrary to
>>>> our previous direction.
>>>> 
>>>> There are a few things to consider if we are going to start moving
>>>> everything into Alerts Ui aren’t there?
>>>> 
>>>> It may be a better road to bring it in on it’s own like the alerts ui
>>>> effort, so it can be released with ‘qualifiers’ and tested with
>>>> the right expectations without effecting the Alerts UI.
>>>> 
>>>> 
>>>> 
>>>> On May 3, 2018 at 17:25:54, Ryan Merriman (merrimanr@gmail.com) wrote:
>>>> 
>>>> Otto,
>>>> 
>>>> I'm assuming just adding it to the Alerts UI is less work but I
>> wouldn't
>>> be
>>>> strongly opposed to it being it's own UI. What are the reasons for
>> doing
>>>> that?
>>>> 
>>>> Mike,
>>>> 
>>>> On using metron-api:
>>>> 
>>>> 1. I'm making an assumption about it not being used much. Maybe it
>>>> still works without issue. I agree, we'll have to test anything we
>> build
>>>> so this is a minor issue.
>>>> 2. Updating metron-api to be asynchronous is a requirement in my
>> opinion
>>>> 3. The MPack work is the major drawback for me. We're essentially
>>>> creating a brand new Metron component. There are a lot of examples we
>>> can
>>>> draw from but it's going to be a large chunk of new MPack code to
>>> maintain
>>>> and MPack development has been painful in the past. I think it will
>>>> include:
>>>> 1. Creating a start script
>>>> 2. Creating master.py and commands.py scripts for managing the
>>>> application lifecycle, service checks, etc
>>>> 3. Creating an -env.xml file for exposing properties in Ambari
>>>> 4. Adding the component to the various MPack files
>>>> (metron_theme.json, metainfo.xml, service_advisor.py, etc.)
>>>> 4. Our Storm topologies are completely different use cases and much
>> more
>>>> complex so I don't understand the comparison. But if you prefer this
>>>> coding style then I think this is a minor issue as well.
>>>> 
>>>> On micro-services:
>>>> 
>>>> 1. Our REST service already includes a lot of dependencies and is
>>>> difficult to manage in it's current state. I just went through this on
>>>> https://github.com/apache/metron/pull/1008. It was painful. When we
>>>> tried to include mapreduce and yarn dependencies it became what seemed
>>> like
>>>> an endless NoSuchMethod, NoClassDef and similar errors. Even if we can
>>> get
>>>> it to work it's going to make managing our REST service that much
>> harder
>>>> than it already is. I think the shaded jars are the source of all this
>>>> trouble and I agree it would be nice to improve our architecture in
>> this
>>>> area. However I don't think it's a simple fix and now we're getting
>> into
>>>> the "will likely take a long time to plan and implement" concern. If
>>>> anyone has ideas on how to solve our shaded jar challenge I would be
>> all
>>>> for it.
>>>> 2. All the MPack work listed above would also be required here. A
>>>> micro-services pattern is a significant shift and can't even give you
>>>> concrete examples of what exactly we would have to do. We would need to
>>> go
>>>> through extensive design and planning to even get to that point.
>>>> 3. It would be a branch new component. See above plus any new
>>>> infrastructure we would need (web server/proxy, service discovery, etc)
>>>> 
>>>> On pcap-query:
>>>> 
>>>> 1. I don't recall any users or customers directly using metron-api but
>>>> if you say so I believe you :)
>>>> 2. As I understand it the pcap topology and pcap query are somewhat
>>>> decoupled. Maybe location of pcap files would be shared? MPack work
>> here
>>>> is likely to include adding a couple properties and moving some around
>>> so
>>>> they can be shared. Deciding between Ambari and global config would be
>>>> similar to properties we add to any component.
>>>> 
>>>> I think you may be underestimating how difficult it's going to be to
>>> solve
>>>> our dependency problem. Or maybe it's me that is overestimating it :)
>> It
>>>> could be something we experiment with before we start on the pcap work.
>>>> There is major upside and it would benefit the whole project. But until
>>>> then we can't fit anymore more screwdrivers in the toolbox. For me the
>>>> only reasonable options are to use the existing metron-api as it's own
>>>> separate service or call out to the pcap_query.sh script from our
>>> existing
>>>> REST app. I could go either way really. I'm just not excited about all
>>>> the MPack code we have to write for a new component. Maybe it won't be
>>>> that bad.
>>>> 
>>>> On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwards@gmail.com>
>>>> wrote:
>>>> 
>>>>> First thought is why the Alerts-UI and Not a dedicated Query UI?
>>>>> 
>>>>> 
>>>>> On May 3, 2018 at 14:36:04, Ryan Merriman (merrimanr@gmail.com)
>>> wrote:
>>>>> 
>>>>> We are planning on adding the pcap query feature to the Alerts UI.
>>> Before
>>>>> we start this work, I think it is important to get community buy in
>> on
>>>> the
>>>>> architectural approach. There are a couple different options.
>>>>> 
>>>>> One option is to leverage the existing metron-api module that exposes
>>>> pcap
>>>>> queries through a REST service. The upsides are:
>>>>> 
>>>>> - some work has already been done
>>>>> - it's part of our build so we know unit and integration tests pass
>>>>> 
>>>>> The downsides are:
>>>>> 
>>>>> - It hasn't been used in a while and will need some end to end
>> testing
>>>>> to make sure it still functions properly
>>>>> - It is synchronous and will block the UI, using up the limited
>> number
>>>>> of concurrent connections available in a browser
>>>>> - It will require significant MPack work to properly set it up on
>>> install
>>>>> - It is a legacy module from OpenSOC and coding style is
>> significantly
>>>>> different
>>>>> 
>>>>> Another option would be moving to a micro-services architecture. We
>>> have
>>>>> experimented with a proof of concept and found it was too hard to add
>>>> this
>>>>> feature into our existing REST services because of all the
>>> dependencies
>>>>> that must coexist in the same application. The upsides are:
>>>>> 
>>>>> - Would provide a platform for future Batch/MR/YARN type features
>>>>> - There would be fewer technical compromises since we are building it
>>>>> from the ground up
>>>>> 
>>>>> The downsides are:
>>>>> 
>>>>> - Will require the most effort and will likely take a long time to
>>> plan
>>>>> and implement
>>>>> - Like the previous option, will require significant MPack work
>>>>> 
>>>>> A third option would be to add an endpoint to our existing REST
>>> service
>>>>> that delegates to the pcap_query.sh script through the Java Process
>>>> class.
>>>>> The upsides to this approach are:
>>>>> 
>>>>> - We know the pcap_query.sh script works and would require minimal
>>>>> changes
>>>>> - Minimal MPack work is required since our REST service is already
>>>>> included
>>>>> 
>>>>> The downsides are:
>>>>> 
>>>>> - Does not set us up to easily add other batch-oriented features in
>>> the
>>>>> future
>>>>> - OS-level security becomes a concern since we are delegating to a
>>>>> script in a separate process
>>>>> 
>>>>> I feel like ultimately we want to transition to a micro-services
>>>>> architecture because it will provide more flexibility and make it
>>> easier
>>>>> to
>>>>> grow our set of features. But in the meantime, wrapping the
>>> pcap_query.sh
>>>>> script would allow us to add this feature with less work and fewer
>>> lines
>>>>> of
>>>>> code. If and when we decide to deploy a separate REST application for
>>>>> batch features, the UI portion would require minimal changes.
>>>>> 
>>>>> What does everyone think?
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 

Mime
View raw message