metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <>
Subject Re: [DISCUSS] Pcap panel architecture
Date Thu, 03 May 2018 19:38:04 GMT
Thanks for the write-up, Ryan. A few questions and comments.

   1. metron-api
      1. "It hasn't been used in a while and will need some end to end
      testing to make sure it still functions properly" > I was probably
      one of the last developers to touch this code a year or more ago
- fwiw, I
      didn't encounter any major issues the last time I ran it up. That aside,
      end to end testing will be much more critical for a new feature.
      2. Mpack work - can you list what you think is needed? Isn't this
      just a jar and some changes to the UI?
      3. This actually might not be a negative, per se. Can you be specific
      about the issues you see in the older code? Personally, I found it much
      quicker to pick up than the Storm topology classes I've been working with
      lately (parser, enrichment, and indexing bolts with 5+ levels of class
      hierarchies with enrichment bolts extending configured indexing bolts, 2+
      types of undocumented initialization routines, BulkMessageWriters,
      BulkMessageComponents, BulkMessageHandlers, AbstractWriters,
      MessageWriters, and a few dozen configuration types used for
      reading/writing to Zookeeper and disc and maintaining an in-memory cache).
   2. microservices
      1. "We have experimented with a proof of concept and found it was too
      hard to add this feature into our existing REST services because of
      all the dependencies that must coexist in the same application."
> Can you
      share the POC example and/or explain the hurdles along with specific
      dependency errors encountered? With Storm not providing
classpath isolation
      and containerization, we have a number of existing
shaded/relocated modules
      in our system. This may be a simple fix and/or an opportunity to improve
      our existing architecture rather than add more non-standard approaches to
      the mix.
      2. What aspects of this approach "will require the most effort?" What
      specifically makes this more work than the other strategies?
      3. Again, can you enumerate the MPack work? What is net-new or does
      not fit with the existing deployment strategy?
      1. "We know the script works and would require
minimal changes"
      > this would be no different from metron-api, no?
      2. How would you manage configuration between the UI and pcap
      topology? Would this go in Ambari, management UI, global config - mpack
      work for this?

My take is that this belongs in the existing REST API, as you mention. I'm
not sure how I feel about calling the from Java - it seems a
bit hacky, like we're taking a shortcut to avoid fixing another problem
that will cause us to provide a solution that is inconsistent with the rest
of our REST app. It's like having a philips head screwdriver that plugs
into a philips screw that adapts to a flat-head screwdriver end and plugs
into a flat-head screw. Just use a flat-head screwdriver man :) One way to
possibly mitigate dependency issues that we've been having is to construct
a new module specifically for managing our external dependencies that have
been perennial problem children, like Guava and Jackson, and exclude them
everywhere else in the project. Either way, we should deprecate the older
metron-api that hosts the stand-alone PCAP REST service as part of this
effort. I don't think we should leave both of them there unless someone has
a good reason otherwise. Pending a better understanding of the dependency
issues encountered, I'm interested to hear what others think of calling the
shell script from REST vs leveraging the PCAP query code directly.

Other feature considerations

   1. Agreed that this should be made asynchronous via the UI. A polling or
   callback mechanism would be useful.
   2. Take care that a user doesn't hit refresh or POST multiple times and
   kick off 50 mapreduce jobs.
   3. Options for managing the YARN queue that is used
   4. Should we provide a "cancel" option that kills the MR job, or tell
   the user to go to the CLI to kill their job?
   5. Managing data if multiple users run queries.
   6. Job cleanup/TTL
   7. Date range limits on queries - PCAP data is massive by comparison to
   other sensors


On Thu, May 3, 2018 at 12:35 PM, Ryan Merriman <> wrote:

> We are planning on adding the pcap query feature to the Alerts UI.  Before
> we start this work, I think it is important to get community buy in on the
> architectural approach.  There are a couple different options.
> One option is to leverage the existing metron-api module that exposes pcap
> queries through a REST service.  The upsides are:
>    - some work has already been done
>    - it's part of our build so we know unit and integration tests pass
> The downsides are:
>    - It hasn't been used in a while and will need some end to end testing
>    to make sure it still functions properly
>    - It is synchronous and will block the UI, using up the limited number
>    of concurrent connections available in a browser
>    - It will require significant MPack work to properly set it up on
> install
>    - It is a legacy module from OpenSOC and coding style is significantly
>    different
> Another option would be moving to a micro-services architecture.  We have
> experimented with a proof of concept and found it was too hard to add this
> feature into our existing REST services because of all the dependencies
> that must coexist in the same application.  The upsides are:
>    - Would provide a platform for future Batch/MR/YARN type features
>    - There would be fewer technical compromises since we are building it
>    from the ground up
> The downsides are:
>    - Will require the most effort and will likely take a long time to plan
>    and implement
>    - Like the previous option, will require significant MPack work
> A third option would be to add an endpoint to our existing REST service
> that delegates to the script through the Java Process class.
> The upsides to this approach are:
>    - We know the script works and would require minimal
>    changes
>    - Minimal MPack work is required since our REST service is already
>    included
> The downsides are:
>    - Does not set us up to easily add other batch-oriented features in the
>    future
>    - OS-level security becomes a concern since we are delegating to a
>    script in a separate process
> I feel like ultimately we want to transition to a micro-services
> architecture because it will provide more flexibility and make it easier to
> grow our set of features.  But in the meantime, wrapping the
> script would allow us to add this feature with less work and fewer lines of
> code.  If and when we decide to deploy a separate REST application for
> batch features, the UI portion would require minimal changes.
> What does everyone think?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message