metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Pcap UI user requirements
Date Wed, 09 May 2018 18:37:10 GMT
I had a feeling it may be that way.  Unless anyone else knows of a better
approach, it's probably most reasonable to push that into a follow-on JIRA
and not over-complicate the current activities.

Jon

On Wed, May 9, 2018 at 2:33 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> We are limited by Yarn and MapReduce applications in the case of
> pause/resume - I could be wrong, but I don't think that's something that's
> supported unless you're talking about multiple MR jobs strung together.
>
>
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application
>
> I don't see anything suggesting "SUSPENDED" or "PAUSED" as we have
> available in workflow engines like Oozie.
>
> "The valid application state can be one of the following:  ALL, NEW,
> NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED"
>
> Same goes for MR job commands:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job
>
> Mike
>
> On Mon, May 7, 2018 at 2:04 PM, Zeolla@GMail.com <zeolla@gmail.com> wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler <ottobackwards@gmail.com>
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsumbul@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler <ottobackwards@gmail.com>:
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report results, and when the
> > > report
> > > > is done it is queued there
> > > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > > info/meta has the yarn details )
> > > > * You can select the report from your queue and view it either in a
> new
> > > UI
> > > > or custom component
> > > > * You can then apply a different ‘view’ to the report or work with
> the
> > > > report data
> > > > * You can print / save etc
> > > > * You can associate the report with the alerts ( again in the report
> > info
> > > )
> > > > with…. a ‘case’ or ‘ticket’ or investigation something or other
> > > >
> > > >
> > > > We can introduce extensibility into the report templates, report
> views
> > (
> > > > thinks that work with the json data of the report )
> > > >
> > > > Something like that.
> > > >
> > > >
> > > > On May 4, 2018 at 09:19:15, Ryan Merriman (merrimanr@gmail.com)
> wrote:
> > > >
> > > > Continuing a discussion that started in a discuss thread about
> exposing
> > > > Pcap query capabilities in the back end. How should we expose this
> > > feature
> > > > to users? Should it be integrated into the Alerts UI or be separate
> > > > standalone UI?
> > > >
> > > > To summarize the general points made in the other thread:
> > > >
> > > > - Adding this capability to the Alerts UI will make it more of a
> > > > composite app. Is that really what we want since we have separate UIs
> > for
> > > > Alerts and management?
> > > > - Would it be better to bring it in on it's own so it can be released
> > > > with qualifiers and tested with the right expectations without
> > affecting
> > > > the Alerts UI?
> > > > - There are some use cases that begin with an infosec analyst doing a
> > > > search on alerts
> > > > followed by them going to query pcap data corresponding to the
> > > > threats they're investigating. Would having these features in the
> same
> > > > UI streamline this process?
> > > >
> > > > There was also mention of some features we should consider:
> > > >
> > > > - Pcap queries should be made asynchronous via the UI
> > > > - Take care that a user doesn't hit refresh or POST multiple times
> and
> > > kick
> > > > off 50 mapreduce jobs
> > > > - Options for managing the YARN queue that is used
> > > > - Provide a "cancel" option that kills the MR job, or tell the user
> to
> > > > go to the CLI to kill their job
> > > > - Managing data if multiple users run queries
> > > > - Strategy for cleaning up jobs and implementing a TTL (I think this
> > one
> > > > will be tricky and definitely needs discussion)
> > > > - Date range or other query limits
> > > >
> > > > A couple other features I would add:
> > > >
> > > > - Ability to paginate through results
> > > > - Ability to download results through the UI
> > > > - Realtime status of a running job in the UI
> > > >
> > > > Let me know if I missed any points or did not correctly capture them
> > > > here. What
> > > > other points do we need to consider? What other features should be
> > > > required? Nice to have?
> > > >
> > >
> > --
> >
> > Jon
> >
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message