metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS] Pcap query branch completion
Date Mon, 13 Aug 2018 21:36:36 GMT
Thanks for the feedback Otto.  I have created a sub task for documenting
the Job cleanup documentation:
https://issues.apache.org/jira/browse/METRON-1737.  I completely agree with
you there, this needs to be documented.  For the others you marked "Follow
on" I will create follow on tasks in Jira.

I have a few questions about a couple others you commented on

- Date range limits on queries

Can you describe what you think is needed here?  Each Metron user could
have different volumes of pcap data spread out over different time
periods.  Are you saying we should limit the data range to something either
constant or configurable?  Are we sure all users would want this?  Am I
misinterpreting this requirement?

- UI should manage a queue/history of jobs

What should we document here?  Reading that bullet point again, it's sort
of vague and not very description.  What I am referring to is a design that
provides users a way to view and manage jobs in the UI.  Currently jobs can
only be run 1 at a time and progress is shown with a status bar, so it's
somewhat interactive.

- Documentation/blueprint for YARN configuration

We are setup for YARN scheduling in that we offer a configuration setting
to submit a Pcap query to a specified YARN queue (this part is
documented).  Any YARN setup or tuning would be out of scope since
everyone's YARN settings will be different and potentially expand beyond
the Metron use case.  I think a Hadoop admin is likely to have this
knowledge and to have already set up YARN queues.  Do you disagree?



On Mon, Aug 13, 2018 at 8:21 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> - Job cleanup/TTL
>
> Documented at least, or a helper script to help yourself if you are in a
> situation
>
>
> - Expose the Query filter (vs Fixed) in the UI
>
> Follow on
>
>
> - Date range limits on queries
>
> I don’t see how this won’t be immediately required. I would do this for
> minimum viable.
>
>
> - Pcap query as a separate UI
>
> Follow on
>
>
> - UI should manage a queue/history of jobs
>
> Follow on, but maybe we need documentation
>
>
> - BPF filtering
>
> This is going to be a PITA, follow on
>
>
> - Sharing PCA jobs with other users
>
> Follow on
>
>
> - Provide a way in the UI to populate a pcap query from an alert/metaalert
>
> Follow on
>
>
> - Documentation/blueprint for YARN configuration
>
> Should have
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message