metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS] Pcap query branch completion
Date Mon, 20 Aug 2018 13:34:56 GMT
The feature branch has been merged into master.

On Thu, Aug 16, 2018 at 5:53 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> I'm +1, thanks for adding that fix, Ryan. (Note, for purposes of vote, I
> was a contributor in the feature branch).
>
> Mike
>
> On Thu, Aug 16, 2018, 4:17 PM Ryan Merriman <merrimanr@gmail.com> wrote:
>
> > We discovered a bug in our testing and felt it should be fixed before we
> > merge.  There is a PR up for review that already has a +1:
> > https://github.com/apache/metron/pull/1168.  I don't anticipate this
> > changing anyone's vote but wanted to be clear about the state of the
> > branch.  If anyone is concerned with this and would like more discussion
> > before we merge, let me know.
> >
> > On Thu, Aug 16, 2018 at 8:25 AM, James Sirota <jsirota@apache.org>
> wrote:
> >
> > > +1 on the merge as well
> > >
> > > 16.08.2018, 05:46, "Casey Stella" <cestella@gmail.com>:
> > > > I'm +1 on the merge. This is great work and congrats to those who
> > > > contributed to it!
> > > >
> > > > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler <ottobackwards@gmail.com
> >
> > > wrote:
> > > >
> > > >>  Looks good, thanks!
> > > >>
> > > >>  On August 15, 2018 at 19:38:12, Ryan Merriman (merrimanr@gmail.com
> )
> > > wrote:
> > > >>
> > > >>  Otto, I believe the items you requested are in the feature branch
> > now.
> > > Is
> > > >>  there anything outstanding that we missed? The Jiras for the Pcap
> > > feature
> > > >>  branch should be up to date:
> > > >>  https://issues.apache.org/jira/browse/METRON-1554
> > > >>
> > > >>  On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <
> merrimanr@gmail.com>
> > > >>  wrote:
> > > >>
> > > >>  > - Date range limits on queries
> > > >>  >
> > > >>  > I will add a warning in the Job cleanup PR. That seems like
an
> > > >>  > appropriate place for it (ie. make sure you don't cause health
> > > issues in
> > > >>  > your cluster).
> > > >>  >
> > > >>  > - UI should manage a queue/history of jobs
> > > >>  >
> > > >>  > I can add some documentation around killing jobs manually with
> the
> > > YARN
> > > >>  > CLI. However if they haven't set up a YARN queue, I'm not sure
> how
> > > you
> > > >>  > would view only Pcap jobs. I'm also not sure how you would get
> the
> > > >>  > application id for the job to kill because it's not displayed
> > > anywhere in
> > > >>  > the UI. However, I believe we are wired for a job name but REST
> > > doesn't
> > > >>  > set this. Maybe we could get a proper job name associated with
> pcap
> > > >>  > queries and then this would be possible to document?
> > > >>  >
> > > >>  > - Documentation/blueprint for YARN configuration
> > > >>  >
> > > >>  > You make a good point. A YARN tuning guide for Metron does sound
> > > useful.
> > > >>  > I will add a follow on Jira.
> > > >>  >
> > > >>  > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <
> > > ottobackwards@gmail.com>
> > > >>  > wrote:
> > > >>  >
> > > >>  >>
> > > >>  >> - Date range limits on queries
> > > >>  >>
> > > >>  >> I took the point the wrong way apparently, sorry, I withdraw.
I
> > > thought
> > > >>  >> you meant allow specifying a limit on the query, not the
system
> > > imposing
> > > >>  a
> > > >>  >> limit.
> > > >>  >> This should be documented with a warning or something
> > > >>  >>
> > > >>  >> - UI should manage a queue/history of jobs
> > > >>  >>
> > > >>  >> I was thinking that if there where multiple users/jobs,
there
> > should
> > > >>  >> be some thought or documentation + script on how to manage
them.
> > > >>  >> “To see all the jobs still running on your cluster, across
users
> > > and ui
> > > >>  >> instances do XXXXX”
> > > >>  >> “If there is an issue with the jobs you can’t resolve
in the UI
> > for
> > > that
> > > >>  >> user, or you are an admin and want to do something then
XXXXX"
> > > >>  >>
> > > >>  >> - Documentation/blueprint for YARN configuration
> > > >>  >>
> > > >>  >> I agree with what you are saying. Although, we offer guidance
on
> > > storm
> > > >>  >> tuning, and that is conceptually the same isn’t it? That
is why
> it
> > > comes
> > > >>  >> to mind.
> > > >>  >> Maybe this can be a follow on, in the tuning guide?
> > > >>  >>
> > > >>  >> On August 13, 2018 at 17:36:41, Ryan Merriman (
> > merrimanr@gmail.com)
> > > >>  >> wrote:
> > > >>  >>
> > > >>  >> - Date range limits on queries
> > > >>  >>
> > > >>  >> Can you describe what you think is needed here? Each Metron
user
> > > could
> > > >>  >> have different volumes of pcap data spread out over different
> time
> > > >>  >> periods. Are you saying we should limit the data range to
> > something
> > > >>  either
> > > >>  >>
> > > >>  >> constant or configurable? Are we sure all users would want
this?
> > Am
> > > I
> > > >>  >> misinterpreting this requirement?
> > > >>  >>
> > > >>  >> - UI should manage a queue/history of jobs
> > > >>  >>
> > > >>  >> What should we document here? Reading that bullet point
again,
> > it's
> > > sort
> > > >>  >> of vague and not very description. What I am referring to
is a
> > > design
> > > >>  that
> > > >>  >>
> > > >>  >> provides users a way to view and manage jobs in the UI.
> Currently
> > > jobs
> > > >>  can
> > > >>  >>
> > > >>  >> only be run 1 at a time and progress is shown with a status
bar,
> > so
> > > it's
> > > >>  >> somewhat interactive.
> > > >>  >>
> > > >>  >> - Documentation/blueprint for YARN configuration
> > > >>  >>
> > > >>  >>
> > > >>  >
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message