metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <jsir...@apache.org>
Subject Re: [DISCUSS] Pcap query branch completion
Date Thu, 16 Aug 2018 13:25:29 GMT
+1 on the merge as well

16.08.2018, 05:46, "Casey Stella" <cestella@gmail.com>:
> I'm +1 on the merge. This is great work and congrats to those who
> contributed to it!
>
> On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler <ottobackwards@gmail.com> wrote:
>
>>  Looks good, thanks!
>>
>>  On August 15, 2018 at 19:38:12, Ryan Merriman (merrimanr@gmail.com) wrote:
>>
>>  Otto, I believe the items you requested are in the feature branch now. Is
>>  there anything outstanding that we missed? The Jiras for the Pcap feature
>>  branch should be up to date:
>>  https://issues.apache.org/jira/browse/METRON-1554
>>
>>  On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman <merrimanr@gmail.com>
>>  wrote:
>>
>>  > - Date range limits on queries
>>  >
>>  > I will add a warning in the Job cleanup PR. That seems like an
>>  > appropriate place for it (ie. make sure you don't cause health issues in
>>  > your cluster).
>>  >
>>  > - UI should manage a queue/history of jobs
>>  >
>>  > I can add some documentation around killing jobs manually with the YARN
>>  > CLI. However if they haven't set up a YARN queue, I'm not sure how you
>>  > would view only Pcap jobs. I'm also not sure how you would get the
>>  > application id for the job to kill because it's not displayed anywhere in
>>  > the UI. However, I believe we are wired for a job name but REST doesn't
>>  > set this. Maybe we could get a proper job name associated with pcap
>>  > queries and then this would be possible to document?
>>  >
>>  > - Documentation/blueprint for YARN configuration
>>  >
>>  > You make a good point. A YARN tuning guide for Metron does sound useful.
>>  > I will add a follow on Jira.
>>  >
>>  > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler <ottobackwards@gmail.com>
>>  > wrote:
>>  >
>>  >>
>>  >> - Date range limits on queries
>>  >>
>>  >> I took the point the wrong way apparently, sorry, I withdraw. I thought
>>  >> you meant allow specifying a limit on the query, not the system imposing
>>  a
>>  >> limit.
>>  >> This should be documented with a warning or something
>>  >>
>>  >> - UI should manage a queue/history of jobs
>>  >>
>>  >> I was thinking that if there where multiple users/jobs, there should
>>  >> be some thought or documentation + script on how to manage them.
>>  >> “To see all the jobs still running on your cluster, across users and
ui
>>  >> instances do XXXXX”
>>  >> “If there is an issue with the jobs you can’t resolve in the UI for
that
>>  >> user, or you are an admin and want to do something then XXXXX"
>>  >>
>>  >> - Documentation/blueprint for YARN configuration
>>  >>
>>  >> I agree with what you are saying. Although, we offer guidance on storm
>>  >> tuning, and that is conceptually the same isn’t it? That is why it comes
>>  >> to mind.
>>  >> Maybe this can be a follow on, in the tuning guide?
>>  >>
>>  >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrimanr@gmail.com)
>>  >> wrote:
>>  >>
>>  >> - Date range limits on queries
>>  >>
>>  >> Can you describe what you think is needed here? Each Metron user could
>>  >> have different volumes of pcap data spread out over different time
>>  >> periods. Are you saying we should limit the data range to something
>>  either
>>  >>
>>  >> constant or configurable? Are we sure all users would want this? Am I
>>  >> misinterpreting this requirement?
>>  >>
>>  >> - UI should manage a queue/history of jobs
>>  >>
>>  >> What should we document here? Reading that bullet point again, it's sort
>>  >> of vague and not very description. What I am referring to is a design
>>  that
>>  >>
>>  >> provides users a way to view and manage jobs in the UI. Currently jobs
>>  can
>>  >>
>>  >> only be run 1 at a time and progress is shown with a status bar, so it's
>>  >> somewhat interactive.
>>  >>
>>  >> - Documentation/blueprint for YARN configuration
>>  >>
>>  >>
>>  >

------------------- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org


Mime
View raw message