metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Metron Management UI
Date Sat, 18 Feb 2017 12:37:56 GMT
I don't expect people to push data through Metron until they have an
expectation that it will be properly parsed and handled, otherwise you'll
flood the cluster with failures and noise.

Here's what's in my head.  A user grabs an upstream log sample (from their
environment), uses the UI to figure out how to parse it, adjusts Metron
accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
triage, optionally index templates) then heads back out of Metron and gives
it some juice by either just turning on the Kafka producer or doing a spot
check first (examine in UI things are working properly) then turning on the
producer (nifi, etc.).

Jon

On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> At the moment the UI relies on Kafka topics already existing, and having
> data flowing into them to pull samples for the parser config panel.
> However, you can paste alternative samples if kafka isn’t already setup. As
> such, the assumption is that the topics would already have data flowing
> from something like nifi, or some other producer before you got round to
> configuring a sensor in this UI. Would that work, or would you expect to be
> creating the parser config before the data was flowing into kafka? In the
> later case, we could probably rely on copy-pasted samples to design the
> parser, and then auto-create topics if not already existing on parser save.
>
> Simon
>
>
> > On 17 Feb 2017, at 23:42, Zeolla@GMail.com <zeolla@gmail.com> wrote:
> >
> > Well, I think that a UI that does what you show is a huge step forward
> and
> > I definitely wouldn't want to stop it from getting into the code because
> of
> > any of the additional features that we are talking about here.
> >
> > That said, I really would like to see the Kafka topic creation as a part
> of
> > the UI because without it, data can't move.  Templates would be a second
> > priority that I see as important, but depending on how Otto's thread goes
> > how that could would may change and I wouldn't push to have that in a MVP
> > UI.
> >
> > Aside from that, I still agree with my previous comments regarding nice
> to
> > have features, but definitely not needed for a first run.
> >
> > Jon
> >
> > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian <hlivian@hortonworks.com>
> > wrote:
> >
> >> Hi Dima and Jon,
> >>
> >> Do you think the UI:
> >>
> >> http://imgur.com/a/QAQyu?
> >>
> >>
> >> ...Is a reasonable place to start as it is?… or do we need to have the
> >> Elastic Search Templates right away?
> >> Are the templates a MUST HAVE in order to add this UI to Metron?
> >>
> >> As Simon said, I think we can file a JIRA on the templates and start
> >> working on what that looks like.
> >>
> >> What are your thoughts?
> >>
> >>
> >>
> >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> simon@simonellistonball.com>
> >> wrote:
> >>
> >>> Seems like the UI discussion has fallen off the inboxes.
> >>>
> >>> We should definitely open up a separate Jira and discussion on the best
> >> way to process elastic search template management. This feels like a
> non-UI
> >> question to me, since it would apply equally to any other parser config
> >> method. I’ll start a separate discuss for that.
> >>>
> >>> In the meantime it would be great to hear more from the community on
> the
> >> UI at http://imgur.com/a/QAQyu?
> >>>
> >>> Is there anything you see missing around sensor config here?
> >>> Does it look useful?
> >>> Does the flow match the way you would currently think about setting up
> >> sensors?
> >>>
> >>> It would be great to get some thoughts.
> >>>
> >>> Simon
> >>>
> >>>
> >>>> On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> >> Simon@simonellistonball.com> wrote:
> >>>>
> >>>> The MPack creates elastic templates for the standard parsers included
> >> in the pack.
> >>>>
> >>>> However, when creating new sensors we should be able to infer Elastic
> >> templates from the schema. There are certain fields that we need to mark
> >> types for to generate those templates (e.g. date / time fields). This
> goes
> >> back to the debate we had a while ago about, for example, typing in the
> >> grok parser, but applies for others too. I suspect that since we should
> be
> >> able to infer all the types needed for templates, this is more an
> excellent
> >> feature enhancement for the REST api layer, unless anyone can think of a
> >> reason we would need explicit type definition in the
> >>>>
> >>>>
> >>>>
> >>>>> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org> wrote:
> >>>>>
> >>>>> The Index templates are currently handled in the Ambari MPack (unless
> >> you
> >>>>> are suggesting some additional functionality beyond what the Ambari
> >> MPack
> >>>>> gives us today.)
> >>>>>
> >>>>> I think there is always going to be a fuzzy, gray line between what
> the
> >>>>> Ambari MPack should do and what the Management UI should do.  I
think
> >> it
> >>>>> works fine in Ambari today, but I could be convinced otherwise.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <
> Dima.Kovalyov@sstech.us
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> Thank you Houshang,
> >>>>>>
> >>>>>> That looks absolutely great, loved it!
> >>>>>> Will it make sense to also add ElasticSearch index  templates
> >>>>>> querying/creation here?
> >>>>>>
> >>>>>> - Dima
> >>>>>>
> >>>>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
> >>>>>>> Hello Metron Community,
> >>>>>>>
> >>>>>>> We have constructed a Management Module UI, built on top
of
> >> METRON-503
> >>>>>> (REST API) (currently under review). This Module gives users
the
> >> ability to
> >>>>>> setup and administer much of the product through the UI.
> >>>>>>>
> >>>>>>> Here are some screens to show you what we are thinking.
Please
> take a
> >>>>>> look:
> >>>>>>> http://imgur.com/a/QAQyu?
> >>>>>>>
> >>>>>>> Does this look like a reasonable place to start?
> >>>>>>> Is there anything that is an absolute MUST have or MUST
NOT have?
> >>>>>>>
> >>>>>>> Houshang Livian
> >>>>>>> Senior User Experience Designer
> >>>>>>> Hortonworks
> >>>>>>>
> >>>>>>> www.hortonworks.com<http://www.hortonworks.com/>
> >>>>>>> ________________________________
> >>>>>>>
> >>>>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
> >>>>>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
> >>>>>>> ________________________________
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Nick Allen <nick@nickallen.org>
> >>>>
> >>>
> >>
> > --
> >
> > Jon
> >
> > Sent from my mobile device
>
> --

Jon

Sent from my mobile device

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message