metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [DISCUSS] Metron Management UI
Date Sat, 18 Feb 2017 13:51:20 GMT
I agree with this.  We could really do something cool here if we have
multiple training topologies, like steps
that fed the ui.

Step one would be kafka to parser with output to some tmp area.  The ui
would be used to train the parser/stellar configs
Step two would add on to enrichments to temp.  The ui would be used to
train the enrichment configurations
etc
etc.

Until it was all set and you were happy then the source was ‘activated’
with the real topologies.



On February 18, 2017 at 07:38:17, Zeolla@GMail.com (zeolla@gmail.com) wrote:

I don't expect people to push data through Metron until they have an
expectation that it will be properly parsed and handled, otherwise you'll
flood the cluster with failures and noise.

Here's what's in my head. A user grabs an upstream log sample (from their
environment), uses the UI to figure out how to parse it, adjusts Metron
accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
triage, optionally index templates) then heads back out of Metron and gives
it some juice by either just turning on the Kafka producer or doing a spot
check first (examine in UI things are working properly) then turning on the
producer (nifi, etc.).

Jon

On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> At the moment the UI relies on Kafka topics already existing, and having
> data flowing into them to pull samples for the parser config panel.
> However, you can paste alternative samples if kafka isn’t already setup.
As
> such, the assumption is that the topics would already have data flowing
> from something like nifi, or some other producer before you got round to
> configuring a sensor in this UI. Would that work, or would you expect to
be
> creating the parser config before the data was flowing into kafka? In the
> later case, we could probably rely on copy-pasted samples to design the
> parser, and then auto-create topics if not already existing on parser
save.
>
> Simon
>
>
> > On 17 Feb 2017, at 23:42, Zeolla@GMail.com <zeolla@gmail.com> wrote:
> >
> > Well, I think that a UI that does what you show is a huge step forward
> and
> > I definitely wouldn't want to stop it from getting into the code
because
> of
> > any of the additional features that we are talking about here.
> >
> > That said, I really would like to see the Kafka topic creation as a
part
> of
> > the UI because without it, data can't move. Templates would be a second
> > priority that I see as important, but depending on how Otto's thread
goes
> > how that could would may change and I wouldn't push to have that in a
MVP
> > UI.
> >
> > Aside from that, I still agree with my previous comments regarding nice
> to
> > have features, but definitely not needed for a first run.
> >
> > Jon
> >
> > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian <hlivian@hortonworks.com>
> > wrote:
> >
> >> Hi Dima and Jon,
> >>
> >> Do you think the UI:
> >>
> >> http://imgur.com/a/QAQyu?
> >>
> >>
> >> ...Is a reasonable place to start as it is?… or do we need to have the
> >> Elastic Search Templates right away?
> >> Are the templates a MUST HAVE in order to add this UI to Metron?
> >>
> >> As Simon said, I think we can file a JIRA on the templates and start
> >> working on what that looks like.
> >>
> >> What are your thoughts?
> >>
> >>
> >>
> >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> simon@simonellistonball.com>
> >> wrote:
> >>
> >>> Seems like the UI discussion has fallen off the inboxes.
> >>>
> >>> We should definitely open up a separate Jira and discussion on the
best
> >> way to process elastic search template management. This feels like a
> non-UI
> >> question to me, since it would apply equally to any other parser
config
> >> method. I’ll start a separate discuss for that.
> >>>
> >>> In the meantime it would be great to hear more from the community on
> the
> >> UI at http://imgur.com/a/QAQyu?
> >>>
> >>> Is there anything you see missing around sensor config here?
> >>> Does it look useful?
> >>> Does the flow match the way you would currently think about setting
up
> >> sensors?
> >>>
> >>> It would be great to get some thoughts.
> >>>
> >>> Simon
> >>>
> >>>
> >>>> On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> >> Simon@simonellistonball.com> wrote:
> >>>>
> >>>> The MPack creates elastic templates for the standard parsers
included
> >> in the pack.
> >>>>
> >>>> However, when creating new sensors we should be able to infer
Elastic
> >> templates from the schema. There are certain fields that we need to
mark
> >> types for to generate those templates (e.g. date / time fields). This
> goes
> >> back to the debate we had a while ago about, for example, typing in
the
> >> grok parser, but applies for others too. I suspect that since we
should
> be
> >> able to infer all the types needed for templates, this is more an
> excellent
> >> feature enhancement for the REST api layer, unless anyone can think of
a
> >> reason we would need explicit type definition in the
> >>>>
> >>>>
> >>>>
> >>>>> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org> wrote:
> >>>>>
> >>>>> The Index templates are currently handled in the Ambari MPack
(unless
> >> you
> >>>>> are suggesting some additional functionality beyond what the Ambari
> >> MPack
> >>>>> gives us today.)
> >>>>>
> >>>>> I think there is always going to be a fuzzy, gray line between what
> the
> >>>>> Ambari MPack should do and what the Management UI should do. I
think
> >> it
> >>>>> works fine in Ambari today, but I could be convinced otherwise.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <
> Dima.Kovalyov@sstech.us
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> Thank you Houshang,
> >>>>>>
> >>>>>> That looks absolutely great, loved it!
> >>>>>> Will it make sense to also add ElasticSearch index templates
> >>>>>> querying/creation here?
> >>>>>>
> >>>>>> - Dima
> >>>>>>
> >>>>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
> >>>>>>> Hello Metron Community,
> >>>>>>>
> >>>>>>> We have constructed a Management Module UI, built on top
of
> >> METRON-503
> >>>>>> (REST API) (currently under review). This Module gives users
the
> >> ability to
> >>>>>> setup and administer much of the product through the UI.
> >>>>>>>
> >>>>>>> Here are some screens to show you what we are thinking.
Please
> take a
> >>>>>> look:
> >>>>>>> http://imgur.com/a/QAQyu?
> >>>>>>>
> >>>>>>> Does this look like a reasonable place to start?
> >>>>>>> Is there anything that is an absolute MUST have or MUST
NOT have?
> >>>>>>>
> >>>>>>> Houshang Livian
> >>>>>>> Senior User Experience Designer
> >>>>>>> Hortonworks
> >>>>>>>
> >>>>>>> www.hortonworks.com<http://www.hortonworks.com/>
> >>>>>>> ________________________________
> >>>>>>>
> >>>>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
> >>>>>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
> >>>>>>> ________________________________
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Nick Allen <nick@nickallen.org>
> >>>>
> >>>
> >>
> > --
> >
> > Jon
> >
> > Sent from my mobile device
>
> --

Jon

Sent from my mobile device

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message