metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: [DISCUSS] Metron Management UI
Date Mon, 20 Feb 2017 20:24:50 GMT
I think its important to keep moving toward a Metron where I can introduce
new telemetry, add new enrichments, tweak the triage process, iteratively,
on-the-fly, and safely in a single, production environment.

If a user wants to have a traditional test-to-production change management
process, they can, but it is not necessary.  I think the base that we have
allows us to do this.

In that regard, I do like the idea, Otto.


On Sat, Feb 18, 2017 at 8:51 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> I agree with this.  We could really do something cool here if we have
> multiple training topologies, like steps
> that fed the ui.
>
> Step one would be kafka to parser with output to some tmp area.  The ui
> would be used to train the parser/stellar configs
> Step two would add on to enrichments to temp.  The ui would be used to
> train the enrichment configurations
> etc
> etc.
>
> Until it was all set and you were happy then the source was ‘activated’
> with the real topologies.
>
>
>
> On February 18, 2017 at 07:38:17, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> I don't expect people to push data through Metron until they have an
> expectation that it will be properly parsed and handled, otherwise you'll
> flood the cluster with failures and noise.
>
> Here's what's in my head. A user grabs an upstream log sample (from their
> environment), uses the UI to figure out how to parse it, adjusts Metron
> accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
> triage, optionally index templates) then heads back out of Metron and gives
> it some juice by either just turning on the Kafka producer or doing a spot
> check first (examine in UI things are working properly) then turning on the
> producer (nifi, etc.).
>
> Jon
>
> On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> > At the moment the UI relies on Kafka topics already existing, and having
> > data flowing into them to pull samples for the parser config panel.
> > However, you can paste alternative samples if kafka isn’t already setup.
> As
> > such, the assumption is that the topics would already have data flowing
> > from something like nifi, or some other producer before you got round to
> > configuring a sensor in this UI. Would that work, or would you expect to
> be
> > creating the parser config before the data was flowing into kafka? In the
> > later case, we could probably rely on copy-pasted samples to design the
> > parser, and then auto-create topics if not already existing on parser
> save.
> >
> > Simon
> >
> >
> > > On 17 Feb 2017, at 23:42, Zeolla@GMail.com <zeolla@gmail.com> wrote:
> > >
> > > Well, I think that a UI that does what you show is a huge step forward
> > and
> > > I definitely wouldn't want to stop it from getting into the code
> because
> > of
> > > any of the additional features that we are talking about here.
> > >
> > > That said, I really would like to see the Kafka topic creation as a
> part
> > of
> > > the UI because without it, data can't move. Templates would be a second
> > > priority that I see as important, but depending on how Otto's thread
> goes
> > > how that could would may change and I wouldn't push to have that in a
> MVP
> > > UI.
> > >
> > > Aside from that, I still agree with my previous comments regarding nice
> > to
> > > have features, but definitely not needed for a first run.
> > >
> > > Jon
> > >
> > > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian <hlivian@hortonworks.com
> >
> > > wrote:
> > >
> > >> Hi Dima and Jon,
> > >>
> > >> Do you think the UI:
> > >>
> > >> http://imgur.com/a/QAQyu?
> > >>
> > >>
> > >> ...Is a reasonable place to start as it is?… or do we need to have the
> > >> Elastic Search Templates right away?
> > >> Are the templates a MUST HAVE in order to add this UI to Metron?
> > >>
> > >> As Simon said, I think we can file a JIRA on the templates and start
> > >> working on what that looks like.
> > >>
> > >> What are your thoughts?
> > >>
> > >>
> > >>
> > >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> > simon@simonellistonball.com>
> > >> wrote:
> > >>
> > >>> Seems like the UI discussion has fallen off the inboxes.
> > >>>
> > >>> We should definitely open up a separate Jira and discussion on the
> best
> > >> way to process elastic search template management. This feels like a
> > non-UI
> > >> question to me, since it would apply equally to any other parser
> config
> > >> method. I’ll start a separate discuss for that.
> > >>>
> > >>> In the meantime it would be great to hear more from the community on
> > the
> > >> UI at http://imgur.com/a/QAQyu?
> > >>>
> > >>> Is there anything you see missing around sensor config here?
> > >>> Does it look useful?
> > >>> Does the flow match the way you would currently think about setting
> up
> > >> sensors?
> > >>>
> > >>> It would be great to get some thoughts.
> > >>>
> > >>> Simon
> > >>>
> > >>>
> > >>>> On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> > >> Simon@simonellistonball.com> wrote:
> > >>>>
> > >>>> The MPack creates elastic templates for the standard parsers
> included
> > >> in the pack.
> > >>>>
> > >>>> However, when creating new sensors we should be able to infer
> Elastic
> > >> templates from the schema. There are certain fields that we need to
> mark
> > >> types for to generate those templates (e.g. date / time fields). This
> > goes
> > >> back to the debate we had a while ago about, for example, typing in
> the
> > >> grok parser, but applies for others too. I suspect that since we
> should
> > be
> > >> able to infer all the types needed for templates, this is more an
> > excellent
> > >> feature enhancement for the REST api layer, unless anyone can think of
> a
> > >> reason we would need explicit type definition in the
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org>
wrote:
> > >>>>>
> > >>>>> The Index templates are currently handled in the Ambari MPack
> (unless
> > >> you
> > >>>>> are suggesting some additional functionality beyond what the
Ambari
> > >> MPack
> > >>>>> gives us today.)
> > >>>>>
> > >>>>> I think there is always going to be a fuzzy, gray line between
what
> > the
> > >>>>> Ambari MPack should do and what the Management UI should do.
I
> think
> > >> it
> > >>>>> works fine in Ambari today, but I could be convinced otherwise.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <
> > Dima.Kovalyov@sstech.us
> > >>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Thank you Houshang,
> > >>>>>>
> > >>>>>> That looks absolutely great, loved it!
> > >>>>>> Will it make sense to also add ElasticSearch index templates
> > >>>>>> querying/creation here?
> > >>>>>>
> > >>>>>> - Dima
> > >>>>>>
> > >>>>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
> > >>>>>>> Hello Metron Community,
> > >>>>>>>
> > >>>>>>> We have constructed a Management Module UI, built on
top of
> > >> METRON-503
> > >>>>>> (REST API) (currently under review). This Module gives
users the
> > >> ability to
> > >>>>>> setup and administer much of the product through the UI.
> > >>>>>>>
> > >>>>>>> Here are some screens to show you what we are thinking.
Please
> > take a
> > >>>>>> look:
> > >>>>>>> http://imgur.com/a/QAQyu?
> > >>>>>>>
> > >>>>>>> Does this look like a reasonable place to start?
> > >>>>>>> Is there anything that is an absolute MUST have or
MUST NOT have?
> > >>>>>>>
> > >>>>>>> Houshang Livian
> > >>>>>>> Senior User Experience Designer
> > >>>>>>> Hortonworks
> > >>>>>>>
> > >>>>>>> www.hortonworks.com<http://www.hortonworks.com/>
> > >>>>>>> ________________________________
> > >>>>>>>
> > >>>>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
> > >>>>>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
> > >>>>>>> ________________________________
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Nick Allen <nick@nickallen.org>
> > >>>>
> > >>>
> > >>
> > > --
> > >
> > > Jon
> > >
> > > Sent from my mobile device
> >
> > --
>
> Jon
>
> Sent from my mobile device
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message