metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <jsir...@apache.org>
Subject Re: [DISCUSS] Metron Management UI
Date Sat, 18 Feb 2017 17:13:36 GMT
Another thing we can use for this is Stellar shell.  We can pull files from kafka (initially
one at a time and later in bulk), apply parser, and see a result (or results).  If there are
any failures you repeat until you get everything parsed.  We can then do the same with the
enrichmens (that capability actually already exists).  

Thanks,
James

18.02.2017, 06:51, "Otto Fowler" <ottobackwards@gmail.com>:
> I agree with this. We could really do something cool here if we have
> multiple training topologies, like steps
> that fed the ui.
>
> Step one would be kafka to parser with output to some tmp area. The ui
> would be used to train the parser/stellar configs
> Step two would add on to enrichments to temp. The ui would be used to
> train the enrichment configurations
> etc
> etc.
>
> Until it was all set and you were happy then the source was ‘activated’
> with the real topologies.
>
> On February 18, 2017 at 07:38:17, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>
> I don't expect people to push data through Metron until they have an
> expectation that it will be properly parsed and handled, otherwise you'll
> flood the cluster with failures and noise.
>
> Here's what's in my head. A user grabs an upstream log sample (from their
> environment), uses the UI to figure out how to parse it, adjusts Metron
> accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
> triage, optionally index templates) then heads back out of Metron and gives
> it some juice by either just turning on the Kafka producer or doing a spot
> check first (examine in UI things are working properly) then turning on the
> producer (nifi, etc.).
>
> Jon
>
> On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
>>  At the moment the UI relies on Kafka topics already existing, and having
>>  data flowing into them to pull samples for the parser config panel.
>>  However, you can paste alternative samples if kafka isn’t already setup.
>
> As
>>  such, the assumption is that the topics would already have data flowing
>>  from something like nifi, or some other producer before you got round to
>>  configuring a sensor in this UI. Would that work, or would you expect to
>
> be
>>  creating the parser config before the data was flowing into kafka? In the
>>  later case, we could probably rely on copy-pasted samples to design the
>>  parser, and then auto-create topics if not already existing on parser
>
> save.
>>  Simon
>>
>>  > On 17 Feb 2017, at 23:42, Zeolla@GMail.com <zeolla@gmail.com> wrote:
>>  >
>>  > Well, I think that a UI that does what you show is a huge step forward
>>  and
>>  > I definitely wouldn't want to stop it from getting into the code
>
> because
>>  of
>>  > any of the additional features that we are talking about here.
>>  >
>>  > That said, I really would like to see the Kafka topic creation as a
>
> part
>>  of
>>  > the UI because without it, data can't move. Templates would be a second
>>  > priority that I see as important, but depending on how Otto's thread
>
> goes
>>  > how that could would may change and I wouldn't push to have that in a
>
> MVP
>>  > UI.
>>  >
>>  > Aside from that, I still agree with my previous comments regarding nice
>>  to
>>  > have features, but definitely not needed for a first run.
>>  >
>>  > Jon
>>  >
>>  > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian <hlivian@hortonworks.com>
>>  > wrote:
>>  >
>>  >> Hi Dima and Jon,
>>  >>
>>  >> Do you think the UI:
>>  >>
>>  >> http://imgur.com/a/QAQyu?
>>  >>
>>  >>
>>  >> ...Is a reasonable place to start as it is?… or do we need to have the
>>  >> Elastic Search Templates right away?
>>  >> Are the templates a MUST HAVE in order to add this UI to Metron?
>>  >>
>>  >> As Simon said, I think we can file a JIRA on the templates and start
>>  >> working on what that looks like.
>>  >>
>>  >> What are your thoughts?
>>  >>
>>  >>
>>  >>
>>  >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
>>  simon@simonellistonball.com>
>>  >> wrote:
>>  >>
>>  >>> Seems like the UI discussion has fallen off the inboxes.
>>  >>>
>>  >>> We should definitely open up a separate Jira and discussion on the
>
> best
>>  >> way to process elastic search template management. This feels like a
>>  non-UI
>>  >> question to me, since it would apply equally to any other parser
>
> config
>>  >> method. I’ll start a separate discuss for that.
>>  >>>
>>  >>> In the meantime it would be great to hear more from the community
on
>>  the
>>  >> UI at http://imgur.com/a/QAQyu?
>>  >>>
>>  >>> Is there anything you see missing around sensor config here?
>>  >>> Does it look useful?
>>  >>> Does the flow match the way you would currently think about setting
>
> up
>>  >> sensors?
>>  >>>
>>  >>> It would be great to get some thoughts.
>>  >>>
>>  >>> Simon
>>  >>>
>>  >>>
>>  >>>> On 1 Feb 2017, at 12:07, Simon Elliston Ball <
>>  >> Simon@simonellistonball.com> wrote:
>>  >>>>
>>  >>>> The MPack creates elastic templates for the standard parsers
>
> included
>>  >> in the pack.
>>  >>>>
>>  >>>> However, when creating new sensors we should be able to infer
>
> Elastic
>>  >> templates from the schema. There are certain fields that we need to
>
> mark
>>  >> types for to generate those templates (e.g. date / time fields). This
>>  goes
>>  >> back to the debate we had a while ago about, for example, typing in
>
> the
>>  >> grok parser, but applies for others too. I suspect that since we
>
> should
>>  be
>>  >> able to infer all the types needed for templates, this is more an
>>  excellent
>>  >> feature enhancement for the REST api layer, unless anyone can think of
>
> a
>>  >> reason we would need explicit type definition in the
>>  >>>>
>>  >>>>
>>  >>>>
>>  >>>>> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org>
wrote:
>>  >>>>>
>>  >>>>> The Index templates are currently handled in the Ambari MPack
>
> (unless
>>  >> you
>>  >>>>> are suggesting some additional functionality beyond what the
Ambari
>>  >> MPack
>>  >>>>> gives us today.)
>>  >>>>>
>>  >>>>> I think there is always going to be a fuzzy, gray line between
what
>>  the
>>  >>>>> Ambari MPack should do and what the Management UI should do.
I
>
> think
>>  >> it
>>  >>>>> works fine in Ambari today, but I could be convinced otherwise.
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <
>>  Dima.Kovalyov@sstech.us
>>  >>>
>>  >>>>> wrote:
>>  >>>>>
>>  >>>>>> Thank you Houshang,
>>  >>>>>>
>>  >>>>>> That looks absolutely great, loved it!
>>  >>>>>> Will it make sense to also add ElasticSearch index templates
>>  >>>>>> querying/creation here?
>>  >>>>>>
>>  >>>>>> - Dima
>>  >>>>>>
>>  >>>>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>  >>>>>>> Hello Metron Community,
>>  >>>>>>>
>>  >>>>>>> We have constructed a Management Module UI, built
on top of
>>  >> METRON-503
>>  >>>>>> (REST API) (currently under review). This Module gives
users the
>>  >> ability to
>>  >>>>>> setup and administer much of the product through the UI.
>>  >>>>>>>
>>  >>>>>>> Here are some screens to show you what we are thinking.
Please
>>  take a
>>  >>>>>> look:
>>  >>>>>>> http://imgur.com/a/QAQyu?
>>  >>>>>>>
>>  >>>>>>> Does this look like a reasonable place to start?
>>  >>>>>>> Is there anything that is an absolute MUST have or
MUST NOT have?
>>  >>>>>>>
>>  >>>>>>> Houshang Livian
>>  >>>>>>> Senior User Experience Designer
>>  >>>>>>> Hortonworks
>>  >>>>>>>
>>  >>>>>>> www.hortonworks.com<http://www.hortonworks.com/>
>>  >>>>>>> ________________________________
>>  >>>>>>>
>>  >>>>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
>>  >>>>>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
>>  >>>>>>> ________________________________
>>  >>>>>>>
>>  >>>>>>>
>>  >>>>>>
>>  >>>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>> --
>>  >>>>> Nick Allen <nick@nickallen.org>
>>  >>>>
>>  >>>
>>  >>
>>  > --
>>  >
>>  > Jon
>>  >
>>  > Sent from my mobile device
>>
>>  --
>
> Jon
>
> Sent from my mobile device

------------------- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Mime
View raw message