metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Re: [DISCUSS] Metron Management UI
Date Sun, 05 Feb 2017 11:48:12 GMT
Thanks Dima.  I think that division does make sense.  We are specifically
talking about the Metron management UI in this conversation (not
necessarily the Analyst or Investigator's day to day console) and so how
this works in use will obviously depend on what roles exist in a company
because they won't always perfectly match Metron personnas.  In my scenario
we will have a hadoop admin team that has very limited access to Metron
directly (I.e. Ambari but not Metron) specifically for sensitivity reasons.

Second, when I say must, I mean must exist for a complete cluster
management experience using the UI, but I agree, it doesn't need to be a
part of the initial merge.  I'm trying to keep a user from needing to
switch between using the UI and CLI to bring a new log source into the

Regarding health monitoring, there is an ongoing dev mailing list thread on
changing the current error reporting, which includes the idea of an ES
dashboard for the errors.  Are you suggesting something in addition to
that?  I would be in favor of more health/performance monitoring details,
but I would need to think carefully about what parts of that to happen in
the management UI as opposed to Ambari.

What about a stoplight area (red, yellow, green) in the mgmt UI that links
to Ambari?  Or are you more interested in metrics like cluster
throughput/performance tuning ("health" as in is the cluster is backing up
or not)?  I could see that part being in the mgmt UI, alongside some of the
nice to haves I suggested to modify the related parameters.


On Sun, Feb 5, 2017, 2:27 AM Dima Kovalyov <> wrote:

Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

Houshang, I am not sure if it is MUST, but it is something that makes
sense if you allow user to create new parser. It is for absolutely OK to
just start here.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep (does
it make sense?). So if we allow Analyst to create/tweak parsers in
Metron UI we should also move ES templates modification feature out of
Ambari UI to Metron UI. If that is so, should Metron service (that
controls topologies like Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

Last thing, should metron-parsers*.jar be split into separate parsers
jar in case if among 50 parsers we need to update just one and we want
to do that from Metron UI?

- Dima

On 02/01/2017 11:16 PM, wrote:
> Ok great, thanks for the feedback Ryan.  I'm going to try and get around
> playing with this next week if I can.  Currently my productivity machine
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
> So if we ignore the current API restrictions, here are my MUST haves:
>  - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
>  - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
> And here are my NICE to haves (eventually):
>  - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>  - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
>  - It would be nice to be able to modify the jobs that purge data from
> and ES via the UI.  Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs?  For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
>  - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured).  This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment.  IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
> Not to get off topic, but I think the ES side of the house could generally
> use some love.  For instance, I don't see a reason why every string field
> shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
> probably do this as a part of METRON-635, but we should make sure to carry
> that setting forward appropriately (i.e. if we infer ES templates from a
> schema).  I could agree with ES templates being inferred when creating new
> parsers/sensors as a nice to have, building off of a must have of being
> able to make/tweak ES templates in the UI/API.
> Ok, that's a lot of text, I'll end it there.  =D
> Jon
> On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
>> wrote:
> The MPack creates elastic templates for the standard parsers included in
> the pack.
> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the schema config element
> of the UI. Thoughts?
> Simon
>> On 1 Feb 2017, at 19:54, Nick Allen <> wrote:
>> The Index templates are currently handled in the Ambari MPack (unless you
>> are suggesting some additional functionality beyond what the Ambari MPack
>> gives us today.)
>> I think there is always going to be a fuzzy, gray line between what the
>> Ambari MPack should do and what the Management UI should do.  I think it
>> works fine in Ambari today, but I could be convinced otherwise.
>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <>
>> wrote:
>>> Thank you Houshang,
>>> That looks absolutely great, loved it!
>>> Will it make sense to also add ElasticSearch index  templates
>>> querying/creation here?
>>> - Dima
>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>>> Hello Metron Community,
>>>> We have constructed a Management Module UI, built on top of METRON-503
>>> (REST API) (currently under review). This Module gives users the ability
> to
>>> setup and administer much of the product through the UI.
>>>> Here are some screens to show you what we are thinking. Please take a
>>> look:
>>>> Does this look like a reasonable place to start?
>>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>>> Houshang Livian
>>>> Senior User Experience Designer
>>>> Hortonworks
>>>> ________________________________
>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
>>>> ________________________________
>> --
>> Nick Allen <>



Sent from my mobile device

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message