metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Kovalyov <Dima.Koval...@sstech.us>
Subject Re: [DISCUSS] Metron Management UI
Date Sun, 05 Feb 2017 07:22:04 GMT
Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep. So
if we allow Analyst to create/tweak parsers in Metron UI we should also
move ES templates modification feature out of Ambari UI to Metron UI. If
that is so, should Metron service that controls default topologies (like
Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

- Dima

On 02/01/2017 11:16 PM, Zeolla@GMail.com wrote:
> Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
> playing with this next week if I can.  Currently my productivity machine is
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
>
> So if we ignore the current API restrictions, here are my MUST haves:
>
>  - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
>  - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some very
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
>
> And here are my NICE to haves (eventually):
>  - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>
>  - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
>  - It would be nice to be able to modify the jobs that purge data from HDFS
> and ES via the UI.  Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs?  For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
>  - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured).  This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment.  IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
>
>
> Not to get off topic, but I think the ES side of the house could generally
> use some love.  For instance, I don't see a reason why every string field
> shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
> probably do this as a part of METRON-635, but we should make sure to carry
> that setting forward appropriately (i.e. if we infer ES templates from a
> schema).  I could agree with ES templates being inferred when creating new
> parsers/sensors as a nice to have, building off of a must have of being
> able to make/tweak ES templates in the UI/API.
>
>
> Ok, that's a lot of text, I'll end it there.  =D
>
>
> Jon
>
> On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> The MPack creates elastic templates for the standard parsers included in
> the pack.
>
> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an excellent
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the schema config element
> of the UI. Thoughts?
>
> Simon
>
>
>> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org> wrote:
>>
>> The Index templates are currently handled in the Ambari MPack (unless you
>> are suggesting some additional functionality beyond what the Ambari MPack
>> gives us today.)
>>
>> I think there is always going to be a fuzzy, gray line between what the
>> Ambari MPack should do and what the Management UI should do.  I think it
>> works fine in Ambari today, but I could be convinced otherwise.
>>
>>
>>
>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <Dima.Kovalyov@sstech.us>
>> wrote:
>>
>>> Thank you Houshang,
>>>
>>> That looks absolutely great, loved it!
>>> Will it make sense to also add ElasticSearch index  templates
>>> querying/creation here?
>>>
>>> - Dima
>>>
>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>>> Hello Metron Community,
>>>>
>>>> We have constructed a Management Module UI, built on top of METRON-503
>>> (REST API) (currently under review). This Module gives users the ability
> to
>>> setup and administer much of the product through the UI.
>>>> Here are some screens to show you what we are thinking. Please take a
>>> look:
>>>> http://imgur.com/a/QAQyu?
>>>>
>>>> Does this look like a reasonable place to start?
>>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>>>
>>>> Houshang Livian
>>>> Senior User Experience Designer
>>>> Hortonworks
>>>>
>>>> www.hortonworks.com<http://www.hortonworks.com/>
>>>> ________________________________
>>>>
>>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
>>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
>>>> ________________________________
>>>>
>>>>
>>>
>>
>> --
>> Nick Allen <nick@nickallen.org>


Mime
View raw message