metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Metron Management UI
Date Wed, 01 Feb 2017 21:16:29 GMT
Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
playing with this next week if I can.  Currently my productivity machine is
"in the shop" getting the battery replaced, so I'm playing man down right
now.

So if we ignore the current API restrictions, here are my MUST haves:

 - We should be able to update search/retrieve (i.e. I agree with Dima).
This means tweaking the ES templates, because when putting together a new
sensor or modifying an existing one, we will likely have a new field or
entirely new template.
 - When creating a new sensor, it should create a new kafka topic (if it
doesn't already - it may, I just haven't played with it yet) with some very
simple/dumb default settings (1 partition, default topic size, 0
replication).

And here are my NICE to haves (eventually):
 - You should have the ability to tune a kafka topic's config from the UI,
such as setting the topic size, # of partitions, and replication number.
This is especially useful if you just created the kafka topic in the UI.
Of course this should have lots of warnings about the issues with
decreasing these numbers in the future, and maybe a "read more" link to
some related documentation in the kafka documentation.

 - Tune and restart the topologies to add parallelism into storm (things
like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
 - It would be nice to be able to modify the jobs that purge data from HDFS
and ES via the UI.  Managing the cluster should include setting your data
retention wants - I think these are still purge cron jobs?  For some
context, I'm thinking about how this could manage METRON-477 (both the
original ticket as well as Nick's comments for additional features), which
could be managed in this UI.
 - The ability to generate a mpack that would install the currently
configured state of the cluster (not including data, of course, but
focusing on setting new, tweaked defaults to be what is currently
configured).  This would be useful for rebuilds, standing up new clusters,
or migrating to a different environment.  IMO, this is a really slick
feature because it allows a nice migration from dev > test > prod.


Not to get off topic, but I think the ES side of the house could generally
use some love.  For instance, I don't see a reason why every string field
shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
probably do this as a part of METRON-635, but we should make sure to carry
that setting forward appropriately (i.e. if we infer ES templates from a
schema).  I could agree with ES templates being inferred when creating new
parsers/sensors as a nice to have, building off of a must have of being
able to make/tweak ES templates in the UI/API.


Ok, that's a lot of text, I'll end it there.  =D


Jon

On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

The MPack creates elastic templates for the standard parsers included in
the pack.

However, when creating new sensors we should be able to infer Elastic
templates from the schema. There are certain fields that we need to mark
types for to generate those templates (e.g. date / time fields). This goes
back to the debate we had a while ago about, for example, typing in the
grok parser, but applies for others too. I suspect that since we should be
able to infer all the types needed for templates, this is more an excellent
feature enhancement for the REST api layer, unless anyone can think of a
reason we would need explicit type definition in the schema config element
of the UI. Thoughts?

Simon


> On 1 Feb 2017, at 19:54, Nick Allen <nick@nickallen.org> wrote:
>
> The Index templates are currently handled in the Ambari MPack (unless you
> are suggesting some additional functionality beyond what the Ambari MPack
> gives us today.)
>
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think it
> works fine in Ambari today, but I could be convinced otherwise.
>
>
>
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov <Dima.Kovalyov@sstech.us>
> wrote:
>
>> Thank you Houshang,
>>
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>>
>> - Dima
>>
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>>
>>> We have constructed a Management Module UI, built on top of METRON-503
>> (REST API) (currently under review). This Module gives users the ability
to
>> setup and administer much of the product through the UI.
>>>
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>>
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>>
>>> Houshang Livian
>>> Senior User Experience Designer
>>> Hortonworks
>>>
>>> www.hortonworks.com<http://www.hortonworks.com/>
>>> ________________________________
>>>
>>> Mobile: (831) 521-4176<tel:(831)%20521-4176>
>>> HLivian@hortonworks.com<mailto:HLivian@hortonworks.com>
>>> ________________________________
>>>
>>>
>>
>>
>
>
> --
> Nick Allen <nick@nickallen.org>

-- 

Jon

Sent from my mobile device

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message