metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rita McKissick <rmckiss...@hortonworks.com>
Subject Re: [DISCUSS] Meta alert Elasticsearch new template requirement ramifications
Date Fri, 29 Sep 2017 23:37:43 GMT
Hi Justin,

I am adding the nested “alert" field requirement to the product documentation, and I had
a question:

* If the user opts to use the default configuration, will they still need to add the nested
“alert” field?

When I’m finished with the documentation, I’ll send you a link to make sure what I’ve
documented is correct.

Thanks,

Rita


Rita McKissick ! Sr. Technical Writer
rmckissick@hortonworks.com
(mobile) 831-234-3676








On 9/29/17, 6:59 AM, "Justin Leet" <justinjleet@gmail.com> wrote:

>As part of building a backend for meta-alerts (
>https://github.com/apache/metron/pull/734), there's an additional
>requirement for the Elasticsearch templates for new sensors.  Although
>seemingly minor, this should be called out explicitly because of the wider
>implications of leaving it out of ANY sensor.  Specifically, this can
>result in the UI and other queries not returning results, because
>Elasticsearch throws an error.
>
>A nested "alert" field must be added in the form:
>
>"alert": {
>   "type": "nested"
>},
>
>This results from Elasticsearch 2.x requiring the type of searches that
>meta alerts wrap to have the fields exist in all indices or the query fails.
>
>In Elasticsearch 5.x, there is a per query parameter that can be set to
>avoid this: see
>https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
>.
>
>The obvious short-term thing that needs to happen with this is improved
>documentation.  A ticket for documenting this (with some more specific
>details and what the error looks like is at
>https://issues.apache.org/jira/browse/METRON-1220).  Where should that
>documentation live?  It seems like our documentation in general around this
>type of stuff is a little lacking.  Right now, I'm putting a prelim version
>into the metron-indexing README, but either now or as a followon we should
>have a more robust version that lays requirements for things like
>templating.
>
>There are a couple options I see to address this more robustly.
>1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
>beginning basis for this change exists in
>https://github.com/apache/metron/pull/619.  More works needs to happen
>there to finalize it
>2) Add in ZK hooks for when a new sensor is added.  The DAOs could receive
>word that a new sensor has been added in ZK and then build and submit the
>modified template itself.  This (or a variant) is probably something that
>should happen anyway, in order to be more consistent with the other pieces
>that monitor and act on ZK updates.
>3) There may be some mitigating that can be done here, e.g. if a query
>fails with the relevant error, rerun a different variant that may not hit
>the meta alerts, but doesn't fail as extravagantly.
>
>Is there a preference on either where the new documentation lives?  And is
>there a preference on how we address this going forward?
>
>Justin
Mime
View raw message