metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Update Metron Elasticsearch index names to metron_
Date Thu, 25 Jan 2018 01:41:27 GMT
One other benefit of this revised approach - we can more effectively use
index template patterns to specify our base set of Metron property types.
Call me crazy, but I think we should be able to do something like:

<metron_default_template>

{
  *"template": "metron_*",*
  "mappings": {
    "metron_doc": {
      "dynamic_templates": [
      {
        "geo_location_point": {
          "match": "enrichments:geo:*:location_point",
          "match_mapping_type": "*",
          "mapping": {
            "type": "geo_point"
          }
        }
      },
      {
        "geo_country": {
          "match": "enrichments:geo:*:country",
          "match_mapping_type": "*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "geo_city": {
          "match": "enrichments:geo:*:city",
          "match_mapping_type": "*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "geo_location_id": {
          "match": "enrichments:geo:*:locID",
          "match_mapping_type": "*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "geo_dma_code": {
          "match": "enrichments:geo:*:dmaCode",
          "match_mapping_type": "*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "geo_postal_code": {
          "match": "enrichments:geo:*:postalCode",
          "match_mapping_type": "*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "geo_latitude": {
          "match": "enrichments:geo:*:latitude",
          "match_mapping_type": "*",
          "mapping": {
            "type": "float"
          }
        }
      },
      {
        "geo_longitude": {
          "match": "enrichments:geo:*:longitude",
          "match_mapping_type": "*",
          "mapping": {
            "type": "float"
          }
        }
      },
      {
        "timestamps": {
          "match": "*:ts",
          "match_mapping_type": "*",
          "mapping": {
            "type": "date",
            "format": "epoch_millis"
          }
        }
      },
      {
        "threat_triage_score": {
          "mapping": {
            "type": "float"
          },
          "match": "threat:triage:*score",
          "match_mapping_type": "*"
        }
      },
      {
        "threat_triage_reason": {
          "mapping": {
            "type": "text",
            "fielddata": "true"
          },
          "match": "threat:triage:rules:*:reason",
          "match_mapping_type": "*"
        }
      },
      {
        "threat_triage_name": {
          "mapping": {
            "type": "text",
            "fielddata": "true"
          },
          "match": "threat:triage:rules:*:name",
          "match_mapping_type": "*"
        }
      }

]}}

That means that for every new sensor we bring on board we can skip
adding that boiler plate mapping config to every new template.



On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> I hear you Ali. I think this type of change would actually ease issues
> with downtime because it offers an easy path to migrating existing indices.
> I'd have to review the specifics in the ES docs again, but I believe you
> could duplicate the old indexes and migrate them to "metron_" in advance of
> the upgrade, and then consume new data to the new index pattern/name after
> the upgrade. That should be pretty seamless, I think. I guess it depends on
> how you're using ES.
>
> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazemian@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I just wanted to say it would be great if we can be careful with these
>> type
>> of changes. From the development point of view, it is just a few lines of
>> code which can provide multiple advantages, but for live large-scale
>> Metron
>> platforms, some of these changes might be really expensive to address with
>> zero-downtime.
>>
>> Cheers,
>> Ali
>>
>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwards@gmail.com>
>> wrote:
>>
>> > +1
>> >
>> >
>> > On January 24, 2018 at 16:28:42, Nick Allen (nick@nickallen.org) wrote:
>> >
>> > +1 to a standard prefix for all Metron indices. I've had the same
>> thought
>> > myself and you laid out the advantages well.
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Jan 24, 2018 at 3:47 PM Zeolla@GMail.com <zeolla@gmail.com>
>> wrote:
>> >
>> > > I agree with having a metron_ prefix for ES indexes, and the timing.
>> > >
>> > > Jon
>> > >
>> > > On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
>> > > michael.miklavcic@gmail.com> wrote:
>> > >
>> > > > With the completion of https://github.com/apache/metron/pull/840
>> > > > (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
>> for
>> > a
>> > > > major release rev of Metron in the upcoming release (currently
>> slotted
>> > to
>> > > > 0.4.3, I believe). Since there are non-backwards compatible changes
>> > > > pertaining to ES indexing, it seems like a good opportunity to
>> revisit
>> > > our
>> > > > index naming standards.
>> > > >
>> > > > I propose we add a simple prefix "metron_" to all Metron indexes.
>> There
>> > > are
>> > > > numerous reasons for doing so
>> > > >
>> > > > - removes the likelihood of index name collisions when we perform
>> > > > operations on index wildcard names, e.g. "enrichment_*, indexing_*,
>> > > > etc.".
>> > > > - ie, this allows us to be more friendly in a multi-tenant ES
>> > > > environment for relatively low engineering cost.
>> > > > - simplifies the Kibana dashboard a bit. We currently needed to
>> > > create a
>> > > > special index pattern in order to accommodate multi-index pattern
>> > > > matching
>> > > > across all metron-specific indexes. Using metron_* would be much
>> > > simpler
>> > > > and less prone to error.
>> > > > - easier for customers to debug and identify Metron-specific indexes
>> > > and
>> > > > associated data
>> > > >
>> > > >
>> > > > The reason for making these changes now is that we already have
>> > breaking
>> > > > changes with ES. Leveraging existing indexed data rather than
>> deleting
>> > > > indexes and starting from scractch already requires a
>> > > re-indexing/migration
>> > > > step, so there is no additional effort on the part of users if they
>> > > choose
>> > > > to attempt a migration. It further makes sense with our current work
>> > > > towards upgrading Solr.
>> > > >
>> > > > We already have a battery of integration and manual tests after the
>> ES
>> > > > upgrade work that can be leveraged to validate the changes.
>> > > >
>> > > > Mike Miklavcic
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Jon
>> > >
>> >
>>
>>
>>
>> --
>> A.Nazemian
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message