metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject [DISCUSS] Update Metron Elasticsearch index names to metron_
Date Wed, 24 Jan 2018 20:20:07 GMT
With the completion of https://github.com/apache/metron/pull/840
(METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for a
major release rev of Metron in the upcoming release (currently slotted to
0.4.3, I believe). Since there are non-backwards compatible changes
pertaining to ES indexing, it seems like a good opportunity to revisit our
index naming standards.

I propose we add a simple prefix "metron_" to all Metron indexes. There are
numerous reasons for doing so

   - removes the likelihood of index name collisions when we perform
   operations on index wildcard names, e.g. "enrichment_*, indexing_*, etc.".
   - ie, this allows us to be more friendly in a multi-tenant ES
   environment for relatively low engineering cost.
   - simplifies the Kibana dashboard a bit. We currently needed to create a
   special index pattern in order to accommodate multi-index pattern matching
   across all metron-specific indexes. Using metron_* would be much simpler
   and less prone to error.
   - easier for customers to debug and identify Metron-specific indexes and
   associated data


The reason for making these changes now is that we already have breaking
changes with ES. Leveraging existing indexed data rather than deleting
indexes and starting from scractch already requires a re-indexing/migration
step, so there is no additional effort on the part of users if they choose
to attempt a migration. It further makes sense with our current work
towards upgrading Solr.

We already have a battery of integration and manual tests after the ES
upgrade work that can be leveraged to validate the changes.

Mike Miklavcic

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message