metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <alinazem...@gmail.com>
Subject Re: [DISCUSS] Update Metron Elasticsearch index names to metron_
Date Wed, 24 Jan 2018 23:08:08 GMT
Hi All,

I just wanted to say it would be great if we can be careful with these type
of changes. From the development point of view, it is just a few lines of
code which can provide multiple advantages, but for live large-scale Metron
platforms, some of these changes might be really expensive to address with
zero-downtime.

Cheers,
Ali

On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> +1
>
>
> On January 24, 2018 at 16:28:42, Nick Allen (nick@nickallen.org) wrote:
>
> +1 to a standard prefix for all Metron indices. I've had the same thought
> myself and you laid out the advantages well.
>
>
>
>
>
> On Wed, Jan 24, 2018 at 3:47 PM Zeolla@GMail.com <zeolla@gmail.com> wrote:
>
> > I agree with having a metron_ prefix for ES indexes, and the timing.
> >
> > Jon
> >
> > On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> > > With the completion of https://github.com/apache/metron/pull/840
> > > (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for
> a
> > > major release rev of Metron in the upcoming release (currently slotted
> to
> > > 0.4.3, I believe). Since there are non-backwards compatible changes
> > > pertaining to ES indexing, it seems like a good opportunity to revisit
> > our
> > > index naming standards.
> > >
> > > I propose we add a simple prefix "metron_" to all Metron indexes. There
> > are
> > > numerous reasons for doing so
> > >
> > > - removes the likelihood of index name collisions when we perform
> > > operations on index wildcard names, e.g. "enrichment_*, indexing_*,
> > > etc.".
> > > - ie, this allows us to be more friendly in a multi-tenant ES
> > > environment for relatively low engineering cost.
> > > - simplifies the Kibana dashboard a bit. We currently needed to
> > create a
> > > special index pattern in order to accommodate multi-index pattern
> > > matching
> > > across all metron-specific indexes. Using metron_* would be much
> > simpler
> > > and less prone to error.
> > > - easier for customers to debug and identify Metron-specific indexes
> > and
> > > associated data
> > >
> > >
> > > The reason for making these changes now is that we already have
> breaking
> > > changes with ES. Leveraging existing indexed data rather than deleting
> > > indexes and starting from scractch already requires a
> > re-indexing/migration
> > > step, so there is no additional effort on the part of users if they
> > choose
> > > to attempt a migration. It further makes sense with our current work
> > > towards upgrading Solr.
> > >
> > > We already have a battery of integration and manual tests after the ES
> > > upgrade work that can be leveraged to validate the changes.
> > >
> > > Mike Miklavcic
> > >
> >
> >
> > --
> >
> > Jon
> >
>



-- 
A.Nazemian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message