metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Update Metron Elasticsearch index names to metron_
Date Thu, 25 Jan 2018 01:34:58 GMT
I hear you Ali. I think this type of change would actually ease issues with
downtime because it offers an easy path to migrating existing indices. I'd
have to review the specifics in the ES docs again, but I believe you could
duplicate the old indexes and migrate them to "metron_" in advance of the
upgrade, and then consume new data to the new index pattern/name after the
upgrade. That should be pretty seamless, I think. I guess it depends on how
you're using ES.

On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazemian@gmail.com> wrote:

> Hi All,
>
> I just wanted to say it would be great if we can be careful with these type
> of changes. From the development point of view, it is just a few lines of
> code which can provide multiple advantages, but for live large-scale Metron
> platforms, some of these changes might be really expensive to address with
> zero-downtime.
>
> Cheers,
> Ali
>
> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
> > +1
> >
> >
> > On January 24, 2018 at 16:28:42, Nick Allen (nick@nickallen.org) wrote:
> >
> > +1 to a standard prefix for all Metron indices. I've had the same thought
> > myself and you laid out the advantages well.
> >
> >
> >
> >
> >
> > On Wed, Jan 24, 2018 at 3:47 PM Zeolla@GMail.com <zeolla@gmail.com>
> wrote:
> >
> > > I agree with having a metron_ prefix for ES indexes, and the timing.
> > >
> > > Jon
> > >
> > > On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
> > > michael.miklavcic@gmail.com> wrote:
> > >
> > > > With the completion of https://github.com/apache/metron/pull/840
> > > > (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
> for
> > a
> > > > major release rev of Metron in the upcoming release (currently
> slotted
> > to
> > > > 0.4.3, I believe). Since there are non-backwards compatible changes
> > > > pertaining to ES indexing, it seems like a good opportunity to
> revisit
> > > our
> > > > index naming standards.
> > > >
> > > > I propose we add a simple prefix "metron_" to all Metron indexes.
> There
> > > are
> > > > numerous reasons for doing so
> > > >
> > > > - removes the likelihood of index name collisions when we perform
> > > > operations on index wildcard names, e.g. "enrichment_*, indexing_*,
> > > > etc.".
> > > > - ie, this allows us to be more friendly in a multi-tenant ES
> > > > environment for relatively low engineering cost.
> > > > - simplifies the Kibana dashboard a bit. We currently needed to
> > > create a
> > > > special index pattern in order to accommodate multi-index pattern
> > > > matching
> > > > across all metron-specific indexes. Using metron_* would be much
> > > simpler
> > > > and less prone to error.
> > > > - easier for customers to debug and identify Metron-specific indexes
> > > and
> > > > associated data
> > > >
> > > >
> > > > The reason for making these changes now is that we already have
> > breaking
> > > > changes with ES. Leveraging existing indexed data rather than
> deleting
> > > > indexes and starting from scractch already requires a
> > > re-indexing/migration
> > > > step, so there is no additional effort on the part of users if they
> > > choose
> > > > to attempt a migration. It further makes sense with our current work
> > > > towards upgrading Solr.
> > > >
> > > > We already have a battery of integration and manual tests after the
> ES
> > > > upgrade work that can be leveraged to validate the changes.
> > > >
> > > > Mike Miklavcic
> > > >
> > >
> > >
> > > --
> > >
> > > Jon
> > >
> >
>
>
>
> --
> A.Nazemian
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message