metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <>
Subject [DISCUSS] Field conversions
Date Mon, 04 Jun 2018 20:55:19 GMT
We've been dealing with a reoccurring challenge in Metron.  It is common
for various fields to contain '.' characters for the purpose of making them
more readable, namespacing, etc.  At one point we only supported
Elasticsearch 2.3 which did not allow dots and forced us to use ':'
instead.  This limitation does not exist in later versions of Elasticsearch
or Solr.

Now we're in a situation where we need to allow a user to use either one
because they may still be using ES 2.3 or have data with ':' characters in
field names.  We've attempted to make this configurable in a couple
different PRs:

The approaches taken in these are not consistent and fall short in
different ways.  The first (METRON-1569 Allow user to change field name
conversion when indexing) only applies to indexing and not querying.  The
others only apply to a single field which does not scale well.  Now we have
an issue with another field in  Rather than continuing
with a patchwork of different fixes I want to attempt to design a
system-wide solution.

My first thought is to expand to
apply globally.  However this is not trivial and would require significant
changes.  It would also make
obsolete and we might end up having to revert all of it.

Does anyone have any ideas or opinions?  I am still researching solutions
but would love some guidance from the community.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message