metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <alinazem...@gmail.com>
Subject Re: Change field separator in Metron to make it Hive and ORC friendly
Date Tue, 14 Aug 2018 09:53:55 GMT
Hi Simon,

We have temporarily decided to just change it with "_" for HDFS to avoid
all the headaches of the bugs and issues that can be raised by using
unsupported separators for ORC/Hive and Spark. However, I am not quite
confident with "_" as an option for the community as it becomes similar to
normal Metron separator. Maybe it would be nice to have an ability to
change the separator to any other character and let users decide what they
want to use.

Cheers,
Ali

On Tue, Aug 14, 2018 at 12:14 AM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> Do you have any suggestions for what would make sense as a delimiter?
>
> On 9 August 2018 at 05:57, Ali Nazemian <alinazemian@gmail.com> wrote:
>
> > Hi All,
> >
> > I was wondering if we can change the field separators in Metron to be
> able
> > to make it Hive/ORC friendly. I could find the following PR, but neither
> > dot nor colon is very Hive and ORC friendly and they will cause some
> > issues. Hence, I wanted to see if it is possible to change the field
> > separator to something else or even give users an ability to define what
> > separator to be used to make the data model consistent across
> Elasticsearch
> > and HDFS.
> >
> > https://github.com/apache/metron/pull/1022
> >
> > Cheers,
> > Ali
> >
>
>
>
> --
> --
> simon elliston ball
> @sireb
>


-- 
A.Nazemian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message