metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: HDFS Compression
Date Tue, 11 Oct 2016 17:00:56 GMT
I don't think we put much thought into how exactly the data should be
landed in HDFS and for what use cases.  It just has not been a priority.

That being said, this might be a good time to gather everyone's thoughts on
how they would use that kind of data and for what purposes.



On Tue, Oct 11, 2016 at 12:11 PM, Owen O'Malley <omalley@apache.org> wrote:

> Be careful of using compressed JSON, since it isn't splittable. JSON is
> also very slow for reading.
>
> .. Owen
>
> On Tue, Oct 11, 2016 at 4:31 AM, Casey Stella <cestella@gmail.com> wrote:
>
> > I'd also tack on to this that the configuration for the hdfs writer
> should
> > be moved to zookeeper rather than done in flux, IMO
> > On Tue, Oct 11, 2016 at 07:20 Otto Fowler <ottobackwards@gmail.com>
> wrote:
> >
> > > The storage format and retrieval from that format should be
> configurable,
> > > that is a ‘boundary’ for Metron so to speak.
> > >
> > > On October 10, 2016 at 16:15:12, Zeolla@GMail.com (zeolla@gmail.com)
> > > wrote:
> > >
> > > Is there a specific reason why the JSON files stored in HDFS are not
> > > compressed? I looked for some related JIRAs and mail conversations but
> > > couldn't find this already mentioned. I'm wondering if there was a good
> > > enough of an argument to keep things uncompressed, or if the subject
> just
> > > hadn't been broached yet.
> > >
> > > Jon
> > > --
> > >
> > > Jon
> > >
> >
>



-- 
Nick Allen <nick@nickallen.org>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message