metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: HDFS Compression
Date Tue, 11 Oct 2016 16:11:55 GMT
Be careful of using compressed JSON, since it isn't splittable. JSON is
also very slow for reading.

.. Owen

On Tue, Oct 11, 2016 at 4:31 AM, Casey Stella <cestella@gmail.com> wrote:

> I'd also tack on to this that the configuration for the hdfs writer should
> be moved to zookeeper rather than done in flux, IMO
> On Tue, Oct 11, 2016 at 07:20 Otto Fowler <ottobackwards@gmail.com> wrote:
>
> > The storage format and retrieval from that format should be configurable,
> > that is a ‘boundary’ for Metron so to speak.
> >
> > On October 10, 2016 at 16:15:12, Zeolla@GMail.com (zeolla@gmail.com)
> > wrote:
> >
> > Is there a specific reason why the JSON files stored in HDFS are not
> > compressed? I looked for some related JIRAs and mail conversations but
> > couldn't find this already mentioned. I'm wondering if there was a good
> > enough of an argument to keep things uncompressed, or if the subject just
> > hadn't been broached yet.
> >
> > Jon
> > --
> >
> > Jon
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message