metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: HDFS Compression
Date Tue, 11 Oct 2016 17:03:26 GMT
I'm somewhat ignorant here, never having used the MaaS stuff yet, but isn't
that the dataset that the models would run against?  I understand there
could be additional use cases, I just wanted to be clear.

Jon

On Tue, Oct 11, 2016 at 1:01 PM Nick Allen <nick@nickallen.org> wrote:

> I don't think we put much thought into how exactly the data should be
> landed in HDFS and for what use cases.  It just has not been a priority.
>
> That being said, this might be a good time to gather everyone's thoughts on
> how they would use that kind of data and for what purposes.
>
>
>
> On Tue, Oct 11, 2016 at 12:11 PM, Owen O'Malley <omalley@apache.org>
> wrote:
>
> > Be careful of using compressed JSON, since it isn't splittable. JSON is
> > also very slow for reading.
> >
> > .. Owen
> >
> > On Tue, Oct 11, 2016 at 4:31 AM, Casey Stella <cestella@gmail.com>
> wrote:
> >
> > > I'd also tack on to this that the configuration for the hdfs writer
> > should
> > > be moved to zookeeper rather than done in flux, IMO
> > > On Tue, Oct 11, 2016 at 07:20 Otto Fowler <ottobackwards@gmail.com>
> > wrote:
> > >
> > > > The storage format and retrieval from that format should be
> > configurable,
> > > > that is a ‘boundary’ for Metron so to speak.
> > > >
> > > > On October 10, 2016 at 16:15:12, Zeolla@GMail.com (zeolla@gmail.com)
> > > > wrote:
> > > >
> > > > Is there a specific reason why the JSON files stored in HDFS are not
> > > > compressed? I looked for some related JIRAs and mail conversations
> but
> > > > couldn't find this already mentioned. I'm wondering if there was a
> good
> > > > enough of an argument to keep things uncompressed, or if the subject
> > just
> > > > hadn't been broached yet.
> > > >
> > > > Jon
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> >
>
>
>
> --
> Nick Allen <nick@nickallen.org>
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message