metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kuba Sienkiewicz <kubasie...@gmail.com>
Subject Re: Removing historical data
Date Fri, 17 Jun 2016 08:22:48 GMT
Thanks for response, but how often metron purges old data?

pt., 17.06.2016 o 04:08 użytkownik Nick Allen <nick@nickallen.org> napisał:

> I don't think we have the functionality that you are looking for yet.  Feel
> free to open a JIRA to help define exactly what that might look like.
>
> I think providing some means of automatically managing data as it ages
> would be interesting.  One approach I've been thinking about is trading
> some fidelity for space as it ages.  For example, once I hit a ceiling on
> the amount of raw pcap that I can store, the system could potentially
> reprocess that raw pcap data to convert it to header-only pcap, flow
> records or some other form that consumes less storage, but loses some
> fidelity.
>
>
> On Thu, Jun 16, 2016 at 2:50 PM, Kuba Sienkiewicz <kubasienki@gmail.com>
> wrote:
>
> > What Merton thinks is old? (I mean when Merton purges data) About backup
> > data I mean having some space for hdfs and having separate partition (or
> > machine) for historical data.
> > On 14 Jun 2016 22:10, "Nick Allen" <nick@nickallen.org> wrote:
> >
> > > The standard deployment does setup some cleanup tasks that purges 'old'
> > > data.  This seems to be different from what you're asking though.  What
> > do
> > > you imagine using as your "backup space" if not /dev/null?
> > >
> > >
> > >
> > >
> > > On Tue, Jun 14, 2016 at 7:45 AM, Kuba Sienkiewicz <
> kubasienki@gmail.com>
> > > wrote:
> > >
> > > > PS. I'm saying about automatically moving data away from hdfs to some
> > > sort
> > > > of backup space. Sorry for inaccuracy in previous email (removing).
> > > >
> > > > wt., 14.06.2016 o 13:32 użytkownik Kuba Sienkiewicz <
> > > kubasienki@gmail.com>
> > > > napisał:
> > > >
> > > > > Hi all,
> > > > > I've tested metron for 2 days and I already have over 20GB of data
> > (w/o
> > > > > any special network traffic, server just stood untouched).
> > > > > What are good practices with such big amounts of data that metron
> > > stores?
> > > > > Also do metron support removing historical data automatically?
> > > > >
> > > > > Best Regards,
> > > > > Jakub Sienkiewicz
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Nick Allen <nick@nickallen.org>
> > >
> >
>
>
>
> --
> Nick Allen <nick@nickallen.org>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message