metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: Removing historical data
Date Fri, 17 Jun 2016 02:08:50 GMT
I don't think we have the functionality that you are looking for yet.  Feel
free to open a JIRA to help define exactly what that might look like.

I think providing some means of automatically managing data as it ages
would be interesting.  One approach I've been thinking about is trading
some fidelity for space as it ages.  For example, once I hit a ceiling on
the amount of raw pcap that I can store, the system could potentially
reprocess that raw pcap data to convert it to header-only pcap, flow
records or some other form that consumes less storage, but loses some
fidelity.


On Thu, Jun 16, 2016 at 2:50 PM, Kuba Sienkiewicz <kubasienki@gmail.com>
wrote:

> What Merton thinks is old? (I mean when Merton purges data) About backup
> data I mean having some space for hdfs and having separate partition (or
> machine) for historical data.
> On 14 Jun 2016 22:10, "Nick Allen" <nick@nickallen.org> wrote:
>
> > The standard deployment does setup some cleanup tasks that purges 'old'
> > data.  This seems to be different from what you're asking though.  What
> do
> > you imagine using as your "backup space" if not /dev/null?
> >
> >
> >
> >
> > On Tue, Jun 14, 2016 at 7:45 AM, Kuba Sienkiewicz <kubasienki@gmail.com>
> > wrote:
> >
> > > PS. I'm saying about automatically moving data away from hdfs to some
> > sort
> > > of backup space. Sorry for inaccuracy in previous email (removing).
> > >
> > > wt., 14.06.2016 o 13:32 u┼╝ytkownik Kuba Sienkiewicz <
> > kubasienki@gmail.com>
> > > napisał:
> > >
> > > > Hi all,
> > > > I've tested metron for 2 days and I already have over 20GB of data
> (w/o
> > > > any special network traffic, server just stood untouched).
> > > > What are good practices with such big amounts of data that metron
> > stores?
> > > > Also do metron support removing historical data automatically?
> > > >
> > > > Best Regards,
> > > > Jakub Sienkiewicz
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <nick@nickallen.org>
> >
>



-- 
Nick Allen <nick@nickallen.org>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message