metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franck Vervial <verv...@gmail.com>
Subject Re: Removing historical data
Date Fri, 17 Jun 2016 11:25:03 GMT
hello,

For HP Arcisght for example, you cannot delete specific logs. You can only
delete all logs before a specified date. The interest of this contraint is
for security and compliance: even if you SIEM is haked or if someone with
an access want to delete his or her traces, it is not possible. Only
deleted all logs is possible and it is not discret.
I guess there should have orher advantages for the optimisation of the
storage of logs.
The downside of this constraint is when by mistake you sent too much logs
in your SIEM, you will not be able to delete them.
On Fri, 17 Jun 2016 at 4:23 PM, Kuba Sienkiewicz <kubasienki@gmail.com>
wrote:

> Thanks for response, but how often metron purges old data?
>
> pt., 17.06.2016 o 04:08 użytkownik Nick Allen <nick@nickallen.org>
> napisał:
>
> > I don't think we have the functionality that you are looking for yet.
> Feel
> > free to open a JIRA to help define exactly what that might look like.
> >
> > I think providing some means of automatically managing data as it ages
> > would be interesting.  One approach I've been thinking about is trading
> > some fidelity for space as it ages.  For example, once I hit a ceiling on
> > the amount of raw pcap that I can store, the system could potentially
> > reprocess that raw pcap data to convert it to header-only pcap, flow
> > records or some other form that consumes less storage, but loses some
> > fidelity.
> >
> >
> > On Thu, Jun 16, 2016 at 2:50 PM, Kuba Sienkiewicz <kubasienki@gmail.com>
> > wrote:
> >
> > > What Merton thinks is old? (I mean when Merton purges data) About
> backup
> > > data I mean having some space for hdfs and having separate partition
> (or
> > > machine) for historical data.
> > > On 14 Jun 2016 22:10, "Nick Allen" <nick@nickallen.org> wrote:
> > >
> > > > The standard deployment does setup some cleanup tasks that purges
> 'old'
> > > > data.  This seems to be different from what you're asking though.
> What
> > > do
> > > > you imagine using as your "backup space" if not /dev/null?
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jun 14, 2016 at 7:45 AM, Kuba Sienkiewicz <
> > kubasienki@gmail.com>
> > > > wrote:
> > > >
> > > > > PS. I'm saying about automatically moving data away from hdfs to
> some
> > > > sort
> > > > > of backup space. Sorry for inaccuracy in previous email (removing).
> > > > >
> > > > > wt., 14.06.2016 o 13:32 użytkownik Kuba Sienkiewicz <
> > > > kubasienki@gmail.com>
> > > > > napisał:
> > > > >
> > > > > > Hi all,
> > > > > > I've tested metron for 2 days and I already have over 20GB of
> data
> > > (w/o
> > > > > > any special network traffic, server just stood untouched).
> > > > > > What are good practices with such big amounts of data that metron
> > > > stores?
> > > > > > Also do metron support removing historical data automatically?
> > > > > >
> > > > > > Best Regards,
> > > > > > Jakub Sienkiewicz
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Nick Allen <nick@nickallen.org>
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <nick@nickallen.org>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message