metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: PCAP payload retention/truncation
Date Wed, 27 Jul 2016 10:06:37 GMT
Yeah, exactly, except in my case there's a need for limited payload in
between full PCAP and header only PCAP.  My team will often use it for
signature validation and false positive analysis.  I'm just a bit unsure of
the best way to handle connections that may be long lived, in the tens of
hours or days, especially if it is longer than my full PCAP retention
policy.

Jon

On Tue, Jul 26, 2016, 22:33 Nick Allen <nick@nickallen.org> wrote:

> Yes, I agree with you Jon.  I think it would be interesting to have a
> mechanism that gradually trades-off fidelity for disk space.
>
> Of course, you would start with raw pcap.  After a threshold is crossed,
> the oldest data would be transformed to header-only pcap. From there you
> could transform the header-only pcap to flow records.  The specific ways in
> which the data is transformed/degraded would have to be configurable.
> Maybe someone doesn't find header-only pcap useful at all and simply goes
> from full pcap to flow records.
>
>
>
> On Wed, Jul 20, 2016 at 2:01 PM, Zeolla@GMail.com <zeolla@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Has there been any discussion on truncating or extracting some payload
> from
> > PCAP on data expiration instead of doing the standard YAF flow IPFIX?  I
> > would like to have longer retention of PCAP, but don't have an empty
> > multi-PB storage array at our disposal to handle full captures.  We have
> > used time-machine to do bpf-based connection cutoffs this in the past
> > (relevant
> > documentation
> > <
> >
> https://github.com/bro/time-machine/blob/master/doc/howto.rst#class-section
> > >
> > ), however that product does not support IPv6 (unless you use the LBNL
> > non-sanctified branch) and seems to be mostly dead.  Time-machine has
> been
> > mostly eclipsed by Stenographer as much as I can tell, but they do not
> > support this feature either (relevant issue
> > <https://github.com/google/stenographer/issues/123>).
> >
> > I spoke with the YAF/SiLK developers earlier today about this and they
> > mentioned that it is possible to have YAF export the first X bytes of the
> > payload and store it in their IPFIX output.  Perhaps this could become a
> > configurable change, where you specify the amount of payload to retain?
> > Interested in getting input on this - if feedback is positive we may be
> > able to dedicate some development cycles to it.
> >
> > Jon
> > --
> >
> > Jon
> >
>
>
>
> --
> Nick Allen <nick@nickallen.org>
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message