metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: PCAP payload retention/truncation
Date Wed, 27 Jul 2016 02:33:10 GMT
Yes, I agree with you Jon.  I think it would be interesting to have a
mechanism that gradually trades-off fidelity for disk space.

Of course, you would start with raw pcap.  After a threshold is crossed,
the oldest data would be transformed to header-only pcap. From there you
could transform the header-only pcap to flow records.  The specific ways in
which the data is transformed/degraded would have to be configurable.
Maybe someone doesn't find header-only pcap useful at all and simply goes
from full pcap to flow records.



On Wed, Jul 20, 2016 at 2:01 PM, Zeolla@GMail.com <zeolla@gmail.com> wrote:

> Hi All,
>
> Has there been any discussion on truncating or extracting some payload from
> PCAP on data expiration instead of doing the standard YAF flow IPFIX?  I
> would like to have longer retention of PCAP, but don't have an empty
> multi-PB storage array at our disposal to handle full captures.  We have
> used time-machine to do bpf-based connection cutoffs this in the past
> (relevant
> documentation
> <
> https://github.com/bro/time-machine/blob/master/doc/howto.rst#class-section
> >
> ), however that product does not support IPv6 (unless you use the LBNL
> non-sanctified branch) and seems to be mostly dead.  Time-machine has been
> mostly eclipsed by Stenographer as much as I can tell, but they do not
> support this feature either (relevant issue
> <https://github.com/google/stenographer/issues/123>).
>
> I spoke with the YAF/SiLK developers earlier today about this and they
> mentioned that it is possible to have YAF export the first X bytes of the
> payload and store it in their IPFIX output.  Perhaps this could become a
> configurable change, where you specify the amount of payload to retain?
> Interested in getting input on this - if feedback is positive we may be
> able to dedicate some development cycles to it.
>
> Jon
> --
>
> Jon
>



-- 
Nick Allen <nick@nickallen.org>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message