nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff <>
Subject Re: S3 Cloudtrail log ingestion to Kafka via NiFi
Date Wed, 07 Jun 2017 22:12:20 GMT
Hello Laurens,

I set up a flow to test this as well, and also saw that the unzipped json
looks like it contains references to other S3Objects.  I'm not familiar
with the formats used by CloudTrail or how the actual logging data is
stored.  I'll have to read up on it, but I think we can set up a flow to
split the json, grab the referenced S3Object values, and route them to
another FetchS3Object processor to pull back the actual logs and decompress

I'll be away from my computer for most of the night, but will hopefully get
back to you tomorrow after doing some more research.

On Wed, Jun 7, 2017 at 1:30 PM James Wing <> wrote:

> Are you seeing errors, or just unexpected results?  ListS3 only returns
> references to objects on S3, but FetchS3Object should return the object
> content.  I recommend looking at the output of FetchS3Object to make sure
> it is right (in size and content type) before trying to unzip it.
> Thanks,
> James
> On Wed, Jun 7, 2017 at 9:56 AM, Laurens Vets <> wrote:
>> Hello,
>> Has anyone been able to ingest S3 Cloudtrail logs into Kafka with NiFi? I
>> got as far ListS3 -> FetchS3Object -> Gunzip, but I'm stuck here. It seems
>> I'm not actually unzipping the logs, but references to the S3 objects?
>> Any help would be appreciated.

View raw message