nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Lamar <adamond...@gmail.com>
Subject Re: ListS3 processor?
Date Sun, 31 Jan 2016 06:57:26 GMT
Kyle/Joe,

I've been meaning to document this process myself, and just finished a 
post with some details:
https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/

Hope that helps,
Adam

On 1/30/16 9:29 PM, Joe Witt wrote:
> Kyle,
>
> The ideal case for communicating how to do this would be both a
> template and an associated doc.  Great for a blog or wiki page or
> something.  We can of course give you perms to write to a wiki page on
> the nifi wiki if interested.  The template itself can also be
> annotated with comments that show up right in the flow itself.  That
> may be a fine option too.
>
> Thanks
> Joe
>
> On Sat, Jan 30, 2016 at 2:52 PM, Kyle Burke <kyle.burke@ignitionone.com> wrote:
>> Joe/Joe,
>>    Thanks for the response. It makes sense to use SNS and SQS to respond to
>> S3 file changes. I’m going see if my company will give me access to those
>> Amazon services. I found an article that explains how to setup on this
>> functionality in the Amazon console. Once that’s setup it seems pretty
>> straight forward to use GetSQS/DeleteSQS. I suspect many will want this
>> functionality but I’m not sure what’s the best method (i.e. Template or user
>> doc) that explains how to solve this in nifi.  I’ll be happy to submit
>> something if you let me know the right method.
>>
>> http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
>>
>> Respectfully,
>>
>> Kyle Burke | Data Science Engineer
>> IgnitionOne - Marketing Technology. Simplified.
>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
>>
>>
>> From: Joe Witt
>> Reply-To: "users@nifi.apache.org"
>> Date: Saturday, January 30, 2016 at 2:06 PM
>> To: "users@nifi.apache.org"
>> Subject: Re: ListS3 processor?
>>
>> Kyle
>>
>> Let us know if that doesn't get you what you need.  We have a decent set of
>> templates but I didn't see one that demonstrates interaction with amazon
>> services.
>>
>> Thanks
>> Joe
>>
>> On Jan 30, 2016 12:56 PM, "Joey Frazee" <joey.frazee@icloud.com> wrote:
>>> Kyle,
>>>
>>> I think you can do what you want right now without ListS3 by using S3
>>> event notifications. You can configure an event notification to publish to
>>> SQS and then use GetSQS to retrieve the events and FetchS3Object to get the
>>> JSON file and the rest of the flow could be written as you have in mind.
>>>
>>> Depending on your scale, this might be preferable because it's
>>> slow/expensive to do listings on S3 prefixes that have a lot of file
>>> matches.
>>>
>>>
>>> -joey
>>>
>>> On Jan 30, 2016, at 11:40 AM, Joe Skora <jskora@gmail.com> wrote:
>>>
>>> Kyle,
>>>
>>> Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the
>>> backlog on ticket NIFI-840 at the moment.  It should fit the List/Fetch
>>> metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc.
>>>
>>> Regards,
>>> Joe Skora
>>>
>>> On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <kyle.burke@ignitionone.com>
>>> wrote:
>>>> All,
>>>>    I'm trying to get Nifi set up to a move data around S3. My first
>>>> attempt is to just monitor a S3 folder where json files are placed and then
>>>> copy the file, convert it to Avro, and the drop it in a different S3 folder.
>>>> The documentation is pretty slim for working with S3. I can't seem to get
it
>>>> working and was wondering if anyone had any S3 examples for monitoring an
S3
>>>> folder (i.e.. something like a ListS3 processer similar to what is available
>>>> on a local file system?)
>>>>
>>>> Respectfully,
>>>>
>>>> Kyle Burke | Data Science Engineer
>>>> IgnitionOne - Marketing Technology. Simplified.
>>>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
>>>> Direct: 404.961.3918
>>>>


Mime
View raw message