nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: ListS3 processor?
Date Sun, 31 Jan 2016 07:01:59 GMT
Ha!  Insert jaw-drop-emoticon-here.

Perfect timing and very well written Adam.

Thanks

On Sun, Jan 31, 2016 at 1:57 AM, Adam Lamar <adamonduty@gmail.com> wrote:
> Kyle/Joe,
>
> I've been meaning to document this process myself, and just finished a post
> with some details:
> https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/
>
> Hope that helps,
> Adam
>
>
> On 1/30/16 9:29 PM, Joe Witt wrote:
>>
>> Kyle,
>>
>> The ideal case for communicating how to do this would be both a
>> template and an associated doc.  Great for a blog or wiki page or
>> something.  We can of course give you perms to write to a wiki page on
>> the nifi wiki if interested.  The template itself can also be
>> annotated with comments that show up right in the flow itself.  That
>> may be a fine option too.
>>
>> Thanks
>> Joe
>>
>> On Sat, Jan 30, 2016 at 2:52 PM, Kyle Burke <kyle.burke@ignitionone.com>
>> wrote:
>>>
>>> Joe/Joe,
>>>    Thanks for the response. It makes sense to use SNS and SQS to respond
>>> to
>>> S3 file changes. I’m going see if my company will give me access to those
>>> Amazon services. I found an article that explains how to setup on this
>>> functionality in the Amazon console. Once that’s setup it seems pretty
>>> straight forward to use GetSQS/DeleteSQS. I suspect many will want this
>>> functionality but I’m not sure what’s the best method (i.e. Template or
>>> user
>>> doc) that explains how to solve this in nifi.  I’ll be happy to submit
>>> something if you let me know the right method.
>>>
>>> http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
>>>
>>> Respectfully,
>>>
>>> Kyle Burke | Data Science Engineer
>>> IgnitionOne - Marketing Technology. Simplified.
>>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
>>>
>>>
>>> From: Joe Witt
>>> Reply-To: "users@nifi.apache.org"
>>> Date: Saturday, January 30, 2016 at 2:06 PM
>>> To: "users@nifi.apache.org"
>>> Subject: Re: ListS3 processor?
>>>
>>> Kyle
>>>
>>> Let us know if that doesn't get you what you need.  We have a decent set
>>> of
>>> templates but I didn't see one that demonstrates interaction with amazon
>>> services.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Jan 30, 2016 12:56 PM, "Joey Frazee" <joey.frazee@icloud.com> wrote:
>>>>
>>>> Kyle,
>>>>
>>>> I think you can do what you want right now without ListS3 by using S3
>>>> event notifications. You can configure an event notification to publish
>>>> to
>>>> SQS and then use GetSQS to retrieve the events and FetchS3Object to get
>>>> the
>>>> JSON file and the rest of the flow could be written as you have in mind.
>>>>
>>>> Depending on your scale, this might be preferable because it's
>>>> slow/expensive to do listings on S3 prefixes that have a lot of file
>>>> matches.
>>>>
>>>>
>>>> -joey
>>>>
>>>> On Jan 30, 2016, at 11:40 AM, Joe Skora <jskora@gmail.com> wrote:
>>>>
>>>> Kyle,
>>>>
>>>> Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in
>>>> the
>>>> backlog on ticket NIFI-840 at the moment.  It should fit the List/Fetch
>>>> metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP,
>>>> etc.
>>>>
>>>> Regards,
>>>> Joe Skora
>>>>
>>>> On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke
>>>> <kyle.burke@ignitionone.com>
>>>> wrote:
>>>>>
>>>>> All,
>>>>>    I'm trying to get Nifi set up to a move data around S3. My first
>>>>> attempt is to just monitor a S3 folder where json files are placed and
>>>>> then
>>>>> copy the file, convert it to Avro, and the drop it in a different S3
>>>>> folder.
>>>>> The documentation is pretty slim for working with S3. I can't seem to
>>>>> get it
>>>>> working and was wondering if anyone had any S3 examples for monitoring
>>>>> an S3
>>>>> folder (i.e.. something like a ListS3 processer similar to what is
>>>>> available
>>>>> on a local file system?)
>>>>>
>>>>> Respectfully,
>>>>>
>>>>> Kyle Burke | Data Science Engineer
>>>>> IgnitionOne - Marketing Technology. Simplified.
>>>>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
>>>>> Direct: 404.961.3918
>>>>>
>

Mime
View raw message