nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Whitaker <russell.whita...@gmail.com>
Subject Re: "Processor requires an upstream connection" for FetchS3Object?
Date Wed, 13 Jan 2016 02:16:21 GMT
On Tue, Jan 12, 2016 at 6:11 PM, Corey Flowers <cflowers@onyxpoint.com> wrote:
> Ha ha! Well that would do it! :)
>

I don't know what that would "do" other than confirm that the
FetchS3Object processor shipped
with v0.4.1 needs its doc to reflect the fact it's not yet useable
untl a ListS3* processor is implemented
and included in the narfile for the distribution.

Russell

> Sent from my iPhone
>
>> On Jan 12, 2016, at 9:10 PM, Russell Whitaker <russell.whitaker@gmail.com>
wrote:
>>
>>> On Tue, Jan 12, 2016 at 6:02 PM, Corey Flowers <cflowers@onyxpoint.com>
wrote:
>>> I haven't worked with this processor but I believe it is looking for
>>> the S3 list processor to generate the list of objects to fetch. Did
>>> you try that yet?
>>
>> I mentioned this: "There's no "ListS3Object" processor type which
>> might hypothetically populate
>> attributes for FetchS3Object to act upon." I should have made this
>> doubly explicit that I checked
>> in the processor creation dialogue.
>>
>> Also, this:
>> https://mail-archives.apache.org/mod_mbox/nifi-users/201510.mbox/%3CD23C06E8.CA0%25chakrader.dewaragatla@lifelock.com%3E
>>
>> "There is already a ticket
>> (NIFI-840<https://issues.apache.org/jira/browse/NIFI-840>)
>> in the hopper to create a ListS3Objects processor that can track
>> bucket contents and trigger
>> FetchS3Object."
>>
>> Oh god, it does appear that v0.4.1 ships with an implemented
>> FetchS3Object processor but no
>> List processor to feed it:
>>
>> https://issues.apache.org/jira/browse/NIFI-840
>>
>> Status: unresolved
>>
>> Description: "A processor is needed that can provide an S3 listing to
>> use in conjunction with FetchS3Object. This is to provide a similar
>> user experience as with the HDFS processors that perform List/Get."
>>
>> I think this means I'm horked. And the Relationships section of the
>> FetchS3Object doc is still wrong.
>>
>> Russell
>>
>>
>>> Sent from my iPhone
>>>
>>>> On Jan 12, 2016, at 8:38 PM, Russell Whitaker <russell.whitaker@gmail.com>
wrote:
>>>>
>>>> I'm running v0.4.1 Nifi, and seeing this (taken from nifi-app.log,
>>>> also seeing on mouseover of the "!" icon on the processor on the
>>>> canvas):
>>>>
>>>> 2016-01-12 17:08:50,357 ERROR [NiFi Web Server-18]
>>>> o.a.nifi.groups.StandardProcessGroup Unable to start
>>>> FetchS3Object[id=f4253204-a2e2-4ce6-ba09-9415e8024dca] due to {}
>>>> java.lang.IllegalStateException: Processor FetchS3Object is not in a
>>>> valid state due to ['Upstream Connections' is invalid because
>>>> Processor requires an upstream connection but currently has none]
>>>>
>>>> Per:
>>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.FetchS3Object/index.html,
>>>> FetchS3Object "Retrieves the contents of an S3 Object and writes it to
>>>> the content of a FlowFile," which would seem to indicate this is an
>>>> "edge" processor that doesn't expect a flowfile from an upstream
>>>> processor.
>>>>
>>>> The "Tags" on the doc are: "Amazon, S3, AWS, Get, Fetch"
>>>>
>>>> The processor configuration settings themselves strongly indicate it
>>>> expects to connect to S3 using the supplied
>>>> credentials/bucket/objectkey settings, with no upstream processor.
>>>>
>>>> But I get this error. What am I missing? There's no GetS3Object
>>>> anymore; surely this is the edge processor for directly downloading S3
>>>> objects, yes? There's no "ListS3Object" processor type which might
>>>> hypothetically populate attributes for FetchS3Object to act upon.
>>>>
>>>> Also, there are these obviously incorrect copy/paste lines in the
>>>> Relationships area of the API doc referenced above:
>>>>
>>>> "success - FlowFiles are routed to success after being successfully
>>>> copied to Amazon S3"
>>>> "failure - FlowFiles are routed to failure if unable to be copied to Amazon
S3"
>>>>
>>>> No, that's obviously lifted from the PutS3Object doc page, where it's
>>>> actually correct:
>>>>
>>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.PutS3Object/index.html
>>>>
>>>> Anyone have any insight into this? Thanks in advance.
>>>>
>>>> Russell
>>
>> --
>> Russell Whitaker
>> http://twitter.com/OrthoNormalRuss
>> http://www.linkedin.com/pub/russell-whitaker/0/b86/329



-- 
Russell Whitaker
http://twitter.com/OrthoNormalRuss
http://www.linkedin.com/pub/russell-whitaker/0/b86/329

Mime
View raw message