nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: nifi is running out of memory
Date Tue, 01 Nov 2016 02:08:07 GMT
Krish,

Did you ever get past this?

Thanks
Joe

On Fri, Oct 28, 2016 at 2:36 PM, Gop Krr <gop.krr@gmail.com> wrote:
> James, permission issue got resolved. I still don't see any write.
>
> On Fri, Oct 28, 2016 at 10:34 AM, Gop Krr <gop.krr@gmail.com> wrote:
>>
>> Thanks James.. I am looking into permission issue and update the thread. I
>> will also make the changes as you per your recommendation.
>>
>> On Fri, Oct 28, 2016 at 10:23 AM, James Wing <jvwing@gmail.com> wrote:
>>>
>>> From the screenshot and the error message, I interpret the sequence of
>>> events to be something like this:
>>>
>>> 1.) ListS3 succeeds and generates flowfiles with attributes referencing
>>> S3 objects, but no content (0 bytes)
>>> 2.) FetchS3Object fails to pull the S3 object content with an Access
>>> Denied error, but the failed flowfiles are routed on to PutS3Object (35,179
>>> files / 0 bytes in the "putconnector" queue)
>>> 3.) PutS3Object is succeeding, writing the 0 byte content from ListS3
>>>
>>> I recommend a couple thing for FetchS3Object:
>>>
>>> * Only allow the "success" relationship to continue to PutS3Object.
>>> Separate the "failure" relationship to either loop back to FetchS3Object or
>>> go to a LogAttibute processor, or other handling path.
>>> * It looks like the permissions aren't working, you might want to
>>> double-check the access keys or try a sample file with the AWS CLI.
>>>
>>> Thanks,
>>>
>>> James
>>>
>>>
>>> On Fri, Oct 28, 2016 at 10:01 AM, Gop Krr <gop.krr@gmail.com> wrote:
>>>>
>>>> This is how my nifi flow looks like.
>>>>
>>>> On Fri, Oct 28, 2016 at 9:57 AM, Gop Krr <gop.krr@gmail.com> wrote:
>>>>>
>>>>> Thanks Bryan, Joe, Adam and Pierre. I went past this issue by switching
>>>>> to 0.71.  Now it is able to list the files from buckets and create those
>>>>> files in the another bucket. But write is not happening and I am getting
the
>>>>> permission issue ( I have attached below for the reference) Could this
be
>>>>> the setting of the buckets or it has more to do with the access key.
All the
>>>>> files which are creaetd in the new bucket are of 0 byte.
>>>>> Thanks
>>>>> Rai
>>>>>
>>>>> 2016-10-28 16:45:25,438 ERROR [Timer-Driven Process Thread-3]
>>>>> o.a.nifi.processors.aws.s3.FetchS3Object FetchS3Object[id=xxxxx] Failed
to
>>>>> retrieve S3 Object for
>>>>> StandardFlowFileRecord[uuid=yyyyy,claim=,offset=0,name=xxxxx.gz,size=0];
>>>>> routing to failure: com.amazonaws.services.s3.model.AmazonS3Exception:
>>>>> Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
>>>>> AccessDenied; Request ID: xxxxxxx), S3 Extended Request ID:
>>>>> lu8tAqRxu+ouinnVvJleHkUUyK6J6rIQCTw0G8G6DB6NOPGec0D1KB6cfUPsj08IQXI8idtiTp4=
>>>>>
>>>>> 2016-10-28 16:45:25,438 ERROR [Timer-Driven Process Thread-3]
>>>>> o.a.nifi.processors.aws.s3.FetchS3Object
>>>>>
>>>>> com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied
>>>>> (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request
ID:
>>>>> 0F34E71C0697B1D8)
>>>>>
>>>>>         at
>>>>> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1219)
>>>>> ~[aws-java-sdk-core-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:803)
>>>>> ~[aws-java-sdk-core-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505)
>>>>> ~[aws-java-sdk-core-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317)
>>>>> ~[aws-java-sdk-core-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
>>>>> ~[aws-java-sdk-s3-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1116)
>>>>> ~[aws-java-sdk-s3-1.10.32.jar:na]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.java:106)
>>>>> ~[nifi-aws-processors-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>>> [nifi-api-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
>>>>> [nifi-framework-core-0.7.1.jar:0.7.1]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> [na:1.8.0_101]
>>>>>
>>>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>>>
>>>>>
>>>>> On Fri, Oct 28, 2016 at 6:31 AM, Pierre Villard
>>>>> <pierre.villard.fr@gmail.com> wrote:
>>>>>>
>>>>>> Quick remark: the fix has also been merged in master and will be
in
>>>>>> release 1.1.0.
>>>>>>
>>>>>> Pierre
>>>>>>
>>>>>> 2016-10-28 15:22 GMT+02:00 Gop Krr <gop.krr@gmail.com>:
>>>>>>>
>>>>>>> Thanks Adam. I will try 0.7.1 and update the community on the
>>>>>>> outcome. If it works then I can create a patch for 1.x
>>>>>>> Thanks
>>>>>>> Rai
>>>>>>>
>>>>>>> On Thu, Oct 27, 2016 at 7:41 PM, Adam Lamar <adamonduty@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hey All,
>>>>>>>>
>>>>>>>> I believe OP is running into a bug fixed here:
>>>>>>>> https://issues.apache.org/jira/browse/NIFI-2631
>>>>>>>>
>>>>>>>> Basically, ListS3 attempts to commit all the files it finds
>>>>>>>> (potentially 100k+) at once, rather than in batches. NIFI-2631
>>>>>>>> addresses the issue. Looks like the fix is out in 0.7.1 but
not yet
>>>>>>>> in
>>>>>>>> a 1.x release.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 27, 2016 at 7:59 PM, Joe Witt <joe.witt@gmail.com>
>>>>>>>> wrote:
>>>>>>>> > Looking at this line [1] makes me think the FetchS3
processor is
>>>>>>>> > properly streaming the bytes directly to the content
repository.
>>>>>>>> >
>>>>>>>> > Looking at the screenshot showing nothing out of the
ListS3
>>>>>>>> > processor
>>>>>>>> > makes me think the bucket has so many things in it that
the
>>>>>>>> > processor
>>>>>>>> > or associated library isn't handling that well and is
just listing
>>>>>>>> > everything with no mechanism of max buffer size.  Krish
please try
>>>>>>>> > with the largest heap you can and let us know what you
see.
>>>>>>>> >
>>>>>>>> > [1]
>>>>>>>> > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java#L107
>>>>>>>> >
>>>>>>>> > On Thu, Oct 27, 2016 at 9:37 PM, Joe Witt <joe.witt@gmail.com>
>>>>>>>> > wrote:
>>>>>>>> >> moving dev to bcc
>>>>>>>> >>
>>>>>>>> >> Yes I believe the issue here is that FetchS3 doesn't
do chunked
>>>>>>>> >> transfers and so is loading all into memory.  I've
not verified
>>>>>>>> >> this
>>>>>>>> >> in the code yet but it seems quite likely.  Krish
if you can
>>>>>>>> >> verify
>>>>>>>> >> that going with a larger heap gets you in the game
can you please
>>>>>>>> >> file
>>>>>>>> >> a JIRA.
>>>>>>>> >>
>>>>>>>> >> Thanks
>>>>>>>> >> Joe
>>>>>>>> >>
>>>>>>>> >> On Thu, Oct 27, 2016 at 9:34 PM, Bryan Bende <bbende@gmail.com>
>>>>>>>> >> wrote:
>>>>>>>> >>> Hello,
>>>>>>>> >>>
>>>>>>>> >>> Are you running with all of the default settings?
>>>>>>>> >>>
>>>>>>>> >>> If so you would probably want to try increasing
the memory
>>>>>>>> >>> settings in
>>>>>>>> >>> conf/bootstrap.conf.
>>>>>>>> >>>
>>>>>>>> >>> They default to 512mb, you may want to try bumping
it up to
>>>>>>>> >>> 1024mb.
>>>>>>>> >>>
>>>>>>>> >>> -Bryan
>>>>>>>> >>>
>>>>>>>> >>> On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr <gop.krr@gmail.com>
>>>>>>>> >>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Hi All,
>>>>>>>> >>>>
>>>>>>>> >>>> I have very simple data flow, where I need
to move s3 data from
>>>>>>>> >>>> one bucket
>>>>>>>> >>>> in one account to another bucket under another
account. I have
>>>>>>>> >>>> attached my
>>>>>>>> >>>> processor configuration.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> 2016-10-27 20:09:57,626 ERROR [Flow Service
Tasks Thread-2]
>>>>>>>> >>>> org.apache.nifi.NiFi An Unknown Error Occurred
in Thread
>>>>>>>> >>>> Thread[Flow Service
>>>>>>>> >>>> Tasks Thread-2,5,main]: java.lang.OutOfMemoryError:
Java heap
>>>>>>>> >>>> space
>>>>>>>> >>>>
>>>>>>>> >>>> I am very new to NiFi and trying ot get
few of the use cases
>>>>>>>> >>>> going. I need
>>>>>>>> >>>> help from the community.
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks again
>>>>>>>> >>>>
>>>>>>>> >>>> Rai
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message