nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: OutOfMemoryError from ListSFTP
Date Wed, 01 Jun 2016 13:43:16 GMT
Unfortunately, the library that we use exposes very few options to us. The full listing will
be done each time,
as we don't really have the ability to filter what was returned. I would recommend going ahead
and increasing
the heap size and see if that gives you what you need.

Thanks
-Mark


> On Jun 1, 2016, at 9:28 AM, Joe Witt <joe.witt@gmail.com> wrote:
> 
> Huagen,
> 
> You can possibly avoid it by giving NiFi a large heap size.  The
> downside is that there is a good library I am aware of which will let
> us handle large remote listing as nicely as we'd like.  It does tend
> to be a 'send small request', 'wait for a large reply', 'then process
> large reply' pattern.  We used to have this too with ListFile or its
> equivalent but then Java exposed a nicer API for us to use.  I do not
> *think* we have that available at this time with SFTP.
> 
> So yes for now you might be happy with updating the heap size in
> conf/boostrap.conf.
> 
> The processor will be smart about not constantly going back for a
> listing while it knows it has a lot of work to do but the first
> listing can be prohibitively expensive.  We definitely should find a
> way to sort that out better if my memory on this is correct.
> 
> Mark Payne; Can you confirm or correct?
> 
> Thanks
> Joe
> 
> On Wed, Jun 1, 2016 at 9:24 AM, Huagen peng <huagen.peng@gmail.com> wrote:
>> Hi,
>> 
>> I tried to use the ListSFTP processor on a server with tens of thousands of files
and the processor tried for a longtime and emit an OutOfMemoryError.  Can I fix this error
by modifying the JVM settings in the conf/bootstrap.conf file?
>> 
>> Thanks,
>> 
>> Huagen


Mime
View raw message