nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff <j.007...@gmail.com>
Subject NiFi flow provides 0 output on large files
Date Fri, 25 Sep 2015 12:22:04 GMT
Hi Adam,


I have a flow that does the following;

GetFile > DDA_Processor > ConvertJSONToAvro > UpdateAttribute > PutFile

My source file has 182897 rows at 1001 bytes per row.  If I do any number of rows under ~15000
an output file is created.  Anything over, the GetFile and DDA_Processor shows data movement
but the no other downstream processor shows movement.  

I confirmed that it is not a data problem by processing a 10,000 row file successfully, then
concatenating 10,000 rows into one file twice.  

Thanks for your insight.

Jeff
 


> On Sep 24, 2015, at 8:40 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
> 
> Jeff,
> 
> This seems to be a bit different as the processor is showing data as having been written
and there is a listing of one FlowFile of 381 MB being transferred out from the processor.
 Could you provide additional information as to how data is not being sent out in the manner
anticipated?  If you can track the issue down more, let us know.  May be helpful to create
another message to help us track the issues separately as we work through them.
> 
> Thanks!
> 
> Adam,
> 
> Found a sizable JSON file to work against and have been doing some initial exploration.
 With the large files, it certainly is a nontrivial process.  At cursory inspection, a good
portion of processing seems to be spent on validation.  There are some ways to tweak the strictness
of this with the supporting library, but will have to dive in a bit more.
> 
> 
> 
> On Thu, Sep 24, 2015 at 8:14 PM, Jeff <j.007ba7@gmail.com <mailto:j.007ba7@gmail.com>>
wrote:
> 
> 
> 
> I’m having a very similar problem.  The process picks up the file, a custom processor
does it’s thing but no data is sent out.
> 
> <unknown.gif>
> 
> 


Mime
View raw message