nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: NiFi flow provides 0 output on large files
Date Fri, 25 Sep 2015 12:30:36 GMT
Jeff,

With regards to:

"Anything over, the GetFile and DDA_Processor shows data movement but the
no other downstream processor shows movement."

Are you referencing downstream processors starting immediately after the
DDA_Processor (ConvertJsonToAvro) or starting immediately after the
ConvertJsonToAvro processor?

In the case of starting immediately after the DDA Processor, as it is a
custom processor, we would need some additional information as to how this
processor is behaving.  In the case of the second condition, if you have
some additional context as to the format of the data that is problematic to
what you are seeing (the effective "schema" of the JSON) would be helpful
in tracking down the issue.

Thanks!
Aldrin

On Fri, Sep 25, 2015 at 8:22 AM, Jeff <j.007ba7@gmail.com> wrote:

> Hi Adam,
>
>
> I have a flow that does the following;
>
> GetFile > DDA_Processor > ConvertJSONToAvro > UpdateAttribute > PutFile
>
> My source file has 182897 rows at 1001 bytes per row.  If I do any number
> of rows under ~15000 an output file is created.  Anything over, the GetFile
> and DDA_Processor shows data movement but the no other downstream processor
> shows movement.
>
> I confirmed that it is not a data problem by processing a 10,000 row file
> successfully, then concatenating 10,000 rows into one file twice.
>
> Thanks for your insight.
>
> Jeff
>
>
>
> On Sep 24, 2015, at 8:40 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
>
> Jeff,
>
> This seems to be a bit different as the processor is showing data as
> having been written and there is a listing of one FlowFile of 381 MB being
> transferred out from the processor.  Could you provide additional
> information as to how data is not being sent out in the manner
> anticipated?  If you can track the issue down more, let us know.  May be
> helpful to create another message to help us track the issues separately as
> we work through them.
>
> Thanks!
>
> Adam,
>
> Found a sizable JSON file to work against and have been doing some initial
> exploration.  With the large files, it certainly is a nontrivial process.
> At cursory inspection, a good portion of processing seems to be spent on
> validation.  There are some ways to tweak the strictness of this with the
> supporting library, but will have to dive in a bit more.
>
>
>
> On Thu, Sep 24, 2015 at 8:14 PM, Jeff <j.007ba7@gmail.com> wrote:
>
>>
>>
>>
>> I’m having a very similar problem.  The process picks up the file, a
>> custom processor does it’s thing but no data is sent out.
>>
>> <unknown.gif>
>>
>>
>>
>

Mime
View raw message