nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohit" <mohit.j...@open-insights.co.in>
Subject ConvertCSVToAvro taking a lot of time when passing single record as an input.
Date Mon, 02 Apr 2018 12:49:53 GMT
Hi,

 

I'm trying to capture bad records from ConvertCSVToAvro processor. For that,
I'm using two SplitText processors in a row to create chunks and then each
record per flow file.

 

My flow is  - ListFile -> FetchFile -> SplitText(10000 records) ->
SplitText(1 record) -> ConvertCSVToAvro -> *(futher processing)

 

I have a 10 MB file with 15 columns per row and 64000 records. Normal flow
(without SplitText) completes in few seconds. But when I'm using the above
flow, ConvertCSVToAvro processor works drastically slow(45-50 rec/sec).

I'm not able to conclude where I'm doing wrong in the flow. 

 

I'm using Nifi 1.5.0 .

 

Any quick input would be appreciated.

 

 

 

Thanks,

Mohit


Mime
View raw message