nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: [EXT] ConvertCSVToAvro vs CSVReader - Value Delimiter
Date Mon, 25 Sep 2017 00:57:46 GMT
Thanks Arun and Peter.  Getting that resolved will be nice.  The
performance difference of the record reader/writer approach in all
this is pretty fantastic so the more we can do to iron out these sorts
of edges the better.  Thanks!

On Sun, Sep 24, 2017 at 8:56 PM, Peter Wicks (pwicks) <pwicks@micron.com> wrote:
> Arun,
>
> I'm also using Ctrl+A as a delimiter and had the same problem.  I haven't had time to
write up a PR but it looked like a pretty easy fix to me too.
>
> I can't merge the change if you submit it, but I'd be happy to review it.
>
> --Peter
>
> -----Original Message-----
> From: Arun Manivannan [mailto:arun@arunma.com]
> Sent: Sunday, September 24, 2017 11:17 PM
> To: Dev@nifi.apache.org
> Subject: [EXT] ConvertCSVToAvro vs CSVReader - Value Delimiter
>
> Hi,
>
> The ConvertCSVToAvro processor have been having performance issues while processing files
which are more than a GB and I was suggested to use the ConvertRecord that leverages the RecordReader
and Writer. Did some tests and they do perform well.
>
> Strangely, the CSVReader doesn't accept unicode character as the value delimiter - Control
A  (\u0001) character is the delimiter of my CSV.
>
> Did some analysis and I see that a minor change needs to be made on the CSVUtils to unescape
the delimiter, like what ConvertCSVToAvro does and also modify the SingleCharacterValidator.
>
> Please let me know if you believe this isn't an issue and there's a workaround for this.
Else, I am more than happy to raise an issue and submit a PR for review.
>
> Best Regards,
> Arun

Mime
View raw message