nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Derek Straka <de...@asterius.io>
Subject CSV record parsing with custom date formats
Date Tue, 13 Feb 2018 22:11:51 GMT
I have a question about the expected behavior of convertSimpleIfPossible in
CSVRecordReader.java (NiFi 1.5.0).

I have a custom CSV file that I am taking to an avro schema using
ValidateRecord.  The schema contains a logical date type and the CSV has
the date in the format MM/DD/YYYY.  I expected to provide the date string
in the controller element for the CSV reader and have everything parse
happily, but it ends up throwing an exception when it tries to parse things
in the avro writer (String->Date).  I don't think I should be blaming the
avro writer because I expected the CSV reader to parse the date for me.

I did a little digging in the CSVRecordReader.java, and I see everything
flows through convertSimpleIfPossible when parsing the data, and each data
type is checked with DataTypeUtils.isCompatibleDataType prior to actually
trying to perform the conversion.

The date string doesn't use the user provided format in the call to
DataTypeUtils.isCompatibleDataType, but instead uses the default for date
types.  The validation ends up failing when it uses the default date string
(YYYY-MM-DD), so it won't use LAZY_DATE_FORMAT as I expected.  Am I totally
off base, or it this unexpected behavior?

Thanks.

-Derek

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message