nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From saloni udani <saloniudani...@gmail.com>
Subject Re: How to redirect invalid lines while parsing record with CSVReader?
Date Mon, 13 Aug 2018 13:12:57 GMT
Thanks Matt.

Validaterecord would be useful. I am using SplitRecord to create batches
for the large input files for efficiency of processing.

On Mon, Aug 13, 2018 at 6:02 PM, Matt Burgess <mattyb149@apache.org> wrote:

> You can use ValidateRecord (with a CSVReader and JSONRecordSetWriter,
> and another "invalid CSV Reader" for invalid records) for that, then
> SplitRecord if you need it. However if you can describe your
> downstream flow, perhaps we can help you avoid the need to split the
> records at all (unless you are using a downstream processor that only
> handles one record/JSON object at a time).
>
> Regards,
> Matt
>
> On Mon, Aug 13, 2018 at 7:41 AM saloni udani <saloniudani.23@gmail.com>
> wrote:
> >
> > Hi
> >
> > I have a bunch of CSV files which I need to convert to JSON.
> > My current flow is
> >
> > GetFile --> SplitRecord (CSVReader and JSONRecordSetWriter)
> >
> >
> > The issue is if the csv contains an invalid records then the file gets
> stuck in the queue. Is there a way to discard the invalid CSV lines
> encountered to failure relationship? Documentation says that only those
> records will be routed to failure which fails csv-->json conversion. But
> here I want even the invalid csv records to be routed to failure.
> >
> >
> > Thanks
> > Saloni Udani
>

Mime
View raw message