nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: Adding schema inference
Date Thu, 23 May 2019 22:25:00 GMT
I think it'd be great to create one-field records for each level that
doesn't exist, for things like LookupRecord and/or UpdateRecord (from
the aforementioned Jira) I had envisioned the same thing. It would
make it easier for other such processors to enrich/enlarge a
newly-existing record level, basically an Upsert pattern (or mkdir -p
:)

Regards,
Matt

On Tue, May 21, 2019 at 2:32 PM Mike Thomsen <mikerthomsen@gmail.com> wrote:
>
> All things considered, it doesn't seem like that much work. My main concern
> is how to handle it if a user does something like nest the data deep into
> multiple schema levels that don't exist. How should that be handled? An
> exception or generating empty record declarations that have one field?
>
> On Tue, May 21, 2019 at 2:07 PM Matt Burgess <mattyb149@apache.org> wrote:
>
> > Mike,
> >
> > Check AbstractRouteRecord, it uses the read-first-get-schema-read-rest
> > pattern. However for that snippet, it is the RecordReader that is
> > possibly updating the schema (currently the only thing that does this
> > is schema inference), then the RecordSetWriter is created using the
> > (possibly updated) schema. For your PR, you might need to update the
> > schema manually and then pass that into the
> > RecordSetWriterFactory.getSchema() call.
> >
> > Now that I think of it, you might be the first to do this
> > automatic-schema-update-on-write thing (if you choose to do so). I
> > thought NIFI-5524 [1] had been implemented but apparently not.
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5524
> >
> >
> > On Tue, May 21, 2019 at 10:08 AM Mike Thomsen <mikerthomsen@gmail.com>
> > wrote:
> > >
> > > Matt left this suggestion:
> > >
> > > https://github.com/apache/nifi/pull/3231#discussion_r285693972
> > >
> > > What would be a good example of that pattern if I wanted to update that
> > PR
> > > and document the process for others?
> > >
> > > Thanks,
> > >
> > > Mike
> >

Mime
View raw message