nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: QueryDatabaseTable - Schema
Date Tue, 12 Sep 2017 01:05:38 GMT
Hi Uwe,

I had a similar expectation when I was using QueryDatabaseTable or any
other processor creating Avro FlowFile which has its schema embedded,
combining new record reader/writer controllers.

Now, NiFi has "Inherit Record Schema" option as "Schema Access
Strategy" of RecordWriter, already merged in master branch.
https://issues.apache.org/jira/browse/NIFI-3921

I was able to reuse the Avro schema at subsequent flow using "Inherit
Record Schema", it's really useful. You can construct a flow like
below:

- QueryDatabaseTable
  - outputs FlowFile with Avro schema embedded
- ConvertRecord
  - AvroReader:
    - "Schema Access Strategy" = "Use Embedded Avro Schema"
  - CSVRecordSetWriter:
    - "Schema Access Strategy" = "Inherit Record Schema"
    - "Schema Write Strategy" = "Set 'avro.schema' Attribute"

This way, you don't have to have the schema in registry, and result
CSV FlowFile has 'avro.schema' attribute inheriting the one created by
QueryDatabaseTable.

Hope this helps.

Thanks,
Koji

On Tue, Sep 12, 2017 at 5:02 AM, Uwe Geercken <uwe.geercken@web.de> wrote:
> Hello,
>
> I was wondering why if the QueryDatabaseTable processor creates internally
> an Avro schema, why is this schema not available as an attribute or saved to
> the registry?
>
> If it would, then one could reuse the schema. E.g. if I use the
> ConvertRecord processor and I specify an AvroReader as RecordReader, then
> this Reader will take the schema from the flowfile the QueryDatabaseTable
> processor creates. But the RecordWriter in the ConvertRecord - in my example
> a CSVRecordSetWriter requires the schema as an attribute or as a reference
> to the schema registry.
>
> I can see there is an ExtractAvroSchema processor but I don'see there is a
> way of combining the metadata into e.g. the ConvertRecord processor.
>
> Any help or ideas?
>
> Rgds,
>
> Uwe

Mime
View raw message