nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: convert avro schema to another schema
Date Wed, 11 Oct 2017 16:18:33 GMT
If you know the input and output schemas, you should be able to use
UpdateRecord for this. You would have a user-defined property for each
output field (including flattened field names), whose value would be
the RecordPath to the field in the map. I believe for any fields that
are in the input schema but not the output schema, they are ignored
(but don't quote me on this, give it a try ;) so you could effectively
"convert" the record by updating outgoing fields with the values from
the nested fields in the input record.

Regards,
Matt

On Wed, Oct 11, 2017 at 12:10 PM, Bryan Bende <bbende@gmail.com> wrote:
> Casting string to integer wouldn't be a problem, but flattening a map would be.
>
> I think there was a previous thread where a FlattenRecord processor
> was discussed, and I believe there was agreement that it would be
> useful, but it just hasn't been developed yet.
>
> If you had FlattenRecord then you would probably bring the data in
> from ConsumeKafkaRecord keeping the original schema (using a writer
> with "inherit schema from reader") and then later in the flow do a
> FlattenRecord.
>
> As a current option, you might be able to do the flattening yourself
> with an ExecuteScript processor, or maybe a scripted record writer
> that did the flattening on the way out.
>
>
> On Wed, Oct 11, 2017 at 11:22 AM, Chris Herssens
> <chris.herssens@gmail.com> wrote:
>> Hello Bryan,
>>
>> In my case the schemas are not compatible, In Schema A I have a map type and
>> I want to flatten this map. Also I want to cast string to integer.
>>
>> Chris
>>
>> On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <bbende@gmail.com> wrote:
>>>
>>> Chris,
>>>
>>> Any processor that uses a record reader and record writer can
>>> inherently do schema conversion by using schema A for the reader, and
>>> schema B for the writer, assuming the schemas are compatible.
>>>
>>> Compatible in this sense would mean one of the following...
>>>
>>> - Schema B has the same field names as schema A, but with some of the
>>> filed types being different
>>> - Schema B has a subset of the fields in schema A, possibly some
>>> changing type as above
>>> - Schema B has additional fields and they have default values since
>>> they won't exist in the records coming from schema A
>>>
>>> If you have Avro data in Kafka that already has the schema embedded in
>>> it, then you can use ConsumeKafkaRecord with an AvroReader and set the
>>> Schema Access Strategy to "Embedded Avro", and then use a
>>> AvroRecordSetWriter and set the Schema Access Strategy to one of the
>>> other options like Schema Name (which needs a schema registry) or
>>> Schema Text which allows you to enter in a full schema.
>>>
>>> You could also do the same thing anywhere else in your flow using
>>> ConvertRecord.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens
>>> <chris.herssens@gmail.com> wrote:
>>> > Hello All,
>>> >
>>> > I would like to convert an avro schema to another avro schema.   Since
>>> > Nifi
>>> > reads the avro data from kafka, I can't use the ConvertAvroSchema
>>> > processor.
>>> > Which processor can I use ?
>>> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes how
>>> > do
>>> > we specify the dynamic properties ?  If possible can you give me an
>>> > example
>>> > ?
>>> >
>>> > Regards,
>>> >
>>> > Chris
>>
>>

Mime
View raw message