Jean-Baptiste Onofré created BEAM-881:
-----------------------------------------
Summary: Provide a PTransform in IOs providing a "standard" Avro IndexedRecord
Key: BEAM-881
URL: https://issues.apache.org/jira/browse/BEAM-881
Project: Beam
Issue Type: New Feature
Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
Now, each IO is using a different data format. For instance, the {{JmsIO.Read}} provides a
{{PCollection}} of {{JmsRecord}} (and {{JmsIO.Write}} expects also a {{JmsRecord}}), {{KafkaIO.Read}}
provides a {{PCollection}} of {{KafkaRecord}}.
It could appear a bit "complex" for users to manipulate such kind of data format: some users
may expect kind of standard format.
Without modifying the existing IO, we could add a {{PTransform}} (as part of the IO) that
an user can optionally use. This transform will convert the IO data format (let say {{JmsRecord}}
for instance) to a standard Avro {{IndexedRecord}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
|