beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <>
Subject [jira] [Commented] (BEAM-1581) JsonIO
Date Sat, 11 Mar 2017 15:43:04 GMT


Aviem Zur commented on BEAM-1581:

Yes, these should definitely be a {{FileBasedSource}}/{{FileBasedSink}}.

I believe the file could either be a JSON array representing the {{PCollection}} or a single
object with one field which is an array representing the {{PCollection}} and other fields
for the user to add metadata (Good for integrations). Both these options should probably be
supported. We should definitely hear from [~eljefe6aa] on this.

We have the {{org.apache.beam.sdk.extensions.jackson}} package which has {{AsJsons}} and {{ParseJsons}}.
IIUC you're suggesting {{JsonIO}} should not call these directly since the user might not
want to use Jackson specifically.
How about we have an abstract {{JsonIO}} with a single implementation (right now) which will
use the Jackson transformations, we can call it {{JacksonIO}}. This will allow extensibility
and new {{JsonIO}} implementations could be introduced in the future.

P.S. {{XmlSource/XmlSink}} do not seem to allow this flexibility and uses JAXB directly and
does not allow the user to change properties of the Marshaller and Unmarshaller.

> JsonIO
> ------
>                 Key: BEAM-1581
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Aviem Zur
>            Assignee: Aviem Zur
> A new IO (with source and sink) which will read/write Json files.
> Similar to {{XmlSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and {{ParseJsons}}

This message was sent by Atlassian JIRA

View raw message