beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <>
Subject [jira] [Commented] (BEAM-1581) JSON sources and sinks
Date Wed, 15 Mar 2017 18:14:42 GMT


Aviem Zur commented on BEAM-1581:

I think we should avoid exposing a contract to the user which promises writing JSONs but accepts
This is a loose contract which will leave JSON validity up to the user. If the user does not
create valid JSON Strings errors can occur.
Errors which might be detected very late in the process, possibly only upon an attempt to
consume the data in another process (which may belong to a different user as JSON is often
used for integration).

I think {{JsonIO}} itself should not be abstract and should not implement any interface (same
as other {{XxxIO}} classes that exist today).
{{JsonIO}} should be the enclosing class of abstract {{JsonSink}} and {{JsonSource}} classes
which extend the existing abstract {{FileBasedSink}} and {{FileBasedSource}}.

These will be implemented by concrete classes such as {{JacksonSink}} and {{JacksonSource}}.
All common JSON file logic regarding how the file should be constructed (As [~jkff] mentioned
this should be better defined) will be in the abstract sink and source, including all file
writing/reading related code (Inherited from {{FileBasedSink}} and {{FileBasedSource}}).

So none of these are actually an abstract {{PTransform}}. The {{PTransform}} the user will
end up using here are {{Write}} and {{Read}} from the {{io}} package.

{code} // for writing
Read.from(JacksonSource.of(...)) // for reading

> JSON sources and sinks
> ----------------------
>                 Key: BEAM-1581
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Aviem Zur
>            Assignee: Aviem Zur
> JSON source and sink to read/write JSON files.
> Similarly to {{XmlSource}}/{{XmlSink}}, these be a {{JsonSource}}/{{JonSink}} which are
a {{FileBaseSource}}/{{FileBasedSink}}.
> Consider using methods/code (or refactor these) found in {{AsJsons}} and {{ParseJsons}}

This message was sent by Atlassian JIRA

View raw message