samoa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwawrzyniak <>
Subject [GitHub] incubator-samoa issue #64: SAMOA-65 JSON serializer/deserializer dedicated f...
Date Tue, 20 Jun 2017 10:00:35 GMT
Github user pwawrzyniak commented on the issue:
    Thank you nicolas-kourtellis for your feedback!
    Regarding your comments, I removed copyright notices from the code, fixed and clened-up
the code according to your comments.
    Regarding AVRO code provided within SAMOA-65, the first difference between avro loader
and our kafka avro mapper is that avro loader works with json files containing schema header
and payload data (json or binary). In our case avro mapper works with byte stream received
from kafka and the schema is defined in separate file. The other difference is that avro loader
produces Instance object while Kafka Mapper was designed to work with InstanceContentEvent.
Moreover kafka avro mapper serializes whole InstanceContentEvent object while avro loader
reads data file, creates avro structure and, based on that, builds Instance object.It should
be possible to use the concept from avro loader for processing kafka but I suppose it would
require implementation of new generic loader, instead of using the old one. Other thing is
that we need to have two way serialization and avro loader is used only for reading data,
not writing.
    And regarding JSON parser, as of now it is prepared to parse messages coming from Apache
Kafka in "one-by-one" style, serializing InstanceContentEvent class. It could be potenatially
used to parse file (in line-by-line manner for example), but I believe currently, as the mapper
accepts byte array as the input, it can be easily used to this task (i.e. as the parser when
reading data from text/json file).

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message