samoa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SAMOA-65) Apache Kafka integration components for SAMOA
Date Wed, 07 Jun 2017 11:17:18 GMT

    [ https://issues.apache.org/jira/browse/SAMOA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040714#comment-16040714
] 

ASF GitHub Bot commented on SAMOA-65:
-------------------------------------

GitHub user pwawrzyniak opened a pull request:

    https://github.com/apache/incubator-samoa/pull/65

    SAMOA-65 AVRO serializer/deserializer dedicated for Kafka components

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pwawrzyniak/incubator-samoa SAMOA-65-AVRO

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-samoa/pull/65.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #65
    
----
commit 0e90fe8c107cd90a7baf16b5d4f334d1650a0afb
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-14T16:43:25Z

    Initial structure of Kafka components.
    Initital code for Kafka Consumer

commit 2168400f86189605b06fb531511235733bf3b6ca
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-17T10:05:14Z

    Code for KafkaEntranceProcessor (consuming messages from Kafka)

commit cf2ff369989fa3560f1570dfaf32bcb746d51c25
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-17T10:54:47Z

    Updated comments

commit 26d86da6ebdd53e440e8b1e65f5ba5ed9f5f609e
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-17T11:09:52Z

    Sample serializer/deserializer for JSON and InstanceContentEvent
    Updates in comments

commit 45b3546f9c525af385f646447a09d3683b70937d
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-17T14:40:25Z

    KafkaDestinationProcessor implementation (sending msg to Kafka

commit 2e6d502a2909731cb9859ba3403407e6a33a2fc2
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-03-24T13:34:49Z

    Tests for kafkaUtils and KafkaEntranceProcessor, minor changes in classes

commit 208febc0524ddc328ae1a785f0642650173f7101
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-04-11T14:44:30Z

    Tests for KafkaDestinationProcessor, minor changes in classes

commit ff6b84dc5403c102388646a31f120c11ee416063
Author: Jakub Jankowski <jakub.jankowski@orange.com>
Date:   2017-04-28T15:08:29Z

    Added Kafka Avro serializer with unit tests. Added Kafka task with unit test

commit 7997fd538c258e8844cbd1a81d4c6c43d8788aa4
Author: Jakub Jankowski <jakub.jankowski@orange.com>
Date:   2017-05-01T09:23:00Z

    Merge branch 'SAMOA-65' of https://github.com/pwawrzyniak/incubator-samoa into SAMOA-65

commit ae7889fce1f52afdaa6d7e6080cff54dd1b01b9b
Author: Jakub Jankowski <jakub.jankowski@orange.com>
Date:   2017-05-10T12:37:25Z

    Fixing avro deserialization

commit 107c4c5f72fcacc38d02bfc7cd1552a1593cdd5a
Author: Jakub Jankowski <jakub.jankowski@orange.com>
Date:   2017-05-10T12:45:16Z

    Fixing avro deserialization

commit cde0642ccc768f7492e8c5811b2f1232e58acec4
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-05-12T15:17:27Z

    Rebuild of Consumer-related classes (added separate thread to read from Kafka, blocking
until sth received)
    Major cleanup in tests

commit f20301a4c64c9487fdbcff17829b6a8018e9a00b
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-05-12T15:22:54Z

    Minor change in tests

commit b4ee754c54ac5be51f9846704947e7c19c186e73
Author: Jakub Jankowski <jakub.jankowski@orange.com>
Date:   2017-05-16T12:11:36Z

    Fixing avro deserializer (using ReflectDatumReader) to be able to read arrays in InstanceData
objects

commit d636ba90851d8424a6d3e651d80c87d0a8d7394e
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-05-16T13:00:33Z

    Changes in JSON mapper

commit 57f80a5bf7d83c31d517a7d98741dd7457a4c84c
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-06-07T10:21:08Z

    repo cleanup, split code into 3 branches

commit 39ba256125a940cf8c4a69f8bbf62881d1fc6715
Author: pwawrzyniak <piotr@wawrzyniak.org.pl>
Date:   2017-06-07T11:00:54Z

    AVRO mapper with test

----


> Apache Kafka integration components for SAMOA
> ---------------------------------------------
>
>                 Key: SAMOA-65
>                 URL: https://issues.apache.org/jira/browse/SAMOA-65
>             Project: SAMOA
>          Issue Type: New Feature
>          Components: SAMOA-API, SAMOA-Instances
>            Reporter: Piotr Wawrzyniak
>              Labels: kafka, sink, source, streaming
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> As of now Apache SAMOA includes no integration components for Apache Kafka, meaning in
particular no possibility to read data coming from Kafka and write data with prediction results
back to Kafka.
> The key assumptions for the development of Kafka-related components are as follows:
> 1)	develop support for input data stream arriving to Apache Samoa via Apache Kafka
> 2)	develop support for output data stream produced by Apache Samoa, including the results
of stream mining and forwarded to Apache Kafka to be provided in this way to other modules
consuming the stream.
> This makes the goal of this issue is to create the following components:
> 1)	KafkaEntranceProcessor in samoa-api. This entrance processor will be able to accept
incoming Kafka stream. It will require KafkaDeserializer interface implementation to be delivered.
The role of Deserializer would be to translate incoming Apache Kafka messages into implementation
of Instance interface of SAMOA.
> 2)	KafkaDestinationProcessor in samoa-api. Similarly to the KafkaEntranceProcessor, this
processor would require KafkaSerializer interface implementation to be delivered. The role
of Serializer would be to create a Kafka message from the underlying Instance class.
> 3)	KafkaStream, as the extension to existing streams (e.g. InstanceStream), would take
similar role to other streams, and will provide the control over Instances flows in the entire
topology.
> Moreover, the following assumptions are considered:
> 1)	Components would be implemented with the use of most up-to-date version of Apache
Kafka, i.e. 0.10
> 2)	Samples of aforementioned Serializer and Deserializer would be delivered, both supporting
AVRO and JSON serialization of Instance objects.
> 3)	Sample testing classes providing reference use of Kafka source and destination would
be included in the project as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message