samoa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishal Karande (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SAMOA-40) Add Kafka stream reader modules to consume data from Kafka framework
Date Tue, 11 Aug 2015 00:09:45 GMT

    [ https://issues.apache.org/jira/browse/SAMOA-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680985#comment-14680985
] 

Vishal Karande commented on SAMOA-40:
-------------------------------------

Hi @gdfm 

Here are the steps to test SAMOA with KAFKA. Let me know if you have any issues to run it.

A] Set up KAFKA:
Step 1: Download the code

Download the 0.8.2.0 release and un-tar it.
> tar -xzf kafka_2.10-0.8.2.0.tgz
> cd kafka_2.10-0.8.2.0

Step 2: Start the server
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have
one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node
ZooKeeper instance.

> bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...
Now start the Kafka server:
> bin/kafka-server-start.sh config/server.properties
[01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)

Step 3: Create a topic

Let's create a topic named "test" with a single partition and only one replica:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions
1 --topic test

We can now see that topic if we run the list topic command:
> bin/kafka-topics.sh --list --zookeeper localhost:2181
test

B] Add data into KAFKA:
Use following code to add data into Kafka.
https://github.com/karande/kafka-producer-file

KafkaFileProducer performs following tasks:

1) Read data from dataset/Processed_subject101.dat file
2) For each line in the file it creates a message of type (Key,Value) where key is line number
and value is actual line

Following default variables can be changed to add data for your choice into kafka.

Default Topic name: test

Default file used: dataset/Processed_subject101.dat

D] Validate data present in Kafka:

Run KafkaFileProducer and verify that data is loaded into Kafka using following command

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
30.25,2.80333,9.5201,4.86089,2.66338,10.2871,5.16317,-0.0774208,-0.117665,0.105727,-9.46798,-54.505,-33.3918,1,0,0,0,33.625,-0.210245,9.76588,-1.38721,-0.335559,9.77131,-1.22859,0.0896249,-0.0378112,-0.037197,0.276133,-59.7344,16.1858,1,0,0,0,33,9.84219,0.511835,-0.651625,9.82576,0.682181,-0.182073,-0.00553086,-0.0257234,0.00726291,-131.076,10.6035,-6.78933,1,0,0,0,5
30.25,2.62452,10.5845,4.9316,2.42503,10.5305,5.25349,0.641135,-0.0649068,0.133527,-8.6943,-54.1086,-33.8593,1,0,0,0,33.625,-0.13506,9.80304,-1.42469,-0.230575,9.68045,-1.18415,0.0136997,-0.0642584,-0.034991,0.381789,-59.5145,16.1877,1,0,0,0,33,9.76984,0.359471,-0.535653,9.84071,0.667041,-0.197239,-0.0178811,0.0137628,0.0107517,-130.856,10.6022,-7.28625,1,0,0,0,5
30.25,2.36427,10.6255,5.04302,2.07469,9.67259,5.67855,1.01104,-0.0771391,0.180579,-8.80487,-53.8846,-33.8556,1,0,0,0,33.625,-0.0571406,9.5774,-1.3451,-0.140908,9.57458,-1.16989,0.0564232,-0.0479727,-0.0264942,0.254743,-59.1773,16.4314,1,0,0,0,33,9.80826,0.549077,-0.536436,9.82569,0.667038,-0.197159,-0.0389267,-0.00497049,0.0120961,-131.191,10.3966,-7.53326,1,0,0,0,5



C] Test KafkaStream for SAMOA:

> bin/samoa local target/SAMOA-Local-0.3.0-incubating-SNAPSHOT.jar "PrequentialEvaluation
-d /tmp/dump.csv -l (org.apache.samoa.learners.classifiers.trees.VerticalHoeffdingTree -p
4) -s (org.apache.samoa.streams.kafka.KafkaStream -r 20 -t test -k 7) -i 100000 -f 10000 -w
0"

evaluation instances,classified instances,classifications correct (percent),Kappa Statistic
(percent),Kappa Temporal Statistic (percent)
10000.0,10000.0,99.99,0.0,0.0
20000.0,20000.0,99.995,0.0,0.0
30000.0,30000.0,96.87666666666667,78.47164931038166,-46750.00000000516
40000.0,40000.0,91.7,79.54710687473718,-165900.00000001822
50000.0,50000.0,92.956,85.62667898764178,-175999.99999982378
60000.0,60000.0,84.89166666666667,74.47144854419265,-302066.6666667
70000.0,70000.0,83.47857142857143,74.5994186500813,-385400.0000001852
80000.0,80000.0,80.72375000000001,72.55641649422608,-385425.0000000423
90000.0,90000.0,79.56444444444445,72.23787238668397,-459700.00000043336
100000.0,100000.0,74.965,67.09652284860583,-500600.000000055

where,
- r  20 :read 20 messages in single read from Kafka
-t  test :read from topic name "test"
-k 7: Total number of classes in the data
-w 0: No delay between instances read
default separator comma is used while parsing values for evaluation


> Add Kafka stream reader modules to consume data from Kafka framework
> --------------------------------------------------------------------
>
>                 Key: SAMOA-40
>                 URL: https://issues.apache.org/jira/browse/SAMOA-40
>             Project: SAMOA
>          Issue Type: Task
>          Components: Infrastructure, SAMOA-API
>         Environment: OS X Version 10.10.3
>            Reporter: Vishal Karande
>            Priority: Minor
>              Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Apache SAMOA is designed to process streaming data and develop streaming machine learning
> algorithm. Currently, SAMOA framework supports stream data read from Arff files only.
> Thus, while using SAMOA as a streaming machine learning component in real time use-cases,
> writing and reading data from files is slow and inefficient.
> A single Kafka broker can handle hundreds of megabytes of reads and writes per second

> from thousands of clients. The ability to read data directly from Apache Kafka into SAMOA
will 
> not only improve performance but also make SAMOA pluggable to many real time machine
> learning use cases such as Internet of Things(IoT).
> GOAL:
> Add code that enables SAMOA to read data from Apache Kafka as a stream data.
> Kafka stream reader supports following different options for streaming:
> a) Topic selection - Kafka topic to read data
> b) Partition selection - Kafka partition to read data
> c) Batching - Number of data instances read from Kafka in one read request to Kafka
> d) Configuration options - Kafka port number, seed information, time delay between two
read requests
> Components:
> KafkaReader - Consists for APIs to read data from Kafka
> KafkaStream - Stream source for SAMOA providing data read from Kafka
> Dependencies for Kafka are added in pom.xml for in samoa-api component. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message