incubator-s4-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mmo...@apache.org
Subject [3/12] git commit: Add documentation for injection
Date Sun, 10 Mar 2013 20:07:10 GMT
Add documentation for injection


Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/bad1e51c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/bad1e51c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/bad1e51c

Branch: refs/heads/dev
Commit: bad1e51c9d123222f9e5920fa7c5d3a9666da776
Parents: cb3c58a
Author: Matthieu Morel <mmorel@apache.org>
Authored: Thu Mar 7 11:50:43 2013 +0100
Committer: Matthieu Morel <mmorel@apache.org>
Committed: Thu Mar 7 11:50:43 2013 +0100

----------------------------------------------------------------------
 website/content/doc/0.6.0/event_injection.md |   37 +++++++++++++++++++++
 website/content/doc/0.6.0/index.md           |    1 +
 2 files changed, 38 insertions(+), 0 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/bad1e51c/website/content/doc/0.6.0/event_injection.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/event_injection.md b/website/content/doc/0.6.0/event_injection.md
new file mode 100644
index 0000000..685ff86
--- /dev/null
+++ b/website/content/doc/0.6.0/event_injection.md
@@ -0,0 +1,37 @@
+---
+title: Inject events into S4
+---
+
+> How do we inject events into S4?
+
+# Problem statement
+There is a data stream somewhere. We want to inject that data into our S4 stream processing
application.
+
+The data stream can be almost anything: user clicks in a web application, alarms in a monitoring
system, stock trade operations, even static data from a database or a file server.
+
+# Requirements
+* We must be able to connect to that stream of data
+* We must be able to chunk the data and isolate events out of it
+
+# Solutions
+Currently S4 does not provide a specific abstraction for a source of event or an external
data stream. This means that the connection to an external data source can be implemented
arbitrarily. Typically, connections to external data sources are initiated in the App object
itself, in the `start()` method. 
+
+## In a single application
+Let us consider a single S4 application. The App class is instantiated on each node (for
instantiating the local topology), and this has to be taken in consideration.
+For instance, if we get the information from the twitter sprinker stream, given that the
stream is provided through a single socket connection, we should connect only from one node.
To ensure that, leader election through ZooKeeper is an option, but a much simpler one is
simply to base this decision on the id of the partition, provided in the `App` class by `getReceiver().getPartitionId()`.
In our example, we would initiate the connection only if we are in partition 0.
+
+~~~
+#!java
+
+
+
+~~~
+
+
+## Using an adapter application
+
+In some cases the above solution is not practical: it introduces some load imbalance between
the partitions, and the extraction of the 
+
+
+
+

http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/bad1e51c/website/content/doc/0.6.0/index.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/index.md b/website/content/doc/0.6.0/index.md
index 27e72b6..b14be0e 100644
--- a/website/content/doc/0.6.0/index.md
+++ b/website/content/doc/0.6.0/index.md
@@ -20,6 +20,7 @@ S4 (Simple Scalable Streaming System) is a general-purpose, distributed,
scalabl
 * [Customize the platform and pass configuration parameters](configuration)
 * Add [application dependencies](application_dependencies)
 * [Dispatch events ](event_dispatch) within an application and between applications
+* [Inject events](event_injection) into S4 applications
 
 ## Running S4
 * [Commands](tools) for creating, running and managing applications


Mime
View raw message