gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karol Brejna <>
Subject Sink/source tickets
Date Wed, 04 May 2016 17:34:43 GMT
We have a series of jira tickets regarding Gearpump sinks/sources: - Compatibility
layer/adapter for Apache Storm - Create MQTT source/sink - Gearpump Redis
Integration - Provide non-persistent
Sink Task so that examples like word count can materialize Sum results
within the Client - Source task that emits
messages per a schedule (interval or otherwise) should be provided - Add parquet datasource
and datasink connectors - Apache Cassandra

We also had a ticket for 'Add a HDFS Sink with secutiry' ( - I am not sure as for
the outcome of this one.

Most of them consider the medium (MQTT, Redis, Casandra, ...). Other talk
about the source mechanics (scheduled/repetative source).

I'd like to discuss the order in wich we plan implementation for them.

In my opinion Redis an MQTT (GEARPUMP-106, GEARPUMP-115) seems most
important to have.
Redis is well known and widely used. MQTT is a de facto standard in IoT

Then I would like to have HDFS sink (if we didn't merged this already).

Non-persistent datasink could be very useful for examples/demo purposes.
(Imagine we have capped collection that the application can send messages
to, kind of application console. In the dashboard there could be a section
that presents lates 'console' messages. This way a user could "watch" the
application progress. Especially if he/she doesn't have access to the
backend - as it happens often in YARN mode. But this is a topic for
dedicated discussion, I think.)

On the other hand, if we start working on GEARPUMP-116, we'd probably
quickly have Redis, JMS, AMQP sources (adapted from Storm)

Please, let me know what do you think.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message