spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: Streaming on different store types
Date Wed, 30 Jul 2014 15:33:33 GMT

I fear you have to write your own transaction logic for it (coordination,e.
.g. via Zookeeper, transaction log, depending on your requirements raft
/paxos etc.).

However, before you embark on this journey question yourself if your
application really needs it and what data load you expect.

Best regards
Le 30 juil. 2014 16:46, "Flavio Pompermaier" <> a écrit

> Hi everybody,
> I have a scenario where I would like to stream data to different
> persistency types (i.e. sql db, graphdb ,hdfs, etc) and perform some
> filtering and trasformation as the the data comes in.
> The problem is to maintain consistency between all datastores (maybe some
> operation could fail) and managing the U and D operations of
> there any suggested solution for that?
> Best,
> FP

View raw message