storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navin Ipe <navin....@searchlighthealth.com>
Subject Re: Emit from a Java application and receive in a Bolt of another Java application?
Date Tue, 26 Apr 2016 10:41:56 GMT
Thanks Cody and Nathan. Kafka it is.

(ps: I know sockets aren't a good option because of not knowing which port
of which node to listen to and having to implement one's own queue etc..)

On Tue, Apr 26, 2016 at 4:06 PM, Nathan Leung <ncleung@gmail.com> wrote:

> Kafka is probably better than redis, and definitely better than sockets.
> Other queues like rabbitmq can work too.
>
> Sockets are a terrible choice, but I will leave why as a mental exercise
> for now :).  If you must know I can respond again.
> On Apr 26, 2016 6:27 AM, "Cody Lee" <codylee@wellaware.us> wrote:
>
> 2 options come to mind without knowing what your code does: 1. join the
> topology code bases for one topology 2. use a distributed queue  (if
> something else needs to use this data )
>
>
>
>
> -------- Original message --------
> From: Navin Ipe <navin.ipe@searchlighthealth.com>
> Date: 04/26/2016 4:59 AM (GMT-06:00)
> To: user@storm.apache.org
> Subject: Emit from a Java application and receive in a Bolt of another
> Java application?
>
> Hi,
>
> A colleague created a Bolt that writes data to a MongoDB application.
> I have a Spout that reads that data from MongoDB.
> My colleague's Bolt is in a separate Storm application he built. Mine is
> on a separate storm application I built. Our applications may run on
> different VM's.
>
> My team lead wants my colleague's Bolt to emit data, and wants a Bolt in
> my application to receive that data. So we basically avoid writing to
> MongoDB.
>
> I suggested doing this via Java sockets, RMI or Redis. On hearing this, my
> team lead tells me "You haven't understood Storm yet".
>
> I've been through a lot of the documentation on Storm, and I haven't seen
> any case where Storm can communicate. Except maybe using DRPC
> <http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>:
> http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
> But there
> But there are complaints of DRPC memory leaks and unexpected behaviour:
> https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
>
> So is DRPC the way to go or does Storm have some other method of emitting
> from one topology and receiving it in another topology?
>
> --
> Regards,
> Navin
>
>


-- 
Regards,
Navin

Mime
View raw message