spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvio Fiorito <>
Subject Re: Java REST custom receiver
Date Thu, 01 Oct 2015 21:40:15 GMT
When you say “receive messages” you mean acting as a REST endpoint, right? If so, it might
be better to use JMS (or Kafka) option for a few reasons:

The receiver will be deployed to any of the available executors, so your REST clients will
need to be made aware of the IP where the receiver is running (or you have some other proxy
doing that). In the event the app restarts or the receiver dies and is restarted the executor
the receiver runs on may change. Now there’s an option to specify a preference for what
hosts to run the receiver on by overriding “preferredLocation” if you still decide to
go that route.

There’s a potential loss of messages when you are deploying or if your receiver or streaming
app dies. You would now have to worry about managing those messages on the client’s side
during a deployment or outage.

There’s also a security issue (may not be relevant in your case) in that you would be exposing
your executors and cluster in order to receive these messages. Worst case would be if the
clients are outside your enterprise network.

My preference would be to use JMS or Kafka or some other messaging systems as a buffer between
the two systems.


From: Pavol Loffay
Date: Thursday, October 1, 2015 at 3:58 PM
To: "<>"
Subject: Java REST custom receiver


is it possible to implement custom receiver [1] which will receive messages from REST calls?

As REST classes in Java(jax-rs) are defined declarative and instantiated by application server
I'm not use if it is possible.

I have tried to implement custom receiver which is inject to REST class via CDI and then is
passed  to JavaStreamingContext. But there is problem receiver instance in REST class is not
the same as in SparkContext (supervisor is null).

Could anyone help me with this? I'm also using JMS in my app so data from REST can be sent
to JMS and then received by spark JMS receiver. But I think there should be more straight
forward solution.

View raw message