spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SerafĂ­n Sedano Arenas <>
Subject Spark Streaming and periodic broadcast
Date Mon, 02 Nov 2015 13:16:00 GMT
Hi all,

I have a long lived Spark Streaming cluster feeding from several Kinesis
streams. There is some data that must be obtained from another data source
to be used in one of the steps on the workers.

My understanding is that the best way to achieve this is by broadcasting
that data. Is that correct? How can I periodically broadcast this info? Is
it a good idea to use StreamingListener.onBatchStarted to achieve this?

Best regards,

View raw message