spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <mailinglist...@gmail.com>
Subject Stream which needs to be “joined” with another Stream of “Reference” data.
Date Mon, 03 May 2021 16:36:18 GMT
I would like to develop a Spark Structured Streaming job that reads
messages in a Stream which needs to be “joined” with another Stream of
“Reference” data.

For example, let’s say I’m reading messages from Kafka coming in from (lots
of) IOT devices. This message has a ‘device_id’. We have a DEVICE table on
a relational database. What I need to do is “join” the ‘device_id’ in the
message with the ‘device_id’ on the table to enrich the incoming message.
Somewhere I read that, this can be done by joining two streams. I guess, we
can create a “Stream” that reads the DEVICE table once every hour or so.

Questions:
1) Is this the right way to solve this use case?
2) Should we use a Stateful Stream for reading DEVICE table with State
timeout set to an hour?
3) What would happen while the DEVICE state is getting updated from the
table on the relational database?

Guidance would be greatly appreciated. Thanks.

Mime
View raw message