spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Join DStream With Other Datasets
Date Sun, 18 Jan 2015 09:36:59 GMT
I think that this problem is not Spark-specific since you are simply side
loading some data into memory. Therefore you do not need an answer that
uses Spark.

Simply load the data and then poll for an update each time it is accessed?
Or some reasonable interval? This is just something you write in Java/Scala.
On Jan 17, 2015 2:06 PM, "Ji ZHANG" <> wrote:

> Hi,
> I want to join a DStream with some other dataset, e.g. join a click
> stream with a spam ip list. I can think of two possible solutions, one
> is use broadcast variable, and the other is use transform operation as
> is described in the manual.
> But the problem is the spam ip list will be updated outside of the
> spark streaming program, so how can it be noticed to reload the list?
> For broadcast variables, they are immutable.
> For transform operation, is it costly to reload the RDD on every
> batch? If it is, and I use RDD.persist(), does it mean I need to
> launch a thread to regularly unpersist it so that it can get the
> updates?
> Any ideas will be appreciated. Thanks.
> --
> Jerry
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message