spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ji ZHANG <zhangj...@gmail.com>
Subject Join DStream With Other Datasets
Date Sat, 17 Jan 2015 14:05:32 GMT
Hi,

I want to join a DStream with some other dataset, e.g. join a click
stream with a spam ip list. I can think of two possible solutions, one
is use broadcast variable, and the other is use transform operation as
is described in the manual.

But the problem is the spam ip list will be updated outside of the
spark streaming program, so how can it be noticed to reload the list?

For broadcast variables, they are immutable.

For transform operation, is it costly to reload the RDD on every
batch? If it is, and I use RDD.persist(), does it mean I need to
launch a thread to regularly unpersist it so that it can get the
updates?

Any ideas will be appreciated. Thanks.

-- 
Jerry

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message