spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Ortiz Fernández <>
Subject Refresh broadcast variable when it isn't the value.
Date Sun, 19 Aug 2018 20:41:41 GMT

I want to set data in a broadcast (Map<key,value>) variable in Spark.
Sometimes there are new data so I have to update/refresh the values but I'm
not sure how I could do this.

My idea is to use accumulators like a flag when a cache error occurs, in
this point I could read the data and reload the broadcast variable for the
next microbach and finally reset the accumulator to 0.
I don't know if there are a better solution or others ideas to do this.

Has anyone faced this problem?

View raw message