spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saravanan Subramanian <>
Subject Maximum Size of Reference Look Up Table in Spark
Date Thu, 14 Jul 2016 21:32:09 GMT
Hello All,
I am in the middle of designing real time data enhancement services using spark streaming.
 As part of this, I have to look up some reference data while processing the incoming stream.
I have below questions:
1) what is the maximum size of look up table / variable can be stored as Broadcast variable
()2) What is the impact of cluster performance, if I store a 10GB data in broadcast variable
Any suggestions and thoughts are welcome.
Thanks,Saravanan S.
View raw message