spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dillon Dukek <>
Subject Re: Unable to broadcast a very large variable
Date Wed, 10 Apr 2019 17:00:27 GMT
You will probably need to do a couple of things. One, you will need to
probably increase the "spark.sql.broadcastTimeout" setting. As well, when
you broadcast a variable it gets replicated once per executor not once per
machine so you will need to increase your executor size and allow more
cores to run per executor. Depending on if you are using pyspark or not,
you will also need to remember that if you are trying to use this large
variable in a python process (RDD functions, UDFs, etc) that that variable
will be transferred to python memory space once per python process that
gets spawned which means that you could ultimately end up with many more
copies of that variable in memory at any given point in time than you may
have intended.

On Wed, Apr 10, 2019 at 9:40 AM V0lleyBallJunki3 <>

> I am using spark.sparkContext.broadcast() to broadcast. Is this even true
> if
> the memory on our machines is 244 Gb a 70 Gb variable can't be broadcasted
> even with high network speed?
> --
> Sent from:
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message