spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pratyush04 <pratyus...@gmail.com>
Subject Re: End of Stream errors in shuffle
Date Mon, 15 Jan 2018 14:30:03 GMT
Hi Fernando,

There is a limit of 2GB on blocks for shuffle, since you say the job fails
while doing shuffle of 200GB data, it might be due to this.
These links give more idea about this:
http://apache-spark-developers-list.1001551.n3.nabble.com/Re-2GB-limit-for-partitions-td10435.html
https://issues.apache.org/jira/browse/SPARK-5928

Thanks,
Pratyush




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message