spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe <>
Subject question about barrier execution mode in Spark 2.4.0
Date Mon, 12 Nov 2018 15:33:35 GMT
I was reading Spark 2.4.0 release docs and I'd like to find out more 
about barrier execution mode.
In particular I'd like to know what happens when number of partitions 
exceeds number of nodes (which I think is allowed, Spark tuning doc 
mentions that)?
Does Spark guarantee that all tasks process all partitions 
simultaneously? If not then how does barrier mode handle partitions that 
are waiting to be processed?
If there are partitions waiting to be processed then I don't think it's 
possible to send all data from given stage to a DL process, even when 
using barrier mode?
Thanks a lot,


To unsubscribe e-mail:

View raw message