spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simona Rabinovici-Cohen" <SIM...@il.ibm.com>
Subject reaasign location of partitions
Date Wed, 01 Mar 2017 21:17:34 GMT
Hello,

We have an application in which specific partitions need to be executed on 
specific worker machines. Thus, we would like to assign for each 
partition, the worker machine that will execute that partition. Moreover 
we would like to do it before each stage of the SPARK app, and be able to 
reassign the worker machine prior to every stage.
Currently, we are able to do so before the first stage by overwriting 
getPrefferedLocation. Is there a way to change the preferred location 
before each stage?

Thanks and Regards,

Simona Rabinovici-Cohen



Mime
View raw message