spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 徐涛 <xutao_u...@163.com>
Subject Question about query local dirs when fetching HostLocalBlocks
Date Tue, 13 Jul 2021 07:24:04 GMT
Hi Experts,
      When I`m reading spark code in version 3.0.0, when external shuffle service is enabled:
      ShuffleBlockFetcherIterator -> 
            fetchHostLocalBlocks ( there is some logic, when there is no record in cache,
then it need to use hostLocalDirManager.getHostLocalDirs to send message to external shuffle
service) 


      My question is :
      1. As the executors are in the same host, the local dir should be sure to be same. Why
it does so much job here( do some cache and RPC), just to fetch local dirs?
      2. Should there be a config to let user just do the same logic as LocalBlocks?


      Thanks a lot.


Best,
Tao
Mime
View raw message