spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rapelly kartheek <kartheek.m...@gmail.com>
Subject [No Subject]
Date Thu, 27 Nov 2014 06:06:57 GMT
Hi,
I've been fiddling with spark/*/storage/blockManagerMasterActor.getPeers()
definition in the context of blockManagerMaster.askDriverWithReply()
sending a request GetPeers().

1) I couldn't understand what the 'selfIndex' is used for?.

2) Also, I tried modifying the 'peers' array by just eliminating some
blockManagerId's and passed the modified one to the tabulate method. The
application gets executed, but I find that the
blockManagerMaster.askDriverWithReply() recieves the sequence of
blockManagerIds that include the ones I have eliminated previously.

For example,

My original 'peers' array contained 5 blockManagerId's: BlockManagerId(2,
s2, 39997, 0), BlockManagerId(1, s4, 35874, 0),BlockManagerId(3, s1, 33738,
0), BlockManagerId(0, s3, 38207, 0), BlockManagerId(<driver>, karthik,
34388, 0).

I modified it to peers1 having 3 blockManagerId's : BlockManagerId(2, s2,
39997, 0), BlockManagerId(1, s4, 35874, 0), BlockManagerId(3, s1, 33738, 0).

Then I passed this modified peers1 array for the sequence conversion:

'Array.tabulate[BlockManagerId](size) { i => peers1((selfIndex + i + 1) %
peers1.length) }.toSeq"

But, finally when the /storage/blockManagerMaster.askDriverWithReply() gets
the result, it contains the blockManagerIds that I have eliminated
purposely.

Can someone please make me understand how this seq[BlockManagerId] is
constructed?
Thank you!

Mime
View raw message