spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Srivastava <>
Subject Spark Shuffle Issue
Date Wed, 12 Oct 2016 06:16:10 GMT

I am upgrading my jobs to Spark 1.6 and am running into shuffle issues. I
have tried all options and now am falling back to legacy memory model but
still running into same issue.

I have set spark.shuffle.blockTransferService to nio.

16/10/12 06:00:10 INFO MapOutputTrackerMaster: Size of output statuses for
shuffle 6 is 57910504 bytes

16/10/12 06:00:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map
output locations for shuffle 6 to

16/10/12 06:00:10 ERROR TransportRequestHandler: Error sending result
body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=57910531
cap=57910531]}} to <hostname>; closing connection

Any other settings that I should tune?



View raw message