spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshat Aranya <>
Subject Task scheduling times
Date Mon, 04 May 2015 18:27:09 GMT

I have been investigating scheduling delays in Spark and I found some
unexplained anomalies.  In my use case, I have two stages after
collapsing the transformations: the first is a mapPartitions() and the
second is a sortByKey().  I found that the task serialization for the
first stage takes much longer than the second.

1. mapPartitions() - this launches 256 tasks in 603 ms (avg. 2.363
ms). Each task serializes to 1220 bytes.
2. sortByKey() - this launches 64 tasks in 12 ms (avg. 0.187 ms). Each
task serializes to 1139 bytes.

Note that the serialized size of the task is similar, but the avg.
scheduling time is very different.  I also instrumented the code to
print out the serialization time, and it seems like it is indeed the
serialization that takes much longer.  This seemed weird to me because
the biggest part of the Task, the taskBinary is actually directly
copied from a byte array.

Any explanation of why this happens?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message