1. Does netty perform better than the basic method for shuffling? I found the latency caused by shuffling in a streaming job is not stable with the basic method.
2. However, after I turn on netty for shuffling, I can only see the results for the first two batches, and then no output at all. I'm not sure whether the way I turn on netty is correct:
val conf = new SparkConf().set("spark.shuffle.use.netty", "true")