spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From onpoq l <onpo...@gmail.com>
Subject shuffling using netty in spark streaming
Date Thu, 12 Jun 2014 06:35:10 GMT
Hi,

1. Does netty perform better than the basic method for shuffling? I found
the latency caused by shuffling in a streaming job is not stable with the
basic method.

2. However, after I turn on netty for shuffling, I can only see the results
for the first two batches, and then no output at all. I'm not sure whether
the way I turn on netty is correct:

val conf = new SparkConf().set("spark.shuffle.use.netty", "true")

Thanks.

Boduo Li

Mime
View raw message