spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoye Sun <sunxiaoy...@gmail.com>
Subject large scheduler delay in OnlineLDAOptimizer, (MLlib and LDA)
Date Thu, 27 Oct 2016 18:05:27 GMT
Hi,

I am running some experiments with OnlineLDAOptimizer in Spark 1.6.1. My
Spark cluster has 30 machines.

However, I found that the Scheduler delay at job/stage "reduce at
LDAOptimizer.scala:452" is extremely large when the LDA model is large. The
delay could be tens of seconds.

Does anyone know the reason for that?

Best,
Xiaoye

Mime
View raw message