spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Zvirinsky <peter.zvirin...@gmail.com>
Subject LDA on a large dataset
Date Mon, 20 Jul 2015 10:21:46 GMT
Hello,

I'm trying to run LDA on a relatively large dataset (size 100-200 G), but
with no luck so far.

At first I made sure that the executors have enough memory with respect to
the vocabulary size and number of topics.

After that I ran LDA with default EMLDAOptimizer, but learning failed after
a few iteration, because the application master ran out of disk. The
learning job used all space available in the usercache of the application
master (cca. 100G). I noticed that this implementation uses some sort of
checkopointing so I made sure it is not used, but it didn't help.

Afterwards, I tried the OnlineLDAOptimizer, but it started failing at
"reduce at LDAOptimizer.scala:421" with error message: "Total size of
serialized results of X tasks (Y GB) is bigger than
spark.driver.maxResultSize (Y GB)". I kept increasing the
spark.driver.maxResultSize to tens of GB but it didn't help, just delayed
this error. I tried to adjust the batch size to very small values so that I
was sure it must fit into memory, but this didn't help at all.

Has anyone experience with learning LDA on such a dataset? Maybe some ideas
what might be wrong?

I'm using spark 1.4.0 in yarn-client mode. I managed to learn a word2vec
model on the same dataset with no problems at all.

Thanks,
Peter

Mime
View raw message