spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anupamme <medira...@gmail.com>
Subject how to save Word2VecModel
Date Wed, 04 Mar 2015 06:16:29 GMT
Hello

I started using spark. I am working with Word2VecModel. However I am not
able to save the trained model. Here is what I am doing:

inp = sc.textFile("/Users/mediratta/code/word2vec/trunk-d/sub-5").map(lambda
row: row.split(" "))
word2vec = Word2Vec()
model = word2vec.fit(inp)
out = open('abc.bin', 'wb')
pickle.dump(model, out, pickle.HIGHEST_PROTOCOL)

But I get error:

"It appears that you are attempting to reference SparkContext from a
broadcast "
Exception: It appears that you are attempting to reference SparkContext from
a broadcast variable, action, or transforamtion. SparkContext can only be
used on the driver, not in code that it run on workers. For more
information, see SPARK-5063.

However when I run pickle.dump of argument of type list instead of
Word2VecModel, then pickle.dumps works fine.

So seems the error is coming because of the type of the first argument
(Word2VecModel in this case). However the error message seems misleading.

Any clue what I am doing wrong?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-save-Word2VecModel-tp21900.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message