spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Congrui Yi <fixed-term.Congrui...@us.bosch.com>
Subject Re: pyspark-Failed to run first
Date Thu, 19 Jun 2014 23:36:40 GMT
I'm starting to develop ADMM for some models using pyspark(Spark version
1.0.0). So I constantly simulated data to test my code. I did simulation in
python but then I ran into the same kind of problems as mentioned above.
Same meaningless error messages show up when I tried methods like first,
take or takeSample. There is no "Out of Memory" so the size should not be a
problem for pyspark.

Again, this is not a problem for Scala. 

I also installed and tried Spark 0.9.1. The same code runs correctly in
pyspark of the older version.

So it is a problem only with pyspark in 1.0.0. 

My code for data simulation:


-Congrui



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-Failed-to-run-first-tp7691p7964.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message