spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Congrui Yi <>
Subject Re: pyspark-Failed to run first
Date Thu, 19 Jun 2014 23:36:40 GMT
I'm starting to develop ADMM for some models using pyspark(Spark version
1.0.0). So I constantly simulated data to test my code. I did simulation in
python but then I ran into the same kind of problems as mentioned above.
Same meaningless error messages show up when I tried methods like first,
take or takeSample. There is no "Out of Memory" so the size should not be a
problem for pyspark.

Again, this is not a problem for Scala. 

I also installed and tried Spark 0.9.1. The same code runs correctly in
pyspark of the older version.

So it is a problem only with pyspark in 1.0.0. 

My code for data simulation:


View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message