spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Beckingham <>
Subject Run times for Spark 1.6.2 compared to 2.1.0?
Date Wed, 27 Jul 2016 20:31:51 GMT
I have a project which runs fine in both Spark 1.6.2 and 2.1.0. It 
calculates a logistic model using MLlib. I compiled the 2.1 today from 
source and took the version 1 as a precompiled version with Hadoop. The 
odd thing is that on 1.6.2 the project produces an answer in 350 sec and 
the 2.1.0 takes 990 sec. Identical code using pyspark. I'm wondering if 
there is something in the setup params for 1.6 and 2.1, say number of 
executors or memory allocation, which might account for this? I'm using 
just the 4 cores of my machine as master and executors.

To unsubscribe e-mail:

View raw message