uhm.sorry...... still same issues. this is hte new version

name := "SparkExamples"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.0.1"

// Add a single dependency
libraryDependencies += "junit" % "junit" % "4.8" % "test"
libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5",
                            "org.slf4j" % "slf4j-simple" % "1.7.5",
                            "org.clapper" %% "grizzled-slf4j" % "1.0.2")
libraryDependencies += "org.apache.spark"%%"spark-core"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-mllib"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming-flume-sink" % "2.0.1"    
libraryDependencies += "org.apache.spark"%%"spark-sql"   % sparkVersion


resolvers += "softprops-maven" at "http://dl.bintray.com/content/softprops/maven"

Still seeing these kinds of errors  which seems to lead to the fact that somehow sbt is getting confused..

C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:2: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.{ Vector, Vectors }
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:3: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.regression.LabeledPoint
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:4: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification.{ RandomForestClassifier, RandomForestClassificationModel }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:6: object feature is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.feature.{ StringIndexer, IndexToString, VectorIndexer, VectorAssembler }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:7: object evaluation is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.evaluation.{ RegressionEvaluator, MulticlassClassificationEvaluator }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:8: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification._
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:9: object tuning is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.tuning.{ CrossValidator, ParamGridBuilder }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:10: object tuning is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.tuning.{ ParamGridBuilder, TrainValidationSplit }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:16: object Pipeline is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.{ Pipeline, PipelineModel }

any other hints?

thanks and regarsd
 marco




On Sun, Nov 13, 2016 at 10:52 PM, Don Drake <dondrake@gmail.com> wrote:
I would upgrade your Scala version to 2.11.8 as Spark 2.0 uses Scala 2.11 by default.

On Sun, Nov 13, 2016 at 3:01 PM, Marco Mistroni <mmistroni@gmail.com> wrote:
HI all
 i have a small Spark-based project which at the moment depends on jar
from Spark 1.6.0
The project has few Spark examples plus one which depends on Flume libraries


I am attempting to move to Spark 2.0, but i am having issues with
my dependencies
The stetup below works fine when compiled against 1.6.0 dependencies

name := "SparkExamples"
version := "1.0"
scalaVersion := "2.10.5"
val sparkVersion = "1.6.0"


// Add a single dependency
libraryDependencies += "junit" % "junit" % "4.8" % "test"
libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5",
                            "org.slf4j" % "slf4j-simple" % "1.7.5",
                            "org.clapper" %% "grizzled-slf4j" % "1.0.2")
libraryDependencies += "org.apache.spark"%%"spark-core"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-mllib"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming-flume" % "1.3.0"    
libraryDependencies += "org.apache.spark"%%"spark-sql"   % sparkVersion


resolvers += "softprops-maven" at "http://dl.bintray.com/content/softprops/maven"



This is the build.sbt version for using Spark 2 dependencies

name := "SparkExamples"
version := "1.0"
scalaVersion := "2.10.6"
val sparkVersion = "2.0.1"


// Add a single dependency
libraryDependencies += "junit" % "junit" % "4.8" % "test"
libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5",
                            "org.slf4j" % "slf4j-simple" % "1.7.5",
                            "org.clapper" %% "grizzled-slf4j" % "1.0.2")
libraryDependencies += "org.apache.spark"%%"spark-core"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-mllib"   % sparkVersion
libraryDependencies += "org.apache.spark"%%"spark-streaming-flume-sink" % "2.0.1"    
libraryDependencies += "org.apache.spark"%%"spark-sql"   % sparkVersion
resolvers += "softprops-maven" at "http://dl.bintray.com/content/softprops/maven"

but the sbt compile fails miserably...below few of the errors (it actually compiles like i forgot all the depencencies as it is complaining on all org.apache.spark.ml and mllib packages

[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
[warn]  * org.apache.spark:spark-core_2.10:(1.6.1, 2.0.1)
[warn]  * org.apache.spark:spark-streaming_2.10:(1.6.1, 2.0.1)
[warn]  * org.apache.spark:spark-sql_2.10:(1.6.1, 2.0.1)
[warn]  * org.apache.spark:spark-mllib_2.10:(1.6.1, 2.0.1)
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
.........
[warn]     * org.apache.spark:spark-mllib_2.10:1.6.1 -> 2.0.1
[warn]     * org.apache.spark:spark-sql_2.10:1.6.1 -> 2.0.1
[warn]     * org.apache.spark:spark-streaming_2.10:1.6.1 -> 2.0.1
[warn]     * org.apache.spark:spark-core_2.10:1.6.1 -> 2.0.1
[warn] Run 'evicted' to see detailed eviction warnings
[info] Compiling 18 Scala sources to C:\Users\marco\SparkExamples\target\scala-2.10\classes...
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:2: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.{ Vector, Vectors }
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:3: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.regression.LabeledPoint
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:4: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification._
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:5: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.tree.DecisionTree
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:6: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.tree.model.DecisionTreeModel
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:7: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.util.MLUtils
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:9: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.stat.{ MultivariateStatisticalSummary, Statistics }
[error]                         ^
[error] C:\Users\marco\SparkExamples\src\main\scala\AnotherDecisionTreeExample.scala:10: object mllib is not a member of package org.apache.spark
[error] import org.apache.spark.mllib.linalg.Vector
........
C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:4: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification.{ RandomForestClassifier, RandomForestClassificationModel }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:6: object feature is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.feature.{ StringIndexer, IndexToString, VectorIndexer, VectorAssembler }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:7: object evaluation is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.evaluation.{ RegressionEvaluator, MulticlassClassificationEvaluator }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:8: object classification is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.classification._
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:9: object tuning is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.tuning.{ CrossValidator, ParamGridBuilder }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:10: object tuning is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.tuning.{ ParamGridBuilder, TrainValidationSplit }
[error]                            ^
[error] C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:16: object Pipeline is not a member of package org.apache.spark.ml
[error] import org.apache.spark.ml.{ Pipeline, PipelineModel }
[error]  
....
and many more

Could anyone assist or pass me the right dependencies? Or am i using wrong resolvers?

kr
 marco



--