mapPartitionsWithSplit was removed in Spark 2.0.0. You can use mapPartitionsWithIndex instead.

On Tue, Mar 28, 2017 at 3:52 PM, Anahita Talebi <anahita.t.amiri@gmail.com> wrote:
Thanks.
I tried this one, as well. Unfortunately I still get the same error.


On Wednesday, March 29, 2017, Marco Mistroni <mmistroni@gmail.com> wrote:
1.7.5

On 28 Mar 2017 10:10 pm, "Anahita Talebi" <anahita.t.amiri@gmail.com> wrote:
Hi,

Thanks for your answer.
What is the version of "org.slf4j" % "slf4j-api" in your sbt file?
I think the problem might come from this part.

On Tue, Mar 28, 2017 at 11:02 PM, Marco Mistroni <mmistroni@gmail.com> wrote:
Hello
 uhm ihave a project whose build,sbt is closest to yours, where i am using spark 2.1, scala 2.11 and scalatest (i upgraded to 3.0.0) and it works fine
in my projects though i don thave any of the following libraries that you mention
- breeze
- netlib,all
-  scoopt

hth

On Tue, Mar 28, 2017 at 9:10 PM, Anahita Talebi <anahita.t.amiri@gmail.com> wrote:
Hi,

Thanks for your answer.

I first changed the scala version to 2.11.8 and kept the spark version 1.5.2 (old version). Then I changed the scalatest version into "3.0.1". With this configuration, I could run the code and compile it and generate the .jar file.

When I changed the spark version into 2.1.0, I get the same error as before. So I imagine the problem should be somehow related to the version of spark.

Cheers,
Anahita

--------------------------------------------------------------------------------------------------------------------------------------------------------
import AssemblyKeys._

assemblySettings

name := "proxcocoa"

version := "0.1"

organization := "edu.berkeley.cs.amplab"

scalaVersion := "2.11.8"

parallelExecution in Test := false

{
  val excludeHadoop = ExclusionRule(organization = "org.apache.hadoop")
  libraryDependencies ++= Seq(
    "org.slf4j" % "slf4j-api" % "1.7.2",
    "org.slf4j" % "slf4j-log4j12" % "1.7.2",
    "org.scalatest" %% "scalatest" % "3.0.1" % "test",
    "org.apache.spark" %% "spark-core" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.spark" %% "spark-mllib" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.spark" %% "spark-sql" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.commons" % "commons-compress" % "1.7",
    "commons-io" % "commons-io" % "2.4",
    "org.scalanlp" % "breeze_2.11" % "0.11.2",
    "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly(),
    "com.github.scopt" %% "scopt" % "3.3.0"
  )
}

{
  val defaultHadoopVersion = "1.0.4"
  val hadoopVersion =
    scala.util.Properties.envOrElse("SPARK_HADOOP_VERSION", defaultHadoopVersion)
  libraryDependencies += "org.apache.hadoop" % "hadoop-client" % hadoopVersion
}

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.1.0"

resolvers ++= Seq(
  "Local Maven Repository" at Path.userHome.asFile.toURI.toURL + ".m2/repository",
  "Typesafe" at "http://repo.typesafe.com/typesafe/releases",
  "Spray" at "http://repo.spray.cc"
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)           => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html"   => MergeStrategy.first
    case "application.conf"                              => MergeStrategy.concat
    case "reference.conf"                                => MergeStrategy.concat
    case "log4j.properties"                              => MergeStrategy.discard
    case m if m.toLowerCase.endsWith("manifest.mf")      => MergeStrategy.discard
    case m if m.toLowerCase.matches("meta-inf.*\\.sf$")  => MergeStrategy.discard
    case _ => MergeStrategy.first
  }
}

test in assembly := {}
--------------------------------------------------------------------------------------------------------------------------------------------------------

On Tue, Mar 28, 2017 at 9:33 PM, Marco Mistroni <mmistroni@gmail.com> wrote:
Hello
 that looks to me like there's something dodgy withyour Scala installation
Though Spark 2.0 is built on Scala 2.11, it still support 2.10... i suggest you change one thing at a time in your sbt
First Spark version. run it and see if it works
Then amend the scala version

hth
 marco

On Tue, Mar 28, 2017 at 5:20 PM, Anahita Talebi <anahita.t.amiri@gmail.com> wrote:
Hello,

Thanks you all for your informative answers.
I actually changed the scala version to the 2.11.8 and spark version into 2.1.0 in the build.sbt

Except for these two guys (scala and spark version), I kept the same values for the rest in the build.sbt file.
---------------------------------------------------------------------------
import AssemblyKeys._

assemblySettings

name := "proxcocoa"

version := "0.1"

scalaVersion := "2.11.8"

parallelExecution in Test := false

{
  val excludeHadoop = ExclusionRule(organization = "org.apache.hadoop")
  libraryDependencies ++= Seq(
    "org.slf4j" % "slf4j-api" % "1.7.2",
    "org.slf4j" % "slf4j-log4j12" % "1.7.2",
    "org.scalatest" %% "scalatest" % "1.9.1" % "test",
    "org.apache.spark" % "spark-core_2.11" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.spark" % "spark-sql_2.11" % "2.1.0" excludeAll(excludeHadoop),
    "org.apache.commons" % "commons-compress" % "1.7",
    "commons-io" % "commons-io" % "2.4",
    "org.scalanlp" % "breeze_2.11" % "0.11.2",
    "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly(),
    "com.github.scopt" %% "scopt" % "3.3.0"
  )
}

{
  val defaultHadoopVersion = "1.0.4"
  val hadoopVersion =
    scala.util.Properties.envOrElse("SPARK_HADOOP_VERSION", defaultHadoopVersion)
  libraryDependencies += "org.apache.hadoop" % "hadoop-client" % hadoopVersion
}

libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.1.0"

resolvers ++= Seq(
  "Local Maven Repository" at Path.userHome.asFile.toURI.toURL + ".m2/repository",
  "Typesafe" at "http://repo.typesafe.com/typesafe/releases",
  "Spray" at "http://repo.spray.cc"
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)           => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html"   => MergeStrategy.first
    case "application.conf"                              => MergeStrategy.concat
    case "reference.conf"                                => MergeStrategy.concat
    case "log4j.properties"                              => MergeStrategy.discard
    case m if m.toLowerCase.endsWith("manifest.mf")      => MergeStrategy.discard
    case m if m.toLowerCase.matches("meta-inf.*\\.sf$")  => MergeStrategy.discard
    case _ => MergeStrategy.first
  }
}

test in assembly := {}
----------------------------------------------------------------

When I compile the code, I get the following error:

[info] Compiling 4 Scala sources to /Users/atalebi/Desktop/new_version_proxcocoa-master/target/scala-2.11/classes...
[error] /Users/atalebi/Desktop/new_version_proxcocoa-master/src/main/scala/utils/OptUtils.scala:40: value mapPartitionsWithSplit is not a member of org.apache.spark.rdd.RDD[String]
[error]     val sizes = data.mapPartitionsWithSplit{ case(i,lines) =>
[error]                      ^
[error] /Users/atalebi/Desktop/new_version_proxcocoa-master/src/main/scala/utils/OptUtils.scala:41: value length is not a member of Any
[error]       Iterator(i -> lines.length)
[error]                           ^
----------------------------------------------------------------
It gets the error in the code. Does it mean that for the different version of the spark and scala, I need to change the main code?

Thanks,
Anahita






On Tue, Mar 28, 2017 at 10:28 AM, Dinko Srkoč <dinko.srkoc@gmail.com> wrote:
Adding to advices given by others ... Spark 2.1.0 works with Scala 2.11, so set:

  scalaVersion := "2.11.8"

When you see something like:

  "org.apache.spark" % "spark-core_2.10" % "1.5.2"

that means that library `spark-core` is compiled against Scala 2.10,
so you would have to change that to 2.11:

  "org.apache.spark" % "spark-core_2.11" % "2.1.0"

better yet, let SBT worry about libraries built against particular
Scala versions:

  "org.apache.spark" %% "spark-core" % "2.1.0"

The `%%` will instruct SBT to choose the library appropriate for a
version of Scala that is set in `scalaVersion`.

It may be worth mentioning that the `%%` thing works only with Scala
libraries as they are compiled against a certain Scala version. Java
libraries are unaffected (have nothing to do with Scala), e.g. for
`slf4j` one only uses single `%`s:

  "org.slf4j" % "slf4j-api" % "1.7.2"

Cheers,
Dinko

On 27 March 2017 at 23:30, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
> check these versions
>
> function create_build_sbt_file {
>         BUILD_SBT_FILE=${GEN_APPSDIR}/scala/${APPLICATION}/build.sbt
>         [ -f ${BUILD_SBT_FILE} ] && rm -f ${BUILD_SBT_FILE}
>         cat >> $BUILD_SBT_FILE << !
> lazy val root = (project in file(".")).
>   settings(
>     name := "${APPLICATION}",
>     version := "1.0",
>     scalaVersion := "2.11.8",
>     mainClass in Compile := Some("myPackage.${APPLICATION}")
>   )
> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.0.0" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
> "1.6.1" % "provided"
> libraryDependencies += "com.google.code.gson" % "gson" % "2.6.2"
> libraryDependencies += "org.apache.phoenix" % "phoenix-spark" %
> "4.6.0-HBase-1.0"
> libraryDependencies += "org.apache.hbase" % "hbase" % "1.2.3"
> libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.2.3"
> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.2.3"
> libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.2.3"
> // META-INF discarding
> mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
>    {
>     case PathList("META-INF", xs @ _*) => MergeStrategy.discard
>     case x => MergeStrategy.first
>    }
> }
> !
> }
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
>
>
> On 27 March 2017 at 21:45, Jörn Franke <jornfranke@gmail.com> wrote:
>>
>> Usually you define the dependencies to the Spark library as provided. You
>> also seem to mix different Spark versions which should be avoided.
>> The Hadoop library seems to be outdated and should also only be provided.
>>
>> The other dependencies you could assemble in a fat jar.
>>
>> On 27 Mar 2017, at 21:25, Anahita Talebi <anahita.t.amiri@gmail.com>
>> wrote:
>>
>> Hi friends,
>>
>> I have a code which is written in Scala. The scala version 2.10.4 and
>> Spark version 1.5.2 are used to run the code.
>>
>> I would like to upgrade the code to the most updated version of spark,
>> meaning 2.1.0.
>>
>> Here is the build.sbt:
>>
>> import AssemblyKeys._
>>
>> assemblySettings
>>
>> name := "proxcocoa"
>>
>> version := "0.1"
>>
>> scalaVersion := "2.10.4"
>>
>> parallelExecution in Test := false
>>
>> {
>>   val excludeHadoop = ExclusionRule(organization = "org.apache.hadoop")
>>   libraryDependencies ++= Seq(
>>     "org.slf4j" % "slf4j-api" % "1.7.2",
>>     "org.slf4j" % "slf4j-log4j12" % "1.7.2",
>>     "org.scalatest" %% "scalatest" % "1.9.1" % "test",
>>     "org.apache.spark" % "spark-core_2.10" % "1.5.2"
>> excludeAll(excludeHadoop),
>>     "org.apache.spark" % "spark-mllib_2.10" % "1.5.2"
>> excludeAll(excludeHadoop),
>>     "org.apache.spark" % "spark-sql_2.10" % "1.5.2"
>> excludeAll(excludeHadoop),
>>     "org.apache.commons" % "commons-compress" % "1.7",
>>     "commons-io" % "commons-io" % "2.4",
>>     "org.scalanlp" % "breeze_2.10" % "0.11.2",
>>     "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly(),
>>     "com.github.scopt" %% "scopt" % "3.3.0"
>>   )
>> }
>>
>> {
>>   val defaultHadoopVersion = "1.0.4"
>>   val hadoopVersion =
>>     scala.util.Properties.envOrElse("SPARK_HADOOP_VERSION",
>> defaultHadoopVersion)
>>   libraryDependencies += "org.apache.hadoop" % "hadoop-client" %
>> hadoopVersion
>> }
>>
>> libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" %
>> "1.5.0"
>>
>> resolvers ++= Seq(
>>   "Local Maven Repository" at Path.userHome.asFile.toURI.toURL +
>> ".m2/repository",
>>   "Typesafe" at "http://repo.typesafe.com/typesafe/releases",
>>   "Spray" at "http://repo.spray.cc"
>> )
>>
>> mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
>>   {
>>     case PathList("javax", "servlet", xs @ _*)           =>
>> MergeStrategy.first
>>     case PathList(ps @ _*) if ps.last endsWith ".html"   =>
>> MergeStrategy.first
>>     case "application.conf"                              =>
>> MergeStrategy.concat
>>     case "reference.conf"                                =>
>> MergeStrategy.concat
>>     case "log4j.properties"                              =>
>> MergeStrategy.discard
>>     case m if m.toLowerCase.endsWith("manifest.mf")      =>
>> MergeStrategy.discard
>>     case m if m.toLowerCase.matches("meta-inf.*\\.sf$")  =>
>> MergeStrategy.discard
>>     case _ => MergeStrategy.first
>>   }
>> }
>>
>> test in assembly := {}
>>
>> -----------------------------------------------------------
>> I downloaded the spark 2.1.0 and change the version of spark and
>> scalaversion in the build.sbt. But unfortunately, I was failed to run the
>> code.
>>
>> Does anybody know how I can upgrade the code to the most recent spark
>> version by changing the build.sbt file?
>>
>> Or do you have any other suggestion?
>>
>> Thanks a lot,
>> Anahita
>>
>