spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Issue with Spark shell and scalatest
Date Thu, 21 Apr 2016 20:23:38 GMT
Hi Ted

I cannot see spark-core_2.11-2.0.0-SNAPSHOT.jar  under

https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/

Sorry where are these artefacts please?



Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 20:24, Ted Yu <yuzhihong@gmail.com> wrote:

> Plug in 1.5.1 for your jars:
>
> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging
> ...
>   1781 Thu Apr 21 08:19:34 PDT 2016
> org/apache/spark/internal/Logging$.class
>
> jar tvf
> external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep
> LeaderOffset
> ...
>   3310 Thu Apr 21 08:38:40 PDT 2016
> org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class
>
> On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> These two are giving me griefs:
>>
>> scala> import org.apache.spark.internal.Logging
>> <console>:26: error: object internal is not a member of package
>> org.apache.spark
>>          import org.apache.spark.internal.Logging
>>
>> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>> <console>:29: error: object KafkaCluster in package kafka cannot be
>> accessed in package org.apache.spark.streaming.kafka
>>          import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 18:29, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>>
>>>
>>> Thanks
>>>
>>> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
>>>   1787 Wed Sep 23 23:34:26 BST 2015
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>>   1780 Wed Sep 23 23:34:26 BST 2015
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>>   3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 18:21, Ted Yu <yuzhihong@gmail.com> wrote:
>>>
>>>> Please replace version number for the release you are using :
>>>>
>>>> spark-core_2.10-1.5.1-tests.jar
>>>>
>>>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> I don't seem to be able to locate
>>>>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>>
>>>>> On 21 April 2016 at 17:45, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>>
>>>>>> It is in core-XX-tests jar:
>>>>>>
>>>>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar
|
>>>>>> grep SparkFunSuite
>>>>>>   1830 Thu Apr 21 08:19:14 PDT 2016
>>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>>>>>   1823 Thu Apr 21 08:19:14 PDT 2016
>>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>>>>>   6232 Thu Apr 21 08:19:14 PDT 2016
>>>>>> org/apache/spark/SparkFunSuite.class
>>>>>>
>>>>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> like war of attrition :)
>>>>>>>
>>>>>>> now I get with sbt
>>>>>>>
>>>>>>> object SparkFunSuite is not a member of package org.apache.spark
>>>>>>>
>>>>>>>
>>>>>>> Dr Mich Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 21 April 2016 at 17:22, Ted Yu <yuzhihong@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Have you tried the following ?
>>>>>>>>
>>>>>>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>>>>>>>>
>>>>>>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Unfortunately this sbt dependency is not working
>>>>>>>>>
>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-core"
%
>>>>>>>>> "1.5.1" % "provided"
>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql"
%
>>>>>>>>> "1.5.1"  % "provided"
>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive"
%
>>>>>>>>> "1.5.1" % "provided"
>>>>>>>>> libraryDependencies += "junit" % "junit" % "4.12"
>>>>>>>>> libraryDependencies += "org.scala-sbt" % "test-interface"
% "1.0"
>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming"
%
>>>>>>>>> "1.6.1" % "provided"
>>>>>>>>> libraryDependencies += "org.apache.spark" %%
>>>>>>>>> "spark-streaming-kafka" % "1.6.1"
>>>>>>>>> libraryDependencies += "org.scalactic" %% "scalactic"
% "2.2.6"
>>>>>>>>> *libraryDependencies += "org.scalatest" %% "scalatest"
% "2.2.6" %
>>>>>>>>> "test"*
>>>>>>>>>
>>>>>>>>> Getting error
>>>>>>>>>
>>>>>>>>> [info] Compiling 1 Scala source to
>>>>>>>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
>>>>>>>>> [error]
>>>>>>>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
>>>>>>>>> object scalatest is not a member of package org
>>>>>>>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>>>>>>> [error]            ^
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dr Mich Talebzadeh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 21 April 2016 at 16:49, Mich Talebzadeh <
>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Ted. It was a typo in my alias and it is sorted
now
>>>>>>>>>>
>>>>>>>>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077
>>>>>>>>>> --jars
>>>>>>>>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dr Mich Talebzadeh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 21 April 2016 at 16:44, Ted Yu <yuzhihong@gmail.com>
wrote:
>>>>>>>>>>
>>>>>>>>>>> I tried on refreshed copy of master branch:
>>>>>>>>>>>
>>>>>>>>>>> $ bin/spark-shell --jars
>>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>>>>>>>>>> ...
>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter,
BeforeAndAfterAll}
>>>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>>>>>>>>>
>>>>>>>>>>> BTW I noticed an extra leading comma after '--jars'
in your
>>>>>>>>>>> email.
>>>>>>>>>>> Not sure if that matters.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu <yuzhihong@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Mich:
>>>>>>>>>>>>
>>>>>>>>>>>> $ jar tvf
>>>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>>>>>>>>>>> | grep BeforeAndAfter
>>>>>>>>>>>>   4257 Sat Dec 26 14:35:48 PST 2015
>>>>>>>>>>>> org/scalatest/BeforeAndAfter$class.class
>>>>>>>>>>>>   2602 Sat Dec 26 14:35:48 PST 2015
>>>>>>>>>>>> org/scalatest/BeforeAndAfter.class
>>>>>>>>>>>>   1998 Sat Dec 26 14:35:48 PST 2015
>>>>>>>>>>>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>>>>>>>>>>>>
>>>>>>>>>>>> FYI
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh
<
>>>>>>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> to Mario, Alonso, Luciano, user
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Following example in
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does anyone know which jar file this
belongs to?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I use *scalatest_2.11-2.2.6.jar *in my
spark-shell
>>>>>>>>>>>>>
>>>>>>>>>>>>>  spark-shell --master spark://50.140.197.217:7077
--jars
>>>>>>>>>>>>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
>>>>>>>>>>>>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>>>>>>>>>>>>>
>>>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter,
BeforeAndAfterAll}
>>>>>>>>>>>>> <console>:28: error: object scalatest
is not a member of
>>>>>>>>>>>>> package org
>>>>>>>>>>>>>          import org.scalatest.{BeforeAndAfter,
>>>>>>>>>>>>> BeforeAndAfterAll}
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dr Mich Talebzadeh
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message