spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Fails to run simple Spark (Hello World) scala program
Date Mon, 22 Sep 2014 06:57:36 GMT
Hi Moshe,

I ran the same code on my machine and it is working without any issues. You
can try running it in local mode and if that is working fine, then the
issue is with your configuration.

[image: Inline image 1]

Thanks
Best Regards

On Sat, Sep 20, 2014 at 3:34 PM, Moshe Beeri <moshe.beeri@gmail.com> wrote:

> Hi Sean,
>
> Thanks a lot for the answer , I loved your excellent book
> *​Mahout in Action
> <http://www.amazon.com/Mahout-Action-Sean-Owen/dp/1935182684> *hope
> you'll keep on writing more books in the field of Big Data.
> The issue was with redundant Hadoop library, But now I am facing some
> other issue (see prev post in this thread)
> java.lang.ClassNotFoundException: com.example.scamel.Nizoz$$anonfun$3
>
> But the class com.example.scamel.Nizoz (in fact Scala object) is the one
> under debugging.
>
>   def main(args: Array[String]) {
>     println(scala.tools.nsc.Properties.versionString)
>     try {
>       //Nizoz.connect
>       val logFile =
> "/home/moshe/store/frameworks/spark-1.1.0-bin-hadoop1/README.md" // Should
> be some file on your system
>       val conf = new SparkConf().setAppName("spark
> town").setMaster("spark://nash:7077"); //spark://master:7077
>       val sc = new SparkContext(conf)
>       val logData = sc.textFile(logFile, 2).cache()
>       *val numAs = logData.filter(line => line.contains("a")).count()
>  // <- here is  where the exception thrown *
>
> Do you have any idea whats wrong?
> Thanks,
> Moshe Beeri.
>
>
> *​*
>
>
> תודה רבה,
> משה בארי.
> 054-3133943
> [hidden email] <http://user/SendEmail.jtp?type=node&node=14731&i=0> |
> linkedin <http://www.linkedin.com/in/mobee>
>
>
>
> On Sat, Sep 20, 2014 at 12:02 PM, sowen [via Apache Spark User List] <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=14731&i=1>> wrote:
>
>> Spark does not require Hadoop 2 or YARN. This looks like a problem with
>> the Hadoop installation as it is not funding native libraries it needs to
>> make some security related system call. Check the installation.
>> On Sep 20, 2014 9:13 AM, "Manu Suryavansh" <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=14724&i=0>> wrote:
>>
>>> Hi Moshe,
>>>
>>> Spark needs a Hadoop 2.x/YARN cluster. Other wise you can run it without
>>> hadoop in the stand alone mode.
>>>
>>> Manu
>>>
>>>
>>>
>>> On Sat, Sep 20, 2014 at 12:55 AM, Moshe Beeri <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=14724&i=1>> wrote:
>>>
>>>> object Nizoz {
>>>>
>>>>   def connect(): Unit = {
>>>>     val conf = new SparkConf().setAppName("nizoz").setMaster("master");
>>>>     val spark = new SparkContext(conf)
>>>>     val lines =
>>>>
>>>> spark.textFile("file:///home/moshe/store/frameworks/spark-1.1.0-bin-hadoop1/README.md")
>>>>     val lineLengths = lines.map(s => s.length)
>>>>     val totalLength = lineLengths.reduce((a, b) => a + b)
>>>>     println("totalLength=" + totalLength)
>>>>
>>>>   }
>>>>
>>>>   def main(args: Array[String]) {
>>>>     println(scala.tools.nsc.Properties.versionString)
>>>>     try {
>>>>       //Nizoz.connect
>>>>       val logFile =
>>>> "/home/moshe/store/frameworks/spark-1.1.0-bin-hadoop1/README.md" //
>>>> Should
>>>> be some file on your system
>>>>       val conf = new SparkConf().setAppName("Simple
>>>> Application").setMaster("spark://master:7077")
>>>>       val sc = new SparkContext(conf)
>>>>       val logData = sc.textFile(logFile, 2).cache()
>>>>       val numAs = logData.filter(line => line.contains("a")).count()
>>>>       val numBs = logData.filter(line => line.contains("b")).count()
>>>>       println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
>>>>
>>>>     } catch {
>>>>       case e => {
>>>>         println(e.getCause())
>>>>         println("stack:")
>>>>         e.printStackTrace()
>>>>       }
>>>>     }
>>>>   }
>>>> }
>>>> Runs with Scala 2.10.4
>>>> The problem is this [vogue] exception:
>>>>
>>>>         at com.example.scamel.Nizoz.main(Nizoz.scala)
>>>> Caused by: java.lang.RuntimeException:
>>>> java.lang.reflect.InvocationTargetException
>>>>         at
>>>>
>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
>>>>         at org.apache.hadoop.security.Groups.<init>(Groups.java:64)
>>>>         at
>>>>
>>>> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
>>>> ...
>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>         at
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>         at
>>>>
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>>>         at
>>>>
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>> ...
>>>>         ... 10 more
>>>> Caused by: java.lang.UnsatisfiedLinkError:
>>>> org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
>>>>         at
>>>> org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native
>>>> Method)
>>>>         at
>>>>
>>>> org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:49)
>>>>
>>>> I have Hadoop 1.2.1 running on Ubuntu 14.04, the Scala console run as
>>>> expected.
>>>>
>>>> What am I doing wrong?
>>>> Any idea will be welcome
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Fails-to-run-simple-Spark-Hello-World-scala-program-tp14718.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=14724&i=2>
>>>> For additional commands, e-mail: [hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=14724&i=3>
>>>>
>>>>
>>>
>>>
>>> --
>>> Manu Suryavansh
>>>
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Fails-to-run-simple-Spark-Hello-World-scala-program-tp14718p14724.html
>>  To unsubscribe from Fails to run simple Spark (Hello World) scala
>> program, click here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Fails to run simple Spark (Hello World)
> scala program
> <http://apache-spark-user-list.1001560.n3.nabble.com/Fails-to-run-simple-Spark-Hello-World-scala-program-tp14718p14731.html>
>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>

Mime
View raw message