spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Error : - No filesystem for scheme: spark
Date Mon, 02 Nov 2015 09:41:06 GMT
Just to be sure: you use yarn cluster (not standalone), right ?

Regards
JB

On 11/02/2015 10:37 AM, Balachandar R.A. wrote:
> Yes. In two different places I use spark://
>
> 1. In my code, while creating spark configuration, I use the code below
>
> val sConf = new
> SparkConf().setAppName("Dummy").setMaster("spark://<myHost>:7077")
> val sConf = val sc = new SparkContext(sConf)
>
>
> 2. I run the job using the command below
>
> spark-submit  --class org.myjob  --jars myjob.jar spark://<myHost>:7077
> myjob.jar
>
> regards
> Bala
>
>
> On 2 November 2015 at 14:59, Romi Kuntsman <romi@totango.com
> <mailto:romi@totango.com>> wrote:
>
>     except "spark.master", do you have "spark://" anywhere in your code
>     or config files?
>
>     *Romi Kuntsman*, /Big Data Engineer/_
>     _
>     http://www.totango.com <http://www.totango.com/>
>
>     On Mon, Nov 2, 2015 at 11:27 AM, Balachandar R.A.
>     <balachandar.ra@gmail.com <mailto:balachandar.ra@gmail.com>> wrote:
>
>
>         ---------- Forwarded message ----------
>         From: "Balachandar R.A." <balachandar.ra@gmail.com
>         <mailto:balachandar.ra@gmail.com>>
>         Date: 02-Nov-2015 12:53 pm
>         Subject: Re: Error : - No filesystem for scheme: spark
>         To: "Jean-Baptiste Onofré" <jb@nanthrax.net
>         <mailto:jb@nanthrax.net>>
>         Cc:
>
>          > HI JB,
>          > Thanks for the response,
>          > Here is the content of my spark-defaults.conf
>          >
>          >
>          > # Default system properties included when running spark-submit.
>          > # This is useful for setting default environmental settings.
>          >
>          > # Example:
>          >  spark.master                     spark://fdoat:7077
>          > # spark.eventLog.enabled           true
>          >  spark.eventLog.dir                /home/bala/spark-logs
>          > # spark.eventLog.dir               hdfs://namenode:8021/directory
>          > # spark.serializer
>         org.apache.spark.serializer.KryoSerializer
>          > # spark.driver.memory              5g
>          > # spark.executor.extraJavaOptions  -XX:+PrintGCDetails
>         -Dkey=value -Dnumbers="one two three"
>          >
>          >
>          > regards
>          > Bala
>
>
>          >
>          > On 2 November 2015 at 12:21, Jean-Baptiste Onofré
>         <jb@nanthrax.net <mailto:jb@nanthrax.net>> wrote:
>          >>
>          >> Hi,
>          >>
>          >> do you have something special in conf/spark-defaults.conf
>         (especially on the eventLog directory) ?
>          >>
>          >> Regards
>          >> JB
>          >>
>          >>
>          >> On 11/02/2015 07:48 AM, Balachandar R.A. wrote:
>          >>>
>          >>> Can someone tell me at what point this error could come?
>          >>>
>          >>> In one of my use cases, I am trying to use hadoop custom
>         input format.
>          >>> Here is my code.
>          >>>
>          >>> |valhConf:Configuration=sc.hadoopConfiguration
>          >>>
>         hConf.set("fs.hdfs.impl",classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)hConf.set("fs.file.impl",classOf[org.apache.hadoop.fs.LocalFileSystem].getName)varjob
>          >>>
>         =newJob(hConf)FileInputFormat.setInputPaths(job,newPath("hdfs:///user/bala/MyBinaryFile"));varhRDD
>          >>>
>         =newNewHadoopRDD(sc,classOf[RandomAccessInputFormat],classOf[IntWritable],classOf[BytesWritable],job.getConfiguration())valcount
>          >>>
>         =hRDD.mapPartitionsWithInputSplit{(split,iter)=>myfuncPart(split,iter)}|
>          >>>
>          >>> |The moment I invoke mapPartitionsWithInputSplit() method,
>         I get the
>          >>> below error in my spark-submit launch|
>          >>>
>          >>> |
>          >>> |
>          >>>
>          >>> |15/10/3011:11:39WARN scheduler.TaskSetManager:Losttask
>         0.0in stage
>          >>> 0.0(TID
>         0,40.221.94.235):java.io.IOException:NoFileSystemforscheme:spark
>          >>> at
>          >>>
>         org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)at
>          >>>
>         org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)at
>          >>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)|
>          >>>
>          >>> Any help here to move towards fixing this will be of great help
>          >>>
>          >>>
>          >>>
>          >>> Thanks
>          >>>
>          >>> Bala
>          >>>
>          >>
>          >> --
>          >> Jean-Baptiste Onofré
>          >> jbonofre@apache.org <mailto:jbonofre@apache.org>
>          >> http://blog.nanthrax.net
>          >> Talend - http://www.talend.com
>          >>
>          >>
>         ---------------------------------------------------------------------
>          >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>         <mailto:user-unsubscribe@spark.apache.org>
>          >> For additional commands, e-mail: user-help@spark.apache.org
>         <mailto:user-help@spark.apache.org>
>          >>
>          >
>
>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message