spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balachandar R.A." <balachandar...@gmail.com>
Subject Re: Error : - No filesystem for scheme: spark
Date Mon, 02 Nov 2015 09:45:44 GMT
No.. I am not using yarn. Yarn is not running in my cluster. So, it is
standalone one.

Regards
Bala
On 02-Nov-2015 3:11 pm, "Jean-Baptiste Onofré" <jb@nanthrax.net> wrote:

> Just to be sure: you use yarn cluster (not standalone), right ?
>
> Regards
> JB
>
> On 11/02/2015 10:37 AM, Balachandar R.A. wrote:
>
>> Yes. In two different places I use spark://
>>
>> 1. In my code, while creating spark configuration, I use the code below
>>
>> val sConf = new
>> SparkConf().setAppName("Dummy").setMaster("spark://<myHost>:7077")
>> val sConf = val sc = new SparkContext(sConf)
>>
>>
>> 2. I run the job using the command below
>>
>> spark-submit  --class org.myjob  --jars myjob.jar spark://<myHost>:7077
>> myjob.jar
>>
>> regards
>> Bala
>>
>>
>> On 2 November 2015 at 14:59, Romi Kuntsman <romi@totango.com
>> <mailto:romi@totango.com>> wrote:
>>
>>     except "spark.master", do you have "spark://" anywhere in your code
>>     or config files?
>>
>>     *Romi Kuntsman*, /Big Data Engineer/_
>>     _
>>     http://www.totango.com <http://www.totango.com/>
>>
>>     On Mon, Nov 2, 2015 at 11:27 AM, Balachandar R.A.
>>     <balachandar.ra@gmail.com <mailto:balachandar.ra@gmail.com>> wrote:
>>
>>
>>         ---------- Forwarded message ----------
>>         From: "Balachandar R.A." <balachandar.ra@gmail.com
>>         <mailto:balachandar.ra@gmail.com>>
>>         Date: 02-Nov-2015 12:53 pm
>>         Subject: Re: Error : - No filesystem for scheme: spark
>>         To: "Jean-Baptiste Onofré" <jb@nanthrax.net
>>         <mailto:jb@nanthrax.net>>
>>         Cc:
>>
>>          > HI JB,
>>          > Thanks for the response,
>>          > Here is the content of my spark-defaults.conf
>>          >
>>          >
>>          > # Default system properties included when running spark-submit.
>>          > # This is useful for setting default environmental settings.
>>          >
>>          > # Example:
>>          >  spark.master                     spark://fdoat:7077
>>          > # spark.eventLog.enabled           true
>>          >  spark.eventLog.dir                /home/bala/spark-logs
>>          > # spark.eventLog.dir
>>  hdfs://namenode:8021/directory
>>          > # spark.serializer
>>         org.apache.spark.serializer.KryoSerializer
>>          > # spark.driver.memory              5g
>>          > # spark.executor.extraJavaOptions  -XX:+PrintGCDetails
>>         -Dkey=value -Dnumbers="one two three"
>>          >
>>          >
>>          > regards
>>          > Bala
>>
>>
>>          >
>>          > On 2 November 2015 at 12:21, Jean-Baptiste Onofré
>>         <jb@nanthrax.net <mailto:jb@nanthrax.net>> wrote:
>>          >>
>>          >> Hi,
>>          >>
>>          >> do you have something special in conf/spark-defaults.conf
>>         (especially on the eventLog directory) ?
>>          >>
>>          >> Regards
>>          >> JB
>>          >>
>>          >>
>>          >> On 11/02/2015 07:48 AM, Balachandar R.A. wrote:
>>          >>>
>>          >>> Can someone tell me at what point this error could come?
>>          >>>
>>          >>> In one of my use cases, I am trying to use hadoop custom
>>         input format.
>>          >>> Here is my code.
>>          >>>
>>          >>> |valhConf:Configuration=sc.hadoopConfiguration
>>          >>>
>>
>> hConf.set("fs.hdfs.impl",classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)hConf.set("fs.file.impl",classOf[org.apache.hadoop.fs.LocalFileSystem].getName)varjob
>>          >>>
>>
>> =newJob(hConf)FileInputFormat.setInputPaths(job,newPath("hdfs:///user/bala/MyBinaryFile"));varhRDD
>>          >>>
>>
>> =newNewHadoopRDD(sc,classOf[RandomAccessInputFormat],classOf[IntWritable],classOf[BytesWritable],job.getConfiguration())valcount
>>          >>>
>>
>> =hRDD.mapPartitionsWithInputSplit{(split,iter)=>myfuncPart(split,iter)}|
>>          >>>
>>          >>> |The moment I invoke mapPartitionsWithInputSplit() method,
>>         I get the
>>          >>> below error in my spark-submit launch|
>>          >>>
>>          >>> |
>>          >>> |
>>          >>>
>>          >>> |15/10/3011:11:39WARN scheduler.TaskSetManager:Losttask
>>         0.0in stage
>>          >>> 0.0(TID
>>         0,40.221.94.235):java.io.IOException:NoFileSystemforscheme:spark
>>          >>> at
>>          >>>
>>
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)at
>>          >>>
>>
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)at
>>          >>>
>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)|
>>          >>>
>>          >>> Any help here to move towards fixing this will be of great
>> help
>>          >>>
>>          >>>
>>          >>>
>>          >>> Thanks
>>          >>>
>>          >>> Bala
>>          >>>
>>          >>
>>          >> --
>>          >> Jean-Baptiste Onofré
>>          >> jbonofre@apache.org <mailto:jbonofre@apache.org>
>>          >> http://blog.nanthrax.net
>>          >> Talend - http://www.talend.com
>>          >>
>>          >>
>>
>> ---------------------------------------------------------------------
>>          >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>         <mailto:user-unsubscribe@spark.apache.org>
>>          >> For additional commands, e-mail: user-help@spark.apache.org
>>         <mailto:user-help@spark.apache.org>
>>          >>
>>          >
>>
>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message