spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Error : - No filesystem for scheme: spark
Date Mon, 02 Nov 2015 06:51:22 GMT
Hi,

do you have something special in conf/spark-defaults.conf (especially on 
the eventLog directory) ?

Regards
JB

On 11/02/2015 07:48 AM, Balachandar R.A. wrote:
> Can someone tell me at what point this error could come?
>
> In one of my use cases, I am trying to use hadoop custom input format.
> Here is my code.
>
> |valhConf:Configuration=sc.hadoopConfiguration
> hConf.set("fs.hdfs.impl",classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)hConf.set("fs.file.impl",classOf[org.apache.hadoop.fs.LocalFileSystem].getName)varjob
> =newJob(hConf)FileInputFormat.setInputPaths(job,newPath("hdfs:///user/bala/MyBinaryFile"));varhRDD
> =newNewHadoopRDD(sc,classOf[RandomAccessInputFormat],classOf[IntWritable],classOf[BytesWritable],job.getConfiguration())valcount
> =hRDD.mapPartitionsWithInputSplit{(split,iter)=>myfuncPart(split,iter)}|
>
> |The moment I invoke mapPartitionsWithInputSplit() method, I get the
> below error in my spark-submit launch|
>
> |
> |
>
> |15/10/3011:11:39WARN scheduler.TaskSetManager:Losttask 0.0in stage
> 0.0(TID 0,40.221.94.235):java.io.IOException:NoFileSystemforscheme:spark
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)|
>
> Any help here to move towards fixing this will be of great help
>
>
>
> Thanks
>
> Bala
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message