spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Kumar <>
Subject Reading custom inputformat from hadoop dfs
Date Mon, 28 Oct 2013 08:52:25 GMT

I am trying to read some custom sequence file from hadoop file system. The
CustomInputFormat Class implements InputFormat<WritableComparable,
Writable> . I am able to read the file in JavaRDD as follows

JobConf job = new JobConf();

FileInputFormat.setInputPaths(job, new Path(input));

JavaPairRDD<WritableComparable, Writable> rdd =
spark.hadoopRDD(job, CustomInputFormat.class, WritableComparable.class,

But I want to read directly from scala api, I am trying as follows

val job = new JobConf();

FileInputFormat.setInputPaths(job, new Path(input));

spark.hadoopRDD(job, classOf[CustomInputFormat],
classOf[WritableComparable[Object]], classOf[Writable]);

I am getting the following error:

[error] argument expression's type is not compatible with formal parameter

[error]  found   : java.lang.Class[CustomInputFormat]

[error]  required: java.lang.Class[_ <:

But my CustomInputFormat Class implements InputFormat<WritableComparable,
Writable>. Is the generics causing the compilation problem?
WritableComparable expects a type parameter.

View raw message