spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Spark Streaming writing to HDFS
Date Sun, 05 Oct 2014 13:39:03 GMT
On Sat, Oct 4, 2014 at 5:28 PM, Abraham Jacob <> wrote:
> import org.apache.hadoop.conf.Configuration;
> import;
> import;
> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

Good. There is also a org.apache.hadoop.mapred.TextOutputFormat, which
is the "old" Hadoop API, and would not work with the method requiring
"new" API classes. Given the identical class names it is a common
error, but that's not the case here.

> I have no experience in Scala.  What does this mean in Scala -
> Class[_ <: NewOutputFormat[_, _]]

It means something like "<? extends NewOutputFormat<?,?>>" in Java.
Here, NewInputFormat is just a renamed import of the new Hadoop API
InputFormat. All is well here.

I can't see the rest of your code to look more closely, but it must be
a simple yet subtle compile error. Scrutinize what is different
between your working and non-working code. There should no real
gotchas here.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message