spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark Streaming writing to HDFS
Date Sun, 05 Oct 2014 13:39:03 GMT
On Sat, Oct 4, 2014 at 5:28 PM, Abraham Jacob <abe.jacobs@gmail.com> wrote:
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.io.IntWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

Good. There is also a org.apache.hadoop.mapred.TextOutputFormat, which
is the "old" Hadoop API, and would not work with the method requiring
"new" API classes. Given the identical class names it is a common
error, but that's not the case here.


> I have no experience in Scala.  What does this mean in Scala -
> Class[_ <: NewOutputFormat[_, _]]

It means something like "<? extends NewOutputFormat<?,?>>" in Java.
Here, NewInputFormat is just a renamed import of the new Hadoop API
InputFormat. All is well here.

I can't see the rest of your code to look more closely, but it must be
a simple yet subtle compile error. Scrutinize what is different
between your working and non-working code. There should no real
gotchas here.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message