sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: 答复: sqoop generate java class in old mapreduce API?
Date Mon, 12 Aug 2013 15:18:46 GMT
Hi sir,
Sqoop generated class implements Hadoop interface called Writable, but do not implements interface
WritableComparable. Whereas implementing Writable interface is sufficient for serialization
it's not sufficient for keys that are serialized from mappers. Instead interface WritableComparable
is needed in such case. My guess is that the generated class is used in a mapper in a job
with reduce phase that is violating this requirement. Would you mind sharing a small code
example that reproduces the issue? (e.g. multiple java classes that together put whole mapreduce
job).

Jarcec

On Mon, Aug 12, 2013 at 09:23:20PM +0800, Wu YongBo wrote:
> I used classes generated by sqoop as Key-Value, and set OutputFormat to
> SequenceFileOutputFormat, and the error occurred.
> However, when I simply changed the Key-Value to Text, and OutputFormat to
> TextOutputFormat, it worked!
> 
>     SqoopGenClass value = ....;
> 
>     context.write(value, NullWritable.get());
> =====>>>>
> 	context.write(new Text(value.toString()), NullWritable.get());
> 
> Our hadoop version is 0.20.2,that's too old so I gusss this is the cause.
> 
> -----邮件原件-----
> 发件人: Jarek Jarcec Cecho [mailto:jarcec@apache.org] 
> 发送时间: 12 August 2013 08:35
> 收件人: user@sqoop.apache.org
> 主题: Re: sqoop generate java class in old mapreduce API?
> 
> Hi Yongbo,
> I believe that the writable itself should be independent on the new vs. old
> mapreduce API. Would you mind sharing more details about your use case?
> Maybe a small code example reproducing the issue could help us move forward.
> 
> Jarcec
> 
> On Thu, Aug 08, 2013 at 09:10:50PM +0800, Wu YongBo wrote:
> > Hi All,
> > 
> >  
> > 
> >   I am using a java class generated by sqoop(1.4.2),  and I get the 
> > following error:
> > 
> >  
> > 
> > 2013-08-08 20:43:10,854 WARN org.apache.hadoop.mapred.TaskTracker: 
> > Error running child
> > 
> > java.lang.ClassCastException: class 
> > com.a.semv.io.semreportbase_keyword
> > 
> >         at java.lang.Class.asSubclass(Class.java:3018)
> > 
> >         at
> > org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:5
> > 99)
> > 
> >         at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:7
> > 91)
> > 
> >         at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.jav
> > a:524)
> > 
> >         at 
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> > 
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > 
> >         at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > 
> >  
> > 
> > My job is written in mapreduce new API.
> > 
> > My hadoop version:0.20.2
> > 
> >  
> > 
> > Thanks,
> > 
> > Yongbo
> > 
> >  
> > 
> 

Mime
View raw message