nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject Re: Upgrading to Hadoop 0.22.0+
Date Wed, 14 Dec 2011 15:05:07 GMT
Andrzej,

I cannot continue with testing migration on 0.20 because things like 
MapFileOutputFormat are missing in the new API. I cannot compile with 0.22 
because it no longer has the old mapred API. And i cannot build with 0.21 
because it is not  in maven central!?

Any help?

Thanks!


On Tuesday 13 December 2011 18:57:48 Andrzej Bialecki wrote:
> On 13/12/2011 18:04, Markus Jelsma wrote:
> > Hi
> > 
> > I did a quick test to see what happens and it won't compile. It cannot
> > find our old mapred API's in 0.22. I've also tried 0.20.205.0 which
> > compiles but won't run and many tests fail with stuff like.
> > 
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/codehaus/jackson/map/JsonMappingException
> > 
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.deduplicator(HostDeduplicat
> > or.java:421)
> 
> Hmm... what's that? I don't see this class (or this package) in the
> Nutch tree. Also, trunk doesn't use JSON for anything as far as I know.
> 
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.run(HostDeduplicator.java:4
> > 43)
> > 
> >          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.main(HostDeduplicator.java:
> > 431) Caused by: java.lang.ClassNotFoundException:
> > org.codehaus.jackson.map.JsonMappingException
> > 
> >          at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >          at java.security.AccessController.doPrivileged(Native Method)
> >          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >          at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >          ... 4 more
> > 
> > I think this can be overcome but we cannot hide from the fact that all
> > jobs must be ported to the new API at some point.
> > 
> > You did some work on the new API's, did you come across any cumbersome
> > issues when working on it?
> 
> It was quite some time ago .. but I don't remember anything being really
> complicated, it was just tedious - and once you've done one class the
> other classes follow roughly the same pattern.

-- 
Markus Jelsma - CTO - Openindex

Mime
View raw message