nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject Re: Upgrading to Hadoop 0.22.0+
Date Wed, 14 Dec 2011 15:58:50 GMT
Hi,

I added Jackson as a dependency and can now build Nutch with Hadoop 
0.20.205.0. Hadoop needs it. Should we commit this? I'd prefer migrating to 
that version before doing all API migrations.

Nutch runs fine with 205 locally and also on a 0.20.203 cluster.

Thanks

On Tuesday 13 December 2011 18:57:48 Andrzej Bialecki wrote:
> On 13/12/2011 18:04, Markus Jelsma wrote:
> > Hi
> > 
> > I did a quick test to see what happens and it won't compile. It cannot
> > find our old mapred API's in 0.22. I've also tried 0.20.205.0 which
> > compiles but won't run and many tests fail with stuff like.
> > 
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/codehaus/jackson/map/JsonMappingException
> > 
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.deduplicator(HostDeduplicat
> > or.java:421)
> 
> Hmm... what's that? I don't see this class (or this package) in the
> Nutch tree. Also, trunk doesn't use JSON for anything as far as I know.
> 
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.run(HostDeduplicator.java:4
> > 43)
> > 
> >          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >          at
> > 
> > org.apache.nutch.util.dupedb.HostDeduplicator.main(HostDeduplicator.java:
> > 431) Caused by: java.lang.ClassNotFoundException:
> > org.codehaus.jackson.map.JsonMappingException
> > 
> >          at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >          at java.security.AccessController.doPrivileged(Native Method)
> >          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >          at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >          ... 4 more
> > 
> > I think this can be overcome but we cannot hide from the fact that all
> > jobs must be ported to the new API at some point.
> > 
> > You did some work on the new API's, did you come across any cumbersome
> > issues when working on it?
> 
> It was quite some time ago .. but I don't remember anything being really
> complicated, it was just tedious - and once you've done one class the
> other classes follow roughly the same pattern.

-- 
Markus Jelsma - CTO - Openindex

Mime
View raw message