nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject Re: Upgrading to Hadoop 0.22.0+
Date Wed, 14 Dec 2011 15:41:36 GMT
0.21 is here, can we use this repo for Hadoop 0.21 poms? If so, how to change 
ivy stuff to do so? I can't seem to tell Ivy to fetch hadoop from here:

https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-
common/

On Wednesday 14 December 2011 16:05:07 Markus Jelsma wrote:
> Andrzej,
> 
> I cannot continue with testing migration on 0.20 because things like
> MapFileOutputFormat are missing in the new API. I cannot compile with 0.22
> because it no longer has the old mapred API. And i cannot build with 0.21
> because it is not  in maven central!?
> 
> Any help?
> 
> Thanks!
> 
> On Tuesday 13 December 2011 18:57:48 Andrzej Bialecki wrote:
> > On 13/12/2011 18:04, Markus Jelsma wrote:
> > > Hi
> > > 
> > > I did a quick test to see what happens and it won't compile. It cannot
> > > find our old mapred API's in 0.22. I've also tried 0.20.205.0 which
> > > compiles but won't run and many tests fail with stuff like.
> > > 
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > org/codehaus/jackson/map/JsonMappingException
> > > 
> > >          at
> > > 
> > > org.apache.nutch.util.dupedb.HostDeduplicator.deduplicator(HostDeduplic
> > > at or.java:421)
> > 
> > Hmm... what's that? I don't see this class (or this package) in the
> > Nutch tree. Also, trunk doesn't use JSON for anything as far as I know.
> > 
> > >          at
> > > 
> > > org.apache.nutch.util.dupedb.HostDeduplicator.run(HostDeduplicator.java
> > > :4 43)
> > > 
> > >          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > >          at
> > > 
> > > org.apache.nutch.util.dupedb.HostDeduplicator.main(HostDeduplicator.jav
> > > a: 431) Caused by: java.lang.ClassNotFoundException:
> > > org.codehaus.jackson.map.JsonMappingException
> > > 
> > >          at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >          at java.security.AccessController.doPrivileged(Native Method)
> > >          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >          at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > >          at
> > >          sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >          at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ...
> > >          4 more
> > > 
> > > I think this can be overcome but we cannot hide from the fact that all
> > > jobs must be ported to the new API at some point.
> > > 
> > > You did some work on the new API's, did you come across any cumbersome
> > > issues when working on it?
> > 
> > It was quite some time ago .. but I don't remember anything being really
> > complicated, it was just tedious - and once you've done one class the
> > other classes follow roughly the same pattern.

-- 
Markus Jelsma - CTO - Openindex

Mime
View raw message