nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirby Bohling <>
Subject The Future of Nutch, reactivated
Date Fri, 15 May 2009 00:23:31 GMT

Sorry that I didn't reply, and thus this isn't threaded properly.
I've lurked on the list via the RSS feed, I subscribed so I could put
in my two cents worth.  I've recently starting using git to maintain a
local branch of Nutch.  My hope is to get my employer to let me
contribute "just engineering" back to Nutch.  We'd like to customize
Nutch in various ways and use that as the basis of internal R&D and
potentially some products that we'd not contribute.  The other things
that just make Nutch more flexible I'd like to contribute to.

I've been working with Nutch on and off since sometime in November or
so for my job.  A couple of thoughts:

1. Nutch is too monolithic
2. Nutch does the heavy lifting of a framework for a distributed system well.
3. Nutch doesn't really keep all the various pieces up to date very well.
4. Nutch requires at least a Bachelors in Nutch to deal with it.
5. Documentation in a Wiki is out of date or is hard to tell which
versions various things work with.
6. Nutch isn't very friendly to simple requests if there a complex
hack could be found. (See recursive file:// handling).

My most recent task was actually to update Tika to use 0.3 and then
use the Tika parsing of the docx format to index.  There were a
several interesting problems, but I want to get permission from my
employer and just show the patches.

I thing we fall into the category of #2 (we wish we could fall into
category #1, but such is life).  We want to make our intranet
searchable on a large scale, and would like to apply the indexing and
retrieval in a number of R&D projects.  We also have an interest in
using Nutch/Lucene/Hadoop in a number of other problems unrelated to
Internet Search.

A couple of things that I'd like to help do (or see done) that would
make Nutch far more framework like so I can assemble the pieces and
parts into what I need:

1. Get Nutch and it's various components into a public Maven
repository, and have public scripts to do the publishing.  Don't care
if that is via Ant with Ivy extensions, or switching to a Maven build
systems.  I've actually started with both approaches.  I'm much better
with Maven, but I think Ivy is more likely to be acceptable to the
project.  I'd like to see this done with Hadoop, and any other core
components.  For now, I'm just maintaining a local POM file that
pushes my builds into our local Maven repository.  I'm going to do
this one way or another, and would love to hear any feedback on an
approach that is acceptable to be contributed back to Nutch.

2. Clearly segregate "Plugins" from "Core" from "Bits that make it an
Application".  I've had fun problems with ClassLoaders, and it seems
that the interface Plugins are allowed to access are "Anything in
Core, or it's existing libraries".  It would seem that it would be
better to have the Core Runtime, which plugins can depend upon, and is
relatively minimal.  Identify the pieces of Nutch which are there to
make it into a program you can run, and push those into a separate
place.  For API's with multiple implementations, it would be nice to
not have be forced to use the same one the Core does when a plugin is

3. As you stated earlier, use OSGi for a plugin system and some type
of dependency injection rather then hand parsed XML files.  I've had
problems with the PluginClassloader (I wanted to use Tika in my
plugin, and because of the plugin/classloader setup, I had to push the
POI libraries into the lib directory rather then in the
src/plugin/plugin-XXX/lib directory).  Well, that was the first
approach, the second was to hack the PluginClassloader to not delegate
to the parent for the "org.apache.tika" package and then provide Tika
in the plugin and it all worked.  Using an well known plug-in system
would have made this much easier.

4. Help transition to using the 3rd party libraries, Nutch still has
an SWF parser that went unmaintained in 2002.  Flash has moved a long
way, it would seem sensible to either jettison that code, or update to
newer versions of the same library by the same project (SWF2).  Not
that I care about Flash, but it seems that parsing isn't something
Nutch proper is focused on.

5. With whatever build system is chosen, figure out how to setup a
Maven build to construct "Out-of-Tree" Nutch plugins without having to
manually deal with all of the various dependencies and packaging

6. Better support for running out of an IDE.  The instructions work,
and are very helpful.  It'd be much nicer to see the use of tools or
scripts to generate a saner system then is currently there (having
each plugin be a project in Eclipse would be a huge help to debugging
weird classpath issues).  Right now, running and compiling inside of
Eclipse isn't at all similar to running it outside, if you have any
time of classloader issues, or multiple conflicting libraries.  Not
that there are any in-tree right now, but I can see how future ones
could exist.

7. Make each plugin be it's own deliverable (even if they are all
maintained inside one tree).  Including the ability to assemble a
".job" from the various internal and external components via Ant
Tasks, or Maven plugins.  It's very tedious to maintain anything
outside of the tree as of right now.

8. Try and reduce the number of "All or Nothing" decisions.  See
NUTCH-407, in order to deal with a straightforward problem, the
suggestion is two switch from Black List out to White List in.  That's
a really significant change to deal with a relatively simple problem.
Especially given that I listed a specific directory I wanted to have
indexed.  If I wanted a higher level to be indexed, I'd have listed
that higher level in my list of seeds.  There are other similar
issues, like the handling of ' ' in URL.  I used the URL Filter
Normalizer to fix that, but I can't see how to apply it to http://,
but not file://.  The handling of the two is very different.  I had
similar problems with "mime.magic".  I can either have it on, or off.
It'd be great if there were much finer granularity there.  The patch
in NUTCH-407 was the first thing applied to my local branch, because
having to remember to modify the urlfilter configuration seems like a
real hassle when I just want to add a new directory.

9. Stop treating Windows as a second class citizen for local-only
configurations.  While it'll work with Cygwin, it is problematic for a
number of my deployments to have to have them install Cygwin.  It'd be
great if we could just write a small abstraction that on Windows
platforms did nothing.  In a non-distributed mode, you can write a
no-op program that does nothing, as long as it returns the proper
error code, it all works fine.  It'd be great if at least that much
could be made to work in Windows without Cygwin.  If I remember right,
there are 3-4 programs (chown, chgrp, whoami, groups?) I think?  They
can do nothing and return garbage and it'll all just work.  I did that
as a cheaters way of getting out of the Cygwin dependency.  I run
Linux on my desktop for 9 of the last 10 years, but it's a simple fact
that to be adopted by many smaller deployments, Window's support is
critical.  I'm going to have to maintain that.  I'm sure others will
too, it'd be really nice if we could push maintain just one copy of it
in the tree.

Anyways, from my perspective, I'd like to help contribute solutions to
various problems.  Nutch is a great concept and even much of the
implementation, but it right now it's just really hard to use from the


View raw message