nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris A. Mattmann (JIRA)" <>
Subject [jira] Updated: (NUTCH-562) Port mime type framework to use Tika mime detection framework
Date Sun, 07 Oct 2007 15:32:50 GMT


Chris A. Mattmann updated NUTCH-562:

    Attachment: NUTCH-562.Mattmann.patch.txt

Initial patch for comments:

1. This patch removes the MimeType system, and its associated java src files, config files
and unit tests from Nutch. This information is in Tika now and is replaced by its TIka counterparts.
2. This patch uses the unreleased 0.1-dev version of Tika. When 0.1 is officially released,
we can convert to that, though I don't anticipate any MimeType API changes between now and
3. All unit tests for core and plugins pass, however, it's probably a good idea to run at
least a small crawl with this patch and see if everything works fine. I don't really have
the time for this now, so anyone want to try? (cough cough Dougacan cough cough ;) )
4. It's worth noting that this MimeType system from Tika changes the traditional Nutch mime
type system (IMO for the better) in a couple of ways. First, whereas the old MimeType system
was very happy to return null in places where it couldn't figure out the MimeType, this system
tries to return a "default" MimeType (which in this case is "application/octet-stream") if
it can't guess the mime type from those that it knows about. Second, this mime type system
uses a different type of XML repo file -- based on the one available from's
shared MIME package. 

Okay, so if someone gets a chance please run a small crawl with this in the next few days
and let us know how it works. Otherwise, I'll do the same myself in a couple days and if there
are no objections, I'd like to commit this then.

> Port mime type framework to use Tika mime detection framework
> -------------------------------------------------------------
>                 Key: NUTCH-562
>                 URL:
>             Project: Nutch
>          Issue Type: Improvement
>          Components: mime_type_detector
>    Affects Versions: 1.0.0
>         Environment: Mac Book Pro, Intel Core Duo 2.0 Ghz, 2.0 GB RAM, Mac OS X 10.4
although improvement is indep of env
>            Reporter: Chris A. Mattmann
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>         Attachments: NUTCH-562.Mattmann.patch.txt
> With Tika ( nearing  a stable 0.1 release candidate,
I think it would be a good time to patch Nutch to use Tika's mime detection system (an improvement
over the existing Nutch one written primarily by Jerome). Tika's mime system is based on the
mime system from and includes several improvements over the existing Nutch
mime system such as:
> 1. reliable XML-based content detection (a clear issue plaguing Nutch for some time now),
ability to delineate between RSS, XML, ATOM, etc.
> 2. mime magic pattern matching, including support for multiple patterns
> 3. glob pattern matches (ability to support > 1)
> I'll get together a patch and then attach it to the list once it's relatively stable.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message