nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koen Smets (JIRA)" <>
Subject [jira] [Created] (NUTCH-1723) nutch updatedb fails due to avro (de)serialization issues on images
Date Tue, 04 Feb 2014 20:08:11 GMT
Koen Smets created NUTCH-1723:

             Summary: nutch updatedb fails due to avro (de)serialization issues on images
                 Key: NUTCH-1723
             Project: Nutch
          Issue Type: Bug
          Components: crawldb, parser
    Affects Versions: 2.2.1, 2.3
         Environment: - Ubuntu 12.04.3 LTS (GNU/Linux 3.2.0-36-generic x86_64)
- DataStax Community Edition Apache Cassandra 2.0.4

            Reporter: Koen Smets
             Fix For: 2.3

Running `bin/crawl` for 2 iterations using either the nutch-2.2.1 release or  the latest 2.x
checkout on a seed file containing for example and
(or any other webpage with image files with no obvious file extensions) causes to throw either
java.lang.IllegalArgument, IOException and/or OutOfBoundsExceptions in the the readFields
function of WebPageWritable:

  public void readFields(DataInput in) throws IOException {
    webPage = IOUtils.deserialize(getConf(), in, webPage, WebPage.class);

  public void write(DataOutput out) throws IOException {
    IOUtils.serialize(getConf(), out, webPage, WebPage.class);

2014-02-04 13:50:15,421 INFO  util.WebPageWritable - Try reading fields: ...
2014-02-04 13:50:15,423 ERROR util.WebPageWritable - Error - Failed to read fields:
2014-02-04 13:50:15,423 ERROR util.WebPageWritable - Error - Reading fields of the WebPage
class failed - java.lang.IllegalArgumentException
2014-02-04 13:50:15,425 ERROR util.WebPageWritable - Error - Printing stacktrace - java.lang.IllegalArgumentException

        at java.nio.Buffer.checkBounds(
        at java.nio.HeapByteBuffer.get(
        at org.apache.gora.mapreduce.FakeResolvingDecoder.readString(
        at org.apache.avro.generic.GenericDatumReader.readString(
        at org.apache.avro.generic.GenericDatumReader.readMap(
        at org.apache.gora.avro.PersistentDatumReader.readMap(
        at org.apache.gora.avro.PersistentDatumReader.readRecord(
        at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(
        at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(
        at org.apache.gora.util.IOUtils.deserialize(
        at org.apache.nutch.util.WebPageWritable.readFields(
        at org.apache.nutch.util.GenericWritableConfigurable.readFields(
        at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(
        at org.apache.hadoop.mapreduce.ReduceContext.nextKey(
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(
        at org.apache.hadoop.mapred.LocalJobRunner$

The exceptions are caused by image files that sneak through the urlfilter (no extension indicating
an image file) and that get (properly?) parsed by tika library.

Note that silently catching the thrown exceptions causes corruption of the Cassandra database,
as the deserializer reads over multiple webpage entries in the DataInput. Resulting in a loss
of several pages of other host present in the seed file.

Moreover, if one makes sure that the image pages don't end up in the DataInput written by
DBUpdateMapper, e.g. by configuring nutch-site.xml to disable the tika parser, the nutch dbupdate
finishes properly.


I highly suspect that the issues are due to gora's dependency on the outdated avro-1.3.3 library.

This message was sent by Atlassian JIRA

View raw message