nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Johnsson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NUTCH-1461) Problem with TableUtil
Date Tue, 28 Aug 2012 01:24:07 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442876#comment-13442876
] 

Christian Johnsson edited comment on NUTCH-1461 at 8/28/12 12:22 PM:
---------------------------------------------------------------------

I did a small workaround just to keep my stuff going.
Not right and will still keep trash in the db but atleast it keeps running and the 0.001%
thats wrong i'll have to live with so far.. until me or someone else figures out a neat way
to handle it.

in /org/apache/nutch/util/TableUtil.java

Change line 98:


buf.append(splits[1]); // add protocol

to

if (splits.length == 1) 
       {
        buf.append("http"); // Add something to the array so it doesnt crash.
       }
else
      {
       buf.append(splits[1]); // add protocol
      }

It would be nice with a tool that harvested trough the hbase and removed all urls that wheren't
legit.
Just in case that someone like me doesn't bother to edit regex-urlfilter.txt to only allow
real urls.
Might be a good idea to have that in the default regex-urlfilter that takes only valid domains.



                
      was (Author: mr.johnsson):
    I did a small workaround just to keep my stuff going.
Not right and will still keep trash in the db but atleast it keeps running and the 0.001%
thats wrong i'll have to live with so far.. until me or someone else figures out a neat way
to handle it.

in /org/apache/nutch/util/TableUtil.java

Change line 98:

- buf.append(splits[1]); // add protocol

to

if (splits.length == 1) 
       {
        buf.append("http"); // Add something to the array so it doesnt crash.
       }
else
      {
       buf.append(splits[1]); // add protocol
      }

It would be nice with a tool that harvested trough the hbase and removed all urls that wheren't
legit.
Just in case that someone like me doesn't bother to edit regex-urlfilter.txt to only allow
real urls.
Might be a good idea to have that in the default regex-urlfilter that takes only valid domains.



                  
> Problem with TableUtil
> ----------------------
>
>                 Key: NUTCH-1461
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1461
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: nutchgora
>         Environment: Debian / CDH3 / Nutch 2.0 Release
>            Reporter: Christian Johnsson
>
> Affects parse and updatedb and parse.
> Think i got some missformated urls into hbase but i can't fin them.
> It generates this error though. If i empty hbase and restart it goes for a couple of
million pages indexed then it comes up again. Any tips on how to locate what row in the table
that genereates this error?
> 2012-08-28 01:48:10,871 WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.ArrayIndexOutOfBoundsException: 1
> 	at org.apache.nutch.util.TableUtil.unreverseUrl(TableUtil.java:98)
> 	at org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:102)
> 	at org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:76)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:260)
> 2012-08-28 01:48:10,875 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the
task

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message