nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cass Pallansch (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2464) Headers That Contain HTML Elements Are Not Parsed
Date Wed, 22 Nov 2017 12:32:00 GMT


Cass Pallansch commented on NUTCH-2464:

The problem was observed in situations where the header markup contained anchors that, in
turn, could contain other markup.  For example:

<h1><a href="{url}">Some Text</a></h1>
<h1><a href="{url}">Some <em>Text</em></a></h1>


I was attempting to commit back a proposed code change for this issue, but ran into some issues
committing the changes.  Anyway, the affected plugin is \apache\nutch\src\plugin\headings\src\java\org\apache\nutch\parse\headings\
 My proposed change was to make the static method getNodeValue() recursive so that it traversed
all of the children nodes until all the text nodes had been parsed.  My proposed update to
this method is as follows:

  protected static String getNodeValue(Node node) {
    StringBuilder buffer = new StringBuilder();

    NodeList children = node.getChildNodes();

    for (int i = 0; i < children.getLength(); i++) {
      if (children.item(i).getNodeType() == Node.TEXT_NODE) {
{color:#8eb021}      } else {
{color}      }

    // Return with stripped surplus whitespace
    Matcher matcher = whitespacePattern.matcher(buffer.toString().trim());
    return matcher.replaceAll(" ").trim();


An example of a page where we were having issues with parsing headers (H1 in particular) is
[link title|].

> Headers That Contain HTML Elements Are Not Parsed
> -------------------------------------------------
>                 Key: NUTCH-2464
>                 URL:
>             Project: Nutch
>          Issue Type: Bug
>          Components: plugin
>    Affects Versions: 2.3
>         Environment: Internal development/test environments.
>            Reporter: Cass Pallansch
>         Attachments: NUTCH-2464-complex-header.html
> Nutch does not appear to traverse the HTML elements that may be contained within header
elements (e.g., H1, H2, H3, etc. tags).  Many times there are anchors and/or <span>
tags within these elements that contain the actual text nodes that should be picked up as
the header value for indexing purposes.

This message was sent by Atlassian JIRA

View raw message