tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Allison (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TIKA-1599) Switch from TagSoup to JSoup
Date Wed, 09 Dec 2015 22:38:11 GMT

    [ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049445#comment-15049445
] 

Tim Allison edited comment on TIKA-1599 at 12/9/15 10:37 PM:
-------------------------------------------------------------

I ran a comparison on our 80k html docs.  It looks like we move from 5.7M common English words
with TagSoup to 6.0M common English words with JSoup for those ~3,800 files where is any content
difference.  The cost of the move is only 4 exceptions, 3 of which are zip bombs.

I'm currently getting far fewer metadata items with JSoup, but that's almost certain to be
the fault of my initial implementation.

If anyone has a chance to look through contents/content_diffs.xlsx, I'd appreciate all feedback.

You can get to the extract root [here|http://162.242.228.174/extracts/].  The "A" run is "tika_1_12_20151208"
and the "B" run is "jsoup".


was (Author: tallison@mitre.org):
I ran a comparison on our 80k html docs.  It looks like we gain about 5% in common English
words if we move to JSoup at the cost of 4 new exceptions.

I'm currently getting far fewer metadata items with JSoup, but that's almost certain to be
the fault of my initial implementation.

If anyone has a chance to look through contents/content_diffs.xlsx, I'd appreciate all feedback.

You can get to the extract root [here|http://162.242.228.174/extracts/].  The "A" run is "tika_1_12_20151208"
and the "B" run is "jsoup".

> Switch from TagSoup to JSoup
> ----------------------------
>
>                 Key: TIKA-1599
>                 URL: https://issues.apache.org/jira/browse/TIKA-1599
>             Project: Tika
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 1.7, 1.8
>            Reporter: Ken Krugler
>            Assignee: Ken Krugler
>            Priority: Minor
>         Attachments: tagsoup_vs_jsoup_reports.zip
>
>
> There are several Tika issues related to how TagSoup cleans up HTML ([TIKA-381], [TIKA-985],
maybe [TIKA-715]), but TagSoup doesn't seem to be under active development.
> On the other hand I know of several projects that are now using [JSoup|https://github.com/jhy/jsoup],
which is an active project (albeit only one main contributor) under the MIT license.
> I haven't looked into how hard it would be to switch this dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message