tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy B. Merrill (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-1602) Detecting standards-non-compliant emails as message/rfc822
Date Thu, 02 Jul 2015 18:23:04 GMT

    [ https://issues.apache.org/jira/browse/TIKA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612321#comment-14612321

Jeremy B. Merrill commented on TIKA-1602:

Thank you, [~chrismattmann], [~tallison@mitre.org]  et al.!

[~tallison@mitre.org] -- got a bunch of normal headers, but also this `Status:` one. The only
possible value in my dataset (a bunch of publicly-released emails from Jeb Bush's tenure as
FL Gov) is `RO`, so the first lines of the emails who were treated improperly by Tika before
this patch was uniformly `Status: RO`. 

I'm going to check the whole dataset once I manage to download it all back down again from
storage to make sure there are no other values than `RO`.

My understanding is that some mail servers use this header internally to keep track of read
status. When emails are exported, they retain the header, and it sometimes appears first --
even though the server would never send this header over the wire. 

> Detecting standards-non-compliant emails as message/rfc822
> ----------------------------------------------------------
>                 Key: TIKA-1602
>                 URL: https://issues.apache.org/jira/browse/TIKA-1602
>             Project: Tika
>          Issue Type: New Feature
>          Components: mime
>            Reporter: Jeremy B. Merrill
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 1.10
>         Attachments: 036491.txt.zip
>   Original Estimate: 1h
>  Remaining Estimate: 1h
> Tika does not properly detect certain emails as `message/rfc822` if they're slightly
standards-non-compliant and begin with `Status: ` as the first header. I've added `Status:
` as a magic detection line in tika-mimetypes.xml. 
> This solves my problem and does not appear to cause unit test failures. I have not yet
run the tika-batch tests.
> As further information, the emails that are processed incorrectly come from dumps directly
from various US public officials' mailservers. The dumps, I believe since they're not intended
to be transmitted over the wire, sometimes are slightly non-compliant. 
> It's important to note that Tika (and the underlying library, James Mime4J) do properly
*parse* these emails, despite the non-compliant header. The problem is getting Tika to *detect*
the file as an email so that Mime4J gets chosen to parse it.
> Pull request on Github at https://github.com/apache/tika/pull/40

This message was sent by Atlassian JIRA

View raw message