tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris A. Mattmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-1331) Find/configure a vm and gather initial corpus
Date Thu, 26 Feb 2015 04:00:07 GMT

    [ https://issues.apache.org/jira/browse/TIKA-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337832#comment-14337832
] 

Chris A. Mattmann commented on TIKA-1331:
-----------------------------------------

Hey Tim, this is an interesting option. I like insulated code and having this capability in
tika would be nice. At the same time, if you see value in a "merging of the minds", I finished
my ImageCatalog thingee with OODT. You can find it here:

https://github.com/chrismattmann/imagecat/

The idea is:

1. Start from a LONG file list (aka 1 million full file paths, or 40million)
2. Chunk it up into smaller file lists
3. For each chunk; curl POST to SolrCell (include Tika 1.7+OCR+Tesseract+EXIF and other metadata)
4. Monitor progress in OODT and Solr

Happy to hear your feedback on this and how it melds with Tika-batch. At the end of the day
you did the work for batch and if you'd like to merge it I'd support it. I wouldnt support
bringing in an ImageCat capability into Tika - it's def. a downstream app.

Another one to check out is DRAT (a distributed version of Release Audit Tool, using OODT):

https://github.com/chrismattmann/drat/



> Find/configure a vm and gather initial corpus
> ---------------------------------------------
>
>                 Key: TIKA-1331
>                 URL: https://issues.apache.org/jira/browse/TIKA-1331
>             Project: Tika
>          Issue Type: Sub-task
>          Components: cli, general, server
>            Reporter: Tim Allison
>            Assignee: Tim Allison
>         Attachments: tika-eval-vm-setup.tar.bz2
>
>
> Let's start with govdocs1 for this issue unless there are other easy options.  Going
forward, we'll want and need to add a more diverse set of documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message