tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Allison (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-2180) Multiple requests on Tika to extract text slows down
Date Wed, 23 Nov 2016 20:12:58 GMT

    [ https://issues.apache.org/jira/browse/TIKA-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691261#comment-15691261

Tim Allison commented on TIKA-2180:

Wait, are you throwing 22 (roughly) concurrent requests at tika-server with only 4 CPUs? 
I haven't tested the following statement, but that seems like a bad idea.  Typically you want
to limit your # of calls to roughly the number of CPUs you have.

Take a look at PDFBOX-3581 for another user who tried the same in-memory with PDFBox.  The
result was really bad performance (no docs parsed!!!).  See esp [~tilman]'s recommendation
to use {{Executors.newFixedThreadPool}}.  Limit the number of threads to ~4, and you should
be good.

What I did in tika-batch was have one thread that crawled the directory structure and put
files on an ArrayBlockingQueue, then each parser thread pulled a file off that queue and processed
it.  This is a simple design pattern, but it takes a not insubstantial amount of code to get
it right.

Good luck!  Let me know if I've misunderstood what you're doing.

> Multiple requests on Tika to extract text slows down
> ----------------------------------------------------
>                 Key: TIKA-2180
>                 URL: https://issues.apache.org/jira/browse/TIKA-2180
>             Project: Tika
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 1.13, 1.14
>         Environment: Windows OS, Open JDK, 4 core 32 GB RAM
>            Reporter: Ashish Basran
> I observed that if I send multiple requests to Tika (eg. http://localhost:8080/tika)
with around 5MB files, Tika is very slow in completing the action. I tried with ~20 random
files, it took 170 seconds to process all the files in sequence. If I pass all files in parallel,
it took around 780 seconds to process same set of files. 

This message was sent by Atlassian JIRA

View raw message