nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (NUTCH-2212) Decrease memory consumption by tuning stack size
Date Wed, 03 Feb 2016 16:25:40 GMT
Markus Jelsma created NUTCH-2212:
------------------------------------

             Summary: Decrease memory consumption by tuning stack size
                 Key: NUTCH-2212
                 URL: https://issues.apache.org/jira/browse/NUTCH-2212
             Project: Nutch
          Issue Type: Task
    Affects Versions: 1.11
            Reporter: Markus Jelsma
            Assignee: Markus Jelsma
            Priority: Minor
             Fix For: 1.12


In today's environments it is common to see a default stack size (-Xss) of 1 MB. This is ridiculous
for a fetcher running many fetcher.threads.fetch. The actual number of threads is much higher
due to parsing and running it on YARN. 

We can decrease stack usage by 75 %, 1 MB to a safe 256 kB. YARN will run out of stack size
if we set it to 128 kB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message