nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <>
Subject [jira] [Created] (NUTCH-2212) Decrease memory consumption by tuning stack size
Date Wed, 03 Feb 2016 16:25:40 GMT
Markus Jelsma created NUTCH-2212:

             Summary: Decrease memory consumption by tuning stack size
                 Key: NUTCH-2212
             Project: Nutch
          Issue Type: Task
    Affects Versions: 1.11
            Reporter: Markus Jelsma
            Assignee: Markus Jelsma
            Priority: Minor
             Fix For: 1.12

In today's environments it is common to see a default stack size (-Xss) of 1 MB. This is ridiculous
for a fetcher running many fetcher.threads.fetch. The actual number of threads is much higher
due to parsing and running it on YARN. 

We can decrease stack usage by 75 %, 1 MB to a safe 256 kB. YARN will run out of stack size
if we set it to 128 kB.

This message was sent by Atlassian JIRA

View raw message