nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Flowers <cflow...@onyxpoint.com>
Subject Re: [External] Re: GitHTTP out of memory error
Date Fri, 04 Sep 2015 15:31:13 GMT
Really the setting of that depends on what you are trying to do in the flow
and what else runs on your systems. I have a cluster of 4 boxes that have
12GB of Ram and I set those to 4096. That cluster runs just fine and pushes
a few hundred gigs a day at about 30% system usage. Depending on what your
custom processors are doing with the heap, I would boost that up a bit. You
should also look at ganglia or something in that family to watch for memory
leaks in the custom code. NIFI has a really good ganglia reporting task to
show you what is happening at a system level within the heap.



On Fri, Sep 4, 2015 at 11:24 AM, Whitmore, Mattie [USA] <
Whitmore_Mattie@bah.com> wrote:

> Thanks for all the speedy replies!
>
>
>    1.  I will up the heap currently it is set at: java.arg.3=-Xmx512m
>    2. I have 15 GB of memory (m3.xlarge)
>    3. I do have a number of custom processors in my workflow
>
> I’ll update ya’ll about how the increase in heap affects this issue.  I
> am also working on serializing  these HTTP calls into one processor, do
> ya’ll have any thoughts on that solution?
>
>
> *Mattie Whitmore*
>
>
> From: Corey Flowers
> Reply-To: "users@nifi.apache.org"
> Date: Friday, September 4, 2015 at 11:05 AM
> To: "users@nifi.apache.org"
> Subject: [External] Re: GitHTTP out of memory error
>
> What is your heap space set to and how much memory do you have on the box?
>
>
> On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA] <
> Whitmore_Mattie@bah.com> wrote:
>
>> Hi,
>>
>> I am 8 GetHTTP processors which run nightly, however 5 hours after these
>> run some threads out of memory:
>>
>> ERROR [Timer-Driven Process Thread-23]
>> o.a.nifi.processors.standard.GetHTTP
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c]
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session
>> due to java.lang.OutOfMemoryError: Java heap space:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11]
>> o.a.nifi.processors.standard.GetHTTP
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0]
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session
>> due to java.lang.OutOfMemoryError: Java heap space:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception:
>> java.lang.OutOfMemoryError: Java heap space
>>
>> I am running on Amazon Linux and have tuned the server as described in
>> the admin page.
>>
>> Thanks!
>>
>> *Mattie Whitmore*
>> Lead Technologist
>>
>> Booz | Allen | Hamilton
>> *________________________*
>>
>>
>>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> cflowers@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>



-- 
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

Mime
View raw message