nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Question about performance tuning for Nifi
Date Thu, 22 Nov 2018 15:23:02 GMT
Have you increased the concurrent tasks on ListenSyslog? If not then going
from 1 to 2 would be the first thing to try.

Since UDP is connection-less, changing the offer to put or changing the
timeout would just move the problem to a different part of the stack. NiFis
code would be blocking waiting for space in the queue, meanwhile the client
would have no idea and keep sending data which would likely get lost
somewhere at the network level, but the client would have no way of knowing.

For TCP it would possibly be nice to block there since that would block all
the way back to the client connection.

On Thu, Nov 22, 2018 at 7:05 AM BINLEI XUE <kongpo0412@gmail.com> wrote:

> Hi,
>
> Recently we received some errors "Internal queue at maximum capacity, could
> not queue event" from our ListenSyslog processor
>
> Currently settings
>
>
>    - using UDP
>    - Max Size of Socket Buffer 2MB
>    - Max Batch Size 10000
>    - Max Size of Message Queue 10000
>    - Parse Message false
>
> By checking the code of EventQueue.java, we found that once the offer
> method timeout, we would lost the data.
>
>
>    1. We could increase the message queue size but it would consume more
>    memory and we not are sure about whether it would solve the problem or
> not.
>    2. Can we increase the timeout, like 2 seconds?
>    3. Can we replace offer method with put?
>
>
> Best,
>
-- 
Sent from Gmail Mobile

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message