nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: puttcp
Date Wed, 16 Nov 2016 14:29:07 GMT
Hello Raf,

The message about attempting to set the socket buffer size is not really an
error that would prevent anything from working, it is just a warning so
that the user knows that NiFI created a new connection and tried to set the
Socket's receive buffer to some value that was specified in the properties
(1 MB in your case), and the OS only let it get set to some smaller value,
which could be important for someone trying to tune various settings.

This happens whenever a new connection is created, which likely happens
your TCP servers goes down and comes back up. It can also happen during
normal operation of the processor, PutTCP will create connections on the
fly as needed and then close them if they have been used in the amount of
time greater than "Idle Connection Expiration".

I definitely agree it would be nice for that message to not print all the
time though, one way to get rid of it would be to reduce the value of "Max
Size of Socket Send Buffer" to meet what the OS is allowing, another way
would be to configure logback.xml so that
org.apache.nifi.processor.util.put.sender.SocketChannelSender only logged
at the ERROR level, since this message is logged at the WARN level, but
this means you could miss other valuable warnings.

When you say "some time later it becomes worse... no flowfiles are
generated", are you saying GenerateFlowFile is no longer generating flow
files? when this happens do you have a lot of flow files in queues, and do
you have back-pressure configured?

Thanks,

Bryan


On Wed, Nov 16, 2016 at 5:59 AM, Raf Huys <raf.huys@gmail.com> wrote:

> I'm having a simple flow GenerateFlowfile -> ReplaceText -> PutTCP which
> should establish a TCP connection and send a small piece of text over that
> connection every 60 seconds. This is established bij scheduling the first
> processor as a cron job.
>
> This pipeline works, until I start restarting our TCP server upstream.
>
>
> What can happen is that the TCP server is unavailable (due to reasons...).
> Nevertheless, when the TCP server becomes available again, the Nifi
> pipeline should continue doing it's job.
>
> However, what I observe is that the PutTCP processor start throwing the
> following errors after the TCP became unavailable, and then available again:
>
> Attempted to set Socket Buffer Size to ... bytes but could only set to ...
> bytes. You may want to consider changing the Operating System's maximum
> receive buffer
>
> I find this message confusing because the messages we are sending are
> about 10 characters wide. Also, the time between between stopping/starting
> the TCP server is a couple of seconds, which means there is no backpressure
> of unsent flowfiles.
>
> Properties of the PutTCP processor
>
>
>    - Hostname localhost
>    - Port 4001
>    - Max Size of Socket Send Buffer 1 MB
>    - Idle Connection Expiration 5 seconds
>    - Connection Per FlowFile true
>    - Outgoing Message Delimiter \r\n
>    - Timeout 10 seconds
>    - SSL Context Service No value set
>    - Character Set UTF-8
>
> I would love some help here.
>
> PS: Some time later it becomes worse, as every processor in the above
> pipeline actually stops doing anything...no flowfiles are generated, no
> errors are thrown...
>
> Thanks,
>
> Raf Huys
>

Mime
View raw message