nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raf Huys <raf.h...@gmail.com>
Subject Re: puttcp
Date Wed, 16 Nov 2016 15:00:09 GMT
I was wrong about it not resuming after connection reset. It does.

I just overlooked, and as I don't have rights to comment in Nabble, I had
to wait for an email to reply on...

Anyway, thanks for clearing out the socket buffer thing!

On Wed, Nov 16, 2016 at 3:31 PM, Joe Witt <joe.witt@gmail.com> wrote:

> In 1.x back pressure happens by default so if that is what is being used
> it is probably why the flow appears stopped.
>
> In the 1.1 release this will he visually more obvious.
>
> Thanks
> Joe
>
> On Nov 16, 2016 9:29 AM, "Bryan Bende" <bbende@gmail.com> wrote:
>
>> Hello Raf,
>>
>> The message about attempting to set the socket buffer size is not really
>> an error that would prevent anything from working, it is just a warning so
>> that the user knows that NiFI created a new connection and tried to set the
>> Socket's receive buffer to some value that was specified in the properties
>> (1 MB in your case), and the OS only let it get set to some smaller value,
>> which could be important for someone trying to tune various settings.
>>
>> This happens whenever a new connection is created, which likely happens
>> your TCP servers goes down and comes back up. It can also happen during
>> normal operation of the processor, PutTCP will create connections on the
>> fly as needed and then close them if they have been used in the amount of
>> time greater than "Idle Connection Expiration".
>>
>> I definitely agree it would be nice for that message to not print all the
>> time though, one way to get rid of it would be to reduce the value of "Max
>> Size of Socket Send Buffer" to meet what the OS is allowing, another way
>> would be to configure logback.xml so that org.apache.nifi.processor.util
>> .put.sender.SocketChannelSender only logged at the ERROR level, since
>> this message is logged at the WARN level, but this means you could miss
>> other valuable warnings.
>>
>> When you say "some time later it becomes worse... no flowfiles are
>> generated", are you saying GenerateFlowFile is no longer generating flow
>> files? when this happens do you have a lot of flow files in queues, and do
>> you have back-pressure configured?
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Wed, Nov 16, 2016 at 5:59 AM, Raf Huys <raf.huys@gmail.com> wrote:
>>
>>> I'm having a simple flow GenerateFlowfile -> ReplaceText -> PutTCP which
>>> should establish a TCP connection and send a small piece of text over that
>>> connection every 60 seconds. This is established bij scheduling the first
>>> processor as a cron job.
>>>
>>> This pipeline works, until I start restarting our TCP server upstream.
>>>
>>>
>>> What can happen is that the TCP server is unavailable (due to
>>> reasons...). Nevertheless, when the TCP server becomes available again, the
>>> Nifi pipeline should continue doing it's job.
>>>
>>> However, what I observe is that the PutTCP processor start throwing the
>>> following errors after the TCP became unavailable, and then available again:
>>>
>>> Attempted to set Socket Buffer Size to ... bytes but could only set to
>>> ... bytes. You may want to consider changing the Operating System's maximum
>>> receive buffer
>>>
>>> I find this message confusing because the messages we are sending are
>>> about 10 characters wide. Also, the time between between stopping/starting
>>> the TCP server is a couple of seconds, which means there is no backpressure
>>> of unsent flowfiles.
>>>
>>> Properties of the PutTCP processor
>>>
>>>
>>>    - Hostname localhost
>>>    - Port 4001
>>>    - Max Size of Socket Send Buffer 1 MB
>>>    - Idle Connection Expiration 5 seconds
>>>    - Connection Per FlowFile true
>>>    - Outgoing Message Delimiter \r\n
>>>    - Timeout 10 seconds
>>>    - SSL Context Service No value set
>>>    - Character Set UTF-8
>>>
>>> I would love some help here.
>>>
>>> PS: Some time later it becomes worse, as every processor in the above
>>> pipeline actually stops doing anything...no flowfiles are generated, no
>>> errors are thrown...
>>>
>>> Thanks,
>>>
>>> Raf Huys
>>>
>>
>>


-- 
Mvg,

Raf Huys

Mime
View raw message