nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: puttcp
Date Wed, 16 Nov 2016 14:31:35 GMT
In 1.x back pressure happens by default so if that is what is being used it
is probably why the flow appears stopped.

In the 1.1 release this will he visually more obvious.

Thanks
Joe

On Nov 16, 2016 9:29 AM, "Bryan Bende" <bbende@gmail.com> wrote:

> Hello Raf,
>
> The message about attempting to set the socket buffer size is not really
> an error that would prevent anything from working, it is just a warning so
> that the user knows that NiFI created a new connection and tried to set the
> Socket's receive buffer to some value that was specified in the properties
> (1 MB in your case), and the OS only let it get set to some smaller value,
> which could be important for someone trying to tune various settings.
>
> This happens whenever a new connection is created, which likely happens
> your TCP servers goes down and comes back up. It can also happen during
> normal operation of the processor, PutTCP will create connections on the
> fly as needed and then close them if they have been used in the amount of
> time greater than "Idle Connection Expiration".
>
> I definitely agree it would be nice for that message to not print all the
> time though, one way to get rid of it would be to reduce the value of "Max
> Size of Socket Send Buffer" to meet what the OS is allowing, another way
> would be to configure logback.xml so that org.apache.nifi.processor.
> util.put.sender.SocketChannelSender only logged at the ERROR level, since
> this message is logged at the WARN level, but this means you could miss
> other valuable warnings.
>
> When you say "some time later it becomes worse... no flowfiles are
> generated", are you saying GenerateFlowFile is no longer generating flow
> files? when this happens do you have a lot of flow files in queues, and do
> you have back-pressure configured?
>
> Thanks,
>
> Bryan
>
>
> On Wed, Nov 16, 2016 at 5:59 AM, Raf Huys <raf.huys@gmail.com> wrote:
>
>> I'm having a simple flow GenerateFlowfile -> ReplaceText -> PutTCP which
>> should establish a TCP connection and send a small piece of text over that
>> connection every 60 seconds. This is established bij scheduling the first
>> processor as a cron job.
>>
>> This pipeline works, until I start restarting our TCP server upstream.
>>
>>
>> What can happen is that the TCP server is unavailable (due to
>> reasons...). Nevertheless, when the TCP server becomes available again, the
>> Nifi pipeline should continue doing it's job.
>>
>> However, what I observe is that the PutTCP processor start throwing the
>> following errors after the TCP became unavailable, and then available again:
>>
>> Attempted to set Socket Buffer Size to ... bytes but could only set to
>> ... bytes. You may want to consider changing the Operating System's maximum
>> receive buffer
>>
>> I find this message confusing because the messages we are sending are
>> about 10 characters wide. Also, the time between between stopping/starting
>> the TCP server is a couple of seconds, which means there is no backpressure
>> of unsent flowfiles.
>>
>> Properties of the PutTCP processor
>>
>>
>>    - Hostname localhost
>>    - Port 4001
>>    - Max Size of Socket Send Buffer 1 MB
>>    - Idle Connection Expiration 5 seconds
>>    - Connection Per FlowFile true
>>    - Outgoing Message Delimiter \r\n
>>    - Timeout 10 seconds
>>    - SSL Context Service No value set
>>    - Character Set UTF-8
>>
>> I would love some help here.
>>
>> PS: Some time later it becomes worse, as every processor in the above
>> pipeline actually stops doing anything...no flowfiles are generated, no
>> errors are thrown...
>>
>> Thanks,
>>
>> Raf Huys
>>
>
>

Mime
View raw message