nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davy De Waele <ddewa...@gmail.com>
Subject Re: PutTCP connector not cleaning up dangling connections
Date Wed, 20 Sep 2017 20:15:03 GMT
The issue is on a CentOS linux server, running Nifi 1.4.0 snapshot. There
it communicates with a dozen of Moxa nports over tcp on various remote
sites. All moxa's are connected over a (slow) cellular connection.

Will do a tcpdump with the new / old PutTCP processor.

On Wed, 20 Sep 2017 at 22:10, Bryan Bende <bbende@gmail.com> wrote:

> The sender/output stream can only ever be in use by one thread since
> it acquires it from the pool at the beginning of onTrigger, which
> makes it unavailable to any other threads until it is placed back in
> the pool.
>
> The flush is there just because now it is writing to an output stream
> and we may be leaving the sender/output stream open depending if
> "connection per flow file" is true or not.
>
> In the case where you do produce the issue, where is NiFi running? Is
> it sending from NiFi on your Mac to the device on Linux?
>
> On Wed, Sep 20, 2017 at 3:45 PM, Davy De Waele <ddewaele@gmail.com> wrote:
> > Don't get it either... new code looks more or less the same. been trying
> > various things all day. Glad I got it working with a "patched" PutTCP.
> > Better from what I got but still not ideal.
> >
> > Can it be a threading issue (we interact with multiple modules) in the
> input
> > / output stream handling ? Also noticed a flush that wasn't there before.
> > Are there other places where charsets can be set ?
> >
> > Been thinking about what could cause the tcp server to react this way
> but no
> > idea. Can reproduce it easily. Will check if I can enable additional
> logging
> > on the moxa.
> >
> > On Wed, 20 Sep 2017 at 21:36, Bryan Bende <bbende@gmail.com> wrote:
> >>
> >> For what it is worth, I can't reproduce this on my Mac laptop either...
> >>
> >> Setup GenerateFlowFile with same sample text your provided, followed
> >> by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same
> >> NiFi instance.
> >>
> >> If I look at the content of the flow files coming out of ListenTCP,
> >> and I put the content viewer in Hex mode, I see:
> >>
> >> 0x00000000 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND..
> >>
> >> The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r  which
> >> would make sense since ListenTCP read-off the \r\n at the end.
> >>
> >>
> >> On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende <bbende@gmail.com> wrote:
> >> > I'll keep looking at this, but just to summarize the changes that were
> >> > made in case something jumps out to someone else...
> >> >
> >> > The changes were to avoid reading the entire content of the flow file
> >> > into memory.
> >> >
> >> > Previous Behavior:
> >> > - Create a ByteArrayOuputStream
> >> > - Use an InputStreamCallback to read entire bytes of flow file, and
> >> > write them to above output stream
> >> > - Get value of the outgoing delimiter property
> >> > - If delimiter is not null, then get the bytes of the delimiter using
> >> > the charset configured in the processor, and write the delimiter bytes
> >> > to the above output stream
> >> > - Get byte array from the ByteArrayOutputStream and pass it to the
> >> > channel sender, which in turns calls write on a SocketChannel
> >> >
> >> > New behavior:
> >> > - Get the direct SockeOutputStream from the sender
> >> > - Get the InputStream of the flow file
> >> > - Call IOUtils.copy to copy the flow file input stream to the socket
> >> > output stream in chunks
> >> > - Perform exact same delimiter logic as above
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele <ddewaele@gmail.com>
> >> > wrote:
> >> >> Correct about the delimiter.
> >> >> Think character set is (was) automatically filled in with utf-8.
> (Never
> >> >> noticed a default option)
> >> >>
> >> >> On Wed, 20 Sep 2017 at 19:11, Bryan Bende <bbende@gmail.com>
wrote:
> >> >>>
> >> >>> Thanks for the info and for digging into this..
> >> >>>
> >> >>> Just to confirm, in your PutTCP configuration, you are saying that
> you
> >> >>> have "Outgoing Message Delimiter" set to \r\n  ?
> >> >>>
> >> >>> And do you have the "Character Set" property set to the default
of
> >> >>> "UTF-8"
> >> >>> ?
> >> >>>
> >> >>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele <ddewaele@gmail.com>
> wrote:
> >> >>> > I've put up a git repo that I use to test the PutTCP behavior.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > (
> https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1
> )
> >> >>> >
> >> >>> > What I've done.
> >> >>> >
> >> >>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my
base
> nifi
> >> >>> > installation
> >> >>> > - Noticed weird characters in the tcp responses
> >> >>> > - Switched to 1.3.0. No more weird characters
> >> >>> > - Isolated the PutTCP processor by creating a custom PutTCP
(see
> >> >>> > github
> >> >>> > repo
> >> >>> > below) to experiment (instead of the built-in PutTCP)
> >> >>> > - Repo has 2 branches. master contains an "older" PutTCP processor
> >> >>> > from
> >> >>> > the
> >> >>> > Nifi code base. With that one I don't have the weird characters
> >> >>> > (even on
> >> >>> > a
> >> >>> > nifi 1.4.0-SNAPSHOT + your PR).
> >> >>> > - If I use the dev_puttcp_1.4 branch on my nifi I get the
weird
> >> >>> > characters.
> >> >>> >
> >> >>> >
> >> >>> > Some info on my setup :
> >> >>> >
> >> >>> > I'm sending commands to PutTCP using a GenerateFlowFile processor.
> >> >>> > payload = ${literal('\r')}@COMMAND${literal('\r')}
> >> >>> > PutTCP adds a delimier (\r\n)
> >> >>> >
> >> >>> > Regarding provenance. PutTCP adds the delimiter to the payload
> >> >>> > before
> >> >>> > sending it to the TCP socket. That  delimiter is not in the
> flowfile
> >> >>> > that is
> >> >>> > put in provenance I think.
> >> >>> >
> >> >>> > Cannot reproduce the issue on my mac with my mock tcp server.
But
> on
> >> >>> > the
> >> >>> > actual environment (Linux CentOS) with the Moxa devices, there
is
> a
> >> >>> > clear
> >> >>> > difference in behaviour. Will see if I can tunnel my through
the
> >> >>> > devices
> >> >>> >
> >> >>> > All the info I have right now ....
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Mime
View raw message