mina-ftpserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Taylor <kenneth.tay...@dataexpress.com>
Subject RE: FTP Data Channels Through Reverse Proxy
Date Thu, 22 Aug 2019 20:14:39 GMT
David,

When you say "a proxy that understands FTP", there is no such proxy.  Remember we are doing
reverse proxy, not proxy.  And the proxy we are using is one we wrote.  And we do have to
tell it when to close the data channel.  There is nothing to magically do that.  We have now
changed our server Ftplet.afterCommand() to look for the 226 reply and to call the proxy to
close the data chnnel. So that part is fixed.

But the client still fails on MLSD response. It does not understand what is coming from the
proxy I guess, but its identical to what's coming from the server.  So the only thing that
could be causing the problem is that Apache is closing the command channel, which we see it
doing, and it should not be doing that.  We set all the timeouts to large values so that’s
not it.

I think we've stumbled upon a bug.  We've been debugging this for 6 weeks so we didn't come
to this conclusion lightly.  Could there be a hard coded timeout that we just happen to be
tripping because the proxy imparts a small delay?   I did see in the debugger a write timeout
that users don't have access to and it was set to 60.  Our entire test session only lasts
about 3 seconds.

Our proxy is very simple. Its just a client side socket and a server side socket with stream
copiers for the 4 streams.  Nothing magical.  It should be completely transparent.  The stream
copiers are doing line-based IO to match FTP with auto-flush on.  We know the socket close
logic is at the heart of the problem.  Apache is closing the command socket and I'll bet there
is some internal hidden error causing that.

Thanks.
Ken

-----Original Message-----
From: David Latorre <dvlato@gmail.com>
Sent: Thursday, August 22, 2019 12:51 PM
To: ftpserver-users@mina.apache.org
Subject: Re: FTP Data Channels Through Reverse Proxy

From your first line I understand that the problem is with the proxy that's not closing the
data channel, isn't it?

I guess that for a proxy that understands FTP ,you shouldn't do anything to ensure the proxy
closes the channel when the transfer is done.

It's not clear from your message but I would think that if the connection gets closed it will
be due to some inactivity timeout being reached. Other than the idle time, I cannot understand
why the control channel would be closed, I can guarantee it's never happened to us.

Overriding PASV to provide a different IP address works fine and is correctly supported. I
don't know why you want to capture anything after MLSD completes. If the data channel is closed
the FTP proxy will close it's own data channel to the client. If it doesn't, you need to see
how to fix the proxy. Why would you try to workaround it for just the MLSD command?
don't you ever need to download files?
I wouldn't try to find a workaround in any case to be honest.


Best regards,
David

On Thu, 22 Aug 2019, 17:33 Kenneth Taylor, <kenneth.taylor@dataexpress.com>
wrote:

> It looks like the problem is in the closing of the data channel. Its
> happening on the server but not on the proxy so the client hangs
> waiting for that to happen, unless the command channel closes.
> Apache, for some reason, closes the command channel after the "226 Closing data connection"
> is sent.  Why?  I found this by preventing the command channel from
> closing on the proxy, and that caused the client to hang after
> receiving the MLSD response.  So the previous behavior was misleading
> because Apache closing the command channel was being seen by the
> client which was giving up the session entirely but complaining about the MLSD failing.
>
> So is this a bug in Apache or just a missing piece?  How would I catch
> the
> 226 response so I could tell the proxy to close the DC?
>
> We can't use afterCommand() for PASV because that would cause Apache
> to open its own DC.  So it seems like we have to do an afterCommand()
> for MLSD, but then we'd have to know this was for a DC and close that
> DC instead of the Command Channel. Ugh!
>
> Thanks.
> Ken
>
> -----Original Message-----
> From: Kenneth Taylor <kenneth.taylor@dataexpress.com>
> Sent: Thursday, August 22, 2019 9:01 AM
> To: ftpserver-users@mina.apache.org
> Subject: RE: FTP Data Channels Through Reverse Proxy
>
> Thanks for the response David.  I have used 3 different clients
> (FileZilla, WinSCP and custom) and they all respond the same.  I can
> see the MLSD response on the Proxy and that its sending it to the client.
> That's why we are stumped.  We cannot see any difference between Proxy
> and No Proxy.  But Apache is closing the Data Channel & Session after
> sending the MLSD response, and it should not.  Connecting direct, the
> Data Channel does not get closed, but I suppose that could be because
> Apache is detecting some kind of error that it is not reporting.  I've
> turned the logs all the way up and don't see anything that would help.
>
> A network trace might be a good idea.  We will try a wireshark scan.
>
> One thing that just occurred to me is that Apache may be opening more
> than one Data Channel and sending the MLSD response down the wrong
> pipe.  Not sure why it would do that since our PASV handler is a
> direct copy of the Apache one (PASV), except with the change of IP and Port in the reponse.
> Our Ftplet returns the ResultType of SKIP in the beforeCommand() for
> PASV which should stop Apache from opening its own data channel, right?
>
> Has Apache ever tested this?  I would think putting an FTP Server
> behind a reverse proxy would be VERY common.  Our customers demand it.
>
> I will have to look at the FTP RFCs again but I don't remember if any
> messages are supposed to go through the data channel from the client
> to the server.  Could it be that Apache is expecting something on that
> stream, even if its just an ACK or disconnect?
>
> Thanks.
> Ken
>
> -----Original Message-----
> From: David Latorre <dvlato@gmail.com>
> Sent: Wednesday, August 21, 2019 6:09 PM
> To: ftpserver-users@mina.apache.org
> Subject: Re: FTP Data Channels Through Reverse Proxy
>
> Ah sorry,
>  I was rechecking something and made a mistake regarding the IP
> address in the PASV response. In any case, unless you have proof (logs
> from the server or network traces of the server <-> proxy bit) that
> the server returns invalid data in the proxy case, I think you should
> focus on verifying why the client is failing and, if it is caused by
> an invalid response, why the proxy is generating it.
>
>
>
>
> On Thu, 22 Aug 2019, 00:03 David Latorre, <dvlato@gmail.com> wrote:
>
> > Hi Kenneth,
> >
> > Shouldn't you receive the proxy address as a response to PASV when
> > going through the proxy? Otherwise it would bypass the proxy, but
> > I'm assuming you should not  connect directly from the client to the
> > server and that's why you need a proxy. I'm a bit puzzled since you
> > state that the proxy receives the response to MLSD but the IP that
> > the client sees for PASV is the same in both cases.
> >
> > Your client log for the proxy case does not show any listing as a
> > response for MLSD, so it's difficult to know what's going on. If you
> > cannot get any logs for that, get some network traces with tcpdump
> > and check what's going on. Enabling more detailed logging in the
> > client might also help (you can configure that in the Filezilla settings).
> > Server logs might also prove useful.
> >
> > In any case  I don't see how this would be caused by Ftpserver with
> > the information you've given.
> >
> >
> >
> > On Wed, 21 Aug 2019, 21:56 Kenneth Taylor,
> > <kenneth.taylor@dataexpress.com>
> > wrote:
> >
> >> We are trying to get Apache FTP Server to work behind a reverse proxy.
> >> The only thing not working is the data channels. We use an Ftplet
> >> to intercept and handle the PASV command and substitute our proxy’s
> >> IP and port in the PASV response. That all seems to work fine.  We
> >> are testing it with FileZilla client and our server works perfectly
> >> when connecting directly to the server but fails when going through
> >> the
> proxy.
> >>
> >>
> >>
> >> We can see the data connections being established on both the
> >> server and the proxy. Our proxy logs every forwarded message and we
> >> can see that all client and server messages are being forwarded both ways.
> >>
> >>
> >>
> >> The problem is that after the PASV command happens the next comand
> >> is MLSD and we can see the server sending the correct response and
> >> the proxy is forwarding the listing. But the client interprets the
> >> response incorrectly and closes the session.  After looking at this
> >> for many many hours we’re stumped.
> >>
> >>
> >>
> >> Our Proxy is a very simple IO streams forwarder, with a control
> >> connection to tell the Proxy when to open data channels that
> >> forward back to the data channel opened by Apache on the Server.
> >>
> >>
> >>
> >> This is the client log when connecting direct:
> >>
> >>
> >>
> >> 14:10:44 Command: PASV
> >>
> >> 14:10:44 Response: 227 Entering Passive Mode (192,168,0,173,40,122)
> >>
> >> 14:10:44 Trace:       Binding data connection source IP to control
> >> connection source IP 192.168.0.92
> >>
> >> 14:10:44 Command: MLSD
> >>
> >> 14:10:44 Response: 150 File status okay; about to open data connection.
> >>
> >> 14:10:44 Response: 226 Closing data connection.
> >>
> >> 14:10:44 Listing:      Size=0;Modify=20190814174646.670;Type=dir;
> Monthly
> >>
> >> 14:10:44 Listing:      Size=0;Modify=20190814180307.721;Type=dir; Weekly
> >>
> >> 14:10:44 Listing:      Size=0;Modify=20190814174833.915;Type=dir; Daily
> >>
> >> 14:10:44 Listing:      Size=0;Modify=20190814175140.777;Type=dir;
> Archive
> >>
> >> 14:10:44 Status:      Directory listing of "/" successful
> >>
> >>
> >>
> >> This is the client log when connecting through the proxy:
> >>
> >>
> >>
> >> 14:37:16 Command: PASV
> >>
> >> 14:37:16 Response: 227 Entering Passive Mode (192,168,0,91,202,228)
> >>
> >> 14:37:16 Trace:       Binding data connection source IP to control
> >> connection source IP 192.168.0.173
> >>
> >> 14:37:16 Command: MLSD
> >>
> >> 14:37:16 Response: 150 File status okay; about to open data connection.
> >>
> >> 14:37:16 Response: 226 Closing data connection.
> >>
> >> 14:37:16 Error:       Disconnected from server: ECONNABORTED -
> Connection
> >> aborted
> >>
> >> 14:37:16 Error:       Failed to retrieve directory listing
> >>
> >>
> >>
> >> One weird thing we see is that the “226 Closing data connection”
> >> shows up on the proxy after the MLSD listing is returned. When
> >> connecting direct it is before the listing in the client log, but
> >> in all cases these messages are coming from the same server so that
> >> may be
> a logging anomaly.
> >>
> >>
> >>
> >> Thanks for any help.
> >>
> >> Ken
> >>
> >>
> >>
> >> Disclaimer: This email from DMBGroup LLC, DMB Consulting Services
> >> LLC, or the personnel associated with either entity (collectively
> >> "*DMB*") and attachments, contain *CONFIDENTIAL, PRIVILEGED AND
> >> PROPRIETARY *information for exclusive use of the addressee
> >> individual(s) or entity. Unauthorized viewing, copying, disclosure,
> >> distribution or use of this e-mail or attachments may be subject to
> >> legal restriction or sanction. If received in error, notify sender
> >> immediately by return e-mail and delete original message and
> >> attachments. Nothing contained in this e-mail or attachments shall
> satisfy the requirements for a writing unless specifically stated.
> >> Nothing contained herein shall constitute a contract or electronic
> >> signature under the Electronic Signatures in Global and National
> >> Commerce Act, any version of the Uniform Electronic Transactions
> >> Act or any other statute governing electronic transactions.
> >> Opinions and statements expressed in this e-mail and any
> >> attachments are those of the individual sender and not necessarily
> >> of DMB. DMB does not guarantee this e-mail transmission is secured, error or
virus-free.
> >> Neither DMB nor the sender of this e-mail accepts liability for
> >> errors or omissions in the contents of this e-mail, which arise as
> >> a
> result of e-mail transmission. .
> >>
> >
>
> Disclaimer: This email from DMBGroup LLC, DMB Consulting Services LLC,
> or the personnel associated with either entity (collectively "DMB")
> and attachments, contain CONFIDENTIAL, PRIVILEGED AND PROPRIETARY
> information for exclusive use of the addressee individual(s) or
> entity. Unauthorized viewing, copying, disclosure, distribution or use
> of this e-mail or attachments may be subject to legal restriction or
> sanction. If received in error, notify sender immediately by return
> e-mail and delete original message and attachments. Nothing contained
> in this e-mail or attachments shall satisfy the requirements for a writing unless specifically
stated.
> Nothing contained herein shall constitute a contract or electronic
> signature under the Electronic Signatures in Global and National
> Commerce Act, any version of the Uniform Electronic Transactions Act
> or any other statute governing electronic transactions. Opinions and
> statements expressed in this e-mail and any attachments are those of
> the individual sender and not necessarily of DMB. DMB does not
> guarantee this e-mail transmission is secured, error or virus-free.
> Neither DMB nor the sender of this e-mail accepts liability for errors
> or omissions in the contents of this e-mail, which arise as a result of e-mail transmission.
.
>
>
> Disclaimer: This email from DMBGroup LLC, DMB Consulting Services LLC,
> or the personnel associated with either entity (collectively "DMB")
> and attachments, contain CONFIDENTIAL, PRIVILEGED AND PROPRIETARY
> information for exclusive use of the addressee individual(s) or
> entity. Unauthorized viewing, copying, disclosure, distribution or use
> of this e-mail or attachments may be subject to legal restriction or
> sanction. If received in error, notify sender immediately by return
> e-mail and delete original message and attachments. Nothing contained
> in this e-mail or attachments shall satisfy the requirements for a writing unless specifically
stated.
> Nothing contained herein shall constitute a contract or electronic
> signature under the Electronic Signatures in Global and National
> Commerce Act, any version of the Uniform Electronic Transactions Act
> or any other statute governing electronic transactions. Opinions and
> statements expressed in this e-mail and any attachments are those of
> the individual sender and not necessarily of DMB. DMB does not
> guarantee this e-mail transmission is secured, error or virus-free.
> Neither DMB nor the sender of this e-mail accepts liability for errors
> or omissions in the contents of this e-mail, which arise as a result of e-mail transmission.
.
>
>

Disclaimer: This email from DMBGroup LLC, DMB Consulting Services LLC, or the personnel associated
with either entity (collectively "DMB") and attachments, contain CONFIDENTIAL, PRIVILEGED
AND PROPRIETARY information for exclusive use of the addressee individual(s) or entity. Unauthorized
viewing, copying, disclosure, distribution or use of this e-mail or attachments may be subject
to legal restriction or sanction. If received in error, notify sender immediately by return
e-mail and delete original message and attachments. Nothing contained in this e-mail or attachments
shall satisfy the requirements for a writing unless specifically stated. Nothing contained
herein shall constitute a contract or electronic signature under the Electronic Signatures
in Global and National Commerce Act, any version of the Uniform Electronic Transactions Act
or any other statute governing electronic transactions. Opinions and statements expressed
in this e-mail and any attachments are those of the individual sender and not necessarily
of DMB. DMB does not guarantee this e-mail transmission is secured, error or virus-free. Neither
DMB nor the sender of this e-mail accepts liability for errors or omissions in the contents
of this e-mail, which arise as a result of e-mail transmission. .

Mime
View raw message