mina-ftpserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brad McEvoy" <b...@bradmcevoy.com>
Subject Re: IODataConnection.getDataInputStream()
Date Wed, 14 Oct 2009 10:08:47 GMT

just one experience i've had with s3 which might impact on the design.
i've found that uploads into s3 (when under load) fail surprisingly
frequently, at a rate of about 1 in a hundred. the s3 docs say that
upload failures are a normal and an expected part of the service, but i
didnt really think it would fail that often.

so my conclusion is that i have to buffer uploads and then load into s3
asynchronously to ensure that end users don't get impacted. if you do
this then you end up doing two seperate file transfers anyway, so
there's no big deal about going inputstream to inputstream.

another consideration is that uploads to s3 are non-transactional. so if
you stream directly from end user to s3, and the end user aborts the
upload, then you'll end up with a completely useless part of a file
stored in s3. if you buffer locally and only load into s3 on success you
won't have this problem.


On Wed, 14 Oct 2009 10:28 +0200, "David Latorre" <dvlato@gmail.com>
> 2009/10/13 Niklas Gustavsson <niklas@protocol7.com>:
> > On Tue, Oct 13, 2009 at 6:21 AM, Charles Karow <charles@karow.com> wrote:
> >> I am using ftpserver to provide a standard way for people to upload files to
> >> a "bucket" on Amazon's S3 service. My users will always be uploading files
> >> in binary mode. I am using code from Amazon that takes an InputStream and
> >> uses it to stream the data to Amazon's servers. Amazon's code does not
> >> expose an OutputStream.
> >>
> >> transferFromClient takes an OutputStream and I do not have access to an
> >> OutputStream.
> >
> > Sounds like this could be solved by an adapter stream which gets
> > written to by DataConnection and is read by S3. Or I might be missing
> > something?
>  This is I what I first thought but I think this might imply  several
> risks in terms of performance, or the need to store the whole
> transferred file locally (be it in memory or disk). This is of course,
> if he cannot use PipedStreams.
> I may not be thinking correctly now but for a solution:
> -  If you are using different threads for the FTP transfer and the
> transfer to Amazon I guess you could use PipedStreams with our current
> code (I haven't looked at it actually).
>  - Otherwise, maybe someone on this list can tell us what their
> approach is. I think some of them are using S3.
> If no one comes up with a solution for this, i don't think we should
> dismiss the possibility of exposing the input stream,  what do you
> think niklas?
  Brad McEvoy

View raw message