spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kamatsuoka <>
Subject s3n > 5GB
Date Mon, 27 Jan 2014 02:18:09 GMT
The  hadoop docs about s3 <>   (linked
to by the Spark docs) say that s3n files are subject to "the 5GB limit on
file size imposed by S3."  However,  limit was raised
about three years ago.  So it wasn't clear to me whether this limit still
applies to Hadoops s3n urls.

Well, I tried running a spark job on a 200GB s3n file, and it ran fine.  Has
this been other people's experience?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message