@sean - I am using latest code from master branch, up to commit#
In my case I have multiple directories with 1024 files(in that sizes of
files may be different). For some directories I always get consistent
result... and for others I can reproduce the inconsistent behavior.
I am not much familiar with S3 protocol or s3 driver in spark. I am
wondering, how does s3 driver verifies that all files(and their content)
under a directory were correctly?
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-from-Amazon-S3-directory-via-textFile-api-behaves-inconsistently-tp11092p11170.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.