spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kamatsuoka <ken...@gmail.com>
Subject Re: How to read a multipart s3 file?
Date Wed, 07 May 2014 03:24:45 GMT
Whereas with s3://, the write takes 32 seconds and the rename takes 33
seconds:

14/05/06 20:23:08 INFO DAGScheduler: Stage 0 (saveAsTextFile at
FileCopy.scala:17) finished in 32.208 s
14/05/06 20:23:08 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
14/05/06 20:23:08 INFO SparkContext: Job finished: saveAsTextFile at
FileCopy.scala:17, took 32.430051749 s
14/05/06 20:23:41 INFO SparkDeploySchedulerBackend: Shutting down all
executors




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-multipart-s3-file-tp5463p5474.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message