spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: is saveAsTextFile in java uses buffered I/O streams?
Date Thu, 09 Jan 2014 23:25:11 GMT
It just uses the Hadoop FileSystem API, I don’t think there’s any extra buffering. That
API itself may do buffering in the HDFS case, though newer versions of HDFS fix that.


On Jan 9, 2014, at 2:54 PM, wrote:

> Can someone provide me details on the spark java implementation of saveAsTextFile API
if it uses buffered I/O streams or not and at what point is flush it buffers if they are used?
> I remember from attending spark summit presentations that current spark release still
uses buffered I/O streams and that an upcoming option to support unbuffered I/O streams upon
writing data to local file or hdfs storage.
> Thanks,
> Hussam

View raw message