spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Holenweger <ro...@lotadata.com>
Subject ioStreams for DataFrameReader/Writer
Date Tue, 26 Jul 2016 15:01:10 GMT
Hello all,

why are all DataFrameReader and Writer methods tied to Path objects, or 
in other words why are there no Reader/Writers that can load data from 
an Input or Output Stream?

After looking at the spark source code, i realize that there is no easy 
way to add methods that can handle ioStreams.

Does anybody know of a solution, without having to write a full-blown 
connector?


I basically want to export a DataFrame as a Parquet ByteStream and then 
take care of persisting the blob.

-- 
Regards
*Roger Holenweger*
LotaData <http://lotadata.com/>
/spatiotemporal intelligence/

Mime
View raw message