spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkumar Chokkalingam <>
Subject Re: Output to a single directory with multiple files rather multiple directories ?
Date Thu, 10 Oct 2013 23:30:05 GMT

Thanks for the mail, Matei. Since, I need to have the output  directory
structure to be same as the input directory structure with some changes
made to the content of those files while parsing [ replacing certain fields
with its encrypted value]. I wouldn't want the union to combine few of the
input files into a single file.

Is there some API which would treat each file as independent and write to a
output file ? That would've been great.

If it doesn't work, then I have to write them each to a folder and process
each of them (using some script) to match my input directory structure.

View raw message