spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen, Kevin" <Kevin.C...@neustar.biz>
Subject SaveAsTextFile to S3 bucket
Date Tue, 27 Jan 2015 04:21:49 GMT
Does anyone know if I can save a RDD as a text file to a pre-created directory in S3 bucket?

I have a directory created in S3 bucket: //nexgen-software/dev

When I tried to save a RDD as text file in this directory:
rdd.saveAsTextFile("s3n://nexgen-software/dev/output");


I got following exception at runtime:

Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
S3 HEAD request failed for '/dev' - ResponseCode=403, ResponseMessage=Forbidden


I have verified /dev has write permission. However, if I grant the bucket //nexgen-software
write permission, I don't get exception. But the output is not created under dev. Rather,
a different /dev/output directory is created directory in the bucket (//nexgen-software).
Is this how saveAsTextFile behalves in S3? Is there anyway I can have output created under
a pre-defied directory.


Thanks in advance.




Mime
View raw message