spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aseem Bansal (JIRA)" <>
Subject [jira] [Commented] (SPARK-17307) Document what all access is needed on S3 bucket when trying to save a model
Date Thu, 01 Sep 2016 08:48:22 GMT


Aseem Bansal commented on SPARK-17307:

I would add that bit of information at

Something like it needs complete read write access when using with S3 should be enough.

> Document what all access is needed on S3 bucket when trying to save a model
> ---------------------------------------------------------------------------
>                 Key: SPARK-17307
>                 URL:
>             Project: Spark
>          Issue Type: Documentation
>            Reporter: Aseem Bansal
>            Priority: Minor
> I faced this lack of documentation when I was trying to save a model to S3. Initially
I thought it should be only write. Then I found it also needs delete to delete temporary files.
Now I requested access for delete and tried again and I am get the error
> Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
S3 PUT failed for '/dev-qa_%24folder%24' XML Error Message
> To reproduce this error the below can be used
> {code}
> SparkSession sparkSession = SparkSession
>                 .builder()
>                 .appName("my app")
>                 .master("local") 
>                 .getOrCreate();
>         JavaSparkContext jsc = new JavaSparkContext(sparkSession.sparkContext());
> jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", <ACCESS_KEY>);
>         jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", <SECRET ACCESS
> //Create a Pipelinemode
>         pipelineModel.write().overwrite().save("s3n://<BUCKET>/dev-qa/modelTest");
> {code}
> This back and forth could be avoided if it was clearly mentioned what all access spark
needs to write to S3. Also would be great if why all of the access is needed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message