spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Forbidded : Error Code: 403
Date Fri, 15 May 2015 13:16:02 GMT
Have you verified that you can download the file from bucket-name without using Spark ?

Seems like permission issue. 

Cheers



> On May 15, 2015, at 5:09 AM, Mohammad Tariq <dontariq@gmail.com> wrote:
> 
> Hello list,
> 
> Scenario : I am trying to read an Avro file stored in S3 and create a DataFrame out of
it using Spark-Avro library, but unable to do so. This is the code which I am using :
> 
> public class S3DataFrame {
> 
> 	public static void main(String[] args) {
> 
> 		System.out.println("START...");
> 		SparkConf conf = new SparkConf().setAppName("DataFrameDemo").setMaster("local");
> 		JavaSparkContext sc = new JavaSparkContext(conf);
> 		Configuration config = sc.hadoopConfiguration();
> 		config.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
> 		config.set("fs.s3a.access.key","****************");
> 		config.set("fs.s3a.secret.key","*****************");
> 		config.set("fs.s3a.endpoint", "s3-us-west-2.amazonaws.com");
> 		SQLContext sqlContext = new SQLContext(sc);
> 		DataFrame df = sqlContext.load("s3a://bucket-name/file.avro", "com.databricks.spark.avro");
> 		df.show();
> 		df.printSchema();
> 		df.select("title").show();
> 		System.out.println("DONE");
> //		df.save("/new/dir/", "com.databricks.spark.avro");
> 	}
> }
> 
> Problem : Getting Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception:
Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; 
> 
> And this is the complete exception trace :
> 
> Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 63A603F1DC6FB900),
S3 Extended Request ID: vh5XhXSVO5ZvhX8c4I3tOWQD/T+B0ZW/MCYzUnuNnQ0R2JoBmJ0MPmUePRiQnPVASTbkonoFPIg=
> 	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1088)
> 	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:735)
> 	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:461)
> 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:296)
> 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3743)
> 	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
> 	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1005)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:688)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:71)
> 	at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> 	at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> 	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> 	at com.databricks.spark.avro.AvroRelation.newReader(AvroRelation.scala:105)
> 	at com.databricks.spark.avro.AvroRelation.<init>(AvroRelation.scala:60)
> 	at com.databricks.spark.avro.DefaultSource.createRelation(DefaultSource.scala:41)
> 	at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:219)
> 	at org.apache.spark.sql.SQLContext.load(SQLContext.scala:697)
> 	at org.apache.spark.sql.SQLContext.load(SQLContext.scala:673)
> 	at org.myorg.dataframe.S3DataFrame.main(S3DataFrame.java:25)
> 
> 
> Would really appreciate some help. Thank you so much for your precious time.
> 
> Software versions used :
> spark-1.3.1-bin-hadoop2.4
> hadoop-aws-2.6.0.jar
> MAC OS X 10.10.3
> java version "1.6.0_65"
>  
> 
> Tariq, Mohammad
> about.me/mti
> 
> 
>  				
>  

Mime
View raw message