spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugen Cepoi <cepoi.eu...@gmail.com>
Subject Re: Bad Digest error while doing aws s3 put
Date Mon, 08 Feb 2016 19:43:05 GMT
I had similar problems with multi part uploads. In my case the real error
was something else which was being masked by this issue
https://issues.apache.org/jira/browse/SPARK-6560. In the end this bad
digest exception was a side effect and not the original issue. For me it
was some library version conflict on EMR.

Depending on the size of the output files, you might try to just disable
the multipart upload using fs.s3n.multipart.uploads.enabled

Cheers,
Eugen

2016-02-07 15:05 GMT-08:00 Steve Loughran <stevel@hortonworks.com>:

>
> > On 7 Feb 2016, at 07:57, Dhimant <dhimant84.jayswal@gmail.com> wrote:
> >
> >        at
> >
> com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245)
> >        ... 15 more
> > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The
> > Content-MD5 you specified did not match what we received. (Service:
> Amazon
> > S3; Status Code: 400; Error Code: BadDigest; Request ID:
> 5918216A5901FCC8),
> > S3 Extended Request ID:
> >
> QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo=
> >        at
> >
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
> >        at
> >
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
> >        at
> >
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> >        at
> > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> >        at
> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796)
> >        at
> >
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482)
> >        at
> >
> com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140)
> >        ... 22 more
> >
>
> This is amazon's own s3 client. nothing in the apache hadoop source tree.
> Normally I'd say "use s3a to make s3n problems go away", but I don't know
> what that does on Amazon's own EMR libraries
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message