spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohamed Taher AlRefaie <m.ta...@msn.com>
Subject Accessing system environment on Spark Worker
Date Sun, 19 Jun 2016 11:46:35 GMT
Hello all:

I have an application that requires accessing DynamoDB tables. Each worker
establishes a connection with the database on its own.

I have added both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to both
master's and workers `spark-env.sh` file. I have also run the file using
`sh` to make sure the variables are exported.

When the code runs, I always get the error:

    Caused by: com.amazonaws.AmazonClientException: Unable to load AWS
credentials from any provider in the chain
    at
com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
    at
com.amazonaws.http.AmazonHttpClient.getCredentialsFromContext(AmazonHttpClient.java:774)
    at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:800)
    at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:695)
    at
com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447)
    at
com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409)
    at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358)
    at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2051)
    at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2021)
    at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1299)
    at
com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.describeTable(DynamoDBDelegate.java:635)
    ... 27 more

It seems that the AWS SDK has failed to load the credentials even though
they're exported. When I do export command, it returns the credentials.
What type of solution should I try?

Thanks,
Mohamed Taher Alrefaie.

Mime
View raw message