drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arjun kr <arjun...@outlook.com>
Subject Re: S3 Connection Issues
Date Fri, 20 Oct 2017 18:12:39 GMT
Hi  Charles,


Any chance you can test s3 connectivity with other tools like hdfs shell or hive in case you
haven't tried already (and these tools available)? This may help to identify if it is Drill
specific issue.


For connecting via hdfs , you may try below command.


hadoop fs -Dfs.s3a.access.key="XXXX" -Dfs.s3a.secret.key="YYYYY" -ls s3a://<bucket-name>/


Enable DEBUG logging if needed.


export HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console


Thanks,


Arjun


________________________________
From: Padma Penumarthy <ppenumarthy@mapr.com>
Sent: Friday, October 20, 2017 3:00 AM
To: user@drill.apache.org
Subject: Re: S3 Connection Issues

Hi Charles,

I tried us-west-2 and it worked fine for me with drill built from latest source.
I did not do anything special.
Just enabled the S3 plugin and updated the plugin configuration like this.

{
  "type": "file",
  "enabled": true,
  "connection": "s3a://<bucket-name>",
  "config": {
    "fs.s3a.access.key": “XXXX",
    "fs.s3a.secret.key": “YYYY"
  },

I am able to do show databases and also can query the parquet files I uploaded to the bucket.

0: jdbc:drill:zk=local> show databases;
+---------------------+
|     SCHEMA_NAME     |
+---------------------+
| INFORMATION_SCHEMA  |
| cp.default          |
| dfs.default         |
| dfs.root            |
| dfs.tmp             |
| s3.default          |
| s3.root             |
| sys                 |
+---------------------+
8 rows selected (2.892 seconds)


Thanks
Padma

On Oct 18, 2017, at 9:18 PM, Charles Givre <cgivre@gmail.com<mailto:cgivre@gmail.com>>
wrote:

Hi Padma,
The bucket is is us-west-2.  I also discovered that some of the variable names in the documentation
on the main Drill site are incorrect.  Do I need to specify the region in the configuration
somewhere?

As an update, after discovering that the variable names are incorrect and that I didn’t
have Jets3t installed properly, I’m now getting the following error:

jdbc:drill:zk=local> show databases;
Error: RESOURCE ERROR: Failed to create schema tree.


[Error Id: e6012aa2-c775-46b9-b3ee-0af7d0b0871d on charless-mbp-2.fios-router.home:31010]

 (org.apache.hadoop.fs.s3.S3Exception) org.jets3t.service.S3ServiceException: Service Error
Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
request signature we calculated does not match the signature you provided. Check your key
and signing method.</Message></Error>
   org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get():175
   org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode():221

Thanks,
— C


On Oct 19, 2017, at 00:14, Padma Penumarthy <ppenumarthy@mapr.com<mailto:ppenumarthy@mapr.com>>
wrote:

Which AWS region are you trying to connect to ?
We have a  problem connecting to regions which support only v4 signature
since the version of hadoop we include in Drill is old.
Last time I tried, using Hadoop 2.8.1 worked for me.

Thanks
Padma


On Oct 18, 2017, at 8:14 PM, Charles Givre <cgivre@gmail.com<mailto:cgivre@gmail.com>>
wrote:

Hello all,
I’m trying to use Drill to query data in an S3 bucket and running into some issues which
I can’t seem to fix.  I followed the various instructions online to set up Drill with S3,
and put my keys in both the conf-site.xml and in the plugin config, but every time I attempt
to do anything I get the following errors:


jdbc:drill:zk=local> show databases;
Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request
ID: 56D1999BD1E62DEB, AWS Error Code: null, AWS Error Message: Forbidden


[Error Id: 65d0bb52-a923-4e98-8ab1-65678169140e on charless-mbp-2.fios-router.home:31010]
(state=,code=0)
0: jdbc:drill:zk=local> show databases;
Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request
ID: 4D2CBA8D42A9ECA0, AWS Error Code: null, AWS Error Message: Forbidden


[Error Id: 25a2d008-2f4d-4433-a809-b91ae063e61a on charless-mbp-2.fios-router.home:31010]
(state=,code=0)
0: jdbc:drill:zk=local> show files in s3.root;
Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request
ID: 2C635944EDE591F0, AWS Error Code: null, AWS Error Message: Forbidden


[Error Id: 02e136f5-68c0-4b47-9175-a9935bda5e1c on charless-mbp-2.fios-router.home:31010]
(state=,code=0)
0: jdbc:drill:zk=local> show schemas;
Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request
ID: 646EB5B2EBCF7CD2, AWS Error Code: null, AWS Error Message: Forbidden


[Error Id: 954aaffe-616a-4f40-9ba5-d4b7c04fe238 on charless-mbp-2.fios-router.home:31010]
(state=,code=0)

I have verified that the keys are correct but using the AWS CLI and downloaded some of the
files, but I’m kind of at a loss as to how to debug.  Any suggestions?
Thanks in advance,
— C




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message