drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Ingoldsby <jack.ingold...@gmail.com>
Subject Re: Connecting to S3 bucket which does not seem to require a key
Date Mon, 12 Jun 2017 16:13:14 GMT
Well, these are for a specific user I created for this bucket. The user
only has read access to this bucket, which only contains this public
citibike data and has no permissions access.
So, I'm fine if anyone can connect (at least until I figure out the problem)

On Mon, Jun 12, 2017 at 11:59 AM, Abhishek Girish <agirish@apache.org>
wrote:

> I hope you haven't shared your actual access / secret keys with the
> community. If not, please work on securing your account [1]!
>
>
> [1] https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
>
>
>
> On Mon, Jun 12, 2017 at 8:34 AM, Jack Ingoldsby <jack.ingoldsby@gmail.com>
> wrote:
>
> > Hi,
> > Thanks. I'm actually more playing around with a proof of concept that I
> can
> > query S3 using our tool via Drill.
> > So, what I did was to download the citibike and data and create my own s3
> > bucket with an accessid,secretket , but I'm having some problem
> connecting
> > I get the following error message when running a query
> >
> > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> > AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request
> > ID: 439EE2E823001E80, AWS Error Code: null, AWS Error Message: Bad
> Request
> > [Error Id: 9da0c6bd-b173-48e0-aeac-47179812e696 on
> > LAP-NY-CHENO.corp.sisense.com:31010]
> >
> > It appears to be a connection issue but i can connect to the bucket
> > sisense.citibike using AWS command line utility, using the same
> accesskey,
> > secretkey
> > Does anything leap out ?
> >
> > The configuration is set to
> >
> > {
> >   "type": "file",
> >   "enabled": true,
> >   "connection": "s3a://sisense.citibike",
> >   "config": {
> >     "fs.s3a.access.key": "ID",
> >     "fs.s3a.secret.key": "SECRET"
> >   },
> >
> >
> > Core-site.xml is set to
> >
> > <configuration>
> >
> >     <property>
> >         <name>fs.s3a.access.key</name>
> >         <value>AKIAJELPGZYEPGRP6VBA</value>
> >     </property>
> >
> >     <property>
> >         <name>fs.s3a.secret.key</name>
> >         <value>h3CyqC/VzpRirOMi3nCImYJL2oNV1xwOcEBiYi02</value>
> >     </property>
> >
> > </configuration>
> >
> > Thanks,
> > Jack
> >
> > On Mon, Jun 12, 2017 at 10:43 AM, Andries Engelbrecht <
> > aengelbrecht@mapr.com
> > > wrote:
> >
> > > You may be better of downloading the NYC bike data set locally and
> > convert
> > > to parquet.
> > > Converting from csv.zip to parquet will result in large improvements in
> > > performance if you do various queries on the data set.
> > >
> > > --Andries
> > >
> > > On 6/11/17, 10:48 PM, "Abhishek Girish" <agirish@apache.org> wrote:
> > >
> > >     Drill connects to to S3 buckets (AWS) via the S3a library. And the
> > > storage
> > >     plugin configuration requires the access & secret keys [1].
> > >
> > >     I'm not sure if Drill can access S3 without the credentials. It
> might
> > > be
> > >     possible via custom authenticators [2]. Hopefully others who have
> > tried
> > >     this will comment.
> > >
> > >
> > >     [1] https://drill.apache.org/docs/s3-storage-plugin/
> > >     [2] http://docs.aws.amazon.com/AmazonS3/latest/API/sig-
> > >     v4-authenticating-requests.html
> > >
> > >     On Wed, Jun 7, 2017 at 3:02 PM, Jack Ingoldsby <
> > > jack.ingoldsby@gmail.com>
> > >     wrote:
> > >
> > >     > Hi,
> > >     > I'm trying to access the NYC Citibike S3 bucket, which seems to
> > > publicly
> > >     > available
> > >     >
> > >     > https://s3.amazonaws.com/tripdata/index.html
> > >     > If I leave the Access Key & Secret Key empty, I get the following
> > > message
> > >     >
> > >     > 0: jdbc:drill:zk=local> !tables
> > >     > Error: Failure getting metadata: Unable to load AWS credentials
> > from
> > > any
> > >     > provider in the chain (state=,code=0)
> > >     >
> > >     > If I try entering random numbers as keys, I get the following
> > message
> > >     >
> > >     > Error: Failure getting metadata: Status Code: 403, AWS Service:
> > > Amazon S3,
> > >     > AWS Request ID: 1C888A3A21D79F87, AWS Error Code:
> > > InvalidAccessKeyId, AWS
> > >     > Error Message: The AWS Access Key Id you provided does not exist
> in
> > > our
> > >     > records. (state=,code=0)
> > >     >
> > >     > Is it possible to connect to a data source that does not seem to
> > > require a
> > >     > key?
> > >     >
> > >     > Thanks,
> > >     > Jack
> > >     >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message