drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <par0...@yahoo.com.INVALID>
Subject Re: Help with Apache Drill - S3 compatible storage connectivity
Date Wed, 27 Jun 2018 00:20:43 GMT
Hi Dummy ID,

Have you tried getting Drill to work with the real S3? There were a few confusing bits in
the S3 docs that Bridget graciously cleaned up. By trying to access the real S3 you'll ensure
that you have gotten the config stuff right.

The key missing bit of information was that you had to provide the fs.s3a.endpoint property.
The HDFS s3a library is the current version you should use (at least for S3) rather than the
older s3n, etc. libraries.

You can also use HDFS to verify that you have proper connectivity to your S3 clone. Configure
HDFS to work with your S3 clone, then check if you can access your files using the "hadoop
fs" commands. If not, then you've probably got a more fundamental issue than Drill.

If "hadoop fs" works, but Drill does not, then you probably have a Drill config issue since
Drill uses the Hadoop libraries for S3 access. Review the docs for what should be in core-site.xml
and what should be in your S3 storage plugin config.

- Paul


    On Tuesday, June 26, 2018, 2:10:11 PM PDT, dummy id <dummyid2124@gmail.com> wrote:
 Can i get an update on this please?

On Fri, Jun 15, 2018 at 11:36 AM, dummy id <dummyid2124@gmail.com> wrote:

> *Team, *
> *I am not sure who can help me out with this, so just adding both of the
> help community..I have followed your  documentation on setting up drill and
> i am able to query the files from local , meaning class path and dfs, but
> not able to do it with s3. I am not using Amazon S3 instead i am using S3
> compatible storage from DELL EMC. I would need your help in setting up the
> storage plugin file so that it uses path style addressing instead of normal
> URL method. I would request you to kindly share an example core-site.xml
> file as well as an example storage plugin file where it uses path style
> addressing method to connect to s3.*
> *I have tried using fs**.**s3a**.**path**.**style**.**access with
> value true in both core-site and storage plugin files, but still the path
> style addressing is not being read by drill and it again tries to connect
> using URL method as it uses in connecting with Amazon S3. Kindly help..*
> *Just an FYI, i have followed steps from "Drill in 10 minutes"
> documentation to install and connect with my S3 compatible storage.*
> *Awaiting for your reply..*
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message