drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Fischmeister <sfisc...@uwaterloo.ca>
Subject Re: Querying multiple s3 buckets
Date Sun, 22 Nov 2020 17:21:46 GMT
But this means that (1) the names have to be known upfront and (2) the list of buckets remains

I'm looking for a method to dynamically search all buckets.

Now, I could use the REST API to dynamically create drill bits after listing all buckets,
but pushes responsibility to the user. It would be nicer to specify in the connection a wildcard
(e.g., "connection" = "s3a://foo*" or "s3a://*") and then drill internally search for matching
buckets and then traverses through them.


Nitin Pawar <nitinpawar432@gmail.com> writes:

> you can create multiple storages with different names to each bucker and
> query them in a single query
> On Sun, Nov 22, 2020 at 10:38 PM Sebastian Fischmeister <
> sfischme@uwaterloo.ca> wrote:
>> Hi,
>> I set up drill with min.io and have multiple buckets in minio. I was able
>> to setup an s3 connection to a single bucket, however, I actually want to
>> run a query on all files in all buckets. Is this possible?
>> Regards,
>>   Sebastian
> -- 
> Nitin Pawar

View raw message