drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Fischmeister <sfisc...@uwaterloo.ca>
Subject Re: Querying multiple s3 buckets
Date Sun, 22 Nov 2020 17:21:46 GMT
But this means that (1) the names have to be known upfront and (2) the list of buckets remains
static.

I'm looking for a method to dynamically search all buckets.

Now, I could use the REST API to dynamically create drill bits after listing all buckets,
but pushes responsibility to the user. It would be nicer to specify in the connection a wildcard
(e.g., "connection" = "s3a://foo*" or "s3a://*") and then drill internally search for matching
buckets and then traverses through them.

  Sebastian

Nitin Pawar <nitinpawar432@gmail.com> writes:

> you can create multiple storages with different names to each bucker and
> query them in a single query
>
> On Sun, Nov 22, 2020 at 10:38 PM Sebastian Fischmeister <
> sfischme@uwaterloo.ca> wrote:
>
>> Hi,
>>
>> I set up drill with min.io and have multiple buckets in minio. I was able
>> to setup an s3 connection to a single bucket, however, I actually want to
>> run a query on all files in all buckets. Is this possible?
>>
>> Regards,
>>   Sebastian
>>
>
>
> -- 
> Nitin Pawar

Mime
View raw message