drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Jaimes III <rafjai...@gmail.com>
Subject Re: exec.queue.enable in drill-embedded
Date Sun, 28 Jun 2020 23:00:19 GMT
What do you consider tiny for a parquet file? Few KB or MB?

16 GB may seem like a lot but it is quite small for a query engine. If
possible, increase the container to 32 GB and try giving it as much
Heap as possible, since that's what it will use for the REST API. Try
> 16 GB Heap and see if you still have issues.

Drill 1.18 is also starting to support Docker/K8s a little better so
you might be able to setup parallel workers (drillbits) with ZooKeeper
and spread the memory usage that way. Though that could take a bit of
setting up. I'd try to increase the resources on your single drillbit
before parallelizing it.

- Rafael

On Sun, Jun 28, 2020 at 6:17 PM Avner Levy <avner.levy@gmail.com> wrote:
>
> Hi,
> I'm using Drill 1.18 (master) docker and trying to configure its memory
> after getting out of heap memory errors:
> "RESOURCE ERROR: There is not enough heap memory to run this query using
> the web interface."
> The docker is serving remote clients through the REST API.
> The queries are simple selects over tiny parquet files that are stored in
> S3.
> It is running on in 16GB container, configured with a heap of 8GB, and 8GB
> direct memory.
> I tried to use:
>   exec.queue.enable=true
>   exec.queue.large=1
>   exec.queue.small=1
>
> and verified it was configured correctly, but I still see queries running
> concurrently.
> In addition, the "drill.queries.enqueued" counter remains zero.
> Is this mechanism supported in drill-embedded?
>
> In addition, it seems there is some memory leak, since after a while even
> with no query running for a while, running a single tiny query still gives
> the same error.
> Any insight would be highly appreciated :)
> Thanks,
>   Avner

Mime
View raw message