drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francis McGregor-Macdonald <fran...@mc-mac.com>
Subject Re: Fwd: Creating a Tableau extracts with Drill 1.12 uses unlimited memory
Date Sat, 27 Jan 2018 00:15:52 GMT
Thanks Paul,

I will update with your suggested memory allocations also and retry.

Zookeeper crashed too which might explain more? I have attached the logs
from Zookeeper too.

Thanks

On Sat, Jan 27, 2018 at 6:45 AM, Paul Rogers <par0328@yahoo.com> wrote:

> Hi Francis,
>
> Thanks much for the log. The log shows running a query, then immediately
> shows entries that occur when starting Drill. I'm guessing that Drill
> literally crashed at this point? This is more severe than the usual error
> in which a query exhausts memory.
>
> Some general observations. The Drill memory is 60 GB, but system memory is
> 61 GB. Perhaps try dropping total Drill memory some to give the OS and
> other tasks more headroom. For a SELECT * memory, Drill needs far less than
> what you have, so maybe try giving Drill 48 GB total.
>
> Then, Drill needs direct memory much more than heap. So, maybe give Drill
> 39 GB direct, 8 GB heap and 1 GB (the default) for code cache. These
> settings are in drill-env.sh.
>
> Kunal, you have more experience with these issues. Can you make additional
> suggestions by looking at the log?
>
> Thanks,
>
> - Paul
>
>
>
> On Thursday, January 25, 2018, 10:20:29 PM PST, Francis McGregor-Macdonald
> <francis@mc-mac.com> wrote:
>
>
> Hi all,
>
> I am guessing that each of your EMR nodes are quite large? EMR nodes are:
> r4.2xlarge ('vcpu': 8, 'memory': 61)
>
> Property "planner.width.max_per_node" is set to = 6
>
> What is the system memory and what are the allocations for heap and direct?
> System Memory: 61GB (EMR nodes above)
> drill_mem_heap: 12G
> drill_mem_max: 48G
>
> The view is simple: SELECT * FROM s3://myparquet.parquet (14GB)
>
> planner.memory.max_query_memor y_per_node = 10479720202
>
> Drillbit.log attached (I think I have the correct selection included).
>
> Thanks
>
> On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <kkhatua@mapr.com> wrote:
>
> What is the system memory and what are the allocations for heap and
> direct? The memory crash might be occurring due to insufficient heap. The
> limits parameter applies to the direct memory and not Heap.
>
> Can you share details in the logs from the crash?
>
> -----Original Message-----
> From: Timothy Farkas [mailto:tfarkas@mapr.com]
> Sent: Thursday, January 25, 2018 2:58 PM
> To: user@drill.apache.org
> Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited
> memory
>
> Hi Francis,
>
> I am guessing that each of your EMR nodes are quite large (32 or 64
> vcpus). On large machines Drill's planner over parallelizes and over
> allocates memory. There is a property "planner.width.max_per_node" which
> limits the number of operators that can simultaneously execute on a
> Drillbit for a query. If you configure the width per node to something like
> 5 or 10 (you may have to play around with it) things should start working.
>
> Thanks,
> Tim
>
> ______________________________ __
> From: Francis McGregor-Macdonald <francis@mc-mac.com>
> Sent: Thursday, January 25, 2018 1:58:22 PM
> To: user@drill.apache.org
> Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory
>
> Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a
> Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set
> by planner.memory.max_query_memor y_per_node.
>
> The extract query consumes all memory and then crashes drill.
>
> Running the same query as a create table memory behaves as expected.
>
> The query complexity is trivial:
> select * from view only a single parquet with no calculated fields.
>
> Has anyone else observed this behavior?
>
>
>
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message