drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Zhu <h...@mapr.com>
Subject Re: Creating a Tableau extracts with Drill 1.12 uses unlimited memory
Date Fri, 26 Jan 2018 19:09:24 GMT
Hi Team,

Could you collect SQL profile for the problematic query and also drillbit.out + /var/log/messages
from OS level?
One finding is:
open files                      (-n) 1024
It should be increased when you setup the cluster. It should be at least:
open files                      (-n) 65535

Thanks,
Hao

From: Francis McGregor-Macdonald <francis@mc-mac.com<mailto:francis@mc-mac.com>>
Reply-To: "user@drill.apache.org<mailto:user@drill.apache.org>" <user@drill.apache.org<mailto:user@drill.apache.org>>
Date: Thursday, January 25, 2018 at 10:19 PM
To: "user@drill.apache.org<mailto:user@drill.apache.org>" <user@drill.apache.org<mailto:user@drill.apache.org>>
Subject: Fwd: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Hi all,

I am guessing that each of your EMR nodes are quite large? EMR nodes are: r4.2xlarge ('vcpu':
8, 'memory': 61)

Property "planner.width.max_per_node" is set to = 6

What is the system memory and what are the allocations for heap and direct?
System Memory: 61GB (EMR nodes above)
drill_mem_heap: 12G
drill_mem_max: 48G

The view is simple: SELECT * FROM s3://myparquet.parquet (14GB)

planner.memory.max_query_memory_per_node = 10479720202

Drillbit.log attached (I think I have the correct selection included).

Thanks

On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <kkhatua@mapr.com<mailto:kkhatua@mapr.com>>
wrote:
What is the system memory and what are the allocations for heap and direct? The memory crash
might be occurring due to insufficient heap. The limits parameter applies to the direct memory
and not Heap.

Can you share details in the logs from the crash?

-----Original Message-----
From: Timothy Farkas [mailto:tfarkas@mapr.com<mailto:tfarkas@mapr.com>]
Sent: Thursday, January 25, 2018 2:58 PM
To: user@drill.apache.org<mailto:user@drill.apache.org>
Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Hi Francis,

I am guessing that each of your EMR nodes are quite large (32 or 64 vcpus). On large machines
Drill's planner over parallelizes and over allocates memory. There is a property "planner.width.max_per_node"
which limits the number of operators that can simultaneously execute on a Drillbit for a query.
If you configure the width per node to something like 5 or 10 (you may have to play around
with it) things should start working.

Thanks,
Tim

________________________________
From: Francis McGregor-Macdonald <francis@mc-mac.com<mailto:francis@mc-mac.com>>
Sent: Thursday, January 25, 2018 1:58:22 PM
To: user@drill.apache.org<mailto:user@drill.apache.org>
Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a Drill (1.12 on EMR)
cluster memory appears not to adhere to the limits set by planner.memory.max_query_memory_per_node.

The extract query consumes all memory and then crashes drill.

Running the same query as a create table memory behaves as expected.

The query complexity is trivial:
select * from view only a single parquet with no calculated fields.

Has anyone else observed this behavior?




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message