drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kunal Khatua <kkha...@mapr.com>
Subject RE: Fwd: Creating a Tableau extracts with Drill 1.12 uses unlimited memory
Date Sat, 27 Jan 2018 01:51:11 GMT
Sorry.. I completely missed the fact that it was 61GB for the System memory. (I’m used to
seeing powers of 2)  O:-)

Paul’s suggestion should be sufficient in my opinion. If Zookeeper crashed.. that is most
likely indicative of excess memory allocation to Drill with no spare on the system.

I’m still not sure why Drill consume so much memory in the first place.

You could add the following flags to the JVM options (drill-env.sh) to generate the GC Logs:
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc -Xloggc:/var/log/drill/gc.log

I also noticed that there were 2 queries in the system when the crash occurred, including
an existing query ID without any reference to what the query was:  25953c99-7e37-fc27-98d1-2cb87b038dfc

Could you share the details with the above flag and no other query running?

Thanks
Kunal

From: Francis McGregor-Macdonald [mailto:francis@mc-mac.com]
Sent: Friday, January 26, 2018 4:16 PM
To: user@drill.apache.org
Subject: Re: Fwd: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Thanks Paul,

I will update with your suggested memory allocations also and retry.

Zookeeper crashed too which might explain more? I have attached the logs from Zookeeper too.

Thanks

On Sat, Jan 27, 2018 at 6:45 AM, Paul Rogers <par0328@yahoo.com<mailto:par0328@yahoo.com>>
wrote:
Hi Francis,

Thanks much for the log. The log shows running a query, then immediately shows entries that
occur when starting Drill. I'm guessing that Drill literally crashed at this point? This is
more severe than the usual error in which a query exhausts memory.

Some general observations. The Drill memory is 60 GB, but system memory is 61 GB. Perhaps
try dropping total Drill memory some to give the OS and other tasks more headroom. For a SELECT
* memory, Drill needs far less than what you have, so maybe try giving Drill 48 GB total.

Then, Drill needs direct memory much more than heap. So, maybe give Drill 39 GB direct, 8
GB heap and 1 GB (the default) for code cache. These settings are in drill-env.sh.

Kunal, you have more experience with these issues. Can you make additional suggestions by
looking at the log?

Thanks,

- Paul


On Thursday, January 25, 2018, 10:20:29 PM PST, Francis McGregor-Macdonald <francis@mc-mac.com<mailto:francis@mc-mac.com>>
wrote:


Hi all,

I am guessing that each of your EMR nodes are quite large? EMR nodes are: r4.2xlarge ('vcpu':
8, 'memory': 61)

Property "planner.width.max_per_node" is set to = 6

What is the system memory and what are the allocations for heap and direct?
System Memory: 61GB (EMR nodes above)
drill_mem_heap: 12G
drill_mem_max: 48G

The view is simple: SELECT * FROM s3://myparquet.parquet (14GB)

planner.memory.max_query_memor y_per_node = 10479720202

Drillbit.log attached (I think I have the correct selection included).

Thanks

On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <kkhatua@mapr.com<mailto:kkhatua@mapr.com>>
wrote:
What is the system memory and what are the allocations for heap and direct? The memory crash
might be occurring due to insufficient heap. The limits parameter applies to the direct memory
and not Heap.

Can you share details in the logs from the crash?

-----Original Message-----
From: Timothy Farkas [mailto:tfarkas@mapr.com<mailto:tfarkas@mapr.com>]
Sent: Thursday, January 25, 2018 2:58 PM
To: user@drill.apache.org<mailto:user@drill.apache.org>
Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Hi Francis,

I am guessing that each of your EMR nodes are quite large (32 or 64 vcpus). On large machines
Drill's planner over parallelizes and over allocates memory. There is a property "planner.width.max_per_node"
which limits the number of operators that can simultaneously execute on a Drillbit for a query.
If you configure the width per node to something like 5 or 10 (you may have to play around
with it) things should start working.

Thanks,
Tim

______________________________ __
From: Francis McGregor-Macdonald <francis@mc-mac.com<mailto:francis@mc-mac.com>>
Sent: Thursday, January 25, 2018 1:58:22 PM
To: user@drill.apache.org<mailto:user@drill.apache.org>
Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory

Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a Drill (1.12 on EMR)
cluster memory appears not to adhere to the limits set by planner.memory.max_query_memor y_per_node.

The extract query consumes all memory and then crashes drill.

Running the same query as a create table memory behaves as expected.

The query complexity is trivial:
select * from view only a single parquet with no calculated fields.

Has anyone else observed this behavior?



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message