drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <par0...@yahoo.com.INVALID>
Subject Re: Memory not coming down after query exeution
Date Thu, 18 Jul 2019 22:01:46 GMT
Hi Preet,

Welcome to the Drill community! You asked "why consumed memory by query is not coming down"?

Perhaps you meant to ask "why is the consumed JVM memory not coming down?" This is a feature
of Java: once the JVM uses an amount of memory, the JVM will not release that memory back
to the operating system. The memory is free within the JVM and will be reused for the next
query.

Also, the memory given to Drill seems excessive: a total of 96 GB for 50 million records.
That is an average of over 1K per record. Are your records that large?

Depending on the structure of your query, Drill may not hold the entire data set in memory.
If you do a simple "SELECT * FROM yourtable" then Drill will hold on only a few "batches"
of data in memory as data streams from your Parquet file, through drill, to your client application.

On the other hand, if you include an ORDER BY, GROUP BY or similar operator, then Drill will
hold all data in memory. If memory is insufficient, Drill will spill to disk.

Some questions to ask you are:

* How big is your Parquet file in MB?
* What client are you using? For this size of data, you should use a CREATE TABLE AS or a
JDBC or ODBC client. This is NOT a good query for the REST API or Drill web console.
* What is your query? Simple SELECT, or does it include sorting, grouping or other operations?

Thanks,
- Paul

 

    On Thursday, July 18, 2019, 02:53:48 PM PDT, preet singh <preetrekhi@gmail.com>
wrote:  
 
 Hello Drill Team

I am evaluating Apache drill for my query on 50 million of records in
parquet files. I updated max direct memory to 64 GB and heap 32 GB.

I am able to execute my query but after complete execution, why consumed
memory by query is not coming down (still be around 64 GB) ?

Thanks
Preet Singh
  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message