drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [drill] HanumathRao opened a new pull request #1677: DRILL-7068: Support memory adjustment framework for resource manageme…
Date Mon, 04 Mar 2019 22:01:50 GMT
HanumathRao opened a new pull request #1677: DRILL-7068: Support memory adjustment framework
for resource manageme…
URL: https://github.com/apache/drill/pull/1677
 
 
   …nt with Queues.
   
   This PR includes the following changes.
   a) Refactoring of the code in SimpleParallelizer to have a mechanism to adjust the memory
for the minor fragments being generated.
   b) There is no change in the logic for generation of minor fragments and adjustment of
the memory in the default case (i.e when no RM is selected). The existing logic of calculating
the memory will be called during the adjustment phase.
   c) A new cost class called PrelCostEstimates is introduced to capture the cost estimates
calculated during physical planning. These estimates are further used in the memory calculation
in case when RM is selected.
   d) Consider this PR as introducing the framework for estimating and adjusting memory when
RM is enabled. The memory estimation based on the cluster state will be further added over
these changes.
   e) Currently only stubs are introduced for estimating the memory in case of Exchange operators.
These needs to be incorporated with the Exchange batch sizing ongoing work. 
   
   @amansinha100  @sohami  Can you please review the changes. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message