drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] ilooner opened a new pull request #1409: DRILL-6644: Don't reserve space for incoming probe batches unnecessarily during the build phase.
Date Mon, 30 Jul 2018 18:27:06 GMT
ilooner opened a new pull request #1409: DRILL-6644: Don't reserve space for incoming probe
batches unnecessarily during the build phase.
URL: https://github.com/apache/drill/pull/1409
 
 
   ## Problem
   
   Previously the memory calculator for HashJoin was reserving space for the worst case sized
incoming probe batch. This is actually completely unnecessary since we won't read probe side
data before the build phase anymore after DRILL-6453, so this effectively causes memory to
be underutilized.
   
   ## Solution
   
   There are two cases we have to handle:
   
   - We received probe side data when fetching the schema. In this case we only have to account
for the memory consumed by the probe data we received in the build phase. Not the worst case
incoming probe batch size.
   - We received NO probe side data when fetching the schema for the probe side. In this case
we don't have to reserve any memory for probe data during the build phase.
   
   ## Prerequisite
   
   The PR for DRILL-6453 must be merged first before this can go in. In this PR only look
at the changes in the last commit.
   
   https://github.com/apache/drill/pull/1408
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message