drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Volodymyr Vysotskyi (Jira)" <j...@apache.org>
Subject [jira] [Created] (DRILL-7372) MethodAnalyzer consumes too many memory
Date Wed, 11 Sep 2019 14:07:00 GMT
Volodymyr Vysotskyi created DRILL-7372:

             Summary: MethodAnalyzer consumes too many memory
                 Key: DRILL-7372
                 URL: https://issues.apache.org/jira/browse/DRILL-7372
             Project: Apache Drill
          Issue Type: Bug
    Affects Versions: 1.16.0
            Reporter: Volodymyr Vysotskyi
            Assignee: Volodymyr Vysotskyi
             Fix For: 1.17.0

In the scope of DRILL-6524 was added logic for determining whether a variable is assigned
in conditional block to prevent incorrect scalar replacement for such cases.

But for some queries, this logic consumes too many memory, for example, for the following
FROM cp.`employee.json`
WHERE employee_id+0 < employee_id
  OR employee_id+1 < employee_id
  AND employee_id+2 < employee_id
  OR employee_id+3 < employee_id
  AND employee_id+4 < employee_id
  OR employee_id+5 < employee_id
  AND employee_id+6 < employee_id
  OR employee_id+7 < employee_id
  AND employee_id+8 < employee_id
  OR employee_id+9 < employee_id
  AND employee_id+10 < employee_id
  OR employee_id+11 < employee_id
  AND employee_id+12 < employee_id
  OR employee_id+13 < employee_id
  AND employee_id+14 < employee_id
  OR employee_id+15 < employee_id
  AND employee_id+16 < employee_id
  OR employee_id+17 < employee_id
  AND employee_id+18 < employee_id
  OR employee_id+19 < employee_id
  AND employee_id+20 < employee_id
  OR employee_id+21 < employee_id
  AND employee_id+22 < employee_id
  OR employee_id+23 < employee_id
  AND employee_id+24 < employee_id
  OR employee_id+25 < employee_id
  AND employee_id+26 < employee_id
  OR employee_id+27 < employee_id
  AND employee_id+28 < employee_id
  OR employee_id+29 < employee_id
  AND employee_id+30 < employee_id
  OR employee_id+31 < employee_id
  AND employee_id+32 < employee_id
  OR employee_id+33 < employee_id
  AND employee_id+34 < employee_id
  OR employee_id+35 < employee_id
  AND employee_id+36 < employee_id
  OR employee_id+37 < employee_id
  AND employee_id+38 < employee_id
  OR employee_id+39 < employee_id
  AND employee_id+40 < employee_id
  OR employee_id+41 < employee_id
  AND employee_id+42 < employee_id
  OR employee_id+43 < employee_id
  AND employee_id+44 < employee_id
  OR employee_id+45 < employee_id
  AND employee_id+46 < employee_id
  OR employee_id+47 < employee_id
  AND employee_id+48 < employee_id
  OR employee_id+49 < employee_id

Drill consumes more than 6 GB memory.

One of the issues to fix is to replace {{Deque<Set<Integer>> localVariablesSet;}}
with {{Deque<BitSet>}}, it will reduce memory usage significantly.
Additionally should be investigated why these objects cannot be collected by GC.

This message was sent by Atlassian Jira

View raw message