drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kunal Khatua" <ku...@apache.org>
Subject Re: Performace issue
Date Fri, 08 Feb 2019 20:07:11 GMT
The memory (heap) would climb as it tries to flatten the JSON data. Have you tried looking
at Drill's LateralJoin-Unnest feature? It was meant to address memory issues for some use
cases of the FLATTEN operator.

On 2/8/2019 5:17:01 AM, PRAVEEN DEVERACHETTY <praveendk@gmail.com> wrote:
I am running a query with UNION ALL. as below

select
from ( select FLATTEN(t.jdata) as record from
((select convert_from(json string, json) union all
(select conver_from(json_string,json) union all
...
) as jdata) ) as t) ems

Reason for giving union all is because we are invoking a call using rest
app, there is limitation of 20,000 when we use convert_from function. Our
heap size is 8GB, server is 8core. From profiling, it shows this perticula
query spikes from 100MB to 8GB continuously. is there anything i am
doing wrong?.

Thanks,
Prveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message