drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Turton <dz...@apache.org>
Subject Re: Soucis Drill
Date Tue, 28 Sep 2021 14:14:10 GMT
I do also see mapr15 reporting a *lot* of data in swap.  It looks like 
the RAM on that box could be over allocated.  I haven't tested this but 
I'm pretty sure it's going to be better to let Drill intentionally spill 
when it thinks RAM is constrained than to tell it can have more RAM than 
there is available and let it thrash the swap partition.  Swapping 
during query processing could also be consistent with your point 1 
(performance degradation when two queries are run) if the two queries 
are competing for seeks from the storage device hosting the swap.  I 
recommend running `smem -atk` to see which processes have been swapped 
out and watching a monitoring tool that can show swapping counters like 
major page faults per second during query execution.

On 2021/09/28 15:52, James Turton wrote:
> Plenty of direct memory there, 50Gb on every Drillbit, far in excess 
> of the modest 134Mb that Drill says it can't get.  Can you share a 
> JSON profile from an execution of this query (Web UI -> Profiles -> 
> Completed queries -> (choose query) -> Full JSON Profile?
>
> Have you made any changes to Drill memory options such as 
> |planner.memory.max_query_memory_per_node|?
>
> On 2021/09/28 13:55, nawfal osman wrote:
>> Hi
>>
>> Please notice  we  have  10 nodes (mapr11-mapr15 )  ( mapr31-35)
>>
>> please find below the output of the command : free -h of the node 
>> mapr15 :
>> [mapr@mapr15 conf]$ free -h
>>               total        used        free      shared  buff/cache
  
>> available
>> Mem:           125G         97G        3.1G        1.3G     25G
      
>>   24G
>> Swap:          117G         54G
>>
>>
>>
>> Find below the output of the command svc :
>> [mapr@mapr15 conf]$ maprcli node list -columns svc
>> 2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ  vmsplice 
>> 406 err = 1   10.148.0.73:7222
>>
>>
>> hostname  service           ip
>> mapr11 
>>  cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver

>>  10.148.0.44,10.32.243.44
>> mapr12 
>>  data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver

>>  10.148.0.45,10.32.243.45
>> mapr13 
>>  httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver

>>                                   10.148.0.46,10.32.243.46
>> mapr14 
>>  cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver

>>   10.148.0.47
>> mapr15 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                          10.148.0.48,10.32.243.48
>> mapr31 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                          10.148.0.70
>> mapr32 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                          10.148.0.71
>> mapr33 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                          10.148.0.72
>> mapr34 
>>  cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver

>>                     10.148.0.73
>> mapr35 
>>  data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>>
>>
>> For the memory setting of drill ; please find  attached the conf of 
>> all nodes
>>
>>
>>
>> Kind Regards;
>
>


Mime
View raw message