drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Turton <dz...@apache.org>
Subject Re: Soucis Drill
Date Wed, 29 Sep 2021 10:05:04 GMT
I think you probably mean options like

planner.width.max_per_node
planner.width.max_per_query

in your last response, given you mention values of 3 and 28.

That aside, I can't see anything in the query profile you shared that 
should be any obstacle for your cluster, or even for a single laptop.  I 
would debug this by investigating memory allocation (including to 
processes other than Drill) on the host machines, as well as swap 
thrashing.  Note that a lot of this investigation is at the operating 
system level, rather than in Drill.

Out of interest, you can also try copying the CSV files you're querying 
here to your laptop, or some other idle machine, where you query them 
with drill-embedded.  This last step would give you an answer to: "how 
quickly can a small, unloaded, single node Drill complete this query?".  
I expect less than 30 seconds, with time to write the results out being 
a big part of that.

On 2021/09/28 18:11, nawfal osman wrote:
> Hi
>
> Please find atatched the json profile of the query
>
>
> Have you made any changes to Drill memory options such 
> as|planner.memory.max_query_memory_per_node|?
>
> Answer : yes  we use   this parameter sometimes we ran queries  with 
> the parameter '3' and some other query with  '28'
>
>
> Kinds Regards :
> ------------------------------------------------------------------------
> *De :* James Turton <dzamo@apache.org>
> *Envoyé :* mardi 28 septembre 2021 14:52
> *À :* user@drill.apache.org <user@drill.apache.org>; nawfal osman 
> <nawfal.osman@corporate-groupe.com>
> *Cc :* Zakaria Rahoui <zakaria.rahoui@corporate-groupe.com>; Amine 
> Elmekki <amine.elmekki@corporate-groupe.com>
> *Objet :* Re: Soucis Drill
> Plenty of direct memory there, 50Gb on every Drillbit, far in excess 
> of the modest 134Mb that Drill says it can't get.  Can you share a 
> JSON profile from an execution of this query (Web UI -> Profiles -> 
> Completed queries -> (choose query) -> Full JSON Profile?
>
> Have you made any changes to Drill memory options such as 
> |planner.memory.max_query_memory_per_node|?
>
> On 2021/09/28 13:55, nawfal osman wrote:
>> Hi
>>
>> Please notice  we  have  10 nodes (mapr11-mapr15 )  ( mapr31-35)
>>
>> please find below the output of the command : free -h of the node 
>> mapr15 :
>> [mapr@mapr15 conf]$ free -h
>>               total        used        free      shared  buff/cache
  
>> available
>> Mem:           125G         97G        3.1G        1.3G      
  25G   
>>       24G
>> Swap:          117G         54G
>>
>>
>>
>> Find below the output of the command svc :
>> [mapr@mapr15 conf]$ maprcli node list -columns svc
>> 2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ  vmsplice 
>> 406 err = 1   10.148.0.73:7222
>>
>>
>> hostname  service                       ip
>> mapr11 
>>  cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver

>>  10.148.0.44,10.32.243.44
>> mapr12 
>>  data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver

>>  10.148.0.45,10.32.243.45
>> mapr13 
>>  httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver

>>                                   10.148.0.46,10.32.243.46
>> mapr14 
>>  cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver

>>       10.148.0.47
>> mapr15 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                              10.148.0.48,10.32.243.48
>> mapr31 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                              10.148.0.70
>> mapr32 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                              10.148.0.71
>> mapr33 
>>  fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver

>>                              10.148.0.72
>> mapr34 
>>  cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver

>>                         10.148.0.73
>> mapr35 
>>  data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>>
>>
>> For the memory setting of drill ; please find  attached the conf of 
>> all nodes
>>
>>
>>
>> Kind Regards;
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message