drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arina Ielchiieva <arina.yelchiy...@gmail.com>
Subject Re: scaling drill in an openshift (K8s) cluster
Date Tue, 24 Mar 2020 16:17:17 GMT
Please see https://issues.apache.org/jira/browse/DRILL-7563 <https://issues.apache.org/jira/browse/DRILL-7563>,
maybe it will be helpful.

Kind regards,
Arina

> On Mar 24, 2020, at 6:04 PM, Dobes Vandermeer <dobesv@gmail.com> wrote:
> 
> I was able to get drill up and running inside a k8s cluster but I didn't connect to it
from outside the cluster, so the DNS names were always resolvable by the client(s).
> 
> I had to run it as a statefulset to ensure the DNS names are stable, otherwise the drillbits
couldn't talk to each other, either.
> 
> On 3/24/2020 6:37:44 AM, Jaimes, Rafael - 0993 - MITLL <rafael.jaimes@ll.mit.edu>
wrote:
> I’m seeing a problem with scaling the number of pod instances in the replication controller
because they aren’t reporting their hostnames properly. This was a common problem that got
fixed in scalable architectures like ZooKeeper and Kafka (see reference at bottom I think
this was related).
>  
> In Drill’s case, ZooKeeper is able to see all of the drillbits, however, the hostnames
are only locally addressable within the cluster, so as soon as you perform a query it fails
since the client can’t find the drillbit that it got assigned, its hostname isn’t externally
addressable.
>  
> Kafka fixes this by allowing an override for advertised names. Has anyone gotten Drill
to scale in a K8s cluster?
>  
> https://issues.apache.org/jira/browse/KAFKA-1070


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message