drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dobes Vandermeer" <dob...@gmail.com>
Subject Re: scaling drill in an openshift (K8s) cluster
Date Tue, 24 Mar 2020 16:04:24 GMT
I was able to get drill up and running inside a k8s cluster but I didn't connect to it from
outside the cluster, so the DNS names were always resolvable by the client(s).

I had to run it as a statefulset to ensure the DNS names are stable, otherwise the drillbits
couldn't talk to each other, either.

On 3/24/2020 6:37:44 AM, Jaimes, Rafael - 0993 - MITLL <rafael.jaimes@ll.mit.edu> wrote:
I’m seeing a problem with scaling the number of pod instances in the replication controller
because they aren’t reporting their hostnames properly. This was a common problem that got
fixed in scalable architectures like ZooKeeper and Kafka (see reference at bottom I think
this was related).
 
In Drill’s case, ZooKeeper is able to see all of the drillbits, however, the hostnames are
only locally addressable within the cluster, so as soon as you perform a query it fails since
the client can’t find the drillbit that it got assigned, its hostname isn’t externally
addressable.
 
Kafka fixes this by allowing an override for advertised names. Has anyone gotten Drill to
scale in a K8s cluster?
 
https://issues.apache.org/jira/browse/KAFKA-1070
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message