drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Girish <agir...@apache.org>
Subject Re: scaling drill in an openshift (K8s) cluster
Date Tue, 24 Mar 2020 18:37:54 GMT
I've added support for auto-scaling and I've tested that it works well.
Please see:
https://github.com/Agirish/drill-helm-charts#autoscaling-drill-clusters
And I have a script to test this:
https://github.com/Agirish/drill-helm-charts/blob/master/scripts/runCPULoadTest.sh

In  case of cloud deployments like  GKE, where I had a Loadbalancer (and an
external IP), I could connect with external clients as well, using  the
JDBC Drillbit direct connection.

On Tue, Mar 24, 2020 at 9:17 AM Arina Ielchiieva <arina.yelchiyeva@gmail.com>
wrote:

> Please see https://issues.apache.org/jira/browse/DRILL-7563 <
> https://issues.apache.org/jira/browse/DRILL-7563>, maybe it will be
> helpful.
>
> Kind regards,
> Arina
>
> > On Mar 24, 2020, at 6:04 PM, Dobes Vandermeer <dobesv@gmail.com> wrote:
> >
> > I was able to get drill up and running inside a k8s cluster but I didn't
> connect to it from outside the cluster, so the DNS names were always
> resolvable by the client(s).
> >
> > I had to run it as a statefulset to ensure the DNS names are stable,
> otherwise the drillbits couldn't talk to each other, either.
> >
> > On 3/24/2020 6:37:44 AM, Jaimes, Rafael - 0993 - MITLL <
> rafael.jaimes@ll.mit.edu> wrote:
> > I’m seeing a problem with scaling the number of pod instances in the
> replication controller because they aren’t reporting their hostnames
> properly. This was a common problem that got fixed in scalable
> architectures like ZooKeeper and Kafka (see reference at bottom I think
> this was related).
> >
> > In Drill’s case, ZooKeeper is able to see all of the drillbits, however,
> the hostnames are only locally addressable within the cluster, so as soon
> as you perform a query it fails since the client can’t find the drillbit
> that it got assigned, its hostname isn’t externally addressable.
> >
> > Kafka fixes this by allowing an override for advertised names. Has
> anyone gotten Drill to scale in a K8s cluster?
> >
> > https://issues.apache.org/jira/browse/KAFKA-1070
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message