drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <par0...@yahoo.com.INVALID>
Subject Re: Drill on YARN Questions
Date Tue, 18 Dec 2018 02:57:32 GMT
Hi Charles,
I'm not quite sure what "dynamic queue allocation" means: all YARN containers are allocated
dynamically through YARN via queues. 
It may be helpful to review how Drill-on-YARN (DoY) works. DoY does NOT attempt to use YARN
for each query. Impala tried that with Llama and discovered that the latency of YARN allocation
is not compatible with the needs of a query engine: YARN takes 10s of seconds to launch containers,
queries must complete in fractions of a second.
Instead, DoY treats YARN as a resource manger for long-running applications. Think of it as
an old-school Kubernetes. That is, you use YARN to launch Drill, and to account for the cluster
resources used by Drill. The DoY UI allows you to grow/shrink the cluster, which turns around
and asks YARN for more or fewer containers.
(Jyothsna; we should integrate your graceful shutdown work into DoY for cluster shrinking.)
To be very clear, Drill is long-running and clusters grow or shrink over long periods of time
(perhaps over a day: more Drill during the day, less at night.) Queries are rapid-fire and
run on the available Drillbits.

DoY is designed for a multi-tenant setup. The only trick is that each tenant cluster must
assign distinct ports and ZK roots. The details are spelled out in the DoY docs. (K8s avoids
the need for mucking with ports via an overlay network, something that YARN does not provide.)

IIf the above leaves questions open, please do provide a bit more detail about what you want
to achieve.

- Paul


    On Monday, December 17, 2018, 8:53:54 AM PST, Charles Givre <cgivre@gmail.com> wrote:
 Hello all, 
We are trying to set up a Drill cluster on our corporate data lake.  Our cluster requires
dynamic YARN queue allocation for multi-tenant environment.  Is this something that Drill
supports or is there a workaround?
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message