drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: query performance with unequal drillbits
Date Mon, 27 Aug 2018 13:16:45 GMT
I will +1 Ted's idea. By doing small drillbits, it does take a bit more
overhead, but you also have an ability to scale your Drill cluster size
(especially using the Drillbit shutdown features added recently).



On Wed, Aug 22, 2018 at 8:23 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Cool
>
> On Wed, Aug 22, 2018, 17:07 scott <tcots8888@gmail.com> wrote:
>
> > Thanks Ted and Paul. I've been experimenting with the "hack" method. It
> > works somewhat, and I guess will have to do.
> >
> > On Tue, Aug 21, 2018 at 2:50 PM Ted Dunning <ted.dunning@gmail.com>
> wrote:
> >
> > > A cheap hack is to use multiple smaller drillbits. Put more drillbits
> on
> > > the hefty machines and fewer on the weaker ones.
> > >
> > > This increases overheads, but it might help you out.
> > >
> > >
> > >
> > > On Tue, Aug 21, 2018 at 1:48 PM scott <tcots8888@gmail.com> wrote:
> > >
> > > > Hi community,
> > > > I am trying to find a way to tune Drill so that weaker drillbits get
> > less
> > > > data to work on so that the weak link doesn't drag my performance
> > down. I
> > > > have drillbits running on a variety of hardware and sometimes these
> > > shared
> > > > resources get really slow. It seems that the query plan always evenly
> > > > divides the data fragments so that each drillbit gets the same data
> to
> > > chew
> > > > on. How do I make it give weaker drillbits less data?
> > > >
> > > > Alternatively, is there a way to limit and queue fragments of the
> query
> > > and
> > > > leave them unassigned, then assign to drillbits as their resources
> > become
> > > > free, similar to MapReduce?
> > > >
> > > > Thanks for you time,
> > > > Scott
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message