drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Karpinski <mkarpin...@opera.com>
Subject Counting large numbers of unique values
Date Tue, 07 Apr 2015 10:30:56 GMT
Hi Guys,

I have a specific use case for Drill, in which I'd like to be able to count
unique values in columns with tens millions of distinct values. The COUNT
DISTINCT method, unfortunately, does not scale both time- and memory-wise
and the idea is to sort the data beforehand by the values of that column
(let's call it ID), to have the row groups split at new a new ID boundary
and to extend Drill with an alternative version of COUNT that would simply
count the number of times the ID changes through out the entire table. This
way, we could expect that counting unique values of pre-sorted columns
could have complexity comparable to that of the regular COUNT operator (a
full scan). So, to sum up, I have three questions:

1. Can such a scenario be realized in Drill?
2. Can it be done in a modular way (eg, a dedicated UDAF or an operator),
so without heavy hacking throughout entire Drill?
3. How to do it?

Our initial experience with Drill was very good - it's an excellent tool.
But in order to be able to adopt it, we need to sort out this one central
issue.

Cheers,
Marcin

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message