drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [drill] paul-rogers commented on issue #1675: DRILL-7055: Revise SELECT * to exclude partitions
Date Mon, 04 Mar 2019 18:14:56 GMT
paul-rogers commented on issue #1675: DRILL-7055: Revise SELECT * to exclude partitions
URL: https://github.com/apache/drill/pull/1675#issuecomment-469357589
 
 
   @arina-ielchiieva, if we want to do this for 2.0 then let's keep the current behavior.
I'll need to do more fixes to the scan framework, but I'll proceed with that work. Far too
difficult to add another option and have to test both paths; doing so would double the effort.
   
   My guess is that the original idea was that the partition directories have data and should
be considered columns. For example, if data is partitioned by year and month, then those might
want to be columns. This is particularly important if the partition columns do not appear
within the data itself.
   
   Using "dir0" and "dir1" has always been a hack: they are meaningless names and require
that the user map from "year" to the partitioning structure and know that, in one table "dir0"
means "year", while in another it might mean "store". Moving forward, it would be better to
use a Hive-like solution: a mapping from partition directories to columns so that the user
sees "year" and "month" as column names, not "dir0" and "dir1". Sounds like a job for the
new metadata system.
   
   If we want a generic directory solution, then allowing a "dir" array would be handy: the
array can allow any number of entries, avoiding the schema change issues inherent with the
current design.
   
   Agree, all these are more than we can do short term, so let's mothball this change for
the Drill 1.x series.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message