drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [drill] cgivre opened a new pull request #2092: DRILL-7763: Add Limit Pushdown to File Based Storage Plugins
Date Fri, 03 Jul 2020 04:18:34 GMT

cgivre opened a new pull request #2092:
URL: https://github.com/apache/drill/pull/2092


   # [DRILL-7763](https://issues.apache.org/jira/browse/DRILL-7763): Add Limit Pushdown to
File Based Storage Plugins
   
   ## Description
   
   As currently implemented, when querying a file, Drill will read the entire file even if
a limit is specified in the query.  This PR does a few things:
   Refactors the `EasyGroupScan`, `EasySubScan`, and `EasyFormatConfig` to allow the option
of pushing down limits.
   Applies this to all the EVF based format plugins which are: LogRegex, PCAP, SPSS, Esri,
Excel and Text (CSV). 
   Due to JSON's fluid schema, it would be unwise to adopt the limit pushdown as it could
result in very inconsistent schemata.
   
   ## Documentation
   No user visible changes.  Queries with limits on large files are considerably faster.
   
   ## Testing
   All existing unit tests are run. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message