drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [drill] vvysotskyi opened a new pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
Date Fri, 13 Mar 2020 20:00:01 GMT
vvysotskyi opened a new pull request #2026: DRILL-7330: Implement metadata usage for all format
plugins
URL: https://github.com/apache/drill/pull/2026
 
 
   # [DRILL-7330](https://issues.apache.org/jira/browse/DRILL-7330): Implement metadata usage
for all format plugins
   
   ## Description
   This pull request adds support of all format plugins for Metastore.
   
   It contains two parts:
   - Making changes to collect metadata for easy group scan
   - Allowing usage Metastore metadata for easy group scan during querying data
   
   To reach the first goal was added AnalyzeFileInfoProviderImpl to handle correctly EasyGroupScan
tables in MetastoreAnalyzeTableHandler and specify correct table type.
   Accommodated EVF scan framework to incorporate Metastore-specific additional implicit columns
and handle the case of empty files when metadata should be returned.
   
   To reach the second goal, EasyGroupScan was updated to extend AbstractGroupScanWithMetadata.
   MetastoreFileTableMetadataProvider based on existing MetastoreParquetTableMetadataProvider
was introduced to provide metastore metadata for file-based tables.
   Accommodated ParquetPushDownFilter rule to work with easy format plugins.
   
   Refactored some code and fixed additional issues.
   
   ## Documentation
   After this PR is merged, Drill Web Site docs should be updated to reflect this feature.
   
   ## Testing
   Added tests to verify the correctness of collected metadata and ensure that new optimizations
work correctly.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message