drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] vvysotskyi commented on a change in pull request #1667: DRILL-7069: Moved version checks outside loops in transformBinaryInMetadataCache
Date Fri, 01 Mar 2019 11:36:10 GMT
vvysotskyi commented on a change in pull request #1667: DRILL-7069: Moved version checks outside
loops in transformBinaryInMetadataCache
URL: https://github.com/apache/drill/pull/1667#discussion_r261518703
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
 ##########
 @@ -292,30 +294,47 @@ public static void transformBinaryInMetadataCache(ParquetTableMetadataBase
parqu
     Set<List<String>> columnsNames = getBinaryColumnsNames(parquetTableMetadata);
     boolean allowBinaryMetadata = allowBinaryMetadata(parquetTableMetadata.getDrillVersion(),
readerConfig);
 
+    // Setting Min / Max values for ParquetTableMetadata_v1
+    if (new MetadataVersion(1, 0).equals(new MetadataVersion(parquetTableMetadata.getMetadataVersion())))
{
+      for (ParquetFileMetadata file : parquetTableMetadata.getFiles()) {
+        for (RowGroupMetadata rowGroupMetadata : file.getRowGroups()) {
+          Long rowCount = rowGroupMetadata.getRowCount();
+          for (ColumnMetadata columnMetadata : rowGroupMetadata.getColumns()) {
+              if (columnMetadata.getPrimitiveType() == PrimitiveTypeName.BINARY || columnMetadata.getPrimitiveType()
== PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+                setMinMaxValues(columnMetadata, rowCount, allowBinaryMetadata, false);
+              }
+          }
+        }
+      }
+      return;
+    }
+
+    // Variables needed for debugging only
+    Stopwatch timer = logger.isDebugEnabled() ? Stopwatch.createStarted() : null;
+    int maxRowGroups = 0, minRowGroups = 99999, maxNumColumns = 0;
+
+    // Setting Min / Max values for V2 and V3 versions; for versions V3_3 and above need
to do decoding
+    boolean needDecoding = new MetadataVersion(parquetTableMetadata.getMetadataVersion()).compareTo(new
MetadataVersion(3, 3)) >= 0;
     for (ParquetFileMetadata file : parquetTableMetadata.getFiles()) {
+      maxRowGroups = Math.max(maxRowGroups,file.getRowGroups().size());
 
 Review comment:
   These two lines will be executed even for disabled debug. Could you please wrap it with
`if`?
   And please add space after comma: `Math.max(maxRowGroups,file.getRowGroups().size())` ->
`Math.max(maxRowGroups, file.getRowGroups().size())`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message