trafodion-codereview mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sureshsubbiah <...@git.apache.org>
Subject [GitHub] incubator-trafodion pull request #637: [TRAFODION-2138] Hive scan on wide ta...
Date Wed, 03 Aug 2016 21:31:25 GMT
Github user sureshsubbiah commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/637#discussion_r73425266
  
    --- Diff: core/sql/optimizer/BindRelExpr.cpp ---
    @@ -7582,6 +7582,24 @@ RelExpr *Scan::bindNode(BindWA *bindWA)
             }
         }
     
    +   if (naTable->isHiveTable() && 
    +       !(naTable->getClusteringIndex()->getHHDFSTableStats()->isOrcFile() ||
    +	 naTable->getClusteringIndex()->getHHDFSTableStats()
    +	 ->isSequenceFile()) &&
    +       (CmpCommon::getDefaultNumeric(HDFS_IO_BUFFERSIZE_BYTES) == 0) && 
    +       (naTable->getRecordLength() >
    +	CmpCommon::getDefaultNumeric(HDFS_IO_BUFFERSIZE)*1024))
    +     {
    +       // do not raise error if buffersize is set though buffersize_bytes.
    --- End diff --
    
    HDFS_IO_BUFFERSIZE_BYTES was added initially in cases where we wanted to specify the buffer
size in granularity less than a KB. The comment indicates that this is for testing. I was
tempted to remove the cqd, but thought we may still need to test with precisely defined buffer
sizes.
    
    I am viewing BUFFERSIZE_BYTES as some kind of internal switch, to be used by expert users
to override the BUFFERSIZE setting. In case users have a wide table but for some reason want
to avoid the new error they can use BUFFERSIZE_BYTES to skip the new error code. Wide tables
can be handled if the actual rows are indeed small enough to fit in a buffer, the user may
have other reasons for not wanting to changing defaults such as HIVE_MAX_STRING_LENGTH.
    
    If we prefer, I can also remove this BUFFERSIZE_BYTES backdoor.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message