trafodion-codereview mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sandhyasun <...@git.apache.org>
Subject [GitHub] trafodion pull request #1503: [TRAFOODION-2949] Performance changes for LOB ...
Date Mon, 02 Apr 2018 17:43:57 GMT
GitHub user sandhyasun opened a pull request:

    https://github.com/apache/trafodion/pull/1503

    [TRAFOODION-2949] Performance changes for LOB and increasing limit for larger varchars/…

    …bobs to use with batch input.
    For LOB internal descriptor files, there was a check in compiler that prevented caching
of NA Table . Removed that restriction for LOB descriptor tables  which affected evey DML
into a table with a LOB column. 
    Fix to allow larger varchar/lob data to be input with batch input.
    1.  	First check for rowset size is in pre codegen phase. This was added during POC since
we were getting 4 GB of rowset data all at once.
    CQD MEMORY_LIMIT_ROWSET_IN_MB  was added and  set to 500MB.   The data + rowset size needed
to fit into this space. I have changed it to be 1GB by defaults now.
    
    2.	Then we check how big the up  queuesize should be to handle this rowset .  
    CQD EXE_MEMORY_FOR_UNPACK_ROWS_IN_MB  is set to 100MB for this.  In this case we dynamically
adjust the queue size to accommodate larger rows.  Eg is we have 100 rows of  2 MB each, we
will limit upqueue size to 50 and allow 50 entries in upqueue. 
    
    I have changed the default value for this CQD to be 1GB to match (1.)
    
    3.	At runtime, the problem is that after passing these 2 checks, we try to allocate double
the size of the up queue size for the full row length .  
    
    Assume we calculate in step(2)  the upqueue size is N,  then we allocate rowlength*N*2
  at runtime. This clearly exceeds what we have estimated at compiletime. 
    This change includes a change in compiler to estimate the memory correctly. 
    
    
    So with these we can handle upto 16MB of long varchar/LOB data for batch input data a
little better.   But driver will still need to ensure that the rowset buffer will fit into
the 1GB limit otherwise an error will be returned at compiletime. So for batch input for large
rows , we need to  limit number of rows in the rowset. 
    
    Also included a fix for TRAFODION-909

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sandhyasun/trafodion traf_lob_global_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/trafodion/pull/1503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1503
    
----
commit 528e9c60345234e554d32db637c82728b63c945b
Author: Sandhya Sundaresan <sandhya@...>
Date:   2018-03-29T05:56:15Z

    Performance changes for LOB and increasing limit for larger varchars/bobs to use with
batch input.

----


---

Mime
View raw message