systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [systemml] mboehm7 opened a new pull request #927: [SYSTEMDS-274] Fix compressed colMins/colMaxs w/ shared dictionary
Date Sat, 30 May 2020 21:12:43 GMT

mboehm7 opened a new pull request #927:
URL: https://github.com/apache/systemml/pull/927


   This patch fixes remaining issues of incorrect results for colMins and
   colMaxs over compressed matrix blocks with shared DDC1 dictionaries.
   Specifically, if the individual column groups have only partial overlap,
   the shared dictionary contains a superset of column group distinct
   values. Since aggregation functions like min and max are executed only
   over the dictionary (without touching the compressed data), it led to
   incorrect results as we find extreme values that do not actually exist
   in the column group.
   
   Three alternatives approaches could solve this: (1) drop shared
   dictionaries, (2) execute colMins and colMaxs over the compressed data,
   or (3) refactor the double array dictionary into a proper class
   hierarchy and maintain additional meta data for shared dictionaries. We
   decided for (3) in order to keep predictable performance, irrespective
   of shared dictionaries and because this class hierarchy allows for
   further improvements of shared dictionaries between any subsets of
   column groups.
   
   Additionally, this fix also cleanups incorrect estimates of the
   individual column groups (because getValueSize was used in the estimates
   as a number of values, although it gave the size in bytes) as well as
   some of the Class-layout size estimation tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message