fluo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] keith-turner opened a new issue #1023: Consider placing notifications in separate table
Date Fri, 09 Mar 2018 15:43:40 GMT
keith-turner opened a new issue #1023: Consider placing notifications in separate table
URL: https://github.com/apache/fluo/issues/1023
 
 
   Fluo's notifications are currently placed in a separate locality group.  This means when
scanning for notifications, user data is not read. Notifications can only be garbage collected
with a full major compaction.    However full major compactions do not happen often.  This
means that over time notification delete markers build up.   This build up over time makes
scanning for notifications more expensive.
   
   Work was done for #457 to drop delete marker for notifications that were inserted and processed
before an Accumulo minor compaction happens.  The work in #457 also has a chance of dropping
delete markers on partial compactions.  However if a delete marker makes it to the largest
files, it may be there for a long time.    The effectiveness of #457 is inversely proportional
to the average time it takes to process a notification.  As the average time goes up, the
effectiveness goes down.  If too many delete markers have built up, then the best work around
is a full compaction of the table.
   
   If notifications were in a separate table, then the full compaction could be cheap.  Also
the work done for [ACCUMULO-4500](https://issues.apache.org/jira/browse/ACCUMULO-4500) could
be leveraged to automatically make smart decisions about when to compact based on the number
of notifications vs the number of delete markers.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message