hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Latham <lat...@davelink.net>
Subject scan column families with different time ranges
Date Sat, 01 Aug 2015 14:17:11 GMT
I have a table with 2 column families, call them A and B, with new data
regularly being added. They are very different sizes: B is 100x the size of
A.  Among other uses for this data, I have a MapReduce job that needs to
read all of A, but only recent data from B (e.g. last day).  Here are some
methods I've considered:

   1. Use a Filter to get throw out older data from B (this is what I
   currently do).  However, all the data from B still needs to be read from
   disk, causing a disk IO bottleneck.
   2. Configure the table input format to read from B only, using a
   TimeRange for recent data, and have each map task open a separate scanner
   for A (without a TimeRange) then merge the data in the map task.  However,
   this adds complexity to the job and gives up the atomicity/consistency
   guarantees as new writes hit both column families.
   3. Add a new column family C to the table with an additional copy of the
   data in B, but set a TTL on it.  All writes duplicate the data written to B
   and C.  Change the scan to include C instead of B.  However, this adds all
   the overhead of another column family, more writes, and having to set the
   TTL to the maximum of any time window I want to scan efficiently.
   4. Implement an enhancement to HBase's Scan to allow giving each column
   family its own TimeRange.  The job would then be able to skip most old
   large store files (hopefully all of them with tiered compaction at some

Does anyone have other suggestions?  Would HBase be willing to accept
updating Scan to have different TimeRange's for each column families?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message