hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject [ANNOUNCE] Apache HBase 1.4.0 is now available for download
Date Tue, 19 Dec 2017 19:32:36 GMT
The HBase team is happy to announce the immediate availability of Apache
​HBase​ 1.4.0!

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

Download through an ASF mirror:

    https://www.apache.org/dyn/closer.lua/hbase/1.4.0

HBase 1.4.0 is the first release in the new HBase 1.4 line, continuing on
the theme of earlier 1.x releases of bringing a stable, reliable database
to the Apache Big Data ecosystem and beyond. As a minor release, 1.4.0
contains a number of new features and improvements that won't appear in
maintenance releases of older code lines. However complete compatibility
with data formats and interoperability with older clients is assured.
There are no special considerations for upgrade or rollback except as
noted in this release announcement.

Maintenance releases of the 1.4.0 code line will occur at roughly a
monthly cadence.

For instructions on verifying ASF release downloads, please see

    https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at

    https://www.apache.org/dist/hbase/KEYS

Thanks to all the contributors who made this release possible!

The complete list of the 660 issues resolved in this release can be found
at https://s.apache.org/OErT . New developer and user-facing
incompatibilities, important issues, features, and major improvements
include:

Critical

- HBASE-15484 Correct the semantic of batch and partial

    Now setBatch doesn't mean setAllowPartialResult(true)

    Scan#setBatch is helpful in paging queries, if you just want to prevent
    OOM at client, use setAllowPartialResults(true) is better.

    We deprecated isPartial and use mayHaveMoreCellsInRow. If it returns
    false, current Result must be the last one of this row.

- HBASE-17287 Master becomes a zombie if filesystem object closes

    If filesystem is not available during log split, abort master server.

- HBASE-17471 Region Seqid will be out of order in WAL if using
  mvccPreAssign

    MVCCPreAssign is added by HBASE-16698, but pre-assign mvcc is only used
    in put/delete path. Other write paths like increment/append still assign
    mvcc in ringbuffer's consumer thread. If put and increment are used
    parallel. Then seqid in WAL may not increase monotonically. Disorder in
    wals will lead to data loss. This patch bring all mvcc/seqid event in
    wal.append, and synchronize wal append and mvcc acquirement. No disorder
    in wal will happen. Performance test shows no regression.

- HBASE-17595 Add partial result support for small/limited scan

    Now small scan and limited scan can also return partial results.

- HBASE-17717 Incorrect ZK ACL set for HBase superuser

    In previous versions of HBase, the system intended to set a ZooKeeper
    ACL on all "sensitive" ZNodes for the user specified in the
    hbase.superuser configuration property. Unfortunately, the ACL was
    malformed which resulted in the hbase.superuser being unable to access
    the sensitive ZNodes that HBase creates. HBase will automatically
    correct the ACLs on start so users do not need to manually correct the
    ACLs.

- HBASE-17887 Row-level consistency is broken for read

    Now we pass on list of memstoreScanners to the StoreScanner along with
    the new files to ensure that the StoreScanner sees the latest memstore
    after flush.

- HBASE-17931 Assign system tables to servers with highest version

    We usually keep compatibility between old client and new server so we
    can do rolling upgrade, HBase cluster first, then HBase client. But we
    don't guarantee new client can access old server. In an HBase cluster,
    we have system tables and region servers will access these tables so for
    servers they are also an HBase client. So if the system tables are in
    region servers with lower version we may get trouble because region
    servers with higher version may can not access them. After this patch,
    we will move all system regions to region servers with highest version.
    So when we do a rolling upgrade across two major or minor versions, we
    should ALWAYS UPGRADE MASTER FIRST and then upgrade region
​ ​
   servers.
​ ​
The new master will handle system tables correctly.

- HBASE-18035 Meta replica does not give any primaryOperationTimeout to
  primary meta region

    When a client is configured to use meta replica, it sends scan request
    to all meta replicas almost at the same time. Since meta replica
    contains stale data, if result from one of replica comes back first, the
    client may get wrong region locations. To fix this,
    "hbase.client.meta.replica.scan.timeout" is introduced, a client will
    always send to primary meta region first, wait the configured timeout
    for reply. If no result is received, it will send request to replica
    meta regions. The unit for "hbase.client.meta.replica.scan.timeout" is
    microsecond, the default value is 1000000 (1 second).

- HBASE-18137 Replication gets stuck for empty WALs

    0-length WAL files can potentially cause the replication queue to get
    stuck.  A new config "replication.source.eof.autorecovery" has been
    added: if set to true (default is false), the 0-length WAL file will be
    skipped after 1) the max number of retries has been hit, and 2) there
    are more WAL files in the queue.  The risk of enabling this is that
    there is a chance the 0-length WAL file actually has some data (e.g.
    block went missing and will come back once a datanode is recovered).

- HBASE-18164 Much faster locality cost function and candidate generator

    New locality cost function and candidate generator that use caching and
    incremental computation to allow the stochastic load balancer to
    consider ~20x more cluster configurations for big clusters.

- HBASE-18192 Replication drops recovered queues on region server shutdown

    If a region server that is processing recovered queue for another
    previously dead region server is gracefully shut down, it can drop the
    recovered queue under certain conditions. Running without this fix on a
    1.2+ release means possibility of continuing data loss in replication,
    irrespective of which WALProvider is used. If a single WAL group (or
    DefaultWALProvider) is used, running without this fix will always cause
    dataloss in replication whenever a region server processing recovered
    queues is gracefully shutdown.

- HBASE-18233 We shouldn't wait for readlock in doMiniBatchMutation in case
  of deadlock

    This patch plus the sort of mutations done in HBASE-17924 fixes a
    performance regression doing increments and/or checkAndPut-style
    operations.

- HBASE-18255 Time-Delayed HBase Performance Degradation with Java 7

    This change sets the JVM property ReservedCodeCacheSize to 256MB in the
    provided hbase-env.sh example file. The specific value for this property
    attempts to prevent performance issues seen when HBase using Java 7. The
    value set is the same as the default when using Java 8.

- HBASE-18469 Correct RegionServer metric of  totalRequestCount

    We introduced a new RegionServer metrics in name of
    "totalRowActionRequestCount" which counts in all row actions and equals
    to the sum of "readRequestCount" and "writeRequestCount". Meantime, we
    have changed "totalRequestCount" to count only once for multi request,
    while previously we will count in action number of the request. As a
    result, existing monitoring system on totalRequestCount will still work
    but see a smaller value, and we strongly recommend to change to use the
    new metrics to monitor server load.

- HBASE-18577 shaded client includes several non-relocated third party
  dependencies

    The HBase shaded artifacts (hbase-shaded-client and hbase-shaded-server)
    no longer contain several non-relocated third party dependency classes
    that were mistakenly included. Downstream users who relied on these
    classes being present will need to add a runtime dependency onto an
    appropriate third party artifact. Previously, we erroneously packaged
    several third party libs without relocating them. In some cases these
    libraries have now been relocated; in some cases they are no longer
    included at all.

    Includes:

      * jaxb
      * jetty
      * jersey
      * codahale metrics (HBase 1.4+ only)
      * commons-crypto
      * jets3t
      * junit
      * curator (HBase 1.4+)
      * netty 3 (HBase 1.1)
      * mokito-junit4 (HBase 1.1)

- HBASE-18665 ReversedScannerCallable invokes getRegionLocations
  incorrectly

    Performing reverse scan on tables used the meta cache incorrectly and
    fetched data from meta table every time. This fix solves this issue and
    which results in performance improvement for reverse scans.

- HBASE-19285 Add per-table latency histograms

    Per-RegionServer table latency histograms have been returned to HBase
    (after being removed due to impacting performance). These metrics are
    exposed via a new JMX bean "TableLatencies" with the typical naming
    conventions: namespace, table, and histogram component.

Major

- HBASE-7621 REST client doesn't support binary row keys

    RemoteHTable now supports binary row keys with any character or byte
    by properly encoding request URLs. This is a both a behavioral change
    from earlier versions and an important fix for protocol correctness.

- HBASE-11013 Clone Snapshots on Secure Cluster Should provide option to
  apply Retained User Permissions

    While creating a snapshot, it will save permissions of the original
    table into .snapshotinfo file(Backward compatibility), which is in the
    snapshot root directory.  For clone_snapshot/restore_snapshot command,
    we provide an additional option (RESTORE_ACL) to decide whether we will
    grant permissions of the origin table to the newly created table.

- HBASE-14548 Expand how table coprocessor jar and dependency path can
  be specified

    Allows a directory containing the jars or some wildcards to be
    specified, such as:

      hdfs://namenode:port/user/hadoop-user/

    or

      hdfs://namenode:port/user/hadoop-user/*.jar

    Please note that if a directory is specified, all jar files directly
    in the directory are added, but it does not search files in the
    subtree rooted in the directory. Do not use a wildcard if you would
    like to specify a directory.

- HBASE-14925 HBase shell command/tool to list table's region info through
  command line

    Added a shell command 'list_regions' for displaying the table's region
    info through command line. List all regions for a particular table as an
    array and also filter them by server name (optional) as prefix and
    maximum locality (optional). By default, it will return all the regions
    for the table with any locality. The command displays server name,
    region name, start key, end key, size of the region in MB, number of
    requests and the locality. The information can be projected out via an
    array as third parameter. By default all these information is displayed.
    Possible array values are SERVER_NAME, REGION_NAME, START_KEY,
    END_KEY, SIZE, REQ and LOCALITY. Values are not case sensitive. If you
    don't want to filter by server name, pass an empty hash or string.

- HBASE-15187 Integrate CSRF prevention filter to REST gateway

    Protection against CSRF attack can be turned on with config
    parameter, hbase.rest.csrf.enabled - default value is false.

    The custom header to be sent can be changed via config parameter,
    hbase.rest.csrf.custom.header whose default value is
    "X-XSRF-HEADER".

    The configuration parameter hbase.rest.csrf.methods.to.ignore
    controls which HTTP methods are not associated with custom header
    check.

    The config parameter hbase.rest-csrf.browser-useragents-regex is a
    comma-separated list of regular expressions used to match against an
    HTTP request's User-Agent header when protection against cross-site
    request forgery (CSRF) is enabled for REST server by setting
    hbase.rest.csrf.enabled to true.

- HBASE-15236 Inconsistent cell reads over multiple bulk-loaded HFiles

    During bulkloading, if there are multiple hfiles corresponding to
    the same region, and if they have same timestamps (which may have
    been set using importtsv.timestamp) and duplicate keys across them,
    then get and scan may return values coming from different hfiles.

- HBASE-15243 Utilize the lowest seek value when all Filters in
  MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT

    When all filters in a MUST_PASS_ONE FilterList return a
    SEEK_USING_NEXT_HINT code, we return SEEK_NEXT_USING_HINT from
    FilterList#filterKeyValue() to utilize the lowest seek value.

- HBASE-15386 PREFETCH_BLOCKS_ON_OPEN in HColumnDescriptor is ignored

    Changes the prefetch TRACE-level loggings to include the word
    'Prefetch' in them so you know what they are about.

    Changes the cryptic logging of the CacheConfig#toString to have
    some preamble saying why and what column family is responsible.

- HBASE-15576 Scanning cursor to prevent blocking long time on
  ResultScanner.next()

    If you don't like scanning being blocked too long because of heartbeat
    and partial result, you can use Scan#setNeedCursorResult(true) to get a
    special result within scanning timeout setting time which will tell you
    where row the server is scanning. See its javadoc for more details.

- HBASE-15633 Backport HBASE-15507 to branch-1

    Adds update_peer_config to the HBase shell and ReplicationAdmin, and
    provides a callback for custom replication endpoints to be notified
    of changes to configuration and peer data

- HBASE-15686 Add override mechanism for the exempt classes when
  dynamically loading table coprocessor

    The
​n​
ew coprocessor table descriptor attribute,
    hbase.coprocessor.classloader.included.classes, is added. A user can
    specify class name prefixes (semicolon separated) which should be
    loaded by CoprocessorClassLoader.

- HBASE-15711 Add client side property to allow logging details for
  batch errors

    A new client side property hbase.client.log.batcherrors.details is
    introduced to allow logging the full stacktrace of exceptions for
    batch errors. It is disabled by default.

- HBASE-15816 Provide client with ability to set priority on Operations

    Added setPriority(int priority) API to Put, Delete, Increment, Append,
    Get and Scan. For all these ops, the user can provide a custom RPC
    priority level.

- HBASE-15924 Enhance hbase services autorestart capability to
  hbase-daemon.sh

    Now one can start hbase services with enabled "autostart/autorestart"
    feature in controlled fashion with the help of "--autostart-window-size"
    to define the window period and the "--autostart-window-retry-limit" to
    define the number of times the hbase services have to be restarted upon
    being killed/terminated abnormally within the provided window perioid.

    The following cases are supported with "autostart/autorestart":

    a) --autostart-window-size=0 and --autostart-window-retry-limit=0,
       indicates infinite window size and no retry limit
    b) not providing the args, will default to a)
    c) --autostart-window-size=0 and --autostart-window-retry-limit=
       <positive value> indicates the autostart process to bail out if the
       retry limit exceeds irrespective of window period
    d) --autostart-window-size=<x> and --autostart-window-retry-limit=<y>
       indicates the autostart process to bail out if the retry limit "y"
       is exceeded for the last window period "x".

- HBASE-15941 HBCK repair should not unsplit healthy splitted region

    A new option -removeParents is now available that will remove an old
    parent when two valid daughters for that parent exist and
    -fixHdfsOverlaps is used. If there is an issue trying to remove the
    parent from META or sidelining the parent from HDFS we will fallback to
    do a regular merge. For now this option only works when the overlap
    group consists only of 3 regions (a parent, daughter A and daughter B)

- HBASE-15950 Fix memstore size estimates to be tighter

    The estimates of heap usage by the memstore objects (KeyValue, object
    and array header sizes, etc) have been made more accurate for heap
    sizes up to 32G (using CompressedOops), resulting in them dropping by
    10-50% in practice. This also results in less number of flushes and
    compactions due to "fatter" flushes. As a result, the actual heap usage
    of the memstore before being flushed may increase by up to 100%. If
    configured memory limits for the region server had been tuned based on
    observed usage, this change could result in worse GC behavior or even
    OutOfMemory errors. Set the environment property (not hbase-site.xml)
    "hbase.memorylayout.use.unsafe" to false to disable.

- HBASE-15994 Allow selection of RpcSchedulers

    Adds a FifoRpcSchedulerFactory so you can try the FifoRpcScheduler by
    setting "hbase.region.server.rpc.scheduler.factory.class"

- HBASE-16052 Improve HBaseFsck Scalability

    Improves the performance and scalability of HBaseFsck, especially for
    large clusters with a small number of large tables.

    Searching for lingering reference files is now a multi-threaded
    operation.  Loading HDFS region directory information is now multi-
    threaded at the region-level instead of the table-level to maximize
    concurrency.  A performance bug in HBaseFsck that resulted in
    redundant I/O and RPCs was fixed by introducing a FileStatusFilter
    that filters FileStatus objects directly.

- HBASE-16213 A new HFileBlock structure for fast random get

    Introduces a new DataBlockEncoding in name of ROW_INDEX_V1, which
    could improve random read (get) performance especially when the
    average record size (key-value size per row) is small. To use this
    feature, please set DATA_BLOCK_ENCODING to ROW_INDEX_V1 for column
    family of newly created table, or change existing CF with below shell
    command:

      alter 'table_name',{NAME => 'cf', DATA_BLOCK_ENCODING =>
         'ROW_INDEX_V1'}.

    Please note that if we turn this DBE on, HFile block will be bigger
    than NONE encoding because it adds some metadata for binary search.

    Seek in row when random reading is one of the main consumers of CPU.
    This helps.

- HBASE-16244 LocalHBaseCluster start timeout should be configurable

    When LocalHBaseCluster is started from the command line the Master
    would give up after 30 seconds due to a hardcoded timeout meant for
    unit tests. This change allows the timeout to be configured via
    hbase-site as well as sets it to 5 minutes when LocalHBaseCluster
    is started from the command line.

- HBASE-16336 Removing peers seems to be leaving spare queues

    Add a ReplicationZKNodeCleaner periodic check and delete any useless
    replication queue belonging to a peer which does not exist.

- HBASE-16388 Prevent client threads being blocked by only one slow
  region server

    Adds a new configuration, hbase.client.perserver.requests.threshold,
    to limit the max number of concurrent request to one region server.
    If the user still create new request after reaching the limit,
    client will throw ServerTooBusyException and do not send the request
    to the server. This is a client side feature and can prevent client's
    threads being blocked by one slow region server resulting in the
    availability of client is much lower than the availability of region
    servers.

- HBASE-16540 Scan should do additional validation on start and stop row

    Scan#setStartRow() and Scan#setStopRow() now validate the argument
    passed for each row key.  If the length of the parameter passed
    exceeds Short.MAX_VALUE, an IllegalArgumentException will be thrown.

- HBASE-16584 Backport the new ipc implementation in HBASE-16432 to
  branch-1

    The netty dependency is upgraded to 4.1.1.Final. And also some
    configurations of the old AsyncRpcClient is gone. Such as
    "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport".

- HBASE-16653 Backport HBASE-11393 to all branches which support namespace

    During HBASE-11393, we have done two things:

      1.  unify tableCFs with peerConfig
      2.  Fix ns not support issue for replication.

    This issue is to backport it to branch-1

    How to rolling update if the replication peer have old table-cfs
    config? Due to we modify proto object of ReplicationPeerConfig (add
    tableCFs field), so when we do rolling update, we have to update
    original ReplicationPeerConfig data on ZK firstly.

      1. Make sure the master have the permission to modify replication
         peer znode.
      2. Disable the replication peer.
      3. Rolling update master first. The master will copy the table-cfs
         config from old table-cfs znode and add it to the new proto
         object of ReplicationPeerConfig.
      4. Rolling update regionservers.
      5. Enable the replication peer.

    If you can't change the replication peer znode permission, you can
    use the TableCFsUpdater tool to copy the table-cfs config.

      1. Disable the replication peer.
      2. bin/hbase
org.apache.hadoop.hbase.replication.master.TableCFsUpdater
         update
      3. Rolling update master and regionservers.
      4. Enable the replication peer.

- HBASE-16672 Add option for bulk load to always copy hfile(s) instead of
  renaming

    This issue adds a config, always.copy.files, to LoadIncrementalHFiles.
    When set to true, source hfiles would be copied. Meaning source hfiles
    would be kept after bulk load is done. Default value is false.

- HBASE-16698] Performance issue: handlers stuck waiting for CountDownLatch
  inside WALKey#getWriteEntry under high writing workload

    Assign sequenceid to an edit before we go on the ringbuffer; undoes
    contention on WALKey latch. Adds a new config
    "hbase.hregion.mvcc.preassign" which defaults to true: i.e. this speedup
    is enabled.

- HBASE-16755 Honor flush policy under global memstore pressure

    Prior to this change, when the memstore low water mark is exceeded on a
    regionserver, the regionserver will force flush all stores on the
    regions selected for flushing until we drop below the low water mark.
    With this change, the regionserver will continue to force flush regions
    when above the memstore low water mark, but will only flush the stores
    returned by the configured FlushPolicy.

- HBASE-16993 BucketCache throw java.io.IOException: Invalid HFile block
  magic when configuring hbase.bucketcache.bucket.sizes

    Any value for hbase.bucketcache.bucket.sizes configuration must be a
    multiple of 256.  If that is not the case, instantiation of L2 Bucket
    cache itself will fail throwing IllegalArgumentException.

- HBASE-17112 Prevent setting timestamp of delta operations the same as
  previous value

    Before this issue, two concurrent Increments/Appends done in same
    millisecond or RS's clock going back will result in two results having
    same TS, which is not friendly to versioning and will get wrong result
    in sink cluster if the replication is disordered. After this issue, the
    result of Increment/Append will always have an incremental TS. There is
    no longer any inconsistency in replication for these operations.

- HBASE-17178 Add region balance throttling

    Add region balance throttling. Master execute every region balance plan
    per balance interval, which is equals to divide max balancing time by
    the size of region balance plan. And Introduce a new config
    hbase.master.balancer.maxRitPercent to protect availability. If config
    this to 0.01, then the max percent of regions in transition is 1% when
    balancing. Then the cluster's availability is at least 99% when
    balancing.

- HBASE-17280 Add mechanism to control cleaner chore behavior

    The HBase cleaner chore process cleans up old WAL files and archived
    HFiles. Cleaner operation can affect query performance when running
    heavy workloads, so disable the cleaner during peak hours. The cleaner
    has the following HBase shell commands:

    - cleaner_chore_enabled: Queries whether cleaner chore is enabled/
      disabled.
    - cleaner_chore_run: Manually runs the cleaner to remove files.
    - cleaner_chore_switch: enables or disables the cleaner and returns
      the previous state of the cleaner. For example, cleaner-switch true
      enables the cleaner.

    Following APIs are added in Admin:

    - setCleanerChoreRunning(boolean on): Enable/Disable the cleaner chore
    - runCleanerChore(): Ask for cleaner chore to run
    - isCleanerChoreEnabled(): Query whether cleaner chore is enabled/
      disabled.

- HBASE-17296 Provide per peer throttling for replication

    Provide per peer throttling for replication. Add the bandwidth upper
    limit to ReplicationPeerConfig and a new shell cmd set_peer_bandwidth
    to update the bandwidth as needed.

- HBASE-17426 Inconsistent environment variable names for enabling JMX

    In bin/hbase-config.sh, if value for HBASE_JMX_BASE is empty, keep
    current behavior. If HBASE_JMX_OPTS is not empty, keep current
    behavior. Otherwise use the value of HBASE_JMX_BASE

- HBASE-17437 Support specifying a WAL directory outside of the root
  directory

    This patch adds support for specifying a WAL directory outside of the
    HBase root directory.

    Multiple configuration variables were added to accomplish this:

    hbase.wal.dir: used to configure where the root WAL directory is
    located. Could be on a different FileSystem than the root directory. WAL
    directory can not be set to a subdirectory of the root directory. The
    default value of this is the root directory if unset.

    hbase.rootdir.perms: Configures FileSystem permissions to set on the
    root directory. This is '700' by default.

    hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL
    directory FileSystem. This is '700' by default.

- HBASE-17472 Correct the semantic of permission grant

    Before this patch, later granted permissions will override previous
    granted permissions, and previous granted permissions will be lost. This
    issue re-defines the grant semantic: for master branch, later granted
    permissions will merge with previous granted permissions. For
    branch-1.4, grant keep override behavior for compatibility purpose, and
    a grant with mergeExistingPermission flag provided.

- HBASE-17508 Unify the implementation of small scan and regular scan for
  sync client

    Now the scan.setSmall method is deprecated. Consider using scan.setLimit
    and scan.setReadType in the future. And we will open scanner lazily when
    you call scanner.next. This is an incompatible change which delays the
    table existence check and permission check.

- HBASE-17578 Thrift per-method metrics should still update in the case of
  exceptions

    In prior versions, the HBase Thrift handlers failed to increment per-
    method metrics when an exception was encountered.  These metrics will
    now always be incremented, whether an exception is encountered or not.
    This change also adds exception-type metrics, similar to those exposed
    in regionservers, for individual exceptions which are received by the
    Thrift handlers.

- HBASE-17583 Add inclusive/exclusive support for startRow and endRow of
  scan for sync client

    Now you can include or exclude the startRow and stopRow for a scan. The
    new methods to specify startRow and stopRow are withStartRow and
    withStopRow. The old methods to specify startRow and Row(include
    constructors) are marked as deprecated as in the old time if startRow
    and stopRow are equal then we will consider it as a get scan and include
    the stopRow implicitly. This is strange after we can set inclusiveness
    explicitly so we add new methods and depredate the old methods. The
    deprecated methods will be removed in the future.

- HBASE-17584 Expose ScanMetrics with ResultScanner rather than Scan

    Now you can use ResultScanner.getScanMetrics to get the scan metrics at
    any time during the scan operation. The old Scan.getScanMetrics is
    deprecated and still work, but if you use ResultScanner.getScanMetrics
    to get the scan metrics and reset it, then the metrics published to the
    Scan instaince will be messed up.

- HBASE-17599 Use mayHaveMoreCellsInRow instead of isPartial

    The word 'isPartial' is ambiguous so we introduce a new method
    'mayHaveMoreCellsInRow' to replace it. And the old meaning of
    'isPartial' is not the same with 'mayHaveMoreCellsInRow' as for batched
    scan, if the number of returned cells equals to the batch, isPartial
    will be false. After this change the meaning of 'isPartial' will be same
    with 'mayHaveMoreCellsInRow'. This is an incompatible change but it is
    not likely to break a lot of things as for batched scan the old
    'isPartial' is just a redundant information, i.e, if the number of
    returned cells reaches the batch limit. You have already know the number
    of returned cells and the value of batch.

- HBASE-17737 Thrift2 proxy should support scan timeRange per column
  family

    Thrift2 proxy now supports scan timeRange per column family.

- HBASE-17757 Unify blocksize after encoding to decrease memory fragment

    Blocksize is set in columnfamily's atrributes. It is used to control
    block sizes when generating blocks. But, it doesn't take encoding into
    count. If you set encoding to blocks, after encoding, the block size
    varies. Since blocks will be cached in memory after encoding (default),
    it will cause memory fragment if using blockcache, or decrease the pool
    efficiency if using bucketCache. This issue introduced a new config
    named 'hbase.writer.unified.encoded.blocksize.ratio'. The default value
    of this config is 1, meaning doing nothing. If this value is set to a
    smaller value like 0.5, and the blocksize is set to 64KB (default value
    of blocksize). It will unify the blocksize after encoding to 64KB * 0.5
    = 32KB. Unified blocksize will relieve the memory problems mentioned
    above.

- HBASE-17817 Make Regionservers log which tables it removed coprocessors
  from when aborting

    Adds table name to exception logging when a coprocessor is removed from
    a table by the region server.

- HBASE-17861 Regionserver down when checking the permission of staging dir
  if hbase.rootdir is on S3

    Some object store does not support unix style permission. This fixes the
    permission check issue when specify staging dir in different file
    system. Currently it covers s3, wasb, swift.

- HBASE-17877 Improve HBase's byte[] comparator

    Updated the lexicographic byte array comparator to use a slightly more
    optimized version similar to the one available in the guava library that
    compares only the first index where left[index] != right[index]. The
    comparator also returns the diff directly instead of mapping it to -1,
    0, +1 range as was being done in the earlier version. We have seen
    significant performance gains, calculated in terms of throughput (ops/
    ms) with these changes ranging from approx 20% for smaller byte arrays
    up to 200 bytes and almost 100% for large byte array sizes that are in
    few KBs. We benchmarked with up to 16KB arrays and the general trend
    indicates that the performance improvement increases as the size of the
    byte array increases.

- HBASE-17956 Raw scan should ignore TTL

    Now raw scan can also read expired cells.

- HBASE-18023 Log multi-requests for more than threshold number of rows

    Introduces a warning message in the RegionServer log when an RPC is
    received from a client that has more than 5000 "actions" (where an
    "action" is a collection of mutations for a specific row) in a single
    RPC. Misbehaving clients who send large RPCs to RegionServers can be
    malicious, causing temporary pauses via garbage collection or denial of
    service via crashes. The threshold of 5000 actions per RPC is defined by
    the property "hbase.rpc.rows.warning.threshold" in hbase-site.xml.

- HBASE-18090 Improve TableSnapshotInputFormat to allow multiple mappers
  per region

    In this task, we make it possible to run multiple mappers per region in
    the table snapshot.

- HBASE-18122 Scanner id should include ServerName of region server

    The first 32 bits are MurmurHash32 of ServerName string "host,port,ts".
    The ServerName contains both host, port, and start timestamp so it can
    prevent collision. The lowest 32bit is generated by atomic int.

- HBASE-18149 The setting rules for table-scope attributes and family-scope
  attributes should keep consistent

    If the table-scope attributes value is false, you need not to enclose
    'false' in single quotation. Both COMPACTION_ENABLED => false and
    COMPACTION_ENABLED => 'false' will take effect.

- HBASE-18226 Disable reverse DNS lookup at HMaster and use the hostname
  provided by RegionServer

    The following config is added:

​    ​
    hbase.regionserver.hostname.disable.master.reversedns

    This config is for experts: don't set its value unless you really know
    what you are doing. When set to true, regionserver will use the current
    node hostname for the servername and HMaster will skip reverse DNS
    lookup and use the hostname sent by regionserver instead. Note that this
    config and hbase.regionserver.hostname are mutually exclusive. See
    https://issues.apache.org/jira/browse/HBASE-18226 for more details.

    Caution: please make sure rolling upgrade succeeds before turning on
    this feature.

- HBASE-18247 Hbck to fix the case that replica region shows as key in the
  meta table

    The hbck tool can now correct the meta table should it get an entry for
    a read replica region.

- HBASE-18374 RegionServer Metrics improvements

    This change adds the latency metrics checkAndPut, checkAndDelete,
    putBatch and deleteBatch . Also the previous regionserver "mutate"
    latency metrics are renamed to "put" metrics. Batch metrics capture the
    latency of the entire batch containing put/delete whereas put/delete
    metrics capture latency per operation. Note this change will break
    existing monitoring based on regionserver "mutate" latency metric.

- HBASE-18520 Add jmx value to determine true Master Start time

    Adds a JMX value to track when the Master has finished initializing.
    The jmx config is 'masterFinishedInitializationTime' and details the
    time in millis that the Master is fully usable and ready to serve
    requests.

- HBASE-18533 Expose BucketCache values to be configured

    This patch exposes configuration for Bucketcache. These configs are very
    similar to those for the LRU cache, but are described below:

    "hbase.bucketcache.single.factor"; /** Single access bucket size */
    "hbase.bucketcache.multi.factor"; /** Multiple access bucket size */
    "hbase.bucketcache.memory.factor"; /** In-memory bucket size */
    "hbase.bucketcache.extrafreefactor"; /** Free this floating point
      factor of extra blocks when evicting. For example free the number of
      blocks requested * (1 + extraFreeFactor) */
    "hbase.bucketcache.acceptfactor"; /** Acceptable size of cache (no
       evictions if size < acceptable) */
    "hbase.bucketcache.minfactor"; /** Minimum threshold of cache (when
       evicting, evict until size < min) */

- HBASE-18675 Making {max,min}SessionTimeout configurable for
  MiniZooKeeperCluster

    Standalone clusters and minicluster instances can now configure the
    session timeout for our embedded ZooKeeper quorum using
    "hbase.zookeeper.property.minSessionTimeout" and
    "hbase.zookeeper.property.maxSessionTimeout".

- HBASE-18786 FileNotFoundException should not be silently handled for
  primary region replicas

    FileNotFoundException opening a StoreFile in a primary replica now
    causes a RegionServer to crash out where before it would be ignored (or
    optionally handled via close/reopen).

- HBASE-18993 Backport patches in HBASE-18410 to branch-1.x branches

    This change fixes bugs in FilterList, and also does a code refactor
     which ensures interface compatibility.

    The primary bug fixes are :

    1. For sub-filter in FilterList with MUST_PASS_ONE, if previous
       filterKeyValue() of sub-filter returns NEXT_COL, we cannot make sure
       that the next cell will be the first cell in next column, because
       FilterList choose the minimal forward step among sub-filters, and it
       may return a SKIP. so here we add an extra check to ensure that the
       next cell will match previous return code for sub-filters.

    2. Previous logic about transforming cell of FilterList is incorrect, we
       should set the previous transform result (rather than the given cell
       in question) as the initial value of transform cell before call
       filterKeyValue() of FilterList.

    3. Handle the ReturnCodes which the previous code did not handle.

    About code refactor, we divided the FilterList into two separated sub-
    classes: FilterListWithOR and FilterListWithAND, The FilterListWithOR
    has been optimized to choose the next minimal step to seek cell rather
    than SKIP cell one by one, and the FilterListWithAND has been optimized
    to choose the next maximal key to seek among sub-filters in filter list.
    All in all, The code in FilterList is clean and easier to follow now.

    Note that ReturnCode NEXT_ROW has been redefined as skipping to next
    row in current family, not to next row in all family. it’s more
    reasonable, because ReturnCode is a concept in store level, not in
    region level.

- HBASE-19035 Miss metrics when coprocessor use region scanner to read
  data

    Move read requests count to region level. Because RegionScanner is
    exposed to CP. Update write requests count in processRowsWithLocks.
    Remove requestRowActionCount in RSRpcServices. This metric can be
    computed by region's readRequestsCount and writeRequestsCount.

- HBASE-19051 Add new split algorithm for num string

    Add new split algorithm DecimalStringSplit,row are decimal-encoded long
    values in the range "00000000" => "99999999".

- HBASE-19131 Add the ClusterStatus hook and cleanup other hooks which can
  be replaced by ClusterStatus hook**

    1) Add preGetClusterStatus() and postGetClusterStatus() hooks
    2) add preGetClusterStatus() to access control check - an admin action

- HBASE-19144 Retry assignments in FAILED_OPEN state when servers (re)join
  the cluster

     When regionserver placement groups (RSGroups) is active, as servers
     join the cluster the Master will attempt to reassign regions in
     FAILED_OPEN state.

- HBASE-19419 Remove hbase-native-client from branch-1

    Removed the hbase-native-client module from branch-1 (it is still in
    Master). It is not complete. Look for a finished C++ client in the near
    future which may be backported to branch-1 at that point.

---
Cheers,
The HBase Dev Team

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message