phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <>
Subject [jira] [Commented] (PHOENIX-3553) Zookeeper connection should be closed immediately after DefaultStatisticsCollector's collecting stats done
Date Wed, 04 Jan 2017 01:40:58 GMT


James Taylor commented on PHOENIX-3553:

The patch looks fine, but I'm confused as to why DefaultStatisticsCollector.initGuidePostDepth()
would be called during a minor compaction given this check in UngroupedAggregateRegionObserver:
    public InternalScanner preCompact(final ObserverContext<RegionCoprocessorEnvironment>
c, final Store store,
            final InternalScanner scanner, final ScanType scanType) throws IOException {
        // Compaction and split upcalls run with the effective user context of the requesting
        // This will lead to failure of cross cluster RPC if the effective user is not
        // the login user. Switch to the login user context to ensure we have the expected
        // security context.
        return User.runAsLoginUser(new PrivilegedExceptionAction<InternalScanner>()
            public InternalScanner run() throws Exception {
                TableName table = c.getEnvironment().getRegion().getRegionInfo().getTable();
                InternalScanner internalScanner = scanner;
                if (scanType.equals(ScanType.COMPACT_DROP_DELETES)) {
                    try {
                        long clientTimeStamp = TimeKeeper.SYSTEM.getCurrentTime();
                        StatisticsCollector stats = StatisticsCollectorFactory.createStatisticsCollector(
                            c.getEnvironment(), table.getNameAsString(), clientTimeStamp,
                        internalScanner = stats.createCompactionScanner(c.getEnvironment(),
store, scanner);

> Zookeeper connection should be closed immediately after DefaultStatisticsCollector's
collecting stats done
> ----------------------------------------------------------------------------------------------------------
>                 Key: PHOENIX-3553
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.9.0
>            Reporter: Yeonseop Kim
>              Labels: stats, zookeeper
>             Fix For: 4.10.0
>         Attachments: PHOENIX-3553.patch
> In every minor compaction job of HBase,
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidePostDepth() is called,
> and SYSTEM.CATALOG table is open to get guidepost width via
> htable = env.getTable(
>                  SchemaUtil.getPhysicalTableName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
> This function call creates one zookeeper connection to get cluster id.
> DefaultStatisticsCollector doesn't close this zookeeper connection immediately after
get guidepost width, and the zookeeper connection remains alive until HRegion is closed.
> This is not a problem with small number of Regions, but when number of Region is large
and upsert operation is frequent, the number of zookeeper connection gradually increases 
to hundreds, and the zookeeper server nodes experience  short of available TCP/IP ports.
> This zookeeper connection should be closed immediately after get guidepost width.

This message was sent by Atlassian JIRA

View raw message