kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoxiang Yu (Jira)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-4427) Wrong FileSystem error when trying to enable system cubes and Dashboard in Kylin 2.6.4
Date Wed, 08 Apr 2020 06:16:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077872#comment-17077872
] 

Xiaoxiang Yu commented on KYLIN-4427:
-------------------------------------

I am testing it at HDI 3.6. Following is cluster's template.

{code:json}
{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "clusters_kylin_system_cube_test_cluster_name": {
            "defaultValue": "123",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.HDInsight/clusters",
            "apiVersion": "2018-06-01-preview",
            "name": "[parameters('123')]",
            "location": "China East",
            "tags": {
                "Owner": "xiaoxiang",
                "Cost Center": "",
                "Period": "1 day"
            },
            "properties": {
                "clusterVersion": "3.6.1000.67",
                "osType": "Linux",
                "clusterDefinition": {
                    "blueprint": "https://blueprints.azurehdinsight.net/hbase-3.6.1000.67.2003240620.json",
                    "kind": "HBASE",
                    "componentVersion": {
                        "HBase": "1.1"
                    }
                },
                "computeProfile": {
                    "roles": [
                        {
                            "name": "headnode",
                            "targetInstanceCount": 2,
                            "hardwareProfile": {
                                "vmSize": "standard_d3_v2"
                            },
                            "osProfile": {
                                "linuxOperatingSystemProfile": {
                                    "username": "sshuser"
                                }
                            }
                        },
                        {
                            "name": "workernode",
                            "targetInstanceCount": 2,
                            "hardwareProfile": {
                                "vmSize": "standard_d4_v2"
                            },
                            "osProfile": {
                                "linuxOperatingSystemProfile": {
                                    "username": "sshuser"
                                }
                            }
                        },
                        {
                            "name": "zookeepernode",
                            "targetInstanceCount": 3,
                            "hardwareProfile": {
                                "vmSize": "standard_d1_v2"
                            },
                            "osProfile": {
                                "linuxOperatingSystemProfile": {
                                    "username": "sshuser"
                                }
                            }
                        }
                    ]
                },
                "tier": "standard",
                "storageProfile": {
                    "storageaccounts": [
                        {
                            "name": "123.blob.core.chinacloudapi.cn",
                            "container": "123",
                            "isDefault": true
                        }
                    ]
                }
            }
        }
    ]
}
{code}

I am using `apache-kylin-2.6.5-bin-hbase1x.tar.gz`, and modify kylin.properties:



{code:java}
kylin.metrics.monitor-enabled=true
kylin.metrics.reporter-job-enabled=true
kylin.metrics.reporter-query-enabled=true
kylin.web.dashboard-enabled=true
{code}


After kylin instance started, I see the logs:


{code:java}
2020-04-08 05:54:48,459 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:54:51,967 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:54:52,954 DEBUG [http-bio-7070-exec-7] badquery.BadQueryHistoryManager:65 :
Loaded 0 Bad Query(s)
2020-04-08 05:54:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:55:24,787 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:55:48,459 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:55:51,967 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:55:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:56:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:56:48,459 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:56:51,968 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:56:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:57:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:57:48,460 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:57:51,968 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:57:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:58:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:58:48,461 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:58:51,969 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:58:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:59:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 05:59:48,461 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 05:59:51,969 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 05:59:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 06:00:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others


2020-04-08 06:00:48,461 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:145
: The number of records in the blocking queue is less than 10 and the duration from last reporting
is less than 600000ms. Will delay to report!
2020-04-08 06:00:51,969 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 06:00:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 06:01:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 06:01:48,462 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:444
: Trying to connect to metastore with URI thrift://hn0-kylin.nert0r5niyaepiqzqwkbwnagve.bx.internal.chinacloudapp.cn:9083
2020-04-08 06:01:48,463 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:540
: Connected to metastore.
2020-04-08 06:01:48,929 INFO  [metrics-blocking-reservoir-scheduler-0] DependencyResolver:1106
: ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/2.6.5.3015-8/0/ivysettings.xml
will be used
2020-04-08 06:01:49,170 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:444
: Trying to connect to metastore with URI thrift://hn1-kylin.nert0r5niyaepiqzqwkbwnagve.bx.internal.chinacloudapp.cn:9083
2020-04-08 06:01:49,171 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:540
: Connected to metastore.
2020-04-08 06:01:50,168 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:701
: Created local directory: /home/sshuser/apache-kylin-2.6.5-bin-hbase1x/tomcat/temp/sshuser
2020-04-08 06:01:50,169 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:701
: Created local directory: /home/sshuser/apache-kylin-2.6.5-bin-hbase1x/tomcat/temp/49f0ff10-87f0-49a9-b873-bd68f81836b0_resources
2020-04-08 06:01:50,255 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:701
: Created HDFS directory: hdfs://mycluster/tmp/hive/sshuser/49f0ff10-87f0-49a9-b873-bd68f81836b0
2020-04-08 06:01:50,648 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:701
: Created local directory: /home/sshuser/apache-kylin-2.6.5-bin-hbase1x/tomcat/temp/sshuser/49f0ff10-87f0-49a9-b873-bd68f81836b0
2020-04-08 06:01:50,690 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:701
: Created HDFS directory: hdfs://mycluster/tmp/hive/sshuser/49f0ff10-87f0-49a9-b873-bd68f81836b0/_tmp_space.db
2020-04-08 06:01:50,704 INFO  [metrics-blocking-reservoir-scheduler-0] tez.TezSessionState:129
: Opening the session with id 49f0ff10-87f0-49a9-b873-bd68f81836b0 for thread metrics-blocking-reservoir-scheduler-0
log trace id -  query id -
2020-04-08 06:01:50,705 INFO  [metrics-blocking-reservoir-scheduler-0] tez.TezSessionState:145
: User of session id 49f0ff10-87f0-49a9-b873-bd68f81836b0 is sshuser
2020-04-08 06:01:50,774 INFO  [metrics-blocking-reservoir-scheduler-0] tez.DagUtils:898 :
Jar dir is null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/sshuser/.hiveJars
2020-04-08 06:01:50,983 INFO  [metrics-blocking-reservoir-scheduler-0] tez.DagUtils:724 :
Resource modification time: 1586318675000
2020-04-08 06:01:51,076 INFO  [metrics-blocking-reservoir-scheduler-0] sqlstd.SQLStdHiveAccessController:95
: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=49f0ff10-87f0-49a9-b873-bd68f81836b0,
clientType=HIVECLI]
2020-04-08 06:01:51,078 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:346
: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl
to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2020-04-08 06:01:51,081 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:444
: Trying to connect to metastore with URI thrift://hn0-kylin.nert0r5niyaepiqzqwkbwnagve.bx.internal.chinacloudapp.cn:9083
2020-04-08 06:01:51,081 INFO  [metrics-blocking-reservoir-scheduler-0] hive.metastore:540
: Connected to metastore.
2020-04-08 06:01:51,113 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClient:177
: Tez Client Version: [ component=tez-api, version=0.7.0.2.6.5.3015-8, revision=99b800b802c2c9737efd7040342c266fd1a53ec5,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2019-12-05T13:33:09Z
]
2020-04-08 06:01:51,114 INFO  [metrics-blocking-reservoir-scheduler-0] tez.TezSessionState:193
: Opening new Tez Session (id: 49f0ff10-87f0-49a9-b873-bd68f81836b0, scratch dir: hdfs://mycluster/tmp/hive/sshuser/_tez_session_dir/49f0ff10-87f0-49a9-b873-bd68f81836b0)
2020-04-08 06:01:51,242 INFO  [metrics-blocking-reservoir-scheduler-0] client.AHSProxy:42
: Connecting to Application History server at headnodehost/10.0.0.15:10200
2020-04-08 06:01:51,248 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClient:351
: Session mode. Starting session.
2020-04-08 06:01:51,251 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClientUtils:173
: Using tez.lib.uris value from configuration: /hdp/apps/2.6.5.3015-8/tez/tez.tar.gz
2020-04-08 06:01:51,341 INFO  [metrics-blocking-reservoir-scheduler-0] client.RequestHedgingRMFailoverProxyProvider:146
: Looking for the active RM in [rm1, rm2]...
2020-04-08 06:01:51,347 INFO  [metrics-blocking-reservoir-scheduler-0] client.RequestHedgingRMFailoverProxyProvider:170
: Found active RM [rm2]
2020-04-08 06:01:51,398 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClient:122
: Tez system stage directory hdfs://mycluster/tmp/hive/sshuser/_tez_session_dir/49f0ff10-87f0-49a9-b873-bd68f81836b0/.tez/application_1586317575543_0033
doesn't exist and is created
2020-04-08 06:01:51,970 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2020-04-08 06:01:52,003 INFO  [metrics-blocking-reservoir-scheduler-0] impl.YarnClientImpl:274
: Submitted application application_1586317575543_0033
2020-04-08 06:01:52,006 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClient:385
: The url to track the Tez Session: http://hn1-kylin.nert0r5niyaepiqzqwkbwnagve.bx.internal.chinacloudapp.cn:8088/proxy/application_1586317575543_0033/
2020-04-08 06:01:54,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
2020-04-08 06:01:55,641 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:55,642 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:55,731 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:436 : We
are setting the hadoop caller context from  to sshuser_20200408060155_3790fa71-111b-41e2-af31-15bd4a0d98f9
2020-04-08 06:01:55,737 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:55,876 INFO  [metrics-blocking-reservoir-scheduler-0] parse.ParseDriver:190
: Parsing command: ALTER TABLE KYLIN.HIVE_METRICS_JOB_EXCEPTION_QA ADD IF NOT EXISTS PARTITION
(kday_date='2020-04-08')
2020-04-08 06:01:56,718 INFO  [metrics-blocking-reservoir-scheduler-0] parse.ParseDriver:218
: Parse Completed
2020-04-08 06:01:56,719 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=parse start=1586325715737 end=1586325716719 duration=982 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:56,719 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,340 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:486 : Semantic
Analysis Completed
2020-04-08 06:01:57,340 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=semanticAnalyze start=1586325716719 end=1586325717340 duration=621 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,351 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:253 : Returning
Hive schema: Schema(fieldSchemas:null, properties:null)
2020-04-08 06:01:57,351 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=compile start=1586325715642 end=1586325717351 duration=1709 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,351 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:568 : We
are resetting the hadoop caller context to
2020-04-08 06:01:57,351 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:173 : Concurrency
mode is disabled, not creating a lock manager
2020-04-08 06:01:57,351 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,352 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:1420 : Setting
caller context to query id sshuser_20200408060155_3790fa71-111b-41e2-af31-15bd4a0d98f9
2020-04-08 06:01:57,352 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:1423 : Starting
command(queryId=sshuser_20200408060155_3790fa71-111b-41e2-af31-15bd4a0d98f9): ALTER TABLE
KYLIN.HIVE_METRICS_JOB_EXCEPTION_QA ADD IF NOT EXISTS PARTITION (kday_date='2020-04-08')
2020-04-08 06:01:57,671 INFO  [metrics-blocking-reservoir-scheduler-0] impl.TimelineClientImpl:302
: Timeline service address: http://headnodehost:8188/ws/v1/timeline/
2020-04-08 06:01:57,836 INFO  [metrics-blocking-reservoir-scheduler-0] hooks.ATSHook:114 :
Created ATS Hook
2020-04-08 06:01:57,837 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,841 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1586325717837
end=1586325717841 duration=4 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,846 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:57,859 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:1763 : Starting
task [Stage-0:DDL] in serial mode
2020-04-08 06:01:58,085 INFO  [ATS Logger 0] hooks.ATSHook:123 : ATS domain created:hive_49f0ff10-87f0-49a9-b873-bd68f81836b0(sshuser,sshuser)
2020-04-08 06:01:58,420 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=runTasks start=1586325717846 end=1586325718420 duration=574 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,422 INFO  [metrics-blocking-reservoir-scheduler-0] hooks.ATSHook:114 :
Created ATS Hook
2020-04-08 06:01:58,422 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,425 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1586325718422
end=1586325718425 duration=3 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,425 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:1655 : Resetting
the caller context to
2020-04-08 06:01:58,425 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=Driver.execute start=1586325717351 end=1586325718425 duration=1074 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,426 INFO  [metrics-blocking-reservoir-scheduler-0] ql.Driver:1106 : OK
2020-04-08 06:01:58,427 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:149
: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,427 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=releaseLocks start=1586325718427 end=1586325718427 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,427 INFO  [metrics-blocking-reservoir-scheduler-0] log.PerfLogger:177
: </PERFLOG method=Driver.run start=1586325715641 end=1586325718427 duration=2786 from=org.apache.hadoop.hive.ql.Driver>
2020-04-08 06:01:58,443 INFO  [metrics-blocking-reservoir-scheduler-0] tez.TezSessionPoolManager:198
: Closing tez session default? false
2020-04-08 06:01:58,443 INFO  [metrics-blocking-reservoir-scheduler-0] tez.TezSessionState:294
: Closing Tez Session
2020-04-08 06:01:58,444 INFO  [metrics-blocking-reservoir-scheduler-0] client.TezClient:518
: Shutting down Tez Session, sessionName=HIVE-49f0ff10-87f0-49a9-b873-bd68f81836b0, applicationId=application_1586317575543_0033
2020-04-08 06:01:58,612 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:790
: Deleted directory: hdfs://mycluster/tmp/hive/sshuser/49f0ff10-87f0-49a9-b873-bd68f81836b0
on fs with scheme hdfs
2020-04-08 06:01:58,613 INFO  [metrics-blocking-reservoir-scheduler-0] session.SessionState:790
: Deleted directory: /home/sshuser/apache-kylin-2.6.5-bin-hbase1x/tomcat/temp/sshuser/49f0ff10-87f0-49a9-b873-bd68f81836b0
on fs with scheme file
2020-04-08 06:01:58,752 ERROR [metrics-blocking-reservoir-scheduler-0] hive.HiveReservoirReporter:119
: Append Support not enabled
java.lang.UnsupportedOperationException: Append Support not enabled
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.append(NativeAzureFileSystem.java:1587)
	at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1188)
	at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:179)
	at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:123)
	at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
	at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
	at org.apache.kylin.metrics.lib.impl.BlockingReservoir.onRecordUpdate(BlockingReservoir.java:93)
	at org.apache.kylin.metrics.lib.impl.BlockingReservoir.access$300(BlockingReservoir.java:33)
	at org.apache.kylin.metrics.lib.impl.BlockingReservoir$ReporterRunnable.run(BlockingReservoir.java:152)
	at java.lang.Thread.run(Thread.java:748)
2020-04-08 06:01:58,753 WARN  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:95
: It fails to notify listener org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener@4a82f1b8
of updated records [{KTIME_HOUR=5, PROJECT=LEARN_KYLIN, JOB_ID=ef1fb713-ee7e-82e0-a433-5ebbb92c45c8,
EVENT_TYPE=METRICS_JOB_EXCEPTION_QA, EXCEPTION=org.apache.kylin.job.exception.ExecuteException,
HOST=hn0-kylin:10.0.0.16, KDAY_TIME=05:51:31, KTIMESTAMP=1586325091451, CUBING_TYPE=NULL,
CUBE_NAME=kylin_sales_cube, KDAY_DATE=2020-04-08, KTIME_MINUTE=51, KTIME_SECOND=31, JOB_TYPE=BUILD,
KWEEK_BEGIN_DATE=2020-04-05, KYEAR_BEGIN_DATE=2020-01-01, KMONTH_BEGIN_DATE=2020-04-01, KUSER=ADMIN},
{KTIME_HOUR=5, PROJECT=LEARN_KYLIN, JOB_ID=ef1fb713-ee7e-82e0-a433-5ebbb92c45c8, EVENT_TYPE=METRICS_JOB_EXCEPTION_QA,
EXCEPTION=org.apache.kylin.job.exception.ExecuteException, HOST=hn0-kylin:10.0.0.16, KDAY_TIME=05:54:45,
KTIMESTAMP=1586325285724, CUBING_TYPE=NULL, CUBE_NAME=kylin_sales_cube, KDAY_DATE=2020-04-08,
KTIME_MINUTE=54, KTIME_SECOND=45, JOB_TYPE=BUILD, KWEEK_BEGIN_DATE=2020-04-05, KYEAR_BEGIN_DATE=2020-01-01,
KMONTH_BEGIN_DATE=2020-04-01, KUSER=ADMIN}]
2020-04-08 06:01:58,753 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:109
: The HA listener org.apache.kylin.metrics.lib.impl.StubReservoirReporter$StubReservoirListener@6e56abc5
for updated records [{KTIME_HOUR=5, PROJECT=LEARN_KYLIN, JOB_ID=ef1fb713-ee7e-82e0-a433-5ebbb92c45c8,
EVENT_TYPE=METRICS_JOB_EXCEPTION_QA, EXCEPTION=org.apache.kylin.job.exception.ExecuteException,
HOST=hn0-kylin:10.0.0.16, KDAY_TIME=05:51:31, KTIMESTAMP=1586325091451, CUBING_TYPE=NULL,
CUBE_NAME=kylin_sales_cube, KDAY_DATE=2020-04-08, KTIME_MINUTE=51, KTIME_SECOND=31, JOB_TYPE=BUILD,
KWEEK_BEGIN_DATE=2020-04-05, KYEAR_BEGIN_DATE=2020-01-01, KMONTH_BEGIN_DATE=2020-04-01, KUSER=ADMIN},
{KTIME_HOUR=5, PROJECT=LEARN_KYLIN, JOB_ID=ef1fb713-ee7e-82e0-a433-5ebbb92c45c8, EVENT_TYPE=METRICS_JOB_EXCEPTION_QA,
EXCEPTION=org.apache.kylin.job.exception.ExecuteException, HOST=hn0-kylin:10.0.0.16, KDAY_TIME=05:54:45,
KTIMESTAMP=1586325285724, CUBING_TYPE=NULL, CUBE_NAME=kylin_sales_cube, KDAY_DATE=2020-04-08,
KTIME_MINUTE=54, KTIME_SECOND=45, JOB_TYPE=BUILD, KWEEK_BEGIN_DATE=2020-04-05, KYEAR_BEGIN_DATE=2020-01-01,
KMONTH_BEGIN_DATE=2020-04-01, KUSER=ADMIN}] will be started
2020-04-08 06:01:58,753 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:140
: There's no record in the blocking queue.
2020-04-08 06:02:24,785 INFO  [FetcherRunner 935914837-51] threadpool.DefaultFetcherRunner:85
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
{code}








> Wrong FileSystem error when trying to enable system cubes and Dashboard in Kylin 2.6.4
> --------------------------------------------------------------------------------------
>
>                 Key: KYLIN-4427
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4427
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metrics
>    Affects Versions: v2.6.4
>            Reporter: Preeti V
>            Assignee: Xiaoxiang Yu
>            Priority: Major
>         Attachments: KylinMetrics.JPG, image-2020-04-03-10-45-15-290.png, image-2020-04-03-10-45-20-859.png,
image-2020-04-03-10-45-45-894.png, image-2020-04-06-15-46-37-356.png, image-2020-04-06-17-14-52-443.png
>
>
>  I am trying to enable system cubes for the Dashboard using Kylin version 2.6.4 The
tables are created correctly and the cube builds successfully but there is no query or job
data on the dashboard, it shows 0. 
>  
> We use Azure storage for Hive(wasb:// file system). I can see that there is no data being
updated on the Hive_Metrics tables in Azure. In Kylin logs I see the below error
>  
>  
> 2020-03-12 20:02:41,790 ERROR [metrics-blocking-reservoir-scheduler-0] hive.HiveReservoirReporter:119
: Wrong FS: wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
expected: hdfs://*****-prod-bn01
> java.lang.IllegalArgumentException: Wrong FS: wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
expected: hdfs://*****-prod-bn01
>         at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:666)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1454)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1448)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:137)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
>         at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
>  
>   
> I checked the hive configs and it has the warehouse metastore dir correctly pointing
to azure. I found another thread with similar problem where they are trying to use S3 instead
of hdfs. [http://apache-kylin.74782.x6.nabble.com/jira-Created-KYLIN-4385-KYLIN-system-cube-failing-to-update-table-when-run-on-EMR-with-S3-as-storageS-td14234.html] 
>  
> I also followed the recommendations here [https://www.mail-archive.com/user@kylin.apache.org/msg04347.html] 
and enabled all the necessary config values.
>  Is this a bug in Kylin or a configuration issue on my cluster? Any help or guidance
is appreciated.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message