kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Preeti V (Jira)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-4427) Wrong FileSystem error when trying to enable system cubes and Dashboard in Kylin 2.6.4
Date Tue, 07 Apr 2020 00:28:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076779#comment-17076779
] 

Preeti V commented on KYLIN-4427:
---------------------------------

[~hit_lacus], thank you for the pointer about it being case-sensitive. I updated it and now
I don't see the wrong FS error anymore. 
I do have a new error now "hive.HiveReservoirReporter:119 : java.io.FileNotFoundException:
/hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file or directory." (detailed log at the end)
and the metrics are still not showing in the dashboard.

I checked on the Kylin machines and the file is under /usr/hdp/2.6.5.1175-1/tez/lib. There
is no "apps' folder.  

 

!image-2020-04-06-15-46-37-356.png!

 

I looked at the Tez URI in Ambari and it is pointed to the folder name with apps in it

 

!image-2020-04-06-17-14-52-443.png!

 

If  I update it to /hdp/${hdp.version}/tez/tez.tar.gz or  /hdp/apps/${hdp.version}/tez/lib/tez.tar.gz,
the BlockingReservoir still gives the file not found error. In addition to that all Kylin
Build jobs started failing as soon as I updated the tez.lib.uris, with a tez error about file
not found. So I assume the original uri was needed for Kylin builds. 

Is there any property I can set in the kylinMetrics.xml for the tez endpoint?

 

Detailed error log

2020-04-06 21:42:36,961 INFO [metrics-blocking-reservoir-scheduler-0] client.AHSProxy:42 :
Connecting to Application History server at bn01eap00002f79.ap.gbl/10.175.241.176:10200
2020-04-06 21:42:36,962 INFO [metrics-blocking-reservoir-scheduler-0] client.TezClient:351
: Session mode. Starting session.
2020-04-06 21:42:36,962 INFO [metrics-blocking-reservoir-scheduler-0] client.TezClientUtils:173
: Using tez.lib.uris value from configuration: /hdp/apps/2.6.5.1175-1/tez/tez.tar.gz
2020-04-06 21:42:36,968 ERROR [metrics-blocking-reservoir-scheduler-0] hive.HiveReservoirReporter:119
: java.io.FileNotFoundException: /hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file or directory.
java.lang.RuntimeException: java.io.FileNotFoundException: /hdp/apps/2.6.5.1175-1/tez/tez.tar.gz:
No such file or directory.
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
 at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:154)
 at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
 at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
 at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
 at org.apache.kylin.metrics.lib.impl.BlockingReservoir.onRecordUpdate(BlockingReservoir.java:93)
 at org.apache.kylin.metrics.lib.impl.BlockingReservoir.access$300(BlockingReservoir.java:33)
 at org.apache.kylin.metrics.lib.impl.BlockingReservoir$ReporterRunnable.run(BlockingReservoir.java:152)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file
or directory.
 at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
 at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
 at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:771)
 at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:132)
 at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)
 at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:831)
 at org.apache.tez.client.TezClient.start(TezClient.java:355)
 at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:197)
 at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
 at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:579)
 ... 8 more

> Wrong FileSystem error when trying to enable system cubes and Dashboard in Kylin 2.6.4
> --------------------------------------------------------------------------------------
>
>                 Key: KYLIN-4427
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4427
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metrics
>    Affects Versions: v2.6.4
>            Reporter: Preeti V
>            Assignee: Xiaoxiang Yu
>            Priority: Major
>         Attachments: KylinMetrics.JPG, image-2020-04-03-10-45-15-290.png, image-2020-04-03-10-45-20-859.png,
image-2020-04-03-10-45-45-894.png, image-2020-04-06-15-46-37-356.png, image-2020-04-06-17-14-52-443.png
>
>
>  I am trying to enable system cubes for the Dashboard using Kylin version 2.6.4 The
tables are created correctly and the cube builds successfully but there is no query or job
data on the dashboard, it shows 0. 
>  
> We use Azure storage for Hive(wasb:// file system). I can see that there is no data being
updated on the Hive_Metrics tables in Azure. In Kylin logs I see the below error
>  
>  
> 2020-03-12 20:02:41,790 ERROR [metrics-blocking-reservoir-scheduler-0] hive.HiveReservoirReporter:119
: Wrong FS: wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
expected: hdfs://*****-prod-bn01
> java.lang.IllegalArgumentException: Wrong FS: wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
expected: hdfs://*****-prod-bn01
>         at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:666)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1454)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1448)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:137)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
>         at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
>         at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
>  
>   
> I checked the hive configs and it has the warehouse metastore dir correctly pointing
to azure. I found another thread with similar problem where they are trying to use S3 instead
of hdfs. [http://apache-kylin.74782.x6.nabble.com/jira-Created-KYLIN-4385-KYLIN-system-cube-failing-to-update-table-when-run-on-EMR-with-S3-as-storageS-td14234.html] 
>  
> I also followed the recommendations here [https://www.mail-archive.com/user@kylin.apache.org/msg04347.html] 
and enabled all the necessary config values.
>  Is this a bug in Kylin or a configuration issue on my cluster? Any help or guidance
is appreciated.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message