hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HADOOP-17471) ABFS to collect IOStatistics
Date Thu, 01 Apr 2021 04:34:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-17471?focusedWorklogId=575309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-575309
]

ASF GitHub Bot logged work on HADOOP-17471:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Apr/21 04:33
            Start Date: 01/Apr/21 04:33
    Worklog Time Spent: 10m 
      Work Description: mehakmeet commented on pull request #2731:
URL: https://github.com/apache/hadoop/pull/2731#issuecomment-811630475


   Extended ```AbfsCounters.java``` interface with ```DurationTrackerFactory.java```, and
converted the try-with-resources DurationTracking into 
   ```
   try {
         IOStatisticsBinding.trackDurationOfInvocation(abfsCounters,
             AbfsStatistic.getStatNameFromHttpCall(method),
             () -> completeExecute());
       } catch (IOException e) {
         throw new RuntimeException("Error while tracking Duration of an "
             + "AbfsRestOperation call", e);
       }
   ```
   in the execute() method of ```AbfsRestOperation.java``` class, but I am facing this exception
now:
   
   ```
   java.lang.RuntimeException: Error while tracking Duration of an AbfsRestOperation call
   
   	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:192)
   	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:767)
   	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:749)
   	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:295)
   	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:786)
   	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:506)
   	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.tryGetFileStatus(AzureBlobFileSystem.java:1035)
   	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:135)
   	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3460)
   	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
   	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3565)
   	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3518)
   	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:592)
   	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:604)
   	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.createFileSystem(AbstractAbfsIntegrationTest.java:271)
   	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:146)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
   	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: Operation failed: "The specified filesystem does not exist.", 404, HEAD, https://abfstorageacc.dfs.core.windows.net/abfs-testcontainer-72fd54ec-fe7b-42bc-a21c-61a175de9286/?upn=false&action=getAccessControl&timeout=90
   	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:225)
   	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:190)
   	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:454)
   	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:188)
   	... 29 more
   ```
   Requests seem to return 404 with FileSystem not found exception. Tried to debug didn't
get much info with that as well. Have you faced this issue before with DurationTrackingFactory
@steveloughran ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 575309)
    Time Spent: 40m  (was: 0.5h)

> ABFS to collect IOStatistics
> ----------------------------
>
>                 Key: HADOOP-17471
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17471
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Steve Loughran
>            Assignee: Mehakmeet Singh
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Add stats collection to ABFS FS operations, especially
> * create
> * open
> * delete
> * rename
> * getFilesStatus
> * list
> * attribute get/set



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message