jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Gaul (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-1475) Exception listing S3 dir where path contains %
Date Fri, 04 Jan 2019 20:39:00 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Gaul updated JCLOUDS-1475:
---------------------------------
    Description: 
Issue 278 was a similar problem, seemingly pretty different code path.

Doing S3BlobStore.list, if directory name is something like Baz%, an exception is thrown.
Encoding the % avoids the exception, but does not find the directory to return results. Interestingly,
%Baz works. I believe the problem is that http.utils.Queries.parseQueryToDecodedMap decodes
the entire path string, when it really only cases about the parameters. I think %Baz works
because %Ba is a valid escape pattern. So it gets put in the parameter map, but never used.

Here's a call stack. My path within an S3 bucket is "dgb6/Foo/Baz%" .

{noformat}
ERROR CloudStoreManager 2018-12-14 11:53:57,546 http-nio-8080-exec-1 : Failed to delete directory
dgb6/Foo/Baz%
java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
 at java.net.URLDecoder.decode(URLDecoder.java:187)
 at org.jclouds.util.Strings2.urlDecode(Strings2.java:131)
 at org.jclouds.http.utils.Queries.parseKeyValueFromStringToDecodedMap(Queries.java:61)
 at org.jclouds.http.utils.Queries.access$000(Queries.java:30)
 at org.jclouds.http.utils.Queries$1.apply(Queries.java:45)
 at org.jclouds.http.utils.Queries$1.apply(Queries.java:32)
 at org.jclouds.aws.s3.filters.AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader(AWSRequestAuthorizeSignatureV4.java:56)
 at org.jclouds.s3.filters.RequestAuthorizeSignatureV4.filter(RequestAuthorizeSignatureV4.java:55)
 at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:90)
 at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
 at com.sun.proxy.$Proxy79.listBucket(Unknown Source)
 at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
 at org.labkey.cloud.store.model.CloudStoreManager.clearContainer(CloudStoreManager.java:1047)
{noformat}
 

Here's the code:

{code:java}
ListContainerOptions listOpts = ListContainerOptions.Builder.prefix(dir.toString());
for (StorageMetadata meta : store.list(bucket, listOpts))
{
 if (!dirString.equals(meta.getName()) &&
 (StorageType.FOLDER.equals(meta.getType()) || StorageType.RELATIVE_PATH.equals(meta.getType())))
 {
 clearContainer(store, bucket, new Path(meta.getName()));
 }
}
{code}

  was:
Issue 278 was a similar problem, seemingly pretty different code path.

Doing S3BlobStore.list, if directory name is something like Baz%, an exception is thrown.
Encoding the % avoids the exception, but does not find the directory to return results. Interestingly,
%Baz works. I believe the problem is that http.utils.Queries.parseQueryToDecodedMap decodes
the entire path string, when it really only cases about the parameters. I think %Baz works
because %Ba is a valid escape pattern. So it gets put in the parameter map, but never used.

Here's a call stack. My path within an S3 bucket is "dgb6/Foo/Baz%" .

ERROR CloudStoreManager 2018-12-14 11:53:57,546 http-nio-8080-exec-1 : Failed to delete directory
dgb6/Foo/Baz%
java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
 at java.net.URLDecoder.decode(URLDecoder.java:187)
 at org.jclouds.util.Strings2.urlDecode(Strings2.java:131)
 at org.jclouds.http.utils.Queries.parseKeyValueFromStringToDecodedMap(Queries.java:61)
 at org.jclouds.http.utils.Queries.access$000(Queries.java:30)
 at org.jclouds.http.utils.Queries$1.apply(Queries.java:45)
 at org.jclouds.http.utils.Queries$1.apply(Queries.java:32)
 at org.jclouds.aws.s3.filters.AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader(AWSRequestAuthorizeSignatureV4.java:56)
 at org.jclouds.s3.filters.RequestAuthorizeSignatureV4.filter(RequestAuthorizeSignatureV4.java:55)
 at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:90)
 at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
 at com.sun.proxy.$Proxy79.listBucket(Unknown Source)
 at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
 at org.labkey.cloud.store.model.CloudStoreManager.clearContainer(CloudStoreManager.java:1047)

 

Here's the code:

ListContainerOptions listOpts = ListContainerOptions.Builder.prefix(dir.toString());
for (StorageMetadata meta : store.list(bucket, listOpts))
{
 if (!dirString.equals(meta.getName()) &&
 (StorageType.FOLDER.equals(meta.getType()) || StorageType.RELATIVE_PATH.equals(meta.getType())))
 {
 clearContainer(store, bucket, new Path(meta.getName()));
 }
}

 

Thanks! 


> Exception listing S3 dir where path contains %
> ----------------------------------------------
>
>                 Key: JCLOUDS-1475
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1475
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-core
>    Affects Versions: 2.1.1
>         Environment: MacBook Pro, OSX High Sierra (10.13.6)
>            Reporter: Dave Bradlee
>            Priority: Major
>
> Issue 278 was a similar problem, seemingly pretty different code path.
> Doing S3BlobStore.list, if directory name is something like Baz%, an exception is thrown.
Encoding the % avoids the exception, but does not find the directory to return results. Interestingly,
%Baz works. I believe the problem is that http.utils.Queries.parseQueryToDecodedMap decodes
the entire path string, when it really only cases about the parameters. I think %Baz works
because %Ba is a valid escape pattern. So it gets put in the parameter map, but never used.
> Here's a call stack. My path within an S3 bucket is "dgb6/Foo/Baz%" .
> {noformat}
> ERROR CloudStoreManager 2018-12-14 11:53:57,546 http-nio-8080-exec-1 : Failed to delete
directory dgb6/Foo/Baz%
> java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern
>  at java.net.URLDecoder.decode(URLDecoder.java:187)
>  at org.jclouds.util.Strings2.urlDecode(Strings2.java:131)
>  at org.jclouds.http.utils.Queries.parseKeyValueFromStringToDecodedMap(Queries.java:61)
>  at org.jclouds.http.utils.Queries.access$000(Queries.java:30)
>  at org.jclouds.http.utils.Queries$1.apply(Queries.java:45)
>  at org.jclouds.http.utils.Queries$1.apply(Queries.java:32)
>  at org.jclouds.aws.s3.filters.AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader(AWSRequestAuthorizeSignatureV4.java:56)
>  at org.jclouds.s3.filters.RequestAuthorizeSignatureV4.filter(RequestAuthorizeSignatureV4.java:55)
>  at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:90)
>  at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
>  at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
>  at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
>  at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>  at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>  at com.sun.proxy.$Proxy79.listBucket(Unknown Source)
>  at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
>  at org.labkey.cloud.store.model.CloudStoreManager.clearContainer(CloudStoreManager.java:1047)
> {noformat}
>  
> Here's the code:
> {code:java}
> ListContainerOptions listOpts = ListContainerOptions.Builder.prefix(dir.toString());
> for (StorageMetadata meta : store.list(bucket, listOpts))
> {
>  if (!dirString.equals(meta.getName()) &&
>  (StorageType.FOLDER.equals(meta.getType()) || StorageType.RELATIVE_PATH.equals(meta.getType())))
>  {
>  clearContainer(store, bucket, new Path(meta.getName()));
>  }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message