jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Gaul (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-1482) Uploading a file to a Swift container create two items inside it: one of type BLOB and other of type FOLDER.
Date Fri, 04 Jan 2019 20:45:00 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Gaul updated JCLOUDS-1482:
---------------------------------
    Description: 
*Usecase*: I'm trying to upload a file(107kb) to an OpenStack Swift container using the api
BlobStore putBlob(String container, Blob blob, PutOptions options). Below code snippet demonstrates
the way I get the BlobStoreContext and upload a file.

{code:java}
//To get the BlobStoreContext for OpenStack Swift

private BlobStoreContext getBlobStoreContext() {
 final Properties override = new Properties();
 override.put(KeystoneProperties.KEYSTONE_VERSION, "3");
 override.put(KeystoneProperties.SCOPE, "project:" + swiftConfig.getProject());
 
 final String credential = swiftConfig.getPassword();
 final String identity = swiftConfig.getUserDomain() + ":" + swiftConfig.getUsername();
 final String authUrl = swiftConfig.getAuthUrl();
 
 return ContextBuilder.newBuilder("openstack-swift")
 .endpoint(authUrl+"/v3")
 .credentials(identity,credential)
 .overrides(override)
 .buildApi(BlobStoreContext.class);
 }

//To upload a file into a container of an objectstore e.g AWS S3, OpenStack Swift

public String uploadFile(String bucketName, byte[] bytes, String fileName) {

BlobStore blobStore = getBlobStoreContext().getBlobStore();

boolean isBlobExist = blobStore.blobExists(bucketName, fileName);
 if(!isBlobExist){

//creating payload
 Payload payload = new ByteArrayPayload(bytes);

//adding user metadata to the blob
 Map<String,String> userMetadata = new HashMap<String,String>();
 userMetadata.put("description", "sample content");

//creating Blob
 Blob blob = blobStore.blobBuilder(fileName)
 .payload(payload)
 .userMetadata(userMetadata)
 .build();
String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);
String message = fileName+" uploaded successfully with etag "+etag;

return message;

}

}
{code}

*Issue:* Once the upload process is complete, I found two items created inside the container.
One is of type *BLOB* and the other is of type *FOLDER* as shown below:

TestFile.DOCX
TestFile.DOCX/slo/1546583793.081000/109256/33554432/00000000

The actual issue is with the below line where I use multipart:

String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);

However the above code works fine in AWS S3 and adds only one item of type BLOB into the required
container.

*Debug:* I debug the code to find the following:
 # step into  putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*)
 # step into RegionScopedSwiftBlobStore > putBlob(String container, Blob blob, PutOptions
options).
 # step into putMultipartBlob(String container, Blob blob, PutOptions overrides). Here i found
the *contentLength:* 109256(file size), *partSize:* 33554432, *part: 0*
 # **step into initiateMultipartUpload(String container, BlobMetadata blobMetadata, long
partSize, PutOptions options). Here I found the *contentLength:* 109256(file size), *uploadId*: TestFile.DOCX/slo/1546583793.081000/109256/33554432
 # step into MultipartUpload.create(container, blobMetadata.getName(), uploadId, blobMetadata,
options)
 # step into new AutoValue_MultipartUpload(containerName, blobName, id, blobMetadata, putOptions).
Here onwards I can't debug any further.

*Analysis*: I observed that if I use SwiftApi instead of the traditional BlobStore then I
get the desired result as it creates only one item of type BLOB inside the container. But
since we are evaluating a java based third party library to develop a multi cloud application
and so far we found Jclouds is the right choice so we would like to use portable apis e.g
BlobStore to perform CRUD operations instead of using provider specific apis e.g SwiftApi.

Any help would be greatly appreciated. 

  was:
*Usecase*: I'm trying to upload a file(107kb) to an OpenStack Swift container using the api
BlobStore putBlob(String container, Blob blob, PutOptions options). Below code snippet demonstrates
the way I get the BlobStoreContext and upload a file.

//To get the BlobStoreContext for OpenStack Swift

private BlobStoreContext getBlobStoreContext() {
 final Properties override = new Properties();
 override.put(KeystoneProperties.KEYSTONE_VERSION, "3");
 override.put(KeystoneProperties.SCOPE, "project:" + swiftConfig.getProject());
 
 final String credential = swiftConfig.getPassword();
 final String identity = swiftConfig.getUserDomain() + ":" + swiftConfig.getUsername();
 final String authUrl = swiftConfig.getAuthUrl();
 
 return ContextBuilder.newBuilder("openstack-swift")
 .endpoint(authUrl+"/v3")
 .credentials(identity,credential)
 .overrides(override)
 .buildApi(BlobStoreContext.class);
 }

//To upload a file into a container of an objectstore e.g AWS S3, OpenStack Swift

public String uploadFile(String bucketName, byte[] bytes, String fileName) {

BlobStore blobStore = getBlobStoreContext().getBlobStore();

boolean isBlobExist = blobStore.blobExists(bucketName, fileName);
 if(!isBlobExist){

//creating payload
 Payload payload = new ByteArrayPayload(bytes);

//adding user metadata to the blob
 Map<String,String> userMetadata = new HashMap<String,String>();
 userMetadata.put("description", "sample content");

//creating Blob
 Blob blob = blobStore.blobBuilder(fileName)
 .payload(payload)
 .userMetadata(userMetadata)
 .build();
String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);
String message = fileName+" uploaded successfully with etag "+etag;

return message;

}

}

*Issue:* Once the upload process is complete, I found two items created inside the container.
One is of type *BLOB* and the other is of type *FOLDER* as shown below:

TestFile.DOCX
TestFile.DOCX/slo/1546583793.081000/109256/33554432/00000000

The actual issue is with the below line where I use multipart:

String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);

However the above code works fine in AWS S3 and adds only one item of type BLOB into the required
container.

*Debug:* I debug the code to find the following:
 # step into  putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*)
 # step into RegionScopedSwiftBlobStore > putBlob(String container, Blob blob, PutOptions
options).
 # step into putMultipartBlob(String container, Blob blob, PutOptions overrides). Here i found
the *contentLength:* 109256(file size), *partSize:* 33554432, *part: 0*
 # **step into initiateMultipartUpload(String container, BlobMetadata blobMetadata, long
partSize, PutOptions options). Here I found the *contentLength:* 109256(file size), *uploadId*: TestFile.DOCX/slo/1546583793.081000/109256/33554432
 # step into MultipartUpload.create(container, blobMetadata.getName(), uploadId, blobMetadata,
options)
 # step into new AutoValue_MultipartUpload(containerName, blobName, id, blobMetadata, putOptions).
Here onwards I can't debug any further.

*Analysis*: I observed that if I use SwiftApi instead of the traditional BlobStore then I
get the desired result as it creates only one item of type BLOB inside the container. But
since we are evaluating a java based third party library to develop a multi cloud application
and so far we found Jclouds is the right choice so we would like to use portable apis e.g
BlobStore to perform CRUD operations instead of using provider specific apis e.g SwiftApi.

Any help would be greatly appreciated. 


> Uploading a file to a Swift container create two items inside it: one of type BLOB and
other of type FOLDER. 
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: JCLOUDS-1482
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1482
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.1.1
>            Reporter: Biswa Ranjan Ray
>            Priority: Major
>              Labels: openstack-swift
>
> *Usecase*: I'm trying to upload a file(107kb) to an OpenStack Swift container using the
api BlobStore putBlob(String container, Blob blob, PutOptions options). Below code snippet
demonstrates the way I get the BlobStoreContext and upload a file.
> {code:java}
> //To get the BlobStoreContext for OpenStack Swift
> private BlobStoreContext getBlobStoreContext() {
>  final Properties override = new Properties();
>  override.put(KeystoneProperties.KEYSTONE_VERSION, "3");
>  override.put(KeystoneProperties.SCOPE, "project:" + swiftConfig.getProject());
>  
>  final String credential = swiftConfig.getPassword();
>  final String identity = swiftConfig.getUserDomain() + ":" + swiftConfig.getUsername();
>  final String authUrl = swiftConfig.getAuthUrl();
>  
>  return ContextBuilder.newBuilder("openstack-swift")
>  .endpoint(authUrl+"/v3")
>  .credentials(identity,credential)
>  .overrides(override)
>  .buildApi(BlobStoreContext.class);
>  }
> //To upload a file into a container of an objectstore e.g AWS S3, OpenStack Swift
> public String uploadFile(String bucketName, byte[] bytes, String fileName) {
> BlobStore blobStore = getBlobStoreContext().getBlobStore();
> boolean isBlobExist = blobStore.blobExists(bucketName, fileName);
>  if(!isBlobExist){
> //creating payload
>  Payload payload = new ByteArrayPayload(bytes);
> //adding user metadata to the blob
>  Map<String,String> userMetadata = new HashMap<String,String>();
>  userMetadata.put("description", "sample content");
> //creating Blob
>  Blob blob = blobStore.blobBuilder(fileName)
>  .payload(payload)
>  .userMetadata(userMetadata)
>  .build();
> String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);
> String message = fileName+" uploaded successfully with etag "+etag;
> return message;
> }
> }
> {code}
> *Issue:* Once the upload process is complete, I found two items created inside the container.
One is of type *BLOB* and the other is of type *FOLDER* as shown below:
> TestFile.DOCX
> TestFile.DOCX/slo/1546583793.081000/109256/33554432/00000000
> The actual issue is with the below line where I use multipart:
> String etag = blobStore.putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*);
> However the above code works fine in AWS S3 and adds only one item of type BLOB into
the required container.
> *Debug:* I debug the code to find the following:
>  # step into  putBlob(bucketName, blob, *PutOptions.Builder.multipart(true)*)
>  # step into RegionScopedSwiftBlobStore > putBlob(String container, Blob blob, PutOptions
options).
>  # step into putMultipartBlob(String container, Blob blob, PutOptions overrides). Here
i found the *contentLength:* 109256(file size), *partSize:* 33554432, *part: 0*
>  # **step into initiateMultipartUpload(String container, BlobMetadata blobMetadata,
long partSize, PutOptions options). Here I found the *contentLength:* 109256(file size),
*uploadId*: TestFile.DOCX/slo/1546583793.081000/109256/33554432
>  # step into MultipartUpload.create(container, blobMetadata.getName(), uploadId, blobMetadata,
options)
>  # step into new AutoValue_MultipartUpload(containerName, blobName, id, blobMetadata,
putOptions). Here onwards I can't debug any further.
> *Analysis*: I observed that if I use SwiftApi instead of the traditional BlobStore then
I get the desired result as it creates only one item of type BLOB inside the container. But
since we are evaluating a java based third party library to develop a multi cloud application
and so far we found Jclouds is the right choice so we would like to use portable apis e.g
BlobStore to perform CRUD operations instead of using provider specific apis e.g SwiftApi.
> Any help would be greatly appreciated. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message