jclouds-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jasdeep Hundal <jasdeep.singh.hun...@gmail.com>
Subject Re: Perfroamnce degrade for JCloud-OpenStack Swift.
Date Tue, 22 Apr 2014 01:48:59 GMT
Hey Sumit,

I have a couple more questions that might help clarify the situation:

1. Are you running the stability test as a single long running Java process
(that just keeps cycling through the 10 uploads/gets/deletes)?

2. Are you always running the test in the same container, or are you
creating new containers for each test iteration?

3. If the answer to #2 is is that the test runs in a single container, how
many objects does that container currently have?

It may also help to time each of the individual blobstore actions as you
run the test to see if any particular one is slowing down.

Jasdeep


On Mon, Apr 21, 2014 at 6:21 PM, Sumit Gaur <sumitkgaur@gmail.com> wrote:

> hi Shri,
> Please find answers below
>
> On Tue, Apr 22, 2014 at 9:23 AM, Shrinand Javadekar <
> shrinand@maginatics.com
> > wrote:
> Few more questions to try and understand this better:
>
> 1) On the Swift instance you are using, how many replicas do you have?
>
> 3 replica
>
> 2) Also, how are you using the curl command in the shell script?
>
> send below command in backgroud for 10 iterations and wait similiar to the
> 10 threads in jclouds.
>
>             curl -X PUT -i -T 100k -H "X-Auth-Token: $OS_AUTH_TOKEN"
> http://
>
> $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt
>             curl -X DELETE -i -H "X-Auth-Token: $OS_AUTH_TOKEN" http://
>
> $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt
>
> I
> think the shell script and jclouds-with-10-parallel-threads may not be
> doing the same amount of work. In 20 hours jclouds might be doing much
> more work than the shell script. If you let the shell script also go
> upto that point, it might see failures too. Do you know how many
> PUT-GET-DEL operations have been performed when you start seeing the
> 409 errors.
>
> Actually 409 errors are coming since the start of the test but TPS start
> degrading after 24-48 hours.
> On Apr 22, 2014 9:23 AM, "Shrinand Javadekar" <shrinand@maginatics.com>
> wrote:
>
> > Few more questions to try and understand this better:
> >
> > 1) On the Swift instance you are using, how many replicas do you have?
> > 2) Also, how are you using the curl command in the shell script? I
> > think the shell script and jclouds-with-10-parallel-threads may not be
> > doing the same amount of work. In 20 hours jclouds might be doing much
> > more work than the shell script. If you let the shell script also go
> > upto that point, it might see failures too. Do you know how many
> > PUT-GET-DEL operations have been performed when you start seeing the
> > 409 errors.
> >
> > -Shri
> >
> >
> > On Mon, Apr 21, 2014 at 4:55 PM, Sumit Gaur <sumitkgaur@gmail.com>
> wrote:
> > > FYI ..This is block of code .....   also I am using jclouds 1.7.1
> (Stable
> > > branch)
> > >      try {
> > > String key = "objkey" + UUID.randomUUID();
> > >                 Blob blob =
> > > Example9.blobStore.blobBuilder(key).payload(Example9.file).build();
> > >
> Example9.blobStore.putBlob(Example9.containerName+count,
> > > blob);
> > >
> Example9.blobStore.getBlob(Example9.containerName+count,
> > > key);
> > >
> > Example9.blobStore.removeBlob(Example9.containerName+count,
> > > key);
> > >         } catch (Exception ace) {
> > >                 System.out.println("Request failed for objkey " + key
> + "
> > >  " + ace);
> > >         }
> > >
> > >
> > >
> > > On Tue, Apr 22, 2014 at 8:32 AM, Sumit Gaur <sumitkgaur@gmail.com>
> > wrote:
> > >
> > >> Hi Shri,
> > >> Thanks for paying attention to it, Please find my answers below:-
> > >>
> > >>
> > >> On Tue, Apr 22, 2014 at 2:31 AM, Shrinand Javadekar <
> > >> shrinand@maginatics.com> wrote:
> > >>
> > >>> Sumit,
> > >>>
> > >>> I realize that you had sent out a similar email sometime ago about
> > >>> performance degradation. I'm not sure if anyone has run these types
> of
> > >>> long running experiments with jclouds. So this may be a first.
> > >>>
> > >> Tried to debug it in last 2 weeks without success. Want to understand
> > more
> > >> how jclouds code handle this use case or any pointers that this is a
> > >> problematic use case would help
> > >>
> > >>>
> > >>> The 409 status is returned because of a conflict [1]. Are you sure
> you
> > >>> didn't have two or more threads trying to delete the same object?
> > >>>
> > >> No two threads share the same object key in my programme (String key =
> > >> "objkey" + UUID.randomUUID();). It is some kind of race between PUT
> and
> > >> DEL call . If I put say 10 ms sleep between call then there is no 409
> > error.
> > >>
> > >>
> > >>> Also, I see that that 409 is returned by Swift if you try to delete
a
> > >>> container that isn't empty[2]. Is that something your test code
> > >>> could've tried?
> > >>>
> > >> I am trying to delete objects .. not containers.
> > >>
> > >>>
> > >>> When you say there was a similar test you're trying with curl, are
> you
> > >>> using the curl command-line utility or the libcurl library?
> > >>
> > >> curl command in shell script with for loops.
> > >>
> > >>
> > >>> How are
> > >>> you specifying the number of threads to use and what object each
> > >>> thread should get/put/delete?
> > >>>
> > >>
> > >> It is a java test programme using ThreadPoolExecutor. Somthing
> similiar
> > as
> > >> here
> > >>
> > >>
> >
> http://www.javacodegeeks.com/2013/01/java-thread-pool-example-using-executors-and-threadpoolexecutor.html
> > >>
> > >> Object is a 5KB file. with  key = "objkey" + UUID.randomUUID(); with
> > Pool
> > >> of 10  threads.
> > >>
> > >>
> > >> Hope this would give a good inside. Let me know if you get any problem
> > >> here.
> > >>
> > >>
> > >>>
> > >>> Thanks.
> > >>> -Shri
> > >>>
> > >>> [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> > >>> [2] https://bugs.launchpad.net/horizon/+bug/1096084
> > >>>
> > >>> On Sun, Apr 20, 2014 at 5:55 PM, Sumit Gaur <sumitkgaur@gmail.com>
> > wrote:
> > >>> > Hi
> > >>> > I using jclouds lib integrated with Openstack Swift+ keystone
> > >>> combinaiton.
> > >>> > Things are working fine except stability test. After 20-30 hours
of
> > test
> > >>> > jclouds/SWIFT start degrading in TPS and keep going down over
the
> > time.
> > >>> >
> > >>> > 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads.
> > >>> > 2) I am getting a lot of 409 and DEL failure for the as response
> too
> > >>> from
> > >>> > SWIFT.
> > >>> > 3) Direct similiar test from curl does not show much impact and
TPS
> > >>> remain
> > >>> > constant.
> > >>> >
> > >>> > Can sombody help me wht is going wrong here ?
> > >>> >
> > >>> > Thanks
> > >>> > sumit
> > >>>
> > >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message