stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Waddell <David.Wadd...@owmobility.com>
Subject RE: autoscaler issue with multiple cartridges + subscriptions
Date Wed, 06 Aug 2014 15:40:27 GMT
Hi Akila

   Please find the rule attached.

   Product version -
netiq@octl-01:/opt/wso2/apache-stratos$ cat bin/version.txt
Apache Stratos v4.0.0
netiq@octl-01:/opt/wso2/apache-stratos$ cat ./bin/wso2carbon-version.txt
WSO2 Carbon Framework v4.2.0

    Shaheed - can you comment on the build origin ?



Thanks
David. 


-----Original Message-----
From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com] 
Sent: 06 August 2014 16:28
To: David Waddell
Cc: dev@stratos.apache.org; Shaheed Haque
Subject: Re: autoscaler issue with multiple cartridges + subscriptions

Hi David,

Seems like the Drool file for scaling rule (scaling.drl) [1] is different in the packs that
you are working with. Have you made changes to the existing scaling rule? Could you please
tell us the Stratos version you're working with? (locally built from source, distribution
release etc.)

Please share the scaling rule found in
<stratos_path>/repository/conf/scaling.drl

[1] https://github.com/apache/stratos/blob/4.0.0/products/stratos/modules/distribution/src/main/conf/scaling.drl

Thanks.

On Wed, Aug 6, 2014 at 6:20 PM, David Waddell <David.Waddell@owmobility.com> wrote:
> .. clarified cartridges/policies inline below..
>
> -----Original Message-----
> From: David Waddell [mailto:David.Waddell@owmobility.com]
> Sent: 06 August 2014 13:46
> To: Akila Ravihansa Perera; dev
> Cc: Shaheed Haque
> Subject: RE: autoscaler issue with multiple cartridges + subscriptions
>
> Hi Guys
>     Sorry for delayed reply.
>
>     Akila - apologies for confusion on deployment policies - I had tested with 2 polices,
and the logs show the mix.   Logs this time are not, and issue exists.
>      Cloud-controller.xml attached.
>
>      I reproduced this with DEBUG on for scaler.  I ran with 2 cartridges, opwv-fe with
a policy static-1,  with min/max instances of 1; and opwv-vos with policy autoscale-1-2, min=1/max=2.
>
>      When launched I then stressed the CPU on the opwv-vos instance to induce scaling.
>
>      What I found was    (log name in brackets)
>            -  when only the opwv-vos cartridge is subscribed, scaling did occur (wso2carbon.log.singlesub)
>            - with the opwv-fe subscribed first, then the opwv-fe cartridge subscribed,
scaling did not occur     (wso2carbon.log.2sub)
>            - with the opwv-vos subscribed first, then the opwv-vos, scaling did occur
  (wso2carbon.log.2sub_reverse).
>
>       I.e. It appeared the order of subscription creation affects.  At a rough glance
it might be that both are sharing the same NetworkPartitionContext instance.
>
>      Please let me know if you would like any more detail, and thanks.
>
>
> Rgds
> David
>
>
>
> -----Original Message-----
> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
> Sent: 01 August 2014 07:58
> To: dev
> Cc: Shaheed Haque
> Subject: Re: autoscaler issue with multiple cartridges + subscriptions
>
> Hi David,
>
> I have few concerns about your deployment policies.
>
> 1. I see only 2 deployment policies defined: static-1, autoscale-1-2 But in the cartridge
subscription I can see a deployment policy named autoscale-1-5. Where did that come from?
Can you share your complete deployment policy/policies?
>
> 2. In deployment policy autoscale-1-2, you have given the provide as:
> "provider":"openstack-Core". Is this correct? Can you share your cloud-controller.xml?
>
> As Nirmal suggested, enabling DEBUG logs will give more insights into what is actually
causing this issue. It's better if you can enable DEBUG logs for the autoscaler package to
get the complete picture.
>
> log4j.logger.org.apache.stratos.autoscaler=INFO
>
> Thanks.
>
>
> On Fri, Aug 1, 2014 at 11:30 AM, Nirmal Fernando <nirmal070125@gmail.com> wrote:
>>
>>
>>
>> On Fri, Aug 1, 2014 at 11:30 AM, Nirmal Fernando 
>> <nirmal070125@gmail.com>
>> wrote:
>>>
>>> Hi David,
>>>
>>> Is there any possibility of enabling following logger in the 
>>> log4j.properties file ?
>>>
>>> log4j.logger.org.apache.stratos.autoscaler.rule.RuleLog=DEBUG
>>>
>>>
>>> For each service cluster, we run a Cluster Monitor and that is 
>>> responsible for monitoring and scaling the cluster.
>>>
>>>
>>>
>>> On Thu, Jul 31, 2014 at 10:56 PM, David Waddell 
>>> <David.Waddell@owmobility.com> wrote:
>>>>
>>>> Hi guys
>>>>
>>>>      We’re experiencing an issue on stratos 4.0 - the autoscaler 
>>>> doesn’t seem to be kicking in when multiple cartridges are subscribed.
>>>>
>>>>       When deploying only one cartridge, the autoscaler works as 
>>>> expected .
>>>>
>>>>
>>>>
>>>>      3 cartridges are defined  :   opwv-oam-01, opwv-oam-02, opwv-vos.
>>>>
>>>>
>>>>
>>>> {"displayName":"opwv-vos","description":"opwv-vos
>>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-vos","activeI
>>>> n 
>>>> stances":0,"provider":"cisco","version":"1","multiTenant":false,"ho
>>>> s tName":"qmog.cisco.com","loadBalancer":false}
>>>>
>>>> {"displayName":"opwv-oam-01","description":"opwv-oam-01
>>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-oam-01","acti
>>>> v 
>>>> eInstances":0,"provider":"cisco","version":"1","multiTenant":false,"
>>>> hostName":"qmog.cisco.com","loadBalancer":false}
>>>>
>>>> {"displayName":"opwv-oam-02","description":"opwv-oam-02
>>>> Cartridge","cartridgeAlias":"-","cartridgeType":"opwv-oam-02","acti
>>>> v 
>>>> eInstances":0,"provider":"cisco","version":"1","multiTenant":false,"
>>>> hostName":"qmog.cisco.com","loadBalancer":false}
>>>>
>>>>
>>>>
>>>>       Deployment policies :
>>>>
>>>>
>>>>
>>>>
>>>> {"id":"static-1","partitionGroup":[{"id":"N1","partitionAlgo":"one-
>>>> a 
>>>> fter-another","partition":[{"id":"RegionOne","partitionMin":1,"part
>>>> i 
>>>> tionMax":1,"provider":"openstack-Core","property":[{"name":"region"
>>>> ,
>>>> "value":"RegionOne"}]}]}]}
>>>>
>>>>
>>>> {"id":"autoscale-1-2","partitionGroup":[{"id":"N1","partitionAlgo":"
>>>> one-after-another","partition":[{"id":"RegionOne","pa
>>>>
>>>>
>>>> rtitionMin":1,"partitionMax":2,"provider":"openstack-Core","propert
>>>> y ":[{"name":"region","value":"RegionOne"}]}]}]}
>>>>
>>>>
>>>>
>>>>      Scaling policy :
>>>>
>>>>
>>>>
>>>>
>>>> {"id":"economyPolicy","loadThresholds":{"requestsInFlight":{"averag
>>>> e 
>>>> ":300.0,"secondDerivative":0.0,"gradient":0.0,"scaleDownMarginOfGra
>>>> d 
>>>> ient":1.0,"scaleDownMarginOfSecondDerivative":0.2},"memoryConsumpti
>>>> o 
>>>> n":{"average":6000.0,"secondDerivative":0.0,"gradient":0.0,"scaleDo
>>>> w 
>>>> nMarginOfGradient":1.0,"scaleDownMarginOfSecondDerivative":0.2},"lo
>>>> a 
>>>> dAverage":{"average":40.0,"secondDerivative":0.0,"gradient":0.0,"sc
>>>> a 
>>>> leDownMarginOfGradient":1.0,"scaleDownMarginOfSecondDerivative":0.2
>>>> }
>>>> }}
>>>>
>>>>
>>>>
>>>>       If we subscribe cartridge opwv-vos by itself :
>>>>
>>>>
>>>>
>>>>         TID: [0] [STRATOS] [2014-07-31 15:15:39,836]  INFO 
>>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager} - 
>>>> Successful Subscription: CartridgeSubscription [subscriptionId=0, 
>>>> type=opwv-vos, alias=opwv-vos, autoscalingPolicyName=economyPolicy,
>>>> deploymentPolicyName=autoscale-1-5, subscriber=Subscriber 
>>>> [adminUserName=admin, tenantId=-1234, tenantDomain=carbon.super], 
>>>> repository=Repository [id=0, url=null, userName=, 
>>>> isPrivateRepository=false], 
>>>> cartridgeInfo=org.apache.stratos.cloud.controller.stub.pojo.Cartrid
>>>> g
>>>> eInfo@5288e5b6,
>>>> payload=SERVICE_NAME=opwv-vos,HOST_NAME=opwv-vos.qmog.cisco.com,MUL
>>>> T 
>>>> ITENANT=false,TENANT_ID=-1234,TENANT_RANGE=-1234,CARTRIDGE_ALIAS=op
>>>> w 
>>>> v-vos,CLUSTER_ID=opwv-vos.opwv-vos.domain,CARTRIDGE_KEY=J5xTyGg9k1o
>>>> d 
>>>> 0Dvl,REPO_URL=null,PORTS=22,PROVIDER=cisco,PUPPET_IP=PUPPET_IP,PUPP
>>>> E 
>>>> T_HOSTNAME=PUPPET_HOSTNAME,PUPPET_ENV=PUPPET_ENV,OPWV_INTEGRA_oam_r
>>>> o 
>>>> =opwv-oam-02,TRUSTSTORE_PASSWORD=wso2carbon,OPWV_INTEGRA_fe_server_
>>>> t 
>>>> ype=VOS,OPWV_INTEGRA_wait_for_hosts=oam01~oam02,CEP_PORT=7611,MONIT
>>>> O
>>>> RING_SERVER_SECURE_PORT=0,NO_CARTRIDGE_SUBSCRIBE=false,MB_PORT=6161
>>>> 6 
>>>> ,MB_IP=octl.qmog.cisco.com,CEP_IP=octl.qmog.cisco.com,DEPLOYMENT=de
>>>> f 
>>>> ault,OPWV_INTEGRA_region=Core,ENABLE_DATA_PUBLISHER=false,OPWV_INTE
>>>> G 
>>>> RA_swap_size=2G,MONITORING_SERVER_ADMIN_PASSWORD=xxxx,MONITORING_SE
>>>> R 
>>>> VER_IP=octl.qmog.cisco.com,COMMIT_ENABLED=false,MONITORING_SERVER_A
>>>> D 
>>>> MIN_USERNAME=xxxx,OPWV_INTEGRA_oam_server_role=,CERT_TRUSTSTORE=/op
>>>> t 
>>>> /apache-stratos-cartridge-agent/security/client-truststore.jks,OPWV
>>>> _ 
>>>> INTEGRA_oam_server_type=OAMClient,MONITORING_SERVER_PORT=0,OPWV_INT
>>>> E GRA_oam_rw=opwv-oam-01,OPWV_INTEGRA_sys_component=Core~CC,
>>>> cluster=opwv-vos.opwv-vos.domain], subscriptionDomainMap={} 
>>>> {org.apache.stratos.manager.manager.CartridgeSubscriptionManager}
>>>>
>>>>
>>>>
>>>>     VM is created, we run a stress load on that VM,  we see the 
>>>> load average increase in aggregator.log, and the autoscaler 
>>>> correctly kicks in and spawns a second instance :
>>>>
>>>>
>>>>
>>>> TID: [0] [STRATOS] [2014-07-31 15:24:42,614]  INFO 
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} -  [scale-up] 
>>>> Partition available, hence trying to spawn an instance to scale up!
>>>> {org.apache.stratos.autoscaler.rule.RuleLog}
>>>>
>>>>
>>>>
>>>>       However when we subscribe all the cartridges together,  a 
>>>> stress on the op-vos instance does not trigger autoscale (although 
>>>> the aggregate log correctly reports the load) .
>>>>
>>>>
>>>>
>>>>       Logs are attached.
>>>>
>>>>       Run with single subscription + successful auto scale is from
>>>> 2014-07-31 16:03:25,320 -> 2014-07-31 16:07:48,220
>>>>
>>>>       Run with multiple subscription and no auto scale occurring, is from
>>>> to   2014-07-31 15:59:50,635 -> 2014-07-31 15:50:08,122
>>>>
>>>>
>>>>
>>>>    Apologies for hitting the dev list but there doesn’t appear to 
>>>> be a user list;  If anyone can take a look appreciated J
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> David.
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
>>> WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
>> WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
>



--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com

Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Mime
View raw message