stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isuru Haththotuwa <isu...@apache.org>
Subject Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils
Date Thu, 03 Dec 2015 02:52:51 GMT
Superb work Akila! Integration test are about three times faster now.
Earlier it used to take ~45 mins, now run in ~15 mins.

On Wed, Dec 2, 2015 at 6:48 PM, Imesh Gunaratne <imesh@apache.org> wrote:

> Fabulous work Akila!! :-)
>
> On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Hi,
>>
>> Currently there are lot of Thread.sleep calls in mock iaas component
>> which makes it slow and cause unexpected behavior due to concurrency
>> issues. Also it has a significant performance overhead when running
>> integration tests since mock iaas is being used for test cases. I've been
>> working on improving this component by doing following changes;
>>
>>  - Remove *all* Thread sleep calls in mock iaas
>>  - Introduce a method named 'initialize' to start event receivers and
>> publishers. This is a synchronous call which grantees that receiver and
>> publisher objects will be created successfully. If not it will throw an
>> exception and startInstance() method call in CC will fail. Earlier this
>> task was delegated to an executor service which made it difficult to check
>> whether mock instance was created successfully.
>>  - Create topology receiver in mock instance and listen for member
>> initialized and member started events. It will publish instance started and
>> instance activated events based on topology events received rather than
>> sleeping for some time interval before publishing.
>>
>> After making those changes I faced multiple integration test failures.
>> This was mainly because integration tests relied heavily on Thread sleep
>> calls to assert various conditions. With these changes, the time taken for
>> a mock instance/app to become active came down to milliseconds, hence test
>> cases could not detect member status or app status correctly. Therefore I
>> had to introduce a new non-blocking mechanism to check app/member status by
>> using thread synchronization.
>>
>> Now the average time taken for complete integration tests is around 16
>> mins (earlier it was more than 30 mins). This is almost 50% performance
>> gain. Created JIRA at [1].
>>
>>
>> Following is a summary of changes.
>>
>> AutoscalerTopologyEventReceiver:
>>  - Fix formatting and log messages
>>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
>> calling destroy() on monitor
>>
>> AutoscalerServiceImpl
>>  - Fix formatting and log messages
>>
>> ClusterStatusActiveProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> GroupStatusActiveProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>   - Print groups map entries and cluster data holder map entries if debug
>> enabled
>>
>> GroupStatusProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> GroupStatusTerminatedProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> CloudControllerServiceComponent
>>   - Increase THREAD_POOL_SIZE from 10 to 20
>>
>>
>> TopologyBuilder
>>   - Fix formatting and log messages
>>   - Move acquire topology lock call outside of try block
>>
>> RestClient
>>  - Add logs for every method to help troubleshooting integration failures
>>
>> StratosTestServerManager
>>  - Call waitForPort method with restart timeout of 600000 ms. This is to
>> avoid test failures due to slow builder machines.
>>
>>
>> RestConstants
>>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>>
>> TopologyHandler
>>  - Update timeout values:
>> APPLICATION_ACTIVATION_TIMEOUT = 300000;
>> APPLICATION_INACTIVATION_TIMEOUT = 120000;
>> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
>> MEMBER_TERMINATION_TIMEOUT = 120000;
>> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>>
>>  - Increase executorService pool size from 10 to 30 to compensate for
>> additional event receivers
>>  - Event receivers with logs for events
>>      - healthStatEventReceiver: MemberFaultEvent
>>
>>      - applicationsEventReceiver:
>> ApplicationInstanceActivatedEventListener,
>> ApplicationInstanceInactivatedEventListener
>>
>>      - topologyEventReceiver: MemberActivatedEventListener,
>> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
>> ClusterInstanceInactivateEventListener
>>
>>  - Added logs for every method to help troubleshoot integration failures
>>  - Asynchronous mechanism to assertApplicationActiveStatus
>>  - Asynchronous mechanism to assertApplicationInactiveStatus
>>
>>
>> Application
>>  - Added log in getStatus() method to print status of all application
>> instances
>>
>>
>> ApplicationCreatedMessageProcessor
>>   - Fix formatting and log messages
>>
>> ApplicationDeletedMessageProcessor
>>   - Fix formatting and log messages
>> ApplicationInstanceActivatedMessageProcessor
>>   - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceInactivatedMessageProcessor
>> - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatingMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ClusterStatusClusterActivatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInactivateMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterResetMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatingMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> Introduce removeEventListener method for message processor chain to
>> remove a registered event listener object. This is needed since integration
>> tests will be registering event listeners on-demand. Those listeners needs
>> to be removed at the end of the test case.
>>  - ApplicationSignUpMessageProcessorChain
>>  - ApplicationsMessageProcessorChain
>>  - ClusterStatusMessageProcessorChain
>>  - DomainMappingMessageProcessorChain
>>  - HealthStatMessageProcessorChain
>>  - InitializerMessageProcessorChain
>>  - InstanceNotifierMessageProcessorChain
>>  - InstanceStatusMessageProcessorChain
>>  - TenantMessageProcessorChain
>>  - TopologyMessageProcessorChain
>>  - MessageProcessorChain
>>  - ApplicationsEventMessageDelegator
>>  - ApplicationsEventReceiver
>>
>> InstanceNotifierEventReceiver
>>  - synchronized blocks for execute() and terminate()
>>  - eventSubscriber object creation moved to constructor from execute()
>> method. This is to avoid possible NPE when calling terminate() method
>>
>> MetadataApi
>>  - Catch generic Exception instead of RegistryException
>>
>> DataStore
>>  - throws MetadataException in addition to RegistryException
>> MetadataApiRegistry
>>  - throw new MetadataException instead of RegistryException
>>
>>
>> MockIaasServiceComponent
>>  - Move mockIaasServiceUtil.startInstancesPersisted() to
>> MockIaasServiceImpl()
>>
>> MockIaasServiceImpl
>>  - start persisted mock instances in the constructor
>>  - Remove all thread sleep calls
>>
>>
>> MockIaasServiceUtil
>>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>>
>>
>> MockInstance
>>  - Create TopologyEventReceiver object and register:
>> MemberInitializedEventListener, MemberStartedEventListener,
>> MemberMaintenanceListener
>>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>>  - Publish InstanceReadyToShutdownEvent upon receiving
>> MemberMaintenanceModeEvent
>>  - synchronized terminate() and initialize() methods
>>  - Introduce an initialize() method to start event receivers and health
>> stat publisher
>>  - Introduce field 'memberStatus' of type MemberStatus to track the
>> life-cycle of mock instance
>>
>> MockIaasServiceTest
>>  - Start embedded MB in unit test case since
>> mockIaasService.startInstance call will start the event receivers as well
>>
>> GitHookTestCase
>>  - Make artifactUpdateEventCount an AtomicInteger
>>  - Replace restClient.doPost call with restClient.addEntity call
>>  - Terminate instanceNotifierEventReceiver at the end of test case
>>  - Shutdown eventListenerExecutorService at the end of test case
>>
>> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>>
>> Thanks.
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Mime
View raw message