Hi Isuru,

Big +1 for this effort. Shall we create a JIRA to track this and get it fixed in next release?

I've noticed few other issues related to Stratos threading behavior.

 - New threads are created in certain places [1] which should be avoided as much as possible.

@Reka: Any idea on why we create new Threads in ClusterLevelPartitionContext at [1]? This would cause Stratos to go out of resources at some point right?

 - New threads are created in component activators [2]. Although level of impact is minimum since this is a one-time creating, we should be using the thread pool allocated at all times.

 - Here is a code piece found in CC service component;

private void shutdownExecutorService(String executorServiceId) {
ExecutorService executorService = StratosThreadPool.getExecutorService(executorServiceId, 1);
if (executorService != null) {
shutdownExecutorService(executorService);
}
}
private void shutdownExecutorService(ExecutorService executorService) {
try {
executorService.shutdownNow();
} catch (Exception e) {
log.warn("An error occurred while shutting down executor service", e);
}
}

It really doesn't make sense to get the thread pool object already created by passing a thread pool size. We should have a fully flexible API in StratosThreadPool class in Stratos common component to perform those operations.

[1] https://github.com/apache/stratos/blob/stratos-4.1.x/components/org.apache.stratos.autoscaler/src/main/java/org/apache/stratos/autoscaler/context/partition/ClusterLevelPartitionContext.java#L116-L121

[2] https://github.com/apache/stratos/blob/stratos-4.1.x/components/org.apache.stratos.autoscaler/src/main/java/org/apache/stratos/autoscaler/internal/AutoscalerServiceComponent.java#L120


Thanks.

On Thu, Dec 3, 2015 at 10:20 AM, Isuru Haththotuwa <isuruh@apache.org> wrote:
Hi Devs,

Currently what we have is a static set of thread pools:
  1. monitor thread pool
  2. autoscaler thread pool
  3. cartridge agent event listener thread pool
  4. cloud controller context thread pool
  5. cloud controller thread pool
  6. cloud controller instance manager thread pool
  7. cloud controller stats publisher thread pool
  8. autoscaler stats publisher thread pool
  9. load balancer thread pool
  10. mock iaas event listener thread pool
  11. stratos manager thread pool

This is in addition to the scheduled executor services we are using. Shall we optimize the StratosThreadPool implementation to dynamically shrink and grow when required? We are currently using fixed thread pools.




On Sun, Nov 22, 2015 at 12:15 PM, Imesh Gunaratne <imesh@apache.org> wrote:
Hi Akila,

On Sat, Nov 21, 2015 at 1:20 PM, Akila Ravihansa Perera <ravihansa@wso2.com> wrote:

Any idea why thread pools are created per monitor object? Is it not possible to share a common thread pool as per the current design?

No, we do not create a thread pool per monitor object, have a look at the code carefully:

executorService = StratosThreadPool.getExecutorService(
                AutoscalerConstants.MONITOR_THREAD_POOL_ID, threadPoolSize);

The above statement ask Stratos Thread Pool class to return an executor service with the name AutoscalerConstants.MONITOR_THREAD_POOL_ID:

public static ExecutorService getExecutorService(String identifier, int threadPoolSize) {
        ExecutorService executorService = executorServiceMap.get(identifier);
        if (executorService == null) {
            synchronized (executorServiceMapLock) {
                if (executorService == null) {
                    executorService = Executors.newFixedThreadPool(threadPoolSize);
                    executorServiceMap.put(identifier, executorService);
                    log.info(String.format("Thread pool created: [type] Executor Service [id] %s [size] %d", identifier, threadPoolSize));
                }
            }
        }
        return executorService;
    }

This will only create an executor service once for a given identifier.

Thanks
 

Stratos server thread count goes beyond 1500 when deploying 2 or 3 apps. And it keeps growing proportional to the deployed app count. This is a very high resource usage.





Thanks.

--
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com



--
Imesh Gunaratne

Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

--
Thanks and Regards,

Isuru H.






--
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com