mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jhcha <jh...@kds.co.kr>
Subject Re: sessionClosed is not called sometimes in heavy stress test contions.
Date Fri, 08 Feb 2008 07:59:42 GMT

Thank Mike for replying my question. 

I read your answers but...

I think MINA already knew the session was being closed in my questioned
situation.
because my server session always close after sending the echo data and 
I debugged MINA source
org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method 
always call doRemove() at (SocketIoProcessor.java:489 line).
then doRemove method always call
session.getServiceListeners().fireSessionDestroyed(session); 
at (SocketIoProcessor.java:188 line).

   try {
        key.cancel();
        ch.close();
    } catch (IOException e) {
        session.getFilterChain().fireExceptionCaught(session, e);
    } finally {
        releaseWriteBuffers(session);
        session.getServiceListeners().fireSessionDestroyed(session);   <---
SocketIoProcessor.java:188 line
    }
            
            
and the above session parameter of the fireSessionDestroyed method was
always not null.

the orgin of the problem is the sessions having serviceAddress is not in
managedSessions hashmap 
in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)

    // Get the session set.
    Set<IoSession> sessions = managedSessions.get(serviceAddress);   <---
IoServiceListenerSupport.java:192 line
    // Ignore if unknown.
    if (sessions == null) {
	return;
    }
        
The fireSessionDestroyed method call propagation stopped only sometimes
Because sessions is not gotten from managedSessions at
(IoServiceListenerSupport.java:192 line).

I used setIdleTimeout with read mode, but 
if sessionClosed method is not called, sessionIdle or sessionClosed method
will never be called after that.

My question is that the filter chain fireSessionClosed method and listener'
sessionDestroyed method 
Should not know if the destroying session be not in the managedSessions
hashmap ?

    // Fire session events.
    session.getFilterChain().fireSessionClosed(session);  <---
IoServiceListenerSupport.java:208 line

    // Fire listener events.
    try {
        for (IoServiceListener listener : listeners) {
            listener.sessionDestroyed(session);           <---
        }
    }
        
        
If fireSessionDestroyed method were called with the parameter the closing
session
but not in managedSessions hashmap, Should filter chain fireSessionClosed
method and 
listener' sessionDestroyed method not called ? 


in the following MINA source below the IoServiceListenerSupport.java:147
line.


  public void fireSessionCreated(IoSession session) {
        SocketAddress serviceAddress = session.getServiceAddress();

        // Get the session set.
        Set<IoSession> s = new IdentityHashSet<IoSession>();
        Set<IoSession> sessions =
managedSessions.putIfAbsent(serviceAddress,
                Collections.synchronizedSet(s));
        boolean firstSession;

        if (null == sessions) {
            sessions = s;
            firstSession = true;
        } else {
            firstSession = false;
        }

        // If already registered, ignore.
        if (!sessions.add(session)) { <---
            return;
        }

        // If the first connector session, fire a virtual service activation
event.
        if (session.getService() instanceof IoConnector && firstSession) {
            fireServiceActivated(session.getService(), session
                    .getServiceAddress(), session.getHandler(), session
                    .getServiceConfig());
        }

        // Fire session events.
        session.getFilterChain().fireSessionCreated(session);
        session.getFilterChain().fireSessionOpened(session);

        // Fire listener events.
        for (IoServiceListener listener : listeners) {
            listener.sessionCreated(session);
        }
    }
    

I think the code "if (!sessions.add(session))" is always true,
So filter chain fireSessionCreated method and listener' sessionCreated
method is 
always called in spite of the heavy unstable tcp stress condition.

I think 
Creaing callback method and destroying callback method should have the same
calling count. 

But creating callback logic and managedSessions hashmap are not related
directly  
in the above fireSessionCreated method. 

"not related directly" means that
whether sessions put managedSessions hashmap or not, fireSessionCreated and
sessionCreated are called.

Then destroying callback logic and managedSessions hashmap should be also
not related directly.
but MINA has the direct relation with managedSessions hashmap 

"direct relation" means that
as the below code shows, if sessions is not found in managedSessions hashmap
then
fireSessionClosed and sessionDestroyed method are not called. 


in the code  "public void fireSessionDestroyed(IoSession session)" method 
of IoServiceListenerSupport.java 

	...

  // Get the session set.
    Set<IoSession> sessions = managedSessions.get(serviceAddress);   <---
IoServiceListenerSupport.java:192 line
		// Ignore if unknown.
		if (sessions == null) {  <---
		    return;
		}
		
		 sessions.remove(session);

    boolean lastSession = false;

    // Try to remove the remaining empty session set after removal.
    if (sessions.isEmpty()) {
        lastSession = managedSessions.remove(serviceAddress, sessions);   
<====
    }

    // Fire session events.
    session.getFilterChain().fireSessionClosed(session);
    
    // Fire listener events.
        try {
            for (IoServiceListener listener : listeners) {
                listener.sessionDestroyed(session);
            }
        } 

	...

My inference is that	
When heavy thread competition (and other session parameters but with the
same serviceAddress),  
the one thread can remove the other using sessions from managedSessions
hashmap. 

 1) the one thread    : sessions.isEmpty() succeeded in fireSessionDestroyed
 2) the other thread  : called fireSessionCreated
 3) the other thread  : sessions.add(session) suceeded in fireSessionCreated
 4) the one thread    : managedSessions.remove(serviceAddress, sessions);    
(above "<===" line)
 5) the other thread  : called fireSessionDestroyed
 6) the other thread  : managedSessions.get(serviceAddress); fail                  
(above "<---" line)
 
other thread fails to get sessions from managedSessions hashmap with the
serviceAddress .
then fireSessionClosed propagation stops. (but should it continue!)
	

It is just my inference. 
and it's possible for me to misunderstand the MINA logic.


Thank you again.


J. H. Cha



Mike Heath-4 wrote:
> 
> It's possible that the server never received the TCP FIN packet and
> therefor did not know that it should close the IoSession.  There are
> situations where a host has no way of knowing that the TCP socket has
> closed unless it tries to write data to the socket and that write fails.
>  This is not a limitation of MINA.  This is just the way socket I/O works.
> 
> If this is the cause of your problem, you have a couple of options.  You
> could use some type of watch dog mechanisms that sends data at a regular
> interval.  If the connection is no longer valid, the write will fail and
> the socket will get closed thereby triggering a sessionClosed event.  Or
> you could use the MINA read idle notification.  This way after a certain
> amount of time has passed and your server hasn't received any data for a
> particular IoSession, you can manually close the IoSession.
> 
> If your problem still persists after implementing one of these
> solutions, please let us know.
> 
> -Mike
> 
> jhcha wrote:
>> Dear Mina developers.
>> I'm trying to use mina 1.1.5 to develop the very big stress server.
>> I test my server with jmeter like some echo test now.
>> 
>> test environment
>> 
>> * jmeter :
>> echo message size : 1024 bytes
>> jmeter threads : 100
>> loop count : 1000
>> sampler : TCP Sampler
>> reuse connection : false
>> 
>> * server : HP-UX 2 cores
>>          java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
>> IA64, mixed mode) 
>>          my server(mina)
>>          
>> * client : pc (dual core).
>>          jmeter 
>> 
>> when I tested the stress, I found sessionCreated called counts are a
>> little
>> bigger than sessionClosed called counts.
>> so, I tried to find out why that mismatch occured during several days.
>> and I found the followings :
>> 
>> 
>> org.apache.mina.common.support.IoServiceListenerSupport
>> {
>> 
>> public void fireSessionCreated(IoSession session) {
>> 		SocketAddress serviceAddress = session.getServiceAddress();
>> 
>> 		// Get the session set.
>> 		Set<IoSession> s = new IdentityHashSet<IoSession>();
>> 		Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
>> Collections
>> 				.synchronizedSet(s));
>> 		boolean firstSession;
>> 
>> 		if (null == sessions) {
>> 			sessions = s;
>> 			firstSession = true;
>> 		} else {
>> 			firstSession = false;
>> 		}
>> 
>> 		// If already registered, ignore.
>> 		if (!sessions.add(session)) {
>> 			return;
>> 		}
>> 
>> 		// If the first connector session, fire a virtual service activation
>> 		// event.
>> 		if (session.getService() instanceof IoConnector && firstSession) {
>> 			fireServiceActivated(session.getService(),
>> session.getServiceAddress(),
>> session
>> 					.getHandler(), session.getServiceConfig());
>> 		}
>> 
>> 		// Fire session events.
>> 		session.getFilterChain().fireSessionCreated(session);  <-- called
>> correctly
>> 		session.getFilterChain().fireSessionOpened(session);  <-- called
>> correctly
>> 
>> 		// Fire listener events.
>> 		for (IoServiceListener listener : listeners) {
>> 			listener.sessionCreated(session);  <-- called correctly
>> 		}
>> 	}
>> 
>> 
>> public void fireSessionDestroyed(IoSession session) {
>> 		boolean lastSession = false;
>> 		SocketAddress serviceAddress = session.getServiceAddress();
>> 
>> 		// Get the session set.
>> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
>> 		// Ignore if unknown.
>> 		if (sessions == null) {
>> 			// logger.error("fireSessionDestroyed() : sessions == null < " +
>> session);
>> 			return;  <-- occured sometimes !!!!!!!!!!!!!!!!!!
>> 		} 
>> 
>>         sessions.remove(session);
>> 
>> 		// Try to remove the remaining empty session set after removal.
>> 		if (sessions.isEmpty()) {
>> 			lastSession = managedSessions.remove(serviceAddress, sessions);
>> 		}
>> 
>> 		// Fire session events.
>> 		session.getFilterChain().fireSessionClosed(session); <-- skipped
>> sometimes
>> 
>> 		// Fire listener events.
>> 		try {
>> 			for (IoServiceListener listener : listeners) {
>> 				listener.sessionDestroyed(session); <-- skipped sometimes
>> 			}
>> 		} finally {
>> 			// Fire a virtual service deactivation event for the last session of
>> 			// the connector.
>> 			// TODO double-check that this is *STILL* the last session. May not
>> 			// be the case
>> 			if (session.getService() instanceof IoConnector && lastSession) {
>> 				fireServiceDeactivated(session.getService(),
>> session.getServiceAddress(), session
>> 						.getHandler(), session.getServiceConfig()); <-- skipped sometimes
>> 			}
>> 		}
>> 	}
>> 
>> the above fireSessionDestroyed method called as the same number as
>> fireSessionCreated method.
>> 
>> but the mysterious event is that the sessions is not gotten from
>> managedSessions in fireSessionDestroyed,
>> i.e. SocketAddress key is not found in managedSessions.
>> then the remaining logic was skipped.
>> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
>> 
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
>> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
>> 0.0.0.0/0.0.0.0:30000)
>> 
>> I think the reason that the 100 client threads tries to proceed "conect,
>> send, receive, close",
>> different sessions has the same client SocketAddress at times in server,
>> and the origin of that problem is client's stress environment.
>> 
>> It is possible the following senario : 
>> 
>> I/O processor thread     I/O processor thread      client thread  client
>> thread
>> one                      two                       one            two
>> --------------------------------------------------------------------------------
>>                                                    1) connect(the same
>> client SocketAddress)
>> 2)fireSessionCreated
>> 3)putIfAbsent
>>                                                    4)disconnect
>>                                                                  
>> 5)connect(the same client SocketAddress)
>>                           6)fireSessionCreated
>>                           7)putIfAbsent
>> 8)fireSessionDestroyed
>> 9)remove
>>                                                                  
>> 10)disconnect
>>                           11)fireSessionDestroyed
>>                           12)get fail
>> 
>> 
>> 
>> I changed the mina source IoServiceListenerSupport.java
>> 
>> org.apache.mina.common.support.IoServiceListenerSupport
>> ...
>> 
>> public void fireSessionDestroyed(IoSession session) {
>> 		boolean lastSession = false;
>> 		SocketAddress serviceAddress = session.getServiceAddress();
>> 
>> 		// Get the session set.
>> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
>> 		// Ignore if unknown.
>> 		if (sessions == null) {
>> 			// return;  <-- delete
>> 		} else {       // <-- add
>> 
>> 			sessions.remove(session);
>> 
>> 			// Try to remove the remaining empty session set after removal.
>> 			if (sessions.isEmpty()) {
>> 				lastSession = managedSessions.remove(serviceAddress, sessions);
>> 			}
>> 		}              // <-- add
>> 		....
>> 
>> then the Create, Destroy call mismatch was cleared.
>> but I don't know this is correct solution.
>> 
>> Would you tell me this is correct or not?
>> 
>> I think this situation is very rare but... 
>> I hope the mina is the best network framework 
>> and this situation shoud be also considered.
>> 
>> 
>> Thank you
>> 
>> J. H. Cha
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/sessionClosed-is-not-called-sometimes-in-heavy-stress-test-contions.-tp15265671s16868p15350882.html
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.


Mime
View raw message