mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 이희승 "(Trustin Lee)" <trus...@gmail.com>
Subject Re: sessionClosed is not called sometimes in heavy stress test contions.
Date Wed, 13 Feb 2008 04:51:02 GMT
JH,

Thank you so much for the detailed analysis.  Let me take a look into
this issue when I get back home.

2008-02-07 (목), 23:59 -0800, jhcha 쓰시길:
> Thank Mike for replying my question. 
> 
> I read your answers but...
> 
> I think MINA already knew the session was being closed in my questioned
> situation.
> because my server session always close after sending the echo data and 
> I debugged MINA source
> org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method 
> always call doRemove() at (SocketIoProcessor.java:489 line).
> then doRemove method always call
> session.getServiceListeners().fireSessionDestroyed(session); 
> at (SocketIoProcessor.java:188 line).
> 
>    try {
>         key.cancel();
>         ch.close();
>     } catch (IOException e) {
>         session.getFilterChain().fireExceptionCaught(session, e);
>     } finally {
>         releaseWriteBuffers(session);
>         session.getServiceListeners().fireSessionDestroyed(session);   <---
> SocketIoProcessor.java:188 line
>     }
>             
>             
> and the above session parameter of the fireSessionDestroyed method was
> always not null.
> 
> the orgin of the problem is the sessions having serviceAddress is not in
> managedSessions hashmap 
> in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)
> 
>     // Get the session set.
>     Set<IoSession> sessions = managedSessions.get(serviceAddress);   <---
> IoServiceListenerSupport.java:192 line
>     // Ignore if unknown.
>     if (sessions == null) {
> 	return;
>     }
>         
> The fireSessionDestroyed method call propagation stopped only sometimes
> Because sessions is not gotten from managedSessions at
> (IoServiceListenerSupport.java:192 line).
> 
> I used setIdleTimeout with read mode, but 
> if sessionClosed method is not called, sessionIdle or sessionClosed method
> will never be called after that.
> 
> My question is that the filter chain fireSessionClosed method and listener'
> sessionDestroyed method 
> Should not know if the destroying session be not in the managedSessions
> hashmap ?
> 
>     // Fire session events.
>     session.getFilterChain().fireSessionClosed(session);  <---
> IoServiceListenerSupport.java:208 line
> 
>     // Fire listener events.
>     try {
>         for (IoServiceListener listener : listeners) {
>             listener.sessionDestroyed(session);           <---
>         }
>     }
>         
>         
> If fireSessionDestroyed method were called with the parameter the closing
> session
> but not in managedSessions hashmap, Should filter chain fireSessionClosed
> method and 
> listener' sessionDestroyed method not called ? 
> 
> 
> in the following MINA source below the IoServiceListenerSupport.java:147
> line.
> 
> 
>   public void fireSessionCreated(IoSession session) {
>         SocketAddress serviceAddress = session.getServiceAddress();
> 
>         // Get the session set.
>         Set<IoSession> s = new IdentityHashSet<IoSession>();
>         Set<IoSession> sessions =
> managedSessions.putIfAbsent(serviceAddress,
>                 Collections.synchronizedSet(s));
>         boolean firstSession;
> 
>         if (null == sessions) {
>             sessions = s;
>             firstSession = true;
>         } else {
>             firstSession = false;
>         }
> 
>         // If already registered, ignore.
>         if (!sessions.add(session)) { <---
>             return;
>         }
> 
>         // If the first connector session, fire a virtual service activation
> event.
>         if (session.getService() instanceof IoConnector && firstSession) {
>             fireServiceActivated(session.getService(), session
>                     .getServiceAddress(), session.getHandler(), session
>                     .getServiceConfig());
>         }
> 
>         // Fire session events.
>         session.getFilterChain().fireSessionCreated(session);
>         session.getFilterChain().fireSessionOpened(session);
> 
>         // Fire listener events.
>         for (IoServiceListener listener : listeners) {
>             listener.sessionCreated(session);
>         }
>     }
>     
> 
> I think the code "if (!sessions.add(session))" is always true,
> So filter chain fireSessionCreated method and listener' sessionCreated
> method is 
> always called in spite of the heavy unstable tcp stress condition.
> 
> I think 
> Creaing callback method and destroying callback method should have the same
> calling count. 
> 
> But creating callback logic and managedSessions hashmap are not related
> directly  
> in the above fireSessionCreated method. 
> 
> "not related directly" means that
> whether sessions put managedSessions hashmap or not, fireSessionCreated and
> sessionCreated are called.
> 
> Then destroying callback logic and managedSessions hashmap should be also
> not related directly.
> but MINA has the direct relation with managedSessions hashmap 
> 
> "direct relation" means that
> as the below code shows, if sessions is not found in managedSessions hashmap
> then
> fireSessionClosed and sessionDestroyed method are not called. 
> 
> 
> in the code  "public void fireSessionDestroyed(IoSession session)" method 
> of IoServiceListenerSupport.java 
> 
> 	...
> 
>   // Get the session set.
>     Set<IoSession> sessions = managedSessions.get(serviceAddress);   <---
> IoServiceListenerSupport.java:192 line
> 		// Ignore if unknown.
> 		if (sessions == null) {  <---
> 		    return;
> 		}
> 		
> 		 sessions.remove(session);
> 
>     boolean lastSession = false;
> 
>     // Try to remove the remaining empty session set after removal.
>     if (sessions.isEmpty()) {
>         lastSession = managedSessions.remove(serviceAddress, sessions);   
> <====
>     }
> 
>     // Fire session events.
>     session.getFilterChain().fireSessionClosed(session);
>     
>     // Fire listener events.
>         try {
>             for (IoServiceListener listener : listeners) {
>                 listener.sessionDestroyed(session);
>             }
>         } 
> 
> 	...
> 
> My inference is that	
> When heavy thread competition (and other session parameters but with the
> same serviceAddress),  
> the one thread can remove the other using sessions from managedSessions
> hashmap. 
> 
>  1) the one thread    : sessions.isEmpty() succeeded in fireSessionDestroyed
>  2) the other thread  : called fireSessionCreated
>  3) the other thread  : sessions.add(session) suceeded in fireSessionCreated
>  4) the one thread    : managedSessions.remove(serviceAddress, sessions);    
> (above "<===" line)
>  5) the other thread  : called fireSessionDestroyed
>  6) the other thread  : managedSessions.get(serviceAddress); fail                  
> (above "<---" line)
>  
> other thread fails to get sessions from managedSessions hashmap with the
> serviceAddress .
> then fireSessionClosed propagation stops. (but should it continue!)
> 	
> 
> It is just my inference. 
> and it's possible for me to misunderstand the MINA logic.
> 
> 
> Thank you again.
> 
> 
> J. H. Cha
> 
> 
> 
> Mike Heath-4 wrote:
> > 
> > It's possible that the server never received the TCP FIN packet and
> > therefor did not know that it should close the IoSession.  There are
> > situations where a host has no way of knowing that the TCP socket has
> > closed unless it tries to write data to the socket and that write fails.
> >  This is not a limitation of MINA.  This is just the way socket I/O works.
> > 
> > If this is the cause of your problem, you have a couple of options.  You
> > could use some type of watch dog mechanisms that sends data at a regular
> > interval.  If the connection is no longer valid, the write will fail and
> > the socket will get closed thereby triggering a sessionClosed event.  Or
> > you could use the MINA read idle notification.  This way after a certain
> > amount of time has passed and your server hasn't received any data for a
> > particular IoSession, you can manually close the IoSession.
> > 
> > If your problem still persists after implementing one of these
> > solutions, please let us know.
> > 
> > -Mike
> > 
> > jhcha wrote:
> >> Dear Mina developers.
> >> I'm trying to use mina 1.1.5 to develop the very big stress server.
> >> I test my server with jmeter like some echo test now.
> >> 
> >> test environment
> >> 
> >> * jmeter :
> >> echo message size : 1024 bytes
> >> jmeter threads : 100
> >> loop count : 1000
> >> sampler : TCP Sampler
> >> reuse connection : false
> >> 
> >> * server : HP-UX 2 cores
> >>          java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> >> IA64, mixed mode) 
> >>          my server(mina)
> >>          
> >> * client : pc (dual core).
> >>          jmeter 
> >> 
> >> when I tested the stress, I found sessionCreated called counts are a
> >> little
> >> bigger than sessionClosed called counts.
> >> so, I tried to find out why that mismatch occured during several days.
> >> and I found the followings :
> >> 
> >> 
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> {
> >> 
> >> public void fireSessionCreated(IoSession session) {
> >> 		SocketAddress serviceAddress = session.getServiceAddress();
> >> 
> >> 		// Get the session set.
> >> 		Set<IoSession> s = new IdentityHashSet<IoSession>();
> >> 		Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
> >> Collections
> >> 				.synchronizedSet(s));
> >> 		boolean firstSession;
> >> 
> >> 		if (null == sessions) {
> >> 			sessions = s;
> >> 			firstSession = true;
> >> 		} else {
> >> 			firstSession = false;
> >> 		}
> >> 
> >> 		// If already registered, ignore.
> >> 		if (!sessions.add(session)) {
> >> 			return;
> >> 		}
> >> 
> >> 		// If the first connector session, fire a virtual service activation
> >> 		// event.
> >> 		if (session.getService() instanceof IoConnector && firstSession) {
> >> 			fireServiceActivated(session.getService(),
> >> session.getServiceAddress(),
> >> session
> >> 					.getHandler(), session.getServiceConfig());
> >> 		}
> >> 
> >> 		// Fire session events.
> >> 		session.getFilterChain().fireSessionCreated(session);  <-- called
> >> correctly
> >> 		session.getFilterChain().fireSessionOpened(session);  <-- called
> >> correctly
> >> 
> >> 		// Fire listener events.
> >> 		for (IoServiceListener listener : listeners) {
> >> 			listener.sessionCreated(session);  <-- called correctly
> >> 		}
> >> 	}
> >> 
> >> 
> >> public void fireSessionDestroyed(IoSession session) {
> >> 		boolean lastSession = false;
> >> 		SocketAddress serviceAddress = session.getServiceAddress();
> >> 
> >> 		// Get the session set.
> >> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> 		// Ignore if unknown.
> >> 		if (sessions == null) {
> >> 			// logger.error("fireSessionDestroyed() : sessions == null < " +
> >> session);
> >> 			return;  <-- occured sometimes !!!!!!!!!!!!!!!!!!
> >> 		} 
> >> 
> >>         sessions.remove(session);
> >> 
> >> 		// Try to remove the remaining empty session set after removal.
> >> 		if (sessions.isEmpty()) {
> >> 			lastSession = managedSessions.remove(serviceAddress, sessions);
> >> 		}
> >> 
> >> 		// Fire session events.
> >> 		session.getFilterChain().fireSessionClosed(session); <-- skipped
> >> sometimes
> >> 
> >> 		// Fire listener events.
> >> 		try {
> >> 			for (IoServiceListener listener : listeners) {
> >> 				listener.sessionDestroyed(session); <-- skipped sometimes
> >> 			}
> >> 		} finally {
> >> 			// Fire a virtual service deactivation event for the last session of
> >> 			// the connector.
> >> 			// TODO double-check that this is *STILL* the last session. May not
> >> 			// be the case
> >> 			if (session.getService() instanceof IoConnector && lastSession) {
> >> 				fireServiceDeactivated(session.getService(),
> >> session.getServiceAddress(), session
> >> 						.getHandler(), session.getServiceConfig()); <-- skipped sometimes
> >> 			}
> >> 		}
> >> 	}
> >> 
> >> the above fireSessionDestroyed method called as the same number as
> >> fireSessionCreated method.
> >> 
> >> but the mysterious event is that the sessions is not gotten from
> >> managedSessions in fireSessionDestroyed,
> >> i.e. SocketAddress key is not found in managedSessions.
> >> then the remaining logic was skipped.
> >> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
> >> 
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> 
> >> I think the reason that the 100 client threads tries to proceed "conect,
> >> send, receive, close",
> >> different sessions has the same client SocketAddress at times in server,
> >> and the origin of that problem is client's stress environment.
> >> 
> >> It is possible the following senario : 
> >> 
> >> I/O processor thread     I/O processor thread      client thread  client
> >> thread
> >> one                      two                       one            two
> >> --------------------------------------------------------------------------------
> >>                                                    1) connect(the same
> >> client SocketAddress)
> >> 2)fireSessionCreated
> >> 3)putIfAbsent
> >>                                                    4)disconnect
> >>                                                                  
> >> 5)connect(the same client SocketAddress)
> >>                           6)fireSessionCreated
> >>                           7)putIfAbsent
> >> 8)fireSessionDestroyed
> >> 9)remove
> >>                                                                  
> >> 10)disconnect
> >>                           11)fireSessionDestroyed
> >>                           12)get fail
> >> 
> >> 
> >> 
> >> I changed the mina source IoServiceListenerSupport.java
> >> 
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> ...
> >> 
> >> public void fireSessionDestroyed(IoSession session) {
> >> 		boolean lastSession = false;
> >> 		SocketAddress serviceAddress = session.getServiceAddress();
> >> 
> >> 		// Get the session set.
> >> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> 		// Ignore if unknown.
> >> 		if (sessions == null) {
> >> 			// return;  <-- delete
> >> 		} else {       // <-- add
> >> 
> >> 			sessions.remove(session);
> >> 
> >> 			// Try to remove the remaining empty session set after removal.
> >> 			if (sessions.isEmpty()) {
> >> 				lastSession = managedSessions.remove(serviceAddress, sessions);
> >> 			}
> >> 		}              // <-- add
> >> 		....
> >> 
> >> then the Create, Destroy call mismatch was cleared.
> >> but I don't know this is correct solution.
> >> 
> >> Would you tell me this is correct or not?
> >> 
> >> I think this situation is very rare but... 
> >> I hope the mina is the best network framework 
> >> and this situation shoud be also considered.
> >> 
> >> 
> >> Thank you
> >> 
> >> J. H. Cha
> > 
> > 
> > 
> 
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Mime
View raw message