mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Heath <mhe...@apache.org>
Subject Re: sessionClosed is not called sometimes in heavy stress test contions.
Date Thu, 07 Feb 2008 03:45:10 GMT
It's possible that the server never received the TCP FIN packet and
therefor did not know that it should close the IoSession.  There are
situations where a host has no way of knowing that the TCP socket has
closed unless it tries to write data to the socket and that write fails.
 This is not a limitation of MINA.  This is just the way socket I/O works.

If this is the cause of your problem, you have a couple of options.  You
could use some type of watch dog mechanisms that sends data at a regular
interval.  If the connection is no longer valid, the write will fail and
the socket will get closed thereby triggering a sessionClosed event.  Or
you could use the MINA read idle notification.  This way after a certain
amount of time has passed and your server hasn't received any data for a
particular IoSession, you can manually close the IoSession.

If your problem still persists after implementing one of these
solutions, please let us know.

-Mike

jhcha wrote:
> Dear Mina developers.
> I'm trying to use mina 1.1.5 to develop the very big stress server.
> I test my server with jmeter like some echo test now.
> 
> test environment
> 
> * jmeter :
> echo message size : 1024 bytes
> jmeter threads : 100
> loop count : 1000
> sampler : TCP Sampler
> reuse connection : false
> 
> * server : HP-UX 2 cores
>          java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> IA64, mixed mode) 
>          my server(mina)
>          
> * client : pc (dual core).
>          jmeter 
> 
> when I tested the stress, I found sessionCreated called counts are a little
> bigger than sessionClosed called counts.
> so, I tried to find out why that mismatch occured during several days.
> and I found the followings :
> 
> 
> org.apache.mina.common.support.IoServiceListenerSupport
> {
> 
> public void fireSessionCreated(IoSession session) {
> 		SocketAddress serviceAddress = session.getServiceAddress();
> 
> 		// Get the session set.
> 		Set<IoSession> s = new IdentityHashSet<IoSession>();
> 		Set<IoSession> sessions = managedSessions.putIfAbsent(serviceAddress,
> Collections
> 				.synchronizedSet(s));
> 		boolean firstSession;
> 
> 		if (null == sessions) {
> 			sessions = s;
> 			firstSession = true;
> 		} else {
> 			firstSession = false;
> 		}
> 
> 		// If already registered, ignore.
> 		if (!sessions.add(session)) {
> 			return;
> 		}
> 
> 		// If the first connector session, fire a virtual service activation
> 		// event.
> 		if (session.getService() instanceof IoConnector && firstSession) {
> 			fireServiceActivated(session.getService(), session.getServiceAddress(),
> session
> 					.getHandler(), session.getServiceConfig());
> 		}
> 
> 		// Fire session events.
> 		session.getFilterChain().fireSessionCreated(session);  <-- called
> correctly
> 		session.getFilterChain().fireSessionOpened(session);  <-- called correctly
> 
> 		// Fire listener events.
> 		for (IoServiceListener listener : listeners) {
> 			listener.sessionCreated(session);  <-- called correctly
> 		}
> 	}
> 
> 
> public void fireSessionDestroyed(IoSession session) {
> 		boolean lastSession = false;
> 		SocketAddress serviceAddress = session.getServiceAddress();
> 
> 		// Get the session set.
> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
> 		// Ignore if unknown.
> 		if (sessions == null) {
> 			// logger.error("fireSessionDestroyed() : sessions == null < " +
> session);
> 			return;  <-- occured sometimes !!!!!!!!!!!!!!!!!!
> 		} 
> 
>         sessions.remove(session);
> 
> 		// Try to remove the remaining empty session set after removal.
> 		if (sessions.isEmpty()) {
> 			lastSession = managedSessions.remove(serviceAddress, sessions);
> 		}
> 
> 		// Fire session events.
> 		session.getFilterChain().fireSessionClosed(session); <-- skipped sometimes
> 
> 		// Fire listener events.
> 		try {
> 			for (IoServiceListener listener : listeners) {
> 				listener.sessionDestroyed(session); <-- skipped sometimes
> 			}
> 		} finally {
> 			// Fire a virtual service deactivation event for the last session of
> 			// the connector.
> 			// TODO double-check that this is *STILL* the last session. May not
> 			// be the case
> 			if (session.getService() instanceof IoConnector && lastSession) {
> 				fireServiceDeactivated(session.getService(),
> session.getServiceAddress(), session
> 						.getHandler(), session.getServiceConfig()); <-- skipped sometimes
> 			}
> 		}
> 	}
> 
> the above fireSessionDestroyed method called as the same number as
> fireSessionCreated method.
> 
> but the mysterious event is that the sessions is not gotten from
> managedSessions in fireSessionDestroyed,
> i.e. SocketAddress key is not found in managedSessions.
> then the remaining logic was skipped.
> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
> 
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> 0.0.0.0/0.0.0.0:30000)
> 
> I think the reason that the 100 client threads tries to proceed "conect,
> send, receive, close",
> different sessions has the same client SocketAddress at times in server,
> and the origin of that problem is client's stress environment.
> 
> It is possible the following senario : 
> 
> I/O processor thread     I/O processor thread      client thread  client
> thread
> one                      two                       one            two
> --------------------------------------------------------------------------------
>                                                    1) connect(the same
> client SocketAddress)
> 2)fireSessionCreated
> 3)putIfAbsent
>                                                    4)disconnect
>                                                                  
> 5)connect(the same client SocketAddress)
>                           6)fireSessionCreated
>                           7)putIfAbsent
> 8)fireSessionDestroyed
> 9)remove
>                                                                  
> 10)disconnect
>                           11)fireSessionDestroyed
>                           12)get fail
> 
> 
> 
> I changed the mina source IoServiceListenerSupport.java
> 
> org.apache.mina.common.support.IoServiceListenerSupport
> ...
> 
> public void fireSessionDestroyed(IoSession session) {
> 		boolean lastSession = false;
> 		SocketAddress serviceAddress = session.getServiceAddress();
> 
> 		// Get the session set.
> 		Set<IoSession> sessions = managedSessions.get(serviceAddress);
> 		// Ignore if unknown.
> 		if (sessions == null) {
> 			// return;  <-- delete
> 		} else {       // <-- add
> 
> 			sessions.remove(session);
> 
> 			// Try to remove the remaining empty session set after removal.
> 			if (sessions.isEmpty()) {
> 				lastSession = managedSessions.remove(serviceAddress, sessions);
> 			}
> 		}              // <-- add
> 		....
> 
> then the Create, Destroy call mismatch was cleared.
> but I don't know this is correct solution.
> 
> Would you tell me this is correct or not?
> 
> I think this situation is very rare but... 
> I hope the mina is the best network framework 
> and this situation shoud be also considered.
> 
> 
> Thank you
> 
> J. H. Cha


Mime
View raw message