thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Napolitano, Diane" <dnapolit...@ets.org>
Subject RE: Question about Multithreaded Thrift Servers
Date Mon, 08 Apr 2013 17:52:59 GMT
Your comment about the Java implementation of TThreadPoolServer got me thinking.  Turns out
this class will use pretty much any Java thread pool/executor configuration you throw at it,
so TThreadPoolServer itself isn't a blocking queue, it just uses a blocking queue by default.
 Because of that, if I queue up 1000 jobs, 10 of them run and the other 990 fail and typically
cause the server to crash.

So, at this point in the code for my server:
TThreadPoolServer.Args args = new TThreadPoolServer.Args(serverTransport);
args.maxWorkerThreads(10);
args.processor(processor);

I added:
args.executorService(new ScheduledThreadPoolExecutor(10));

Then created the server object:
TServer server = new TThreadPoolServer(args);

Now Java is smart enough to queue (schedule) the other 990 jobs, so I no longer have this
issue. :)  (Also setting -Xmx2048m and -XX:-UseGCOverheadLimit helped too.)

Thanks for your help!

Best,
Diane
________________________________________
From: Ben Craig [ben.craig@ni.com]
Sent: Friday, April 05, 2013 4:34 PM
To: user@thrift.apache.org
Subject: RE: Question about Multithreaded Thrift Servers

Are you running out of ports / file descriptors?  I have attempted to
write performance tests in the past that stressed connect / disconnect,
and I have routinely run into the problem of exhausting ports.  This isn't
because I failed to call close, but because "closing" a socket doesn't
immediately free up the resources in the TCP/IP stack.  It typically takes
a few minutes for those resources to be freed.



From:   "Napolitano, Diane" <dnapolitano@ets.org>
To:     "user@thrift.apache.org" <user@thrift.apache.org>,
Date:   04/05/2013 03:32 PM
Subject:        RE: Question about Multithreaded Thrift Servers



Hmm, interesting.  These requests are actually clients, several thousands
of them actually :) , calling methods on the server.  We have a cluster
setup here, and I have one cluster job that is the server, and several
thousand cluster jobs that are each one client, or one request to the
server.

So, hmm.

Thanks,
Diane
________________________________________
From: Ben Craig [ben.craig@ni.com]
Sent: Friday, April 05, 2013 4:20 PM
To: user@thrift.apache.org
Subject: RE: Question about Multithreaded Thrift Servers

You only mention requests, so I'm not sure if these are all going over the
same connection or not.  One other thing that gets strange is that naming
across the different languages isn't consistent.  I am most comfortable
with the C++ implementation, and so that's the behavior that I described.
I vaguely recall that the Java implementation of TThreadPoolServer is more
similar to C++'s NonBlockingServer, and I'm not sure what that does.



From:   "Napolitano, Diane" <dnapolitano@ets.org>
To:     "user@thrift.apache.org" <user@thrift.apache.org>,
Date:   04/05/2013 03:02 PM
Subject:        RE: Question about Multithreaded Thrift Servers



Really?  So if threads 0 thru 9 are running, and 0-8 finish, the remaining
requests in the queue (is there even a queue?) won't make use of 0-8?

It's fine if they sit there waiting for 9 to finish, but in this case,
even after 9 finishes, additional requests aren't being executed.  If > 10
requests are sent to it, the server throws an exception, and only executes
the first 10.  Also, sometimes it isn't the 10th request that is the
slowest, it's some other one (the 2nd or whatever).

Sorry, I'm confused because your answer makes it sound like what I'm
seeing shouldn't be happening. :/

Thanks,
Diane
________________________________________
From: Ben Craig [ben.craig@ni.com]
Sent: Friday, April 05, 2013 3:55 PM
To: user@thrift.apache.org
Subject: Re: Question about Multithreaded Thrift Servers

The threaded server uses one thread per connection.  It does not get a
request, and have an available thread process the request.  This means
that if you send 20 requests, and the 10th request takes a long time to
process, the last ten requests will be stalled.



From:   "Napolitano, Diane" <dnapolitano@ets.org>
To:     "user@thrift.apache.org" <user@thrift.apache.org>,
Date:   04/05/2013 02:44 PM
Subject:        Question about Multithreaded Thrift Servers



Hello, I have a Thrift server (just "NameOfServer" here) written in Java
which is initialized with the following inside of an inner class called
ServerThread:

TThreadPoolServer.Args args = new TThreadPoolServer.Args(serverTransport);
args.maxWorkerThreads(10);
args.processor(processor);
TServer server = new TThreadPoolServer(args);
server.serve();

Then NameOfServer has two objects:
public static NameOfServerHandler handler;
public static NameOfServer.Processor processor;

And then a ServerThread object is created in my server's main:
handler = new NameOfServerHandler();
processor = new NameOfServer.Processor(handler);
Runnable r = new ServerThread(processor, portNum);
new Thread(r).start();

My question is: am I doing this right?  Because when I send more than ten
requests to the server, it throws an exception and dies, and all requests
beyond the first 10 are stalled.  Unless I'm misunderstanding threads,
Java, and everything else, aren't these additional requests supposed to be
queued and then served when one of the 10 threads becomes available?

If you need more code/context than this, definitely let me know.

Thanks,
Diane







Mime
View raw message