thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dvir Volk <dvir...@gmail.com>
Subject Re: best solution for java client and python server maintaining high concurrents
Date Tue, 25 Jan 2011 11:01:10 GMT
No prob,
It might be worth not using a proxy handler, but rather deriving TServer and
creating a TProcessPoolServer

On Tue, Jan 25, 2011 at 12:45 PM, dean forever
<deanforwever2010@gmail.com>wrote:

> ok thank you for your so kind reply
>  a proxy handler  is a good idea
> I'am so busy for my other works,I didnot make a program now
> I will try your way as soon as possible .
> Thank you very much
>
> 2011/1/25 Dvir Volk <dvirsky@gmail.com>
>
> > if you want to get multiple processes working with one port per machine,
> > you
> > also have 2 other options:
> > 1. there's a forking server in pything that forks a worker child for each
> > request.  that's a bit resource consuming, forking per each request, so
> I'm
> > not using it.
> >
> > 2. you can implement a proxy handler for all thrift requests (using
> python
> > descriptors), that pushes the request on a multiprocessing queue.
> > then you have a pool of subprocess workers that fetches items from the
> > pool,
> > processes them and returns the response.
> > this will also utilize all the CPUs of your machine without the added
> > complexity of multiple ports.
> >
> > On Tue, Jan 25, 2011 at 4:22 AM, dean forever <
> deanforwever2010@gmail.com
> > >wrote:
> >
> > > different ports didnot work for my situation.
> > > I putted python server code in different server machines ,and it
> > performed
> > > a
> > > little better.
> > > Thank you very much for your kind advice .
> > > I will try your suggestion!
> > >
> > > 2011/1/24 Dvir Volk <dvirsky@gmail.com>
> > >
> > > > I'm sure it's completely solvable, the only question is why you want
> > > python
> > > > in the first place.
> > > >
> > > > Here's a little trick I did that boosted my own performance a lot:
> > > > my setup is a PHP API server that handles REST requests, and is a
> > thrift
> > > > client. It passes the parameters to thrift, serializes the response
> > back
> > > to
> > > > JSON, and sends it to the user.
> > > >
> > > > The main problem was the python GIL that bounds all threads to a
> single
> > > CPU
> > > > Core
> > > > So I did a very simple thing: since my server is mostly stateless,
> what
> > > it
> > > > does is use the python multiprocessing module to spawn 4-6 thread
> pool
> > > > servers on consecutive ports.
> > > > the php client selects a random port, and I'm utilizing the CPU much
> > > better
> > > > and Increased the RPS by a lot.
> > > >
> > > > Another thing with python is, that you can optimize CPU heavy parts
> of
> > > your
> > > > program and offload them to C code, either by loading a DLL,
> > > > by building a binary module (like the TBinaryProtocolAccelerated), or
> > by
> > > > using some C precompiler like Cython.
> > > >
> > > > I wonder how difficult it would be to wrap the whole thrift server as
> a
> > > > python module and have only the handler written in python.
> > > >
> > > > On Mon, Jan 24, 2011 at 5:03 PM, dean forever <
> > > deanforwever2010@gmail.com
> > > > >wrote:
> > > >
> > > > > thanks , Dvir!
> > > > > finally ,by testing the server, I found the server didnot perform
> > > > well.Only
> > > > > 10% CPU usage,comparing to 100%  CPU usage of Java Tomcat
> > > > > I think i need to find a better server program instead of the
> thrift
> > > > > default
> > > > > server.
> > > > >
> > > > >
> > > > > 2011/1/21 Dvir Volk <dvirsky@gmail.com>
> > > > >
> > > > > > I don't know about the compact transport, but just a python
> server
> > > tip
> > > > -
> > > > > > use TBinaryProtocolAccelerated, handles serialization with C
code
> > > which
> > > > > > makes it faster.
> > > > > > Also, try not to have more jmeter threads than max threads on
the
> > > > server.
> > > > > > I'm currently testing thrift with jmeter via a PHP proxy rest
> > server,
> > > > but
> > > > > > these tips should apply when testing directly with java, I guess.
> > > > > >
> > > > > > On Fri, Jan 21, 2011 at 7:23 AM, dean forever <
> > > > > deanforwever2010@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > When I use the default example settings for java client
and
> > python
> > > > > > server(
> > > > > > > like tutorial shows) to do Jmeter Tests,lots of errors
such as
> > > > > > > "TTransportException: Socket already connected"  "'Connection
> > reset
> > > > by
> > > > > > > peer"
> > > > > > > "Broken pipe"
> > > > > > > happened
> > > > > > >
> > > > > > > when i try TFramedTransport TCompactProtocol in java, it's
all
> > > right
> > > > > > >
> > > > > > > in python  when I use TFramedTransport ,it shows that no
listen
> > and
> > > > > > accept
> > > > > > > method in it.When I added them by
> > > > > > > def listen(self):
> > > > > > > return self.__trans.listen()
> > > > > > >
> > > > > > > the python server can run normally
> > > > > > >
> > > > > > >
> > > > > > > but when I start Java Client to request, the python server
show
> > > lots
> > > > of
> > > > > > > NameError in TCompactProtocol.py
> > > > > > >
> > > > > > > Anyone has a better solution  to this combine???Please
help me
> > find
> > > a
> > > > > > best
> > > > > > > solution for java Client and python Server  maintaining
> > > > > > >  high concurrents
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message