thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Slee <ms...@facebook.com>
Subject RE: Short question re. threading with a Python server
Date Fri, 06 Aug 2010 17:58:25 GMT
Gotcha. Slightly weird that threading.local would fix that. It sounds like what you may have
potentially just done is create N MySQL clients, one-per each thread (resulting in each thread
just blocking on its own query, rather than each thread blocking on getting access to that
same MySQL connection when someone else is using it).

Out of curiosity, does "causing error responses for other clients" mean causing timeouts?
Or was there actually some kind of error being generated from the internals?

-----Original Message-----
From: Phillip B Oldham [mailto:phillip.oldham@gmail.com] 
Sent: Friday, August 06, 2010 6:08 AM
To: thrift-user@incubator.apache.org
Subject: Re: Short question re. threading with a Python server

Seems my error was due to threading, specifically with the MySQL
client. I had to alter the handler class so that it extended
`threading.local` otherwise a MySQL query would block the whole
process, causing error responses for other clients.

On Tue, Aug 3, 2010 at 2:41 PM, Phillip B Oldham
<phillip.oldham@gmail.com> wrote:
> Thanks Mark, that's really helpful. We're seeing a very odd bottleneck
> and thought it could've been down to issues with threading in our
> Python server. We've no shared state except what is persisted through
> memcached and MySQL, so it mustn't be that. I'll continue to dig
> around.
>
> Cheers!
>
> On Mon, Aug 2, 2010 at 8:57 PM, Mark Slee <mslee@facebook.com> wrote:
>> Generally, no. Python's threading.local is a specific way of offering thread-local
storage.
>>
>> You should NOT use threading.local if you're trying to safely share data between
threads. That's not what it does -- rather it actually exposes a *different* copy of the data
to each thread.
>>
>> So, you should use it if you actually want N copies of some data (where N is the
number of threads), but you want a simple interface to refer to them all by the same name
(with the clear understanding that you are actually accessing N different pieces of data).
>>
>> There are certainly situations in which this can be useful in a threaded Thrift server,
but it's by no means a general rule.
>>
>> -----Original Message-----
>> From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
>> Sent: Friday, July 30, 2010 2:39 AM
>> To: thrift-user
>> Subject: Short question re. threading with a Python server
>>
>> Should Python classes that are run via the
>> TThreaded/TThreadPool/TNonBlocking server inherit from
>> `threading.local`, in order to be thread safe?
>>
>> --
>> Phillip B Oldham
>> phillip.oldham@gmail.com
>> +44 (0) 7525 01 09 01
>>
>
>
>
> --
> Phillip B Oldham
> phillip.oldham@gmail.com
> +44 (0) 7525 01 09 01
>



-- 
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01
Mime
View raw message