thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Reiss <dre...@facebook.com>
Subject Re: PHP Client fails to read correct return values from the Java Service
Date Tue, 09 Feb 2010 16:15:48 GMT
After a read timeout, you have to throw the connection away and create a new one.

Utku Can Topçu wrote:
> Hello Again,
> 
> I'm re-volting the issue,
> 
> I've conducted several experiments and the problem does seem to continue.
> 
> First of all I tried using the TFramedTransport, at first it worked fine but
> this only lasted until several client timeouts occurred.
> 
> Let me explain the problem again,
> 
> Say, there are n consecutive php requests on the web server working on the
> same php-process, calling the service for the for a simple function like
> Item getItem(1:i64 requestedItemId)
> the getItem function connects to the DB and fetches and forms an object for
> generating the Item requested.
> The contract for the Item object result is, Item.id = requestedItemId
> 
> If I'm using a persistent connection, all requests running under the same
> php-process defined above use the same socket and buffer for service calls.
> 
> Now,
> at time t1 php client calls getItem(1:i64 id1) with the timeout 1000 ms.
> at time t1+1000 php client times out and exits
> at time t2 (t2>t1+1000) Java service returns the Item
> at time t3 (t3 is just after t2) php client calls getItem(1:i64 id2) with
> the timeout 1000 ms.
> at time t4 (t4 is just after t3) php client reads the Item instance which
> was the response for the call at "time t1", so what I get is Item.id = id1
> whereas it should have beed, Item.id = id2
> 
> If I'm not using persistent connections, such problem never happens, but
> this time I seriously face a connection overhead which fails eventually.
> 
> Can you suggest me scheme for working around this?
> 
> Best Regards,
> Utku
> 
> 
> On Sat, Jan 16, 2010 at 1:17 AM, Mark Slee <mslee@facebook.com> wrote:
> 
>> Yes, you're right. That's definitely possible, and it sounds like that's
>> probably what is happening here.
>>
>> I was referring to was the fact that persistent sockets are never shared
>> across separate apache processes (in the OS sense), so the pfsockopen() call
>> should only return a socket that's done being used by the last page request.
>>
>> I was wrongly assuming the last page request properly completed use of the
>> socket. If the client side experiences a read timeout, it'll close up making
>> it possible for a subsequent request to get that same socket later, and then
>> the server response will come back.
>>
>> The right fix for this is to build proper sequence-id validation into the
>> PHP client code. We actually do send a unique sequence id with each RPC
>> request that should be echoed back by the server, but currently we're not
>> tracking state in the client to match these.
>>
>> Cheers,
>> Mark
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, January 14, 2010 9:12 PM
>> To: thrift-user@incubator.apache.org
>> Subject: Re: PHP Client fails to read correct return values from the Java
>> Service
>>
>> On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <mslee@facebook.com> wrote:
>>
>>> Would need more info to debug this. Do you mind sharing more of your
>> server
>>> implementation code? The client code looks fine to me. I'm not really
>> clear
>>> on how you'd get the result of an old call when you're using a new socket
>>> each time, unless there is a bug with PHP's underlying persistent socket
>>> implementation leaving old data in the socket buffer (I've never seen
>> this
>>> before, and it seems like it would create major problems if this could
>> ever
>>> happen -- I would have assumed that pfsockopen() would always reset the
>> recv
>>> buffer).
>>>
>>>
>> Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
>> *think* the buffer's clear, then hand off the socket to the client, and
>> then
>> the packets arrive.
>>
>> -Todd
>>
>>
>>> Can you try running this without using PHP persistent sockets? See if it
>>> fixes the issue to open/close a new socket every time?
>>>
>>> -----Original Message-----
>>> From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
>>> Sent: Tuesday, January 05, 2010 12:02 AM
>>> To: thrift-user@incubator.apache.org
>>> Subject: PHP Client fails to read correct return values from the Java
>>> Service
>>>
>>> Dear Thrift users and Developers,
>>>
>>> I've written a simple service which has the following thrift definition:
>>>
>>> service CounterService {
>>>        string increment(1:i64 itemId),
>>>        string getCount(1:i64 itemId),
>>> }
>>>
>>> In our system, we're using PHP as the client and Java as the Server,
>>> Following the sample client and server stub examples included in the
>>> tutorial package, I haven't made any big differences in both the Client
>> and
>>> Server code (and of course never touched the auto-generated code);
>>>
>>> The Java Server makes use of the following simple stub:
>>>
>>> --Server.java--
>>> CounterHandler handler = new CounterHandler();
>>> CounterService.Processor processor = new
>> CounterService.Processor(handler);
>>> TServerTransport serverTransport;
>>> serverTransport = new TServerSocket(8888);
>>> server = new TThreadPoolServer(processor, serverTransport);
>>> ---
>>>
>>> The CounterHandler implements CounterService.Iface
>>>
>>> In the PHP Client code I have,
>>>
>>> --Client.php--
>>> $server = '192.168.1.64';
>>> $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
>>> to
>>> the Java Service
>>> $transport = new TBufferedTransport($socket, 1024, 1024);
>>> $protocol = new TBinaryProtocol($transport);
>>> $client = new CounterServiceClient($protocol);
>>> $transport->open();
>>> $jcount = $client->increment($auctionId);
>>> $transport->close();
>>> ---
>>>
>>> The problem is the fact that; under our production load (1500 persistent
>>> socket connections from 7 PHP clients, about a total of  300 requests per
>>> second), the PHP Client fails to get the correct increment(id) value from
>>> the Buffer, it somehow gets previous replies from the Server.
>>>
>>> I'm logging the Java Server transactions, the Java Server seems to work
>>> fine; I'm suspecting that the problem occurs because of the PHP client
>>> libraries.
>>>
>>> We are running PHP in lighttpd and fastcgi btw.
>>>
>>> Any comments on solving the problem is welcome.
>>>
>>> Regards,
>>> Utku
>>>

Mime
View raw message