thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Dodd <andrewdod...@gmail.com>
Subject Re: Asynchronous RPC
Date Mon, 11 Aug 2014 19:36:33 GMT
Hi Jens,

The situation came about due to the race condition above. My client made
two requests one after the other, which were dispatched immediately in a
thread pool on the server. Since I'm on a multicore system, the two
responses were written to the wire at exactly the same time. The first
thing written to the wire is the int32 VERSION_1 (0x80010000). My client
ended up receiving two of these concurrently.

You can probably imagine the rest yourself, but as such, when the client
receives this, we go into TBinaryProtocol::readMessageBegin. It reads off
the version as VERSION_1, fine. This method then calls readString(), which
does a readI32 expecting the length of the string. However, the server has
written another VERSION_1 to the wire, which then gets interpreted as the
length of the string, and since it's greater than 0x7FFFFFFF, it's
interpreted as being negative and the bytes "remaining" in the buffer is
therefore larger.

A fun one indeed!

Andrew


On 11 August 2014 12:17, Jens Geyer <jensgeyer@hotmail.com> wrote:

> Hi Andrew,
>
> great to hear the good news!
>
>  if you send readString a negative size on a TSocket,
>> it will throw a string-out-of-bounds exception since
>> [...]
>> I don't know if that's a bug worth fixing [...]
>>
>
> IMHO a negative size is not, well, a common use case. If I don't overlook
> anything, the only place where this may happen is when the length exceeds
> 0x7FFFFFFF, then getting casted into a signed int. So (again IMHO) this
> looks not like a real bug to me. I just wonder how you did run into that
> scenario?
>
>
> Have fun,
> JensG
>
>
>
> -----Urspr√ľngliche Nachricht----- From: Andrew Dodd
> Sent: Saturday, August 9, 2014 8:30 PM
> To: user@thrift.apache.org
> Subject: Re: Asynchronous RPC
>
>
> Hi Jens,
>
> Thanks for the reply - I'd actually forgotten about this issue for a while.
> I did some digging tonight, and the problem is that the server was sending
> two messages at once on the same client, which breaks horribly. If I
> synchronize on the client, then the problem goes away.
>
> Cheers,
> Andrew
>
> PS. One slightly strange thing that happens is in TBinaryProtocol.java - if
> you send readString a negative size on a TSocket, it will throw a
> string-out-of-bounds exception since getBytesRemainingInBuffer (which would
> normally catch this) is returning -1. I don't know if that's a bug worth
> fixing or not, since my real-world problem was elsewhere.
>
>
>
> On 9 August 2014 09:51, Jens Geyer <jensgeyer@hotmail.com> wrote:
>
>  Hi,
>>
>>
>>  My thought on this is that the one-way message isn't actually
>>
>>> completely one-way; that we do send some sort of acknowledgement
>>> packet, and this is causing a mess on the receiving end. I also notice
>>> that if I put a breakpoint inside my call-back to the client (inside the
>>> server's code) and let it run a while after, that I don't get an error
>>> and
>>> the message does get received on the client, further strengthening
>>> my suspicions.
>>>
>>> Can anyone confirm that this could be the issue?
>>>
>>>
>> If you have a test case and the issue is reproducible with current trunk,
>> you should file a JIRA ticket.
>>
>> And if (by accident :-) you already have a patch/pull request,
>> see http://thrift.apache.org/docs/HowToContribute
>>
>> Have fun,
>> JensG
>>
>>
>> -----Urspr√ľngliche Nachricht----- From: Andrew Dodd
>> Sent: Monday, July 21, 2014 7:18 AM
>> To: user@thrift.apache.org
>> Subject: Asynchronous RPC
>>
>>
>> Morning,
>>
>> Over the last few days I've been experimenting with the code provided by
>> this article:
>> http://joelpm.com/2009/04/03/thrift-bidirectional-async-rpc.html .
>> However,
>> that code doesn't exactly fit my needs. I checked out the source from
>> GitHub and did the following things:
>>
>> 1. Updated it to work with Thrift 0.9.1 - everything is OK here.
>> 2. Changed the Service to use a different callback method (it does mean
>> several no-op implementations, but that is fine).
>> 3. Changed the MessageDistributor to be instantiated once per client. This
>> means that rather than sending one response to each client, it is only
>> sent
>> to the client which sent the request.
>>
>> Up to this point everything works fine. However, I don't really want one
>> thread per client as it's potentially wasteful. So rather than do that, I
>> made the MessageDistributor submit a runnable into a shared Executor
>> Service. At this point, problems start to occur. When the client receives
>> it the first response, I get this message:
>>
>> Exception in thread "Thread-2" java.lang.StringIndexOutOfBoundsException:
>> String index out of range: -2147418111
>>    at java.lang.String.checkBounds(String.java:371)
>>    at java.lang.String.<init>(String.java:415)
>>    at
>> org.apache.thrift.protocol.TBinaryProtocol.readString(
>> TBinaryProtocol.java:326)
>>    at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(
>> TBinaryProtocol.java:197)
>>    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
>>    at
>> com.joelpm.bidiMessages.client.MessageReceiver.run(
>> MessageReceiver.java:31)
>>    at java.lang.Thread.run(Thread.java:745)
>>
>> My thought on this is that the one-way message isn't actually completely
>> one-way; that we do send some sort of acknowledgement packet, and this is
>> causing a mess on the receiving end. I also notice that if I put a
>> breakpoint inside my call-back to the client (inside the server's code)
>> and
>> let it run a while after, that I don't get an error and the message does
>> get received on the client, further strengthening my suspicions.
>>
>> Can anyone confirm that this could be the issue?
>>
>> Thanks,
>> Andrew
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message