thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "郑华斌" <huabin.zh...@qq.com>
Subject Re:Re: Best way to sniffer thrift requests
Date Tue, 11 Jul 2017 23:41:45 GMT
Thank you Randy. Yes it's convenient to searilize thrift request, but where to do it?

I can implement it in each api handle:

def Sum(req):
    // serialize and write out
    // business code
but it intrudes business code, s
o it's not an ideal solution.

------------------ 原始邮件 ------------------
发件人: "Randy Abernethy" <randy.abernethy@gmail.com>;
发送时间: 2017年7月12日(星期三) 5:45
收件人: "user@thrift.apache.org" <user@thrift.apache.org>;
主题: Re: Best way to sniffer thrift requests



Hi Huabin,

I might attack the problem at a higher level and define the API in terms of
messages, structs in Apache Thrift. Then when a server receives a requests
in the form of of a struct, it can serialize the struct to a memory buffer
and send the buffer out as a message queue message.

Apache Thrift makes this very easy because all structs have read() and
write() methods that can be used to serialize them to/from any I/O stack.
So if you create a TMemoryBuffer transport and add a TCompactProtocol to it
you can write your struct out to the mem buf and then email it (or
whatever).

Here's an example in Python but the concept is the same with any language:

https://github.com/RandyAbernethy/ThriftBook/blob/master/part3/mq/QuoteGen.py

If you really need a low level solution, I would using message framing and
then create a custom framing layer that forks a copy of every frame out to
the message queue while passing another copy up the stack to the protocol
(e.g. TCompactProtocol/TMyFrameDupTrans/TSocket). There's a custom tee
transport example in Java here:

https://github.com/RandyAbernethy/ThriftBook/blob/master/part2/servers/factories/TTeeTransport.java

Hope this helps.

Best,
Randy


On Tue, Jul 11, 2017 at 2:16 AM, 郑华斌 <huabin.zheng@qq.com> wrote:

> For service S, it has N apis, what's the best way to copy requests of
> specified apis and resend them to a message queue, without affecting the
> normal service?
>
>
> Maybe there are following possible ways:
> 1) tcp copy: with tcp copy tools that copy requests on the server side,
> and redirect the copied flow to a fake server, there requests of different
> apis are send to message queue, in the format of api:serialized_request.
> But deploying is difficult because tcpcopy requires root permission.
> 2) develop a transport to wrapped current transport on server side, and
> send buffers to mq as well as call underline transport. But how to identify
> which api a request buffer belong to?
>
>
> Any ideas?
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message