thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Abernethy <>
Subject Re: Thrift RPC database query results
Date Mon, 25 Apr 2016 20:37:17 GMT
While I am the largest of thrift fans, it is worth noting that Thrift is
great for RPC, that is to say, creating really fast cross language
microservice interfaces. It is not great for large scale data transfer.
Cassandra was originally built with a Thrift API but moved to CQL and a
native protocol, purpose built for returning large datasets. API
interactions that are characterized by fast, modestly sized transfers work
great with Thrift but transferring anything bigger than a few megs, while
possible, may not be practical or optimal. I think you could say this about
RPC systems in general (protobuf/grpc, MSRPC, RMI, etc.).

On Fri, Apr 22, 2016 at 5:43 AM, Edward Capriolo <>

> Thrif  messages have to be buffered into memory. I suggest to use paging n
> rows at a time to keep latency predictable.
> On Friday, April 22, 2016, Tenghuan He <> wrote:
> > Hi there
> >
> >     I have a PostgreSQL database on my server machine, which can only be
> > queried locally. Now I want to query the data from another machine. It
> > seems that Thrift RPC is a natural and good choice. However I found there
> > is some problem when the ResultSet is very large, say there are millions
> of
> > rows in the ResultSet. Since ResultSet is not serializable and
> serializing
> > it is meaningless. Returning the materialized rows all at once consumes
> two
> > much memory.
> > I consider returning such as 1000 rows each time
> > Is there any other idea or advice?
> >
> > Thanks in advance
> >
> > Tenghuan He
> >
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message