thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Moreno <jwellington.mor...@gmail.com>
Subject Re: Thrift RPC database query results
Date Sat, 28 May 2016 15:41:49 GMT
Agree with Randy. A more indirect option is to use Thrift to send a
"pointer" to the actual file, and then use another protocol to serve the
data (HTTP, FTP, etc).

Kind Regards,
Wellington Moreno
*​Software Emperor*
*​RedRoma*

On Mon, Apr 25, 2016 at 1:37 PM, Randy Abernethy <randy.abernethy@gmail.com>
wrote:

> While I am the largest of thrift fans, it is worth noting that Thrift is
> great for RPC, that is to say, creating really fast cross language
> microservice interfaces. It is not great for large scale data transfer.
> Cassandra was originally built with a Thrift API but moved to CQL and a
> native protocol, purpose built for returning large datasets. API
> interactions that are characterized by fast, modestly sized transfers work
> great with Thrift but transferring anything bigger than a few megs, while
> possible, may not be practical or optimal. I think you could say this about
> RPC systems in general (protobuf/grpc, MSRPC, RMI, etc.).
>
> On Fri, Apr 22, 2016 at 5:43 AM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>
> > Thrif  messages have to be buffered into memory. I suggest to use paging
> n
> > rows at a time to keep latency predictable.
> >
> > On Friday, April 22, 2016, Tenghuan He <tenghuanhe@gmail.com> wrote:
> >
> > > Hi there
> > >
> > >     I have a PostgreSQL database on my server machine, which can only
> be
> > > queried locally. Now I want to query the data from another machine. It
> > > seems that Thrift RPC is a natural and good choice. However I found
> there
> > > is some problem when the ResultSet is very large, say there are
> millions
> > of
> > > rows in the ResultSet. Since ResultSet is not serializable and
> > serializing
> > > it is meaningless. Returning the materialized rows all at once consumes
> > two
> > > much memory.
> > > I consider returning such as 1000 rows each time
> > > Is there any other idea or advice?
> > >
> > > Thanks in advance
> > >
> > > Tenghuan He
> > >
> >
> >
> > --
> > Sorry this was sent from mobile. Will do less grammar and spell check
> than
> > usual.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message