thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Craig <ben.cr...@ni.com>
Subject Re: very large request to HsHa server
Date Mon, 06 Jan 2014 22:10:14 GMT
The maximum length for TFramedTransport is configurable. 
TFramedTransport's ctor accepts a max length argument, and 
TFramedTransport::Factory also accepts a maxLength.

Jules Cisek <jules@luminate.com> wrote on 01/06/2014 03:55:45 PM:

> From: Jules Cisek <jules@luminate.com>
> To: user@thrift.apache.org, 
> Date: 01/06/2014 03:56 PM
> Subject: very large request to HsHa server
> 
> hello,
> 
> TL;DR version: i need to transfer a huge amount of data (millions of 
small
> objects) using a single call to a THsHaServer (or 
TThreaderSelectorServer)
> 
> The long version:
> 
> i built a java server around TThreadPoolServer that accepts an extremely
> large amount of data (millions of small objects) via a single call every
> hour or so (from a python script) and also accepts various requests for
> small amounts of data tens of times per second.  the requests come from
> python, java, and perl hence why i went with thrift in the first place.
> 
> this worked great using TBufferedTransport until i had to modify the 
java
> server to support async clients.  i've rewritten my service to use the
> THsHaServer and switched to TFramedTransport on the clients.
> 
> everything worked great (my test suite was happy) but it turns out in
> production the hourly data update call far far exceeds the maximum size
> allowed in TFramedTransport:
> 
> Frame size (150283373) larger than max length (16384000)!
> 
> yes i realize this is not what thrift was really designed for but it was
> working great until the async requirement came about.
> 
> the data is a list of millions of small objects, not one large piece of
> data that could be streamed, by the way.
> 
> my options at this point are (in order of difficulty):
> . run a second thrift server in my java server that uses 
TThreadPoolServer
> on a different port to allow using TBufferedTransport for the update 
call
> . find a different (non-thrift) mechanism to transfer data from the 
python
> update script to the java service
> . rewrite the update logic to chunk the data
> 
> none of these are particularly appealing due to the nature of how my
> service works (the update needs to be fairly atomic, the objects are not
> trivial, etc.).
> 
> i'm going to go with option a unless there is a way to transfer a huge
> amount of data in a single call that does work with HsHa (or
> ThreadedSelector) and was just wondering if anyone had any better ideas.
> 
> thanks,
> ~j
> 
> -- 
> jules cisek | jules@luminate.com


Mime
View raw message