thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Craig <>
Subject Re: very large request to HsHa server
Date Mon, 06 Jan 2014 22:10:14 GMT
The maximum length for TFramedTransport is configurable. 
TFramedTransport's ctor accepts a max length argument, and 
TFramedTransport::Factory also accepts a maxLength.

Jules Cisek <> wrote on 01/06/2014 03:55:45 PM:

> From: Jules Cisek <>
> To:, 
> Date: 01/06/2014 03:56 PM
> Subject: very large request to HsHa server
> hello,
> TL;DR version: i need to transfer a huge amount of data (millions of 
> objects) using a single call to a THsHaServer (or 
> The long version:
> i built a java server around TThreadPoolServer that accepts an extremely
> large amount of data (millions of small objects) via a single call every
> hour or so (from a python script) and also accepts various requests for
> small amounts of data tens of times per second.  the requests come from
> python, java, and perl hence why i went with thrift in the first place.
> this worked great using TBufferedTransport until i had to modify the 
> server to support async clients.  i've rewritten my service to use the
> THsHaServer and switched to TFramedTransport on the clients.
> everything worked great (my test suite was happy) but it turns out in
> production the hourly data update call far far exceeds the maximum size
> allowed in TFramedTransport:
> Frame size (150283373) larger than max length (16384000)!
> yes i realize this is not what thrift was really designed for but it was
> working great until the async requirement came about.
> the data is a list of millions of small objects, not one large piece of
> data that could be streamed, by the way.
> my options at this point are (in order of difficulty):
> . run a second thrift server in my java server that uses 
> on a different port to allow using TBufferedTransport for the update 
> . find a different (non-thrift) mechanism to transfer data from the 
> update script to the java service
> . rewrite the update logic to chunk the data
> none of these are particularly appealing due to the nature of how my
> service works (the update needs to be fairly atomic, the objects are not
> trivial, etc.).
> i'm going to go with option a unless there is a way to transfer a huge
> amount of data in a single call that does work with HsHa (or
> ThreadedSelector) and was just wondering if anyone had any better ideas.
> thanks,
> ~j
> -- 
> jules cisek |

View raw message