thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jules Cisek <ju...@luminate.com>
Subject very large request to HsHa server
Date Mon, 06 Jan 2014 21:55:45 GMT
hello,

TL;DR version: i need to transfer a huge amount of data (millions of small
objects) using a single call to a THsHaServer (or TThreaderSelectorServer)

The long version:

i built a java server around TThreadPoolServer that accepts an extremely
large amount of data (millions of small objects) via a single call every
hour or so (from a python script) and also accepts various requests for
small amounts of data tens of times per second.  the requests come from
python, java, and perl hence why i went with thrift in the first place.

this worked great using TBufferedTransport until i had to modify the java
server to support async clients.  i've rewritten my service to use the
THsHaServer and switched to TFramedTransport on the clients.

everything worked great (my test suite was happy) but it turns out in
production the hourly data update call far far exceeds the maximum size
allowed in TFramedTransport:

Frame size (150283373) larger than max length (16384000)!

yes i realize this is not what thrift was really designed for but it was
working great until the async requirement came about.

the data is a list of millions of small objects, not one large piece of
data that could be streamed, by the way.

my options at this point are (in order of difficulty):
. run a second thrift server in my java server that uses TThreadPoolServer
on a different port to allow using TBufferedTransport for the update call
. find a different (non-thrift) mechanism to transfer data from the python
update script to the java service
. rewrite the update logic to chunk the data

none of these are particularly appealing due to the nature of how my
service works (the update needs to be fairly atomic, the objects are not
trivial, etc.).

i'm going to go with option a unless there is a way to transfer a huge
amount of data in a single call that does work with HsHa (or
ThreadedSelector) and was just wondering if anyone had any better ideas.

thanks,
~j

-- 
jules cisek | jules@luminate.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message