mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fernando Padilla <f...@alum.mit.edu>
Subject Re: [vysper] cloning and forwarding stanzas
Date Wed, 02 Sep 2009 16:06:43 GMT
Right.  The idea is that you should have methods and returns be as low 
as possible ( Iterable ), instead of higher requirements ( Collection, 
List, Set, etc ).  This just gives you some flexibility on how to 
implement things, and reduce the number of data structures you have to 
maintain or even iterate through.  It's really neat.  But like you said, 
it's theoretically more efficient, and maybe a good programming 
practice, but I don't have any hard numbers..

I just wanted to make sure to mention it early, because it might be one 
of those things that's easy to bake in, but a little hard to apply later 
( since it changes lots of apis )..

But you can look at Google Collections.  They have some nice utilities 
under the Iterables class, for filtering or transforming the stream of 
objects in flight. :)

ps - But ultimately the memory footprint for most cases won't be an 
issue.  But I'm really interested in streamlining the code as much as 
possible because my use case is for sports fans following a live game ( 
play by play, chat, etc ), so there might be a large number of MUC room 
participants or PubSub subscribers (thousands? tens of thousands?) And 
the system simply can't load up full lists room participants or 
subscribers to be effective..

So I'm thinking into the future :)

Heck I'm already pondering how you can make Vysper cluster aware.  It 
might be stable enough to start to bake in clustering/sharding 
concepts.. not sure if anyone else would be interested in chatting on 
this. :)

On 9/2/09 12:31 AM, Bernd Fondermann wrote:
> Fernando Padilla wrote:
>> I wonder, would it be too complicated to create something like
>> CopyOnWriteStanzaClone?
>> I also wonder if there is some way to not generate all of the stanzas
>> right away, but only as they are processed.  Basically following the
>> google-collections ideas of creating Iterables whose items are
>> generated/instanciated on the fly instead of a fully instanciated
>> collection.  This won't fix the cpu use on cloning, but it could at
>> least reduce the max memory used depending on how many threads are
>> devoted to consuming the Iterable, versus processing the in-flight
>> stanzas.. etc.
> Interesting. IIUC, that would mean lazy generation of stanzas: instead
> of having 100 stanzas waiting for 10 threads to process them, only 1
> stanza + 100 to addresses sit there. Each time a thread starts
> processing a stanza, the actual forwarding stanza is generated from the
> original one + one 'to' address.
> Yet, I don't think our memory penalty is high enough to justify
> implementing this non-trivial logic. In fact, our stanza copies are
> pretty lean already.
> This whole discussion is rather theoretical anyway because we have no
> measured data to judge from.
>    Bernd
>> On 9/1/09 2:47 AM, Bernd Fondermann wrote:
>>> Michael Jakl wrote:
>>>> Hi!
>>>> On Tue, Sep 1, 2009 at 11:27, Bernd
>>>> Fondermann<bf_jak@brainlounge.de>   wrote:
>>>>> It's not 'easier'. It would use less objects, yet not neccessarily much
>>>>> less memory though. Think about it: Queueing up the character data
>>>>> needs
>>>>> much more memory: what's now stored as 1 immutable inner object (and
>>>>> it's children) is then multiplexed into 1000s of character streams.
>>>>> Probably this would need much more memory! (And don't tell me now to
>>>>> store in special string objects what's unchanged, because this is what
>>>>> we are doing now right now with immutable objects!)
>>>> :D, no I won't. You're right the character data would take a lot more
>>>> space than the objects do now.
>>>> Sorry for taking a bit  longer to grasp that, and thanks for the
>>>> explanation ;)
>>> No problem. You don't know how long it took me to figure this out in the
>>> first place! ;-)
>>>     Bernd

View raw message