james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <e...@apache.org>
Subject Re: HBase message implementation
Date Mon, 01 Aug 2011 17:07:02 GMT
On 01/08/11 18:52, Norman Maurer wrote:
> Hi Eugen,
>
> comments inside..
>
> 2011/8/1 Ioan Eugen Stan<stan.ieugen@gmail.com>:
>> Hello,
>>
>> I'm having some issues putting things together when it comes to
>> Message implementation. I will explain things right away.
>> The current code [1], [2], and [3] is based on JPA implementation and
>> it copies the message content into byte arrays.
>> This does not seem to scale too well.
>>
>> There is also the issue of getting the info back from HBase. I agree
>> with Eric and Norman that large content should be split and I'm
>> thinking of providing a HBaseMessage implementation that should read
>> data from HBase as demanded (ChunkedInputStream and
>> ChunkedOutputStream as suggested by Norman [4] ).
>>
>> Do I have to copy the data when someone creates a new HBaseMessage, or
>> as suggested by the streaming alternative, I can save a reference of
>> SharedInputStream and when I save the message to the mailbox I can
>> move the bytes to HBase.
>>
>> The way I see things is that HBaseMessage implementation is only used
>> for retrieving data from HBase. It should not store the message body
>> (in the constructor).
>
> Exactly.. The only special case is the MessageMapper.copy(..) but
> maybe thiscan be handled in a more efficient way in hbase...
>

The byte loading into memory can be further deferred from the 
HBaseMessage constructor to a HBaseMessageMapper method, but at the end, 
the memory will be used by the bytes because there is no native 
streaming support on hbase side.

To mitigate the effect, you can implement ChunkedInput/OuputStream that 
will split the stream in more limited chunks, saving some peak memory usage.

The call to these Chunk classes that transforms the stream to the bytes 
is probably best placed in the HBaseMessageMapper to avoid pollute the 
HBaseMessage constructor with many byte arrays, also loosing the 
positive effect of the chunks.

You will need to loop and instantiate one HBase Put per chunk in the 
HBaseMessageMapper.


>>
>> Please, I need some clarification.
>>
>> Thanks,
>>
>> [1] http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/source/browse/src/main/java/org/apache/james/mailbox/hbase/mail/model/hbase/HBaseMessage.java
>> [2] http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/source/browse/src/main/java/org/apache/james/mailbox/hbase/mail/model/hbase/AbstractHBaseMessage.java
>> [3] http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/source/browse/src/main/java/org/apache/james/mailbox/hbase/mail/model/hbase/HBaseStreamingMessage.java
>> [4] https://github.com/rantav/hector/tree/master/core/src/main/java/me/prettyprint/cassandra/io
>>
>> --
>> Ioan Eugen Stan
>> http://ieugen.blogspot.com/
>
>
> Bye,
> Norman
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>


-- 
Eric Charles
http://about.echarles.net

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message