hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sela <am...@infolinks.com>
Subject Re: Bulk load from OSGi running client
Date Sun, 22 Sep 2013 13:10:15 GMT
I think I got it.
Algorithm has a Configuration member that is instantiated when Algorithm is
created. Since Algorithm is static, when I update a jar that doesn't cause
the hbase bundle to update, configuration still holds to the old CL.
I tested with GZ and changed the code to call new Configuration (in
getCodec(Configuration conf) GZ override) and it works. Makes sense because
it probably takes the current TCCL.

Since Algorithm.getCodec(Configuration conf) is only called from inside
Algorithm and the only (apparent) reason to have configuration as a member
of Algorithm is to avoid code duplication of
true)* for every codec, I think it would be better to duplicate code here
even if it's not as pretty as it is now, because on top of cloning the
configuration properties, the clone constructor also clones the CL and that
may cause problems where CL changes like in OSGi environment.

The NPE causing recovery is still a mystery to me :)

On Wed, Sep 11, 2013 at 3:34 PM, Amit Sela <amits@infolinks.com> wrote:

> I did some more digging and I got this:
> When the HBase bundle is loaded (system start) the Compression.Algorithm
> is probably created for the first time and the constructor calls new
> Configuration().
> When I update a bundle (but not the HBase bundle) it refreshes the
> relevant packages, which doesn't include the HBase bundle, and once I try
> to use getCodec() (GZ in my case) it creates a
> new ReusableStreamGzipCodec() and sets new Configuration(*conf*) - where *conf
> *is a private final member in Algorithm. Since *conf *holds the old class
> loader (referring to the pre-update bundle) it passes that CL to the new
> configuration created for the codec.
> I still have NO IDEA why NPE would cause Compression.Algorithm to
> re-instantiate itself...
> I think that calling new Configuration() and
> setting this.conf.setBoolean("hadoop.native.lib", true) for each codec
> would solve it since the class loader that would be set is the TCCL.
> I'll give it a try an keep updating.
> Thanks,
> Amit.
> On Mon, Sep 9, 2013 at 9:12 PM, Stack <stack@duboce.net> wrote:
>> On Mon, Sep 9, 2013 at 12:14 AM, Amit Sela <amits@infolinks.com> wrote:
>> ...
>> > The main issue still remains, it looks like Compression.Algortihm
>> > configuration's class loader had reference to the bundle in revision 0
>> > (before jar update) instead of revision 1 (after jar update). This
>> could be
>> > because of caching (or static) but then, why should it work after I get
>> > NullPointerException (it does, immediately, no restarts or bundle
>> updates).
>> >
>> When you say configuration above, you mean Compression.Algorithm's
>> reference to an Hadoop Configuration instance?  I've not looked at code.
>>  Is it coming in via static?
>> I am not sure what it would then start working after NPE.  I would expect
>> that it would stay broke rather than recover.
>> St.Ack

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message