hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: custom filter on hbase 0.96
Date Wed, 22 Oct 2014 01:53:24 GMT
See this blog post:
http://www.flurry.com/2012/12/06/exploring-dynamic-loading-of-custom-filters-i#.VEcNtNR4rZg

Cheers

On Tue, Oct 21, 2014 at 6:48 PM, Kevin <kevin.macksamie@gmail.com> wrote:

> Also, if you do end up using dynamic loading, you'll need a way to version
> your filters because the RS will not reload a JAR if it changes.
>
> On Tue, Oct 21, 2014 at 9:46 PM, Kevin <kevin.macksamie@gmail.com> wrote:
>
> > I haven't tried dynamic loading of filters on RS, but I know it does
> > exist. See https://issues.apache.org/jira/browse/HBASE-9301.
> >
> > If you still can't get it to work, then I suggest distributing your
> > filters to the RS and restart them. Let us know how everything works out.
> >
> > On Tue, Oct 21, 2014 at 9:02 PM, Matt K <matvey1414@gmail.com> wrote:
> >
> >> Thanks Kevin!
> >>
> >> I was under impression, probably mistakingly, that as of 0.96 placing
> >> the filter on hdfs under hbase lib directory is sufficient and RS should
> >> load the filter dynamically from hdfs. Is that not the case?
> >>
> >> On Tuesday, October 21, 2014, Kevin <kevin.macksamie@gmail.com> wrote:
> >>
> >> > BTW, the error looks like you didn't distribute your custom filter to
> >> your
> >> > region servers.
> >> >
> >> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> >> > <javascript:;>> wrote:
> >> >
> >> > > Matt,
> >> > >
> >> > > You should create your own proto file and compile that with the
> Google
> >> > > Protocol Buffer compiler. Take a look at the
> SingleColumnValueFilter's
> >> > > code:
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >> > >
> >> > > You will need to override `public byte[] toByteArray()` and `public
> >> > static
> >> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray()
> should
> >> be
> >> > > the byte[] from serializing with the protocol buffer. This will also
> >> be
> >> > the
> >> > > input to parseFrom(byte[]), which is called using reflection on the
> >> > > server-side to instantiate your custom filter and use it.
> >> > >
> >> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >
> >> > >> Anyone?
> >> > >>
> >> > >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >>
> >> > >> > Hi, can anyone help with above? Feels like I'm missing something
> >> > >> obvious.
> >> > >> >
> >> > >> > On Wednesday, October 15, 2014, Nishanth S <
> >> nishanth.2884@gmail.com
> >> > <javascript:;>
> >> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> >> <javascript:;>');>>
> >> > wrote:
> >> > >> >
> >> > >> >> Thanks Ted .I will take a look.
> >> > >> >>
> >> > >> >> -Nishanth
> >> > >> >>
> >> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> >> > <javascript:;>> wrote:
> >> > >> >>
> >> > >> >> > Nishanth:
> >> > >> >> > Good question.
> >> > >> >> >
> >> > >> >> > As a general coding guide, writing unit test is
always a good
> >> > start.
> >> > >> >> Using
> >> > >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> > >> >> >
> >> > >> >> > There're various unit tests for Filters in hbase
code.
> >> > >> >> >
> >> > >> >> > Cheers
> >> > >> >> >
> >> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> >> > nishanth.2884@gmail.com <javascript:;>
> >> > >> >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Hi Ted ,
> >> > >> >> > >  Since I am also working on similar thing is
there a way we
> >> can
> >> > >> first
> >> > >> >> > test
> >> > >> >> > > the filter on client side?.You know what I
 mean without
> >> > disrupting
> >> > >> >> > others
> >> > >> >> > > who are using the same cluster for other work?
> >> > >> >> > >
> >> > >> >> > > Thanks,
> >> > >> >> > > Nishanth
> >> > >> >> > >
> >> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <
> yuzhihong@gmail.com
> >> > <javascript:;>>
> >> > >> wrote:
> >> > >> >> > >
> >> > >> >> > > > bq. Or create a new file, compile it into
...
> >> > >> >> > > >
> >> > >> >> > > > You should go with the above approach.
> >> > >> >> > > >
> >> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt
K <
> >> matvey1414@gmail.com
> >> > <javascript:;>>
> >> > >> >> wrote:
> >> > >> >> > > >
> >> > >> >> > > > > Hi all,
> >> > >> >> > > > >
> >> > >> >> > > > > I'm trying to get a custom filter
to work on HBase 0.96.
> >> > After
> >> > >> >> some
> >> > >> >> > > > > searching, I found that starting
from 0.96, the
> >> implementer
> >> > is
> >> > >> >> > required
> >> > >> >> > > > to
> >> > >> >> > > > > implement "toByteArray" and "parseFrom"
methods, using
> >> > Protocol
> >> > >> >> > > Buffers.
> >> > >> >> > > > > But I'm having trouble with the "how".
> >> > >> >> > > > >
> >> > >> >> > > > > The "proto" file for the existing
filters is located
> here:
> >> > >> >> > > > >
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >>
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >> > >> >> > > > >
> >> > >> >> > > > > Am I supposed to modify that file?
Or create a new file,
> >> > >> compile
> >> > >> >> it
> >> > >> >> > > into
> >> > >> >> > > > > Java, and package it up with the
filter?
> >> > >> >> > > > >
> >> > >> >> > > > > In the meantime, I've taken a shortcut
that's not
> working.
> >> > >> Here's
> >> > >> >> my
> >> > >> >> > > > code:
> >> > >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> > >> >> > > > >
> >> > >> >> > > > > I'm using "PrefixFilter", which comes
with HBase, since
> >> I'm
> >> > >> also
> >> > >> >> > > > filtering
> >> > >> >> > > > > by "prefix". However, that errors
out with the
> following:
> >> > >> >> > > > > http://pastebin.com/zBg47p6Z
> >> > >> >> > > > >
> >> > >> >> > > > > Thanks in advance for helping!
> >> > >> >> > > > >
> >> > >> >> > > > > -Matt
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > www.calcmachine.com - easy online calculator.
> >> > >> >
> >> > >>
> >> > >>
> >> > >> --
> >> > >> www.calcmachine.com - easy online calculator.
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message