hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈加俊 <cjjvict...@gmail.com>
Subject Re: I can't get many versions of the specified column,but only get the latest version of the specified column
Date Thu, 24 Feb 2011 07:34:40 GMT
HTable object has not setAutoFlush. It's default value is true at my
cluster.So I set it true as follows ,but error is still the same.

public class GetRowVersionsTest extends TestCase
{
    private final byte[] family    = Bytes.toBytes("log");

    private final byte[] qualifier = Bytes.toBytes("siteUrl");

    private final byte[] rowKey    = Bytes.toBytes(1);

    private final long   ts1       = 1298529542218L;

    private final long   ts2       = ts1 + 100;

    private final long   ts3       = ts1 + 100;

    private final byte[] value1    = Bytes.toBytes("value1");

    private final byte[] value2    = Bytes.toBytes("value2");

    private final byte[] value3    = Bytes.toBytes("value3");

    private void insert(final long ts, final byte[] value) throws
IOException
    {
        final HTable table =
IDMHBaseConfiguration.getTable(BigTableName.getSite());
        table.setAutoFlush(false);
        final Put put = new Put(rowKey);
        put.add(family, qualifier, ts, value);
        table.put(put);
    }

    private void sleep()
    {
        try
        {
            Thread.sleep(1000);
        }
        catch (final InterruptedException e)
        {
            e.printStackTrace();
        }
    }

    @Test
    public void testGetRowMultipleVersions() throws Exception
    {
        insert(ts1, value1);
        sleep();
        insert(ts2, value2);
        sleep();
        insert(ts3, value3);
        sleep();

        // check getRow with multiple versions
        final HTable table =
IDMHBaseConfiguration.getTable(BigTableName.getSite());
        final Get get = new Get(rowKey);
        get.setMaxVersions();
        final Result r = table.get(get);

        final List<KeyValue> list = r.list();
        for (final KeyValue kv : list)
        {
            System.err.println(kv.getKey());
            System.err.println(Bytes.toString(kv.getValue()));
        }

        final NavigableMap<byte[], NavigableMap<byte[], NavigableMap<Long,
byte[]>>> map = r.getMap();
        final NavigableMap<byte[], NavigableMap<Long, byte[]>> familyMap =
map.get(family);
        final NavigableMap<Long, byte[]> versionMap =
familyMap.get(qualifier);
        for (final Map.Entry<Long, byte[]> entry : versionMap.entrySet())
        {
            System.err.println(entry.getKey());
            System.err.println(Bytes.toString(entry.getValue()));
        }
        //        assertTrue(versionMap.size() == 3);

        //        assertTrue(value1 == versionMap.get(ts1));
        //        assertTrue(value2 == versionMap.get(ts2));
        //        assertTrue(value3 == versionMap.get(ts3));

        //        table.delete(new Delete(rowKey));
        //        assertTrue(table.get(get).size() == 0);
        //        table.close();
    }

On Thu, Feb 24, 2011 at 3:26 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> Does the HTable object have setAutoFlush(false) turned on by any chance?
>
> On Wed, Feb 23, 2011 at 11:22 PM, 陈加俊 <cjjvictory@gmail.com> wrote:
> > line 89:        final NavigableMap<byte[], NavigableMap<Long, byte[]>>
> > familyMap = map.get(family);
> > map is null ,
> > and strangely  I use r.list() instead,
> > final List<KeyValue> list = r.list();
> > r is null !
> >
> >
> > 2011/2/24 Ryan Rawson <ryanobjc@gmail.com>
> >>
> >> Which line is line 89?
> >>
> >> Also it's preferable to do:
> >> assertEquals(3, versionMap.size());
> >> vs:
> >> assertTrue(versionMap.size() == 3);
> >>
> >> since the error messages from the former are more descriptive
> >> "expected 3 was 2".
> >>
> >> looking at the code it looks like it should work...
> >>
> >> On Wed, Feb 23, 2011 at 11:07 PM, 陈加俊 <cjjvictory@gmail.com> wrote:
> >> >  This is my test case ,but I get NPE some times .
> >> >
> >> > java.lang.NullPointerException
> >> > at
> >> >
> >> >
> com.uuwatch.idm.hbase.GetRowVersionsTest.testGetRowMultipleVersions(GetRowVersionsTest.java:89)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at
> >> >
> >> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >> > at
> >> >
> >> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >> > at
> >> >
> >> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >> > at
> >> >
> >> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >> > at
> >> >
> >> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> >> > at
> >> >
> >> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> >> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> >> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> >> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> >> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> >> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> >> > at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> >> > at
> >> >
> >> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> >> >
> >> > public class GetRowVersionsTest extends TestCase
> >> > {
> >> >    private final byte[] family    = Bytes.toBytes("log");
> >> >
> >> >    private final byte[] qualifier = Bytes.toBytes("siteUrl");
> >> >
> >> >    private final byte[] rowKey    = Bytes.toBytes(1);
> >> >
> >> >    private final HTable table     =
> >> > IDMHBaseConfiguration.getTable(BigTableName.getSite());
> >> >
> >> >    private final long   ts1       = 1298529542218L;
> >> >
> >> >    private final long   ts2       = ts1 + 100;
> >> >
> >> >    private final long   ts3       = ts1 + 100;
> >> >
> >> >    private final byte[] value1    = Bytes.toBytes("value1");
> >> >
> >> >    private final byte[] value2    = Bytes.toBytes("value2");
> >> >
> >> >    private final byte[] value3    = Bytes.toBytes("value3");
> >> >
> >> >    private void insert(final long ts, final byte[] value) throws
> >> > IOException
> >> >    {
> >> >        final Put put = new Put(rowKey);
> >> >        put.add(family, qualifier, ts, value);
> >> >        table.put(put);
> >> >    }
> >> >
> >> >    private void sleep()
> >> >    {
> >> >        try
> >> >        {
> >> >            Thread.sleep(1000);
> >> >        }
> >> >        catch (final InterruptedException e)
> >> >        {
> >> >            e.printStackTrace();
> >> >        }
> >> >    }
> >> >
> >> >    @Test
> >> >    public void testGetRowMultipleVersions() throws Exception
> >> >    {
> >> >        insert(ts1, value1);
> >> >        sleep();
> >> >        insert(ts2, value2);
> >> >        sleep();
> >> >        insert(ts3, value3);
> >> >        sleep();
> >> >
> >> >        // check getRow with multiple versions
> >> >        final Get get = new Get(rowKey);
> >> >        get.setMaxVersions();
> >> >        final Result r = table.get(get);
> >> >        final NavigableMap<byte[], NavigableMap<byte[],
> >> > NavigableMap<Long,
> >> > byte[]>>> map = r.getMap();
> >> >        final NavigableMap<byte[], NavigableMap<Long, byte[]>>
> familyMap
> >> > =
> >> > map.get(family);
> >> >        final NavigableMap<Long, byte[]> versionMap =
> >> > familyMap.get(qualifier);
> >> >        for (final Map.Entry<Long, byte[]> entry :
> versionMap.entrySet())
> >> >        {
> >> >            System.err.println(entry.getKey());
> >> >            System.err.println(Bytes.toString(entry.getValue()));
> >> >        }
> >> >        //        assertTrue(versionMap.size() == 3);
> >> >
> >> >        //        assertTrue(value1 == versionMap.get(ts1));
> >> >        //        assertTrue(value2 == versionMap.get(ts2));
> >> >        //        assertTrue(value3 == versionMap.get(ts3));
> >> >
> >> >        //        table.delete(new Delete(rowKey));
> >> >        //        assertTrue(table.get(get).size() == 0);
> >> >        //        table.close();
> >> >    }
> >> > }
> >> >
> >> >
> >> > On Thu, Feb 24, 2011 at 11:07 AM, Tatsuya Kawano
> >> > <tatsuya6502@gmail.com>wrote:
> >> >
> >> >> Hi Jiajun,
> >> >>
> >> >> Make sure you don't have the same timestamp on every versions of
> puts;
> >> >> try
> >> >> to put Thread.sleep() in your test(?) codes when necessary.
> >> >>
> >> >> You might not want to specify the timestamp by yourself but want to
> let
> >> >> HBase to store appropriate ones.
> >> >>
> >> >> --
> >> >> Tatsuya Kawano (Mr.)
> >> >> Tokyo, Japan
> >> >>
> >> >>
> >> >> On Feb 24, 2011, at 11:30 AM, Ryan Rawson <ryanobjc@gmail.com>
> wrote:
> >> >>
> >> >> > There are test cases for this, the functionality DOES work,
> something
> >> >> > is
> >> >> up...
> >> >> >
> >> >> > Without full code and full descriptions of your tables, debugging
> is
> >> >> > harder than it needs to be.  It's probably a simple typo or
> >> >> > something,
> >> >> > check your code and table descriptions again. Many people rely
on
> the
> >> >> > multi version query capabilities and it is very unlikely to be
> broken
> >> >> > in a released version of hbase.
> >> >> >
> >> >> > On Wed, Feb 23, 2011 at 6:27 PM, 陈加俊 <cjjvictory@gmail.com>
wrote:
> >> >> >>  final List<KeyValue> list = result.list();
> >> >> >>           for (final KeyValue it : list)
> >> >> >>           {
> >> >> >>               System.out.println(Bytes.toString(it.getKey()));
> >> >> >>               System.out.println(Bytes.toString(it.getValue()));
> >> >> >>           }
> >> >> >> I can only get the last version!  why ? Is there any testcase
in
> >> >> >> Hbase?
> >> >> >>
> >> >> >> On Thu, Feb 24, 2011 at 9:56 AM, 陈加俊 <cjjvictory@gmail.com>
> wrote:
> >> >> >>
> >> >> >>> /**
> >> >> >>>   * Create a sorted list of the KeyValue's in this result.
> >> >> >>>   *
> >> >> >>>   * @return The sorted list of KeyValue's.
> >> >> >>>   */
> >> >> >>>  public List<KeyValue> list() {
> >> >> >>>    if(this.kvs == null) {
> >> >> >>>      readFields();
> >> >> >>>    }
> >> >> >>>    return isEmpty()? null: Arrays.asList(sorted());
> >> >> >>>  }
> >> >> >>>
> >> >> >>> I will try it . Thank you very much!
> >> >> >>>
> >> >> >>> On Thu, Feb 24, 2011 at 9:45 AM, Buttler, David <
> buttler1@llnl.gov>
> >> >> wrote:
> >> >> >>>
> >> >> >>>> Result.list() ?
> >> >> >>>> Putting the hbase source into your IDE of choice (yay
Eclipse!)
> is
> >> >> really
> >> >> >>>> helpful
> >> >> >>>>
> >> >> >>>> Dave
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> -----Original Message-----
> >> >> >>>> From: 陈加俊 [mailto:cjjvictory@gmail.com]
> >> >> >>>> Sent: Wednesday, February 23, 2011 5:42 PM
> >> >> >>>> To: user@hbase.apache.org
> >> >> >>>> Cc: Buttler, David
> >> >> >>>> Subject: Re: I can't get many versions of the specified
> column,but
> >> >> only
> >> >> >>>> get the latest version of the specified column
> >> >> >>>>
> >> >> >>>> Thank you David !
> >> >> >>>>
> >> >> >>>> I alter the table schema as follow:
> >> >> >>>>
> >> >> >>>>> alter 'cjjIndexPageModify', {NAME => 'log'
, VERSIONS => 5 ,
> >> >> >>>>> METHOD
> >> >> =>
> >> >> >>>> 'add'}
> >> >> >>>>
> >> >> >>>> How to iterate over KeyValues?  which method in Result?
> >> >> >>>>
> >> >> >>>> On Thu, Feb 24, 2011 at 9:27 AM, Buttler, David
> >> >> >>>> <buttler1@llnl.gov>
> >> >> >>>> wrote:
> >> >> >>>>
> >> >> >>>>> What is your table schema set to?  By default
it holds 3
> >> >> >>>>> versions.
> >> >> >>>>> Also, you might iterating over KeyValues instead
of using the
> Map
> >> >> since
> >> >> >>>> you
> >> >> >>>>> don't really care about the organization, just
the time.
> >> >> >>>>>
> >> >> >>>>> Dave
> >> >> >>>>>
> >> >> >>>>> -----Original Message-----
> >> >> >>>>> From: 陈加俊 [mailto:cjjvictory@gmail.com]
> >> >> >>>>> Sent: Wednesday, February 23, 2011 5:22 PM
> >> >> >>>>> To: user@hbase.apache.org
> >> >> >>>>> Cc: Stack
> >> >> >>>>> Subject: Re: I can't get many versions of the
specified
> >> >> >>>>> column,but
> >> >> only
> >> >> >>>> get
> >> >> >>>>> the latest version of the specified column
> >> >> >>>>>
> >> >> >>>>> I execute it five times at diffrent time.
> >> >> >>>>>
> >> >> >>>>> //put data by version
> >> >> >>>>>
> >> >> >>>>> final Put p = new Put(key); // key
> >> >> >>>>> final long ts = System.currentTimeMillis();
> >> >> >>>>>      p.add(FAMILY, q1, ts,v1);
> >> >> >>>>>      p.add(FAMILY, q2, ts,v2);
> >> >> >>>>>      p.add(FAMILY, q3, ts,v3);
> >> >> >>>>>     table.put(p);
> >> >> >>>>>
> >> >> >>>>> So I can get five versions ,right?
> >> >> >>>>>
> >> >> >>>>> On Thu, Feb 24, 2011 at 2:06 AM, Stack <stack@duboce.net>
> wrote:
> >> >> >>>>>
> >> >> >>>>>> What do you get for a result?
> >> >> >>>>>>
> >> >> >>>>>> You are only entering a single version of
each column, a
> single
> >> >> >>>>>> version of FAMILY:q1, a single version FAMILY:q2,
and a
> >> >> >>>>>> FAMILY:q3.
> >> >> >>>>>>
> >> >> >>>>>> St.Ack
> >> >> >>>>>>
> >> >> >>>>>> On Wed, Feb 23, 2011 at 2:54 AM, 陈加俊
<cjjvictory@gmail.com>
> >> >> >>>>>> wrote:
> >> >> >>>>>>> I can't get many versions of the specified
column,but only
> get
> >> >> >>>>>>> the
> >> >> >>>>> latest
> >> >> >>>>>>> version of the specified column. Is there
anyone  help me?
> >> >> >>>>>>>
> >> >> >>>>>>> //put data by version
> >> >> >>>>>>>
> >> >> >>>>>>> final Put p = new Put(key); // key
> >> >> >>>>>>> final long ts = System.currentTimeMillis();
> >> >> >>>>>>>      p.add(FAMILY, q1, ts,v1);
> >> >> >>>>>>>      p.add(FAMILY, q2, ts,v2);
> >> >> >>>>>>>      p.add(FAMILY, q3, ts,v3);
> >> >> >>>>>>>
> >> >> >>>>>>>     table.put(p);
> >> >> >>>>>>>
> >> >> >>>>>>>
> >> >> >>>>>>> //get all versions of data
> >> >> >>>>>>>
> >> >> >>>>>>> final Get get = new Get(key);
> >> >> >>>>>>>          get.addFamily(FAMILY);
> >> >> >>>>>>>          get.setMaxVersions(10);
> >> >> >>>>>>>          final Result result = htable.get(get);
> >> >> >>>>>>>
> >> >> >>>>>>>          final NavigableMap<byte[],
NavigableMap<byte[],
> >> >> >>>>>>> NavigableMap<Long, byte[]>>>
map = result.getMap();
> >> >> >>>>>>>          final Set<Entry<byte[],
NavigableMap<byte[],
> >> >> >>>>>>> NavigableMap<Long, byte[]>>>>
entrySet = map.entrySet();
> >> >> >>>>>>>          final Iterator<Entry<byte[],
NavigableMap<byte[],
> >> >> >>>>>>> NavigableMap<Long, byte[]>>>>
iterator = entrySet
> >> >> >>>>>>>                  .iterator();
> >> >> >>>>>>>          while (iterator.hasNext())
> >> >> >>>>>>>          {
> >> >> >>>>>>>              final Entry<byte[], NavigableMap<byte[],
> >> >> >>>>>>> NavigableMap<Long, byte[]>>>
next = iterator.next();
> >> >> >>>>>>>
>  System.out.println(Bytes.toString(next.getKey()));
> >> >> >>>>> //family
> >> >> >>>>>>>              for (final Entry<byte[],
NavigableMap<Long,
> >> >> >>>>>>> byte[]>>
> >> >> >>>>>>> item : next.getValue().entrySet())
> >> >> >>>>>>>              {
> >> >> >>>>>>>
> >> >> >>>>>>>  System.out.println(Bytes.toString(item.getKey()));
> >> >> >>>>>>>                  for (final Entry<Long,
byte[]> _item :
> >> >> >>>>>>> item.getValue().entrySet())
> >> >> >>>>>>>                  {
> >> >> >>>>>>>                      System.out.println(_item.getKey());
//q
> >> >> >>>>>>>
> >> >> >>>>>> System.out.println(Bytes.toString(_item.getValue()));
> >> >> >>>>>>> //value
> >> >> >>>>>>>                  }
> >> >> >>>>>>>              }
> >> >> >>>>>>>
> >> >> >>>>>>>          }
> >> >> >>>>>>>
> >> >> >>>>>>> --
> >> >> >>>>>>> Thanks & Best regards
> >> >> >>>>>>> jiajun
> >> >> >>>>>>>
> >> >> >>>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> --
> >> >> >>>>> Thanks & Best regards
> >> >> >>>>> jiajun
> >> >> >>>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Thanks & Best regards
> >> >> >>>> jiajun
> >> >> >>>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> --
> >> >> >>> Thanks & Best regards
> >> >> >>> jiajun
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Thanks & Best regards
> >> >> >> jiajun
> >> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks & Best regards
> >> > jiajun
> >> >
> >
> >
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
> >
>



-- 
Thanks & Best regards
jiajun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message