phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jaanai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-5171) SkipScan incorrectly filters composite primary key which the key range contains all values
Date Mon, 11 Mar 2019 06:21:00 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

jaanai updated PHOENIX-5171:
----------------------------
    Description: 
Running the below SQL:
{code:sql}
create table if not exists aiolos(
vdate varchar,
tab varchar,
dev tinyint not null,
app varchar,
target varchar,
channel varchar,
one varchar,
two varchar,
count1 integer,
count2 integer,
CONSTRAINT PK PRIMARY KEY (vdate,tab,dev,app,target,channel,one,two));

upsert into aiolos values('2018-02-14','channel_agg',2,null,null,'A004',null,null,2,2);
upsert into aiolos values('2018-02-14','channel_agg',2,null,null,null,null,null,2,2);

SELECT * FROM aiolos WHERE dev = 2 AND vdate BETWEEN '2018-02-10' AND '2019-02-19' AND tab
= 'channel_agg' and channel='A004';
{code}
Throws exception:
{code:java}
Caused by: java.lang.IllegalStateException: The next hint must come after previous hint (prev=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
next=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0, kv=2018-02-14\x00channel_agg\x00\x82/0:\x00\x00\x00\x00/1550642992223/Put/vlen=4/seqid=5445463)
	at org.apache.phoenix.filter.SkipScanFilter.setNextCellHint(SkipScanFilter.java:171)
	at org.apache.phoenix.filter.SkipScanFilter.filterKeyValue(SkipScanFilter.java:145)
	at org.apache.hadoop.hbase.filter.FilterList.filterKeyValue(FilterList.java:264)
	at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:418)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:557)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6308)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6459)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6246)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6232)
	at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:175)
	... 8 more
{code}
The caused by incorrect next cell hint, due to we  have skipped the rest of solt that some
key ranges contains all values(EVERYTHING_RANGE) in ScanUtil.setKey method. The next cell
hint of this case is `kv=2018-02-14\x00channel_agg\x00\x82/0:\x00\x00\x00\x00/1550642992223/Put/vlen=4/seqid=5445463`


adding skipped row into nextCellHintMap. Actually,  due to we don't store NULL at the end
of the key for the variable data type,  these keys should be skipped when invokes filterKeyValue, 
because they are smaller than the rest of the positions of the slots.

  was:
Running the below SQL:
{code:sql}
create table if not exists aiolos(
vdate varchar,
tab varchar,
dev tinyint not null,
app varchar,
target varchar,
channel varchar,
one varchar,
two varchar,
count1 integer,
count2 integer,
CONSTRAINT PK PRIMARY KEY (vdate,tab,dev,app,target,channel,one,two));

upsert into aiolos values('2018-02-14','channel_agg',2,null,null,'A004',null,null,2,2);
upsert into aiolos values('2018-02-14','channel_agg',2,null,null,null,null,null,2,2);

SELECT * FROM aiolos WHERE dev = 2 AND vdate BETWEEN '2018-02-10' AND '2019-02-19' AND tab
= 'channel_agg' and channel='A004';
{code}
Throws exception:
{code:java}
Caused by: java.lang.IllegalStateException: The next hint must come after previous hint (prev=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
next=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0, kv=2018-02-14\x00channel_agg\x00\x82/0:\x00\x00\x00\x00/1550642992223/Put/vlen=4/seqid=5445463)
	at org.apache.phoenix.filter.SkipScanFilter.setNextCellHint(SkipScanFilter.java:171)
	at org.apache.phoenix.filter.SkipScanFilter.filterKeyValue(SkipScanFilter.java:145)
	at org.apache.hadoop.hbase.filter.FilterList.filterKeyValue(FilterList.java:264)
	at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:418)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:557)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6308)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6459)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6246)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6232)
	at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:175)
	... 8 more
{code}
The caused by incorrect next cell hint. 

adding skipped row into nextCellHintMap. Actually,  due to we don't store NULL at the end
of the key for the variable data type,  these keys should be skipped when invokes filterKeyValue, 
because they are smaller than the rest of the positions of the slots.


> SkipScan incorrectly filters composite primary key which the key range contains all values
> ------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-5171
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5171
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 5.0.0, 4.14.1
>            Reporter: jaanai
>            Assignee: jaanai
>            Priority: Critical
>             Fix For: 5.1.0
>
>         Attachments: PHOENIX-5171-master-v2.patch, PHOENIX-5171-master.patch
>
>
> Running the below SQL:
> {code:sql}
> create table if not exists aiolos(
> vdate varchar,
> tab varchar,
> dev tinyint not null,
> app varchar,
> target varchar,
> channel varchar,
> one varchar,
> two varchar,
> count1 integer,
> count2 integer,
> CONSTRAINT PK PRIMARY KEY (vdate,tab,dev,app,target,channel,one,two));
> upsert into aiolos values('2018-02-14','channel_agg',2,null,null,'A004',null,null,2,2);
> upsert into aiolos values('2018-02-14','channel_agg',2,null,null,null,null,null,2,2);
> SELECT * FROM aiolos WHERE dev = 2 AND vdate BETWEEN '2018-02-10' AND '2019-02-19' AND
tab = 'channel_agg' and channel='A004';
> {code}
> Throws exception:
> {code:java}
> Caused by: java.lang.IllegalStateException: The next hint must come after previous hint
(prev=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0, next=2018-02-14\x00channel_agg\x00\x82//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0,
kv=2018-02-14\x00channel_agg\x00\x82/0:\x00\x00\x00\x00/1550642992223/Put/vlen=4/seqid=5445463)
> 	at org.apache.phoenix.filter.SkipScanFilter.setNextCellHint(SkipScanFilter.java:171)
> 	at org.apache.phoenix.filter.SkipScanFilter.filterKeyValue(SkipScanFilter.java:145)
> 	at org.apache.hadoop.hbase.filter.FilterList.filterKeyValue(FilterList.java:264)
> 	at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:418)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:557)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6308)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6459)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6246)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6232)
> 	at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:175)
> 	... 8 more
> {code}
> The caused by incorrect next cell hint, due to we  have skipped the rest of solt that
some key ranges contains all values(EVERYTHING_RANGE) in ScanUtil.setKey method. The next
cell hint of this case is `kv=2018-02-14\x00channel_agg\x00\x82/0:\x00\x00\x00\x00/1550642992223/Put/vlen=4/seqid=5445463`

> adding skipped row into nextCellHintMap. Actually,  due to we don't store NULL at the
end of the key for the variable data type,  these keys should be skipped when invokes filterKeyValue, 
because they are smaller than the rest of the positions of the slots.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message