phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PHOENIX-1232) java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I on JOIN using org.apache.phoenix.pig.PhoenixHBaseLoader
Date Wed, 03 Sep 2014 01:02:51 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119095#comment-14119095
] 

Andrew Purtell edited comment on PHOENIX-1232 at 9/3/14 1:02 AM:
-----------------------------------------------------------------

HashCacheClient wants to use snappy compression internally during serialization. Looks like
the snappy native library is not available on the mapreduce cluster where the Pig job is executing.

Phoenix probably should not hardcode compression codec selection. It might be useful to replace
use of Snappy static methods with a Compression singleton, where the actual algorithm is selected
with site configuration and runtime environment testing using reflection. Alternative algorithms
may demonstrate superior performance, e.g. https://gist.github.com/kevsmith/4217004 . If snappy
or the desired codec (given by configuration) is not available, we can fall back on gzip.
Although a slower alternative we know the JRE will always support it. Or no compression.


was (Author: apurtell):
HashCacheClient wants to use snappy compression internally during serialization. Looks like
the snappy native library is not available on the mapreduce cluster where the Pig job is executing.

Phoenix probably should not hardcode compression codec selection. It might be useful to replace
use of Snappy static methods with a Compression singleton, where the actual algorithm is selected
with site configuration and runtime environment testing (via reflection). Alternative algorithms
may demonstrate superior performance, e.g. https://gist.github.com/kevsmith/4217004 . If snappy
is not available, we can fall back on gzip. Although a slower alternative we know the JRE
will always support it. Or no compression.

> java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
on JOIN using org.apache.phoenix.pig.PhoenixHBaseLoader
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-1232
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1232
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.0.0
>            Reporter: Brian Johnson
>
> When using the using the pig PhoenixHBaseLoader, I get the following error when doing
a JOIN on two tables. The tables are defined as:
> 1.9.3-p194 :002 > describe 'TEST'
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/hadoop/.versions/hbase-0.98.5/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2014-09-02 20:11:25,402 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
> DESCRIPTION                                                                         
                                                                                         ENABLED
                                                                                        
>  'TEST', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|1|',
coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregate true             
                                                                              
>  RegionObserver|1|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEnd                    
                                                                            
>  pointImpl|1|', coprocessor$5 => '|org.apache.phoenix.hbase.index.Indexer|1073741823|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,ind
                                                                                         
      
>  ex.builder=org.apache.phoenix.index.PhoenixIndexBuilder', coprocessor$6 => '|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|1|'},
{NAME => '0', DATA_BLOCK_ENCODING                                                     
                                           
>   => 'FAST_DIFF', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION
=> 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS
=> 'true'                                                                             
                   
>  , BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME
=> 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE
=> '0', COMPRESS                                                                      
                          
>  ION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS
=> 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
                                                                                         
           
> 1 row(s) in 1.4710 seconds
> 1.9.3-p194 :003 > describe 'TEST2'
> DESCRIPTION                                                                         
                                                                                         ENABLED
                                                                                        
>  'TEST2', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|1|',
coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregat true              
                                                                             
>  eRegionObserver|1|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEn                     
                                                                           
>  dpointImpl|1|', coprocessor$5 => '|org.apache.phoenix.hbase.index.Indexer|1073741823|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,in
                                                                                         
      
>  dex.builder=org.apache.phoenix.index.PhoenixIndexBuilder', coprocessor$6 => '|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|1|'},
{NAME => '0', DATA_BLOCK_ENCODIN                                                      
                                          
>  G => 'FAST_DIFF', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION
=> 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS
=> 'true                                                                              
                  
>  ', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME
=> 'cf1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE
=> '0', COMPRES                                                                       
                         
>  SION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0',
KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}                                                                            
                        
> 1 row(s) in 0.0240 seconds
> and the query is
> SELECT TEST.ID FROM TEST LEFT OUTER JOIN TEST2 ON TEST.ID = TEST2.ID
> It works fine from sqlline.py and neither table has compression enabled.
> Caused by: java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
> 	at org.xerial.snappy.SnappyNative.maxCompressedLength(Native Method)
> 	at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:316)
> 	at org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:119)
> 	at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:77)
> 	at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:119)
> 	at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:114)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> 	at java.lang.Thread.run(Thread.java:695)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message