hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: snappy error during completebulkload
Date Mon, 09 Jan 2012 16:11:27 GMT
Sounds like the snappy library isn't installed on the machine or that java can't find the native

library.  I think you need the hadoop-0.20-native installed (via apt or yum).

~Jeff

On 1/9/2012 3:42 AM, Oliver Meyn (GBIF) wrote:
> Hi all,
>
> I'm trying to do bulk loading into a table with snappy compression enabled and I'm getting
an exception complaining about missing native snappy library, namely:
>
> 12/01/09 11:16:53 WARN snappy.LoadSnappy: Snappy native library not loaded
> Exception in thread "main" java.io.IOException: java.lang.RuntimeException: native snappy
library not available
> 	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:89)
>
> First, to be clear, everything in this chain works fine if I don't use compression, and
using hbase shell to 'put' to the compression-enable table also works fine.
>
> Here's what I'm doing:
> - use importtsv to generate the hfiles.  I'm passing -Dhfile.compression=snappy on the
command line, as per a mailing list email from lars g i found while googling.  The import
runs without errors, but I don't know how to test whether the hfiles are actually compressed.
> - use completebulkload to move the hfiles into the cluster.  This is where I get the
exception.  I'm running the command from my os x workstation, targeting a remote hdfs and
hbase (both running on the same cluster).
>
> My environment:
> - hbase/hadoop are cdh3u2, fully distributed
> - workstation is os x, 10.6
>
> It seems really weird that compression (native compression even moreso) should be required
by a command that is in theory moving files from one place on a remote filesystem to another.
 Any light shed would be appreciated.
>
> Thanks,
> Oliver
> --
> Oliver Meyn
> Software Developer
> Global Biodiversity Information Facility (GBIF)
> +45 35 32 15 12
> http://www.gbif.org
>

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com


Mime
View raw message