spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uddin, Nasir M." <>
Subject Spark 1.0: Reading JSON LZH Compressed File
Date Mon, 30 Jun 2014 20:09:50 GMT

Spark 1.0 has been installed as Standalone - But it can't read any compressed (CMX/Snappy)
and Sequence file residing on HDFS. The key notable message is: "Unable to load native-hadoop
library.....". Other related messages are -

Caused by: java.lang.IllegalStateException: Cannot load
without native library! at<clinit>(

Here is the core-site.xml's key part:

Here is the
export SCALA_HOME=/opt/spark/scala-2.11.1
export JAVA_HOME=/opt/spark/jdk1.7.0_55
export SPARK_HOME=/opt/spark/spark-0.9.1-bin-hadoop2
export ADD_JARS=/opt/IHC/lib/compression.jar
export SPARK_CLASSPATH=/opt/IHC/lib/compression.jar
export SPARK_LIBRARY_PATH=/opt/IHC/lib/native/Linux-amd64-64/
export HADOOP_CONF_DIR=/opt/IHC/hadoop-conf

Note: CMX is an IBM branded splittable LZO based compression codec.

Any help is appreciated.

DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed. If you have received
this email in error, please notify us immediately and delete the email and any attachments
from your system. The recipient should check this email and any attachments for the presence
of viruses.  The company accepts no liability for any damage caused by any virus transmitted
by this email.

View raw message