spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prashant Sharma <sharma.prashant.1...@gmail.com>
Subject UnsatisfiedLinkError related to libgfortran when running MLLIB code on RHEL 5.8
Date Tue, 03 Mar 2015 19:21:05 GMT
Hi Folks,

We are trying to run the following code from the spark shell in a CDH 5.3
cluster running on RHEL 5.8.

*spark-shell --master yarn --deploy-mode client --num-executors 15
--executor-cores 6 --executor-memory 12G *
*import org.apache.spark.mllib.recommendation.ALS *
*import org.apache.spark.mllib.recommendation.Rating *
*val users_item_score_clean =
sc.textFile("/tmp/spark_mllib_test").map(_.split(",")) *
*val ratings = users_item_score_clean.map(x=> Rating(x(0).toInt,
x(1).toInt, x(2).toDouble)) *
*val rank = 10 *
*val numIterations = 20 *
*val alpha = 1.0 *
*val lambda = 0.01 *
*val model = ALS.trainImplicit(ratings, rank, numIterations, lambda,alpha) *



We are getting the following error (detailed error is attached in
error.log):


*-- org.jblas ERROR Couldn't load copied link file:
java.lang.UnsatisfiedLinkError: *
*/u08/hadoop/yarn/nm/usercache/sharma.p/appcache/application_1425015707226_0128/*
*container_e12_1425015707226_0128_01_000010/tmp/jblas7605371780736016929libjblas_arch_flavor.so:
libgfortran.so.3: *
*cannot open shared object file: No such file or directory. *

*On Linux 64bit, you need additional support libraries. *
*You need to install libgfortran3. *

*For example for debian or Ubuntu, type "sudo apt-get install
libgfortran3" *

*For more information,
see https://github.com/mikiobraun/jblas/wiki/Missing-Libraries
<https://github.com/mikiobraun/jblas/wiki/Missing-Libraries> *
*15/03/02 14:50:25 ERROR executor.Executor: Exception in task 22.0 in stage
6.0 (TID 374) *
*java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dposv(CII[DII[DII)I *
*at org.jblas.NativeBlas.dposv(Native Method) *
*at org.jblas.SimpleBlas.posv(SimpleBlas.java:369) *


This exact code runs fine on another CDH 5.3 cluster which runs on RHEL
6.5.

"libgfortran.so.3" is not present on the problematic cluster.

*[root@node04 ~]# find / -name libgfortran.so.3 2>/dev/null *


I am able to find "*libgfortran.so.3 *" on the cluster where the above code
works:

*[root@workingclusternode04 ~]# find / -name libgfortran.so.3 2>/dev/null *
*/usr/lib64/libgfortran.so.3 *



The following output shows that the fortran packages are installed on both
the clusters:

*On the cluster where this is not working *

*[root@node04 ~]# yum list | grep -i fortran *
*gcc-gfortran.x86_64 4.1.2-52.el5_8.1 installed *
*libgfortran.i386 4.1.2-52.el5_8.1 installed *
*libgfortran.x86_64 4.1.2-52.el5_8.1 installed *


*On the cluster where the spark job is this working *

*[root@** workingclusternode04** ~]# yum list | grep -i fortran *
*Repository 'bda' is missing name in configuration, using id *
*compat-libgfortran-41.x86_64 4.1.2-39.el6 @bda *
*gcc-gfortran.x86_64 4.4.7-4.el6 @bda *
*libgfortran.x86_64 4.4.7-4.el6 @bda *


Has anybody run into this? Any pointers are much appreciated.

Regards,
Prashant

Mime
View raw message