spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kuromatsu nobuyuki (JIRA)" <>
Subject [jira] [Created] (SPARK-4132) Spark uses incompatible HDFS API
Date Wed, 29 Oct 2014 09:08:33 GMT
kuromatsu nobuyuki created SPARK-4132:

             Summary: Spark uses incompatible HDFS API
                 Key: SPARK-4132
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.1.0
         Environment: Spark1.1.0 on Hadoop1.2.1
CentOS 6.3 64bit
            Reporter: kuromatsu nobuyuki
            Priority: Minor

When I enable event logging and set it to output to HDFS, initialization fails with 'java.lang.ClassNotFoundException'
(see trace below).

I found that an API incompatibility in org.apache.hadoop.fs.permission.FsPermission between
Hadoop 1.0.4 and Hadoop 1.1.0 (and above) causes this error (org.apache.hadoop.fs.permission.FsPermission$2
is used in 1.0.4 but doesn't exist in my 1.2.1 environment).

I think that the Spark jar file pre-built for Hadoop1.X should be built on Hadoop Stable version(Hadoop

2014-10-24 10:43:22,893 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: 
readAndProcess threw exception java.lang.RuntimeException: 
readObject can't find class org.apache.hadoop.fs.permission.FsPermission$2. Count of bytes
read: 0
java.lang.RuntimeException: readObject can't find class org.apache.hadoop.fs.permission.FsPermission$2
        at org.apache.hadoop.ipc.RPC$Invocation.readFields(
        at org.apache.hadoop.ipc.Server$Connection.processData(
        at org.apache.hadoop.ipc.Server$Connection.processOneRpc(
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(
        at org.apache.hadoop.ipc.Server$Listener.doRead(
        at org.apache.hadoop.ipc.Server$Listener$
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.permission.FsPermission$2
        at Method)
        at java.lang.ClassLoader.loadClass(
        at sun.misc.Launcher$AppClassLoader.loadClass(
        at java.lang.ClassLoader.loadClass(
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(
        at org.apache.hadoop.conf.Configuration.getClassByName(
        ... 9 more

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message