spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hakanilter <hakanil...@gmail.com>
Subject Re: Problem with loading files: Loss was due to java.io.EOFException java.io.EOFException
Date Wed, 21 May 2014 22:19:35 GMT
The problem is solved after hadoop-core dependency added. But I think there
is a misunderstanding about local files. I found this one:

"Note that if you've connected to a Spark master, it's possible that it will
attempt to load the file on one of the different machines in the cluster, so
make sure it's available on all the cluster machines. In general, in future
you will want to put your data in HDFS, S3, or similar file systems to avoid
this problem."

http://docs.sigmoidanalytics.com/index.php/Using_the_Spark_Shell

This means that you can't use local files with spark. I don't understand
why, because after calling addFile() or textFile(), the file can be
downloaded by every node on the cluster and became accessible. 

Anyway, if you got "Loss was due to java.io.EOFException", you have to make
sure that hadoop libs are available.

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>0.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.0.0-mr1-cdh4.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.0.0-cdh4.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.0.0-cdh4.6.0</version>
        </dependency>

Cheers!




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-loading-files-Loss-was-due-to-java-io-EOFException-java-io-EOFException-tp6090p6201.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message