spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Guller <moham...@glassbeam.com>
Subject RE: saveAsTextFile is not writing to local fs
Date Tue, 02 Feb 2016 01:45:37 GMT
You should not be saving an RDD to local FS if Spark is running on a real cluster. Essentially,
each Spark worker will save the partitions that it processes locally.

Check the directories on the worker nodes and you should find pieces of your file on each
node.

Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>

From: Siva [mailto:sbhavanari@gmail.com]
Sent: Friday, January 29, 2016 5:40 PM
To: Mohammed Guller
Cc: spark users
Subject: Re: saveAsTextFile is not writing to local fs

Hi Mohammed,

Thanks for your quick response. I m submitting spark job to Yarn in "yarn-client" mode on
a 6 node cluster. I ran the job by turning on DEBUG mode. I see the below exception, but this
exception occurred after saveAsTextfile function is finished.

16/01/29 20:26:57 DEBUG HttpParser:
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.read(SocketInputStream.java:190)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at org.spark-project.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
        at org.spark-project.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
        at org.spark-project.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
        at org.spark-project.jetty.http.HttpParser.fill(HttpParser.java:1044)
        at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:280)
        at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.spark-project.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.spark-project.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)
16/01/29 20:26:57 DEBUG HttpParser:
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.read(SocketInputStream.java:190)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at org.spark-project.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
        at org.spark-project.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
        at org.spark-project.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
        at org.spark-project.jetty.http.HttpParser.fill(HttpParser.java:1044)
        at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:280)
        at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.spark-project.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.spark-project.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)
16/01/29 20:26:57 DEBUG HttpParser: HttpParser{s=-14,l=0,c=-3}
org.spark-project.jetty.io.EofException

Do you think this one this causing this?

Thanks,
Sivakumar Bhavanari.

On Fri, Jan 29, 2016 at 3:55 PM, Mohammed Guller <mohammed@glassbeam.com<mailto:mohammed@glassbeam.com>>
wrote:
Is it a multi-node cluster or you running Spark on a single machine?

You can change Spark’s logging level to INFO or DEBUG to see what is going on.

Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>

From: Siva [mailto:sbhavanari@gmail.com<mailto:sbhavanari@gmail.com>]
Sent: Friday, January 29, 2016 3:38 PM
To: spark users
Subject: saveAsTextFile is not writing to local fs

Hi Everyone,

We are using spark 1.4.1 and we have a requirement of writing data local fs instead of hdfs.

When trying to save rdd to local fs with saveAsTextFile, it is just writing _SUCCESS file
in the folder with no part- files and also no error or warning messages on console.

Is there any place to look at to fix this problem?

Thanks,
Sivakumar Bhavanari.

Mime
View raw message