spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: saveAsTextFile to s3 on spark does not work, just hangs
Date Mon, 25 Aug 2014 22:31:39 GMT
Got it. Another thing that would help is if you spot any exceptions or failed tasks in the
web UI (http://<driver>:4040).

Matei

On August 25, 2014 at 3:07:41 PM, amnonkhen (amnon.is@gmail.com) wrote:

Hi Matei, 
The original issue happened on a spark-1.0.2-bin-hadoop2 installation. 
I will try the synthetic operation and see if I get the same results or not. 
Amnon 


On Mon, Aug 25, 2014 at 11:26 PM, Matei Zaharia [via Apache Spark 
Developers List] <ml-node+s1001551n8000h46@n3.nabble.com> wrote: 

> Was the original issue with Spark 1.1 (i.e. master branch) or an earlier 
> release? 
> 
> One possibility is that your S3 bucket is in a remote Amazon region, which 
> would make it very slow. In my experience though saveAsTextFile has worked 
> even for pretty large datasets in that situation, so maybe there's 
> something else in your job causing a problem. Have you tried other 
> operations on the data, like count(), or saving synthetic datasets (e.g. 
> sc.parallelize(1 to 100*1000*1000, 20).saveAsTextFile(...)? 
> 
> Matei 
> 
> On August 25, 2014 at 12:09:25 PM, amnonkhen ([hidden email] 
> <http://user/SendEmail.jtp?type=node&node=8000&i=0>) wrote: 
> 
> Hi jerryye, 
> Maybe if you voted up my question on Stack Overflow it would get some 
> traction and we would get nearer to a solution. 
> Thanks, 
> Amnon 
> 
> 
> 
> -- 
> View this message in context: 
> http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p7991.html

> 
> Sent from the Apache Spark Developers List mailing list archive at 
> Nabble.com. 
> 
> --------------------------------------------------------------------- 
> To unsubscribe, e-mail: [hidden email] 
> <http://user/SendEmail.jtp?type=node&node=8000&i=1> 
> For additional commands, e-mail: [hidden email] 
> <http://user/SendEmail.jtp?type=node&node=8000&i=2> 
> 
> 
> 
> ------------------------------ 
> If you reply to this email, your message will be added to the discussion 
> below: 
> 
> http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p8000.html

> To unsubscribe from saveAsTextFile to s3 on spark does not work, just 
> hangs, click here 
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7795&code=YW1ub24uaXNAZ21haWwuY29tfDc3OTV8LTkxODIwMjYzNg==>

> . 
> NAML 
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

> 




-- 
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/saveAsTextFile-to-s3-on-spark-does-not-work-just-hangs-tp7795p8008.html

Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message