spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuti Awasthi <stutiawas...@hcl.com>
Subject RE: Not able to write output to local filsystem from Standalone mode.
Date Tue, 24 May 2016 12:00:11 GMT
<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">hi Jacek,
<div><br>
</div>
<div>Parent directory already present, its my home directory. Im using Linux (Redhat)
machine 64 bit.</div>
<div>Also I noticed that &quot;test1&quot; folder is created in my master with
subdirectory as &quot;_temporary&quot; which is empty. but on slaves, no such directory
is created under /home/stuti.</div>
<div><br>
</div>
<div>Thanks</div>
<div>Stuti&nbsp;<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF895641" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b>
Jacek Laskowski [jacek@japila.pl]<br>
<b>Sent:</b> Tuesday, May 24, 2016 5:27 PM<br>
<b>To:</b> Stuti Awasthi<br>
<b>Cc:</b> user<br>
<b>Subject:</b> Re: Not able to write output to local filsystem from Standalone
mode.<br>
</font><br>
</div>
<div></div>
<div>
<p dir="ltr">Hi, </p>
<p dir="ltr">What happens when you create the parent directory /home/stuti? I think
the failure is due to missing parent directories. What's the OS?
</p>
<p dir="ltr">Jacek</p>
<div class="gmail_quote">On 24 May 2016 11:27 a.m., &quot;Stuti Awasthi&quot;
&lt;<a href="mailto:stutiawasthi@hcl.com" target="_blank">stutiawasthi@hcl.com</a>&gt;
wrote:<br type="attribution">
<blockquote class="quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div lang="EN-US">
<div>
<p class="MsoNormal">Hi All,<u></u><u></u></p>
<p class="MsoNormal">I have 3 nodes Spark 1.6 Standalone mode cluster with 1 Master
and 2 Slaves. Also Im not having Hadoop as filesystem . Now, Im able to launch shell , read
the input file from local filesystem and perform transformation successfully. When
 I try to write my output in local filesystem path then I receive below error .<u></u><u></u></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<p class="MsoNormal">I tried to search on web and found similar Jira : <a href="https://issues.apache.org/jira/browse/SPARK-2984"
target="_blank">
https://issues.apache.org/jira/browse/SPARK-2984</a> . Even though it shows resolved
for Spark 1.3&#43; but already people have posted the same issue still persists in latest
versions.<u></u><u></u></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<p class="MsoNormal"><b>ERROR<u></u><u></u></b></p>
<p class="MsoNormal">scala&gt; data.saveAsTextFile(&quot;/home/stuti/test1&quot;)<u></u><u></u></p>
<p class="MsoNormal">16/05/24 05:03:42 WARN TaskSetManager: Lost task 1.0 in stage 1.0
(TID 2, server1): java.io.IOException: The temporary job-output directory file:/home/stuti/test1/_temporary
doesn't exist!<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:250)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:244)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:116)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.scheduler.Task.run(Task.scala:89)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<u></u><u></u></p>
<p class="MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at java.lang.Thread.run(Thread.java:745)<u></u><u></u></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<p class="MsoNormal">What is the best way to resolve this issue if suppose I don’t
want to have Hadoop installed OR is it mandatory to have Hadoop to write the output from Standalone
cluster mode.<u></u><u></u></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<p class="MsoNormal">Please suggest.<u></u><u></u></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt">Thanks &amp;Regards<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt">Stuti Awasthi<u></u><u></u></span></p>
<p class="MsoNormal"><u></u>&nbsp;<u></u></p>
</div>
<span style="font-family:'Arial','sans-serif'; color:gray; font-size:7.5pt">
<p class="MsoNormal" style="margin:0in 0in 0pt"><br>
<br>
<font style="font-family:'Arial','sans-serif'; color:gray; font-size:9px">::DISCLAIMER::<br>
----------------------------------------------------------------------------------------------------------------------------------------------------</font></p>
<p class="MsoNormal" style="margin:0in 0in 0pt"><font style="font-family:'Arial','sans-serif';
color:gray; font-size:11px">The contents of this e-mail and any attachment(s) are confidential
and intended for the named recipient(s) only.<br>
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted,
corrupted,
<br>
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e
mail and its contents
<br>
(with or without referred errors) shall therefore not attach any liability on the originator
or HCL or its affiliates.
<br>
Views or opinions, if any, presented in this email are solely those of the author and may
not necessarily reflect the
<br>
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying,
disclosure, modification,
<br>
distribution and / or publication of this message without the prior written consent of authorized
representative of
<br>
HCL is strictly prohibited. If you have received this email in error please delete it and
notify the sender immediately.
<br>
Before opening any email and/or attachments, please check them for viruses and other defects.</font></p>
<p class="MsoNormal" style="margin:0in 0in 0pt"><font style="font-family:'Arial','sans-serif';
color:gray; font-size:11px">----------------------------------------------------------------------------------------------------------------------------------------------------</font></p>
</span></div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message