spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 정재부 <itsjb.j...@samsung.com>
Subject Re: Shuffle write increases in spark 1.2
Date Mon, 05 Jan 2015 05:07:23 GMT
<HTML><HEAD>
<META content="text/html; charset=euc-kr" http-equiv=Content-Type>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY style="-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; -ms-word-wrap:
break-word">
<P>Sure, here is a ticket. <A href="https://issues.apache.org/jira/browse/SPARK-5081">https://issues.apache.org/jira/browse/SPARK-5081</A></P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Josh Rosen&lt;rosenville@gmail.com&gt;</P>
<P><B>Date</B> : 2015-01-05 06:14 (GMT+09:00)</P>
<P><B>Title</B> : Re: Shuffle write increases in spark 1.2</P>
<P>&nbsp;</P>
<STYLE>BODY {
	FONT-SIZE: 13px; FONT-FAMILY: Helvetica,Arial
}
</STYLE>

<DIV id=bloop_customfont style="FONT-SIZE: 13px; FONT-FAMILY: Helvetica,Arial; COLOR: ;
MARGIN: 0px">If you have a small reproduction for this issue, can you open a ticket at&nbsp;<A
href="https://issues.apache.org/jira/browse/SPARK">https://issues.apache.org/jira/browse/SPARK</A>&nbsp;?
</DIV><BR>
<DIV id=bloop_sign_1420406071401256192 class=bloop_sign>
<DIV style="FONT-SIZE: 13px; FONT-FAMILY: helvetica,arial"><BR></DIV></DIV><BR>
<P style="COLOR: rgb(0,0,0)">On December 29, 2014 at 7:10:02 PM, Kevin Jung (<A href="mailto:itsjb.jung@samsung.com">itsjb.jung@samsung.com</A>)
wrote:</P>
<BLOCKQUOTE class=clean_bq type="cite"><SPAN>
<DIV>
<DIV></DIV>
<DIV>Hi all, <BR>The size of shuffle write showing in spark web UI is mush different
when I <BR>execute same spark job on same input data(100GB) in both spark 1.1 and spark
<BR>1.2. <BR>At the same sortBy stage, the size of shuffle write is 39.7GB in
spark 1.1 <BR>but 91.0GB in spark 1.2. <BR>I set spark.shuffle.manager option
to hash because it's default value is <BR>changed but spark 1.2 writes larger file than
spark 1.1. <BR>Can anyone tell me why this happened? <BR><BR>Thanks <BR>Kevin
<BR><BR><BR><BR>-- <BR>View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-write-increases-in-spark-1-2-tp20894.html
<BR>Sent from the Apache Spark User List mailing list archive at Nabble.com. <BR><BR>---------------------------------------------------------------------
<BR>To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <BR>For additional
commands, e-mail: user-help@spark.apache.org <BR><BR></DIV></DIV></SPAN></BLOCKQUOTE></BODY></HTML><img
src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=e6c8c890fe3919e244c20355f093e54383719247f2e786e6fc144f529aba3c5c911590d97d85df7fd4b5cb504b28632862e1ac75b522795a07805447a154a46fcf878f9a26ce15a0'
border=0 width=0 height=0 style='display:none'>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Mime
View raw message