spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 정재부 <>
Subject Re: Shuffle write increases in spark 1.2
Date Mon, 05 Jan 2015 05:07:23 GMT
<META content="text/html; charset=euc-kr" http-equiv=Content-Type>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY style="-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; -ms-word-wrap:
<P>Sure, here is a ticket. <A href=""></A></P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Josh Rosen&lt;;</P>
<P><B>Date</B> : 2015-01-05 06:14 (GMT+09:00)</P>
<P><B>Title</B> : Re: Shuffle write increases in spark 1.2</P>
	FONT-SIZE: 13px; FONT-FAMILY: Helvetica,Arial

<DIV id=bloop_customfont style="FONT-SIZE: 13px; FONT-FAMILY: Helvetica,Arial; COLOR: ;
MARGIN: 0px">If you have a small reproduction for this issue, can you open a ticket at&nbsp;<A
<DIV id=bloop_sign_1420406071401256192 class=bloop_sign>
<DIV style="FONT-SIZE: 13px; FONT-FAMILY: helvetica,arial"><BR></DIV></DIV><BR>
<P style="COLOR: rgb(0,0,0)">On December 29, 2014 at 7:10:02 PM, Kevin Jung (<A href=""></A>)
<BLOCKQUOTE class=clean_bq type="cite"><SPAN>
<DIV>Hi all, <BR>The size of shuffle write showing in spark web UI is mush different
when I <BR>execute same spark job on same input data(100GB) in both spark 1.1 and spark
<BR>1.2. <BR>At the same sortBy stage, the size of shuffle write is 39.7GB in
spark 1.1 <BR>but 91.0GB in spark 1.2. <BR>I set spark.shuffle.manager option
to hash because it's default value is <BR>changed but spark 1.2 writes larger file than
spark 1.1. <BR>Can anyone tell me why this happened? <BR><BR>Thanks <BR>Kevin
<BR><BR><BR><BR>-- <BR>View this message in context:
<BR>Sent from the Apache Spark User List mailing list archive at <BR><BR>---------------------------------------------------------------------
<BR>To unsubscribe, e-mail: <BR>For additional
commands, e-mail: <BR><BR></DIV></DIV></SPAN></BLOCKQUOTE></BODY></HTML><img
border=0 width=0 height=0 style='display:none'>
To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message