From user-return-37429-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Jul 8 01:39:29 2015 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4C98318A6F for ; Wed, 8 Jul 2015 01:39:29 +0000 (UTC) Received: (qmail 94892 invoked by uid 500); 8 Jul 2015 01:39:25 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 94803 invoked by uid 500); 8 Jul 2015 01:39:25 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 94793 invoked by uid 99); 8 Jul 2015 01:39:25 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2015 01:39:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5C3FE182839 for ; Wed, 8 Jul 2015 01:39:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.802 X-Spam-Level: * X-Spam-Status: No, score=1.802 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, UNPARSEABLE_RELAY=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id MkVANFjSUkD5 for ; Wed, 8 Jul 2015 01:39:19 +0000 (UTC) Received: from f02smg01.tsmc.com.tw (f02smg01.tsmc.com.tw [203.66.245.11]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTP id 51E612122D for ; Wed, 8 Jul 2015 01:39:17 +0000 (UTC) X-AuditID: 0a3c2143-b7f7b6d00000449a-a7-559c7f3c5aa2 Received: from a08aveamail04.tw.tsmc.com ( [10.211.108.105]) by f02smg01.tsmc.com.tw (Symantec Messaging Gateway) with SMTP id B2.EA.17562.C3F7C955; Wed, 8 Jul 2015 09:39:08 +0800 (CST) Received: from a08avmrelay01.tsmc.com.tw (a08avmrelay01.tsmc.com.tw) by a08aveamail04.tw.tsmc.com (Clearswift SMTPRS 5.6.0) with ESMTP id ; Wed, 8 Jul 2015 09:39:08 +0800 Received: from ev14s04.tsmc.com.tw (ev14s04.tsmc.com.tw [10.168.48.59]) by a08avmrelay01.tsmc.com.tw (8.14.4/8.13.8) with ESMTP id t681d8EE002202; Wed, 8 Jul 2015 09:39:08 +0800 In-Reply-To: References: Subject: Re: SparkSQL OOM issue X-KeepSent: 6FBEDA48:C3166210-48257E7C:0008FEB9; type=4; name=$KeepSent To: Xiaoyu Ma Cc: user@spark.apache.org, mike_shen@tsmc.com, linchzs@tsmc.com X-Mailer: Lotus Notes Release 8.5.3FP2 July 03, 2012 Message-ID: From: shshann@tsmc.com Date: Wed, 8 Jul 2015 09:39:05 +0800 X-MIMETrack: Serialize by Router on EV14S04/TSMC(Release 9.0.1FP3HF517 | April 28, 2015) at 2015/07/08 09:39:08 MIME-Version: 1.0 Content-type: text/plain; charset=Big5 Content-transfer-encoding: base64 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprOIsWRmVeSWpSXmKPExsXCZaPorGtTPyfU4OhEG4sv+wIcGD2+POhg DmCMUrRJSc3JLEst0rezSaosSCwu1k1OU0jMybFVKikqTVXSt0tQzJg6pYWxoMm6YuWl00wN jHssuxg5OSQETCSW/prFDmGLSVy4t56ti5GLQ0hgD6PEncdtLBDOQ0aJR/NnQ2W2Mkq8a/vE CtLCKeAi8ebGBqiqDkaJvp17wGYJC8hLbLhzmQ1irqfEho1nmUFsEQFtiSk3DoLZzAL2ErdW HmOEuWP5w49MIDavQIBE+7F7YDYb0E2NZ5aCzWQRUJF4u3IX1K2FEju3vmKGqBeUODnzCQvE TDWJ3rY+dghbUWLaogbGCYzCs5CUzUJSNgtJ2QJG5lWMImkGRsW56QaGeiXFucl6yfm5eiXl mxihob+D8eEBi0OMVUCnTGSWEk3OB8ZOXkm8obGxmZmhuamlkbmxCVWElcR5+3ZPDhUSSE8s Sc1OTS1ILYovKs1JLT7EyMTBKdXA2Kr99ZLirow7vrGLmr17DT+bMv5Rcdde2XVv9+VH6/Yl CS89+DnrYevU9X0mHMu3JqfcEJu+fcuHbOaJ7Y8bTrOy35ewnLxZZ5nMx22TD/XErzMW6ja7 +Fo8WD/xmblx3/TG9tD/55yCdz7oCG6PjJi35JOZbUmxa62vnIpqR/hKXiu2CaZGSizFGYmG WsxFxYkACXiLi6oCAAA= SGkgU2hhd24sDQoNClRoYW5rIGFsb3QgdGhhdCdzIGFjdHVhbGx5IHRoZSBsYXN0IHBhcmFtZXRl ciB3ZSBvdmVybG9va2VkISENCkknbSBhYmxlIHRvIHJ1biB0aGUgc2FtZSBzcWwgb24gc3Bhcmsg bm93IGlmIEkgc2V0IHRoZSBzcGFyay5kcml2ZXIubWVtb29yeQ0KbGFyZ2VyLA0KdGhhbmtzIGFn YWluISENCg0KLS0NCkJlc3QgUmVnYXJkcywNCkZlbGljaWEgU2hhbm4NCrPmrnayWw0KKzg4Ni0z LTU2MzY2ODggRXh0LiA3MTI0MzAwDQoNCg0KfC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0+DQp8ICAgICAgICAgICAgWGlhb3l1IE1hICAgICAgICAgICAgICAgIHwNCnwgICAg ICAgICAgICA8aHptYXhpYW95dUBjb3JwLm5ldGVhc2UufA0KfCAgICAgICAgICAgIGNvbT4gICAg ICAgICAgICAgICAgICAgICB8DQp8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwNCnwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQp8ICAgICAgICAgICAgMjAxNS8wNy8wNyCkVaTI IDA0OjAzICAgIHwNCnwtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tPg0KICA+ LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS18DQogIHwgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVG98DQogIHwgICAgICAgIHNoc2hhbm5AdHNt Yy5jb20gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNjfA0KICB8 ICAgICAgICB1c2VyQHNwYXJrLmFwYWNoZS5vcmcsIG1pa2Vfc2hlbkB0c21jLmNvbSwgbGluY2h6 c0B0c21jLmNvbSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogIHwgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgU3ViamVjdHwNCiAgfCAgICAgICAgUmU6IFNwYXJrU1FMIE9PTSBpc3N1ZSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogIHwgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogIHwgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgPi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tfA0KDQoN Cg0KDQpIaSwNCldoZXJlIGRpZCBPT00gaGFwcGVuZWQ/DQpJbiBEcml2ZXIgb3IgZXhlY3V0b3I/ DQpTb21ldGltZXMgU3BhcmtTUUwgRHJpdmVyIE9PTSBvbiB0YWJsZXMgd2l0aCBsYXJnZSBudW1i ZXIgcGFydGl0aW9ucy4NCklmIHNvLCB5b3UgbWlnaHQgd2FudCB0byBpbmNyZWFzZSBpdCBpbiBz cGFyay1kZWZhdWx0cy5jb25mDQpzcGFyay5kcml2ZXIubWVtb3J5DQoNClNoYXduDQoNCg0KDQo+ IE9uIEp1bCA3LCAyMDE1LCBhdCAzOjU4IFBNLCBzaHNoYW5uQHRzbWMuY29tIHdyb3RlOg0KPg0K Pg0KPiBEZWFyIGFsbCwNCj4NCj4gV2UndmUgdHJpZWQgdG8gdXNlIHNwYXJrU3FsIHRvIGRvIHNv bWUgaW5zZXJ0IGZyb20gQSB0YWJsZSB0byBCIHRhYmxlDQo+IGFjdGlvbiB3aGVyZSB1c2luZyB0 aGUgZXhhY3Qgc2FtZSBTUUwgc2NyaXB0LA0KPiBoaXZlIGlzIGFibGUgdG8gZmluaXNoIGl0IGJ1 dCBTcGFyayAxLjMuMSB3b3VsZCBhbHdheXMgZW5kIHdpdGggT09NDQppc3N1ZTsNCj4gd2UgdHJp ZWQgc2V2ZXJhbCBjb25maWd1cmF0aW9uIGluY2x1ZGluZzoNCj4NCj4gLS1leGVjdXRvci1jb3Jl cyAyDQo+IC0tbnVtLWV4ZWN1dG9ycyAzMDANCj4gLS1leGVjdXRvci1tZW1vcnkgN2cNCj4gc2Nv bmYuc2V0KCJzcGFyay5zdG9yYWdlLm1lbW9yeUZyYWN0aW9uIiwgIjAiKQ0KPg0KPiBidXQgbm9u ZSBvZiB0aGVtIGNhbiBjaGFuZ2UgdGhlIHJlc3VsdCBvZiBlcnJvcjoNCj4gamF2YS5sYW5nLk91 dE9mTWVtb3J5RXJyb3I6IEdDIG92ZXJoZWFkIGxpbWl0IGV4Y2VlZGVkDQo+IGlzIHRoZXJlIGFu eSBvdGhlciBjb25maWd1cmF0aW9uIHdlIGNhbiBtYWtlPyBUaGFua3MhDQo+DQotLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0NCg0KPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIFRTTUMgUFJPUEVSVFkNCg0KPiBUaGlzIGVtYWlsIGNvbW11bmljYXRpb24g KGFuZCBhbnkgYXR0YWNobWVudHMpIGlzIHByb3ByaWV0YXJ5IGluZm9ybWF0aW9uDQoNCj4gZm9y IHRoZSBzb2xlIHVzZSBvZiBpdHMNCg0KPiBpbnRlbmRlZCByZWNpcGllbnQuIEFueSB1bmF1dGhv cml6ZWQgcmV2aWV3LCB1c2Ugb3IgZGlzdHJpYnV0aW9uIGJ5DQphbnlvbmUNCj4gb3RoZXIgdGhh biB0aGUgaW50ZW5kZWQNCg0KPiByZWNpcGllbnQgaXMgc3RyaWN0bHkgcHJvaGliaXRlZC4gIElm IHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCByZWNpcGllbnQsDQoNCj4gcGxlYXNlIG5vdGlmeSB0 aGUgc2VuZGVyIGJ5DQoNCj4gcmVwbHlpbmcgdG8gdGhpcyBlbWFpbCwgYW5kIHRoZW4gZGVsZXRl IHRoaXMgZW1haWwgYW5kIGFueSBjb3BpZXMgb2YgaXQNCg0KPiBpbW1lZGlhdGVseS4gVGhhbmsg eW91Lg0KDQo+DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCg0KPg0KPg0KPg0KPg0KPg0KPiAtLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0NCj4gVG8gdW5zdWJzY3JpYmUsIGUtbWFpbDogdXNlci11bnN1YnNjcmliZUBzcGFyay5h cGFjaGUub3JnDQo+IEZvciBhZGRpdGlvbmFsIGNvbW1hbmRzLCBlLW1haWw6IHVzZXItaGVscEBz cGFyay5hcGFjaGUub3JnDQo+DQoNCg0KDQoNCiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0gDQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUU01DIFBS T1BFUlRZICAgICAgIA0KIFRoaXMgZW1haWwgY29tbXVuaWNhdGlvbiAoYW5kIGFueSBhdHRhY2ht ZW50cykgaXMgcHJvcHJpZXRhcnkgaW5mb3JtYXRpb24gICANCiBmb3IgdGhlIHNvbGUgdXNlIG9m IGl0cyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg DQogaW50ZW5kZWQgcmVjaXBpZW50LiBBbnkgdW5hdXRob3JpemVkIHJldmlldywgdXNlIG9yIGRp c3RyaWJ1dGlvbiBieSBhbnlvbmUgIA0KIG90aGVyIHRoYW4gdGhlIGludGVuZGVkICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiByZWNpcGllbnQg aXMgc3RyaWN0bHkgcHJvaGliaXRlZC4gIElmIHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCByZWNp cGllbnQsICAgDQogcGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGJ5ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KIHJlcGx5aW5nIHRvIHRoaXMgZW1haWws IGFuZCB0aGVuIGRlbGV0ZSB0aGlzIGVtYWlsIGFuZCBhbnkgY29waWVzIG9mIGl0ICAgICANCiBp bW1lZGlhdGVseS4gVGhhbmsgeW91LiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgDQogLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tIA0KDQo= DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0NClRvIHVuc3Vic2NyaWJlLCBlLW1haWw6IHVzZXItdW5zdWJz Y3JpYmVAc3BhcmsuYXBhY2hlLm9yZw0KRm9yIGFkZGl0aW9uYWwgY29tbWFuZHMsIGUtbWFp bDogdXNlci1oZWxwQHNwYXJrLmFwYWNoZS5vcmcNCg0K