From dev-return-1544-apmail-systemml-dev-archive=systemml.apache.org@systemml.incubator.apache.org Sat Apr 1 04:14:25 2017 Return-Path: X-Original-To: apmail-systemml-dev-archive@minotaur.apache.org Delivered-To: apmail-systemml-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3B62A1946C for ; Sat, 1 Apr 2017 04:14:25 +0000 (UTC) Received: (qmail 16838 invoked by uid 500); 1 Apr 2017 04:14:25 -0000 Delivered-To: apmail-systemml-dev-archive@systemml.apache.org Received: (qmail 16796 invoked by uid 500); 1 Apr 2017 04:14:25 -0000 Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list dev@systemml.incubator.apache.org Received: (qmail 16782 invoked by uid 99); 1 Apr 2017 04:14:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Apr 2017 04:14:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 59750188A17 for ; Sat, 1 Apr 2017 04:14:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.496 X-Spam-Level: X-Spam-Status: No, score=-1.496 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-2.796, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 0eCQ9Dwm1ze5 for ; Sat, 1 Apr 2017 04:14:22 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id D7E615F177 for ; Sat, 1 Apr 2017 04:14:21 +0000 (UTC) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v314Dx9G049549 for ; Sat, 1 Apr 2017 00:14:21 -0400 Received: from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208]) by mx0a-001b2d01.pphosted.com with ESMTP id 29hy9ja8sf-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Sat, 01 Apr 2017 00:14:20 -0400 Received: from localhost by e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 1 Apr 2017 00:14:19 -0400 Received: from b01cxnp23033.gho.pok.ibm.com (9.57.198.28) by e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sat, 1 Apr 2017 00:14:18 -0400 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v314EKgp44367936 for ; Sat, 1 Apr 2017 04:14:20 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BFF25112051 for ; Sat, 1 Apr 2017 00:14:18 -0400 (EDT) Received: from d50lp02.ny.us.ibm.com (unknown [146.89.104.208]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTPS id 9989311205E for ; Sat, 1 Apr 2017 00:14:18 -0400 (EDT) Received: from localhost by d50lp02.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 1 Apr 2017 00:14:17 -0400 Received: from smtp.notes.na.collabserv.com (192.155.248.91) by d50lp02.ny.us.ibm.com (158.87.18.21) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Sat, 1 Apr 2017 00:14:16 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Sat, 1 Apr 2017 04:14:15 -0000 Received: from us1a3-smtp03.a3.dal06.isc4sb.com (10.106.154.98) by smtp.notes.na.collabserv.com (10.106.227.143) with smtp.notes.na.collabserv.com ESMTP; Sat, 1 Apr 2017 04:14:13 -0000 Received: from us1a3-mail146.a3.dal06.isc4sb.com ([10.146.38.94]) by us1a3-smtp03.a3.dal06.isc4sb.com with ESMTP id 2017040104141304-529664 ; Sat, 1 Apr 2017 04:14:13 +0000 In-Reply-To: To: dev@systemml.incubator.apache.org Subject: Re: Java compiler for code generation From: "Berthold Reinwald" Date: Fri, 31 Mar 2017 20:14:10 -0800 References: MIME-Version: 1.0 X-KeepSent: CFBB87C4:05F16269-882580F5:0014F093; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF766 December 14, 2016 X-LLNOutbound: False X-Disclaimed: 57211 X-TNEFEvaluated: 1 Content-Type: multipart/alternative; boundary="=_alternative 001708E8882580F5_=" x-cbid: 17040104-0044-0000-0000-000002F20566 X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.429909; ST=0; TS=0; UL=0; ISC=; MB=0.022292 X-IBM-SpamModules-Versions: BY=3.00006875; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000207; SDB=6.00841490; UDB=6.00414334; IPR=6.00619568; BA=6.00005250; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014874; XFM=3.00000013; UTC=2017-04-01 04:14:14 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-04-01 02:01:05 - 6.00006530 x-cbparentid: 17040104-9952-0000-0000-000009D7E0F8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006875; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000207; SDB=6.00841490; UDB=6.00414334; IPR=6.00619568; BA=6.00005250; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014874; XFM=3.00000013; UTC=2017-04-01 04:14:19 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-03-31_18:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704010040 --=_alternative 001708E8882580F5_= Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="US-ASCII" Sounds like a good idea. Wrt codegen, in a pure Java scoring environment w/o Spark and Hadoop, will = the dependency on Janino still be there (that question applies to JDK as=20 well), and what is the footprint? Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinwald@us.ibm.com From: Matthias Boehm To: dev@systemml.incubator.apache.org Date: 03/31/2017 08:17 PM Subject: Java compiler for code generation Hi all, currently, our new code generator for operator fusion, uses the programmatic javax.tools.JavaCompiler, which is Java's standard API for compilation. Despite a plan cache that mitigates unnecessary compilation and recompilation overheads, we still see significant end-to-end overhead especially for small input data. Moving forward, I'd like to switch to Janino (org.codehaus.janino.SimpleCompiler), which is a fast in-memory Java compiler with restricted language support. The advantages are (1) Reduced compilation overhead: On end-to-end scenarios for L2SVM, GLM, and MLogreg, Janino improved total javac compilation time from 2.039 to 0.195 (14 operators), from 8.134 to 0.411 (82 operators), and from 4.854=20 to 0.283 (46 operators), respectively. At the same time, there was no measurable impact on runtime efficiency, but even slightly reduced JIT compilation overhead. (2) Removed JDK requirement: Using the standard javax.tools.JavaCompiler requires the existence of a JDK, while Janino only requires a JRE, which means it makes it easier to apply code generation by default. However, I'm raising this here as Janino would add another explicit dependency (with BSD license). Fortunately, Spark also uses Janino for whole-stage-codegen. So we should be able to mark Janino as provided library. The only issue is a pure Hadoop environment, where we still want to use code generation for CP operations. To simplify the build, I could imagine using the javax.tools.JavaCompiler for hadoop execution types, but Janino by default. If you have any concerns, please let me know by Monday; otherwise I'd like to push this change into our upcoming 0.14 release. Regards, Matthias --=_alternative 001708E8882580F5_=--