From dev-return-1774-apmail-systemml-dev-archive=systemml.apache.org@systemml.incubator.apache.org Tue May 16 18:05:14 2017 Return-Path: X-Original-To: apmail-systemml-dev-archive@minotaur.apache.org Delivered-To: apmail-systemml-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CE2C118F32 for ; Tue, 16 May 2017 18:05:14 +0000 (UTC) Received: (qmail 85401 invoked by uid 500); 16 May 2017 18:05:14 -0000 Delivered-To: apmail-systemml-dev-archive@systemml.apache.org Received: (qmail 85361 invoked by uid 500); 16 May 2017 18:05:14 -0000 Mailing-List: contact dev-help@systemml.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.incubator.apache.org Delivered-To: mailing list dev@systemml.incubator.apache.org Received: (qmail 85120 invoked by uid 99); 16 May 2017 18:05:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 May 2017 18:05:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E12C7D02D2 for ; Tue, 16 May 2017 18:05:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.997 X-Spam-Level: X-Spam-Status: No, score=-0.997 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-2.796, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Va0PPEA9UtgX for ; Tue, 16 May 2017 18:05:11 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7FF735F3BC for ; Tue, 16 May 2017 18:05:10 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v4GHxSAm053663 for ; Tue, 16 May 2017 14:05:08 -0400 Received: from e15.ny.us.ibm.com (e15.ny.us.ibm.com [129.33.205.205]) by mx0a-001b2d01.pphosted.com with ESMTP id 2ag5p2j96m-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 16 May 2017 14:05:08 -0400 Received: from localhost by e15.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 May 2017 14:05:06 -0400 Received: from b01cxnp23032.gho.pok.ibm.com (9.57.198.27) by e15.ny.us.ibm.com (146.89.104.202) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 16 May 2017 14:05:04 -0400 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v4GI54V343909342 for ; Tue, 16 May 2017 18:05:04 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AA9F52804C for ; Tue, 16 May 2017 14:04:58 -0400 (EDT) Received: from d50lp01.ny.us.ibm.com (unknown [146.89.104.207]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTPS id 9FBC52804A for ; Tue, 16 May 2017 14:04:58 -0400 (EDT) Received: from localhost by d50lp01.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 May 2017 14:05:03 -0400 Received: from smtp.notes.na.collabserv.com (192.155.248.75) by d50lp01.ny.us.ibm.com (158.87.18.20) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Tue, 16 May 2017 14:05:01 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Tue, 16 May 2017 18:05:00 -0000 Received: from us1a3-smtp01.a3.dal06.isc4sb.com (10.106.154.95) by smtp.notes.na.collabserv.com (10.106.227.123) with smtp.notes.na.collabserv.com ESMTP; Tue, 16 May 2017 18:04:58 -0000 Received: from us1a3-mail56.a3.dal09.isc4sb.com ([10.142.3.44]) by us1a3-smtp01.a3.dal06.isc4sb.com with ESMTP id 2017051618045751-721433 ; Tue, 16 May 2017 18:04:57 +0000 MIME-Version: 1.0 To: dev@systemml.incubator.apache.org In-Reply-To: Subject: Re: Improve SystemML execution speed in Spark From: "Niketan Pansare" Date: Tue, 16 May 2017 18:04:57 +0000 X-Mailer: IBM Traveler 9.0.1.18 Build 201704241101_20 on behalf of device with id PL6...DNO, type 445 (ios) and description iPhone 7:Apple-iPhone9C3/1404.27 (OS 10) X-KeepSent: 434A9487:6C7C30E8-00258122:006354AD; type=4; name=$KeepSent X-LLNOutbound: False X-Disclaimed: 48299 X-TNEFEvaluated: 1 Content-type: multipart/alternative; Boundary="0__=8FBB0BB1DFF0D23D8f9e8a93df938690918c8FBB0BB1DFF0D23D" Content-Disposition: inline x-cbid: 17051618-0036-0000-0000-000002082290 X-IBM-SpamModules-Scores: BY=0.248957; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.417846; ST=0; TS=0; UL=0; ISC=; MB=0.261727 X-IBM-SpamModules-Versions: BY=3.00007073; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000212; SDB=6.00861366; UDB=6.00427217; IPR=6.00641029; BA=6.00005352; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015482; XFM=3.00000015; UTC=2017-05-16 18:04:59 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-05-16 17:20:10 - 6.00006751 x-cbparentid: 17051618-3816-0000-0000-00000F55CB9C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007073; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000212; SDB=6.00861366; UDB=6.00427216; IPR=6.00641029; BA=6.00005352; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015482; XFM=3.00000015; UTC=2017-05-16 18:05:05 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705160142 --0__=8FBB0BB1DFF0D23D8f9e8a93df938690918c8FBB0BB1DFF0D23D Content-Transfer-Encoding: quoted-printable Content-type: text/plain; charset=US-ASCII Thanks Arijit for sharing your solution. Appreciate your participation :) > On May 15, 2017, at 11:29 AM, arijit chakraborty wrote: > > Hi Niketan, > > > I came up a round about to increase my speed of systemML execution through spark. I'm using these 2 commands. > > > SparkContext.setSystemProperty('spark.executor.memory', '15g') > > sc =3D SparkContext("local[*]", "test") > > > > This not only increased my speed drastically (a code which took to run around 12 mins now running in 3 mins), but also can run with much larger dataset. > > > Also as a technique for debugging I'm now using "print" statement in code, so the output now showing in cmd. This will tell me at which step code failed. > > > So thank yo for nudging me towards looking at cmd. > > > Sharing all these information here for letting you know that I could solve the issue raised here. And thank you for that. Secondly, can help someone if they have the same issue. > > > Thanks again! > > Arijit > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F > From: arijit chakraborty > Sent: Friday, May 12, 2017 8:41:26 PM > To: dev@systemml.incubator.apache.org > Subject: Re: Improve SystemML execution speed in Spark > > HI Niketan, > > > You are right. I was actually testing 2 seperate code in the same environment. Maybe that's why having this high stats were coming. Please find the stats in standalone environment. > > > SystemML Statistics: > Total elapsed time: 0.000 sec. > Total compilation time: 0.000 sec. > Total execution time: 0.000 sec. > Number of compiled Spark inst: 0. > Number of executed Spark inst: 0. > Cache hits (Mem, WB, FS, HDFS): 2/0/0/0. > Cache writes (WB, FS, HDFS): 0/0/0. > Cache times (ACQr/m, RLS, EXP): 0.000/0.001/0.000/0.000 sec. > HOP DAGs recompiled (PRED, SB): 0/0. > HOP DAGs recompile time: 0.000 sec. > Spark ctx create time (lazy): 0.000 sec. > Spark trans counts (par,bc,col):0/0/0. > Spark trans times (par,bc,col): 0.000/0.000/0.000 secs. > Total JIT compile time: 0.0 sec. > Total JVM GC count: 168. > Total JVM GC time: 0.345 sec. > Heavy hitter instructions (name, time, count): > -- 1) ba+* 0.010 sec 1 > -- 2) rand 0.007 sec 2 > -- 3) createvar 0.001 sec 3 > -- 4) rmvar 0.000 sec 3 > -- 5) cpvar 0.000 sec 1 > > > > Thanks a lot! > > Arijit > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F > From: Niketan Pansare > Sent: Friday, May 12, 2017 8:04:41 PM > To: dev@systemml.incubator.apache.org > Subject: Re: Improve SystemML execution speed in Spark > > > > Hi Arijit > > The second statistics is bit surprising. Is it possible for you to share > the exact setup may be via git: bash script (for example run.sh with Jvm > sizes, number of executors, etc), pyspark script (containing mlcontext > code) and DML script. This will help me reproduce those numbers on my > cluster. > > May be Matthias can comment on buildTree part. > > Thanks > > Niketan > >> On May 12, 2017, at 5:07 AM, arijit chakraborty > wrote: >> >> Hi Niketan, >> >> >> Sorry for asking the nuisance question. Please find the output from > "setStatistics(True)" for my model. >> >> >> SystemML Statistics: >> >> Total elapsed time: 0.000 sec. >> >> Total compilation time: 0.000 sec. >> >> Total execution time: 0.000 sec. >> >> Number of compiled Spark inst: 583. >> >> Number of executed Spark inst: 29. >> >> Cache hits (Mem, WB, FS, HDFS): 180563/0/0/3. >> >> Cache writes (WB, FS, HDFS): 36070/0/0. >> >> Cache times (ACQr/m, RLS, EXP): 4.349/0.077/0.729/0.000 sec. >> >> HOP DAGs recompiled (PRED, SB): 0/496. >> >> HOP DAGs recompile time: 4.976 sec. >> >> Functions recompiled: 46. >> >> Functions recompile time: 3.016 sec. >> >> Spark ctx create time (lazy): 0.008 sec. >> >> Spark trans counts (par,bc,col):29/0/3. >> >> Spark trans times (par,bc,col): 0.021/0.000/4.178 secs. >> >> ParFor loops optimized: 1. >> >> ParFor optimize time: 0.158 sec. >> >> ParFor initialize time: 0.070 sec. >> >> ParFor result merge time: 0.001 sec. >> >> ParFor total update in-place: 0/4446/53253 >> >> Total JIT compile time: 1.083 sec. >> >> Total JVM GC count: 212. >> >> Total JVM GC time: 0.844 sec. >> >> Heavy hitter instructions (name, time, count): >> >> -- 1) dev 7.381 sec 1 >> >> -- 2) buildTree=5Ft10 5.524 sec 1 >> >> -- 3) buildTree=5Ft11 5.423 sec 1 >> >> -- 4) buildTree=5Ft9 5.225 sec 1 >> >> -- 5) rangeReIndex 4.551 sec 60268 >> >> -- 6) findBestSplitSC=5Ft10 3.737 sec 15 >> >> -- 7) findBestSplitSC=5Ft11 3.635 sec 15 >> >> -- 8) findBestSplitSC=5Ft9 3.410 sec 12 >> >> -- 9) append 1.509 sec 197 >> >> -- 10) leftIndex 0.994 sec 53253 >> >> >> >> "buildTree" part is taking quite a bit of time. >> >> >> I also tested the following basic code. This is also taking high time. >> >> >> A =3D matrix(1, 10,10) >> B =3D matrix(1,5,10) >> C =3D B %*% A >> >> >> The log output is the following >> >> >> SystemML Statistics: >> Total elapsed time: 0.000 sec. >> Total compilation time: 0.000 sec. >> Total execution time: 0.000 sec. >> Number of compiled Spark inst: 0. >> Number of executed Spark inst: 270. >> Cache hits (Mem, WB, FS, HDFS): 518/0/0/114. >> Cache writes (WB, FS, HDFS): 414/0/0. >> Cache times (ACQr/m, RLS, EXP): 32.094/0.003/0.034/0.000 sec. >> HOP DAGs recompiled (PRED, SB): 0/132. >> HOP DAGs recompile time: 0.165 sec. >> Spark ctx create time (lazy): 0.000 sec. >> Spark trans counts (par,bc,col):58/54/118. >> Spark trans times (par,bc,col): 0.040/0.225/32.187 secs. >> Total JIT compile time: 4.431 sec. >> Total JVM GC count: 1136. >> Total JVM GC time: 15.553 sec. >> Heavy hitter instructions (name, time, count): >> -- 1) append 23.917 sec 60 >> -- 2) sp=5Frblk 8.201 sec 54 >> -- 3) sp=5Fctable 1.915 sec 54 >> -- 4) sp=5Fsample 1.597 sec 54 >> -- 5) sp=5Fmapmm 0.995 sec 54 >> -- 6) sp=5Fseq 0.195 sec 54 >> -- 7) rmvar 0.071 sec 916 >> -- 8) rangeReIndex 0.010 sec 72 >> -- 9) createvar 0.010 sec 576 >> -- 10) rmempty 0.007 sec 54 >> >> >> I can see JVM GC time is high (is pretty low in above case) & append is > taking time (even though we are not appending anything). >> >> >> Can you please help me to understand why this can be the case? >> >> >> >> >> Thanks a lot! >> >> Arijit >> >> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F >> From: arijit chakraborty >> Sent: Friday, May 12, 2017 2:32:07 AM >> To: dev@systemml.incubator.apache.org >> Subject: Re: Improve SystemML execution speed in Spark >> >> Hi Niketan, >> >> >> Thank you for your suggestion! >> >> >> I tried what you suggested. >> >> >> ## Changed it here: >> >> >> from pyspark.sql import SQLContext >> import systemml as sml >> sqlCtx =3D SQLContext(sc) >> ml =3D sml.MLContext(sc).setStatistics(True) >> >> >> # And then : >> >> >> scriptUrl =3D "C:/systemml-0.13.0-incubating-bin/scripts/model=5Fcode.dm= l" >> %%time >> script =3D sml.dml(scriptUrl).input(bdframe=5Ftrain =3Dtrain=5Fdata , > bdframe=5Ftest =3D test=5Fdata).output("check=5Ffunc") >> >> beta =3D ml.execute(script).get("check=5Ffunc").toNumPy() >> >> pd.DataFrame(beta).head(1) >> >> >> >> It gave me output: >> >> >> Wall time: 16.3 s >> >> >> >> But how I can get this "time is spent in converters" or "some instruction > in SystemML"? >> >> >> Just want to add I'm running this code through jupyter notebook. >> >> >> Thanks again! >> >> >> Arijit >> >> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F >> From: Niketan Pansare >> Sent: Friday, May 12, 2017 2:02:52 AM >> To: dev@systemml.incubator.apache.org >> Subject: Re: Improve SystemML execution speed in Spark >> >> Ok, then the next step would be to set statistics: >>>> ml =3D sml.MLContext(sc).setStatistics(True) >> >> It will help you identify whether the time is spent in converters or some > instruction in SystemML. >> >> Also, since dataframe creation is lazy, you may to do persist() followed > by an action such as count() to ensure you are measuring it correctly. >> >>> On May 11, 2017, at 1:27 PM, arijit chakraborty > wrote: >>> >>> Thank you Niketan for your reply! I was actually putting the timer in > the dml code part. Rest of the portion were almost instantaneous. The dml > code part was taking time. And I could not able to figure out why it could > be. >>> >>> >>> Thanks again! >>> >>> Arijit >>> >>> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F >>> From: Niketan Pansare >>> Sent: Thursday, May 11, 2017 1:33:15 AM >>> To: dev@systemml.incubator.apache.org >>> Subject: Re: Improve SystemML execution speed in Spark >>> >>> Hi Arijit, >>> >>> Can you please put timing counters around below code to understand 20-30 > seconds you observe: >>> 1. Creation of SparkContext: >>> sc =3D SparkContext("local[*]", "test") >>> 2. Converting pandas to Pyspark dataframe: >>>> train=5Fdata=3D pd.read=5Fcsv("data1.csv") >>>> test=5Fdata =3D pd.read=5Fcsv("data2.csv") >>>> train=5Fdata =3D sqlCtx.createDataFrame(pd.DataFrame(train=5Fdata)) >>>> test=5Fdata =3D sqlCtx.createDataFrame(pd.DataFrame(test=5Fdata)) >>> >>> >>> Also, you can pass pandas data frame directly to MLContext :) >>> >>> Thanks >>> >>> Niketan >>> >>>> On May 10, 2017, at 10:31 AM, arijit chakraborty > wrote: >>>> >>>> Hi, >>>> >>>> >>>> I'm creating a process in SystemML, and running it through spark. I'm > running the code in the following way: >>>> >>>> >>>> # Spark Specifications: >>>> >>>> >>>> import os >>>> import sys >>>> import pandas as pd >>>> import numpy as np >>>> >>>> spark=5Fpath =3D "C:\spark" >>>> os.environ['SPARK=5FHOME'] =3D spark=5Fpath >>>> os.environ['HADOOP=5FHOME'] =3D spark=5Fpath >>>> >>>> sys.path.append(spark=5Fpath + "/bin") >>>> sys.path.append(spark=5Fpath + "/python") >>>> sys.path.append(spark=5Fpath + "/python/pyspark/") >>>> sys.path.append(spark=5Fpath + "/python/lib") >>>> sys.path.append(spark=5Fpath + "/python/lib/pyspark.zip") >>>> sys.path.append(spark=5Fpath + "/python/lib/py4j-0.10.4-src.zip") >>>> >>>> from pyspark import SparkContext >>>> from pyspark import SparkConf >>>> >>>> sc =3D SparkContext("local[*]", "test") >>>> >>>> >>>> # SystemML Specifications: >>>> >>>> >>>> from pyspark.sql import SQLContext >>>> import systemml as sml >>>> sqlCtx =3D SQLContext(sc) >>>> ml =3D sml.MLContext(sc) >>>> >>>> >>>> # Importing the data >>>> >>>> >>>> train=5Fdata=3D pd.read=5Fcsv("data1.csv") >>>> test=5Fdata =3D pd.read=5Fcsv("data2.csv") >>>> >>>> >>>> >>>> train=5Fdata =3D sqlCtx.createDataFrame(pd.DataFrame(train=5Fdata)) >>>> test=5Fdata =3D sqlCtx.createDataFrame(pd.DataFrame(test=5Fdata)) >>>> >>>> >>>> # Finally executing the code: >>>> >>>> >>>> scriptUrl =3D "C:/systemml-0.13.0-incubating-bin/scripts/model=5Fcode.= dml" >>>> >>>> script =3D sml.dml(scriptUrl).input(bdframe=5Ftrain =3Dtrain=5Fdata , > bdframe=5Ftest =3D test=5Fdata).output("check=5Ffunc") >>>> >>>> beta =3D ml.execute(script).get("check=5Ffunc").toNumPy() >>>> >>>> pd.DataFrame(beta).head(1) >>>> >>>> The datasize are 1000 & 100 rows for train and test respectively. I'm > testing it on small dataset during development. Later will test in larger > dataset. I'm running on my local system with 4 cores. >>>> >>>> The problem is, if I run the model in R, it's taking fraction of > second. But when I'm running like this, it's taking around 20-30 seconds. >>>> >>>> Could anyone please suggest me how to improve the execution speed? In > case there are any other way I can execute the code, which can improve the > execution speed. >>>> >>>> Also, thank you all you guyz for releasing the 0.14 version. There are > fewimprovements we found extremely helpful. >>>> >>>> Thank you! >>>> Arijit >>>> >>> >> --0__=8FBB0BB1DFF0D23D8f9e8a93df938690918c8FBB0BB1DFF0D23D--