systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshu Mohan <himanshu.moha...@gmail.com>
Subject Re: Update Spark Configuration to improve SystemML performance
Date Tue, 25 Jul 2017 15:06:01 GMT
I am also interested in doing some real life hands on work in SystemML

Thanks and Regards
Himanshu

> On 25-Jul-2017, at 2:57 PM, arijit chakraborty <akc14@hotmail.com> wrote:
> 
> Hi Matthias,
> 
> 
> Thanks for your mail. I'm attaching again the server configurations. I'm also adding
your personal email id, just to be double sure you can see the images. Pardon me for that.
I could improve the setup further so that now I can run the code at the same speed as R (around
40 mins). But this setup I'm sharing is the older setup. So most probably the performance
of my code was dependent on spark configuration. So if you can help me on that.
> 
> 
> Also, currently I'm mainly working on CNN works. And I've decent programming experience
in python & R. But I would request you to share with me project which is among the least
priority one. This will help me to get accustomed with this project setup without getting
bothered about time lines.
> 
> 
> Thank you!
> 
> Arijit<cluster performance.png>
> <cluster specs.png>
> <cores.png>
> From: Matthias Boehm <mboehm7@googlemail.com>
> Sent: Tuesday, July 25, 2017 2:10:52 PM
> To: dev@systemml.apache.org
> Subject: Re: Update Spark Configuration to improve SystemML performance
>  
> great to hear that - we welcome additional contributions. Just let us know
> in which area you're most interested in (e.g., algorithms, APIs, optimizer,
> runtime, etc) and we could identifying a couple of tasks to get you started.
> 
> Regarding the performance numbers, I am not able to see the details. Also
> could you share which operation was causing the large GC overhead - maybe
> we can improve the runtime for the specific scenario. Thanks.
> 
> Regards,
> Matthias
> 
> On Mon, Jul 24, 2017 at 12:17 PM, arijit chakraborty <akc14@hotmail.com>
> wrote:
> 
> > Hi,
> >
> >
> > I tried to work on spark configuration file to improve the systemML
> > performance. Even after much tuning R code is running in 40 mins, but
> > SystemML is taking 2.2 hours. Please find the spark configuration
> > screenshots. Please let me know if I'm making some mistake in tuning of the
> > spark configuration. One problem we could rectify is garbage time error.
> > Now, it's completely not there. That was one major bottleneck which was
> > making the code extremely slow.
> >
> >
> > I'm  working in local system and created a standalone version of spark,
> > with master and workers. The following are the details:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > I also wants to know is it possible to get involved with systemML
> > development? My project is almost on the verge of completion and I learned
> > a lot from you all people. And I really liked this project. So I want to
> > contribute more fruitfully in it.
> >
> >
> > Thank you!
> >
> > Arijit
> >

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message