From dev-return-1928-apmail-systemml-dev-archive=systemml.apache.org@systemml.apache.org Tue Jul 18 02:49:32 2017 Return-Path: X-Original-To: apmail-systemml-dev-archive@minotaur.apache.org Delivered-To: apmail-systemml-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DCFB019AA5 for ; Tue, 18 Jul 2017 02:49:31 +0000 (UTC) Received: (qmail 79229 invoked by uid 500); 18 Jul 2017 02:49:31 -0000 Delivered-To: apmail-systemml-dev-archive@systemml.apache.org Received: (qmail 79183 invoked by uid 500); 18 Jul 2017 02:49:31 -0000 Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@systemml.apache.org Delivered-To: mailing list dev@systemml.apache.org Received: (qmail 79164 invoked by uid 99); 18 Jul 2017 02:49:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jul 2017 02:49:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D62B11A0901 for ; Tue, 18 Jul 2017 02:49:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.63 X-Spam-Level: ** X-Spam-Status: No, score=2.63 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_IMAGE_ONLY_32=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=googlemail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id kH8utYwyKiCa for ; Tue, 18 Jul 2017 02:49:28 +0000 (UTC) Received: from mail-ua0-f174.google.com (mail-ua0-f174.google.com [209.85.217.174]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6AB4960DB8 for ; Tue, 18 Jul 2017 02:49:27 +0000 (UTC) Received: by mail-ua0-f174.google.com with SMTP id z22so8556849uah.1 for ; Mon, 17 Jul 2017 19:49:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=bdNX9478uySxyTiRJ47QRo+jCb5/gRcIHSxJHRUaIlw=; b=JsI+FEaZWNvkFgIyGka8jQ+KnjEs7+MJuFl9tPX4pugiUHCYr177dECY+egkZ3Qcr+ gG8K4ooyhdQcXHslXMZPxYm7NY8gddpSVk49fYUxeDQblRR7TJTcAR+DF+ypxK0TOKAP icE6susxWyvc61kXC5F5dcmP3AAsnKUdEyxyVaKz++SABW5+dpNE5w2zKLwGlwx7m+9Y kZXI3MzpLYzKCeXMDbZcpitHnQchlH3mHH64H0uk7VzjfTu380bQMWccKvLkmEI6najv LvdTvpCrbuxxBwoyPXc/ePyz7dAhRit7FcKSM0OoWCy5XhkrAP/2N+CHhnXMdt0W+pa8 OelQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=bdNX9478uySxyTiRJ47QRo+jCb5/gRcIHSxJHRUaIlw=; b=ee9EQzKm80jNfzScBw1pT4oQTEEaianVK/XhNuysNGUuUjYzCxJtr7OHg8w6/9M4n/ e2o2RZlcItsY09YXiDDX2sTeE1wpedCfBCR4+S1tOYskWItJgsjwS8K67qE5tNjVuJTK afwMa4kuJMiEVRHJDr7314YbfD+Qrt3VKZJO/FBCFsEirwK/RkEsrZLtO1sNrzaSb+UF XgB9pNMEEvZi2ncUdQS6HO7Dp/xVspb50g6L53ef6v6O8ooRv3RqmfcYkWCsVJ9cGS+W IQtLHXEceDAN7giU5GA4900jdif6CYprjbyV4jiMjidBJHm4bK7/cyjxsSUoNp1z9VrO KSmw== X-Gm-Message-State: AIVw110qRM8RdYp03AKlRyljQs/GtCNnWR9SYa9HVVL01SRlDJtOcUNC h+qYqqC7csVxG6VuKIevHw5QG0JPU4tQ X-Received: by 10.31.4.147 with SMTP id 141mr318808vke.151.1500346166128; Mon, 17 Jul 2017 19:49:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.32.166 with HTTP; Mon, 17 Jul 2017 19:49:25 -0700 (PDT) From: Matthias Boehm Date: Mon, 17 Jul 2017 19:49:25 -0700 Message-ID: Subject: Re: spark hybrid mode on HDFS To: dev@systemml.apache.org Content-Type: multipart/related; boundary="001a11423620463a7a05548e903e" --001a11423620463a7a05548e903e Content-Type: multipart/alternative; boundary="001a11423620463a7605548e903d" --001a11423620463a7605548e903d Content-Type: text/plain; charset="UTF-8" well, at a high-level, resource negotiation and distributed storage are orthogonal concepts. Yarn, Mesos, Standalone, and Kubernetes are resource schedulers, which you can configure via master and a separate deploy mode (client/cluster). Under the covers of the HDFS API, you can also use various alternative file system implementations such as HDFS, local file, object stores (e.g., swift/s3), etc. At a bare minimum, you need to have some hadoop jars in your classpath, which would already allow you to run local/standalone and the local file system implementation. Regarding the attached error, it looks like your HDFS is configured with local FS as the default file system implementation but you're trying to write to a filename with prefix hdfs. It also looks like you're running a stale version of SystemML (according to the given line numbers in your stacktrace). Note that up until SystemML 0.14 (inclusive), we always used the default file system implementation, but in master, we create the correct file system according to the given file schemes (see SYSTEMML-1696). So please try to (1) use a recent build of SystemML master, or (2) reconfigure your hdfs-site.xml to use hdfs as the default fs implementation. Regards, Matthias On Sun, Jul 16, 2017 at 11:22 PM, Krishna Kalyan wrote: > Hello All, > I have some questions running systemml scripts on HDFS (with hybrid_spark > execution mode). > > My Current Configuration: > Standalone HDFS on OSX (version 2.8) > and Spark Pre-Built for hadoop 2.7 (version 2.1.0) > > *jps* out from my system > [image: Inline image 1] > > > Both of them have been installed separately. > As far as I understand, to enable hdfs support we need to run spark master > on yarn-client | yarn-cluster. (Is this understanding correct?) > > My question: > I dont have access to a cluster, is there a way to set up a yarn-client / > yarn-cluster or my local system so that I can run systemml scripts on > hybrid_spark mode with HDFS?. If yes could you please point to some > documentation?. > > Thank you so much, > Krishna > > > PS : sysout of what I have tried already attached below. > > --001a11423620463a7605548e903d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
well, at a high-level, resource negotiation and = distributed storage are orthogonal concepts. Yarn, Mesos, Standalone, and K= ubernetes are resource schedulers, which you can configure via master and a= separate deploy mode (client/cluster). Under the covers of the HDFS API, y= ou can also use various alternative file system implementations such as HDF= S, local file, object stores (e.g., swift/s3), etc. At a bare minimum, you = need to have some hadoop jars in your classpath, which would already allow = you to run local/standalone and the local file system implementation.
Regarding the attached error, it looks like your HDFS is confi= gured with local FS as the default file system implementation but you'r= e trying to write to a filename with prefix hdfs. It also looks like you= 9;re running a stale version of SystemML (according to the given line numbe= rs in your stacktrace). Note that up until SystemML 0.14 (inclusive), we al= ways used the default file system implementation, but in master, we create = the correct file system according to the given file schemes (see SYSTEMML-1= 696). So please try to (1) use a recent build of SystemML master, or (2) re= configure your hdfs-site.xml to use hdfs as the default fs implementation.<= br>

Regards,
Matthias
<= div class=3D"gmail_extra">
On Sun, Jul 16, 20= 17 at 11:22 PM, Krishna Kalyan <krishnakalyan3@gmail.com> wrote:
Hello All,
I have some questions running systemml scripts on H= DFS (with hybrid_spark execution mode).

My Current= Configuration:
Standalone HDFS on OSX (version 2.8)
and Spark= Pre-Built for hadoop 2.7 (version 2.1.0)

jps out from my system
3D=


Both of them have been installed separately.
As f= ar as I understand, to enable hdfs support we need to run spark master on= =C2=A0yarn-client | yarn-cluster. (Is this understanding correct?)

My question:
I dont have access to a cluster= , is there a way to set up a yarn-client / yarn-cluster or my local system = so that I can run systemml scripts on hybrid_spark mode with HDFS?. If yes = could you please point to some documentation?.

Tha= nk you so much,
Krishna


P= S : sysout of what I have tried already attached below. =C2=A0=C2=A0
<= div>


--001a11423620463a7605548e903d-- --001a11423620463a7a05548e903e--