From mapreduce-dev-return-4749-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 20 21:21:04 2011 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3D71E7BB4 for ; Tue, 20 Sep 2011 21:21:04 +0000 (UTC) Received: (qmail 30683 invoked by uid 500); 20 Sep 2011 21:21:03 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 30630 invoked by uid 500); 20 Sep 2011 21:21:03 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 30621 invoked by uid 99); 20 Sep 2011 21:21:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Sep 2011 21:21:03 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of criccomini@linkedin.com designates 69.28.149.25 as permitted sender) Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Sep 2011 21:20:57 +0000 DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns; h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic: Thread-Index:Date:Message-ID:References:In-Reply-To: Accept-Language:Content-Language:X-MS-Has-Attach: X-MS-TNEF-Correlator:x-originating-ip:Content-Type: Content-Transfer-Encoding:MIME-Version; b=mVsfG0VdUUns626tSK5uiS+Fm454+fV/SKtnR86acQDJW9lTylznsld7 wvPms7cYYVgJwlar0kh4HlhF5HW6qsjZxTAUmp4MxJP4y4Tjk9QyBOifx pwbzkIbKQ+KsIoD; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linkedin.com; i=criccomini@linkedin.com; q=dns/txt; s=proddkim; t=1316553657; x=1348089657; h=from:to:subject:date:message-id:references:in-reply-to: content-transfer-encoding:mime-version; bh=TGh6udMqBIfhisowJZ2wYk2n3P4uIC0GD3PRk6b0dJA=; b=wia3gy4Ne+aTRe2XaTezAfP21PSidiX0Kq0fxXtnGEy/fi81ckKt9f57 IRewBqhuTaL0NAaZkJ+wyS/OoNry987JEU9Xrh9zk7FKbGjHFyKTkg24X R6UlU+7mdvKOpP0; X-IronPort-AV: E=Sophos;i="4.68,413,1312182000"; d="scan'208";a="27784637" Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0289.001; Tue, 20 Sep 2011 14:20:35 -0700 From: Chris Riccomini To: "mapreduce-dev@hadoop.apache.org" Subject: RE: YARN, LocalResource, and Classpaths Thread-Topic: YARN, LocalResource, and Classpaths Thread-Index: Acx3ulhyd9Tk4TCCTfuvnM9TKYY5MgAACBPrABAFPoD//6dvo4AAAerZgAAXsAI= Date: Tue, 20 Sep 2011 21:20:34 +0000 Message-ID: References: ,,, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.18.46.247] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hey Guys, Thanks for the help. For the record, this solved the problem: new ApplicationMasterExecutor(packagePath) .addCommand("java -cp './package/*' kafka.yarn.ApplicationMaster " + = streamerClass + " " + tasks + " " + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout " + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr") .execute(new Configuration) Java is so sensitive about the classpath. Sigh. Cheers, Chris ________________________________________ From: Chris Riccomini [criccomini@linkedin.com] Sent: Tuesday, September 20, 2011 12:59 PM To: mapreduce-dev@hadoop.apache.org Subject: RE: YARN, LocalResource, and Classpaths Hey Vinod, Interesting follow up. find ./package/* Shows: package/activation-1.1.jar package/aopalliance-1.0.jar package/asm-3.2.jar package/aspectjrt-1.6.5.jar package/avro-1.5.3.jar package/avro-ipc-1.5.3.jar ... Appears it's unzipped, after all. I think the :./package/*: syntax is buste= d for me, fixing. Cheers, Chris ________________________________________ From: Chris Riccomini [criccomini@linkedin.com] Sent: Tuesday, September 20, 2011 12:49 PM To: mapreduce-dev@hadoop.apache.org Subject: RE: YARN, LocalResource, and Classpaths Hey Vinod, Yea, I considered that. find . . ./.task.sh.crc ./.container_tokens.crc ./container_tokens ./task.sh ./package ls -l package lrwxrwxrwx 1 criccomi eng 125 Sep 20 12:47 package -> /tmp/nm-local-dir/use= rcache/criccomi/appcache/application_1316468926404_0016/filecache/257397245= 5544981641/kafka-streamer.tgz Am I supposed to unzip the tgz myself? I saw code in the FSDownload to unpa= ck it, I thought. Thanks! Chris ________________________________________ From: Vinod Kumar Vavilapalli [vinodkv@hortonworks.com] Sent: Tuesday, September 20, 2011 11:05 AM To: mapreduce-dev@hadoop.apache.org Subject: Re: YARN, LocalResource, and Classpaths The .tgz archives are untarred and symlinked to in the working directory. I= n your code, you mentioned the key to be "package" in the resource description. I'd suggest you try to add files under the "package" dir (actually a symlink). Also, it really depends on how you package really is built. For more debugging, I'd replace the java command with a shell command and list the files recursively under the current work-dir to stdout. HTH, +Vinod On Tue, Sep 20, 2011 at 10:56 PM, Chris Riccomini wrote: > Hey All, > > I'm trying to use YARN's LocalResource stuff to ship a tgz file that has > all the jars that I need on my Classpath (for my ApplicationMaster and my > container tasks). I'm having trouble figuring out what my client should p= ut > in the ApplicationMaster exec command that it sends to the Resource manag= er. > I tried -cp :./*:, but it didn't seem to pick anything up. > > Here are some code snippets: > > // tell resource manager to execute kafka's application master > new ApplicationMasterExecutor(packagePath) > .addCommand("java -cp :./*: kafka.yarn.ApplicationMaster " + > streamerClass + " " + tasks + " " > + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout " > + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr") > .execute(new Configuration) > > When I run that, I see in my log output dir that it could not find > kafka.yarn.ApplicationMaster. I'm the code sample above, packagePath is a > local FS path to my .tgz file. It's uploaded by my ApplicationMasterExecu= tor > with: > > val packageFile =3D new File(packagePath); > val packageUrl =3D > ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualifie= d(new > Path(packagePath))); > > packageResource.setResource(packageUrl); > packageResource.setSize(packageFile.length()); > packageResource.setTimestamp(packageFile.lastModified()); > packageResource.setType(LocalResourceType.ARCHIVE); > packageResource.setVisibility(LocalResourceVisibility.APPLICATION); > > resource.setMemory(memory) > containerCtx.setResource(resource) > containerCtx.setCommands(cmds.toList) > containerCtx.setLocalResources(Collections.singletonMap("package", > packageResource)) > appCtx.setApplicationId(appId) > appCtx.setUser(user.getShortUserName) > appCtx.setAMContainerSpec(containerCtx) > request.setApplicationSubmissionContext(appCtx) > applicationsManager.submitApplication(request) > > When I poke around in /tmp/nm-local-dir/usercache/criccomi, I see nothing > in there after a run. Before, when I was running with visibility set to U= SER > (instead of APPLICATION), I could see the unzipped data in the filecache, > but that's gone now that I switched to APPLICATION. > > I'm probably just not putting the proper classpath. How do I do this? > > Cheers, > Chris >