hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: YARN, LocalResource, and Classpaths
Date Tue, 20 Sep 2011 18:14:30 GMT
As Vinod mentioned, package your dirs so that they untar into the CWD. 

If you have a directory structure in your .tgz  (e.g. lib.tgz -> lib/foo.jar) you'll need
to set your classpath to be ./foo/*.

hth,
Arun

On Sep 20, 2011, at 10:26 AM, Chris Riccomini wrote:

> Hey All,
> 
> I'm trying to use YARN's LocalResource stuff to ship a tgz file that has all the jars
that I need on my Classpath (for my ApplicationMaster and my container tasks). I'm having
trouble figuring out what my client should put in the ApplicationMaster exec command that
it sends to the Resource manager. I tried -cp :./*:, but it didn't seem to pick anything up.
> 
> Here are some code snippets:
> 
>    // tell resource manager to execute kafka's application master
>    new ApplicationMasterExecutor(packagePath)
>      .addCommand("java -cp :./*: kafka.yarn.ApplicationMaster " + streamerClass + " "
+ tasks + " "
>        + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout "
>        + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
>      .execute(new Configuration)
> 
> When I run that, I see in my log output dir that it could not find kafka.yarn.ApplicationMaster.
I'm the code sample above, packagePath is a local FS path to my .tgz file. It's uploaded by
my ApplicationMasterExecutor with:
> 
>    val packageFile = new File(packagePath);
>    val packageUrl = ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualified(new
Path(packagePath)));
> 
>    packageResource.setResource(packageUrl);
>    packageResource.setSize(packageFile.length());
>    packageResource.setTimestamp(packageFile.lastModified());
>    packageResource.setType(LocalResourceType.ARCHIVE);
>    packageResource.setVisibility(LocalResourceVisibility.APPLICATION);
> 
>    resource.setMemory(memory)
>    containerCtx.setResource(resource)
>    containerCtx.setCommands(cmds.toList)
>    containerCtx.setLocalResources(Collections.singletonMap("package", packageResource))
>    appCtx.setApplicationId(appId)
>    appCtx.setUser(user.getShortUserName)
>    appCtx.setAMContainerSpec(containerCtx)
>    request.setApplicationSubmissionContext(appCtx)
>    applicationsManager.submitApplication(request)
> 
> When I poke around in /tmp/nm-local-dir/usercache/criccomi, I see nothing in there after
a run. Before, when I was running with visibility set to USER (instead of APPLICATION), I
could see the unzipped data in the filecache, but that's gone now that I switched to APPLICATION.
> 
> I'm probably just not putting the proper classpath. How do I do this?
> 
> Cheers,
> Chris


Mime
View raw message