tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Achal Soni <as...@twitter.com>
Subject Re: Tez Jars/Lib Exception
Date Tue, 23 Jul 2013 21:24:17 GMT
Oh I was wondering why do we need that jar to begin with? Is it because it
needs it for packaging the mapper and reducer classes?


On Tue, Jul 23, 2013 at 1:22 PM, Hitesh Shah <hitesh@apache.org> wrote:

> Hi Achal,
>
> I am not sure if that use-case is doable at the moment. It seems like what
> you are asking for is the ability to create a jar on the fly if a jar is
> not found. Feel free to go ahead and open a jira to address this use-case.
>
> In any case, the code does not handle a jar not being found. I filed
> TEZ-315 if you want to take that up.
>
> thanks
> -- Hitesh
>
>
> On Jul 23, 2013, at 12:13 PM, Achal Soni wrote:
>
> > I was referring to this:
> >
> >    String jarPath = ClassUtil.findContainingJar(getClass());
> >
> >    Path remoteJarPath = remoteFs.makeQualified(
> >
> >        new Path(remoteStagingDir, "dag_job.jar"));
> >
> >    remoteFs.copyFromLocalFile(new Path(jarPath), remoteJarPath);
> >
> >    FileStatus jarFileStatus = remoteFs.getFileStatus(remoteJarPath);
> >
> >
> >    Map<String, LocalResource> commonLocalResources =
> >
> >        new HashMap<String, LocalResource>();
> >
> >    LocalResource dagJarLocalRsrc = LocalResource.newInstance(
> >
> >        ConverterUtils.getYarnUrlFromPath(remoteJarPath),
> >
> >        LocalResourceType.FILE,
> >
> >        LocalResourceVisibility.APPLICATION,
> >
> >        jarFileStatus.getLen(),
> >
> >        jarFileStatus.getModificationTime());
> >
> >    commonLocalResources.put("dag_job.jar", dagJarLocalRsrc);
> >
> >
> > This causes issues in Eclipse because there is no jar per se.
> >
> >
> > On Mon, Jul 22, 2013 at 11:25 PM, Hitesh Shah <hitesh@apache.org> wrote:
> >
> >> I am not sure what you mean by the jar containing the submission. All
> the
> >> tez jars ( from the distribution tarball ) are added as local resources
> for
> >> the AM as well as all vertices by default.
> >>
> >> -- Hitesh
> >>
> >> On Jul 22, 2013, at 10:59 PM, Achal Soni wrote:
> >>
> >>> Ah I see. And is it necessary to add the jar containing the submission
> >> as a
> >>> local resource? This is causing issues when running from Eclipse. I
> guess
> >>> it is necessary for the map and reduce classes?
> >>>
> >>> - Achal
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message