tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: Tez Jars/Lib Exception
Date Tue, 23 Jul 2013 21:44:01 GMT
Yes - for the mapper/reduce to run within a tez task on the cluster, the jar has to be localized
for it to work.

-- Hitesh

On Jul 23, 2013, at 2:24 PM, Achal Soni wrote:

> Oh I was wondering why do we need that jar to begin with? Is it because it
> needs it for packaging the mapper and reducer classes?
> 
> 
> On Tue, Jul 23, 2013 at 1:22 PM, Hitesh Shah <hitesh@apache.org> wrote:
> 
>> Hi Achal,
>> 
>> I am not sure if that use-case is doable at the moment. It seems like what
>> you are asking for is the ability to create a jar on the fly if a jar is
>> not found. Feel free to go ahead and open a jira to address this use-case.
>> 
>> In any case, the code does not handle a jar not being found. I filed
>> TEZ-315 if you want to take that up.
>> 
>> thanks
>> -- Hitesh
>> 
>> 
>> On Jul 23, 2013, at 12:13 PM, Achal Soni wrote:
>> 
>>> I was referring to this:
>>> 
>>>   String jarPath = ClassUtil.findContainingJar(getClass());
>>> 
>>>   Path remoteJarPath = remoteFs.makeQualified(
>>> 
>>>       new Path(remoteStagingDir, "dag_job.jar"));
>>> 
>>>   remoteFs.copyFromLocalFile(new Path(jarPath), remoteJarPath);
>>> 
>>>   FileStatus jarFileStatus = remoteFs.getFileStatus(remoteJarPath);
>>> 
>>> 
>>>   Map<String, LocalResource> commonLocalResources =
>>> 
>>>       new HashMap<String, LocalResource>();
>>> 
>>>   LocalResource dagJarLocalRsrc = LocalResource.newInstance(
>>> 
>>>       ConverterUtils.getYarnUrlFromPath(remoteJarPath),
>>> 
>>>       LocalResourceType.FILE,
>>> 
>>>       LocalResourceVisibility.APPLICATION,
>>> 
>>>       jarFileStatus.getLen(),
>>> 
>>>       jarFileStatus.getModificationTime());
>>> 
>>>   commonLocalResources.put("dag_job.jar", dagJarLocalRsrc);
>>> 
>>> 
>>> This causes issues in Eclipse because there is no jar per se.
>>> 
>>> 
>>> On Mon, Jul 22, 2013 at 11:25 PM, Hitesh Shah <hitesh@apache.org> wrote:
>>> 
>>>> I am not sure what you mean by the jar containing the submission. All
>> the
>>>> tez jars ( from the distribution tarball ) are added as local resources
>> for
>>>> the AM as well as all vertices by default.
>>>> 
>>>> -- Hitesh
>>>> 
>>>> On Jul 22, 2013, at 10:59 PM, Achal Soni wrote:
>>>> 
>>>>> Ah I see. And is it necessary to add the jar containing the submission
>>>> as a
>>>>> local resource? This is causing issues when running from Eclipse. I
>> guess
>>>>> it is necessary for the map and reduce classes?
>>>>> 
>>>>> - Achal
>>>> 
>>>> 
>> 
>> 


Mime
View raw message