tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: Tez HIve problems.
Date Tue, 13 May 2014 14:54:25 GMT
@Archit, if the hive shell was shut down, can you obtain all the tez logs using:

$yarn logs -applicationId <appId> 

Or, for a running query, go to the RM UI, find the application id and use the logs link to
view/download the logs. 

thanks
— Hitesh


On May 12, 2014, at 11:57 PM, Archit Thakur <archit279thakur@gmail.com> wrote:

> @Vikram: I have attached the complete hive.log. PLease go in the last to see that it
keeps on going and going.
> Applications logs: There are no tez logs generated.
> 
> So, No exception or stack trace.
> 
> @Bikas: Yes, it does in the release notes of https://issues.apache.org/jira/browse/HIVE-6098.
> There should be enough resources left because I am able to run it when I set set hive.execution.engine=mr;
> Thanks and Regards,
> Archit Thakur.
> 
> 
> On Mon, May 12, 2014 at 10:27 PM, Bikas Saha <bikas@hortonworks.com> wrote:
> Do the hive on tez instruction not specify that the hive-exec jar needs to be uploaded
first?
> 
>  
> About the stuck job, you can check the resources in the cluster via the RM web UI. Please
confirm that there are enough resources left in the cluster to run tasks. It may be that the
cluster only has enough resources to run the App Master (controller process) and not the tasks.
> 
>  
> From: Archit Thakur [mailto:archit279thakur@gmail.com] 
> Sent: Sunday, May 11, 2014 9:59 PM
> To: user@tez.incubator.apache.org
> Cc: user@hive.apache.org
> Subject: Re: Tez HIve problems.
> 
>  
> Hi Bikas,
> 
> I was able to resolve this issue, by copying hive-exec.jar to hdfs://user/admin/, It
was not able to localize it while running Tez.
> 
> Though, I am facing some other problem now. My Tez job is not progressing,
> 
> My Tez job doesn't progress, it just keeps printing:
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4: 0/1
> 
> In hive logs also, it keeps on printing the same.
> 
> Any idea, where should I look into or how should I debug the problem. Or 
> pointers on problem itself would help.
> 
> Thanks Archit.
> 
>  
> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bikas@hortonworks.com> wrote:
> 
> Setting execution engine to mr and framework name to yarn = Hive compiles to MR and runs
on MR.
> 
> Setting execution engine to mr and framework name to yarn-tez = Hive compiles to MR and
runs on Tez.
> 
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
> 
>  
> Did you change the execution engine in the same shell that was earlier running with execution
engine set to mr? If yes, then try and set the execution engine in hive-site.xml and use a
new hive shell.
> 
> Have you set all required hive configurations?
> 
>  
> If you can still reproduce the issue, then you should create a jira in the Apache hive
project to track the bug.
> 
>  
> Bikas
> 
>  
> From: Archit Thakur [mailto:archit279thakur@gmail.com] 
> Sent: Thursday, May 08, 2014 4:42 AM
> To: user@hive.apache.org; user@tez.incubator.apache.org
> Subject: Tez HIve problems.
> 
>  
> Hi,
> 
> I am facing few problems while using Tez on Hive.
> 
> I am able to run Tez through YARN submitting independent MR procesing job.
> 
> Also, when I run SELECT query on hive shell by setting
> 
> set hive.execution.engine=mr;
> 
> but setting mapreduce.framework.name=yarn-tez
> 
> This launches tez job, I checked by seeing AM's UI of machine.
> 
> But when I set hive.execution.engine=tez;
> 
> It gives me 
> 
> Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
> 
> I checked the hive logs:
> 
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>         at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>         at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>         at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>         at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>         at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>         at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>         at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> 
> Am I doing something wrong? The Tez I am running by setting hive.execution.engine=mr;
but setting mapreduce.framework.name=yarn-tez. is it correct? Does it launches Tez job or
is it MR?
> 
> Thanks and Regards,
> 
> Archit Thakur.
> 
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.
> 
>  
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.
> 
> <hive.log>


Mime
View raw message