tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Archit Thakur <archit279tha...@gmail.com>
Subject Fwd: Tez HIve problems.
Date Mon, 12 May 2014 17:02:37 GMT
Forwarding to dev.

Hi Bikas,

I was able to resolve this issue, by copying hive-exec.jar to
hdfs://user/admin/, It was not able to localize it while running Tez.

Though, I am facing some other problem now. My Tez job is not progressing,

My Tez job doesn't progress, it just keeps printing:
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1
Map 1: 0/2      Map 5: 0/2      Reducer 2: 0/1  Reducer 3: 0/1  Reducer 4:
0/1

In hive logs also, it keeps on printing the same.

Any idea, where should I look into or how should I debug the problem. Or
pointers on problem itself would help.
Thanks Archit.


On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <bikas@hortonworks.com> wrote:

> Setting execution engine to mr and framework name to yarn = Hive compiles
> to MR and runs on MR.
>
> Setting execution engine to mr and framework name to yarn-tez = Hive
> compiles to MR and runs on Tez.
>
> Setting execution engine to tez = Hive compiles to Tez and runs on Tez.
>
>
>
> Did you change the execution engine in the same shell that was earlier
> running with execution engine set to mr? If yes, then try and set the
> execution engine in hive-site.xml and use a new hive shell.
>
> Have you set all required hive configurations?
>
>
>
> If you can still reproduce the issue, then you should create a jira in the
> Apache hive project to track the bug.
>
>
>
> Bikas
>
>
>
> *From:* Archit Thakur [mailto:archit279thakur@gmail.com]
> *Sent:* Thursday, May 08, 2014 4:42 AM
> *To:* user@hive.apache.org; user@tez.incubator.apache.org
> *Subject:* Tez HIve problems.
>
>
>
> Hi,
>
> I am facing few problems while using Tez on Hive.
>
> I am able to run Tez through YARN submitting independent MR procesing job.
>
> Also, when I run SELECT query on hive shell by setting
>
> *set hive.execution.engine=mr;*
>
> but setting *mapreduce.framework.name
> <http://mapreduce.framework.name>=yarn-tez*
>
> This launches tez job, I checked by seeing AM's UI of machine.
>
> But when I *set hive.execution.engine=tez;*
>
>
>
> *It gives me *Query ID =
> admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
>
> *I checked the hive logs:*
>
> exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph.
> java.io.FileNotFoundException: File does not exist: hdfs:/user/admin
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>         at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>         at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638)
>         at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169)
>         at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
>         at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
>         at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Am I doing something wrong? The Tez I am running by setting*
> hive.execution.engine=mr;* but setting
>
> *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is
> it correct? Does it launches Tez job or is it MR? *
>
> Thanks and Regards,
>
> Archit Thakur.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message