tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Tzoumas <ktzou...@apache.org>
Subject Re: Problems with earlier Hadoop versions
Date Mon, 16 Feb 2015 18:58:44 GMT
Hi,

I tried again with the full tarball and I am getting the same error. The
code was indeed compiled on a different machine, this might be the issue.

Another issue I am having is related to classpath issues. How is the user
code (for example in the Tez examples) loaded into the classpath of the
containers that the job runs? Is there some functionality in TezClient that
I can use to ship the user code classes to the containers they will be
executed in?

Best,
Kostas

On Fri, Feb 13, 2015 at 7:09 AM, Hitesh Shah <hitesh@apache.org> wrote:

> Hi Kostas,
>
> 2.4.0 is something which we are looking to support as I believe there are
> quite a few users using it currently. Based on an email survey I sent out
> sometime back, I think there were users using a wide spectrum from 2.2
> onwards. Though, at some point down the line, we would like to stop
> supporting 2.2 depending on the level of complexity reached to support the
> various hadoop versions and their divergent feature sets.
>
> IAC, with respect to what you are seeing, at times, there are changes
> which go in that break builds against older versions of hadoop
> unintentionally. Most of the developers usually tend to use the latest
> version of hadoop ( and also the recent change to make 2.6.0 default ) has
> not helped on that front. The 2.6.0 change was mainly made with respect to
> the Tez UI and the dependence on YARN timeline server. From a security
> point of view, neither 2.4 nor 2.5 have a proper secure Timeline server
> implementation in place.
>
> In most cases, we have a couple of daily builds (
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/ and
> https://builds.apache.org/job/Tez-Build-Hadoop-2.2/ ) which usually catch
> these issues though the turnaround time on these bugs is dependent on
> someone picking them up quickly. Filed TEZ-2095 for the build failure -
> introduced a couple of days back.
>
> However, the 0.6.0 release is something that I don’t believe most of us
> were aware of. I just tried a local deploy of 0.6.0 and hadoop 2.4.0 and
> did not hit any issues with the simple orderedwordcount job that I ran
> though I tried with the full tarball and not the minimal one. Would you
> mind trying with the full tarball and let me know whether the error is
> reproducible? The only thing I can think of here is that there is an
> incompatibility between what was compiled against and what is on the
> classpath on the cluster.
>
> thanks
> — Hitesh
>
>
> On Feb 12, 2015, at 3:32 AM, Kostas Tzoumas <ktzoumas@apache.org> wrote:
>
> > Hi folks,
> >
> > I would like to report the following:
> >
> > (1) the current master does not compile with hadoop 2.4.0
> >
> > (mvn -DskipTests clean package -Dhadoop.version=2.4.0 -Phadoop24
> > -P\!hadoop26)
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > (default-testCompile) on project tez-dag: Compilation failure
> > [ERROR]
> >
> /.../tez/tez-dag/src/test/java/org/apache/tez/dag/app/TestTaskAttemptListenerImplTezDag.java:[161,42]
> > cannot find symbol
> > [ERROR] symbol:   method
> >
> newContainerId(org.apache.hadoop.yarn.api.records.ApplicationAttemptId,long)
> > [ERROR] location: class org.apache.hadoop.yarn.api.records.ContainerId
> >
> > (2) I am trying to run the examples of tez 0.6.0 on a cluster with hadoop
> > 2.4.0 and I am getting runtime exceptions:
> >
> > 2015-02-12 11:59:01,353 FATAL [main] app.DAGAppMaster: Error starting
> > DAGAppMaster
> > java.lang.AbstractMethodError:
> > org.apache.hadoop.yarn.api.records.ContainerId.setContainerId(J)V
> >        at
> >
> org.apache.hadoop.yarn.api.records.ContainerId.newInstance(ContainerId.java:60)
> >        at
> >
> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
> >        at
> org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:1821)
> >
> > The error appears when both using the cluster's hadoop jars (with the
> > tez-minimal jar) and using the hadoop jars shipped with tez-0.6.0.
> >
> > I compiled the 0.6.0 release with the -Dhadoop.version=2.4.0 -Phadoop24
> > -P\!hadoop26 options. I also tried editing the pom file and changed the
> > hadoop version and removed tez-plugins.
> >
> > Any advice? Is hadoop 2.4.0 supported in the long term, or would you
> > recommend to upgrade to 2.6.0?
> >
> > Best,
> > Kostas
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message