trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Selva Govindarajan <selva.govindara...@esgyn.com>
Subject RE: Change in sqenvcom.sh - Response required
Date Thu, 07 Apr 2016 17:23:38 GMT
In the install_local_hadoop installation, the class
org/apache/hadoop/mapred/MRVersion is in
hadoop-mapreduce-client-common-2.6.0-cdh5.4.4.jar.   But, I think this class
is not found in Jenkins installation in
hadoop-mapreduce-client-common-2.6.0-cdh5.4.4.jar. This  jar is part of
class path as seen from the console output in regression logs. But we need
to check if it is part of CLASSPATH in ms.env.  We need to consider saving
off etc directory too as part of regression. This class path is definitely
found in the MR1 directory.  So there is some installation difference.

Selva

----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Thursday, April 7, 2016 9:50 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: Change in sqenvcom.sh - Response required

I did not intentionally do anything strange with the CDH set-up, just
install CDH5.4.

A little googling tells me that the Hadoop-0.20-mapred package is MR1. It is
optional, but valid to have both MR1 and MR2 installed on a system at the
same time. So I'm not sure if this is a mismatch between hive and trafodion
classpaths.

--Steve


> -----Original Message-----
> From: Dave Birdsall [mailto:dave.birdsall@esgyn.com]
> Sent: Thursday, April 7, 2016 8:32 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: Change in sqenvcom.sh - Response required
>
> Hi Selva,
>
> I'm no expert in this, but your short-term change to sqenvcom.sh
> sounds good.
>
> I wonder if Steve could comment on the Jenkins angle?
>
> Dave
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Wednesday, April 6, 2016 3:56 PM
> To: dev@trafodion.incubator.apache.org
> Subject: Change in sqenvcom.sh - Response required
>
>  Change in sqenvcom.sh - Response required
>
> Changed the subject line to avoid filtering….
>
> _____________________________________________
> *From:* Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <selva.govindarajan@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:25 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> dev@trafodion.incubator.apache.org>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still
> Failing - Change in sqenvcom.sh
>
> It looks like color didn’t show up in the message correctly. Line 351
> was added.
>
> _____________________________________________
> *From:* Selva Govindarajan [*mailto:selva.govindarajan@esgyn.com*
> <selva.govindarajan@esgyn.com>]
> *Sent:* Wednesday, April 6, 2016 3:21 PM
> *To:* 'dev@trafodion.incubator.apache.org' <
> *dev@trafodion.incubator.apache.org*
> <dev@trafodion.incubator.apache.org>>
> *Subject:* RE: Trafodion master Daily Test Result - 167 - Still
> Failing - Change in sqenvcom.sh
>
> I added the line in red to sqenvcom.sh and removed ms.env and
> regenerated ms.env.
>
>     347   # directories with jar files and list of jar files
>
>     348   # (could try to reduce the number of jars in the classpath)
>
>     349   export HADOOP_JAR_DIRS="/usr/lib/hadoop
>
>     350                           /usr/lib/hadoop/lib
>
>     351                           /usr/lib/hadoop-0.20-mapreduce"
>
>
> This makes hive to work from Trafodion. But, I am not sure if the
> Jenkins installation is done differently from other CDH installation.
> Also, I am not sure if it is ok to change sqenvcom.sh this way. If so,
> we can make this change as a short term solution. So, please provide
> your feedback on this.
>
> However, as a long term solution I feel sqenvcom.sh is becoming quite
> complex. There is a need to simplify it. One of the suggestion would
> be to generate sqenvcom.sh based on the distro rather than making
> sqenvcom.sh to cater to different distros. Even if we don’t generate
> the file fully, we should consider generating  CLASSPATH for different
> distro and source in to sqenvcom.sh only the generated installed
> distro CLASSPATH.
>
> To generate the CLASSPATH, Narendra said we can do the following
>
> hadoop classpath
>
> hbase classpath
>
> And expand all the * in the resulting class path to include all the
> jar files in the path (We don’t expand *, the jar files are not seen
> from JNI).
> This would make it better to cater to installation specific classpath
> changes too.
>
> Selva
>
> -----Original Message-----
> From: Steve Varnau [*mailto:steve.varnau@esgyn.com*
> <steve.varnau@esgyn.com> ]
> Sent: Wednesday, April 6, 2016 9:55 AM
> To: *dev@trafodion.incubator.apache.org*
> <dev@trafodion.incubator.apache.org>
> Subject: RE: Trafodion master Daily Test Result - 167 - Still Failing
>
> Yes. Before we hung trying to load the tpc-ds data into hive. That
> script seems to be running fine now, and the test suite actually runs,
> though all the hive related tests fail.  So something else is not right.
>
> Test automation saves the hbase logs, but not hive logs, so I think we
> will need to spin up a vm to debug it.  Who would like a crack at this
> one?
>
> --Steve
>
> > -----Original Message-----
>
> > From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com
> <selva.govindarajan@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 9:19 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still
> > Failing
>
> >
>
> > I think it looks like a class path issue for hive tests in CDH.
> > Need
>
> > to be analyzed in CDH vm.
>
> >
>
> > Selva
>
> >
>
> > -----Original Message-----
>
> > From: Sandhya Sundaresan [mailto:sandhya.sundaresan@esgyn.com
> <sandhya.sundaresan@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 7:48 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: RE: Trafodion master Daily Test Result - 167 - Still
> > Failing
>
> >
>
> > The executor suite passed after weeks but the problematic test
>
> > (TEST106) has been commented out until the new  hang for "drop table"
>
> > is fixed  just so the rest of the tests in the suite get run. The
> > old
>
> > hang seems to be fixed now.
>
> >
>
> > Steve, looks like the Hive metadata is corrupt or not setup properly
>
> > on the VMs where the core and hive suites are run.
>
> >
>
> > Arvind, I suppose you're following up on the phoenix failures ?
>
> >
>
> > Sandhya
>
> >
>
> > -----Original Message-----
>
> > From: steve.varnau@esgyn.com [mailto:steve.varnau@esgyn.com
> <steve.varnau@esgyn.com>]
>
> > Sent: Wednesday, April 6, 2016 4:02 AM
>
> > To: dev@trafodion.incubator.apache.org
>
> > Subject: Trafodion master Daily Test Result - 167 - Still Failing
>
> >
>
> > Daily Automated Testing master
>
> >
>
> > Jenkins Job:   https://jenkins.esgyn.com/job/Check-Daily-master/167/
>
> > Archived Logs: http://traf-testlogs.esgyn.com/Daily-master/167
>
> > Bld Downloads: http://traf-builds.esgyn.com
>
> >
>
> > Changes since previous daily build:
>
> > [Steve Varnau] [TRAFODION-1816] License info for binary distribution
>
> > of REST bundle
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove auto flush concept in
>
> > Trafodion
>
> >
>
> > [selvaganesang] [TRAFODION-1918] Remove disabling auto flush concept
>
> > in Trafodion
>
> >
>
> >
>
> >
>
> > Test Job Results:
>
> >
>
> > FAILURE core-regress-core-cdh (1 hr 34 min) FAILURE
>
> > core-regress-hive-cdh
>
> > (22 min) FAILURE core-regress-privs1-cdh (41 min) FAILURE
>
> > phoenix_part1_T2-cdh (11 min) FAILURE phoenix_part1_T2-hdp (39 min)
>
> > FAILURE phoenix_part2_T2-cdh (11 min) FAILURE phoenix_part2_T2-hdp
> > (38
>
> > min) SUCCESS build-master-debug (25 min) SUCCESS
> > build-master-release
>
> > (26 min) SUCCESS core-regress-charsets-cdh (27 min) SUCCESS
>
> > core-regress-charsets-hdp (1 hr
>
> > 29 min) SUCCESS core-regress-compGeneral-cdh (1 hr 25 min) SUCCESS
>
> > core-regress-compGeneral-hdp (47 min) SUCCESS core-regress-core-hdp
> > (1
>
> > hr
>
> > 1
>
> > min) SUCCESS core-regress-executor-cdh (49 min) SUCCESS
>
> > core-regress-executor-hdp (1 hr 3 min) SUCCESS
>
> > core-regress-fullstack2-cdh
>
> > (1 hr 2 min) SUCCESS core-regress-fullstack2-hdp (14 min) SUCCESS
>
> > core-regress-hive-hdp (41 min) SUCCESS core-regress-privs1-hdp (1 hr
> > 1
>
> > min)
>
> > SUCCESS core-regress-privs2-cdh (35 min) SUCCESS
>
> > core-regress-privs2-hdp
>
> > (43
>
> > min) SUCCESS core-regress-qat-cdh (14 min) SUCCESS
>
> > core-regress-qat-hdp (1 hr 13 min) SUCCESS core-regress-seabase-cdh
> > (1
>
> > hr 45 min) SUCCESS core-regress-seabase-hdp (2 hr 7 min) SUCCESS
>
> > core-regress-udr-cdh (18
>
> > min)
>
> > SUCCESS core-regress-udr-hdp (40 min) SUCCESS jdbc_test-cdh (50 min)
>
> > SUCCESS jdbc_test-hdp (57 min) SUCCESS phoenix_part1_T4-cdh (42 min)
>
> > SUCCESS phoenix_part1_T4-hdp (1 hr 26 min) SUCCESS
>
> > phoenix_part2_T4-cdh (44 min) SUCCESS phoenix_part2_T4-hdp (1 hr 4
>
> > min) SUCCESS pyodbc_test-cdh (10 min) SUCCESS pyodbc_test-hdp (38
> > min)

Mime
View raw message