trafodion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arvind" <narain.arv...@gmail.com>
Subject RE: Parallel Make Failures
Date Tue, 08 Mar 2016 02:43:20 GMT
Could you check on the output of traf_tools_setup.sh ? As per the code if the bin directory
is existing then the package is assumed to be already installed.

		echo "INFO: Installing Thrift on $(date)" | tee -a $LOGFILE
		if [ -d $TOOLSDIR/thrift-0.9.0/bin ]; then
		  echo "INFO: Thrift is already installed, skipping to next tool" | tee -a $LOGFILE
		else
		  downloadSource http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz thrift-0.9.0

And if $TOOLSDIR/thrift-0.9.0 is existing then the environment variables mentioned by Roberta
get set in sqenvcom.sh:

		# Common for all distros
		if [[ -d $TOOLSDIR/thrift-${THRIFT_DEP_VER} ]]; then
		  # this is for a build environment, where we need
		  # thrift from TOOLSDIR
		  export THRIFT_LIB_DIR=$TOOLSDIR/thrift-${THRIFT_DEP_VER}/lib
		  export THRIFT_INC_DIR=$TOOLSDIR/thrift-${THRIFT_DEP_VER}/include
		fi

Regards
Arvind

-----Original Message-----
From: Gunnar Tapper [mailto:tapper.gunnar@gmail.com] 
Sent: Monday, March 7, 2016 6:21 PM
To: dev@trafodion.incubator.apache.org
Subject: Re: Parallel Make Failures

My problems is that the include and lib directories are created under ~/trafodion-build-tools/thrift-0.9.0/.
So, I'd be pointing these variables to non-existing directories. The effect is the same:

[centos@gtdev incubator-trafodion]$ make all -j 1 Building all Trafodion components cd core
&& make all
make[1]: Entering directory `/home/centos/incubator-trafodion/core'
./bldenvchk.sh;
*** Warning: QT_TOOLKIT does not point to an existing directory.
*** Warning: SQL Compiler Debugger will not be built.
*** Error: THRIFT_LIB_DIR directory not found:
/home/centos/trafodion-build-tools/thrift-0.9.0/lib
*** Error: THRIFT_INC_DIR directory not found:
/home/centos/trafodion-build-tools/thrift-0.9.0/include
make[1]: *** [sqroot] Error 1
make[1]: Leaving directory `/home/centos/incubator-trafodion/core'
make: *** [all] Error 2
[centos@gtdev incubator-trafodion]$ env | grep THRIFT THRIFT_LIB_DIR=/home/centos/trafodion-build-tools/thrift-0.9.0/lib
THRIFT_INC_DIR=/home/centos/trafodion-build-tools/thrift-0.9.0/include
THRIFT_DEP_VER=0.9.0


On Mon, Mar 7, 2016 at 7:12 PM, Roberta Marton <roberta.marton@esgyn.com <mailto:roberta.marton@esgyn.com>
>
wrote:

> I was wondering if you did setup them up before building if it fixed 
> the build issue.
>
>     Roberta
>
> -----Original Message-----
> From: Gunnar Tapper [mailto:tapper.gunnar@gmail.com]
> Sent: Monday, March 7, 2016 6:07 PM
> To: dev@trafodion.incubator.apache.org <mailto:dev@trafodion.incubator.apache.org>

> Subject: Re: Parallel Make Failures
>
> Do I set these variables before running the tools setup?
>
> On Mon, Mar 7, 2016 at 6:55 PM, Roberta Marton 
> <roberta.marton@esgyn.com <mailto:roberta.marton@esgyn.com> >
> wrote:
>
> > When I was testing release 1.3 last year, I remember getting a 
> > similar issue.  Thrift was installed but something in the make was 
> > not working correctly.
> > To get around the problem temporarily, I manually exported the 
> > THRIFT_INC_DIR and THRIFT_LIB_DIR:
> >
> > export THRIFT_LIB_DIR=<location>
> > export THRIFT_INC_DIR=<location>
> >
> > The build then succeeded.
> > There were several later changes made in the setup files that 
> > eventually did not require this workaround.
> >
> > If you manually set these envvars, does the build work?
> >
> >      Roberta
> >
> > -----Original Message-----
> > From: Gunnar Tapper [mailto:tapper.gunnar@gmail.com]
> > Sent: Monday, March 7, 2016 5:28 PM
> > To: dev@trafodion.incubator.apache.org <mailto:dev@trafodion.incubator.apache.org>

> > Subject: Re: Parallel Make Failures
> >
> > I created a new VM and did a new git clone. As stated, include and 
> > lib are not built for Thrift. Also, it seems to me that the qmake 
> > change might be broken. See below.
> >
> > [centos@trafbuild ~]$ cd incubator-trafodion/ [centos@trafbuild 
> > incubator-trafodion]$ source ./env.sh
> >
> >     If you are ready to build Trafodion, perform one of the 
> > following
> > options:
> >
> >       make all         (Build Trafodion, DCS, and REST) OR
> >       make package     (Build Trafodion, DCS, REST, and Client drivers)
> > OR
> >       make package-all (Build Trafodion, DCS, REST, Client drivers, 
> > and
> > Tests)
> >
> >     If Trafodion has been built and you want test:
> >
> >        Execute the install_local_hadoop script which performs a 
> > single node
> >        install using a popular Hadoop distribution
> >
> >           cd /home/centos/incubator-trafodion/core/sqf/sql/scripts
> >           install_local_hadoop [-p <port option>]
> >           install_traf_components
> >           configure Trafodion and start the processes
> >           HAVE FUN!
> >
> >    You can also choose to install_local_hadoop before building Trafodion.
> >
> > [centos@trafbuild incubator-trafodion]$ make all -j 1 Building all 
> > Trafodion components cd core && make all
> > make[1]: Entering directory `/home/centos/incubator-trafodion/core'
> > ./bldenvchk.sh;
> > *** Warning: QT_TOOLKIT does not point to an existing directory.
> > *** Warning: SQL Compiler Debugger will not be built.
> > *** Error: THRIFT_LIB_DIR directory not found:
> > //home/centos/trafodion-build-tools/thrift-0.9.0/lib
> > *** Error: THRIFT_INC_DIR directory not found:
> > //home/centos/trafodion-build-tools/thrift-0.9.0/include
> > make[1]: *** [sqroot] Error 1
> > make[1]: Leaving directory `/home/centos/incubator-trafodion/core'
> > make: *** [all] Error 2
> >
> > [centos@trafbuild incubator-trafodion]$ echo $QT_TOOLKIT
> >
> > [centos@trafbuild incubator-trafodion]$ ls 
> > ~/trafodion-build-tools/thrift-0.9.0/
> > bin
> >
> >
> > On Mon, Mar 7, 2016 at 3:16 PM, Steve Varnau 
> > <steve.varnau@esgyn.com <mailto:steve.varnau@esgyn.com> >
> > wrote:
> >
> > > The traf_tools_setup.sh script has not changed in the in the last 
> > > few days.
> > > I hope that the download location for thrift have not changed.  
> > > The only thing I can see that has been changing is the wiki 
> > > instructions for setting up the environment.
> > >
> > > I see at least one thing wrong on wiki. For instance, it says that 
> > > CentOS
> > > 6.7 won't work.  That is not correct. There was a mistaken 
> > > impression that
> > > 6.7 has newer gcc. Not true.
> > >
> > > With the very latest code (as of this morning), the qt-devel and 
> > > qt-config should no longer be needed.
> > >
> > > --Steve
> > >
> > >
> > > > -----Original Message-----
> > > > From: Gunnar Tapper [mailto:tapper.gunnar@gmail.com]
> > > > Sent: Monday, March 7, 2016 2:03 PM
> > > > To: dev@trafodion.incubator.apache.org <mailto:dev@trafodion.incubator.apache.org>

> > > > Subject: Re: Parallel Make Failures
> > > >
> > > > Alas, things are getting worse rather than better... :(
> > > >
> > > > I create a new VM, installed all required packages, did a git 
> > > > clone of trafodion, and then I ran install/traf_tools_setup.sh.
> > > > Now, Thrift isn't building the include and lib folders. I run 
> > > > the exact same command on the version that I downloaded last 
> > > > Thursday and that works just fine.
> > > >
> > > > Did anything change that I should be aware of?
> > > >
> > > > Gunnar
> > > >
> > > > On Mon, Mar 7, 2016 at 12:17 PM, Gunnar Tapper 
> > > > <tapper.gunnar@gmail.com <mailto:tapper.gunnar@gmail.com> >
> > > > wrote:
> > > >
> > > > > Well, today, the build doesn't work at all regardless of what 
> > > > > -j or -l options I use. It constantly fails on:
> > > > >
> > > > > Generating C++ code from yacc file ../sqlci/sqlci_yacc.y
> > > > > ##(SQL)
> > > > > ../sqlci/sqlci_yacc.y: warning: 1 reduce/reduce conflict 
> > > > > [-Wconflicts-rr]
> > > > >       ##(SQL)
> > > > > /home/centos/traf-tools/bison_3_linux/bin/bison:
> > > > >
> /home/centos/trafodion//bison_3_linux/share/bison/m4sugar/m4sugar.m4:
> > > > > cannot open: No such file or directory   ##(SQL)
> > > > > mv: cannot stat `sqlcilib/linux/64bit/debug/sqlci_yacc.cpp': 
> > > > > No such file or directory  ##(SQL)
> > > > > sed: can't read sqlcilib/linux/64bit/debug/sqlci_yacc.cpp.tmp:
> > > > > No
> > such
> > > > > file or directory        ##(SQL)
> > > > > sed: can't read sqlcilib/linux/64bit/debug/sqlci_yacc.hpp: No 
> > > > > such file or
> > > > > directory    ##(SQL)
> > > > > rm: cannot remove `sqlcilib/linux/64bit/debug/sqlci_yacc.hpp':
> > > > > No
> > such
> > > > > file or directory        ##(SQL)
> > > > > rm: cannot remove `sqlcilib/linux/64bit/debug/sqlci_yacc.cpp.tmp':
> > > > > No such
> > > > > file or directory    ##(SQL)
> > > > > make[4]: *** [sqlcilib/linux/64bit/debug/sqlci_yacc.h] Error 1
> > > > > ##(SQL)
> > > > > make[4]: Leaving directory
> > > > > `/home/centos/incubator-trafodion/core/sql/nskgmake' ##(SQL)
> > > > > make[3]: *** [all] Error 2      ##(SQL)
> > > > > make[3]: Leaving directory
> > > > > `/home/centos/incubator-trafodion/core/sqf/sql'
> > > > >      ##(SQL)
> > > > > make[2]: *** [make_sql] Error 2
> > > > > make[2]: Leaving directory
> > `/home/centos/incubator-trafodion/core/sqf'
> > > > > make[1]: *** [foundation] Error 2
> > > > > make[1]: Leaving directory `/home/centos/incubator-trafodion/core'
> > > > > make: *** [all] Error 2
> > > > >
> > > > >
> > > > > On Mon, Mar 7, 2016 at 12:05 PM, Amanda Moran
> > > > <amanda.moran@esgyn.com <mailto:amanda.moran@esgyn.com> >
> > > > > wrote:
> > > > >
> > > > >> On redhat 7.1 this command still works:
> > > > >>
> > > > >> [ec2-user@ip-10-0-0-175 ~]$ cat /etc/redhat-release Red Hat 
> > > > >> Enterprise Linux Server release 7.1 (Maipo)
> > > > >>
> > > > >> [ec2-user@ip-10-0-0-175 ~]$ grep processor /proc/cpuinfo | wc

> > > > >> -l
> > > > >> 4
> > > > >>
> > > > >>
> > > > >> On Mon, Mar 7, 2016 at 9:56 AM, Steve Varnau 
> > > > >> <steve.varnau@esgyn.com <mailto:steve.varnau@esgyn.com>
>
> > > > >> wrote:
> > > > >>
> > > > >> > > It seems that the parallel make fails on 8 GB machines.
> > > > >> >
> > > > >> > I think your first sentence overstates the determinism of

> > > > >> > the
> > > problem
> > > > >> > a
> > > > >> > bit.
> > > > >> > I ran a normal, default build on 8GB machine last week and

> > > > >> > had no
> > > > >> problem.
> > > > >> > There must be an environmental problem, but I don't think

> > > > >> > we fully understand it yet.
> > > > >> >
> > > > >> > The aggressiveness of the make parallelism is set in
> > > > >> core/sqf/sqenvcom.sh.
> > > > >> > It sets the parallel factor based on how many CPUs are on

> > > > >> > your
> > > > >> > machine:
> > > > >> >
> > > > >> > # Set default build parallelism # Can be overridden on make

> > > > >> > commandline cpucnt=$(grep processor /proc/cpuinfo | wc -l)
> > > > >> > #     no number means unlimited, and will swamp the system
> > > > >> > export MAKEFLAGS="-j$cpucnt"
> > > > >> >
> > > > >> > If that calculation is wrong, maybe that could cause a problem.
> > > > >> >
> > > > >> > --Steve
> > > > >> >
> > > > >> >
> > > > >> > > -----Original Message-----
> > > > >> > > From: Gunnar Tapper [mailto:tapper.gunnar@gmail.com]
> > > > >> > > Sent: Monday, March 7, 2016 9:35 AM
> > > > >> > > To: dev@trafodion.incubator.apache.org <mailto:dev@trafodion.incubator.apache.org>

> > > > >> > > Subject: Parallel Make Failures
> > > > >> > >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > It seems that the parallel make fails on 8 GB machines.

> > > > >> > > At least,
> > > > >> Nitin
> > > > >> > > and
> > > > >> > > I both ran into make failures that did not appear when

> > > > >> > > running serial make.
> > > > >> > > I've also seen similar failures when building the code
on
> > > > >> > > 12 GB
> > > > >> machines.
> > > > >> > >
> > > > >> > > Based on previous discussions, the Trafodion Contributor

> > > > >> > > Guide
> > > > >> recommends
> > > > >> > > rerunning make a few times if running issues.
> > > > >> > >
> > > > >> > > I most wonder if there's a way to reduce the 
> > > > >> > > aggressiveness of the
> > > > >> make
> > > > >> > in
> > > > >> > > general. Could we, for example, come up with a table
that 
> > > > >> > > correlates system size to define the -l option or 
> > > > >> > > something similar?
> > > > >> > >
> > > > >> > > --
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Gunnar
> > > > >> > > *If you think you can you can, if you think you can't

> > > > >> > > you're
> > > > >> > > right.*
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Thanks,
> > > > >>
> > > > >> Amanda Moran
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > >
> > > > > Gunnar
> > > > > *If you think you can you can, if you think you can't you're
> > > > > right.*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > >
> > > > Gunnar
> > > > *If you think you can you can, if you think you can't you're
> > > > right.*
> > >
> >
> >
> >
> > --
> > Thanks,
> >
> > Gunnar
> > *If you think you can you can, if you think you can't you're right.*
> >
>
>
>
> --
> Thanks,
>
> Gunnar
> *If you think you can you can, if you think you can't you're right.*
>



--
Thanks,

Gunnar
*If you think you can you can, if you think you can't you're right.*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message