hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guo Gang <paul...@gmail.com>
Subject Re: Replace git submodule with git clone + file with commit number?
Date Fri, 01 Jul 2016 07:31:07 GMT
Those submodules will be git-cloned and then compiled only with specific
options
are enabled in configure command. If users want to build them without
Internet
connection they will have to download related git repo with specific
commit number in advance and copy them to hawq repo. Or they can set up
internal git repos and modify the git addresses in hawq source.



2016-07-01 14:07 GMT+08:00 Radar Da lei <rlei@pivotal.io>:

> One thought is if they all have released source tarball, we can let user
> download these tarballs to a specified directory.
>
> Then when compiling, we do not do 'git clone' but extract these tarballs if
> the specified version tarballs are detected.
>
> But for user who don't have internet access, when compiling, maven download
> would be a problem. See:
>
> Downloading:
>
> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-ao/1.1.0/hawq-mapreduce-ao-1.1.0.jar
> Downloading:
>
> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-parquet/1.1.0/hawq-mapreduce-parquet-1.1.0.jar
> [INFO] hawq-hadoop ........................................ FAILURE [
>  1.757 s]
>
> Thanks.
>
> Regards,
> Radar
>
> On Fri, Jul 1, 2016 at 12:30 AM, Roman Shaposhnik <roman@shaposhnik.org>
> wrote:
>
> > Good discussion so far, but I wanted to make sure that our source
> > tarball has an option of being built in an environment completely
> > disconnected from the internet (you can mandate certain shared
> > librarires to be available in the environment, of course). Will this
> > be possible with the current proposal?
> >
> > Thanks,
> > Roman.
> >
> > On Thu, Jun 30, 2016 at 3:51 AM, Guo Gang <paulguo@gmail.com> wrote:
> > > Discussed more with Jiali, I realised that to make hawk repo be more
> > > friendly to gporca developers, I could store git address, branch and a
> > > commit number in one file so that gporca developers could easier switch
> > > gporca code.
> > >
> > > 2016-06-30 15:35 GMT+08:00 Guo Gang <paulguo@gmail.com>:
> > >
> > >> 1. HAWQ repo is not a good place for gp-xerces development. My change
> > just
> > >>    use a stable and working version of gp-xerces. If gp-xerces
> > developers
> > >> want to
> > >>    test it with hawq, they can easily modify code under hawk:
> gp-xerces
> > >> (e.g. pull from
> > >>     a branch, etc.), then remove the timestamp file
> > >> depends/thirdpaty/gp-xerces_build_timestamp
> > >>    to trigger a rebuild.
> > >>
> > >> 2. Correct.
> > >>
> > >> 2016-06-30 11:31 GMT+08:00 Jiali Yao <jyao@pivotal.io>:
> > >>
> > >>> Hi Paul
> > >>>
> > >>> Generally agree on the approach while I have some questions:
> > >>>
> > >>> 1. Do we need to add branch handling?  For example, I am a developer
> > >>> of gp-xerces,
> > >>>  I need to test my branch code.
> > >>>
> > >>> 2. Let me double confirm about the build process.  Below is gp-xerces
> > repo
> > >>> and now HAWQ will be commit a. So I will set commit file
> > gp-xerces.commit
> > >>> in HAWQ repo which indicate we will use commit a. And then sometime
> > later,
> > >>> we want to use b, then we commit   gp-xerces.commit to b right?  And
> we
> > >>> will have multiple .commit file for each submodule now. right?
> > >>> gp-xerces:
> > >>>      commit c
> > >>>      ....
> > >>>      commit b
> > >>>      ....
> > >>>      commit a
> > >>>
> > >>> Thanks
> > >>> Jiali
> > >>>
> > >>> On Thu, Jun 30, 2016 at 11:03 AM, Radar Da lei <rlei@pivotal.io>
> > wrote:
> > >>>
> > >>> > +1
> > >>> > Since 'git clone' is called automatically in Makefile, I feel
this
> is
> > >>> much
> > >>> > better than use submodule.
> > >>> >
> > >>> > Regards,
> > >>> > Radar
> > >>> >
> > >>> > On Thu, Jun 30, 2016 at 10:07 AM, Lili Ma <lma@pivotal.io>
wrote:
> > >>> >
> > >>> > > Hi Paul,
> > >>> > >
> > >>> > > I have one question, you mentioned "if I update gp-xerces
commit
> > >>> number
> > >>> > in
> > >>> > > the commit file gp-xerces.commit make will trigger an
> auto-build".
> > >>> Could
> > >>> > > we set the $(gp_xerces_commit) to any commit number? What
I mean
> > is,
> > >>> > could
> > >>> > > we set it to a commit number already existing in my local
code,
> or
> > >>> from
> > >>> > > remote code which I have not fetched?
> > >>> > >
> > >>> > > If the answer is yes, I totally agree this suggestion.
> > >>> > >
> > >>> > > Thanks
> > >>> > > Lili
> > >>> > >
> > >>> > > On Wed, Jun 29, 2016 at 11:52 PM, Kavinder Dhaliwal <
> > >>> > kdhaliwal@pivotal.io>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > +1. I am in favor of this approach, especially if the
> submodules
> > >>> make
> > >>> > the
> > >>> > > > source tarball difficult to build. I also agree it will
make a
> > >>> > developers
> > >>> > > > life much easier.
> > >>> > > >
> > >>> > > > On Wed, Jun 29, 2016 at 3:18 AM, Guo Gang <paulguo@gmail.com>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > I'm proposing this change because:
> > >>> > > > >
> > >>> > > > > 1) We are ready to release the first apache release
with
> > "source
> > >>> > > > tarball",
> > >>> > > > > but
> > >>> > > > >     submodule is not friendly to source tarball
since git
> > >>> submodule
> > >>> > > > >    require a git parent.
> > >>> > > > >
> > >>> > > > > 2) With more and more development, I found that
submodule
> > >>> mechanism
> > >>> > > > >     is not that friendly for development, e.g.
> > >>> > > > >
> > >>> > > > >     If the commit number of one submodule is modified,
it is
> > hard
> > >>> to
> > >>> > > > detect
> > >>> > > > >     this for Makefile. We need to manually update
it in an
> old
> > >>> repo.
> > >>> > If
> > >>> > > > >     using "git clone", we could easily detect update
by set a
> > >>> commit
> > >>> > > > number
> > >>> > > > > file
> > >>> > > > >     as dependency and thus easily update the code.
> > >>> > > > >
> > >>> > > > >    Some developers have complained the annoying
output in
> "git
> > >>> > status"
> > >>> > > > >    after building submodules. With "git clone"
we can easily
> > mask
> > >>> > those
> > >>> > > > >    directories via .gitignore.
> > >>> > > > >
> > >>> > > > >    It is hard for developers who is not familiar
with
> submodule
> > >>> > > (Frankly
> > >>> > > > > speaking
> > >>> > > > >    I really do not think submodule is friendly)
to manipulate
> > >>> related
> > >>> > > > > directories
> > >>> > > > >    when necessary.
> > >>> > > > >
> > >>> > > > > While with the "git clone" solution, we save the
commit of
> > >>> previous
> > >>> > > > > submodule in a file,
> > >>> > > > > and the related Makefile code change is rather
simple, e.g.
> for
> > >>> > > > gp-xerces.
> > >>> > > > >
> > >>> > > > > $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp:
> > >>> > > > > $(ORCA_SRC_PATH)/gp-xerces.commit
> > >>> > > > >     rm -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
> > >>> > > > >
> > >>> > > > > gp-xerces_prepare:
> $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
> > >>> > > > >     if [ ! -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
];
> > >>> then \
> > >>> > > > >         [ "x$(gp_xerces_commit)" != "x" ] || exit
1; \
> > >>> > > > >         cd $(abs_top_srcdir)/$(subdir); mkdir -p
gp-xerces;
> cd
> > >>> > > > gp-xerces; \
> > >>> > > > >         [ ! -d .git ] && git clone
> > >>> > > > > https://github.com/greenplum-db/gp-xerces.git .;
\
> > >>> > > > >         git reset --hard $(gp_xerces_commit) ||
exit 2; \
> > >>> > > > >         touch $(ORCA_BLD_PATH)/gp_xerces_prepare_timestamp;
\
> > >>> > > > >     fi
> > >>> > > > >
> > >>> > > > > With above code change, if I update gp-xerces commit
number
> in
> > the
> > >>> > > commit
> > >>> > > > > file
> > >>> > > > > gp-xerces.commit make will trigger an auto-build,
If I messed
> > up
> > >>> the
> > >>> > > > > gp-xerces directory
> > >>> > > > > I can easily remove the whole gp-xerces, or just
remove the
> > >>> timestamp
> > >>> > > > file
> > >>> > > > > gp-xerces_prepare_timestamp to trigger an auto
build.
> > >>> > > > >
> > >>> > > > > Any suggestion? Thanks.
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message