buildr-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Assaf Arkin <ar...@intalio.com>
Subject Re: Welcome to Rebase Hell!
Date Sun, 01 Mar 2009 09:00:48 GMT
On Sat, Feb 28, 2009 at 11:02 PM, Daniel Spiewak <djspiewak@gmail.com>wrote:

> Actually, I don't think that it is a restriction that Apache has placed on
> their SVN server (AFAIK).  The way that git-svn works is it grabs every
> single change set back to the beginning of time.  This means a really, *
> really* huge checkout from the server, all at once.  While this is not
> strictly a problem for SVN, it does pose some issues when SVN is being run
> through mod_dav.  Basically, mod_svn_dav (or mod_svn, whichever it is)
> keeps
> accruing objects in memory as the checkout progresses.  At some point, the
> process handling your HTTP request actually crashes because it runs out of
> heap space.
>
> I ran into this issue at my previous job, though it would manifest itself
> as
> a PROPSET error on svn checkout.  Because we had one particularly huge
> changeset, I was never able to do a full checkout of all revisions from
> scratch.  Using the svn client, things were at least passable (since you
> could repeatedly "svn up"), but it wasn't pretty.  The really depressing
> thing is that this is a known issue.  The mod_dav folks say that it's SVN's
> fault, while the SVN folks keep bouncing the problem back.  I've also heard
> it said that no one would ever run into this issue when using SVN
> "properly", so it's not a serious concern.  :-S  The officially
> "workaround"
> is to just allocate more memory to httpd.
>
> I'm not sure why this wouldn't affect incubator, but it may be partially
> due
> to the fact that you performed your initial clone of the Buildr repo at a
> much earlier point than this.  Alternatively, there may be some other
> factors in play here beyond annoying bugs associated with SVN+HTTPS.


Apache has one repository shared by all the projects, the first Buildr
commit was made at rev 591150 (we're now at 748722). So even on day one, you
have to go through half a million commits before you get to the first line
of Buildr code.

The only way to have git-svn complete in a life time is to select a specific
version to start with (-r) and then grab the rest with git fetch, which is
pretty snappy. So starting with rev 594940, you can quickly clone incubator:
  git svn clone
https://svn.apache.org/repos/asf/incubator/buildr/--revision 594940

But you can't clone trunk from the top-level Buildr, this will fail:
  git svn clone https://svn.apache.org/repos/asf/buildr/ --revision 594940

Same thing happens with sandboxes, you can clone a sandbox (say
/asf/buildr/sandbox/cool-new-stuff) but not the top-level projects, so
something tells me paths immediately under /asf/ are blocked.

Assaf



>
>
> Daniel
>
> On Sun, Mar 1, 2009 at 12:40 AM, Assaf Arkin <arkin@intalio.com> wrote:
>
> > On Sat, Feb 28, 2009 at 8:52 PM, Daniel Spiewak <djspiewak@gmail.com>
> > wrote:
> >
> > > Those of you following development progress using Git are probably
> > starting
> > > to notice that the classical "Vic Master" is no longer the all-knowing
> > > source of data.  Actually, Assaf's GitHub fork has become the more
> > > trustworthy one.  This is because upon its exit from incubation, Buildr
> > > gets
> > > to move its SVN repository to a new URL.  This is good for the project,
> > but
> > > bad for the Git forks since git-svn stores the URL information in its
> > > commit
> > > messages.
> > >
> > > The solution of course is to re-clone from SVN, which I assume exactly
> > what
> > > Assaf did.  There result is a repository which contains all of the same
> > SVN
> > > commits as Vic's, but different messages and very different SHA1
> > revisions,
> > > meaning that Git has a much harder time merging between the two.  I
> > > discovered this when I attempted to merge Assaf's latest changes with
> my
> > > master (forked from Vic's).  57 conflicts later (all petty, little
> issues
> > > unrelated to my additions), I finally had a working master with the
> > latest
> > > commits.  Unfortunately, when I cloned Assaf's repository directly and
> > > attempted to merge back some of my changes, it became very apparent
> that
> > I
> > > would need to fix the issue in a more scientific manner.
> > >
> > > Long story short, the solution is to rebase all of your branches onto
> > > Assaf's master.  I did this by finding the exact commit where I
> diverged
> > > from vic (I had it tagged, actually) as well as the corresponding
> commit
> > in
> > > Assaf's master.  These commits I tagged "branch-point" and
> > > "new-branch-point", respectively.  Then for each branch, I did
> something
> > > like the following:
> > >
> > > git checkout scala-joint-compilation
> > > git rebase new-branch-point
> > >
> > > Once this was done, I went back to my master and performed a similar
> > > procedure:
> > >
> > > git checkout master
> > > git rebase -s ours new-branch-point
> > >
> > > This effectively wiped out all of my changes in that branch (it's
> > possible
> > > that some commits may remain if you try it, but none did in my case).
> >  Once
> > > this was done, I went and picked through my origin/master log to see
> what
> > I
> > > was missing.  This meant re-merging all of my branches:
> > >
> > > git merge scala-joint-compilation
> > > git merge clojure
> > > ...
> > >
> > > Also, I had to cherry-pick a few commits that I had done on master
> (like
> > > four or five):
> > >
> > > git cherry-pick all-your-ant-base
> > > ...
> > >
> > > Once this was done, I pushed the result back to GitHub:
> > >
> > > git push -f origin
> > >
> > > The one caveat to this approach is I had all of my changes on numerous
> > > separate branches (for patching reasons).  All of these branches were
> > > branched off of the same point on vic/master.  Since there hadn't been
> > any
> > > merging *between* the branches (only onto master), it was fairly easy
> to
> > > just rebase these branches onto the new trunk (I only had three
> conflicts
> > > in
> > > the whole process, all easily resolved).  Just judging by GitHub, not
> > many
> > > people are managing their repositories in this fashion.  However, this
> > does
> > > mean that you could be able to just rebase without the "-s ours" on
> your
> > > master and come to the same result.
> > >
> > > The point is that you will need to perform some conniptions of this
> sort
> > in
> > > order to fix your repositories, otherwise your changes will remain
> > > incompatible with the Buildr mainline trunk: you won't be able to
> > (easily)
> > > merge assaf/master, and he won't be able to (easily) pull from you.
> > >
> > > Incidentally, if anyone has a *better* way of doing this (particularly
> > one
> > > where the entire master history doesn't get wiped out), I'm all ears!
>  I
> > do
> > > still have the unmerged repository sitting in Time Machine, so I'm
> > > perfectly
> > > willing to roll-forward a copy and try again if the result turns out to
> > be
> > > more correct.
> >
> >
> > Thanks.
> >
> >
> > AFAIK it's not possible to git svn clone directly from svn.apache.orgdue
> > to
> > some weird restriction they placed on the SVN server, it will just keep
> > git-svn hanging forever. Somehow that doesn't affect incubator projects,
> or
> > svn.eu.apache.org, although my starting point was Jukka's unofficial but
> > somewhat official git mirror[1].
> >
> >
> > When you git log, check the git-svn-id:
> >
> > commit a3ab30a66a092bf730950bd95a1394253ebd2f39
> >
> > Author: Assaf Arkin <assaf@apache.org>
> >
> > Date:   Fri Feb 27 22:24:50 2009 +0000
> >
> >
> > >     Fixed RDoc 2.3/2.4 conflict on rake setup.
> >
> >
> >
> >    git-svn-id:
> >
> >
> https://svn.eu.apache.org/repos/asf/buildr/trunk@74872213f79535-47bb-0310-9956-ffa450edef68
> >
> >
> > If two repositories use a different URL -- look for http vs https,
> > svn.eu.apache.org vs svn.apache.org, asf/buildr vs asf/incubator/buildr
> --
> > Git considers them distinct trees (branches). Any time you merge, Git
> will
> > have to merge the entire history of these two branches, leading to a lot
> of
> > conflicts.
> >
> > So if you have branches with changes, follow Daniel's instructions. If
> you
> > don't, you can still use the -s ours strategy to "switch" from one branch
> > to
> > another.
> >
> > Until Apache comes with a better solution, I'm going to keep my
> > repository synchronized against
> > https://svn.eu.apache.org/repos/asf/buildr.
> > I suggest you all do the same, that way we have the same history and can
> > easily merge with each other.
> >
> > Assaf
> >
> > [1] http://jukka.zitting.name/git/
> >
> >
> >
> > >
> > >
> > > Daniel
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message