Generate forward ports without working copies #803
Labels
Comments
xmo-odoo
added a commit
that referenced
this issue
Aug 25, 2023
It has been a consideration for a while, but the pain of subtly interacting with git via the ignominous CLI kept it back. Then ~~the fire nation attacked~~ github got more and more tight-fisted (and in some ways less reliable) with their API. Staging pretty much just interacts with the git database, so it's both a facultative github operator (it can just interact with git directly) and a big consumer of API requests (because the git database endpoints are very low level so it takes quite a bit of work to do anything especially when high-level operations like rebase have to be replicated by hand). Furthermore, an issue has also been noticed which can be attributed to using the github API (and that API's reliability getting worse): in some cases github will fail to propagate a ref update / reset, so when staging 2 PRs it's possible that the second one is merged on top of the temporary branch of the first one, yielding a kinda broken commit (in that it's a merge commit with a broken error message) instead of the rebase / squash commit we expected. As it turns out it's a very old issue but only happened very early so was misattributed and not (sufficiently) guarded against: - 41bd82244bb976bbd4d4be5e7bd792417c7dae6b (October 8th 2018) was spotted but thought to be a mergebot issue (might have been one of the opportunities where ref-checks were added though I can't find any reference to the commit in the runbot repo). - 2be25052e147b151d1d8a5bc73cceb351586ce03 (October 15th, 2019) was missed (or ignored). - 5a9fe7a7d05a9df7186072a7bffd60c6b428fd0e (July 31st, 2023) was spotted, but happened at a moment where everything kinda broke because of github rate-limiting ref updates, so the forensics were difficult and it was attributed to rate limiting issues. - f10d03bf0f2e8f88f62a5d8356b84f714196130f (August 24th, 2023) broke the camel's back (and the head block): the logs were not too interspersed with other garbage and pretty clear that github ack'd a ref update, returned the correct oid when checking the ref, then returned the wrong oid when fetching it later on. No Working Copy =============== The working copy turns out to not be necessary, the plumbing commands we *need* work just fine on a bare repository. Working without a WC means we had to reimplement the high level operations (rebase) by hand much as we'd done previously, *but* we needed to do that anyway as git doesn't seem to provide any way to retrieve the mapping when rebasing/cherrypicking, and cherrypicking by commit doesn't work well as it can't really find the *merge base* it needs. Forward-porting can almost certainly be implemented similarly (with some overhead), issue #803 has been opened to keep track of the idea. No TMP ====== The `tmp.` branches are no more, the process of creating stagings is based entirely around oids, if staging something fails we can just abandon the oids (they'll be collected by the weekly GC), we only need to update the staging branches at the very end of the process. This simplifies things a fair bit. For now we have stopped checking for visibility / backoff as we're pushing via git, hopefully it is a more reliable reference than the API. Commmit Message Formatting ========================== There's some unfortunate churn in the test, as the handling of trailing newlines differs between github's APIs and git itself. Fixes #247 PS: It might be a good idea to use pygit2 instead of the CLI eventually, the library is typed which is nice, and it avoids shelling out although that's really unlikely to be a major cost.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Taking a better look at the plumbing in the context of #247 it turns out basic plumbing operations like
merge-treeandcommit-treecan operate without a working copy.This means it should be possible to perform forward ports without needing a separate working copy, this likely would create a bit more garbage, but tracking oids explicitly is generally easier than working copy state, so that might be a good idea. Also would avoid having leftover working copies from time to time (although clones are not too much of an issue given git hardlinks pretty much all the repository contents).
The text was updated successfully, but these errors were encountered: