Context
- Git annex does a lot under the hood: replace files with symlinks, check and adjust moved links, etc.
- Git annex allows to mix and match with plain git commands.
- Operations may be interrupted by signal (Ctrl-C), by network failure (especially in mobile situation), by power failure (laptop on battery).
Git itself is mostly transactional: most actions can be interrupted and everything looks like they were never started.
I'm confident that you Joey have an eye for details and do the safe thing. Yet parameters above open up for a lot of combinations, some of which might not actually do what's intended, or are perhaps safe only for someone who knows where not to put a foot.
Example
For example, nearly 5 years ago you wrote about interrupted git annex add: in git annex add crash and subsequent recovery
Thought I'd mention how to clean up from interrupting git annex add. When you do that, it doesn't get a chance to git add the files it's added (this is normally done at the end, or sometimes at points in the middle when you're adding a lot of files). Which is also why fsck, whereis, and unannex wouldn't operate on them, since they only deal with files in git. So the first step is to manually use git add on any symlinks. I've made git annex add recover when ran a second time.
Questions
- Which operations are safe to interrupt? What will be the result? No change? Partial but valid result? Messy result? With what consequences?
- What operations should not be performed after an interrupted one?
For example, let's say I've started a git annex sync. While it works, I realize there are unwanted changes. If I interrupt what is the result? Can I safely use manual git commands to add or remove some files to index, make separated commits, then git sync again?
I'm using a v5 repository, indirect mode, mixed content. Is it different in v6? In direct mode? In mixed vs non-mixed content repository?
Wish a dedicated page in git-annex documentation
IMHO this is an important, difficult and changing topic, which would deserve its own "when things go wrong" topic pages, like fsck: when things go wrong and transferring files: When things go wrong.
Perhaps just starting with broad rules like "Commands foo and bar are ok, baz is mostly ok just repeat it" or "in any on-trivial case, do git annex sync last after you check twice that any necessary commit is done" would be very appreciated.
Thank you.
Note that git is not entirely robust against being interrupted. In particular, interrupting a commit can leave a stale lock file in place, and committing will then fail until the user manually goes in and removes the lock file.
I'd say that git-annex is overall more robust against interruptions than git is, except that git-annex uses git so of course inherits its limitations.
git-annex tries to handle all interruptions sanely. Ie, files are always moved into place atomically so partial writes are not a problem, stale lock files are not a problem, and any recordkeeping that might be lost by an interruption should be able to be recovered by running the interrupted command again, or perhaps
git annex fsck.The only problem that I can think of with interuptions is that since git-annex uses temp files in
.git/annex/tmpextensively, if it's interrupted nothing will clean up those temp files.This issue is quite important when you reach a certain point/scenarios.
I'm using a git annex repo that have ~300GB of annexed files; I wanted to push them to a remote server in another country. This server has limited bandwidth around 1MB/s. I also have a limited data plan at home. That is why I was planning to
git annex copy --to my-slow-remote-serverat work, were the data plan is unlimited. But after spending around 4 hours during the copy I realized that when I doCtrl+Cthen the copy operation interrupts and it seems like it's not resumable, because "git annex" does not record any state in git after interrupting the copy operation.To solve this issue I'm planning to write a custom script - similar to finding duplicate files. The goal of this script would be:
git annex find=*, and upload them individually.cd <dir> & git annex copy . --to my-slow-remote-serverIt will allow me to upload all files in batches in a course of several days/weeks. But I it will also probably take me a weekend or more to handle it correctly.
So, the questions are:
Git annex can normally resume any operation seamlessly. It can resume from aborted copying and shouldn't need to start over. I suggest:
git-annexbranch (I think that it does that)git annex --copy --to=slowremote --not --in=slowremote, that should speed up the initial phase of finding what to copy and also not display the files that are already copied (maybe that confused you)Aha, that what I missed!
Indeed, my original observations weren't entirely correct. Seems like git annex does record it's state periodically. For some big files where copying took ~1 hour, after running
git annex copyagain it was done almost immediately. And as you suggested--not --in=slowremoteshould speedup the initial process.It laterally saved me hours, thanks a lot!
P.S.: Still, it may be probably a good idea to record the progress state to git annex on
ctrl+cand show a friendly message e.g. "The operation is interrupted and the progress is recorded. You can safely resume it by runninggit annex copyagain."