Please describe the problem.
previously reported and partially addressed in here
What steps will reproduce the problem?
try any presumably read-only annex operation (whereis, wanted, etc) on a repository where you have no write permissions.
-c annex.merge-annex-branches=false configuration switch was introduced to allow for running whereis in "read-only" mode. But it requires user knowhow and good memory.
It is also inconsistent with how git behaves in that it doesn't require any extra option and just handles it automagically/gracefully.
If git-annex could automagically switch to such mode when encountering read-only repository, may be announcing that on stderr, that would make life so much easier.
Also it seems that some commands are not even fully support it:
[d31548v@discovery7 1076_spacetop]$ git annex wanted here
git-annex: /dartfs/rc/lab/D/DBIC/DBIC/archive/BIDS/Wager/Wager/1076_spacetop/.git/annex: openTempFile template locktmp: permission denied (Permission denied)
git-annex: /dartfs/rc/lab/D/DBIC/DBIC/archive/BIDS/Wager/Wager/1076_spacetop/.git/annex: openTempFile template locktmp: permission denied (Permission denied)
[d31548v@discovery7 1076_spacetop]$ git -c annex.merge-annex-branches=false annex wanted here
git-annex: /dartfs/rc/lab/D/DBIC/DBIC/archive/BIDS/Wager/Wager/1076_spacetop/.git/annex: openTempFile template locktmp: permission denied (Permission denied)
[d31548v@discovery7 1076_spacetop]$ git annex version | head -n 1
git-annex version: 8.20211117+git14-ge1f38b9dd-1~ndall+1
Firstly, you're overstating the problem. This does not affect queries in all read-only repositories. If the read-only repository does not have any remote git-annex branches that have not been merged into the local branch, the query will succeed. Only when there is such an unmerged branch will there be a permission error when it merges the annex branch.
So, if you eg use
git anenx sync, you would never end up in a situation where annex.merge-annex-branches is needed, because the remote git-annex branches are always merged by it.Setting annex.merge-annex-branches false manually is one thing, you know you are changing its behavior. Setting it implicitly due to permissions is another thing, what would result in git-annex query commands changing their behavior based on permissions. The previously linked thread discussed and rejected that idea, and rightly so.
What I would be willing to do would be to have git-annex detect when it cannot merge remote annex branches, and skip the merging, and instead when querying the value of a file from the git-annex branch, also query the values from the unmerged remote branches, and combine them in memory the same as if a merge had been done. This would make git-annex query operations slower in this unusual situation, but they would always have the same result as if it was able to perform the merge.
I have not been able to reproduce the
git -c annex.merge-annex-branches=false annex wanted hereproblem. You apparently have annex.pidlock set since "locktmp" is used when doing pid locks, but despite setting that and having an unmerged remote git-annex branch, this is as close as I can get:Anyway, this is an entirely different problem and should be in its own bug report, once you determine how to reproduce it..
+1 on the idea of "combine them in memory the same as if a merge had been done." -- sounds great to me, even though indeed might cost some performance. I would assume that with
annex.merge-annex-branches=falsesuch operation would be skipped, and then read-only operations would operate on current, possibly lacking some merges git-annex branchre "wanted": tried on the same repository, but it was updated so its git-annex branch is now up to date with the remote one... For now just managed to unravel a more of a UX issue: https://git-annex.branchable.com/bugs/wanted_on_read-only_repo_crashes_/?updated
Yes, it seems like annex.merge-annex-branches=false should prevent it looking at unmerged sibling branches. Since it would be effectively doing an in-memory merge.
I have this mostly implemented, in the
readonly-annex-mergebranch. It seems to work well. Eg:One sticking point is transitions. When a remote git-annex branch has a transition in it, and the transition has not yet been applied to the local branch and/or other remote git-annex branches, then as this is currently implemented, the transition will not be applied when doing the in-memory merge.
Since the only transitions so far forget old history and dead remotes, it's not a major effect. It would only mean that some additional information about a dead remote would be displayed, or that
git-annex logwould display more history.Also, there are several couple of other things that look at the git-annex branch, beyond simply reading a single log file, and so do not look at unmerged branches.
git-annex logdoes, --all does, etc. There are several more.These things break the invariant I am shooting for, that git-annex commands will behave the same whether they are in this readonly mode or not. They will need to be addressed before this is merged..
Update: Completed and merged!
Current list of items that need to be fixed before this is mergeable:
git-annex logruns git log on the git-annex branch, and so it will not be able to show information coming from the other, not yet merged branches. Fixed by making it error out in that situation.--in=repository@{date}uses loggedLocationsHistorical which uses Annex.Branch.getHistorical, which does not look at unmerged branches. But perhaps this is ok.. This uses the reflog, so when used to query eg files that were present in the past, it means ones that the local git-annex branch thought were present; other git-annex branches may have had a different opinion but it will not show those. And another way to look at it, the result for this will be the same before and after merging those branches.sudoin a few spots for similar needs).Bravo! Readonly annex merge support seems to mostly work on Windows, too, which frankly surprised me a bit. I was under the impression that Windows was kind of a lost cause at times and no readonly-type support would be forthcoming on this platform. :D So colour me surprised when
wanted,whereisandlogworked just like that. Onlyinfogave me trouble when I targeted a directory or file with it:An argumentless
infoworks just fine:Should I file a bug, Joey? Or do you still think this is hardly worth pursuing?
@jkniiv that may somehow be windows-specific, if opening a sqlite database on windows in order to only query it needs write access.
I did try this on linux with an unlocked file and it did not crash, but the file was already present in the database. I also tried with the keysdb deleted, and it still worked. I'm sure that there are situations though where the database is not updated where it will try to populate it before querying it, which will fail when it cannot write.
It would be worth filing new bug reports for such issues. Although when write access is unavoidable, I might just have to close them.
@yoh I'm also planning to extend git-annex's test suite with checks that some of the basic queries work in readonly repos.