It seems that the ?issue with parallel git-annex-add is not yet quite fixed; I'm seeing it in the git-annex version 7.20190507-g0c2cc3d . Repeating the git-annex-add command succeeds in adding the files that failed the first time.
add benchmarks/rabies200/analysis-FK7BF7Q0761xKv6V3z3gzZgK/benchmark_variants/v1.23.0_spades-3.13.0-trinity-metaspades-01/inputs-local.json
git-annex: .git/annex/othertmp/inge11793-2931: rename: permission denied (Permission denied)
failed
add benchmarks/rabies200/analysis-FK7BKGj0761VgPb22550gfBX/benchmark_variants/v1.23.0_spades-3.13.0-trinity-metaspades-01/inputs-local.json
git-annex: .git/annex/othertmp/inge11793-3330: rename: permission denied (Permission denied)
failed
git-annex: add: 2 failed
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git-annex has become an essential tool for organizing my bioinformatics data analyses. It scales well to large repos. It lets me keep track of analyses without locking myself into any specific cloud platform. It lets me quickly free up space on demand by moving data to cheaper storage, with the assurance that nothing will get lost. Thanks @joeyh for all your work on it.
This may be somehow related to the other bug in that it might also have to do with the same temp file, but it does not seem to be the same problem.
Why would rename() be failing with a permissions problem due to a temp directory cleanup?
It would be helpful to see a strace of the failed
rename()call, because I can't immadiately see where a temp file with that name would be exposed torename()in a way that could crash like that.I'll try to reproduce this and re-run under
strace. Maybe also, take another look at add tests under concurrency? I think that re-running concurrent operations on a small repo many times should test all relevant operation interleavings, and bring out the concurrency-specific bugs.With large repos, concurrency is the only way to make operations run in sane time. So fixing this would help a lot.
This seems to be the same error message as in these 2 bugs:
OSX: https://git-annex.branchable.com/bugs/__34__rename__58___permission_denied__34__/
Windows: https://git-annex.branchable.com/bugs/WSL1__58___git-annex-add_fails_in_DrvFs_filesystem/
So it would be especially helpful if I could reproduce it on Linux. Is anything else needed other than running git-annex add concurrently? And is that multiple processes, or with -J?