Please describe the problem.
git-annex assistant starts to drop files in remote repos, even when they are set to manual.
What steps will reproduce the problem?
Create 3 repos:
- A -- standard, archive
- m1 -- standard, manual
- m2 -- standard, manual
System has 3 files annexed: file1, file2, file3. Repo "A" has all three files, m1 has none, m2 has file3.
So, while in m1:
m1 $ git annex find --want-drop --in m2
file3
file3 shouln't be dropped from m2. There is no reason to do this. m2 is set as manual, and it shouldn't be touched in any case.
Now, let's get this file in m1:
m1 $ git annex get file3
get file3 (from m2...) ok
(Recording state in git...)
m1 $ git annex find --want-drop --in m2
So when 'file3' is present in local repo, it's not going to be dropped from m2.
I guess that rule 'present' works in local repo context while 'drop' acts on remote files.
What version of git-annex are you using? On what operating system?
I'm using latest version in Debian Jessie (5.20141024) on amd64 and armhf. I've also reproduced the bug with manually compiled 5.20141105-g8d8b248 on amd64.
The command
git annex find --want-drop --in m2finds files that meet 2 criteria:Combining criterion 1 and criterion 2, the command finds file3. Not that you expected, but only I think because you misunderstood what --in does.
So, I don't see a bug here?
Joey, thank you very much for the explanation. I completely agree that what I described in "how to reproduce" works as it should.
Actually I was trying to isolate an issue I had where assistant drops files in a remote manual repo. I don't quite get what circumstances are, but sometimes assistant running in m1 starts to drop files in m2 (while still leaving a copy in A). In this test setting the issue doesn't appear... In my actual setup there is an archive and 3 manual live repos, everything connected via ssh, sometimes on slow links, one manual repo is on a linux phone. Assistants were running on several repos.
I'll try to find a convinient way to reproduce this situation, or maybe discover that bug is in my head again ) Anyway I'll return with an update soon.
This might not be a very helpful comment, but a while ago I also had a situation where files would be dropped from a remote manual repo.
Unfortunately, I was not able to get this to happen reliably, but from memory
(on repo 1)
would result in
filebeing gotten, and then dropped, from repo2, even though both repos were set to manual.But, the unhelpful part, is I can't even remember exactly when this happened (maybe a year ago), so this might not help narrow it down.
This is the same problem I try to report here: http://git-annex.branchable.com/bugs/file_in_manual_mode_repository_is_dropped_when_it_is_copied_to_another_manual_mode_repository/
It is as Walter said,
git annex get file --from repo2will add the file to the current repo but will drop it in repo2Thanks Walter, renaud.
I was finally able to consitently reproduce the bug and record clean debug logs.
There are 3 repos, "arch", "man1", "man2". They are on different machines connected via ssh. "arch" is archive, "man1" and "man2" are in manual mode. Assistant is started only at "man1" with args -v -d, after that some files are added. Assistant detects them, adds and then transfers to "arch", this works as expected.
After that in "man2" I run command:
File is successfully transferred, and then assistant removes it from "man1". Here is corresponding part of logs at "man1":
Here is git annex vicfg at "man1":
Please let me know if I can provide more information, some extra logs, traces, recompile git-annex, etc.
Well, you've provided some great information, and your setup seems sane, and I'd expect it to keep the file in manual mode and not drop it.
The key part of the log is this:
So, after transferring the file to the remote where you're getting it, it has apparently checked the local repo's preferred content, and thinks it says that the local repo doesn't want the file anymore. Which certianly doesn't make sense if the repo is configured to be in manual mode.
The only time I'd expect this to happen is if the file in question is one that a "client" repository wouldn't normally want. So, if "random_1_24" was in an archive directory, the preferred content expression would let it drop it then.
I tried to replicate your configuration, and wasn't able to reproduce it deciding to drop the file, when testing with git-annex version 5.20150618.
Can you reproduce this with the most recent release of git-annex?
If so, can you verify that running "git annex drop --auto" doesn't try to drop any files when you run it in the repository that's confgured to be in manual mode?
I have also not been able to reproduce this using version 5.20141024. So, there must be some peice of information left out that's needed to reproduce this, or possibly it's an intermittent problem.
I did verify that my attempts to reproduce it caused the code path that handles drops after transfers to fire; in my tests it decided not to drop the file.
Hi Joey, I've failed to reproduce it now on Debian's 5.20141105-g8d8b248 and have already deleted dirs from previous setup. Maybe I'll try again later if I get some ideas what to tune.
This time I've written a script to setup repos for better reproducibility and while result is negative, I think it'll still may be useful to post it here:
Seems as if the problem still exists in 6.20161118 (Debian).
I have three repositories (among others),
jolla,sts-3xx, andhere.jollaandhereare in groupmanual,sts-3xxisbackup;hereandsts-3xxhave assistants running,jollanot.jollaandsts-3xxhave slightly older versions of git-annex installed.Now, when I copy a file from
heretojollalike thisthe file is subsequently dropped by the assistant:
However, I failed to reproduce the problem by replicating my setup with fresh repositories …
Please let me know if you need more information, and so many thanks for git-annex!