Also seems possible that the keys db contains an inode cache for the object that is somehow out of date. It would then only stat the object once, which would succeed, but would differ from the previous value.

(This stat check is not strictly necessary, at least when annex.verify is enabled, so removing it is a possibility. Or perhaps not making a failure to stat be treated as a problem. But I need to understand what's happening first.)

This request seems like it might be related to https://git-annex.branchable.com/bugs/__34__failed_to_send_content_to_remote__34__/. Same basic problem and I think Michael is involved with both. Are these the same problem?

I don't think it's a case of the file being deleted file it's being transferred. It is possible to do that, just git annex drop while a transfer is in progress. But that transfer fails in a different way:

  .git/annex/objects/Pg/8g/SHA256E-s1048576000--81a66cc6944beade4f98430ad1993bfe98a773fd0db76dd7818b2486b7c33399/SHA256E-s1048576000--81a66cc6944beade4f98430ad1993bfe98a773fd0db76dd7818b2486b7c33399: getFileStatus: does not exist (No such file or directory)

  .git/annex/objects/Pg/8g/SHA256E-s1048576000--81a66cc6944beade4f98430ad1993bfe98a773fd0db76dd7818b2486b7c33399/SHA256E-s1048576000--81a66cc6944beade4f98430ad1993bfe98a773fd0db76dd7818b2486b7c33399: openBinaryFile: does not exist (No such file or directory)

  failed to send content to remote

  content not available

@mih so the new version's output confirms my suspicion that git-annex sees a change in the stat information of the file (mtime, size, inode) from before it started the transfer to when it finished.

I could imagine eg, that somehow the filesystem is not preserving stable inode numbers, and so the inode might appear different, without anything having actually changed.

It also seems possible that git-annex might somehow fail to stat the file either before or afterwards. In either case that would result in the same message. Maybe there's some way that the file gets deleted at the same time it's being transferred. Maybe the stat call fails for some other reason.

A strace of git-annex seems like the next step, if you can reproduce this somewhat reliably. I suggest a strace -v, which will display the full results of the stat() calls. The stats of the object file done before and after are the only part of the strace that we should need.

I posted this in response to your reference (which points here). What is being reproducible is this report

...
[2021-07-22 16:48:57.422143438] (Utility.Process) process [4094853] read: cp ["--reflink=always","--preserve=timestamps","/data/project/brainpeach/memento-sss/.git/annex/objects/3M/zW/MD5E-s2145810855--9aed9592cd3a6c1917b7c4c031aad4fa.fif/MD5E-s2145810855--9aed9592cd3a6c1917b7c4c031aad4fa.fif",".git/annex/tmp/MD5E-s2145810855--9aed9592cd3a6c1917b7c4c031aad4fa.fif"]
[2021-07-22 16:48:57.432008244] (Utility.Process) process [4094853] done ExitFailure 1
  content changed while it was being sent
  content changed while it was being sent
  failed to retrieve content from remote
[2021-07-22 16:49:08.991227722] (Utility.Process) process [4094849] done ExitSuccess
[2021-07-22 16:49:08.991545471] (Utility.Process) process [4094850] done ExitSuccess
  content changed while it was being sent
  content changed while it was being sent
  failed to retrieve content from remote

I have a repository with a file where this happens reliably for get requests by different users to different locations on that system (but not for get requests issued from different systems).

If it helps, we could arrange for access Joey.

Enabling the ssh agent avoids password prompts, but not the latency of making a ssh connection per transfer.

You can try setting the environment variable GIT_ANNEX_SSH_SOCKET_DIR to point to a directory where ssh can try to put its socket. I don't know if ssh on windows is able to use a socket or not. It might just work.

Turns out ssh-agent works just fine in the git bash shell on Windows, and at least I'm not prompted over and over.

Is there any chance of getting ssh connection caching supported on Windows?

I didn’t realize the full impact of this until I started syncing a larger repo, but now I understand if you start a sync you have to hand around and be ready with a passphrase when asked, because the sync will pause multiple times waiting for that input. For testing that’s fine but in production that’s too inconvenient.