Please describe the problem.

Sync / Assistant between a source and transfer repo results in zero copies for whereis, but data is available.

What steps will reproduce the problem?

See a script below...

#!/bin/sh

TMP=$(mktemp -d)


##########################################
# Set up the repos
##########################################


###
# Create working directories
cd ${TMP}
mkdir source transfer


###
# Set up the source repo
cd ${TMP}/source
git init
git annex init source
git-annex group here source
git annex wanted here standard


###
# Set up the transfer repo
# N.B. It can see the source repo
cd ${TMP}/transfer
git init
git remote add source ${TMP}/source
git annex init transfer
git-annex group here transfer
git annex wanted here \
  'not (inallgroup=client and copies=client:1) and ((include=* and ((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1)))) or approxlackingcopies=1)'


##########################################
# Add some files and run the assistant
##########################################

###
# Make sure it works by adding some files to the source...
cd ${TMP}/source
date > file.1
date > file.2
date > file.3

# Run the assistant in the source...
(cd ${TMP}/source && \
    git annex assistant && \
    sleep 30 && \
    git annex assistant --stop && \
    git annex whereis file.*)
# PASS: All files are in source.

# Now run it in the transfer group
(cd ${TMP}/transfer && \
    git annex assistant && \
    sleep 30 && \
    git annex assistant --stop && \
    git annex whereis file.*)
# PASS: All files are in transfer

# Check the source
(cd ${TMP}/source && \
    git annex whereis file.*)
# FAIL: Only file.1 was updated to be in source.

# whereis file.1 (1 copy)
#        64b193e0-86f0-4349-b073-70af919ce628 -- transfer
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed


##########################################
# Start to scratch head and ask 'Why?'
##########################################

##########
# What's happening???

cd ${TMP}/source && git annex sync
# Source thinks it is updated - working tree clean

cd ${TMP}/transfer && git annex sync
# Transfer pulls from source, so run again
git annex sync
# OK... working tree clean


# Check again
cd ${TMP}/source && git annex whereis file.*
# whereis file.1 (1 copy)
#         64b193e0-86f0-4349-b073-70af919ce628 -- transfer
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed

cd ${TMP}/transfer && git annex whereis file.*
# whereis file.1 (1 copy)
#         64b193e0-86f0-4349-b073-70af919ce628 -- transfer [here]
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed


# HOWEVER...  Files are definitely in the transfer repo

cd ${TMP}/source && cat file.*
# cat: file.1: No such file or directory
# cat: file.2: No such file or directory
# cat: file.3: No such file or directory


cd ${TMP}/transfer && cat file.*
# Sun May 14 21:22:02 PDT 2017
# Sun May 14 21:22:03 PDT 2017
# Sun May 14 21:22:06 PDT 2017

What version of git-annex are you using? On what operating system?

Version:

git-annex version: 6.20170510
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.6.1 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: darwin x86_64

OS:

MacOS 10.12.4, but seems to apply to Debian Jessie too.

Please provide any additional information below.

Annex assistant daemon.log for source repo:

[2017-05-15 14:11:24.150202] main: starting assistant version 6.20170510
[2017-05-15 14:11:24.196597] Cronner: You should enable consistency checking to protect your data.
(scanning...) [2017-05-15 14:11:24.257123] Watcher: Performing startup scan
(started...)
[2017-05-15 14:11:45.505855] Committer: Adding file.1 file.2 file.3
add file.1 ok
add file.2 ok
add file.3 ok
[2017-05-15 14:11:45.544277] Committer: Committing changes to git
(recording state in git...)
(recording state in git...)

Annex assistant daemon.log for transfer repo:

[2017-05-15 14:12:09.127406] main: starting assistant version 6.20170510
[2017-05-15 14:12:09.157611] Cronner: You should enable consistency checking to protect your data.
[2017-05-15 14:12:09.177452] TransferScanner: Syncing with source
(scanning...) [2017-05-15 14:12:09.210605] Watcher: Performing startup scan
(started...)
warning: no common commits
From /Users/olaf/tmp/git-annex/play/bug-play/source
 * [new branch]      git-annex     -> source/git-annex
 * [new branch]      master        -> source/master
 * [new branch]      synced/master -> source/synced/master
(merging source/git-annex into git-annex...)
(recording state in git...)


Already up-to-date.
[2017-05-15 14:12:10.256195] Committer: Committing changes to git
(recording state in git...)
(checksum...) [2017-05-15 14:12:10.425494] Transferrer: Downloaded file.1
[2017-05-15 14:12:11.17664] Pusher: Syncing with source
(recording state in git...)
[2017-05-15 14:12:11.356447] Committer: Committing changes to git
(recording state in git...)
To /Users/olaf/tmp/git-annex/play/bug-play/source
 * [new branch]      git-annex -> synced/git-annex
(checksum...) [2017-05-15 14:12:11.428676] Transferrer: Downloaded file.2
(checksum...) [2017-05-15 14:12:11.451457] Transferrer: Downloaded file.3
drop source file.1 ok
drop source file.2 ok
drop source file.3 ok
[2017-05-15 14:12:13.477636] Pusher: Syncing with source
(recording state in git...)
To /Users/olaf/tmp/git-annex/play/bug-play/source
   825d3dd..60c96cb  git-annex -> synced/git-annex

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

Yes. Love it. Donated. Have been using it for years. Recommend it and get(/force) my collaborators to use it.

Since this bug involved direct mode, and direct mode was known to be rather buggy and has been removed, I think it's best to close it. done --Joey