MacOSX: archive folders not working as expected

Please describe the problem.

I've added a backup USB Drive and a small archive USB drive and created a archive folder in the root of my ~/annex folder.

If I move a file into the ~/annex folder and then move the same file into the ~/annex/archive folder the file gets uploaded to the small archive USB drive, but not dropped from my ~/annex/archive folder.

[2015-08-14 19:03:47 CEST] Committer: Adding git-annex..iving.ogv
ok
(recording state in git...)
(recording state in git...)
(recording state in git...)
add /Users/markus/annex/git-annex-assistant-archiving.ogv [2015-08-14 19:03:47 CEST] Committer: Committing changes to git
[2015-08-14 19:03:47 CEST] Pusher: Syncing with annexbackup, annexarchive

git-annex-assistant-archiving.ogv
To /Volumes/annex-archive/annex-archive0
   6ef9226..6c87a7d  git-annex -> synced/git-annex
   b8855cc..bc20ee2  annex/direct/master -> synced/master
To /Volumes/annex-backup/annex-backup
   6ef9226..6c87a7d  git-annex -> synced/git-annex
   b8855cc..bc20ee2  annex/direct/master -> synced/master
    15428954 100%   20.71MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 15430948 bytes  received 42 bytes  10287326.67 bytes/sec
total size is 15428954  speedup is 1.00
[2015-08-14 19:03:48 CEST] Transferrer: Uploaded git-annex..iving.ogv
[2015-08-14 19:03:49 CEST] Pusher: Syncing with annexbackup, annexarchive
To /Volumes/annex-backup/annex-backup
   6c87a7d..d24edd5  git-annex -> synced/git-annex
To /Volumes/annex-archive/annex-archive
   6c87a7d..d24edd5  git-annex -> synced/git-annex
[2015-08-14 19:04:23 CEST] Committer: Adding git-annex..iving.ogv
ok
(recording state in git...)
(recording state in git...)
(recording state in git...)
add /Users/markus/annex/archive/git-annex-assistant-archiving.ogv [2015-08-14 19:04:23 CEST] Committer: Committing changes to git
[2015-08-14 19:04:23 CEST] Pusher: Syncing with annexbackup, annexarchive

git-annex-assistant-archiving.ogv
To /Volumes/annex-backup/annex-backup:00
   d24edd5..507be0c  git-annex -> synced/git-annex
   bc20ee2..8fea81a  annex/direct/master -> synced/master
To /Volumes/annex-archive/annex-archive
   d24edd5..507be0c  git-annex -> synced/git-annex
   bc20ee2..8fea81a  annex/direct/master -> synced/master
    15428954 100%   30.72MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 15430948 bytes  received 42 bytes  30861980.00 bytes/sec
total size is 15428954  speedup is 1.00
[2015-08-14 19:04:23 CEST] Transferrer: Uploaded git-annex..iving.ogv
[2015-08-14 19:04:25 CEST] Pusher: Syncing with annexbackup, annexarchive
To /Volumes/annex-archive/annex-archive
   507be0c..ba58f77  git-annex -> synced/git-annex
To /Volumes/annex-backup/annex-backup
   507be0c..ba58f77  git-annex -> synced/git-annex

If I directly move a file e.g. from ~/Downloads to ~/annex/archive the file gets uploaded to the small archive USB drive and then dropped from my drive (as expected).

[2015-08-14 19:04:25 CEST] Pusher: Syncing with annexbackup, annexarchive
To /Volumes/annex-archive/annex-archive
   507be0c..ba58f77  git-annex -> synced/git-annex
To /Volumes/annex-backup/annex-backup
   507be0c..ba58f77  git-annex -> synced/git-annex
[2015-08-14 19:09:44 CEST] Committer: Adding Oracle_VM..x-extpack
ok
(recording state in git...)
(recording state in git...)
(recording state in git...)
add /Users/markus/annex/archive/Oracle_VM_VirtualBox_Extension_Pack-5.0.0-101573.vbox-extpack [2015-08-14 19:09:44 CEST] Committer: Committing changes to git
[2015-08-14 19:09:44 CEST] Pusher: Syncing with annexbackup, annexarchive

Oracle_VM_VirtualBox_Extension_Pack-5.0.0-101573.vbox-extpack
To /Volumes/annex-backup/annex-backup:00
   ba58f77..73eed9e  git-annex -> synced/git-annex
   8fea81a..68526f7  annex/direct/master -> synced/master
To /Volumes/annex-archive/annex-archive
   ba58f77..73eed9e  git-annex -> synced/git-annex
   8fea81a..68526f7  annex/direct/master -> synced/master
    18028050 100%   25.65MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 18030392 bytes  received 42 bytes  12020289.33 bytes/sec
total size is 18028050  speedup is 1.00
[2015-08-14 19:09:45 CEST] Transferrer: Uploaded Oracle_VM..x-extpack
ok
(recording state in git...)
(recording state in git...)
drop /Users/markus/annex/archive/Oracle_VM_VirtualBox_Extension_Pack-5.0.0-101573.vbox-extpack [2015-08-14 19:09:47 CEST] Pusher: Syncing with annexbackup, annexarchive
To /Volumes/annex-backup/annex-backup
   73eed9e..93dc4a3  git-annex -> synced/git-annex
To /Volumes/annex-archive/annex-archive
   73eed9e..93dc4a3  git-annex -> synced/git-annex

But if I move the file-link from ~/annex/archive to ~/annex the file stays a link and don't get transferred back to my drive.

[2015-08-14 19:12:39 CEST] Committer: Committing changes to git
[2015-08-14 19:12:39 CEST] Pusher: Syncing with annexbackup, annexarchive
To /Volumes/annex-backup/annex-backup
   68526f7..082e0d8  annex/direct/master -> synced/master
To /Volumes/annex-archive/annex-archive
   68526f7..082e0d8  annex/direct/master -> synced/master

What version of git-annex are you using? On what operating system?

Im using MacOSX 10.11 Beta (15A243d) with the most resend MacOSX App.

$ /Applications/git-annex.app/Contents/MacOS/git-annex version
git-annex version: 5.20150812-ga6f7b36
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA TorrentParser Database
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

confirmed —marked as confirmed on 2018-12-30

RSS Atom

comment 1

Here's some more infos from running git-annex with debug:

Moving a 'symlinked' file from ~/annex/archive to ~/annex. I expected that the file get's automagically pulled from the archive drive and the symlink is replaced with the real file. but what happens is this:

[2015-08-17 13:29:48 CEST] Watcher: file deleted /Users/markus/annex/archive/Crisi\'s Trail und Stubenfelsen Jam.gpx
[2015-08-17 13:29:48 CEST] Committer: committing 1 changes
[2015-08-17 13:29:48 CEST] Committer: Committing changes to git
(recording state in git...)
[2015-08-17 13:29:48 CEST] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","update-index","-z","--index-info"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","--hash","refs/heads/annex/direct/master"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","write-tree"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","rev-parse","df537468c185961a9785d9288d00b0bc5f628a28:"]
[2015-08-17 13:29:48 CEST] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","commit-tree","be3e4a4a4477d6a1aad9fcbd333b43d84295e9b6","--no-gpg-sign","-p","df537468c185961a9785d9288d00b0bc5f628a28"]
[2015-08-17 13:29:48 CEST] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","update-ref","refs/heads/annex/direct/master","3436556450500cf342154e9b03bfd1825402f5eb"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2015-08-17 13:29:48 CEST] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","branch","-f","synced/master"]
[2015-08-17 13:29:48 CEST] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","branch","-f","master"]
[2015-08-17 13:29:48 CEST] Pusher: Syncing with annexbackup, annexarchive
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2015-08-17 13:29:48 CEST] Pusher: pushing to [Remote { name ="annexbackup" },Remote { name ="annexarchive" }]
[2015-08-17 13:29:48 CEST] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","push","annexbackup","+git-annex:synced/git-annex","annex/direct/master:synced/master"]
[2015-08-17 13:29:48 CEST] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","push","annexarchive","+git-annex:synced/git-annex","annex/direct/master:synced/master"]
To /Volumes/annex-archive/annex-archive
   df53746..3436556  annex/direct/master -> synced/master
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","push","annexarchive","git-annex","master"]
To /Volumes/annex-backup/annex-backup
   464ebc5..2413398  git-annex -> synced/git-annex
   df53746..3436556  annex/direct/master -> synced/master
[2015-08-17 13:29:48 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","push","annexbackup","git-annex","master"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/annex/direct/master..refs/heads/synced/master","-n1","--pretty=%H"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/annex/direct/master..refs/remotes/annexarchive/synced/master","-n1","--pretty=%H"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","git-annex"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","--hash","refs/heads/git-annex"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/git-annex..24133980364c5ae86e939c214deea1b6ab63135c","-n1","--pretty=%H"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/git-annex..d42c35efe8b1333930587a41aec8744aa0e90ae3","-n1","--pretty=%H"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/git-annex..796db681152a16121be271668d5f79c845e24b8c","-n1","--pretty=%H"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","ls-tree","--full-tree","-z","--","refs/heads/git-annex","uuid.log","remote.log","trust.log","group.log","numcopies.log","schedule.log","preferred-content.log","required-content.log","group-preferred-content.log"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2015-08-17 13:29:49 CEST] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/annex/direct/master..refs/remotes/annexbackup/synced/master","-n1","--pretty=%H"]

Comment by mhubig — Mon Aug 17 11:42:18 2015

Remove comment

comment 2

If this is OSX specific, which it seems to be (I've tested archive folders working as expected plenty on Linux), it might have to do with the assistant not seeing an event for the file being added when it's renamed to/from the archive folder. In the --debug log, it looks like the file got deleted but never added back.

If this is the case, git annex status should show the file as not being checked into git.

Comment by joey — Mon Sep 21 19:29:26 2015

Remove comment

comment 3

I think this is the same bug as Moved files are not picked up by the assistant on OS X. To reproduce, run the first transcript there but drop the file before moving it.

FSEvents fails to produce add events for dangling symlinks (https://github.com/facebook/watchman/issues/172), although it does produce removal events. Recompiling and using Kqueue with git-annex instead seems to detect dangling symlinks and add them correctly.

Comment by nanotech — Thu Mar 17 08:39:36 2016

Remove comment

comment 4

I can confirm this is still a problem.

OS:

macOS Sierra, version 10.12.5 (16F73)

git-annex versions:

OSX DMG install

Version: 6.20170611-gb493ac8
Build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi Dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.21 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.6.1 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5

Homebrew install

git-annex version: 6.20170521
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: darwin x86_64

Question

@nanotech - can you elaborate on your recompile with Kqueue comment above?

Comment by olaf — Thu Jun 22 04:50:13 2017

Remove comment

Still a problem

Not wanting to repeat myself, but this is still an issue.

ISSUE Moving files to the archive folder on MacOS does not result in them being dropped.

To reproduce

Created two linked repos with the webapp, one is client the other is full archive type.
Using MacOS' GUI drag some files into the client directory.
- All good. They get added and copied over to the full archive (looking at the webapp and git annex status)
Drag one of the files into the archive folder the webapp/assistant helpfully created for me.
- File move is noticed and git m[o]v[ed] (looking at the webapp and git annex status)
- However, file is not dropped from client

Version

git-annex version: 6.20171109
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents TorrentParser MagicMime Feeds Quvi
dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.24 DAV-1.3.1 feed-1.0.0.0 ghc-8.2.1 http-client-0.5.7.0 persistent-sqlite-2.6.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 6
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: darwin x86_64

I want to help!

What can I do to help debug this Joey?

Comment by olaf — Thu Nov 23 04:34:17 2017

Remove comment

comment 6

Bumping this bug up in hopes it gets some attention. This feature would be very useful for mac users.

Comment by flpgdt — Sat Dec 22 20:06:32 2018

Remove comment

comment 7

Just tested with 7.20181106-g352f88226 on macOS 10.12.6. Archive groups work fine with v5 repo but do not work with a v7 repo. With a v7 repo content is synced, but content is never dropped from the source archive folder. If I disable assistant and run git annex drop --auto from the client repo this will in fact drop the files from the archive folder. With the v5 repo I get drop logs in daemon.log with the v7 repos I get no errors in the logs and also no drop logs.

Also reported by a user here: [https://git-annex.branchable.com/bugs/OSX_Assistant_will_not_automatically_drop/]

Comment by andrew — Sun Dec 30 23:23:46 2018

Remove comment

comment 8

Would that be a possible workaround? Is there a way to force git-annex to use only v5 repos?

Comment by flpgdt — Mon Dec 31 16:53:50 2018

Remove comment

comment 9

I believe all repos are created at v5 by default currently both from the command-line and the assistant webapp, but there is no guarantee that will always be the case. With the latest version of git-annex a v6 repo will automatically be upgraded to a v7 repo whenever you run any git-annex command (or start the assistant or webapp).

Also just for help troubleshooting Joey: I just tested this with 6.20180914-g13dc117e9 and v6 indirect repos. Same issue, files moved to an archive folder (unlocked and locked) are never dropped by the assistant even after they have reached an archive repo. As with v7 repos, running git-annex drop --auto does drop them.

Comment by andrew — Tue Jan 1 19:00:07 2019

Remove comment

Add a comment