OSX Assistant will not automatically drop

Please describe the problem.

I cannot reproduce the archive behaviour explained in the screencast: git annex assistant archiving.

What steps will reproduce the problem?

Install the OSX dmg. Open the webapp. Make a client repo ~/annex on the laptop and a full archive repo /Volumes/Seagate/annex on an external hard drive. Disregard the errors in the webapp. Restart daemon. Finish configuration of the archive repo as full archive. Put a test file in ~/annex, watch it copy over in the webapp. Verify copies with git-annex whereis.
Use the GUI to move the test file to ~/annex/archive. The file does not turn into a symlink as expected.
Run the command git-annex drop --auto. the archived file is now replaced by a symlink. git annex whereis verifies that the only copy is in the archive repo. Now move the symlink back to ~/annex. The symlink is not replaced by the file. Restart the assistant using the webapp. The symlink is replaced by the file.
Repeat above steps with the same effect.

What version of git-annex are you using? On what operating system?

OSX 10.11.6 El Capitan
git-annex version: 6.20180409-gfb0780266
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external

Version 6.20180426 also produces the same result, however the webapp does not launch a browser window.

Please provide any additional information below.

# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

[2018-04-29 22:12:25.754703] main: starting assistant version 6.20180409-gfb0780266
[2018-04-29 22:12:25.780218] Cronner: You should enable consistency checking to protect your data. 
(scanning...) [2018-04-29 22:12:25.901493] Watcher: Performing startup scan
(started...) 
gpg: assuming signed data in '/var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex.tmpvDEpq7/info'
gpg: Signature made Fri Apr 27 20:03:18 2018 EAT using DSA key ID 89C809CB
gpg: /var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex-gpg.tmpBe1DxC/trustdb.gpg: trustdb created
gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
[2018-04-29 22:12:28.884998] Upgrader: An upgrade of git-annex is available.  (version 6.20180426)
(recording state in git...)
29/Apr/2018:22:13:13 +0300 [Error#yesod-core] there is no available git remote named "SeagateBlack" @(yesod-core-1.4.37-GCI7RasEcSMIU2vku0fHJ1:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:693:5)
fatal: remote SeagateBlack already exists.
29/Apr/2018:22:13:21 +0300 [Error#yesod-core] there is no available git remote named "SeagateBlack" @(yesod-core-1.4.37-GCI7RasEcSMIU2vku0fHJ1:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:693:5)
fatal: remote SeagateBlack already exists.
29/Apr/2018:22:13:24 +0300 [Error#yesod-core] there is no available git remote named "SeagateBlack" @(yesod-core-1.4.37-GCI7RasEcSMIU2vku0fHJ1:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:693:5)

##restarted daemon via webapp##

[2018-04-29 22:13:33.426914] main: starting assistant version 6.20180409-gfb0780266
[2018-04-29 22:13:33.474857] Cronner: You should enable consistency checking to protect your data. 
[2018-04-29 22:13:33.494878] TransferScanner: Syncing with SeagateBlack 
(scanning...) [2018-04-29 22:13:33.646686] Watcher: Performing startup scan
(started...) 
warning: no common commits
From /Volumes/SeagateBlack/annex
 * [new branch]      git-annex  -> SeagateBlack/git-annex
(merging SeagateBlack/git-annex into git-annex...)
(recording state in git...)
To /Volumes/SeagateBlack/annex
 * [new branch]      git-annex -> synced/git-annex
 * [new branch]      annex/direct/master -> synced/master
gpg: assuming signed data in '/var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex.tmpIrPVZb/info'
gpg: Signature made Fri Apr 27 20:03:18 2018 EAT using DSA key ID 89C809CB
gpg: /var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex-gpg.tmptALBpe/trustdb.gpg: trustdb created
gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
[2018-04-29 22:13:38.218204] Upgrader: An upgrade of git-annex is available.  (version 6.20180426)
[2018-04-29 22:14:33.641316] Pusher: Syncing with SeagateBlack 
Everything up-to-date

##restarted daemon via webapp##

[2018-04-29 22:14:04.840648] main: starting assistant version 6.20180409-gfb0780266
[2018-04-29 22:14:04.914622] Cronner: You should enable consistency checking to protect your data. 
[2018-04-29 22:14:04.938372] TransferScanner: Syncing with SeagateBlack 
(scanning...) [2018-04-29 22:14:05.129225] Watcher: Performing startup scan
(recording state in git...)
(started...) 
To /Volumes/SeagateBlack/annex
   96eb102..56b72d6  git-annex -> synced/git-annex
gpg: assuming signed data in '/var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex.tmp6iw9TV/info'
gpg: Signature made Fri Apr 27 20:03:18 2018 EAT using DSA key ID 89C809CB
gpg: /var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex-gpg.tmpFugMM8/trustdb.gpg: trustdb created
gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
[2018-04-29 22:14:13.443848] Upgrader: An upgrade of git-annex is available.  (version 6.20180426)
[2018-04-29 22:14:29.453797] Committer: Adding testing.txt
add /Users/jhnichol/Desktop/annex/testing.txt ok
[2018-04-29 22:14:29.458807] Committer: Committing changes to git
(recording state in git...)
[2018-04-29 22:14:29.508458] Pusher: Syncing with SeagateBlack 
(recording state in git...)

testing.txt
          13 100%    0.00kB/s    0:00:00
          13 100%    0.00kB/s    0:00:0(checksum...) 0 (xfer#1, to-check=0/1)

sent 105 bytes  received 42 bytes  294.00 bytes/sec
total size is 13  speedup is 0.09
[2018-04-29 22:14:29.908335] Transferrer: Uploaded testing.txt
To /Volumes/SeagateBlack/annex
   56b72d6..97076de  git-annex -> synced/git-annex
   f981f0e..903f744  annex/direct/master -> synced/master
[2018-04-29 22:14:31.993469] Pusher: Syncing with SeagateBlack 
(recording state in git...)
To /Volumes/SeagateBlack/annex
   97076de..351ae72  git-annex -> synced/git-annex
[2018-04-29 22:14:37.442779] Committer: Adding testing.txt
add /Users/jhnichol/Desktop/annex/archive/testing.txt ok
[2018-04-29 22:14:37.451934] Committer: Committing changes to git
(recording state in git...)
[2018-04-29 22:14:37.506885] Pusher: Syncing with SeagateBlack 
To /Volumes/SeagateBlack/annex
   903f744..0ace0a4  annex/direct/master -> synced/master
[2018-04-29 22:15:05.080814] Pusher: Syncing with SeagateBlack 
To /Volumes/SeagateBlack/annex
   351ae72..acc6f66  git-annex -> synced/git-annex
[2018-04-29 22:15:07.643433] Committer: Committing changes to git
(recording state in git...)
[2018-04-29 22:15:07.716942] Pusher: Syncing with SeagateBlack 
To /Volumes/SeagateBlack/annex
   0ace0a4..216adc1  annex/direct/master -> synced/master

##restarted daemon via webapp##

[2018-04-29 22:15:12.721042] main: starting assistant version 6.20180409-gfb0780266
[2018-04-29 22:15:12.780184] Cronner: You should enable consistency checking to protect your data. 
[2018-04-29 22:15:12.802662] TransferScanner: Syncing with SeagateBlack 
(scanning...) [2018-04-29 22:15:12.995192] Watcher: Performing startup scan
Everything up-to-date
(started...) 
[2018-04-29 22:15:13.807031] Committer: Committing changes to git
(recording state in git...)

SHA256E-s13--091dfb50686a31a1137099832271eb582fbb745aeca5c0c8153ffc9acc7cd0fe.txt
          13 100%    0.00kB/s    0:00:00
          13 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)

sent 175 bytes  received 42 bytes  434.00 bytes/sec
total size is 13  speedup is 0.06
(checksum...) [2018-04-29 22:15:13.980745] Transferrer: Downloaded testing.txt
[2018-04-29 22:15:14.805402] Pusher: Syncing with SeagateBlack 
(recording state in git...)
To /Volumes/SeagateBlack/annex
   acc6f66..f62818b  git-annex -> synced/git-annex
   216adc1..bebd3d1  annex/direct/master -> synced/master
[2018-04-29 22:15:15.384765] Committer: Adding testing.txt
add /Users/jhnichol/Desktop/annex/testing.txt ok
[2018-04-29 22:15:15.447495] Committer: Committing changes to git
(recording state in git...)
gpg: assuming signed data in '/var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex.tmpxY52C5/info'
gpg: Signature made Fri Apr 27 20:03:18 2018 EAT using DSA key ID 89C809CB
gpg: /var/folders/vt/jdjpydy51cb4c9743nvk2s7m0000gp/T/git-annex-gpg.tmpqPe64c/trustdb.gpg: trustdb created
gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
[2018-04-29 22:15:15.908187] Upgrader: An upgrade of git-annex is available.  (version 6.20180426)
[2018-04-29 22:15:16.983499] Pusher: Syncing with SeagateBlack 
Everything up-to-date

# End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

I have not used it before, but I'm determined to make it work! I'm nomadic and archive my pictures/videos/files and therefore this software must be perfect for me!

RSS Atom

comment 1

Hmmm. I am testing on macOS 10.12.6 with git-annex 6.20180409-gfb0780266 installed from dmg in /Applications

The webapp launch behavior seems a bit odd. If I launch from the command-line git-annex webapp from my /Users/Shared/a1 directory, then a webpage loads (for me in Chrome) and no assistant daemons are started, or at least I can't find any with ps aux. If I add files to /Users/Shared/a1 the assistant sees them, but it doesn't transfer to my backup repo at /Users/Shared/a3. It seems to only see files from the repo I started git-annex webapp from. Is this the intended behavior Joey?

I have three repos listed at ~/.config/git-annex/autostart. If I run git-annex assistant --autostart from one of my repo directories, then an assistant daemon is started for each repo, I can verify that with ps aux | grep "git-annex assistant --startdelay" which lists one assistant process per repository (3 total). If I start the assistants this way, they seem to stick around. Then, I can follow the mentioned archiving video and successfully sync/archive as shown.

I have these errors in system.log: bumblebee git-annex[766]: no path for address 0x107d40000, I don't have anything interesting in any of my daemon.log files. I can't find any other interesting system logs.

—Andrew

Comment by andrew — Fri May 4 15:58:59 2018

Remove comment

thoughts?

Any thoughts Joey? I am happy to troubleshoot this further, but I am not really sure what to test. I do have git-annex building from source on a mac, but I don't have any experience with Haskell, so don't really know what I could be checking.

Thanks. Andrew

Comment by andrew — Sun May 20 17:41:28 2018

Remove comment

comment 3

Andrew: " I have three repos listed at ~/.config/git-annex/autostart. If I run git-annex assistant --autostart from one of my repo directories, then an assistant daemon is started for each repo, I can verify that with ps aux | grep "git-annex assistant --startdelay" which lists one assistant process per repository (3 total). If I start the assistants this way, they seem to stick around. Then, I can follow the mentioned archiving video and successfully sync/archive as shown. "

I have only one repo listed at ~/.config/git-annex/autostart which is my laptop. The other repo was created with the webapp on a USB drive; it is bare and will not allow me to start an assistant on it. Again, I cannot successfully sync/archive as shown in the archiving video.

I tried using git-annex version: 6.20180509-gd801a3318 but no help.

FYI, I also noticed that clicking the .app now successfully opens the webapp again.

Comment by jhnichol — Thu May 24 19:28:27 2018

Remove comment

comment 4

So bare repositories should work fine with Assistant.

You said it is bare and will not allow me to start an assistant on it. Do you get an error message from the webapp?

What happens if you manually add the path to your bare USB drive repo to the ~/.config/git-annex/autostart file, then run git-annex assistant --autostart? Then is assistant running on it?

Comment by andrew — Wed May 30 15:24:47 2018

Remove comment

comment 5

I started over, making new repos with the webapp & version: 6.20180527-gc3064edac . I still get an error in the webapp when adding the USB repo . The USB repo /Volumes/SeagateBlack/annex was not in ~/.config/git-annex/autostart , so I added it . I enabled debugging in the webapp, shutdown the webapp and assistants, and used the terminal:

JamesMac:annex jhnichol$ cd /Volumes/SeagateBlack/annex 
JamesMac:annex jhnichol$ git-annex assistant --autostart
git-annex: You cannot run this command in a bare repository.
JamesMac:annex jhnichol$ cd /Users/jhnichol/Desktop/annex 
JamesMac:annex jhnichol$ git-annex assistant --autostart
git-annex autostart in /Users/jhnichol/Desktop/annex
[2018-05-30 20:13:45.005716] call: /Applications/git-annex.app/Contents/MacOS/bundle/git-annex ["assistant","--stop"]
[2018-05-30 20:13:45.064878] process done ExitSuccess
[2018-05-30 20:13:45.065006] call: /Applications/git-annex.app/Contents/MacOS/bundle/git-annex ["assistant","--startdelay=5s"]
[2018-05-30 20:13:45.117653] logging to .git/annex/daemon.log
[2018-05-30 20:13:45.507796] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","git-annex"]
[2018-05-30 20:13:45.520686] process done ExitSuccess
[2018-05-30 20:13:45.520806] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","--hash","refs/heads/git-annex"]
[2018-05-30 20:13:45.526297] process done ExitSuccess
[2018-05-30 20:13:45.526857] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/git-annex..2f0d4e1b6fe058a5cc899012ef99870cd2877d2c","--pretty=%H","-n1"]
[2018-05-30 20:13:45.537446] process done ExitSuccess
[2018-05-30 20:13:45.537572] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","log","refs/heads/git-annex..31d29d92a8a48146cdf9dd4517f5a034daf4a467","--pretty=%H","-n1"]
[2018-05-30 20:13:45.543984] process done ExitSuccess
[2018-05-30 20:13:45.544626] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","cat-file","--batch"]
[2018-05-30 20:13:45.546186] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2018-05-30 20:13:45.550793] read: git ["config","--null","--list"]
[2018-05-30 20:13:45.556288] process done ExitSuccess
[2018-05-30 20:13:45.557233] logging to .git/annex/daemon.log
[2018-05-30 20:13:45.565149] process done ExitSuccess
ok
git-annex autostart in /Volumes/SeagateBlack/annex 
failed

so you see I got the "You cannot run this command in a bare repository" error from the assistant. Also, the assistant for the USB drive just fails with no error details.

Thank you so much for your comments! I feel so close to having working software!

Comment by jhnichol — Wed May 30 17:36:30 2018

Remove comment

bare remotes, workflow

Aaah.

So although git-annex does support storing files in bare repositories it treats them in much the same way as git in general. Namely, they are meant to be added as remotes from other full repositories. Assistant is responsible for monitoring a working directory for changes and automatically committing and syncing those changes. Bare repositories don't actually need assistant monitoring them because they don't have a working directory where users are adding and removing files from.

In your case, you want assistant monitoring ~/Desktop/annex which it now is. If you then add /Volumes/SeagateBlack/annex as a remote to ~/Desktop/annex then assistant, by watching just ~/Desktop/annex will sync with all remotes whenever you add remove files in ~/Desktop/annex. I imagine you already have your USB drive correctly setup as a remote. Just run git annex info from your ~/Desktop/annex directory to see what remotes it knows about.

Once USB is a remote of ~/Desktop/annex, then if you launch the webapp from your ~/Desktop/annex directory,git-annex webapp you should actually see both ~/Desktop/annex and /Volumes/SeagateBlack/annex listed. The webapp automatically lists all remotes your repo knows about.

In terms of workflow, I am not actually sure that using a bare repo is what you want on your USB drive. I always setup normal full git repositories on USB drives (not bare repos). That way I can use Finder to actually browse out to the files on the USB drive, add files when I am on the go, copy files etc… If you use a bare repo, then the files won't actually be listed in a usable form. But, if you don't actually want to ever look at the files on your USB drive (from Finder) then using a bare remote on a USB drive has the advantage that you don't need assistant monitoring it, and is probably marginally faster in some extreme cases since there is no working tree to update.

Comment by andrew — Thu May 31 12:50:55 2018

Remove comment

Add a comment