Recent changes to this wiki:

Added a comment
diff --git a/doc/forum/whereis__58___Why_not_here__63__/comment_3_7de87ad1fd8a0f53562d3083af1410d6._comment b/doc/forum/whereis__58___Why_not_here__63__/comment_3_7de87ad1fd8a0f53562d3083af1410d6._comment
new file mode 100644
index 0000000..9b53cd0
--- /dev/null
+++ b/doc/forum/whereis__58___Why_not_here__63__/comment_3_7de87ad1fd8a0f53562d3083af1410d6._comment
@@ -0,0 +1,33 @@
+[[!comment format=mdwn
+ username="Horus"
+ avatar="http://cdn.libravatar.org/avatar/8f0ee08b98ea5bba76c3fe112c08851c"
+ subject="comment 3"
+ date="2017-05-29T18:10:55Z"
+ content="""
+Sorry, just saw your comment...
+
+Yes, I'm using the assistant, as I pointed out in my original posting. Just right now, I'm having the same situation:
+
+```
+florian@horus ~/.synced_configuration (git)-[annex/direct/master] % git annex whereis .emacs
+whereis .emacs (2 copies) 
+        c7fc42ca-49f7-4688-b81d-c9cfd4a42564 -- Asaru
+        d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
+ok
+```
+
+```
+florian@horus ~/.synced_configuration (git)-[annex/direct/master] % git annex log .emacs    
++ Mon, 29 May 2017 20:04:11 CEST .emacs | c7fc42ca-49f7-4688-b81d-c9cfd4a42564 -- Asaru
+- Mon, 29 May 2017 20:04:10 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
++ Mon, 29 May 2017 20:02:37 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
++ Mon, 29 May 2017 20:02:37 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
+- Mon, 29 May 2017 20:02:36 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
+- Mon, 29 May 2017 20:02:35 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
++ Mon, 29 May 2017 20:02:35 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
++ Mon, 29 May 2017 20:02:33 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
+```
+
+Thanks!
+Florian
+"""]]

Added a comment: Working again
diff --git a/doc/forum/Git_annex_hangs/comment_13_f28528c9b9766fe8e2d344e5f27ade65._comment b/doc/forum/Git_annex_hangs/comment_13_f28528c9b9766fe8e2d344e5f27ade65._comment
new file mode 100644
index 0000000..a528672
--- /dev/null
+++ b/doc/forum/Git_annex_hangs/comment_13_f28528c9b9766fe8e2d344e5f27ade65._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="Working again"
+ date="2017-05-29T15:19:18Z"
+ content="""
+Hi,
+
+I'm not sure what exactly has changed, but it's working again!
+
+Until now, I was using git-annex version: 6.20170321-gf3dee9d65 from the standalone-amd64.tar.gz. In the meantime I convinced the administrator to install git-annex system wide. She installed git-annex version: 5.20140221 from the packet-system.
+
+Now, the repository works again with both, git-annex 5 and 6.
+
+However, I cannot exactly tell if anything else has changed in the meantime, since I am not the administrator of the system.
+
+Thanks for your help, anyway.
+
+"""]]

Added a comment
diff --git a/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_12_5090e41cf96dfe542d1d2326aebc556c._comment b/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_12_5090e41cf96dfe542d1d2326aebc556c._comment
new file mode 100644
index 0000000..af336a1
--- /dev/null
+++ b/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_12_5090e41cf96dfe542d1d2326aebc556c._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="http://schnouki.net/"
+ nickname="Schnouki"
+ avatar="http://cdn.libravatar.org/avatar/5f6406e9db28564121169f0051645b8c30a12a20ca7bc40287ac9bf2cd3ad283"
+ subject="comment 12"
+ date="2017-05-29T09:56:59Z"
+ content="""
+Not sure how I did it, but I have 2 repos that give me the same error. (Perhaps it happened when killing the assistant in the middle of a sync? No idea…)
+
+However I \"solved\" the issue for one repo:
+
+1. cloned the gcrypt remote to a temporary directory
+2. removed all the (encrypted) files with `git rm`, committed that, and pushed it (the repo is hosted on Gitlab and I can't just remove the branch or do something like that)
+3. removed the `gcrypt-id| from my `.git/config`
+4. ran `git annex sync` again
+
+And voilà! It works :)
+
+I don't have any content in this remote, only metadata, so it's easy to do. But I still don't know what caused this.
+"""]]

Added a comment: Standalone git-annex on Synology DS216+ NAS
diff --git a/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_4_33087a19874447a1cb07aa794c0f030b._comment b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_4_33087a19874447a1cb07aa794c0f030b._comment
new file mode 100644
index 0000000..041fe92
--- /dev/null
+++ b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_4_33087a19874447a1cb07aa794c0f030b._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="ewen"
+ avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
+ subject="Standalone git-annex on Synology DS216+ NAS"
+ date="2017-05-28T01:24:43Z"
+ content="""
+For the benefit of future readers, I did manage to [get the stand alone `git-annex` running on my Synology DS216+](http://ewen.mcneill.gen.nz/blog/entry/2017-05-28-git-annex-on-synology-ds216+/).  With a little bit of sorting out details, including adding some symlinks in `/usr/bin`, it runs like `git-annex` on any other Linux-based system.  At this time that is definitely much easier than trying to use a SMB share if you are able to enable `ssh` access -- but note that `ssh` access to the Synology NAS requires an administrator account, so may not be available to everyone.  (Plus of course this work around is useful mainly to relatively open Linux-based NAS solutions.)
+
+Ewen
+"""]]

Added a comment: git-annex on Synology DS216+ (x86-64)
diff --git a/doc/tips/Synology_NAS_and_git_annex/comment_11_ab2487da061a23e14198875a00ae801e._comment b/doc/tips/Synology_NAS_and_git_annex/comment_11_ab2487da061a23e14198875a00ae801e._comment
new file mode 100644
index 0000000..6a4e912
--- /dev/null
+++ b/doc/tips/Synology_NAS_and_git_annex/comment_11_ab2487da061a23e14198875a00ae801e._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="ewen"
+ avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
+ subject="git-annex on Synology DS216+ (x86-64)"
+ date="2017-05-28T01:19:16Z"
+ content="""
+I used a similar approach, with a bit of refinement to not require a custom ssh key/user, to [get `git-annex` running on a Synology DS216+ NAS](http://ewen.mcneill.gen.nz/blog/entry/2017-05-28-git-annex-on-synology-ds216+/), which is based around a Celeron chip (and thus needs the `x86-64` build).  Once all the `PATH` related issues are taken care of (which some symlinks into `/usr/bin`) it appears to work like any other Linux/Unix-based `git-annex` install.  (Definitely much more successful than [trying to use `git-annex` via a SMB share](http://git-annex.branchable.com/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/) at this point.)
+
+Ewen
+"""]]

diff --git a/doc/forum/Preferred_content_settings_for_sets_of_backup_drives__63__.mdwn b/doc/forum/Preferred_content_settings_for_sets_of_backup_drives__63__.mdwn
new file mode 100644
index 0000000..6ddb1e0
--- /dev/null
+++ b/doc/forum/Preferred_content_settings_for_sets_of_backup_drives__63__.mdwn
@@ -0,0 +1,3 @@
+I have one main 4TB annex and 4 2TB drives that I would like to use for long-term backups. The plan is to have two complete backups, one at home and one in a different state, and swap + update them occasionally. If I put an annex on each drive, how can I tell git annex that every file should be duplicated to one or the other from each set? Like this:
+
+main and (backup1a or backup1b) and (backup2a or backup2b)

Added a comment: woohoo
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_5_03e474b00b1cb2a5b8244d9161857b9b._comment b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_5_03e474b00b1cb2a5b8244d9161857b9b._comment
new file mode 100644
index 0000000..abc1d70
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_5_03e474b00b1cb2a5b8244d9161857b9b._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="woohoo"
+ date="2017-05-26T02:03:57Z"
+ content="""
+Thank you Joey!  It seems to work very nice now!  Not a single one lost out of 1550!
+
+[[!format sh \"\"\"
+$> datalad get -J5 38*
+[INFO   ] Getting 100 items of dataset <Dataset path=/tmp/travis-buildlogs> ... 
+[INFO   ] Actually getting 1550 files 
+Tried to get 1550 files. Got 1550. 
+\"\"\"]]
+"""]]

Avoid concurrent git-config setting problem when running concurrent threads.
See my comment. This only avoids the problem for -J; two git-annex
processes started at the same time could still both try to write to
.git/config and one fail. That would be very unlikely though, and it
doesn't really seem worth adding an additional layer of locking around
.git/config.
This commit was supported by the NSF-funded DataLad project.
diff --git a/CHANGELOG b/CHANGELOG
index bf91667..1232b09 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,6 +6,8 @@ git-annex (6.20170520) UNRELEASED; urgency=medium
     a transfer does not resume.
   * Fix transfer log file locking problem when running concurrent
     transfers.
+  * Avoid concurrent git-config setting problem when running concurrent
+    threads.
 
  -- Joey Hess <id@joeyh.name>  Wed, 24 May 2017 14:03:40 -0400
 
diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
index 27621e4..75c9e94 100644
--- a/CmdLine/Action.hs
+++ b/CmdLine/Action.hs
@@ -16,6 +16,7 @@ import Types.Command
 import Types.Concurrency
 import Messages.Concurrent
 import Types.Messages
+import Remote.List
 
 import Control.Concurrent.Async
 import Control.Exception (throwIO)
@@ -57,6 +58,12 @@ commandAction a = go =<< Annex.getState Annex.concurrency
 		ws <- Annex.getState Annex.workers
 		(st, ws') <- if null ws
 			then do
+				-- Generate the remote list now, to avoid
+				-- each thread generating it, which would
+				-- be more expensive and could cause
+				-- threads to contend over eg, calls to
+				-- setConfig.
+				_ <- remoteList
 				st <- dupState
 				return (st, replicate (n-1) (Left st))
 			else do
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
index 4725f9a..10dbd55 100644
--- a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
@@ -29,3 +29,5 @@ SHA256E-s328--c2eb8088cdc71a0d4cbd660312bef5421a47ce7da3655efdb17712e7188be4a1.t
 """]]
 
 [[!meta author=yoh]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_4_480df575c68bfb37b8bb4fb43737726f._comment b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_4_480df575c68bfb37b8bb4fb43737726f._comment
new file mode 100644
index 0000000..3827077
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_4_480df575c68bfb37b8bb4fb43737726f._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2017-05-25T21:52:37Z"
+ content="""
+The .git/config concurrent access happens because the remote
+list is only generated on demand, and nothing demands it when running with
+-J until all the threads are spun up and each thread has its own state
+then, so each generates the remote list.
+
+There don't look to be any other git-config settings that
+would cause problems for concurrency other than ones run while
+generating the remote list.
+
+So, generating the remote list before starting concurrent threads
+would avoid that problem, and also leads to a slightly faster startup
+since the remote git config only has to be read once, etc.
+
+The only risk in doing that would be if generating a Remote
+opens some kind of handle, which can't be used concurrently, or
+is less efficient if only opened once and then used by multiple threads.
+
+I've audited all the Remote.*.gen methods, and they all
+seem ok. For example, Remote.External.gen sets up a worker pool
+of external special remote processes, and new ones are allocated as needed.
+And Remote.P2P.chainGen sets up a connection pool.
+
+Ok, gone ahead with this fix.
+"""]]

Fix transfer log file locking problem when running concurrent transfers.
orElse is great, but was not the right thing to use here because
waitTakeLock could retry for other reasons than the lock being held,
which made tryTakeLock fail when it shouldn't.
Instead, move the code to tryTakeLock and implement waitTakeLock using
tryTakeLock and retry.
(Also, in runTransfer, when checkSaneLock fails, dropLock to avoid leaking a
lock handle.)
This commit was supported by the NSF-funded DataLad project.
diff --git a/Annex/Transfer.hs b/Annex/Transfer.hs
index 87480b2..3fcf1a1 100644
--- a/Annex/Transfer.hs
+++ b/Annex/Transfer.hs
@@ -118,7 +118,9 @@ runTransfer' ignorelock t afile shouldretry transferaction = checkSecureHashes t
 					void $ liftIO $ tryIO $
 						writeTransferInfoFile info tfile
 					return (Just lockhandle, False)
-				, return (Nothing, True)
+				, do
+					liftIO $ dropLock lockhandle
+					return (Nothing, True)
 				)
 #else
 	prep tfile _mode info = catchPermissionDenied (const prepfailed) $ do
diff --git a/CHANGELOG b/CHANGELOG
index dd408a0..bf91667 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,8 @@ git-annex (6.20170520) UNRELEASED; urgency=medium
     exclamation mark, which forces gpg to use a specific subkey.
   * Improve progress display when watching file size, in cases where
     a transfer does not resume.
+  * Fix transfer log file locking problem when running concurrent
+    transfers.
 
  -- Joey Hess <id@joeyh.name>  Wed, 24 May 2017 14:03:40 -0400
 
diff --git a/Utility/LockPool/STM.hs b/Utility/LockPool/STM.hs
index d1ee0db..5cb7b88 100644
--- a/Utility/LockPool/STM.hs
+++ b/Utility/LockPool/STM.hs
@@ -49,9 +49,9 @@ type LockPool = TMVar (M.Map LockFile LockStatus)
 
 -- A shared global variable for the lockPool. Avoids callers needing to
 -- maintain state for this implementation detail.
+{-# NOINLINE lockPool #-}
 lockPool :: LockPool
 lockPool = unsafePerformIO (newTMVarIO M.empty)
-{-# NOINLINE lockPool #-}
 
 -- Updates the LockPool, blocking as necessary if another thread is holding
 -- a conflicting lock.
@@ -62,23 +62,23 @@ lockPool = unsafePerformIO (newTMVarIO M.empty)
 -- Keeping the whole Map in a TMVar accomplishes this, at the expense of
 -- sometimes retrying after unrelated changes in the map.
 waitTakeLock :: LockPool -> LockFile -> LockMode -> STM LockHandle
-waitTakeLock pool file mode = do
-	m <- takeTMVar pool
-	v <- case M.lookup file m of
-		Just (LockStatus mode' n closelockfile)
-			| mode == LockShared && mode' == LockShared ->
-				return $ LockStatus mode (succ n) closelockfile
-			| n > 0 -> retry -- wait for lock
-		_ -> return $ LockStatus mode 1 noop
-	putTMVar pool (M.insert file v m)
-	newTMVar (pool, file)
+waitTakeLock pool file mode = maybe retry return =<< tryTakeLock pool file mode
 
 -- Avoids blocking if another thread is holding a conflicting lock.
 tryTakeLock :: LockPool -> LockFile -> LockMode -> STM (Maybe LockHandle)
-tryTakeLock pool file mode =
-	(Just <$> waitTakeLock pool file mode)
-		`orElse`
-	return Nothing
+tryTakeLock pool file mode = do
+	m <- takeTMVar pool
+	let success v = do
+		putTMVar pool (M.insert file v m)
+		Just <$> newTMVar (pool, file)
+	case M.lookup file m of
+		Just (LockStatus mode' n closelockfile)
+			| mode == LockShared && mode' == LockShared ->
+				success $ LockStatus mode (succ n) closelockfile
+			| n > 0 -> do
+				putTMVar pool m
+				return Nothing
+		_ -> success $ LockStatus mode 1 noop
 
 -- Call after waitTakeLock or tryTakeLock, to register a CloseLockFile
 -- action to run when releasing the lock.
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_3_6674e4dbc7437ce941bcef6272c3433b._comment b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_3_6674e4dbc7437ce941bcef6272c3433b._comment
new file mode 100644
index 0000000..a004b75
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_3_6674e4dbc7437ce941bcef6272c3433b._comment
@@ -0,0 +1,40 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-25T19:08:59Z"
+ content="""
+That looks like concurrent `git config` setting remote.origin.annex-uuid
+are failing.
+
+I have not reproduced the `.git/config` error, but with a local
+clone of a repository, I have been able to reproduce some intermittent
+"transfer already in progress, or unable to take transfer lock" failures
+with `git annex get -J5`, happening after remote.origin.annex-uuid has been
+cached.
+
+So, two distinct bugs I think..
+
+---
+
+Debugging, the lock it fails to take always seems to be the lock on
+the remote side, which points to the local clone being involved somehow.
+
+Debugging further, Utility.LockPool.STM.tryTakeLock is what's failing.
+That's supposed to only fail when another thread holds a conflicting lock,
+but as it's implemented with `orElse`, if the main STM
+transaction retries due to other STM activity on the same TVar,
+it will give up when it shouldn't.
+
+That's probably why this is happening under heavier concurrency loads;
+it makes that failure case much more likely. And with a local clone,
+twice as much locking is done.
+
+I've fixed this part of it! 
+
+---
+
+The concurrent `git config` part remains.
+Since git-annex can potentially have multiple threads doing different `git
+config` for their own reasons concurrently, it seems it will need to add
+its own locking around that.
+"""]]

Added a comment
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_2_2ba8101ed0e91df7079744d1b37779fd._comment b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_2_2ba8101ed0e91df7079744d1b37779fd._comment
new file mode 100644
index 0000000..c9a6f13
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_2_2ba8101ed0e91df7079744d1b37779fd._comment
@@ -0,0 +1,51 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 2"
+ date="2017-05-25T18:39:03Z"
+ content="""
+didn't need to go far ;)
+
+[[!format sh \"\"\"
+$> git annex get -J5 
+(merging origin/git-annex into git-annex...)
+(recording state in git...)
+get 10/10.1-None.txt.gz get 1000/1000.2-0.txt.gz get 1000/1000.1-0.txt.gz get 1000/1000.3-0.txt.gz get 1000/1000.4-1.txt.gz error: could not lock config file .git/config: File exists
+(from origin...) (from origin...) (from origin...) 
+
+(transfer already in progress, or unable to take transfer lock) 
+  Unable to access these remotes: origin
+(from origin...) 
+
+  Try making some of these repositories available:
+  	3db23446-8c40-441e-97ec-55ffc86b4fc0 -- yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]
+failed
+SHA256E-s55194--48621840b3a869ac27dfdb2a201c202ee1382aa0114337563da2c38262a0c9e8.txt.gz
+         55,194 100%   21.39MB/s    0:00:00 (xfr#1, to-chk=0/1)
+SHA256E-s56291--48af9865017aca9c8931c3b91c4f0ff9954273d21007ff25de1a2c73ca0df95cSHA256E-s7813--7861750b475c620012f1a76635749d89b86b7777130352b6adb844753d3822c2.txt.gz
+.txt.gz
+         32,768  58%    0.00kB/s    0:00:00  (checksum...)    0:00:00 (xfr#1, to-chk=0/1)
+         56,291 100%   22.43MB/s    0:00:00 (xfr#1, to-chk=0/1)
+(checksum...) (checksum...) ok
+ok
+ok
+git-annex: git [Param \"config\",Param \"remote.origin.annex-uuid\",Param \"3db23446-8c40-441e-97ec-55ffc86b4fc0\"] failed
+CallStack (from HasCallStack):
+  error, called at ./Git/Command.hs:39:17 in main:Git.Command
+
+$> git annex get 2>&1 | head
+get 1000/1000.2-0.txt.gz (from origin...) 
+SHA256E-s56206--1fa38270b23b4c700c4a998712549c65782613d4567b8650b1ebaa3fcfc884f4.txt.gz
+         56,206 100%   22.35MB/s    0:00:00 (xfr#1, to-chk=0/1)
+(checksum...) ok
+get 1000/1000.3-0.txt.gz (from origin...) 
+SHA256E-s55880--dea4463f7e42c51463e236d29c4c40486b56754cbd8340c260e481bdd506115b.txt.gz
+         55,880 100%   22.04MB/s    0:00:00 (xfr#1, to-chk=0/1)
+(checksum...) ok
+get 1000/1000.5-1.txt.gz (from origin...) 
+SHA256E-s55028--f6f8d5f25bed2bb51d9ba17275a956a97276cdcc74c6538c7e590919426e97df.txt.gz
+
+\"\"\"]]
+
+FWIW, the repository in question is this one: http://datasets.datalad.org/devel/travis-buildlogs/.git/
+"""]]

Improve progress display when watching file size, in cases where a transfer does not resume.
This commit was supported by the NSF-funded DataLad project.
diff --git a/CHANGELOG b/CHANGELOG
index 0d65af1..dd408a0 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,8 @@ git-annex (6.20170520) UNRELEASED; urgency=medium
 
   * initremote, enableremote: Support gpg subkeys suffixed with an
     exclamation mark, which forces gpg to use a specific subkey.
+  * Improve progress display when watching file size, in cases where
+    a transfer does not resume.
 
  -- Joey Hess <id@joeyh.name>  Wed, 24 May 2017 14:03:40 -0400
 
diff --git a/Utility/Metered.hs b/Utility/Metered.hs
index 626aa2c..a5dda54 100644
--- a/Utility/Metered.hs
+++ b/Utility/Metered.hs
@@ -171,22 +171,27 @@ defaultChunkSize = 32 * k - chunkOverhead
 	k = 1024
 	chunkOverhead = 2 * sizeOf (1 :: Int) -- GHC specific
 
-{- Runs an action, watching a file as it grows and updating the meter. -}
+{- Runs an action, watching a file as it grows and updating the meter.
+ -
+ - The file may already exist, and the action could throw the original file
+ - away and start over. To avoid reporting the original file size followed
+ - by a smaller size in that case, wait until the file starts growing
+ - before updating the meter for the first time.
+ -}
 watchFileSize :: (MonadIO m, MonadMask m) => FilePath -> MeterUpdate -> m a -> m a
 watchFileSize f p a = bracket 
-	(liftIO $ forkIO $ watcher zeroBytesProcessed)
+	(liftIO $ forkIO $ watcher =<< getsz)
 	(liftIO . void . tryIO . killThread)
 	(const a)
   where
 	watcher oldsz = do
-		v <- catchMaybeIO $ toBytesProcessed <$> getFileSize f
-		newsz <- case v of
-			Just sz | sz /= oldsz -> do
-				p sz
-				return sz
-			_ -> return oldsz
 		threadDelay 500000 -- 0.5 seconds
-		watcher newsz
+		sz <- getsz
+		when (sz > oldsz) $
+			p sz
+		watcher sz
+	getsz = catchDefaultIO zeroBytesProcessed $
+		toBytesProcessed <$> getFileSize f
 
 data OutputHandler = OutputHandler
 	{ quietMode :: Bool
diff --git a/doc/bugs/__34__byte-progress__34___could_jump_down_upon_initiating_re-download_--_report_actual_one_first__63__/comment_1_ee95564fafba601246df3de57500eb1c._comment b/doc/bugs/__34__byte-progress__34___could_jump_down_upon_initiating_re-download_--_report_actual_one_first__63__/comment_1_ee95564fafba601246df3de57500eb1c._comment
new file mode 100644
index 0000000..7f87e22
--- /dev/null
+++ b/doc/bugs/__34__byte-progress__34___could_jump_down_upon_initiating_re-download_--_report_actual_one_first__63__/comment_1_ee95564fafba601246df3de57500eb1c._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-25T17:56:48Z"
+ content="""
+That looks like a git remote accessed perhaps by rsync, or perhaps locally?
+
+I'd be surprised if a rsync transfer did this, because AFAIK all progress
+updates come from rsync's own progress display, and that does not jump
+backward.
+
+Local file copies (when not using rsync), and some other types of remotes,
+poll the size of the temp file to determine how much data has been
+received, and so if the transfer doesn't resume, they will do this. **I've
+made it avoid reporting the file size until the file size has changed once,
+which avoids the problem in this case.**
+
+Another way it could happen is when a transfer fails partway and git-annex
+immediately retries and the retry fails to resume. In
+this case, the progress would go to some percent for the first transfer,
+and then could reset to a lower percent for the retry, and that
+reflects what's really happening. Eg, 50% of it transferred and now
+we've unfortunately started over at 0%.
+
+I could make the reported progress always be monotonically increasing, but
+then in that retry cases it would just seem to stall, perhaps for a long
+period of time. Not sure that's better than a progress display that while
+annoying, reflects what's really going on.
+"""]]

Added a comment
diff --git a/doc/forum/Git_annex_hangs/comment_12_053666d256e26803b911d68622257cd5._comment b/doc/forum/Git_annex_hangs/comment_12_053666d256e26803b911d68622257cd5._comment
new file mode 100644
index 0000000..abe9947
--- /dev/null
+++ b/doc/forum/Git_annex_hangs/comment_12_053666d256e26803b911d68622257cd5._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="comment 12"
+ date="2017-05-25T18:25:59Z"
+ content="""
+I checked the file system, it's nfs.
+
+`git config --get annex.pidlock` gives nothing, so I guess it's not enabled. Do I have to enable it, if the repo is on an nfs mount?
+
+"""]]

response
diff --git a/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__/comment_1_26cb6d46a408c1002aa2fbc6618242f3._comment b/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__/comment_1_26cb6d46a408c1002aa2fbc6618242f3._comment
new file mode 100644
index 0000000..33a4806
--- /dev/null
+++ b/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__/comment_1_26cb6d46a408c1002aa2fbc6618242f3._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-25T17:51:26Z"
+ content="""
+You must have rsynced over the .git/config from A. That is where the UUID
+of the repository is stored. So B now has whatever git config settings you
+had in A. The reinit fixed the UUID; there could be other things, like
+remotes, that are configured wrong. Otherwise, the way you recovered seems
+fine.
+"""]]

comment
diff --git a/doc/forum/Git_annex_hangs/comment_11_e0b4616c25405fa3c496f7e85800ac43._comment b/doc/forum/Git_annex_hangs/comment_11_e0b4616c25405fa3c496f7e85800ac43._comment
new file mode 100644
index 0000000..7c11079
--- /dev/null
+++ b/doc/forum/Git_annex_hangs/comment_11_e0b4616c25405fa3c496f7e85800ac43._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2017-05-25T17:47:53Z"
+ content="""
+So there is some kind of networked filesystem involved? What kind?
+
+Do you have annex.pidlock enabled in the repository's `git config`?
+"""]]

comment
diff --git a/doc/forum/Different_branches_for_different_devices__63__/comment_3_534f1e2cfcc0b6d184ef4968ec33dc89._comment b/doc/forum/Different_branches_for_different_devices__63__/comment_3_534f1e2cfcc0b6d184ef4968ec33dc89._comment
new file mode 100644
index 0000000..54642fd
--- /dev/null
+++ b/doc/forum/Different_branches_for_different_devices__63__/comment_3_534f1e2cfcc0b6d184ef4968ec33dc89._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-25T17:44:59Z"
+ content="""
+You'd need to checkout the phone branch on the laptop and make the changes
+to it. So there can be some extra work involved since you have to maintain
+two branches. In some cases `git cherry-pick` could be used to pick a
+change from one branch to the other.
+"""]]

improve docs, mention git config receive.denyCurrentBranch updateInstead
diff --git a/doc/git-annex-sync.mdwn b/doc/git-annex-sync.mdwn
index cabe5fe..a27d315 100644
--- a/doc/git-annex-sync.mdwn
+++ b/doc/git-annex-sync.mdwn
@@ -26,8 +26,9 @@ versions of a file have been committed, both will be added to the tree,
 under different filenames. For example, file "foo" would be replaced
 with "foo.somekey" and "foo.otherkey".
 
-Note that syncing with a remote will not update the remote's working
-tree with changes made to the local repository. However, those changes
+Note that syncing with a remote will not normally update the remote's working
+tree with changes made to the local repository. (Unless it's configured
+with receive.denyCurrentBranch=updateInstead.) However, those changes
 are pushed to the remote, so they can be merged into its working tree
 by running "git annex sync" on the remote.
 

response
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_1_1173126bd91dcb7fd13c57f06fd16c2b._comment b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_1_1173126bd91dcb7fd13c57f06fd16c2b._comment
new file mode 100644
index 0000000..4695182
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_/comment_1_1173126bd91dcb7fd13c57f06fd16c2b._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-25T17:29:08Z"
+ content="""
+Let's not diagnose a concurrency problem prematurely.
+
+I've run many concurrent gets and never seen masses of failures, or indeed any
+failures without either an error message explaining why it failed, or something
+obviously wrong (like the drive not being mounted).
+
+The json output is making it harder than necessary to understand what's going on.
+It seems you should be easily able to replicate the same problem without --json.
+"""]]

original complaint
diff --git a/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
new file mode 100644
index 0000000..4725f9a
--- /dev/null
+++ b/doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
@@ -0,0 +1,31 @@
+### Please describe the problem.
+
+Well, I remember whining before on this issue.  It does cause some notable inconvenience when trying to automate use of annex -- I am left without a choice but try to reget files multiple times without knowing what was actual cause for it to fail to start with.  Originally I observed it while content had to be wget'ed, to which I thought "oh well -- might be some connection overload etc".  But now I have tried on a repository which is local to that drive -- there must be no problem accessing multiple files at once whatsoever.  But on a trial X with -J5 I have error: 613, ok: 255, on next call  error: 415,  ok: 198 and so on -- kinda converging but imho it must not be so difficult.  I do suspect some race condition in annex itself preventing correct operation.
+
+### What steps will reproduce the problem?
+
+git annex get -J10   on some well sized annex.  Possibly with original annex to fetch content from just right on the same drive
+
+### What version of git-annex are you using? On what operating system?
+
+6.20170408+gitg804f06baa-1~ndall+1
+
+### Please provide any additional information below.
+
+[[!format sh """
+$> git annex get -J5 --json | grep -v '"success":true' 2>&1 | head
+{"command":"get","wanted":[{"here":false,"uuid":"3db23446-8c40-441e-97ec-55ffc86b4fc0","description":"yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]"}],"note":"Try making some of these repositories available:\n\t3db23446-8c40-441e-97ec-55ffc86b4fc0 -- yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]\n","skipped":[],"success":false,"key":"SHA256E-s328--c2eb8088cdc71a0d4cbd660312bef5421a47ce7da3655efdb17712e7188be4a1.txt.gz","file":"3728/3728.9-None.txt.gz"}
+{"command":"get","wanted":[{"here":false,"uuid":"3db23446-8c40-441e-97ec-55ffc86b4fc0","description":"yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]"}],"note":"Try making some of these repositories available:\n\t3db23446-8c40-441e-97ec-55ffc86b4fc0 -- yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]\n","skipped":[],"success":false,"key":"SHA256E-s62840--c20189a229fac622bb781650af394cf40367b5563a833885480f
+825fdbf29b47.txt.gz","file":"3729/3729.1-0.txt.gz"}
+
+...
+$> git annex get --key SHA256E-s328--c2eb8088cdc71a0d4cbd660312bef5421a47ce7da3655efdb17712e7188be4a1.txt.gz
+get SHA256E-s328--c2eb8088cdc71a0d4cbd660312bef5421a47ce7da3655efdb17712e7188be4a1.txt.gz (from origin...)
+SHA256E-s328--c2eb8088cdc71a0d4cbd660312bef5421a47ce7da3655efdb17712e7188be4a1.txt.gz
+            328 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)
+(checksum...) ok
+(recording state in git...)
+
+"""]]
+
+[[!meta author=yoh]]

Added a comment
diff --git a/doc/forum/git-annex_sync_here/comment_2_1b0543443759202ec6a85a26088ddb43._comment b/doc/forum/git-annex_sync_here/comment_2_1b0543443759202ec6a85a26088ddb43._comment
new file mode 100644
index 0000000..ba3ff18
--- /dev/null
+++ b/doc/forum/git-annex_sync_here/comment_2_1b0543443759202ec6a85a26088ddb43._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="comment 2"
+ date="2017-05-25T12:47:18Z"
+ content="""
+git-annex sync does more than that. Especially merging the synced branches and, thus, showing files that were pushed into a repo from a remote. Also, git commit -a cannot be done in direct mode.
+But, I think I can just create an alias for git-annex sync --no-push --no-pull to save me some typing.
+"""]]

Added a comment
diff --git a/doc/forum/Different_branches_for_different_devices__63__/comment_2_214c71d2ab43e4d1bf564ea8337909b7._comment b/doc/forum/Different_branches_for_different_devices__63__/comment_2_214c71d2ab43e4d1bf564ea8337909b7._comment
new file mode 100644
index 0000000..f06be60
--- /dev/null
+++ b/doc/forum/Different_branches_for_different_devices__63__/comment_2_214c71d2ab43e4d1bf564ea8337909b7._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="comment 2"
+ date="2017-05-25T12:01:46Z"
+ content="""
+Hi, thanks for your help.
+
+But I don't get it entirely. Let's say I move photos from the phone folder into another folder (on the laptop, after merging, in the master branch). How do I propagate these changes back into the phone branch? Since, I guess, merging would only work one-way in this setting. Or let's say I want to push a new picture from my laptop into the phone branch, how would I do this?
+"""]]

thoughts
diff --git a/doc/todo/export/comment_3_38e0b7bac8f2cfad492704ab6ab81e2b._comment b/doc/todo/export/comment_3_38e0b7bac8f2cfad492704ab6ab81e2b._comment
new file mode 100644
index 0000000..307ff40
--- /dev/null
+++ b/doc/todo/export/comment_3_38e0b7bac8f2cfad492704ab6ab81e2b._comment
@@ -0,0 +1,42 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-24T18:59:30Z"
+ content="""
+`storeKey` could not be used for this, so it would need a new remote method
+to store a file on the remote under a specified name. Call it `storeFile`.
+
+What should `storeFile` with an encrypted special remote do? Encrypting
+the data does not seem very useful, especially for hybrid and shared
+where the secret key is embedded in the git repo. Not encrypting the data
+is surely a least surprise violation that would be a security hole.
+So probably best to not support exporting to encrypted special remotes.
+
+`git annex export --to specialremote` cannot handle incremental updates,
+resuming uploads etc, because special remotes can be so limited they only
+support putting the whole content of a file. Even resuming interrupted
+uploads is problimatic, because we don't know for sure what key was
+(partially or completely) exported before. The best that seems doable
+is to make `storeFile` avoid resending the file if the remote has the file
+on it already, and move the file into place atomically once it's all sent. 
+
+Then `git annex export --to remote` could be run repeatedly to export files
+until everything gets exported.
+
+But, when a file has been modified, that would prevent the modified version
+being exported. Unless state is stored somewhere to say that the given file
+on a remote is the content of a given key.
+That state could take the form of a .git-annex/filename.exported stored
+on the remote, which contains the name of the key. When exporting a new
+key over an existing file, that would be overwritten. (Really needs to be
+done atomically..)
+
+What about deleted files? Should export somehow notice that a file
+has been deleted locally, and remove it from the remote? How?
+
+----
+
+Alternatively, leave the incremental updating, deletion etc to
+third-party tools, and have `git annex export` simply export the current
+files to a directory.
+"""]]

more consistent backticks with other man pages
diff --git a/doc/git-annex-adjust.mdwn b/doc/git-annex-adjust.mdwn
index 772d27b..6f32f30 100644
--- a/doc/git-annex-adjust.mdwn
+++ b/doc/git-annex-adjust.mdwn
@@ -4,7 +4,7 @@ git-annex adjust - enter an adjusted branch
 
 # SYNOPSIS
 
-`git annex adjust --unlock|--fix`
+git annex adjust `--unlock|--fix`
 
 # DESCRIPTION
 

initremote, enableremote: Support gpg subkeys suffixed with an exclamation mark, which forces gpg to use a specific subkey.
This commit was sponsored by Peter Hogg on Patreon.
diff --git a/CHANGELOG b/CHANGELOG
index bd9cb32..0d65af1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,10 @@
+git-annex (6.20170520) UNRELEASED; urgency=medium
+
+  * initremote, enableremote: Support gpg subkeys suffixed with an
+    exclamation mark, which forces gpg to use a specific subkey.
+
+ -- Joey Hess <id@joeyh.name>  Wed, 24 May 2017 14:03:40 -0400
+
 git-annex (6.20170519) unstable; urgency=medium
 
   * Ssh password prompting improved when using -J for concurrency.
diff --git a/Utility/Gpg.hs b/Utility/Gpg.hs
index 336711b..c20be75 100644
--- a/Utility/Gpg.hs
+++ b/Utility/Gpg.hs
@@ -157,7 +157,11 @@ pipeLazy (GpgCmd cmd) params feeder reader = do
  - a key id, or a name; See the section 'HOW TO SPECIFY A USER ID' of
  - GnuPG's manpage.) -}
 findPubKeys :: GpgCmd -> String -> IO KeyIds
-findPubKeys cmd for = KeyIds . parse . lines <$> readStrict cmd params
+findPubKeys cmd for
+	-- "subkey!" tells gpg to force use of a specific subkey,
+	-- so pass it through as-is rather than looking up the master key.
+	| "!" `isSuffixOf` for = return $ KeyIds [for]
+	| otherwise = KeyIds . parse . lines <$> readStrict cmd params
   where
 	params = [Param "--with-colons", Param "--list-public-keys", Param for]
 	parse = mapMaybe (keyIdField . splitc ':')
diff --git a/doc/encryption/comment_10_6416ee43ffad1c306ef71247ae71a6c5._comment b/doc/encryption/comment_10_6416ee43ffad1c306ef71247ae71a6c5._comment
new file mode 100644
index 0000000..ae2fb88
--- /dev/null
+++ b/doc/encryption/comment_10_6416ee43ffad1c306ef71247ae71a6c5._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2017-05-24T17:47:46Z"
+ content="""
+@Yurt, git-annex will let you specify the gpg key id using anything that gpg
+accepts, including a keyid with a appended '!'. However, when I tried that,
+gpg seemed to still pick the master key instead of the subkey. That
+happens because git-annex runs the input through `gpg --list-public-keys`
+(in order to convert eg, email addresses to key ids)
+which always lists the master key even when given a subkey.
+
+I made a small change to git-annex to special case this '!' suffix
+behavior. Seems to work in my very limited testing.
+
+Please file bug reports about this kind of thing!
+"""]]

comment
diff --git a/doc/forum/Different_branches_for_different_devices__63__/comment_1_a7d2c26298e8da715e1deb13baaf002e._comment b/doc/forum/Different_branches_for_different_devices__63__/comment_1_a7d2c26298e8da715e1deb13baaf002e._comment
new file mode 100644
index 0000000..edeee3b
--- /dev/null
+++ b/doc/forum/Different_branches_for_different_devices__63__/comment_1_a7d2c26298e8da715e1deb13baaf002e._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-24T17:41:08Z"
+ content="""
+Don't underestimate the simple power of branches for this kind of thing.
+If you have a limited `phone` branch, you can `git merge` it into a larger
+branch at home.
+"""]]

response
diff --git a/doc/forum/Malicious_autoenabled_remotes/comment_3_608213f5d0df482b731ae1502cdd87af._comment b/doc/forum/Malicious_autoenabled_remotes/comment_3_608213f5d0df482b731ae1502cdd87af._comment
new file mode 100644
index 0000000..702be5f
--- /dev/null
+++ b/doc/forum/Malicious_autoenabled_remotes/comment_3_608213f5d0df482b731ae1502cdd87af._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-24T17:27:20Z"
+ content="""
+The server can certainly do filtering or blocking of changes to the
+git-annex branch to prevent this kind of abuse.
+
+Marking a repository as dead will indeed prevent it from being
+auto-enabled. It will not cause later synchronisation problems. It seems
+like a perhaps too big hammer though. Cloning from such a server, and then
+pushing back to it would make your clone be marked as dead on the next
+pull!
+
+And marking dead doesn't prevent malicious changes to preferred
+content settings etc.
+
+Filtering in the `pre-receive` hook should be very doable. See
+[[internals]] for the git-annex branch documentation.
+"""]]

mention autoenable=true
diff --git a/doc/internals.mdwn b/doc/internals.mdwn
index 00b65d2..4ed8001 100644
--- a/doc/internals.mdwn
+++ b/doc/internals.mdwn
@@ -119,6 +119,8 @@ The file format is one line per remote, starting with the uuid of the
 remote, followed by a space, and then a series of var=value pairs,
 each separated by whitespace, and finally a timestamp.
 
+Special remotes that are autoenabled have autoenable=true here.
+
 Encrypted special remotes store their encryption key here,
 in the "cipher" value. It is base64 encoded, and unless shared [[encryption]]
 is used, is encrypted to one or more gpg keys. The first 256 bytes of

comment
diff --git a/doc/forum/git-annex_help_import_outdated__63__/comment_2_ca898237f42af1205acab509e4d67d38._comment b/doc/forum/git-annex_help_import_outdated__63__/comment_2_ca898237f42af1205acab509e4d67d38._comment
new file mode 100644
index 0000000..f5a9c8a
--- /dev/null
+++ b/doc/forum/git-annex_help_import_outdated__63__/comment_2_ca898237f42af1205acab509e4d67d38._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-05-24T17:03:11Z"
+ content="""
+The man page is not outdated, see
+<https://git-annex.branchable.com/git-annex-import/>
+
+It seem that `git help`'s handling of MANPATH makes it not look
+first at the man pages from the standalone tarball in this case,
+and instead it displays the system man page.
+
+`git annex importfeed --help` will display the shorter documentation that's
+built into the binary.
+"""]]

comment
diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_3_61924cf10643a6d2d011899d95b4ae56._comment b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_3_61924cf10643a6d2d011899d95b4ae56._comment
new file mode 100644
index 0000000..0025126
--- /dev/null
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_3_61924cf10643a6d2d011899d95b4ae56._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-24T17:08:59Z"
+ content="""
+Well, there's no way I know of to make git treat a directory as one item in
+history. So even if git-annex didn't, it would not help much.
+
+You can check a tar archive into git-annex as a single file, and have hooks
+to handle unpacking it into the directory and repacking the directory back
+to it.
+"""]]

response
diff --git a/doc/forum/git-annex_sync_here/comment_1_3ebef27614bd72b38404c1940007e68c._comment b/doc/forum/git-annex_sync_here/comment_1_3ebef27614bd72b38404c1940007e68c._comment
new file mode 100644
index 0000000..7003ce5
--- /dev/null
+++ b/doc/forum/git-annex_sync_here/comment_1_3ebef27614bd72b38404c1940007e68c._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-24T17:06:29Z"
+ content="""
+Well, that does the same thing as `git commit -a`, so I don't see
+any benefit complicating the command with additional syntax.
+"""]]

diff --git a/doc/forum/Different_branches_for_different_devices__63__.mdwn b/doc/forum/Different_branches_for_different_devices__63__.mdwn
new file mode 100644
index 0000000..4b9154a
--- /dev/null
+++ b/doc/forum/Different_branches_for_different_devices__63__.mdwn
@@ -0,0 +1,17 @@
+Hi,
+
+I'm currently trying to figure out how to best use git-annex for my needs. And I have a specific use case on which I can't really wrap my head around:
+
+Having a large photo library that collects all photos from all my devices.
+
+What's special about that? Some devices (e.g. mobile phone, laptop, laptop at work) should have/keep their own "view" on the data. Meaning that, e.g., on my laptop at work I don't want to have private pictures, but I might want to have work pictures on my private laptop. Similar for my mobile phone: I want to have all photos for my mobile phone on my laptop, but I don't want all pictures I have ever made on my (tiny) phone.
+
+So, first of all, git-annex supports get, drop, preferred content, etc. which makes it perfect for this use case. Still there is one problem: It's not only that I don't want the content on some devices, I also don't want the respective directory names, simlinks, etc. Say, a directory name like "crazy party pictures" should not be visible on my work laptop, at all...
+
+For me, using directories like "phone", "work", etc. would be a start and a reasonable way to sort which content should be available on these devices. However, as far as I know, it's not possible with GIT to just check out a single (or a few) directories. So I thought different branches could be what I want. But I don't understand how I can copy/move content from one branch to another. Or even, how can I see all the content at once on some machines (e.g. private laptop) without switching branches all the time. I could have several copies of the same repo all set to a different branch, but does this make sense..?
+
+I think what I want is similar to git-annex view, but only for its filter capabilities. Still, subdirectories should work the same as before and filenames should not change. Also the repository should still be fully functional and behaving as usual but just hiding some of the content.
+
+Do you people have similar use cases? What are your best practices?
+
+Best, Mario

Added a comment
diff --git a/doc/forum/Malicious_autoenabled_remotes/comment_2_37456c0a159453fa984c5a003578d1eb._comment b/doc/forum/Malicious_autoenabled_remotes/comment_2_37456c0a159453fa984c5a003578d1eb._comment
new file mode 100644
index 0000000..6de3506
--- /dev/null
+++ b/doc/forum/Malicious_autoenabled_remotes/comment_2_37456c0a159453fa984c5a003578d1eb._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="achilleas.k@14be77d42a1252fab5ec9dbf4e5ea03c5833e8c8"
+ nickname="achilleas.k"
+ avatar="http://cdn.libravatar.org/avatar/ed6c67c4d8e6c6850930e16eaf85a771"
+ subject="comment 2"
+ date="2017-05-22T14:40:21Z"
+ content="""
+Hey, thanks for the feedback and your thoughts. Should have gotten back to you sooner on this.
+
+I wanted to share with you my thoughts about getting around this issue, from the point of view of the `trustedserver` administrator, and get your input on this.
+
+I want to run a server that uses git and git annex for data storage. I want users of this server to feel safe that when they clone a repository and sync content, they're not pulling things from an untrusted server. I was thinking of modifying annex configurations serverside, perhaps as a *post-receive* hook. The idea would be to go through the remotes on the serveride, bare git repository, and mark all unknown (ssh, rsync, etc) remotes as a `dead`.
+
+Would this cause any issues for the receiver or the sender? Other than potentially making files for the receiver unavailable (which is what I want), would it possibly put the repository in a state where the original sender can't push more changes, because of a disagreement between configurations?
+
+I've played around with the idea a bit and I think the idea is pretty safe, but I might be missing something.
+
+Thanks!
+
+Achilleas
+"""]]

Added a comment
diff --git a/doc/bugs/glacier/comment_2_5475986eed257c535c21833c23b7661c._comment b/doc/bugs/glacier/comment_2_5475986eed257c535c21833c23b7661c._comment
new file mode 100644
index 0000000..a622333
--- /dev/null
+++ b/doc/bugs/glacier/comment_2_5475986eed257c535c21833c23b7661c._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="namsgorf@6b5ce57fbe9dc2a2c65d6817151f107dc22f438c"
+ nickname="namsgorf"
+ avatar="http://cdn.libravatar.org/avatar/b85cacece5b347853dc35d97371b9e0c"
+ subject="comment 2"
+ date="2017-05-21T13:40:07Z"
+ content="""
+IIRC, at the time I wrote glacier-cli, boto (the Python AWS library it uses) had no Python 3 support, so I targeted Python 2 only. I see that boto supports Python 3 now. Patches to glacier-cli for Python 3 support welcome.
+
+Your Python 2 error looks like https://bugs.python.org/issue11898 to me. I know that others have hit the same error in the past. It's a problem between boto and httplib and Python, not a bug in glacier-cli. I don't think it affects Debian or Ubuntu, so you may need to take it up with Arch developers (perhaps Debian or Ubuntu carries a patch?)
+"""]]

add news item for git-annex 6.20170519
diff --git a/doc/news/version_6.20170228.mdwn b/doc/news/version_6.20170228.mdwn
deleted file mode 100644
index 7318d0a..0000000
--- a/doc/news/version_6.20170228.mdwn
+++ /dev/null
@@ -1,67 +0,0 @@
-News for git-annex 6.20170228:
-
-   This version of git-annex has mitigations for SHA1 hash collision
-   problems.
-   A new annex.securehashesonly configuration, when used in combination with
-   signed git commits, avoids potential hash collision problems in git-annex
-   repositories. For details, see this web page:
-   &lt;https://git-annex.branchable.com/tips/using\_signed\_git\_commits/&gt;
-
-git-annex 6.20170228 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Cryptographically secure hashes can be forced to be used in a
-     repository, by setting annex.securehashesonly.
-     This does not prevent the git repository from containing links
-     to insecure hashes, but it does prevent the content of such files
-     from being added to .git/annex/objects by any method.
-   * Tighten key parser to prevent SHA1 collision attacks generating
-     two keys that have the same SHA1. (Only done for keys that contain
-     a hash). This ensures that signed git commits of annexed files
-     will remain secure, as long as git-annex is using a secure hashing
-     backend.
-   * fsck: Warn about any files whose content is present, that don't
-     use secure hashes, when annex.securehashesonly is set.
-   * init: When annex.securehashesonly has been set with git-annex config,
-     copy that value to the annex.securehashesonly git config.
-   * Added --securehash option to match files using a secure hash function,
-     and corresponding securehash preferred content expression.
-   * sync, merge: Fail when the current branch has no commits yet, instead
-     of not merging in anything from remotes and appearing to succeed.
-   * Run ssh with -n whenever input is not being piped into it,
-     to avoid it consuming stdin that it shouldn't.
-     This fixes git-annex-checkpresentkey --batch remote,
-     which didn't output results for all keys passed into it. Other
-     git-annex commands that communicate with a remote over ssh may also
-     have been consuming stdin that they shouldn't have, which could have
-     impacted using them in eg, shell scripts.
-   * sync: Improve integration with receive.denyCurrentBranch=updateInstead,
-     displaying error messages from the remote then it fails to update
-     its checked out branch.
-   * Added post-recieve hook, which makes updateInstead work with direct
-     mode and adjusted branches.
-   * init: Set up the post-receive hook.
-   * sync: When syncing with a local repository located on a crippled
-     filesystem, run the post-receive hook there, since it wouldn't get run
-     otherwise. This makes pushing to repos on FAT-formatted removable
-     drives update them when receive.denyCurrentBranch=updateInstead.
-   * config group groupwanted numcopies schedule wanted required:
-     Avoid displaying extraneous messages about repository auto-init,
-     git-annex branch merging, etc, when being used to get information.
-   * adjust: Fix behavior when used in a repository that contains
-     submodules.
-   * Run wget with -nv instead of -q, so it will display HTTP errors.
-   * Run curl with -S, so HTTP errors are displayed, even when
-     it's otherwise silent.
-   * When downloading in --json or --quiet mode, use curl in preference
-     to wget, since curl is able to display only errors to stderr, unlike
-     wget.
-   * status: Pass --ignore-submodules=when option on to git status.
-   * config --set: As well as setting value in git-annex branch,
-     set local gitconfig. This is needed especially for
-     annex.securehashesonly, which is read only from local gitconfig and not
-     the git-annex branch.
-   * Removed support for building with the old cryptohash library.
-     Building with that library made git-annex not support SHA3; it's time
-     for that to always be supported in case SHA2 dominoes.
-   * git-annex.cabal: Make crypto-api a dependency even when built w/o
-     webapp and test suite."""]]
\ No newline at end of file
diff --git a/doc/news/version_6.20170519.mdwn b/doc/news/version_6.20170519.mdwn
new file mode 100644
index 0000000..9886299
--- /dev/null
+++ b/doc/news/version_6.20170519.mdwn
@@ -0,0 +1,12 @@
+git-annex 6.20170519 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Ssh password prompting improved when using -J for concurrency.
+     When ssh connection caching is enabled (and when GIT\_ANNEX\_USE\_GIT\_SSH
+     is not set), only one ssh password prompt will be made per host, and
+     only one ssh password prompt will be made at a time.
+   * When built with concurrent-output 1.9, ssh password prompts will no
+     longer interfere with the -J display.
+   * Removed dependency on MissingH, instead depending on the split library.
+   * Progress is displayed for transfers of files of unknown size.
+   * Work around bug in git 2.13.0 involving GIT\_COMMON\_DIR that broke
+     merging changes into adjusted branches."""]]
\ No newline at end of file

Added a comment: (compress/tar)ing the files
diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_2_6889f4695bca77c18efe0000fc491b57._comment b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_2_6889f4695bca77c18efe0000fc491b57._comment
new file mode 100644
index 0000000..7ca756b
--- /dev/null
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_2_6889f4695bca77c18efe0000fc491b57._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="dav02.git@4d920fe040aa5df86fcd9f4dd57d3fcf85ee5641"
+ nickname="dav02.git"
+ avatar="http://cdn.libravatar.org/avatar/c452f62e60af3c49990973d008ca1b76"
+ subject="(compress/tar)ing the files"
+ date="2017-05-19T12:14:14Z"
+ content="""
+I considered this, but unfortunately in this form I cannot work with files.
+
+There are hacks like archivemount, but they are extremely slow for large number of files and not available on many systems.
+"""]]

Added a comment
diff --git a/doc/forum/git-annex_help_import_outdated__63__/comment_1_ada30e2fead4b5b8f5e53388f941eb07._comment b/doc/forum/git-annex_help_import_outdated__63__/comment_1_ada30e2fead4b5b8f5e53388f941eb07._comment
new file mode 100644
index 0000000..b054e15
--- /dev/null
+++ b/doc/forum/git-annex_help_import_outdated__63__/comment_1_ada30e2fead4b5b8f5e53388f941eb07._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="comment 1"
+ date="2017-05-19T12:00:01Z"
+ content="""
+Okay, I did the same on a machine when git-annex was not installed before. Here everything works fine.
+
+So does git-annex help [...] open the locally installed manpages?
+
+I think it should show the help message that belongs to the running binary..
+
+"""]]

Added a comment
diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_1_89d2819893ae954041d65a5758993be8._comment b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_1_89d2819893ae954041d65a5758993be8._comment
new file mode 100644
index 0000000..5d13ae7
--- /dev/null
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item/comment_1_89d2819893ae954041d65a5758993be8._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="leavingchicago@c04c893e78d1c4c76cb3e32b5c227cf42bbf7682"
+ nickname="leavingchicago"
+ avatar="http://cdn.libravatar.org/avatar/4ae498d3d6ee558d6b65caa658f72572"
+ subject="comment 1"
+ date="2017-05-19T01:02:57Z"
+ content="""
+Have you thought about (compress/tar)ing the files and then only include the single object in the annex?
+
+Would give you the benefits of deduping any data too.
+"""]]

diff --git a/doc/forum/git-annex_help_import_outdated__63__.mdwn b/doc/forum/git-annex_help_import_outdated__63__.mdwn
new file mode 100644
index 0000000..6367fe1
--- /dev/null
+++ b/doc/forum/git-annex_help_import_outdated__63__.mdwn
@@ -0,0 +1,13 @@
+Hi,
+
+I just saw that my installed version of git-annex is too old for:
+
+    git-annex import --reinject-duplicates [...]
+
+Thus, I downloaded the precompiled binary: git-annex-standalone-amd64.tar.gz	2017-05-10 16:19 	54M
+
+First of all, it works fine with this version. However, the "--reinject-duplicates" option is not shown, when calling:
+
+    ./git-annex help import  ## in ~/git-annex.linux/
+
+So, is the help page just outdated, or does git-annex help gets confused because I still have an older version of git-annex installed in the system?

item in history -> single item
diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
index 7fb9631..4adbedb 100644
--- a/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
@@ -4,4 +4,4 @@ In my analyses I often have multiple (>10k) generated  small files in a single d
 I would like to store this in git annex, in order to version them and probably even synchronize. The problem is that if a huge number of files is stored inside the repository, the repository itself becomes huge and slow. There are some ways to improve the performance ([[1|tips/Repositories_with_large_number_of_files]], [[2|forum/Handling_a_large_number_of_files]], [[3|forum/__34__git_annex_sync__34___synced_after_8_hours]]), but it doesn't solve the issue completely.
 
 
-I was wondering if it is possible to force git annex to treat a single directory as one item in history? Probably with abandoning the checksum verification.
+I was wondering if it is possible to force git annex to treat a single directory with multiple files as a single item? Probably with abandoning the checksum verification.

diff --git a/doc/forum/git-annex_sync_here.mdwn b/doc/forum/git-annex_sync_here.mdwn
new file mode 100644
index 0000000..9aa91ed
--- /dev/null
+++ b/doc/forum/git-annex_sync_here.mdwn
@@ -0,0 +1,19 @@
+Hello,
+
+git-annex does a lot of convenient stuff, also locally. With git-annex sync [remote-name] the synchronization can be restricted to a single remote.
+
+However, I think being able to restrict sync to no remote at all, only doing the local stuff, would be helpful.
+
+The natural syntax for me would be git-annex sync here. But it says:
+
+    $ git-annex sync here
+    commit  ok
+    git-annex: there is no available git remote named "here"
+
+I think a workaround would be to do:
+
+    $ git-annex sync --no-push --no-pull
+
+Is that correct?
+
+Still, I think "git-annex sync here" would be a convenient shortcut.

update links
diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
index f91cfb3..7fb9631 100644
--- a/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
@@ -1,7 +1,7 @@
-In my analyses I often have multiple generated (>10k) small files in a single directory.
+In my analyses I often have multiple (>10k) generated  small files in a single directory.
 
 
-I would like to store this in git annex, in order to version them and probably even synchronize. The problem is that if a huge number of files is stored inside the repository, the repository itself becomes huge and slow. There are some ways to improve the performance ([[tips/Repositories_with_large_number_of_files]], [[forum/Handling_a_large_number_of_files]], [[forum/__34__git_annex_sync__34___synced_after_8_hours]]), but it doesn't solve the issue completely.
+I would like to store this in git annex, in order to version them and probably even synchronize. The problem is that if a huge number of files is stored inside the repository, the repository itself becomes huge and slow. There are some ways to improve the performance ([[1|tips/Repositories_with_large_number_of_files]], [[2|forum/Handling_a_large_number_of_files]], [[3|forum/__34__git_annex_sync__34___synced_after_8_hours]]), but it doesn't solve the issue completely.
 
 
 I was wondering if it is possible to force git annex to treat a single directory as one item in history? Probably with abandoning the checksum verification.

diff --git a/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
new file mode 100644
index 0000000..f91cfb3
--- /dev/null
+++ b/doc/forum/treat_directory_with_multiple_files_as_a_single_item.mdwn
@@ -0,0 +1,7 @@
+In my analyses I often have multiple generated (>10k) small files in a single directory.
+
+
+I would like to store this in git annex, in order to version them and probably even synchronize. The problem is that if a huge number of files is stored inside the repository, the repository itself becomes huge and slow. There are some ways to improve the performance ([[tips/Repositories_with_large_number_of_files]], [[forum/Handling_a_large_number_of_files]], [[forum/__34__git_annex_sync__34___synced_after_8_hours]]), but it doesn't solve the issue completely.
+
+
+I was wondering if it is possible to force git annex to treat a single directory as one item in history? Probably with abandoning the checksum verification.

Added a comment: p.s.
diff --git a/doc/forum/Git_annex_hangs/comment_10_ce5355b358f1acc63270331f3749fdcd._comment b/doc/forum/Git_annex_hangs/comment_10_ce5355b358f1acc63270331f3749fdcd._comment
new file mode 100644
index 0000000..e07ab04
--- /dev/null
+++ b/doc/forum/Git_annex_hangs/comment_10_ce5355b358f1acc63270331f3749fdcd._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="p.s."
+ date="2017-05-17T14:34:40Z"
+ content="""
+Actually the situation is even more complex.
+
+\"The server\" where my repo is stored are actually 4 servers for load balancing that share a single file system.
+
+I just found out that on one of the servers the were still some git-annex-shell processes running. Probably the move command with -J10 logged in on different servers; maybe this confused git-annex.
+
+Now, I logged-in on all four servers and killed any remaining processes. Still the problem persists.
+
+    git-annex info --debug
+    [2017-05-17 16:29:07.197350961] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
+    [2017-05-17 16:29:07.216435634] process done ExitSuccess
+    [2017-05-17 16:29:07.216539092] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+    [2017-05-17 16:29:07.2214318] process done ExitSuccess
+    [2017-05-17 16:29:07.2220868] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..76cf27fbf3697af8843ddb82f1fffde35aaba0b2\",\"--pretty=%H\",\"-n1\"]
+    [2017-05-17 16:29:07.23077964] process done ExitSuccess
+
+What's interesting is that these killing actions apparently have changed the hash number on which the hang happens (76cf..). What I also did was git-annex sync at another host, this actually seemed to work. Maybe this changed the hash.. I'm not entirely sure..
+
+"""]]

Added a comment: Hanging problem still there
diff --git a/doc/forum/Git_annex_hangs/comment_9_07ae4582a4403b4f0185dbc1830642ad._comment b/doc/forum/Git_annex_hangs/comment_9_07ae4582a4403b4f0185dbc1830642ad._comment
new file mode 100644
index 0000000..55de021
--- /dev/null
+++ b/doc/forum/Git_annex_hangs/comment_9_07ae4582a4403b4f0185dbc1830642ad._comment
@@ -0,0 +1,54 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="Hanging problem still there"
+ date="2017-05-17T14:05:41Z"
+ content="""
+Hi,
+
+it seems my git-annex repository is also affected by the \"hanging\" bug.
+
+Any operation of git-annex just hangs and doesn't do anything. Problem occurs with: sync, whereis, fsck (and with everything I tested).
+
+    git-annex whereis file --debug
+    [2017-05-17 15:57:34.895221948] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
+    [2017-05-17 15:57:34.901449143] process done ExitSuccess
+    [2017-05-17 15:57:34.901536719] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+    [2017-05-17 15:57:34.905894352] process done ExitSuccess
+    [2017-05-17 15:57:34.906485313] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..6ccd8cbac5fffa4763294ed88cb1ebc0920c3812\",\"--pretty=%H\",\"-n1\"]
+    [2017-05-17 15:57:34.917489743] process done ExitSuccess
+
+Then it just hangs..
+
+    git-annex sync --debug
+    [2017-05-17 15:58:26.366522677] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
+    [2017-05-17 15:58:26.371989858] process done ExitSuccess
+    [2017-05-17 15:58:26.372083975] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+    [2017-05-17 15:58:26.384517659] process done ExitSuccess
+    [2017-05-17 15:58:26.385061668] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..6ccd8cbac5fffa4763294ed88cb1ebc0920c3812\",\"--pretty=%H\",\"-n1\"]
+    [2017-05-17 15:58:26.390054368] process done ExitSuccess
+
+Also here, hangs..
+
+I \"installed\" git-annex from the pre-compiled binaries (i.e., copied it somewhere and put a simlink to git-annex and git-annex-shell into $HOME/bin).
+
+Versions are:
+
+    git --version
+    git version 1.8.3.1
+
+    git-annex version
+    git-annex version: 6.20170321-gf3dee9d65
+    build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+    local repository version: 5
+    supported repository versions: 3 5 6
+    upgrade supported from repository versions: 0 1 2 3 4 5
+    operating system: linux x86_64
+
+The setup was working quite fine for a while and now the problem just appeared. Actually, right before I noticed the problem I did git-annex move [...] -J10 from another machine to this repo but it didn't seem to work correctly, so I pressed CTRL-C..
+
+I cannot reboot the server, since I don't have root access.
+
+"""]]

update
diff --git a/doc/devblog/day_459__git_bug.mdwn b/doc/devblog/day_459__git_bug.mdwn
index 3079904..3ec5f16 100644
--- a/doc/devblog/day_459__git_bug.mdwn
+++ b/doc/devblog/day_459__git_bug.mdwn
@@ -10,7 +10,8 @@ workaround out. It's not a super bad bug, but it does make the test suite
 fail and I've already had 3 people report the problem.
 
 Seems it would be good to have an integration test that runs git-annex's
-test suite against new commits to git.
+test suite against new commits to git. Ævar Arnfjörð Bjarmason has stepped
+up to add that to git's test suite.
 
 ----
 

Added a comment: Encrypt to different subkeys?
diff --git a/doc/encryption/comment_9_5ca10891d642392aaff342c1478b0550._comment b/doc/encryption/comment_9_5ca10891d642392aaff342c1478b0550._comment
new file mode 100644
index 0000000..723e3df
--- /dev/null
+++ b/doc/encryption/comment_9_5ca10891d642392aaff342c1478b0550._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Yurt"
+ avatar="http://cdn.libravatar.org/avatar/28d2177f7678c6cc0b677c34927eba06"
+ subject="Encrypt to different subkeys?"
+ date="2017-05-16T20:25:01Z"
+ content="""
+Is it possible to encrypt with subkeys. I have a few subkeys distributed to different computers and I'd like to be able to sync to a special remote with all of them. Right now, if the master key is stripped, I get an error from gpg.
+
+I do this exact thing with password-store. Appending \"!\" to the subkey id should force gpg to use that specific key: [https://lists.zx2c4.com/pipermail/password-store/2014-September/001131.html](https://lists.zx2c4.com/pipermail/password-store/2014-September/001131.html).
+"""]]

devblog
diff --git a/doc/devblog/day_459__git_bug.mdwn b/doc/devblog/day_459__git_bug.mdwn
new file mode 100644
index 0000000..3079904
--- /dev/null
+++ b/doc/devblog/day_459__git_bug.mdwn
@@ -0,0 +1,28 @@
+Seems that the recent release of git 2.13.0 contained a reversion that
+broke `git-annex sync` in an adjusted branch. After bisecting git,
+producing a minimal test case, and reporting that to the git developers, I
+was able to work around it in git-annex. The workaround is normally not
+expensive, but could be when a repository has thousands of unpacked refs.
+So I hope this will get fixed in git and I can remove the workaround.
+
+I think I will hurry up the next git-annex release somewhat to get the
+workaround out. It's not a super bad bug, but it does make the test suite
+fail and I've already had 3 people report the problem.
+
+Seems it would be good to have an integration test that runs git-annex's
+test suite against new commits to git.
+
+----
+
+Also, I dealt with some fallout from removing MissingH; a exponential
+speed blowup in a directory traversal function.
+
+Getting back to the ssh password prompting with -J I was working on last
+week, dealt with the ssh prompt interfering with the regional display
+manager. The fix is not perfect, but good enough; before ssh prompts (and
+only if it prompts), git-annex temporarily clears the regional display.
+Then the display gets redrawn under the ssh output. That needed some
+changes to concurrent-output (which I did over the weekend), so will only
+be done when it's built with a new enough version. A better approach would
+be to save and restore the cursor position, but the ansi-terminal library
+does not yet support that.

clear regions before ssh prompt
When built with concurrent-output 1.9, ssh password prompts will no longer
interfere with the -J display.
To avoid flicker, only done when ssh actually does need to prompt;
ssh is first run in batch mode and if that succeeds the connection is up
and no need to clear regions.
This commit was supported by the NSF-funded DataLad project.
diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs
index a9ff917..50a5163 100644
--- a/Annex/Ssh.hs
+++ b/Annex/Ssh.hs
@@ -191,17 +191,27 @@ prepSocket socketfile gc sshparams = do
 	liftIO $ createDirectoryIfMissing True $ parentDir socketfile
 	let socketlock = socket2lock socketfile
 
-	prompt $ \c -> case c of
-		Concurrent {} -> do
-			-- If the LockCache already has the socketlock in it,
-			-- the connection has already been started. Otherwise,
-			-- get the connection started now.
-			whenM (isNothing <$> fromLockCache socketlock) $
-				void $ liftIO $ boolSystem "ssh" $
-					sshparams ++ startSshConnection gc
+	c <- Annex.getState Annex.concurrency
+	case c of
+		Concurrent {} -> makeconnection socketlock
 		NonConcurrent -> return ()
 	
 	lockFileCached socketlock
+  where
+	-- When the LockCache already has the socketlock in it,
+	-- the connection has already been started. Otherwise,
+	-- get the connection started now.
+	makeconnection socketlock =
+		whenM (isNothing <$> fromLockCache socketlock) $ do
+			let startps = sshparams ++ startSshConnection gc
+			-- When we can start the connection in batch mode,
+			-- ssh won't prompt to the console.
+			(_, connected) <- liftIO $ processTranscript "ssh"
+				(["-o", "BatchMode=true"] ++ toCommand startps)
+				Nothing
+			unless connected $ 
+				prompt $ void $ liftIO $
+					boolSystem "ssh" startps
 
 -- Parameters to get ssh connected to the remote host,
 -- by asking it to run a no-op command.
diff --git a/CHANGELOG b/CHANGELOG
index 3096b80..5e816ca 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,8 @@ git-annex (6.20170511) UNRELEASED; urgency=medium
     When ssh connection caching is enabled (and when GIT_ANNEX_USE_GIT_SSH
     is not set), only one ssh password prompt will be made per host, and
     only one ssh password prompt will be made at a time.
+  * When built with concurrent-output 1.9, ssh password prompts will no
+    longer interfere with the -J display.
   * Removed dependency on MissingH, instead depending on the split library.
   * Progress is displayed for transfers of files of unknown size.
   * Work around bug in git 2.13.0 involving GIT_COMMON_DIR that broke
diff --git a/Messages.hs b/Messages.hs
index 83ea91d..ff13b31 100644
--- a/Messages.hs
+++ b/Messages.hs
@@ -56,6 +56,7 @@ import Types.Messages
 import Types.ActionItem
 import Types.Concurrency
 import Messages.Internal
+import Messages.Concurrent
 import qualified Messages.JSON as JSON
 import qualified Annex
 
@@ -227,10 +228,13 @@ implicitMessage = whenM (implicitMessages <$> Annex.getState Annex.output)
  - that the action is the only thing using the console, and can eg prompt
  - the user.
  -}
-prompt :: (Concurrency -> Annex a) -> Annex a
+prompt :: Annex a -> Annex a
 prompt a = go =<< Annex.getState Annex.concurrency
   where
-	go NonConcurrent = a NonConcurrent
-	go c@(Concurrent {}) = withMessageState $ \s -> do
+	go NonConcurrent = a
+	go (Concurrent {}) = withMessageState $ \s -> do
 		let l = promptLock s
-		bracketIO (takeMVar l) (putMVar l) (const (a c))
+		bracketIO
+			(takeMVar l)
+			(putMVar l)
+			(const $ hideRegionsWhile a)
diff --git a/Messages/Concurrent.hs b/Messages/Concurrent.hs
index 41153d0..78eed3b 100644
--- a/Messages/Concurrent.hs
+++ b/Messages/Concurrent.hs
@@ -1,6 +1,6 @@
 {- git-annex output messages, including concurrent output to display regions
  -
- - Copyright 2010-2016 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2017 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU GPL version 3 or higher.
  -}
@@ -10,8 +10,9 @@
 
 module Messages.Concurrent where
 
-import Annex
+import Types
 import Types.Messages
+import qualified Annex
 
 #ifdef WITH_CONCURRENTOUTPUT
 import Common
@@ -136,3 +137,22 @@ concurrentOutputSupported = return True -- Windows is always unicode
 #else
 concurrentOutputSupported = return False
 #endif
+
+{- Hide any currently displayed console regions while running the action,
+ - so that the action can use the console itself.
+ - This needs a new enough version of concurrent-output; otherwise
+ - the regions will not be hidden, but the action still runs, garbling the
+ - display. -}
+hideRegionsWhile :: Annex a -> Annex a
+#if MIN_VERSION_concurrent_output(1,9,0)
+hideRegionsWhile a = bracketIO setup cleanup go
+  where
+	setup = Regions.waitDisplayChange $ swapTMVar Regions.regionList []
+	cleanup = void . atomically . swapTMVar Regions.regionList
+	go _ = do
+		liftIO $ hFlush stdout
+		a
+#else
+#warning Building with concurrent-output older than 1.9.0 so expect some display glitches when password prompts occur in concurrent mode
+hideRegionsWhile = id
+#endif
diff --git a/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
index 1df0694..a8e0871 100644
--- a/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
+++ b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
@@ -47,3 +47,5 @@ Some approaches to fix it:
    See <https://github.com/feuerbach/ansi-terminal/issues/7>
 
 --[[Joey]]
+
+> [[fixed|done]] using option #3. --[[Joey]]

workaround is in place
(and remove debug print)
diff --git a/Annex/AdjustedBranch.hs b/Annex/AdjustedBranch.hs
index 43677d7..c022554 100644
--- a/Annex/AdjustedBranch.hs
+++ b/Annex/AdjustedBranch.hs
@@ -369,7 +369,6 @@ updateAdjustedBranch tomerge (origbranch, adj) mergeconfig commitmode = catchBoo
 				liftIO $ forM_ refs' $ \src ->
 					whenM (doesFileExist src) $ do
 						dest <- relPathDirToFile git_dir src
-						print (src, dest)
 						let dest' = tmpgit </> dest
 						createDirectoryIfMissing True (takeDirectory dest')
 						void $ createLinkOrCopy src dest'
diff --git a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
index 2d3eebe..53e12b5 100644
--- a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
+++ b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
@@ -355,3 +355,5 @@ FAIL (1.50s)
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 git-annex is an essential building block in my digital life style! It keeps backups of all my precious family photos. I'm a big git-annex shill when I get the chance, especially to nix, guix and decentralized hacker types.
+
+> Workaround is in [[!commit 9bcaef1ec496b4ffd3033ae5080949bd8cc3edd5]]. [[done]] --[[Joey]]

glacier-cli python3 bug nearly for sure
diff --git a/doc/bugs/glacier.mdwn b/doc/bugs/glacier.mdwn
index b6887a8..cca9d39 100644
--- a/doc/bugs/glacier.mdwn
+++ b/doc/bugs/glacier.mdwn
@@ -1,3 +1,5 @@
+[[!meta title="glacier-cli fails with backtrace"]]
+
 ### Please describe the problem.
 
 Amazon Glacier remote doesn't work as expected. It seems like glacier-cli no longer works with git-annex.
@@ -97,3 +99,5 @@ git-annex: copy: 1 failed
 Yes it's a great tool :). Thanks!
 AWS Glacier integration would be a perfect addition and I could employ git-annex at my company.
 
+> Closing as it's not a bug in git-annex, but glacier-cli. [[done]]
+> --[[Joey]]
diff --git a/doc/bugs/glacier/comment_1_257c401ee74fd48f0b2930f18b82c8c9._comment b/doc/bugs/glacier/comment_1_257c401ee74fd48f0b2930f18b82c8c9._comment
new file mode 100644
index 0000000..ad6c3e9
--- /dev/null
+++ b/doc/bugs/glacier/comment_1_257c401ee74fd48f0b2930f18b82c8c9._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-16T18:19:13Z"
+ content="""
+That looks rather more like a bug in glacier-cli than a bug in git-annex to
+me.
+
+"expected a bytes-like object, str found" has a ton of google hits, 
+and looks to be breakage related to python 3's changes to string handling.
+
+Please file a bug on glacier-cli, and consider making it use python 2 until
+it can be fixed. I'll bet python 2 will avoid this problem.
+"""]]

forwardeed to git ML
diff --git a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment
index fc0304b..9e7d192 100644
--- a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment
+++ b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment
@@ -30,5 +30,9 @@ In particular, this just seems outright wrong:
 So, it's just pulled synced/master to r1/synced/master, but then it claims
 refs/remotes/r1/synced/master is not mergable.
 
-Need to boil this down to a useful bug report against git..
+This only affects adjusted branches because git has a reversion in the
+`GIT_COMMON_DIR` that they use.
+
+Posted on the git ML about this reversion, message-id
+<20170516171028.5eagqr2sw5a2i77d@kitenet.net>
 """]]

close dup
diff --git a/doc/bugs/8_unit_tests_fail.mdwn b/doc/bugs/8_unit_tests_fail.mdwn
index 7d8769a..9bc53c9 100644
--- a/doc/bugs/8_unit_tests_fail.mdwn
+++ b/doc/bugs/8_unit_tests_fail.mdwn
@@ -26,3 +26,4 @@ nix-build and inside a nix shell with manual building.
 
 Yep, works apart from the few tests that fail.
 
+> Duplicate of [[git-annex_in_nixpkgs_fails_with_git-2.13.0]]; closing. [[done]] --[[Joey]]

bisection
diff --git a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment
new file mode 100644
index 0000000..fc0304b
--- /dev/null
+++ b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/comment_1_364f69b47889486ab006811406324c44._comment
@@ -0,0 +1,34 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-16T15:42:53Z"
+ content="""
+Not nix-specific, I can reproduce with self-built git.
+
+Bisecting git itself (rather than nix):
+
+	f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 is the first bad commit
+	commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0
+	Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
+	Date:   Sun Mar 26 09:42:24 2017 +0700
+	
+	    files-backend: remove the use of git_path()
+
+That commit seems has to do with paths used for refs, and the test suite is
+failing due to something involving pushing/merging the
+synced/master branch. So, it's looking like a git bug.
+
+In particular, this just seems outright wrong:
+
+	From ../../.t/tmprepo44
+	 * [new branch]      git-annex     -> r1/git-annex
+	 * [new branch]      master        -> r1/master
+	 * [new branch]      synced/master -> r1/synced/master
+	merge: refs/remotes/r1/master - not something we can merge
+	merge: refs/remotes/r1/synced/master - not something we can merge
+
+So, it's just pulled synced/master to r1/synced/master, but then it claims
+refs/remotes/r1/synced/master is not mergable.
+
+Need to boil this down to a useful bug report against git..
+"""]]

failing unit tests bug
diff --git a/doc/bugs/8_unit_tests_fail.mdwn b/doc/bugs/8_unit_tests_fail.mdwn
new file mode 100644
index 0000000..7d8769a
--- /dev/null
+++ b/doc/bugs/8_unit_tests_fail.mdwn
@@ -0,0 +1,28 @@
+### Please describe the problem.
+
+8 out of 293 tests failed (200.39s)
+  (This could be due to a bug in git-annex, or an incompatibility
+   with utilities, such as git, installed on this system.)
+
+full log https://headcounter.org/hydra/build/1819294/nixlog/1
+
+### What steps will reproduce the problem?
+
+./make test
+
+
+### What version of git-annex are you using? On what operating system?
+
+6.20170510
+
+nix-build and inside a nix shell with manual building.
+
+
+### Please provide any additional information below.
+
+
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Yep, works apart from the few tests that fail.
+

devblog
diff --git a/doc/devblog/day_458__adeiu_MissingH.mdwn b/doc/devblog/day_458__adeiu_MissingH.mdwn
new file mode 100644
index 0000000..5d6a787
--- /dev/null
+++ b/doc/devblog/day_458__adeiu_MissingH.mdwn
@@ -0,0 +1,18 @@
+Wasn't planning to, but spent the day making git-annex not depend on the
+MissingH library. This has been a long-term goal, as MissingH pulls in
+several other libraries and is not modern or principled.
+
+The first part was to using cryptonite for MD5 calculation. While
+converting to the form git-annex uses to make hash directories involved
+some math, this did make git-annex garbage-collect less, and
+probably made it faster.
+
+Then I had to write my own progress meter display, since git-annex was
+using MissingH's display. That was fairly simple (73 LoC), and let me
+make it more efficient and tuned for the git-annex use case. As a bonus, it
+got progress displays when transferring files of unknown sizes, which
+wasn't done before.
+
+MissingH was handy training wheels when I was coming over from perl,
+but it's been training wheels on some old cars in the middle of a 
+500 car train for a while, so glad that's over.

diff --git a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
index 42721f8..2d3eebe 100644
--- a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
+++ b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
@@ -354,4 +354,4 @@ FAIL (1.50s)
 
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
-git-annex is an essential building block in my digital life style! It keeps backups of all my precious family photos.
+git-annex is an essential building block in my digital life style! It keeps backups of all my precious family photos. I'm a big git-annex shill when I get the chance, especially to nix, guix and decentralized hacker types.

stop using MissingH for MD5
Cryptonite is faster and allocates less, and I want to get rid of
MissingH use.
Note that the new dependency on memory is free; it's a dependency of
cryptonite.
This commit was supported by the NSF-funded DataLad project.
diff --git a/Annex/DirHashes.hs b/Annex/DirHashes.hs
index 82d751e..f843848 100644
--- a/Annex/DirHashes.hs
+++ b/Annex/DirHashes.hs
@@ -1,6 +1,6 @@
 {- git-annex file locations
  -
- - Copyright 2010-2015 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2017 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU GPL version 3 or higher.
  -}
@@ -19,14 +19,15 @@ module Annex.DirHashes (
 
 import Data.Bits
 import Data.Word
-import Data.Hash.MD5
 import Data.Default
+import qualified Data.ByteArray
 
 import Common
 import Key
 import Types.GitConfig
 import Types.Difference
 import Utility.FileSystemEncoding
+import Utility.Hash
 
 type Hasher = Key -> FilePath
 
@@ -62,15 +63,24 @@ hashDirs :: HashLevels -> Int -> String -> FilePath
 hashDirs (HashLevels 1) sz s = addTrailingPathSeparator $ take sz s
 hashDirs _ sz s = addTrailingPathSeparator $ take sz s </> drop sz s
 
+hashDirLower :: HashLevels -> Hasher
+hashDirLower n k = hashDirs n 3 $ take 6 $ show $ md5 $
+	encodeBS $ key2file $ nonChunkKey k
+
+{- This was originally using Data.Hash.MD5 from MissingH. This new version
+- is faster, but ugly as it has to replicate the 4 Word32's that produced. -}
 hashDirMixed :: HashLevels -> Hasher
-hashDirMixed n k = hashDirs n 2 $ take 4 $ display_32bits_as_dir =<< [a,b,c,d]
+hashDirMixed n k = hashDirs n 2 $ take 4 $ concatMap display_32bits_as_dir $
+	encodeWord32 $ map fromIntegral $ Data.ByteArray.unpack $
+		Utility.Hash.md5 $ encodeBS $ key2file $ nonChunkKey k
   where
-	ABCD (a,b,c,d) = md5 $ md5FilePath $ key2file $ nonChunkKey k
-
-hashDirLower :: HashLevels -> Hasher
-hashDirLower n k = hashDirs n 3 $ take 6 $ md5s $ md5FilePath $ key2file $ nonChunkKey k
+	encodeWord32 (b1:b2:b3:b4:rest) =
+		(shiftL b4 24 .|. shiftL b3 16 .|. shiftL b2 8 .|. b1)
+		: encodeWord32 rest
+	encodeWord32 _ = []
 
 {- modified version of display_32bits_as_hex from Data.Hash.MD5
+ - in MissingH
  -   Copyright (C) 2001 Ian Lynagh 
  -   License: Either BSD or GPL
  -}
diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs
index c538029..a9ff917 100644
--- a/Annex/Ssh.hs
+++ b/Annex/Ssh.hs
@@ -33,6 +33,7 @@ import Config
 import Annex.Path
 import Utility.Env
 import Utility.FileSystemEncoding
+import Utility.Hash
 import Types.CleanupActions
 import Types.Concurrency
 import Git.Env
@@ -42,7 +43,6 @@ import Annex.Perms
 import Annex.LockPool
 #endif
 
-import Data.Hash.MD5
 import Control.Concurrent.STM
 
 {- Some ssh commands are fed stdin on a pipe and so should be allowed to
@@ -287,7 +287,7 @@ hostport2socket host Nothing = hostport2socket' host
 hostport2socket host (Just port) = hostport2socket' $ host ++ "!" ++ show port
 hostport2socket' :: String -> FilePath
 hostport2socket' s
-	| length s > lengthofmd5s = md5s (Str s)
+	| length s > lengthofmd5s = show $ md5 $ encodeBS s
 	| otherwise = s
   where
 	lengthofmd5s = 32
diff --git a/Annex/VariantFile.hs b/Annex/VariantFile.hs
index 17658a9..8365073 100644
--- a/Annex/VariantFile.hs
+++ b/Annex/VariantFile.hs
@@ -9,8 +9,7 @@ module Annex.VariantFile where
 
 import Annex.Common
 import Utility.FileSystemEncoding
-
-import Data.Hash.MD5
+import Utility.Hash
 
 variantMarker :: String
 variantMarker = ".variant-"
@@ -42,4 +41,4 @@ variantFile file key
 	doubleconflict = variantMarker `isInfixOf` file
 
 shortHash :: String -> String
-shortHash = take 4 . md5s . md5FilePath
+shortHash = take 4 . show . md5 . encodeBS
diff --git a/Backend/Utilities.hs b/Backend/Utilities.hs
index d1fb94f..1691fa2 100644
--- a/Backend/Utilities.hs
+++ b/Backend/Utilities.hs
@@ -7,10 +7,9 @@
 
 module Backend.Utilities where
 
-import Data.Hash.MD5
-
 import Annex.Common
 import Utility.FileSystemEncoding
+import Utility.Hash
 
 {- Generates a keyName from an input string. Takes care of sanitizing it.
  - If it's not too long, the full string is used as the keyName.
@@ -20,7 +19,8 @@ genKeyName :: String -> String
 genKeyName s
 	-- Avoid making keys longer than the length of a SHA256 checksum.
 	| bytelen > sha256len =
-		truncateFilePath (sha256len - md5len - 1) s' ++ "-" ++ md5s (Str s)
+		truncateFilePath (sha256len - md5len - 1) s' ++ "-" ++ 
+			show (md5 (encodeBS s))
 	| otherwise = s'
   where
 	s' = preSanitizeKeyName s
diff --git a/Utility/FileSystemEncoding.hs b/Utility/FileSystemEncoding.hs
index ae3bd35..785b078 100644
--- a/Utility/FileSystemEncoding.hs
+++ b/Utility/FileSystemEncoding.hs
@@ -12,7 +12,6 @@ module Utility.FileSystemEncoding (
 	useFileSystemEncoding,
 	fileEncoding,
 	withFilePath,
-	md5FilePath,
 	decodeBS,
 	encodeBS,
 	decodeW8,
@@ -27,7 +26,6 @@ import qualified GHC.IO.Encoding as Encoding
 import Foreign.C
 import System.IO
 import System.IO.Unsafe
-import qualified Data.Hash.MD5 as MD5
 import Data.Word
 import Data.Bits.Utils
 import Data.List
@@ -101,10 +99,6 @@ _encodeFilePath fp = unsafePerformIO $ do
 	GHC.withCString enc fp (GHC.peekCString Encoding.char8)
 		`catchNonAsync` (\_ -> return fp)
 
-{- Encodes a FilePath into a Md5.Str, applying the filesystem encoding. -}
-md5FilePath :: FilePath -> MD5.Str
-md5FilePath = MD5.Str . _encodeFilePath
-
 {- Decodes a ByteString into a FilePath, applying the filesystem encoding. -}
 decodeBS :: L.ByteString -> FilePath
 #ifndef mingw32_HOST_OS
diff --git a/Utility/Hash.hs b/Utility/Hash.hs
index b6bf996..70f826b 100644
--- a/Utility/Hash.hs
+++ b/Utility/Hash.hs
@@ -1,8 +1,4 @@
-{- Convenience wrapper around cryptohash/cryptonite.
- -
- - SHA3 hashes are currently only enabled when using cryptonite,
- - because of https://github.com/vincenthz/hs-cryptohash/issues/36
- -}
+{- Convenience wrapper around cryptonite's hashing. -}
 
 module Utility.Hash (
 	sha1,
diff --git a/Utility/LockFile/PidLock.hs b/Utility/LockFile/PidLock.hs
index 87c11c0..23560fa 100644
--- a/Utility/LockFile/PidLock.hs
+++ b/Utility/LockFile/PidLock.hs
@@ -25,6 +25,8 @@ import Utility.Path
 import Utility.FileMode
 import Utility.LockFile.LockStatus
 import Utility.ThreadScheduler
+import Utility.Hash
+import Utility.FileSystemEncoding
 import qualified Utility.LockFile.Posix as Posix
 
 import System.IO
@@ -33,7 +35,6 @@ import Data.Maybe
 import Data.List
 import Network.BSD
 import System.FilePath
-import Data.Hash.MD5
 import Control.Applicative
 import Prelude
 

(Diff truncated)
glacier-cli and git-annex doesn't seem to work together right
diff --git a/doc/bugs/glacier.mdwn b/doc/bugs/glacier.mdwn
new file mode 100644
index 0000000..b6887a8
--- /dev/null
+++ b/doc/bugs/glacier.mdwn
@@ -0,0 +1,99 @@
+### Please describe the problem.
+
+Amazon Glacier remote doesn't work as expected. It seems like glacier-cli no longer works with git-annex.
+
+### What steps will reproduce the problem?
+
+1. use git-annex from archlinux repo
+2. download and install this with pip: https://github.com/basak/glacier-cli
+3. follow these instructions https://git-annex.branchable.com/tips/using_Amazon_Glacier/
+4. the problems occurs when trying to move or copy content to glacier
+
+### What version of git-annex are you using? On what operating system?
+
+version: 6.20160418-geff8673
+on: Linux  4.10.13-1-ARCH #1 SMP PREEMPT Thu Apr 27 12:15:09 CEST 2017 x86_64 GNU/Linux
+
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+# with python 3.6
+% git annex copy file.mp3 --to glacier
+copy file.mp3 (checking glacier...) (to glacier...) 
+98%           0.0 B/s 0sTraceback (most recent call last):
+  File "/home/me/.local/bin/glacier", line 736, in <module>
+    main()
+  File "/home/me/.local/bin/glacier", line 732, in main
+    App().main()
+  File "/home/me/.local/bin/glacier", line 718, in main
+    self.args.func()
+  File "/home/me/.local/bin/glacier", line 500, in archive_upload
+    file_obj=self.args.file, description=name)
+  File "/usr/lib/python3.6/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
+    writer.close()
+  File "/usr/lib/python3.6/site-packages/boto/glacier/writer.py", line 231, in close
+    self.partitioner.flush()
+  File "/usr/lib/python3.6/site-packages/boto/glacier/writer.py", line 79, in flush
+    self._send_part()
+  File "/usr/lib/python3.6/site-packages/boto/glacier/writer.py", line 64, in _send_part
+    data = b''.join(self._buffer)
+TypeError: sequence item 0: expected a bytes-like object, str found
+failed                  
+git-annex: copy: 1 failed
+
+# --------------------------
+# with python2.7
+% git annex copy file.m3 --to glacier
+copy file.mp3 (checking glacier...) (to glacier...) 
+98%           0.0 B/s 0sTraceback (most recent call last):
+  File "/home/me/.local/bin/glacier", line 736, in <module>
+    main()
+  File "/home/me/.local/bin/glacier", line 732, in main
+    App().main()
+  File "/home/me/.local/bin/glacier", line 718, in main
+    self.args.func()
+  File "/home/me/.local/bin/glacier", line 500, in archive_upload
+    file_obj=self.args.file, description=name)
+  File "/usr/lib/python2.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
+    writer.close()
+  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 228, in close
+    self.partitioner.flush()
+  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 79, in flush
+    self._send_part()
+  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part
+    self.send_fn(part)
+  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part
+    self.uploader.upload_part(self.next_part_index, part_data)
+  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part
+    content_range, part_data)
+  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 1279, in upload_part
+    response_headers=response_headers)
+  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 114, in make_request
+    data=data)
+  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
+    retry_handler=retry_handler)
+  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 943, in _mexe
+    request.body, request.headers)
+  File "/usr/lib/python2.7/httplib.py", line 1042, in request
+    self._send_request(method, url, body, headers)
+  File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request
+    self.endheaders(body)
+  File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
+    self._send_output(message_body)
+  File "/usr/lib/python2.7/httplib.py", line 880, in _send_output
+    msg += message_body
+UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 0: ordinal not in range(128)
+failed                  
+git-annex: copy: 1 failed
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+Yes it's a great tool :). Thanks!
+AWS Glacier integration would be a perfect addition and I could employ git-annex at my company.
+

diff --git a/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
new file mode 100644
index 0000000..42721f8
--- /dev/null
+++ b/doc/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0.mdwn
@@ -0,0 +1,357 @@
+### Please describe the problem.
+
+Tests fail on Nix when git is upgraded to 2.13.
+
+
+### What steps will reproduce the problem?
+
+On NixOS or a system with Nix installed, check out nixpkgs ce8662e693b5756e8457d94bb1765d853310afdb, try to build git-annex (`nix-build -I . -A pkgs.gitAndTools.gitAnnex`).
+
+
+### What version of git-annex are you using? On what operating system?
+
+git-annex 6.20170321 on Nix ce8662e6.
+
+
+### Please provide any additional information below.
+
+I ran a git bisect that concluded:
+
+[[!format sh """
+8 out of 285 tests failed (281.73s)
+  (This could be due to a bug in git-annex, or an incompatibility
+   with utilities, such as git, installed on this system.)
+builder for ‘/nix/store/908y9923fnjmi87apji6q14smgc2rf3d-git-annex-6.20170321.drv’ failed with exit code 1
+error: build of ‘/nix/store/908y9923fnjmi87apji6q14smgc2rf3d-git-annex-6.20170321.drv’ failed
+ce8662e693b5756e8457d94bb1765d853310afdb is the first bad commit
+commit ce8662e693b5756e8457d94bb1765d853310afdb
+Author: Tim Steinbach <tim@nequissimus.com>
+Date:   Tue May 9 21:57:24 2017 -0400
+
+    git: 2.12.2 -> 2.13.0
+
+:040000 040000 4155c091e7156a4e15cf073640e0be3c76b1b7f3 48ca917e5578b405cf8517e1f113433b2d076e15 M      pkgs
+bisect run success
+"""]]
+
+
+[[!format sh """
+    adjusted branch merge regression:                     Switched to branch 'adjusted/master(unlocked)'
+[adjusted/master(unlocked) ca9719e] git-annex in .t/tmprepo39
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+From ../../.t/tmprepo40
+ * [new branch]      git-annex  -> r2/git-annex
+ * [new branch]      master     -> r2/master
+To ../../.t/tmprepo40
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+Switched to branch 'adjusted/master(unlocked)'
+[adjusted/master(unlocked) b29e8de] git-annex in .t/tmprepo40
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+merge: refs/heads/synced/master - not something we can merge
+From ../../.t/tmprepo39
+ * [new branch]      adjusted/master(unlocked) -> r1/adjusted/master(unlocked)
+ * [new branch]      git-annex                 -> r1/git-annex
+ * [new branch]      master                    -> r1/master
+ * [new branch]      synced/master             -> r1/synced/master
+merge: refs/remotes/r1/master - not something we can merge
+merge: refs/remotes/r1/synced/master - not something we can merge
+To ../../.t/tmprepo39
+ * [new branch]      git-annex -> synced/git-annex
+ ! [rejected]        master -> synced/master (non-fast-forward)
+error: failed to push some refs to '../../.t/tmprepo39'
+hint: Updates were rejected because a pushed branch tip is behind its remote
+hint: counterpart. Check out this branch and integrate the remote changes
+hint: (e.g. 'git pull ...') before pushing again.
+hint: See the 'Note about fast-forwards' in 'git push --help' for details.
+To ../../.t/tmprepo39
+ ! [rejected]        master -> master (non-fast-forward)
+error: failed to push some refs to '../../.t/tmprepo39'
+hint: Updates were rejected because a pushed branch tip is behind its remote
+hint: counterpart. Check out this branch and integrate the remote changes
+hint: (e.g. 'git pull ...') before pushing again.
+hint: See the 'Note about fast-forwards' in 'git push --help' for details.
+  Pushing to r1 failed.
+FAIL (2.32s)
+"""]]
+
+
+[[!format sh """
+    conflict resolution (adjusted branch):                [master b0dd758] git-annex in .t/tmprepo44
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+[master 8295bae] git-annex in .t/tmprepo45
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+Switched to branch 'adjusted/master(unlocked)'
+On branch master
+nothing to commit, working tree clean
+From ../../.t/tmprepo45
+ * [new branch]      adjusted/master(unlocked) -> r2/adjusted/master(unlocked)
+ * [new branch]      git-annex                 -> r2/git-annex
+ * [new branch]      master                    -> r2/master
+ * [new branch]      synced/master             -> r2/synced/master
+Auto-merging conflictor
+CONFLICT (add/add): Merge conflict in conflictor
+Automatic merge failed; fix conflicts and then commit the result.
+conflictor: needs merge
+[master 14ffa24] git-annex automatic merge conflict fix
+Already up-to-date.
+To ../../.t/tmprepo45
+   8295bae..14ffa24  master -> synced/master
+ * [new branch]      git-annex -> synced/git-annex
+On branch adjusted/master(unlocked)
+nothing to commit, working tree clean
+merge: refs/heads/synced/master - not something we can merge
+From ../../.t/tmprepo44
+ * [new branch]      git-annex     -> r1/git-annex
+ * [new branch]      master        -> r1/master
+ * [new branch]      synced/master -> r1/synced/master
+merge: refs/remotes/r1/master - not something we can merge
+merge: refs/remotes/r1/synced/master - not something we can merge
+FAIL (2.34s)
+"""]]
+
+[[!format sh """
+    adjusted branch merge regression:                     Switched to branch 'adjusted/master(unlocked)'
+[adjusted/master(unlocked) 7ccad36] git-annex in .t/tmprepo39
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+From ../../.t/tmprepo40
+ * [new branch]      git-annex  -> r2/git-annex
+ * [new branch]      master     -> r2/master
+To ../../.t/tmprepo40
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+Switched to branch 'adjusted/master(unlocked)'
+[adjusted/master(unlocked) cfb04fd] git-annex in .t/tmprepo40
+ 1 file changed, 1 insertion(+)
+ create mode 100644 conflictor
+merge: refs/heads/synced/master - not something we can merge
+From ../../.t/tmprepo39
+ * [new branch]      adjusted/master(unlocked) -> r1/adjusted/master(unlocked)
+ * [new branch]      git-annex                 -> r1/git-annex
+ * [new branch]      master                    -> r1/master
+ * [new branch]      synced/master             -> r1/synced/master
+merge: refs/remotes/r1/master - not something we can merge
+merge: refs/remotes/r1/synced/master - not something we can merge
+To ../../.t/tmprepo39
+ * [new branch]      git-annex -> synced/git-annex
+ ! [rejected]        master -> synced/master (non-fast-forward)
+error: failed to push some refs to '../../.t/tmprepo39'
+hint: Updates were rejected because a pushed branch tip is behind its remote
+hint: counterpart. Check out this branch and integrate the remote changes
+hint: (e.g. 'git pull ...') before pushing again.
+hint: See the 'Note about fast-forwards' in 'git push --help' for details.
+To ../../.t/tmprepo39
+ ! [rejected]        master -> master (non-fast-forward)
+error: failed to push some refs to '../../.t/tmprepo39'
+hint: Updates were rejected because a pushed branch tip is behind its remote
+hint: counterpart. Check out this branch and integrate the remote changes
+hint: (e.g. 'git pull ...') before pushing again.
+hint: See the 'Note about fast-forwards' in 'git push --help' for details.
+  Pushing to r1 failed.
+FAIL (1.79s)
+"""]]
+
+[[!format sh """
+    conflict resolution (adjusted branch):                [master ea63ee5] git-annex in .t/tmprepo44
+ 1 file changed, 1 insertion(+)
+ create mode 120000 conflictor
+[master 07d129c] git-annex in .t/tmprepo45
+ 1 file changed, 1 insertion(+)
+ create mode 120000 conflictor
+Switched to branch 'adjusted/master(unlocked)'
+On branch master
+nothing to commit, working tree clean
+From ../../.t/tmprepo45
+ * [new branch]      adjusted/master(unlocked) -> r2/adjusted/master(unlocked)
+ * [new branch]      git-annex                 -> r2/git-annex
+ * [new branch]      master                    -> r2/master
+ * [new branch]      synced/master             -> r2/synced/master
+Auto-merging conflictor
+CONFLICT (add/add): Merge conflict in conflictor
+Automatic merge failed; fix conflicts and then commit the result.
+conflictor: needs merge
+[master 5d5a674] git-annex automatic merge conflict fix
+Already up-to-date.
+To ../../.t/tmprepo45
+   07d129c..5d5a674  master -> synced/master
+ * [new branch]      git-annex -> synced/git-annex
+On branch adjusted/master(unlocked)
+nothing to commit, working tree clean
+merge: refs/heads/synced/master - not something we can merge
+From ../../.t/tmprepo44
+ * [new branch]      git-annex     -> r1/git-annex
+ * [new branch]      master        -> r1/master
+ * [new branch]      synced/master -> r1/synced/master
+merge: refs/remotes/r1/master - not something we can merge
+merge: refs/remotes/r1/synced/master - not something we can merge
+FAIL (1.35s)
+"""]]
+

(Diff truncated)
Added a comment: Related...
diff --git a/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking/comment_1_562cf3ab3398c1ef874c0bfe5d89e47b._comment b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking/comment_1_562cf3ab3398c1ef874c0bfe5d89e47b._comment
new file mode 100644
index 0000000..7511c36
--- /dev/null
+++ b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking/comment_1_562cf3ab3398c1ef874c0bfe5d89e47b._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="leavingchicago@c04c893e78d1c4c76cb3e32b5c227cf42bbf7682"
+ nickname="leavingchicago"
+ avatar="http://cdn.libravatar.org/avatar/4ae498d3d6ee558d6b65caa658f72572"
+ subject="Related..."
+ date="2017-05-15T04:43:44Z"
+ content="""
+I think this bug is related to the problems that I'm having with [/forum/Get source group to automatically drop with assistant](https://git-annex.branchable.com/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/)
+
+"""]]

diff --git a/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn
new file mode 100644
index 0000000..3223c61
--- /dev/null
+++ b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn
@@ -0,0 +1,215 @@
+### Please describe the problem.
+Sync / Assistant between a `source` and `transfer` repo results in zero copies for whereis, but data is available.
+
+
+### What steps will reproduce the problem?
+See a script below...
+
+``` sh
+#!/bin/sh
+
+TMP=$(mktemp -d)
+
+
+##########################################
+# Set up the repos
+##########################################
+
+
+###
+# Create working directories
+cd ${TMP}
+mkdir source transfer
+
+
+###
+# Set up the source repo
+cd ${TMP}/source
+git init
+git annex init source
+git-annex group here source
+git annex wanted here standard
+
+
+###
+# Set up the transfer repo
+# N.B. It can see the source repo
+cd ${TMP}/transfer
+git init
+git remote add source ${TMP}/source
+git annex init transfer
+git-annex group here transfer
+git annex wanted here \
+  'not (inallgroup=client and copies=client:1) and ((include=* and ((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1)))) or approxlackingcopies=1)'
+
+
+##########################################
+# Add some files and run the assistant
+##########################################
+
+###
+# Make sure it works by adding some files to the source...
+cd ${TMP}/source
+date > file.1
+date > file.2
+date > file.3
+
+# Run the assistant in the source...
+(cd ${TMP}/source && \
+    git annex assistant && \
+    sleep 30 && \
+    git annex assistant --stop && \
+    git annex whereis file.*)
+# PASS: All files are in source.
+
+# Now run it in the transfer group
+(cd ${TMP}/transfer && \
+    git annex assistant && \
+    sleep 30 && \
+    git annex assistant --stop && \
+    git annex whereis file.*)
+# PASS: All files are in transfer
+
+# Check the source
+(cd ${TMP}/source && \
+    git annex whereis file.*)
+# FAIL: Only file.1 was updated to be in source.
+
+# whereis file.1 (1 copy)
+#        64b193e0-86f0-4349-b073-70af919ce628 -- transfer
+# ok
+# whereis file.2 (0 copies) failed
+# whereis file.3 (0 copies) failed
+# git-annex: whereis: 2 failed
+
+
+##########################################
+# Start to scratch head and ask 'Why?'
+##########################################
+
+##########
+# What's happening???
+
+cd ${TMP}/source && git annex sync
+# Source thinks it is updated - working tree clean
+
+cd ${TMP}/transfer && git annex sync
+# Transfer pulls from source, so run again
+git annex sync
+# OK... working tree clean
+
+
+# Check again
+cd ${TMP}/source && git annex whereis file.*
+# whereis file.1 (1 copy)
+#         64b193e0-86f0-4349-b073-70af919ce628 -- transfer
+# ok
+# whereis file.2 (0 copies) failed
+# whereis file.3 (0 copies) failed
+# git-annex: whereis: 2 failed
+
+cd ${TMP}/transfer && git annex whereis file.*
+# whereis file.1 (1 copy)
+#         64b193e0-86f0-4349-b073-70af919ce628 -- transfer [here]
+# ok
+# whereis file.2 (0 copies) failed
+# whereis file.3 (0 copies) failed
+# git-annex: whereis: 2 failed
+
+
+# HOWEVER...  Files are definitely in the transfer repo
+
+cd ${TMP}/source && cat file.*
+# cat: file.1: No such file or directory
+# cat: file.2: No such file or directory
+# cat: file.3: No such file or directory
+
+
+cd ${TMP}/transfer && cat file.*
+# Sun May 14 21:22:02 PDT 2017
+# Sun May 14 21:22:03 PDT 2017
+# Sun May 14 21:22:06 PDT 2017
+```
+
+
+
+### What version of git-annex are you using? On what operating system?
+
+#### Version:
+```
+git-annex version: 6.20170510
+build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.6.1 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+local repository version: 5
+supported repository versions: 3 5 6
+upgrade supported from repository versions: 0 1 2 3 4 5
+operating system: darwin x86_64
+```
+
+#### OS:
+MacOS 10.12.4, but seems to apply to Debian Jessie too.
+
+
+### Please provide any additional information below.
+
+Annex assistant daemon.log for `source` repo:
+
+``` sh
+[2017-05-15 14:11:24.150202] main: starting assistant version 6.20170510
+[2017-05-15 14:11:24.196597] Cronner: You should enable consistency checking to protect your data.
+(scanning...) [2017-05-15 14:11:24.257123] Watcher: Performing startup scan
+(started...)
+[2017-05-15 14:11:45.505855] Committer: Adding file.1 file.2 file.3
+add file.1 ok
+add file.2 ok
+add file.3 ok
+[2017-05-15 14:11:45.544277] Committer: Committing changes to git
+(recording state in git...)
+(recording state in git...)
+```
+
+Annex assistant daemon.log for `transfer` repo:
+
+``` sh
+[2017-05-15 14:12:09.127406] main: starting assistant version 6.20170510
+[2017-05-15 14:12:09.157611] Cronner: You should enable consistency checking to protect your data.
+[2017-05-15 14:12:09.177452] TransferScanner: Syncing with source
+(scanning...) [2017-05-15 14:12:09.210605] Watcher: Performing startup scan
+(started...)
+warning: no common commits
+From /Users/olaf/tmp/git-annex/play/bug-play/source
+ * [new branch]      git-annex     -> source/git-annex
+ * [new branch]      master        -> source/master
+ * [new branch]      synced/master -> source/synced/master
+(merging source/git-annex into git-annex...)
+(recording state in git...)
+
+
+Already up-to-date.
+[2017-05-15 14:12:10.256195] Committer: Committing changes to git
+(recording state in git...)
+(checksum...) [2017-05-15 14:12:10.425494] Transferrer: Downloaded file.1
+[2017-05-15 14:12:11.17664] Pusher: Syncing with source

(Diff truncated)
Add missing backtick around code
diff --git a/doc/git-annex-adjust.mdwn b/doc/git-annex-adjust.mdwn
index 140de63..772d27b 100644
--- a/doc/git-annex-adjust.mdwn
+++ b/doc/git-annex-adjust.mdwn
@@ -4,7 +4,7 @@ git-annex adjust - enter an adjusted branch
 
 # SYNOPSIS
 
-git annex adjust --unlock|--fix`
+`git annex adjust --unlock|--fix`
 
 # DESCRIPTION
 

Describe issue, actions done, question.
diff --git a/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__.mdwn b/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__.mdwn
new file mode 100644
index 0000000..ea4e30c
--- /dev/null
+++ b/doc/forum/Fix_wrong_rsync_with_git_annex_reinit__63__.mdwn
@@ -0,0 +1,38 @@
+# Context
+
+* Had two repo A and B. 
+* A had a lot of content not yet tracked and thus not available on B.
+* Filesystem problem on A: (relatively small) part of big repo content is lost.
+* In emergency, used rsync to propagate yet untracked content from failing A to sane B.
+* But forgot `ignore .git`, so rsync also partly rsync'ed `.git/annex` .
+* Now sane repo B believes it is the source one (same UUID A) as shown by `git annex info --fast`. 
+
+Both A and B show:
+
+    UUID-of-A -- A [here]
+    UUID-of-B -- B
+
+# Actions done
+
+Search on Google, see [[https://git-annex.branchable.com/git-annex-reinit/]] .
+
+On B:
+
+    git annex reinit UUID-of-B
+    git annex fsck
+
+Now `git annex info -fast` on B shows as expected:
+
+    UUID-of-A -- A
+    UUID-of-B -- B [here]
+
+
+I'm currently doing `git annex copy --from B` to repropagate lost parts of A.
+
+# Question
+
+* Were actions done best thing to do?
+* Is there something more to do?
+* Something to check?
+
+Thanks.

Added a comment: dtruss OS X 10.11 git annex on SMB mount
diff --git a/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment
new file mode 100644
index 0000000..c549c19
--- /dev/null
+++ b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment
@@ -0,0 +1,86 @@
+[[!comment format=mdwn
+ username="ewen"
+ avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
+ subject="dtruss OS X 10.11 git annex on SMB mount"
+ date="2017-05-14T05:04:29Z"
+ content="""
+A tar file with:
+
+    ewen@ashram:~/Desktop$ tar -tzf dtruss-git-annex-osx-smb-2017-05-14.tar.gz
+    dtruss-git-annex-osx-smb-2017-05-14/
+    dtruss-git-annex-osx-smb-2017-05-14/00-ga-test-smb-osx-symlinks-2017-05-14.script
+    dtruss-git-annex-osx-smb-2017-05-14/01-ga-test-smb-osx-symlinks-dtruss-git-annex-init-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/02-ga-test-smb-osx-symlinks-dtruss-git-annex-sync-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/03-ga-test-smb-osx-symlinks-dtruss-git-annex-get-all-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/04-ga-test-smb-osx-symlinks-dtruss-git-annex-sync-other-way-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/05-ga-test-smb-osx-symlinks-dtruss-git-annex-copy-to-smb-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/dtruss-ga
+    ewen@ashram:~/Desktop$ 
+
+is available at [dtruss-git-annex-osx-smb-2017-05-14.tar.gz](http://ewen.mcneill.gen.nz/temp/dtruss-git-annex-osx-smb-2017-05-14.tar.gz) (at least until I next tidy up the `/temp/` directory :-) ).
+
+The script file is the output of \"`script ....`\" showing all the commands (including typos :-) ) run, except the `dtruss` ones themselves.  The various `0n-ga-tst-smb-osx-symlinks-dtruss...` files contain the output of \"`sudo dtruss -af PID`\" where the `PID` was printed out by the script `dtruss-ga` that is included in the archive.  `dtruss-ga` was used to enable running git-annex as non-root (to ensure that the SMB access wasn't altered by running as a different user), since `dtruss` itself can only run as root.  The script is just:
+
+    ewen@ashram:~/Desktop$ cat dtruss-git-annex-osx-smb-2017-05-14/dtruss-ga 
+    #! /bin/sh
+    # Allow dtruss of non-root git annex
+    #
+    # Based on hints at:
+    # http://sec.cs.ucl.ac.uk/users/smurdoch/blog/2015/08/fixing-slow-start-mc-on-osx.html
+    #---------------------------------------------------------------------------
+
+    echo \"Run: sudo dtruss -af -p $$\"
+    echo \"Then press enter\"
+    read
+    exec git annex \"$@\"
+    ewen@ashram:~/Desktop$ 
+
+so basically \"print PID, wait for go ahead, run git annex command\".
+
+From a quick glance through, it looks like:
+
+    7003/0x16c3126:    121576      14     11 write(0x12, \"PidLock {lockingPid = 7003, lockingHost = \\"ashram\\"}\0\", 0x33)           = 51 0
+    7003/0x16c3126:    121775    3350    191 close(0x12)            = 0 0
+    7003/0x16c3126:    121792      10      6 stat64(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249\0\", 0x200004940, 0x33)                 = 0 0
+    7003/0x16c3126:    122041    2341    235 link(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249\0\", \"/Volumes/music/ga-test/.git/annex/locktmp16807282475249.lnk\0\")             = -1 Err#45
+    7003/0x16c3126:    122115     885     49 unlink(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249.lnk\0\", 0x200004D80, 0x33)             = -1 Err#2
+
+(in `03-...`, the first `get`) probably isn't helping -- that's trying to do a *hardlink* on a SMB file system, and appearing to give up when it fails... (`/Volumes/music` is the SMB mount I was using for testing).
+
+Also since I didn't realise until after the `script` file/`tar` file were made, there *is* an `annex/pidlock` file:
+
+    ewen@ashram:~$ ls -l /Volumes/music/ga-test/.git/annex/pidlock 
+    -rwx------  1 ewen  staff  51 14 May 16:25 /Volumes/music/ga-test/.git/annex/pidlock
+    ewen@ashram:~$ cat /Volumes/music/ga-test/.git/annex/pidlock 
+    PidLock {lockingPid = 7143, lockingHost = \"ashram\"}ewen@ashram:~$ 
+    ewen@ashram:~$ 
+
+created within the SMB `git-annex`, which looks vaguely sensible.  And FTR, there is *not* one in the HFS (local OS X) `git-annex`:
+
+    ewen@ashram:~$ ls -l /var/tmp/ga-test/.git/annex/pidlock
+    ls: /var/tmp/ga-test/.git/annex/pidlock: No such file or directory
+    ewen@ashram:~$ 
+
+so it definitely seems to *try* to do pidlocking on the SMB mount on transfers, but it looks like maybe it's also trying to do something *not* supported.
+
+Looking through the OS X `open(2)` manpage, it appears that:
+
+           O_EXCL          error if O_CREAT and the file exists
+           [...]
+           O_EXLOCK        atomically obtain an exclusive lock
+
+might be plausible non-hardlink locking primitives that'd work better than nothing.  The manpage also says:
+
+    When opening a file, a lock with flock(2) semantics can be obtained by
+    setting O_SHLOCK for a shared lock, or O_EXLOCK for an exclusive lock.
+    If creating a file with O_CREAT, the request for the lock will never fail
+    (provided that the underlying filesystem supports locking).
+
+so maybe that's something to work with?  (SMB as a protocol supports All Kinds Of Locking (tm), and hopefully the OS X SMB file system driver would also translate O_EXLOCK, etc, through to those...)
+
+Hope that's of some help.
+
+Ewen
+
+PS: Yes, I do plan to try to install the stand alone x86-64 git-annex on my Synology when I get a spare few hours to sort out all the details.  It seems like it would be the most robust solution for my specific problem.  But generalised SMB support could also be useful, even if it was \"user needs to be careful about running commands in parallel as there are races\".
+"""]]

Added a comment
diff --git a/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_9_9b91f93a66d3e38edc4490be20445a6d._comment b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_9_9b91f93a66d3e38edc4490be20445a6d._comment
new file mode 100644
index 0000000..586dd6b
--- /dev/null
+++ b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_9_9b91f93a66d3e38edc4490be20445a6d._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="t.z.mates"
+ avatar="http://cdn.libravatar.org/avatar/90f15fad216078fd08d62cc676487925"
+ subject="comment 9"
+ date="2017-05-13T20:59:34Z"
+ content="""
+Thanks for the response! I've [linked](https://github.com/tzmates/GitAnnexStrace) to a repo where I posted the strace results (only for the offending line, otherwise there were a lot of files) as well as the code that generated it. I'm not very familiar with strace results, so let me know if I should supply anything else.
+
+It certainly could be a BTRFS bug, though it doesn't seem to be tied to a specific kernel version; I've been continually updating kernels for the last 2 years now.
+
+If it's any help, I've never had this sort of problem using `cp --reflink` day-to-day. I have no experience with `mmap` though, so that's a distinct possibility.
+
+Thanks for all your input on this so far!
+"""]]

pointers
diff --git a/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
index b106e66..1df0694 100644
--- a/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
+++ b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
@@ -39,7 +39,11 @@ Some approaches to fix it:
    we then know ssh will prompt, we can display the hostname as context for
    the "Password:" prompt it uses.
 
+   Needs concurrent-output 1.8.0
+
 4. Find a way to add cursor position querying to ansi-terminal. Can it be
    done portably?
 
+   See <https://github.com/feuerbach/ansi-terminal/issues/7>
+
 --[[Joey]]

break out bug report
diff --git a/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment
index 1de5890..8860bc5 100644
--- a/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment
+++ b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment
@@ -6,5 +6,6 @@
 Current status: It's implemented, but not for `GIT_SSH` yet.
 
 The display is a bit ugly, because the ssh password prompt line
-confuses the concurrent-output region manager.
+confuses the concurrent-output region manager. Opened
+[[minor_display_glitch_with_ssh_password_prompting_and_-J]] bug for that.
 """]]
diff --git a/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
new file mode 100644
index 0000000..b106e66
--- /dev/null
+++ b/doc/bugs/minor_display_glitch_with_ssh_password_prompting_and_-J.mdwn
@@ -0,0 +1,45 @@
+When using -J and there's a ssh password prompt (or other prompt eg ssh
+host key), the region-based display gets messed up by the ssh output. This
+is a minor display glitch; it's still fairly clear what git-annex is doing.
+
+The root problem is that the regional display code does not know the
+absolute cursor position. All cursor movement is relative. So when ssh
+display moves the cursor, all subsequent output goes to the wrong place.
+
+ansi-terminal has absolute cursor movement, but no way to query position.
+
+Some approaches to fix it:
+
+1. Allocate a slave pty and run ssh in there, forwarding IO from the slave
+   pty to the master pty. The ssh output is then added to the region that
+   it's prompting for the password for.
+   
+   Unix-specific and somewhat heavyweight solution.
+
+2. Set position to eg 0,0 when starting git-annex, and then the
+   absolute position can be calculated, and after ssh runs it can reset the
+   cursor to the previous position. 
+   
+   Would make -J take over the whole screen even if it's only transferring
+   1 file.
+
+3. Clear all regions before running the ssh command that can prompt,
+   (moving the cursor to the start of the first region),
+   and redraw them when it's done. So the ssh output would appear above the
+   redrawn regions.
+
+   This would cause some flicker in the common case where ssh does not have
+   any output. The N regions would display briefly, then be cleared, then
+   be redrawn. It might flicker multiple times, when multiple different
+   hosts are being accessed. 
+   
+   One way to avoid the flicker would be to first
+   try to ssh with password prompting disabled, and only if that fails do
+   regions need to be cleared for the ssh that will prompt. Also, since
+   we then know ssh will prompt, we can display the hostname as context for
+   the "Password:" prompt it uses.
+
+4. Find a way to add cursor position querying to ansi-terminal. Can it be
+   done portably?
+
+--[[Joey]]

misplaced comment
diff --git a/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment
new file mode 100644
index 0000000..1de5890
--- /dev/null
+++ b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_8_5da63cf5fa93120c85b98077fba51488._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""status"""
+ date="2017-05-11T21:57:24Z"
+ content="""
+Current status: It's implemented, but not for `GIT_SSH` yet.
+
+The display is a bit ugly, because the ssh password prompt line
+confuses the concurrent-output region manager.
+"""]]
diff --git a/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment b/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
deleted file mode 100644
index 0527904..0000000
--- a/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
+++ /dev/null
@@ -1,10 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 1"""
- date="2017-05-11T21:57:24Z"
- content="""
-Current status: It's implemented, but not for `GIT_SSH` yet.
-
-The display is a bit ugly, because the ssh password prompt line
-confuses the concurrent-output region manager.
-"""]]

Added a comment
diff --git a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_3_553542a4a2b2530863a006a1afe91637._comment b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_3_553542a4a2b2530863a006a1afe91637._comment
new file mode 100644
index 0000000..3f3dae5
--- /dev/null
+++ b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_3_553542a4a2b2530863a006a1afe91637._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="olaf"
+ avatar="http://cdn.libravatar.org/avatar/4ae498d3d6ee558d6b65caa658f72572"
+ subject="comment 3"
+ date="2017-05-11T23:45:36Z"
+ content="""
+Oops.  I committed the greatest sin...
+
+```
+git-annex version: 6.20170321-gf3dee9d65
+build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+local repository version: 5
+supported repository versions: 3 5 6
+upgrade supported from repository versions: 0 1 2 3 4 5
+operating system: linux i386
+```
+"""]]

Added a comment
diff --git a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_2_835dba9de7a5eea868635dc793d4e75b._comment b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_2_835dba9de7a5eea868635dc793d4e75b._comment
new file mode 100644
index 0000000..34d3b46
--- /dev/null
+++ b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_2_835dba9de7a5eea868635dc793d4e75b._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="leavingchicago@c04c893e78d1c4c76cb3e32b5c227cf42bbf7682"
+ nickname="leavingchicago"
+ avatar="http://cdn.libravatar.org/avatar/4ae498d3d6ee558d6b65caa658f72572"
+ subject="comment 2"
+ date="2017-05-11T23:23:45Z"
+ content="""
+&lt;ASIDE&gt;</br>
+*Laughing* I wish the data collection was interesting.  Just a machine with locked down networking so that it's a tad more secure.  Closest example is an EC2 instance sitting in a VPC...</br> 
+&lt;/ASIDE&gt;
+
+Thanks for the feedback Joey.  I tried again and this approach works **except** where the downstream repo is direct mode - in this case the transfer repo doesn't drop.  I'm not sure if it's a bug or a feature I didn't read about.
+
+Happy to submit a bug report with steps if it's unexpected.
+"""]]

fix sshCleanup race using STM
diff --git a/Annex.hs b/Annex.hs
index 3c47789..597a5dd 100644
--- a/Annex.hs
+++ b/Annex.hs
@@ -71,6 +71,7 @@ import Utility.Url
 import "mtl" Control.Monad.Reader
 import Control.Concurrent
 import Control.Concurrent.Async
+import Control.Concurrent.STM
 import qualified Data.Map as M
 import qualified Data.Set as S
 
@@ -124,7 +125,7 @@ data AnnexState = AnnexState
 	, groupmap :: Maybe GroupMap
 	, ciphers :: M.Map StorableCipher Cipher
 	, lockcache :: LockCache
-	, sshstalecleaned :: MVar ()
+	, sshstalecleaned :: TMVar Bool
 	, flags :: M.Map String Bool
 	, fields :: M.Map String String
 	, cleanup :: M.Map CleanupAction (Annex ())
@@ -147,7 +148,7 @@ newState :: GitConfig -> Git.Repo -> IO AnnexState
 newState c r = do
 	emptyactiveremotes <- newMVar M.empty
 	o <- newMessageState
-	sc <- newMVar ()
+	sc <- newTMVarIO False
 	return $ AnnexState
 		{ repo = r
 		, repoadjustment = return
diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs
index 09df676..e0cc4a0 100644
--- a/Annex/Ssh.hs
+++ b/Annex/Ssh.hs
@@ -43,7 +43,7 @@ import Annex.LockPool
 #endif
 
 import Data.Hash.MD5
-import Control.Concurrent
+import Control.Concurrent.STM
 
 {- Some ssh commands are fed stdin on a pipe and so should be allowed to
  - consume it. But ssh commands that are not piped stdin should generally
@@ -173,23 +173,31 @@ prepSocket :: FilePath -> RemoteGitConfig -> [CommandParam] -> Annex ()
 prepSocket socketfile gc sshparams = do
 	-- There could be stale ssh connections hanging around
 	-- from a previous git-annex run that was interrupted.
-	-- This must run only once, before we have made any ssh connection.
-	whenM (isJust <$> (liftIO . tryTakeMVar =<< Annex.getState Annex.sshstalecleaned)) $
-		sshCleanup
+	-- This must run only once, before we have made any ssh connection,
+	-- and any other prepSocket calls must block while it's run.
+	tv <- Annex.getState Annex.sshstalecleaned
+	join $ liftIO $ atomically $ do
+		cleaned <- takeTMVar tv
+		if cleaned
+			then do
+				putTMVar tv cleaned
+				return noop
+			else return $ do
+				sshCleanup
+				liftIO $ atomically $ putTMVar tv True
 	-- Cleanup at shutdown.
 	Annex.addCleanup SshCachingCleanup sshCleanup
 	
 	liftIO $ createDirectoryIfMissing True $ parentDir socketfile
 	let socketlock = socket2lock socketfile
 
-	prompt $ \s -> when (concurrentOutputEnabled s) $
+	prompt $ \s -> when (concurrentOutputEnabled s) $ do
 		-- If the LockCache already has the socketlock in it,
 		-- the connection has already been started. Otherwise,
 		-- get the connection started now.
 		whenM (isNothing <$> fromLockCache socketlock) $
 			void $ liftIO $ boolSystem "ssh" $
 				sshparams ++ startSshConnection gc
-	
 	lockFileCached socketlock
 
 -- Parameters to get ssh connected to the remote host,
diff --git a/doc/devblog/day_457__improved_ssh_password_prompting.mdwn b/doc/devblog/day_457__improved_ssh_password_prompting.mdwn
index 0b99cbc..df38304 100644
--- a/doc/devblog/day_457__improved_ssh_password_prompting.mdwn
+++ b/doc/devblog/day_457__improved_ssh_password_prompting.mdwn
@@ -15,6 +15,3 @@ up several ssh's at the same time, so connection caching didn't kick in,
 and there could be a bunch of ssh password prompts at the same time. Now
 there will never be more than one ssh password prompt at once, and only one
 prompt per host. (As long as connection caching is enabled.)
-
-That mostly works, but `git annex drop -J` exposes a bug in it, which I'll
-have to get back to tomorrow.
diff --git a/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment b/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
index 4e0c93f..0527904 100644
--- a/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
+++ b/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
@@ -5,15 +5,6 @@
  content="""
 Current status: It's implemented, but not for `GIT_SSH` yet.
 
-`git annex get -J 3` seems to work well.
-
-`git annex drop -J 3` sometimes (once per 5 or so tries) runs
-2 ssh processes both prompting at once. For example:
-
-	21555 pts/3    S+     0:00      |           \_ ssh localhost -S .git/annex/ssh/localhost -o ControlMaster=auto -o ControlPersist=yes -T git-annex-shell 'lockcontent' '/~/tmp/b/a' 'SHA256E-s30--ec748fe42378798b82ddf4e6b3e778a0cbc5d76d458685af01dc1f09cc423eb7' --uuid ad19d177-519e-4440-a9c9-0c9cddd99db2
-	21556 pts/3    S+     0:00      |           \_ ssh localhost -S .git/annex/ssh/localhost -o ControlMaster=auto -o ControlPersist=yes -n -T git-annex-shell inannex .
-
-The inannex is what's used to bring up the ssh connection, so
-I don't immediately understand how the lockcontent could run before
-the ssh connection is brought up.
+The display is a bit ugly, because the ssh password prompt line
+confuses the concurrent-output region manager.
 """]]

devblog
diff --git a/doc/devblog/day_457__improved_ssh_password_prompting.mdwn b/doc/devblog/day_457__improved_ssh_password_prompting.mdwn
new file mode 100644
index 0000000..0b99cbc
--- /dev/null
+++ b/doc/devblog/day_457__improved_ssh_password_prompting.mdwn
@@ -0,0 +1,20 @@
+After a month away building [debug-me](https://debug-me.branchable.com/)
+I'm back working on git-annex. I hope debug-me will be useful for debugging
+git-annex in some situations BTW.
+
+Pushed a release yesterday that was mostly changes from back in March.
+It also updated the git bundled with git-annex to fix the recent git-shell
+security hole.
+
+After work on Monday and today, I am caught up with all the recent month's
+backlog, but still have 230 old backlogged messages to get to.
+
+The first consequental thing I got back to was improving ssh password
+prompting when git-annex is running concurrently with -J. It used to start
+up several ssh's at the same time, so connection caching didn't kick in,
+and there could be a bunch of ssh password prompts at the same time. Now
+there will never be more than one ssh password prompt at once, and only one
+prompt per host. (As long as connection caching is enabled.)
+
+That mostly works, but `git annex drop -J` exposes a bug in it, which I'll
+have to get back to tomorrow.
diff --git a/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment b/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
new file mode 100644
index 0000000..4e0c93f
--- /dev/null
+++ b/get_-J_cannot_be_used_with_password-based_authentication/comment_1_5da63cf5fa93120c85b98077fba51488._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-11T21:57:24Z"
+ content="""
+Current status: It's implemented, but not for `GIT_SSH` yet.
+
+`git annex get -J 3` seems to work well.
+
+`git annex drop -J 3` sometimes (once per 5 or so tries) runs
+2 ssh processes both prompting at once. For example:
+
+	21555 pts/3    S+     0:00      |           \_ ssh localhost -S .git/annex/ssh/localhost -o ControlMaster=auto -o ControlPersist=yes -T git-annex-shell 'lockcontent' '/~/tmp/b/a' 'SHA256E-s30--ec748fe42378798b82ddf4e6b3e778a0cbc5d76d458685af01dc1f09cc423eb7' --uuid ad19d177-519e-4440-a9c9-0c9cddd99db2
+	21556 pts/3    S+     0:00      |           \_ ssh localhost -S .git/annex/ssh/localhost -o ControlMaster=auto -o ControlPersist=yes -n -T git-annex-shell inannex .
+
+The inannex is what's used to bring up the ssh connection, so
+I don't immediately understand how the lockcontent could run before
+the ssh connection is brought up.
+"""]]

Ssh password prompting improved when using -J
When ssh connection caching is enabled (and when GIT_ANNEX_USE_GIT_SSH is
not set), only one ssh password prompt will be made per host, and only one
ssh password prompt will be made at a time.
This also fixes a race in prepSocket's stale ssh connection stopping
when run with -J. It was possible for one thread to start a cached ssh
connection, and another thread to immediately stop it, resulting in excess
connections being made.
This commit was supported by the NSF-funded DataLad project.
diff --git a/Annex.hs b/Annex.hs
index 2a372f1..3c47789 100644
--- a/Annex.hs
+++ b/Annex.hs
@@ -124,6 +124,7 @@ data AnnexState = AnnexState
 	, groupmap :: Maybe GroupMap
 	, ciphers :: M.Map StorableCipher Cipher
 	, lockcache :: LockCache
+	, sshstalecleaned :: MVar ()
 	, flags :: M.Map String Bool
 	, fields :: M.Map String String
 	, cleanup :: M.Map CleanupAction (Annex ())
@@ -145,6 +146,8 @@ data AnnexState = AnnexState
 newState :: GitConfig -> Git.Repo -> IO AnnexState
 newState c r = do
 	emptyactiveremotes <- newMVar M.empty
+	o <- newMessageState
+	sc <- newMVar ()
 	return $ AnnexState
 		{ repo = r
 		, repoadjustment = return
@@ -152,7 +155,7 @@ newState c r = do
 		, backend = Nothing
 		, remotes = []
 		, remoteannexstate = M.empty
-		, output = def
+		, output = o
 		, concurrency = NonConcurrent
 		, force = False
 		, fast = False
@@ -175,6 +178,7 @@ newState c r = do
 		, groupmap = Nothing
 		, ciphers = M.empty
 		, lockcache = M.empty
+		, sshstalecleaned = sc
 		, flags = M.empty
 		, fields = M.empty
 		, cleanup = M.empty
diff --git a/Annex/LockFile.hs b/Annex/LockFile.hs
index cb1d232..1f35444 100644
--- a/Annex/LockFile.hs
+++ b/Annex/LockFile.hs
@@ -11,6 +11,7 @@ module Annex.LockFile (
 	lockFileCached,
 	unlockFile,
 	getLockCache,
+	fromLockCache,
 	withExclusiveLock,
 	tryExclusiveLock,
 ) where
diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs
index ae14a56..09df676 100644
--- a/Annex/Ssh.hs
+++ b/Annex/Ssh.hs
@@ -23,10 +23,6 @@ module Annex.Ssh (
 	runSshAskPass
 ) where
 
-import qualified Data.Map as M
-import Data.Hash.MD5
-import System.Exit
-
 import Annex.Common
 import Annex.LockFile
 import qualified Build.SysConfig as SysConfig
@@ -38,6 +34,7 @@ import Annex.Path
 import Utility.Env
 import Utility.FileSystemEncoding
 import Types.CleanupActions
+import Types.Messages
 import Git.Env
 import Git.Ssh
 #ifndef mingw32_HOST_OS
@@ -45,6 +42,9 @@ import Annex.Perms
 import Annex.LockPool
 #endif
 
+import Data.Hash.MD5
+import Control.Concurrent
+
 {- Some ssh commands are fed stdin on a pipe and so should be allowed to
  - consume it. But ssh commands that are not piped stdin should generally
  - not be allowed to consume the process's stdin. -}
@@ -71,16 +71,18 @@ sshCommand cs (host, port) gc remotecmd = ifM (liftIO safe_GIT_SSH)
 sshOptions :: ConsumeStdin -> (String, Maybe Integer) -> RemoteGitConfig -> [CommandParam] -> Annex [CommandParam]
 sshOptions cs (host, port) gc opts = go =<< sshCachingInfo (host, port)
   where
-	go (Nothing, params) = ret params
+	go (Nothing, params) = return $ mkparams cs params
 	go (Just socketfile, params) = do
-		prepSocket socketfile
-		ret params
-	ret ps = return $ concat
+		prepSocket socketfile gc
+			(Param host : mkparams NoConsumeStdin params)
+			
+		return $ mkparams cs params
+	mkparams cs' ps = concat
 		[ ps
 		, map Param (remoteAnnexSshOptions gc)
 		, opts
 		, portParams port
-		, consumeStdinParams cs
+		, consumeStdinParams cs'
 		, [Param "-T"]
 		]
 
@@ -158,21 +160,52 @@ portParams :: Maybe Integer -> [CommandParam]
 portParams Nothing = []
 portParams (Just port) = [Param "-p", Param $ show port]
 
-{- Prepare to use a socket file for ssh connection caching. 
- - Locks a lock file to prevent other git-annex processes from
- - stopping the ssh multiplexer on this socket. -}
-prepSocket :: FilePath -> Annex ()
-prepSocket socketfile = do
-	-- If the lock pool is empty, this is the first ssh of this
-	-- run. There could be stale ssh connections hanging around
+{- Prepare to use a socket file for ssh connection caching.
+ -
+ - When concurrency is enabled, this blocks until a ssh connection
+ - has been made to the host. So, any password prompting by ssh will
+ - happen in this call, and only one ssh process will prompt at a time.
+ -
+ - Locks the socket lock file to prevent other git-annex processes from
+ - stopping the ssh multiplexer on this socket.
+ -}
+prepSocket :: FilePath -> RemoteGitConfig -> [CommandParam] -> Annex ()
+prepSocket socketfile gc sshparams = do
+	-- There could be stale ssh connections hanging around
 	-- from a previous git-annex run that was interrupted.
-	whenM (not . any isLock . M.keys <$> getLockCache)
+	-- This must run only once, before we have made any ssh connection.
+	whenM (isJust <$> (liftIO . tryTakeMVar =<< Annex.getState Annex.sshstalecleaned)) $
 		sshCleanup
-	-- Cleanup at end of this run.
+	-- Cleanup at shutdown.
 	Annex.addCleanup SshCachingCleanup sshCleanup
-
+	
 	liftIO $ createDirectoryIfMissing True $ parentDir socketfile
-	lockFileCached $ socket2lock socketfile
+	let socketlock = socket2lock socketfile
+
+	prompt $ \s -> when (concurrentOutputEnabled s) $
+		-- If the LockCache already has the socketlock in it,
+		-- the connection has already been started. Otherwise,
+		-- get the connection started now.
+		whenM (isNothing <$> fromLockCache socketlock) $
+			void $ liftIO $ boolSystem "ssh" $
+				sshparams ++ startSshConnection gc
+	
+	lockFileCached socketlock
+
+-- Parameters to get ssh connected to the remote host,
+-- by asking it to run a no-op command.
+--
+-- Could simply run "true", but the remote host may only
+-- allow git-annex-shell to run. So, run git-annex-shell inannex
+-- with the path to the remote repository and no other parameters,
+-- which is a no-op supported by all versions of git-annex-shell.
+startSshConnection :: RemoteGitConfig -> [CommandParam]
+startSshConnection gc =
+	[ Param "git-annex-shell"
+	, Param "inannex"
+	, File $ Git.repoPath $ gitConfigRepo $
+		remoteGitConfig gc
+	]
 
 {- Find ssh socket files.
  -
@@ -324,12 +357,20 @@ sshOptionsTo remote gc localr
 		Just host -> ifM (liftIO $ safe_GIT_SSH <&&> gitSshEnvSet)
 			( unchanged
 			, do
-				(msockfile, _) <- sshCachingInfo (host, Git.Url.port remote)
+				let port = Git.Url.port remote
+				(msockfile, cacheparams) <- sshCachingInfo (host, port)
 				case msockfile of
 					Nothing -> use []
 					Just sockfile -> do
-						prepSocket sockfile
-						use (sshConnectionCachingParams sockfile)
+						prepSocket sockfile gc $
+							Param host : concat
+								[ cacheparams
+								, map Param (remoteAnnexSshOptions gc)
+								, portParams port
+								, consumeStdinParams NoConsumeStdin
+								, [Param "-T"]
+								]
+						use cacheparams
 			)
   where
 	unchanged = return localr
diff --git a/CHANGELOG b/CHANGELOG
index e6b6d45..8edf324 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,12 @@

(Diff truncated)
hmm
diff --git a/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_7_0118a107147f6b94a7da907e599e58db._comment b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_7_0118a107147f6b94a7da907e599e58db._comment
new file mode 100644
index 0000000..24f34bc
--- /dev/null
+++ b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_7_0118a107147f6b94a7da907e599e58db._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2017-05-11T19:23:02Z"
+ content="""
+What about when `GIT_SSH` is used? `prepSocket` is not used then,
+and git-annex can only use the `GIT_SSH` interface to ssh to the host.
+So, the approach above won't work.
+
+git-annex could then try to use `GIT_SSH` to ssh to the host and run eg `true`,
+in hopes that `GIT_SSH` is enabling ssh connection caching and that will
+get the ssh connection set up. If `GIT_SSH` is not enabling connection
+caching, that might add an additional password prompt, and not avoid
+other password prompts from overlapping.
+"""]]

plan
diff --git a/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_6_f5fe8d4cecfceec5cb4a03dd054d2e0a._comment b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_6_f5fe8d4cecfceec5cb4a03dd054d2e0a._comment
new file mode 100644
index 0000000..cd692ca
--- /dev/null
+++ b/doc/bugs/get_-J_cannot_be_used_with_password-based_authentication/comment_6_f5fe8d4cecfceec5cb4a03dd054d2e0a._comment
@@ -0,0 +1,45 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2017-05-11T18:09:04Z"
+ content="""
+All cases could be dealt with by having a single process-level prompt lock
+(not a lock file, but an MVar), that's taken when doing something
+that might prompt for input.
+
+Then `Annex.Ssh.prepSocket` could block to take the prompt lock, and once
+it has the prompt lock, start the ssh connection multiplexer and wait for
+the the ssh connection to be established.
+
+Thus, even if `git annex get -J` is connecting to multiple hosts that each
+need passwords, password prompting would be serialized.
+
+All message output could also be blocked while the prompt lock is held,
+and then concurrent output would not scramble with the ssh password prompt.
+
+`ssh -S path -O check` does indeed exit nonzero when ssh has not yet
+connected and is at a password prompt. Also, I noticed that the socket file
+is only created after the password prompt, so a less expensive check
+(though perhaps not as accurate) is to see if the socket file exists.
+(But, it seems we don't need to check, see below.)
+
+The real problem is starting the ssh connection multiplexer without
+blocking for eg a whole rsync transfer to run. There's
+not a `-O` command that starts the multiplexer. The only way to do it seems
+to be something like `ssh -S path -o ControlMaster=auto -o 
+ControlPersist=yes host true`. So, run a no-op command on the remote host just
+to get the connection up. Then prepSocket will know the cached connection
+is up, and can drop the prompt lock and return.
+
+It would only need to do this when concurrency is enabled, so
+non-concurrent uses the current, faster path.
+
+prepSocket takes a shared 
+file level lock of the socket's lock file, which is used to tell when
+another git-annex process is using the connection multiplexer.
+So, an optimisation would be for prepSocket to try to take a non-blocking
+exclusive file-level lock. If it fails, it knows some process has
+already taken the shared lock, and so the multiplexer is started and no
+password prompting needs to be done. So it does not need to try to start
+the multiplexer in this case.
+"""]]

patchneeds to be sent to git
diff --git a/doc/tips/Faster_bash_autocompletion_with_big_annex_repos/comment_1_a41fb1037186319bbb44f7a67df6120f._comment b/doc/tips/Faster_bash_autocompletion_with_big_annex_repos/comment_1_a41fb1037186319bbb44f7a67df6120f._comment
new file mode 100644
index 0000000..3a44fef
--- /dev/null
+++ b/doc/tips/Faster_bash_autocompletion_with_big_annex_repos/comment_1_a41fb1037186319bbb44f7a67df6120f._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-11T17:29:47Z"
+ content="""
+Could you send this improvement to the git mailing list?
+"""]]

comment
diff --git a/doc/todo/export/comment_2_b13f12dd00347114f99fd11d85d236a0._comment b/doc/todo/export/comment_2_b13f12dd00347114f99fd11d85d236a0._comment
new file mode 100644
index 0000000..4e394dd
--- /dev/null
+++ b/doc/todo/export/comment_2_b13f12dd00347114f99fd11d85d236a0._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-05-11T17:25:48Z"
+ content="""
+Note that making `git annex export` need a remote to export to would be
+somewhat asymmetric compared to `git annex import` which can operate on any
+directory full of files.
+
+Still, if you want to export to a directory full of files, setting up a
+directory special remote before exporting is not too big a pain, probably.
+
+I suppose the question is whether this asymetricity and complication of
+perhaps needing changes to the remote interface for `export` is a price
+worth paying to be able to have export upload to lots of sorts of remotes.
+"""]]

followup
diff --git a/doc/forum/whereis__58___Why_not_here__63__/comment_2_e55ce8cbb6bbabc654ba743d755a6eee._comment b/doc/forum/whereis__58___Why_not_here__63__/comment_2_e55ce8cbb6bbabc654ba743d755a6eee._comment
new file mode 100644
index 0000000..cffe96e
--- /dev/null
+++ b/doc/forum/whereis__58___Why_not_here__63__/comment_2_e55ce8cbb6bbabc654ba743d755a6eee._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-05-11T17:22:01Z"
+ content="""
+Your log shows that the .emacs file reached Horus, but was then dropped
+from it some 46 minutes later.
+
+Something had to drop it. Are you using the git-annex assistant or what?
+"""]]

response
diff --git a/doc/forum/deploy/comment_1_bd9855cc19f0f49620258564845f1f77._comment b/doc/forum/deploy/comment_1_bd9855cc19f0f49620258564845f1f77._comment
new file mode 100644
index 0000000..1f4732b
--- /dev/null
+++ b/doc/forum/deploy/comment_1_bd9855cc19f0f49620258564845f1f77._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-11T17:13:35Z"
+ content="""
+I'm not sure that a bare repository is the best way to go about it.
+
+One way to do it is to have a non-bare repository, which gets updated when
+you push (by eg setting receive.denyNonFastforwards=false), and have that
+non-bare repository be the one you push to deploy to.
+
+While you could push first to a bare repository and have it then push on
+the the non-bare repository, that seems to mostly only complicate things,
+since you will need to find a way to copy the annexed file contents from
+the bare to the non-bare repository.
+"""]]

response
diff --git a/doc/forum/Synchronize_large_files___40__VM_images__41__/comment_3_6488bbe6f39e7154f950530498e9b548._comment b/doc/forum/Synchronize_large_files___40__VM_images__41__/comment_3_6488bbe6f39e7154f950530498e9b548._comment
new file mode 100644
index 0000000..9c55e48
--- /dev/null
+++ b/doc/forum/Synchronize_large_files___40__VM_images__41__/comment_3_6488bbe6f39e7154f950530498e9b548._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-11T17:06:23Z"
+ content="""
+The reason `git annex unlock` is slow is because it makes a copy
+of the entire file. The file is left as-is in the annex so the
+old version is available later, and the unlocked copy is made available for
+modification.
+
+More recent versions of git-annex support v6 mode, which has a annex.thin
+configuration that makes `git annex unlock` not do this copy, so it's very
+fast. But then no copy of the old version of the file will be made,
+and so you won't be able to revert to the old version. Which seems to be
+an important part of your workflow.
+
+Another way to make `git annex unlock` fast is to use a file system that
+supports Copy On Write (CoW). git-annex will use CoW automatically when
+available, and then unlocking doesn't need to actually copy the file,
+but the old version will still be preserved. Btrfs is the only filesystem
+I know of that supports CoW, although there may be others.
+"""]]

followup
diff --git a/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_2_8831965ea7abf5ac6fe860d93133fbb1._comment b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_2_8831965ea7abf5ac6fe860d93133fbb1._comment
new file mode 100644
index 0000000..c422dc9
--- /dev/null
+++ b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_2_8831965ea7abf5ac6fe860d93133fbb1._comment
@@ -0,0 +1,42 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-05-11T16:37:35Z"
+ content="""
+`git-annex init` is probably detecting a problem with the filesystem
+allowing writes to files whose permissions don't allow them to be written
+to. Since indirect mode relies on being able to prevent writing to files,
+it enables direct mode. The disabling of symlinks is a side effect of that.
+You can probably get away with using indirect mode, but there is the risk
+of annexed files getting modified and so the old version of files being
+lost.
+
+The pid lock is used for transfer locks when that's enabled.
+
+So, the pid lock creation is failing for some reason. The two likely causes
+would be:
+
+* The pid locking code not working well on OSX. This would not be too
+  surprising.
+
+  But, I gave pid locking on OSX a quick test, by making a repository
+  in my OSX home directory, `git config annex.pidlock true` and doing the
+  same in a clone. Transfers between the repositories worked ok so pid
+  locking seems to at least basically work on OSX.
+
+* Something to do with the SMB filesystem breaking the pid locking code.
+  Perhaps something to do with permissions since as noted `git annex init`
+  seems to have identified a problem with permissions handling.
+
+Could you possibly get a `dtruss` dump of git-annex when this error
+happens? That would narrow down the problem a lot.
+
+I could try to set up SMB etc, but I don't have root access to an OSX
+machine, so am not confident I could replicate the problem.
+
+----
+
+Also, it might overall be better to run git-annex on the Synology NAS.
+Then you don't have to mess with networked filesystems. People have done
+this successfully before.
+"""]]

comment
diff --git a/doc/tips/splitting_a_repository/comment_1_654527ef2350fe871e2d7ff6addc6713._comment b/doc/tips/splitting_a_repository/comment_1_654527ef2350fe871e2d7ff6addc6713._comment
new file mode 100644
index 0000000..3c04c20
--- /dev/null
+++ b/doc/tips/splitting_a_repository/comment_1_654527ef2350fe871e2d7ff6addc6713._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-11T16:28:32Z"
+ content="""
+This is a simple way to split a repository, but the resulting split git
+repository will be larger than is really necessary.
+
+When you `dropunused` all the hard links that are not present in the
+repository, git-annex will commit a log to the git-annex branch saying "I
+don't have this content" for each of them. That seems unnecessary since
+it probably does not have an earlier log saying it contained the content
+that was hard linked into it, and perhaps could be improved in git-annex
+to not record that unncessarily, but that's what it does currently.
+
+So I suggest running `git annex forget` after the dropunused or at some
+later point. That will delete all traces of those log files from
+the git-annex branch.
+"""]]

response
diff --git a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_1_ead4b5b7ec0d085d4fa87fb4e29c4d55._comment b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_1_ead4b5b7ec0d085d4fa87fb4e29c4d55._comment
new file mode 100644
index 0000000..dec09cf
--- /dev/null
+++ b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant/comment_1_ead4b5b7ec0d085d4fa87fb4e29c4d55._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-11T16:06:03Z"
+ content="""
+Sounds like an interesting data-gathering application, I have to say
+I'm curious what it is. ;)
+
+If Repo1 is configured like this:
+
+	git-annex group here source
+	git annex wanted here standard
+
+Then it should want to drop the contents of files from Repo1 once it knows
+they have reached any other repository. (Sometimes people put a repository
+in a group but forget to set wanted to "standard" ...)
+
+Looks like Repo1 cannot make outgoing connections to Repo2?
+
+So, you need to run the assistant on Repo2 and probably on Repo1.
+Then it works like this:
+
+1. The assistant on Repo1 will commit files as they're put in the repository.
+2. The assistant on Repo2 will notice changes to Repo1, pull down the
+   files, see that Repo1 is configured as a source repository, and drop
+   the files from Repo1.
+3. The assistant on Repo1 will also notice when it's sent a file to Repo2
+   and can drop it from Repo1 then -- but this may take a while for it to
+   notice and #2 will probably happen first.
+
+Note that if you get the files added and committed by some other process,
+you don't really need to run the assistant on Repo1.
+
+The USB drives need to be client, so that once content reaches one of them,
+the content will be dropped from the transfer repositories. The way that
+part should work:
+
+1. Content reaches Repo2.
+2. The assistant in Repo3 notices the change to Repo2, pulls down
+   the files.
+3. The assistant in Repo3 pushes the files to a connected USB drive.
+4. The assistant in Repo3 drops content from itself and Repo2, since
+   it knows both are transfer repos and the content has reached a client
+   (USB).
+
+If you're having trouble getting any of this to work, I recommend
+running `git annex sync --content` manually while testing it, and make sure
+it does what you would expect to happen at each step.
+"""]]

comment
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_7_bd6b81815e5888c26d7a8a96e565519e._comment b/doc/forum/Lots_of_4k_symlinks/comment_7_bd6b81815e5888c26d7a8a96e565519e._comment
new file mode 100644
index 0000000..148778f
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_7_bd6b81815e5888c26d7a8a96e565519e._comment
@@ -0,0 +1,23 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2017-05-11T15:57:10Z"
+ content="""
+Note that the analysis in my earlier comment assumes that the
+.git/annex/objects/xx/yy/key/ directory is removed. As long as those
+per-key directories are used, the symlinks cannot possibly be made short enough
+to pack.
+
+There have been some
+other requests for that (datalad requested it because all those per-key
+directories use disk space, add to the size of the git repo, and slow down
+traversal). However, git-annex relies on those directories to prevent
+accidential rm -rf deleting the annexed objects and prevent some symlink
+following programs from editing/corrupting the annexed objects (the
+per-key directories are left mode 400 most of the time). So it would be
+fairly complicated to add a tuning that eliminated those while locking
+down the permissions some other way (eg, making the `yy` directories mode
+400 except when one or more thread/process needs to write to them), and since
+it would have to be a tuning, it would introduce a lot of conditional
+complexity into the code.
+"""]]

add news item for git-annex 6.20170510
diff --git a/doc/news/version_6.20170214.mdwn b/doc/news/version_6.20170214.mdwn
deleted file mode 100644
index bc40e75..0000000
--- a/doc/news/version_6.20170214.mdwn
+++ /dev/null
@@ -1,56 +0,0 @@
-git-annex 6.20170214 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Increase default cost for p2p remotes from 200 to 1000.
-     This makes git-annex prefer transferring data from special
-     remotes when possible.
-   * Remove -j short option for --json-progress; that option was already
-     taken for --json.
-   * vicfg: Include the numcopies configuation.
-   * config: New command for storing configuration in the git-annex branch.
-   * annex.autocommit can be configured via git-annex config, to control
-     the default behavior in all clones of a repository.
-   * New annex.synccontent config setting, which can be set to true to make
-     git annex sync default to --content. This may become the default at
-     some point in the future. As well as being configuable by git config,
-     it can be configured by git-annex config to control the default
-     behavior in all clones of a repository.
-   * stack.yaml: Update to lts-7.18.
-   * Some optimisations to string splitting code.
-   * unused: When large files are checked right into git, avoid buffering
-     their contents in memory.
-   * unused: Improved memory use significantly when there are a lot
-     of differences between branches.
-   * Wormhole pairing will start to provide an appid to wormhole on
-     2021-12-31. An appid can't be provided now because Debian stable is going
-     to ship a older version of git-annex that does not provide an appid.
-     Assumption is that by 2021-12-31, this version of git-annex will be
-     shipped in a Debian stable release. If that turns out to not be the
-     case, this change will need to be cherry-picked into the git-annex in
-     Debian stable, or its wormhole pairing will break.
-   * Fix build with aws 0.16. Thanks, aristidb.
-   * assistant: Make --autostart --foreground wait for the children it
-     starts. Before, the --foreground was ignored when autostarting.
-   * initremote: When a uuid= parameter is passed, use the specified
-     UUID for the new special remote, instead of generating a UUID.
-     This can be useful in some situations, eg when the same data can be
-     accessed via two different special remote backends.
-   * import: Changed how --deduplicate, --skip-duplicates, and
-     --clean-duplicates determine if a file is a duplicate.
-     Before, only content known to be present somewhere was considered
-     a duplicate. Now, any content that has been annexed before will be
-     considered a duplicate, even if all annexed copies of the data have
-     been lost.
-     Note that --clean-duplicates and --deduplicate still check
-     numcopies, so won't delete duplicate files unless there's an annexed
-     copy.
-   * import: --deduplicate and --skip-duplicates were implemented
-     inneficiently; they unncessarily hashed each file twice. They have
-     been improved to only hash once.
-   * import: Added --reinject-duplicates.
-   * Added git template directory to Linux standalone tarball and OSX
-     app bundle.
-   * Improve pid locking code to work on filesystems that don't support hard
-     links.
-   * S3: Fix check of uuid file stored in bucket, which was not working.
-   * Work around sqlite's incorrect handling of umask when creating
-     databases."""]]
\ No newline at end of file
diff --git a/doc/news/version_6.20170510.mdwn b/doc/news/version_6.20170510.mdwn
new file mode 100644
index 0000000..b22af5e
--- /dev/null
+++ b/doc/news/version_6.20170510.mdwn
@@ -0,0 +1,27 @@
+git-annex 6.20170510 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * When a http remote does not expose an annex.uuid config, only warn
+     about it once, not every time git-annex is run.
+   * multicast: New command, uses uftp to multicast annexed files, for eg
+     a classroom setting.
+   * Added remote.&lt;name&gt;.annex-push and remote.&lt;name&gt;.annex-pull
+     which can be useful to make remotes that don't get fully synced with
+     local changes.
+   * Disable git-annex's support for GIT\_SSH and GIT\_SSH\_COMMAND, unless
+     GIT\_ANNEX\_USE\_GIT\_SSH=1 is also set in the environment. This is
+     necessary because as feared, the extra -n parameter that git-annex
+     passes breaks uses of these environment variables that expect exactly
+     the parameters that git passes.
+   * enableremote: When enabling a non-special remote, param=value
+     parameters can't be used, so error out if any are provided.
+   * enableremote: Fix re-enabling of special remotes that have a git
+     url, so that eg, encryption key changes take effect. They were silently
+     ignored, a reversion introduced in 6.20160527.
+   * gcrypt: Support re-enabling to change eg, encryption parameters.
+     This was never supported before.
+   * git annex add -u now supported, analagous to git add -u
+   * version: Added "dependency versions" line.
+   * Keys marked as dead are now skipped by --all.
+   * annex.backend is the new name for what was annex.backends, and takes
+     a single key-value backend, rather than the unncessary and confusing
+     list. The old option still works if set."""]]
\ No newline at end of file

Added a comment
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_6_fa72056ee788ac0f2c15bb57d9876cf6._comment b/doc/forum/Lots_of_4k_symlinks/comment_6_fa72056ee788ac0f2c15bb57d9876cf6._comment
new file mode 100644
index 0000000..6e4c896
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_6_fa72056ee788ac0f2c15bb57d9876cf6._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="CandyAngel"
+ avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8"
+ subject="comment 6"
+ date="2017-05-10T12:45:59Z"
+ content="""
+Oops,
+
+    find -name .git -prune -o -type l -printf '%s\n' | awk '{sum+=$1} END {print sum/1024**3}'
+
+should have been
+
+    find -name .git -prune -o -type l -printf '%s\n' | awk '{sum+=$1} END {print sum/1024**2}'
+
+That'll teach me to prematurely copy it :P
+"""]]

Added a comment
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_5_ab3884748f3271a77ba3320f25d74414._comment b/doc/forum/Lots_of_4k_symlinks/comment_5_ab3884748f3271a77ba3320f25d74414._comment
new file mode 100644
index 0000000..c1f4289
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_5_ab3884748f3271a77ba3320f25d74414._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="CandyAngel"
+ avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8"
+ subject="comment 5"
+ date="2017-05-10T12:44:08Z"
+ content="""
+    $ find -name .git -prune -o -type l | wc -l
+    1034886
+
+Just over a million symlinks.. very convenient :)
+
+    $ find -name .git -prune -o -type l -printf '%s\n' | awk '{sum+=$1} END {print sum/1024**3}'
+    195.9 # 195MB actual size
+    $ find -name .git -prune -o -type l -print0 | du -ch --files0-from=- | tail -n1
+    4.0G    total # 4GB disk usage
+
+And in comparison to my earlier comment 2 weeks ago:
+
+    $ du -shc *-* | tail -n3
+    33M     fd79bbd4-d41e-4ea8-acc8-86437c5eed7c
+    33M     ffbd042e-f6d9-4450-9a57-8ed1086f587c
+    4.1G    total
+
+So directory inode sizes are dwarfed by the 4K disk usage but ~198b actual usage of the symlinks (~96% wasted space?).
+"""]]

Added a comment
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_4_be12d26936b3502445e880be997b8877._comment b/doc/forum/Lots_of_4k_symlinks/comment_4_be12d26936b3502445e880be997b8877._comment
new file mode 100644
index 0000000..8737c0c
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_4_be12d26936b3502445e880be997b8877._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="CandyAngel"
+ avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8"
+ subject="comment 4"
+ date="2017-05-10T09:21:34Z"
+ content="""
+> And I doubt CandyAngel was counting only the sizes of symlinks and not git repos or at least directory inodes to hold all the symlinks.)
+
+In that repository, it is only top level directories (no sub directories) and each directory in it only has symlinks (up to 8000 of them). Directories are **mkdir $(uuidgen -r)**, hence the wildcard for du.
+
+It would be including the directory size to hold all the inodes, but it definitely *isn't counting .git* as this annex spans 3 drives with 6TB of content so far. Well, 6 drives because of \"numcopies 2\" :P
+
+I will calculate this a different way and only count symlinks, when I have access to it again.
+"""]]

diff --git a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn
index e5138d5..be87682 100644
--- a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn
+++ b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn
@@ -48,6 +48,10 @@ If it would help I can share the git-annex commands I've been using, but as I'm
 I've put some details about my thoughts on the repositories and restrictions below.
 
 
+Thanks - Olaf
+
+
+
 
 Repository 1
 ------------

diff --git a/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn
new file mode 100644
index 0000000..e5138d5
--- /dev/null
+++ b/doc/forum/Get___39__source__39___group_to_automatically_drop_with_assistant.mdwn
@@ -0,0 +1,84 @@
+Hi,
+
+I'm trying to get my head around groups, wanted, etc. for a particular use case.
+
+**Problem:** I can't work out how to get a source(?) repository to automatically drop files when they hit a transfer repository.
+
+
+I have a machine (`Machine 1`) that is used for data acquisition but it is behind a strict firewall (both physical and virtual).  I usually physically carry a USB drive over, set up a rsync ssh -> local-USB-drive from the one machine (`Machine 2`) that is able to connect over the network to `Machine 1`.  As it is a pain to lug the drive over, I only do this rsync maybe weekly, so the rsync takes many hours (~24) to complete.  Then (when I remember) I visit and I carry the USB drive back...  Naturally, this slows down my work process.
+
+What I was hoping to do was set up git-annex with the assistant to help me.  I am able to run the assistant, but not the webapp on `Machines 1 and 2`.  :-(
+
+My thought was - as these have to be disconnected network transfers...
+
+- `Repository 1 -> Repository 2` (when space permits)
+- `Repository 2 -> Repository 3` (when space permits) `-> Repository 4` (USB drive(s))
+
+
+Another limitation is that `Repos/Machines 2 & 3` have limited storage space.
+
+
+As a test case I can set up (`Repo1 -> Repo2`) and (`Repo2 -> Repo3`) (on other machines, but the commands should be the same...)
+
+After reading a bit I made a changed [preferred content](/preferred_content/standard_groups/) for a transfer repo to:
+
+```
+not (inallgroup=client and copies=client:1) and ($client)
+```
+
+i.e. `copies` from `2` to `1`.
+ 
+---
+
+
+Finally...The question
+----------------------
+
+**BUT** I can't work out how to get `Repo1` (the source) to automatically drop the files when they hit `Repo2` (what I'm guessing should be a transfer repository).
+
+Can anyone suggest how to automagically do this with the assistant?
+
+
+---
+
+
+If it would help I can share the git-annex commands I've been using, but as I'm only doing testing up at the moment, I'm happy to start from scratch if there is a RTFM page out there.  :-)
+
+
+I've put some details about my thoughts on the repositories and restrictions below.
+
+
+
+Repository 1
+------------
+- Type: source (Data collection)
+- Human readable directory structure
+- Physically: Machine 1
+- Strict firewall only incoming network connections from Machine 2
+- Storage: 50Gb
+
+
+Repository 2
+------------
+- Type: transfer
+- Physically: Machine 2
+- Reasonably relaxed firewall, can talk to Repository 3
+- Limited storage: 10Gb
+
+
+Repository 3
+------------
+- Type: transfer
+- Pysically: Machine 3
+- Reasonably relaxed firewall, can talk to Repository 2
+- Limited storage: 10Gb
+- Connected to USB drive(s)
+
+
+Repository 4, 5, ...
+--------------------
+- Type: ? Client ?
+- Human readable directory structure
+- Physically: USB drive
+- Usually (but not always) connected to machine 3
+- Large storage (2Tb) + Additional drives

Added a comment
diff --git a/doc/forum/sneakernet_with_a___34__directory__34___special_remote_on_fat/comment_3_02eae41ba9ad2f2fb15cbd20069bf1da._comment b/doc/forum/sneakernet_with_a___34__directory__34___special_remote_on_fat/comment_3_02eae41ba9ad2f2fb15cbd20069bf1da._comment
new file mode 100644
index 0000000..f980358
--- /dev/null
+++ b/doc/forum/sneakernet_with_a___34__directory__34___special_remote_on_fat/comment_3_02eae41ba9ad2f2fb15cbd20069bf1da._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="https://launchpad.net/~barthelemy"
+ nickname="barthelemy"
+ avatar="http://cdn.libravatar.org/avatar/e99cb15f6029de3225721b3ebdd0233905eb69698e9b229a8c4cc510a4135438"
+ subject="comment 3"
+ date="2017-05-09T23:38:27Z"
+ content="""
+Hi Joel,
+thank you for the precision (and for git annex, and for all the rest!)
+Cheers
+"""]]

hmm
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_3_80b3a84e6d79871bbbef13d4bc87cc37._comment b/doc/forum/Lots_of_4k_symlinks/comment_3_80b3a84e6d79871bbbef13d4bc87cc37._comment
new file mode 100644
index 0000000..d350dc2
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_3_80b3a84e6d79871bbbef13d4bc87cc37._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-05-09T22:49:01Z"
+ content="""
+My analysis above assumes no subdirectories.
+
+To leave space for even a single "../" would need to drop to 13 bytes
+of hash state. 1/79228162514264337593543950336 chance of 2 files colliding.
+Not comfortable with something so worse than md5, and that still doesn't help
+when files are 2 directories deep. Droping to 11 bytes for that,
+1/1208925819614629174706176 chance is starting to get into could really
+happen territory.
+"""]]

update
diff --git a/doc/forum/Lots_of_4k_symlinks/comment_2_be9617e8cbc231069c44bc9f077ce673._comment b/doc/forum/Lots_of_4k_symlinks/comment_2_be9617e8cbc231069c44bc9f077ce673._comment
new file mode 100644
index 0000000..1b2d91e
--- /dev/null
+++ b/doc/forum/Lots_of_4k_symlinks/comment_2_be9617e8cbc231069c44bc9f077ce673._comment
@@ -0,0 +1,32 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-05-09T21:07:11Z"
+ content="""
+You also get better seek speed with packed inodes.
+
+With default 256 byte inodes, there seems to be 59 bytes to play with.
+(Determined experimentally.) 
+
+Note that disks over 4 tb default to 32 kilobyte inodes, so probably most
+spinning hard disks these days *do* pack regular git-annex symlinks
+efficiently. (I don't have a 4 tb disk online to check this.. And I doubt
+CandyAngel was counting only the sizes of symlinks and not git repos
+or at least directory inodes to hold all the symlinks.)
+
+With a prefix like ".git/annex/objects/zX/Wx/S-s1000000000-"
+that leaves 20 bytes out of the 59 for the hash.
+
+That's not enough data to be cryptographically secure, but if
+we use SHA1 or MD5 as the base hash, it wouldn't be anyway. 15 bytes
+of hash state will base64 encode to 20 bytes. SHA1 is a 20 byte hash;
+MD5 is a 16 byte hash. So even MD5 would need to be truncated a little bit.
+Chances of (non-malicious) collision would still be small, only 256
+times as likely as a (non-malicious) MD5 collision. It could easily be made
+harder than MD5/SHA1 to maliciously collide by using truncated SHA2.
+
+(Files larger than 9.3 gb would still have too long symlinks due to the size
+field. The size field could also be omitted or encoded more efficiently,
+but omitting it would reduce git-annex's ability to not overfill disk
+and I don't think re-encoding buys enough to bother.)
+"""]]

Added a comment
diff --git a/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_2_c2846031bd79fa60cd903fb4d5bcebaf._comment b/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_2_c2846031bd79fa60cd903fb4d5bcebaf._comment
new file mode 100644
index 0000000..7a381bf
--- /dev/null
+++ b/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_2_c2846031bd79fa60cd903fb4d5bcebaf._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="supernaught"
+ avatar="http://cdn.libravatar.org/avatar/55f92a50f2617099e2dc7509130ce158"
+ subject="comment 2"
+ date="2017-05-09T22:07:59Z"
+ content="""
+Your solution is reasonable. Simpler is better.
+
+Thanks.
+"""]]

comment
diff --git a/doc/forum/Malicious_autoenabled_remotes/comment_1_c82464f40eb1442aadd0f1e89b72f41e._comment b/doc/forum/Malicious_autoenabled_remotes/comment_1_c82464f40eb1442aadd0f1e89b72f41e._comment
new file mode 100644
index 0000000..0a1d230
--- /dev/null
+++ b/doc/forum/Malicious_autoenabled_remotes/comment_1_c82464f40eb1442aadd0f1e89b72f41e._comment
@@ -0,0 +1,42 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-05-09T20:45:17Z"
+ content="""
+`git-annex init` will try to auto-enable special remotes that have
+been configured with autoenable=true.
+
+So if someone can push to the repository on trustedserver, they can
+set up such a special remote and cause your later clones of it to enable
+the special remote. Sync will then push content to their special remote.
+They could also check in additional annexed file to the git repository,
+and put their contents on their special remote, and sync would then
+download the contents from there.
+
+Of course, someone who can do this has to have write access to the
+git repository on trustedserver, and if they can write to the git repository,
+they can also send annexed file to there, unless you've prevented that
+somehow.
+
+I had not really considered the autoenable=true as a potential security
+problem, so it's good to think about it that way. I don't know if we have a
+real security problem here though. It seems to rely on the attacker
+having write access to the trustedserver so far.
+
+I suppose the attacker could instead convince you to pull from a clone that
+they control, and after you've pulled once, clones made from your
+repository (or trustedserver after you push to there) will then autoenable
+their special remote unexpectedly. Perhaps the goal then is to get git
+annex sync to unexpectedly send file contents there, so they can collect
+all your annexed files. Pulling from their repository once thus turns into
+sending them all your annexed files going forward.
+
+So I am starting to see this as a security problem.. 
+
+Note that pulling from someone untrusted can also change other settings in
+the git-annex branch (since it's automatically merged), which can probably
+screw up the repository fairly well in other ways, like setting numcopies
+to 0 and messing with preferred content expressions such that git-annex
+wants to drop all files, or copy files to repositories where you don't want
+them to go, etc.
+"""]]

annex.backend is the new name for what was annex.backends
It takes a single key-value backend, rather than the unncessary and confusing list.
The old option still works if set.
Simplified some old old code too.
This commit was sponsored by Thomas Hochstein on Patreon.
diff --git a/Annex.hs b/Annex.hs
index 95709fa..2a372f1 100644
--- a/Annex.hs
+++ b/Annex.hs
@@ -98,7 +98,7 @@ data AnnexState = AnnexState
 	{ repo :: Git.Repo
 	, repoadjustment :: (Git.Repo -> IO Git.Repo)
 	, gitconfig :: GitConfig
-	, backends :: [BackendA Annex]
+	, backend :: Maybe (BackendA Annex)
 	, remotes :: [Types.Remote.RemoteA Annex]
 	, remoteannexstate :: M.Map UUID AnnexState
 	, output :: MessageState
@@ -149,7 +149,7 @@ newState c r = do
 		{ repo = r
 		, repoadjustment = return
 		, gitconfig = c
-		, backends = []
+		, backend = Nothing
 		, remotes = []
 		, remoteannexstate = M.empty
 		, output = def
diff --git a/Backend.hs b/Backend.hs
index da7d9f6..c39141f 100644
--- a/Backend.hs
+++ b/Backend.hs
@@ -7,7 +7,7 @@
 
 module Backend (
 	list,
-	orderedList,
+	defaultBackend,
 	genKey,
 	getBackend,
 	chooseBackend,
@@ -33,40 +33,29 @@ import qualified Data.Map as M
 list :: [Backend]
 list = Backend.Hash.backends ++ Backend.WORM.backends ++ Backend.URL.backends
 
-{- List of backends in the order to try them when storing a new key. -}
-orderedList :: Annex [Backend]
-orderedList = do
-	l <- Annex.getState Annex.backends -- list is cached here
-	if not $ null l
-		then return l
-		else do
-			f <- Annex.getState Annex.forcebackend
-			case f of
-				Just name | not (null name) ->
-					return [lookupname name]
-				_ -> do
-					l' <- gen . annexBackends <$> Annex.getGitConfig
-					Annex.changeState $ \s -> s { Annex.backends = l' }
-					return l'
+{- Backend to use by default when generating a new key. -}
+defaultBackend :: Annex Backend
+defaultBackend = maybe cache return =<< Annex.getState Annex.backend
   where
-	gen [] = list
-	gen ns = map lookupname ns
+	cache = do
+		n <- maybe (annexBackend <$> Annex.getGitConfig) (return . Just)
+			=<< Annex.getState Annex.forcebackend
+		let b = case n of
+			Just name | valid name -> lookupname name
+			_ -> Prelude.head list
+		Annex.changeState $ \s -> s { Annex.backend = Just b }
+		return b
+	valid name = not (null name)
 	lookupname = lookupBackendVariety . parseKeyVariety
 
-{- Generates a key for a file, trying each backend in turn until one
- - accepts it. -}
+{- Generates a key for a file. -}
 genKey :: KeySource -> Maybe Backend -> Annex (Maybe (Key, Backend))
-genKey source trybackend = do
-	bs <- orderedList
-	let bs' = maybe bs (: bs) trybackend
-	genKey' bs' source
-genKey' :: [Backend] -> KeySource -> Annex (Maybe (Key, Backend))
-genKey' [] _ = return Nothing
-genKey' (b:bs) source = do
+genKey source preferredbackend = do
+	b <- maybe defaultBackend return preferredbackend
 	r <- B.getKey b source
-	case r of
-		Nothing -> genKey' bs source
-		Just k -> return $ Just (makesane k, b)
+	return $ case r of
+		Nothing -> Nothing
+		Just k -> Just (makesane k, b)
   where
 	-- keyNames should not contain newline characters.
 	makesane k = k { keyName = map fixbadchar (keyName k) }
@@ -82,13 +71,14 @@ getBackend file k = case maybeLookupBackendVariety (keyVariety k) of
 		return Nothing
 
 {- Looks up the backend that should be used for a file.
- - That can be configured on a per-file basis in the gitattributes file. -}
+ - That can be configured on a per-file basis in the gitattributes file,
+ - or forced with --backend. -}
 chooseBackend :: FilePath -> Annex (Maybe Backend)
 chooseBackend f = Annex.getState Annex.forcebackend >>= go
   where
-	go Nothing =  maybeLookupBackendVariety . parseKeyVariety
+	go Nothing = maybeLookupBackendVariety . parseKeyVariety
 		<$> checkAttr "annex.backend" f
-	go (Just _) = Just . Prelude.head <$> orderedList
+	go (Just _) = Just <$> defaultBackend
 
 {- Looks up a backend by variety. May fail if unsupported or disabled. -}
 lookupBackendVariety :: KeyVariety -> Backend
diff --git a/CHANGELOG b/CHANGELOG
index c1d9dc7..2bb2d08 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -22,6 +22,9 @@ git-annex (6.20170322) UNRELEASED; urgency=medium
   * git annex add -u now supported, analagous to git add -u
   * version: Added "dependency versions" line.
   * Keys marked as dead are now skipped by --all.
+  * annex.backend is the new name for what was annex.backends, and takes
+    a single key-value backend, rather than the unncessary and confusing
+    list. The old option still works if set.
 
  -- Joey Hess <id@joeyh.name>  Wed, 29 Mar 2017 12:41:46 -0400
 
diff --git a/Command/Migrate.hs b/Command/Migrate.hs
index 8dfee98..8924b17 100644
--- a/Command/Migrate.hs
+++ b/Command/Migrate.hs
@@ -36,15 +36,13 @@ start file key = do
 		Nothing -> stop
 		Just oldbackend -> do
 			exists <- inAnnex key
-			newbackend <- choosebackend =<< chooseBackend file
+			newbackend <- maybe defaultBackend return 
+				=<< chooseBackend file
 			if (newbackend /= oldbackend || upgradableKey oldbackend key || forced) && exists
 				then do
 					showStart "migrate" file
 					next $ perform file key oldbackend newbackend
 				else stop
-  where
-	choosebackend Nothing = Prelude.head <$> orderedList
-	choosebackend (Just backend) = return backend
 
 {- Checks if a key is upgradable to a newer representation.
  - 
diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs
index da548d4..7140380 100644
--- a/Types/GitConfig.hs
+++ b/Types/GitConfig.hs
@@ -48,7 +48,7 @@ data GitConfig = GitConfig
 	, annexNumCopies :: Maybe NumCopies
 	, annexDiskReserve :: Integer
 	, annexDirect :: Bool
-	, annexBackends :: [String]
+	, annexBackend :: Maybe String
 	, annexQueueSize :: Maybe Int
 	, annexBloomCapacity :: Maybe Int
 	, annexBloomAccuracy :: Maybe Int
@@ -98,7 +98,12 @@ extractGitConfig r = GitConfig
 	, annexDiskReserve = fromMaybe onemegabyte $
 		readSize dataUnits =<< getmaybe (annex "diskreserve")
 	, annexDirect = getbool (annex "direct") False
-	, annexBackends = getwords (annex "backends")
+	, annexBackend = maybe
+		-- annex.backends is the old name of the option, still used
+		-- when annex.backend is not set.
+		(headMaybe $ getwords (annex "backends"))
+		Just
+		(getmaybe (annex "backend"))
 	, annexQueueSize = getmayberead (annex "queuesize")
 	, annexBloomCapacity = getmayberead (annex "bloomcapacity")
 	, annexBloomAccuracy = getmayberead (annex "bloomaccuracy")
diff --git a/doc/backends.mdwn b/doc/backends.mdwn
index f69f655..c5b047e 100644
--- a/doc/backends.mdwn
+++ b/doc/backends.mdwn
@@ -45,9 +45,8 @@ Note that the various 512 and 384 length hashes result in long paths,
 which are known to not work on Windows. If interoperability on Windows is a
 concern, avoid those.
 
-The `annex.backends` git-config setting can be used to list the backends
-git-annex should use when adding new files. The first one listed will
-be used.
+The `annex.backend` git-config setting can be used to configure the
+default backend to use when adding new files.
 
 For finer control of what backend is used when adding different types of
 files, the `.gitattributes` file can be used. The `annex.backend`
diff --git a/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_1_5730767d5247e8997b4fa5b20c4cb281._comment b/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_1_5730767d5247e8997b4fa5b20c4cb281._comment
new file mode 100644
index 0000000..1b6ebca
--- /dev/null
+++ b/doc/forum/Can_git-annex-import_--clean-duplicates_honour_multiple_backends__63__/comment_1_5730767d5247e8997b4fa5b20c4cb281._comment
@@ -0,0 +1,35 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""

(Diff truncated)