Recent changes to this wiki:

Added a comment: Thank you
diff --git a/doc/git-annex-adjust/comment_3_c5af3a73af8925839413467a7d1e0e14._comment b/doc/git-annex-adjust/comment_3_c5af3a73af8925839413467a7d1e0e14._comment
new file mode 100644
index 000000000..9e0cfbb34
--- /dev/null
+++ b/doc/git-annex-adjust/comment_3_c5af3a73af8925839413467a7d1e0e14._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="mario"
+ avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
+ subject="Thank you"
+ date="2020-01-23T19:52:47Z"
+ content="""
+This is is a great feature, especially `--hide-missing`! I really missed this in the past. (Strangely it took me until now to notice that you implemented it.) Thank you.
+"""]]

Added a comment
diff --git a/doc/bugs/2_ssh_connection_prompts_for_password/comment_5_27b5669cecba4863ce296049804e82af._comment b/doc/bugs/2_ssh_connection_prompts_for_password/comment_5_27b5669cecba4863ce296049804e82af._comment
new file mode 100644
index 000000000..10f405fd1
--- /dev/null
+++ b/doc/bugs/2_ssh_connection_prompts_for_password/comment_5_27b5669cecba4863ce296049804e82af._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 5"
+ date="2020-01-23T17:51:58Z"
+ content="""
+Thank you Joey!  I can only confirm that the file system was likely a crippled/NFS one... So we would likely need to do some sensing on DataLad side and instruct git-annex. Will continue on our end at https://github.com/datalad/datalad/issues/4075
+"""]]

Added a comment: thanks
diff --git a/doc/devblog/day_617__remote_config_parsing_continued/comment_1_d3ab2e85ea775cc6f39f12210374c35f._comment b/doc/devblog/day_617__remote_config_parsing_continued/comment_1_d3ab2e85ea775cc6f39f12210374c35f._comment
new file mode 100644
index 000000000..541bb6487
--- /dev/null
+++ b/doc/devblog/day_617__remote_config_parsing_continued/comment_1_d3ab2e85ea775cc6f39f12210374c35f._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="thanks"
+ date="2020-01-23T16:51:44Z"
+ content="""
+\"the only user-visible improvement is these error messages\" -- FWIW, I've been bitten by the lack of config param checking in the past (thought I had set a chunk size but didn't due to misspelled param name, had to re-create the remote.)
+"""]]

Display a warning when concurrency is enabled but ssh connection caching is not enabled or won't work due to a crippled filesystem
A warning message is unsatisfying. But erroring out is too hard a failure,
especially since it may well work fine if the user has enabled passwordless
ssh.
I did think about falling back to one ssh connection at a time in this
case, but it would have needed a rework of every ssh call, which
seems far overboard for such a niche problem. There's no single place where
git-annex runs ssh, so no one place that it could block a concurrent call
on a semaphore. And, even if it did fall back to one ssh connection at a
time, it seems to me that doing so without warning the user about the
problem just invites bug reports like "git-annex is ignoring my -J2 and
only doing one download at a time". So a warning is needed, and I suppose
is good enough.
diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs
index 9fea51a92..7d3d99a57 100644
--- a/Annex/Ssh.hs
+++ b/Annex/Ssh.hs
@@ -1,6 +1,6 @@
 {- git-annex ssh interface, with connection caching
  -
- - Copyright 2012-2017 Joey Hess <id@joeyh.name>
+ - Copyright 2012-2020 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -100,11 +100,44 @@ consumeStdinParams NoConsumeStdin = [Param "-n"]
 sshCachingInfo :: (SshHost, Maybe Integer) -> Annex (Maybe FilePath, [CommandParam])
 sshCachingInfo (host, port) = go =<< sshCacheDir
   where
-	go Nothing = return (Nothing, [])
 	go (Just dir) =
 		liftIO (bestSocketPath $ dir </> hostport2socket host port) >>= return . \case
 			Nothing -> (Nothing, [])
 			Just socketfile -> (Just socketfile, sshConnectionCachingParams socketfile)
+	-- No connection caching with concurrency is not a good
+	-- combination, so warn the user.
+	go Nothing = do
+		Annex.getState Annex.concurrency >>= \case
+                	NonConcurrent -> return ()
+			Concurrent {} -> warnnocaching
+			ConcurrentPerCpu -> warnnocaching
+		return (Nothing, [])
+	
+	warnnocaching = do
+		warning nocachingwarning
+		ifM (fromMaybe True . annexSshCaching <$> Annex.getGitConfig)
+			( whenM crippledFileSystem $
+				warning crippledfswarning
+			, warning enablecachingwarning
+			)
+	
+	nocachingwarning = unwords
+		[ "You have enabled concurrency, but ssh connection caching"
+		, "is not enabled. This may result in multiple ssh processes"
+		, "prompting for passwords at the same time."
+		]
+	
+	crippledfswarning = unwords
+		[ "This repository is on a crippled filesystem, so unix named"
+		, "pipes probably don't work, and ssh connection caching"
+		, "relies on those. One workaround is to set"
+		, sshSocketDirEnv
+		, "to point to a directory on a non-crippled filesystem."
+		, "(Or, disable concurrency.)"
+		]
+	
+	enablecachingwarning = 
+		"Enable annex.sshcaching (or disable concurrency) to avoid this problem."
 
 {- Given an absolute path to use for a socket file,
  - returns whichever is shorter of that or the relative path to the same
@@ -133,9 +166,11 @@ sshConnectionCachingParams socketfile =
 	, Param "-o", Param "ControlPersist=yes"
 	]
 
+sshSocketDirEnv :: String
+sshSocketDirEnv = "GIT_ANNEX_SSH_SOCKET_DIR"
+
 {- ssh connection caching creates sockets, so will not work on a
- - crippled filesystem. A GIT_ANNEX_TMP_DIR can be provided to use
- - a different filesystem. -}
+ - crippled filesystem. -}
 sshCacheDir :: Annex (Maybe FilePath)
 sshCacheDir
 	| BuildInfo.sshconnectioncaching = 
@@ -148,7 +183,7 @@ sshCacheDir
 			)
 	| otherwise = return Nothing
   where
-	gettmpdir = liftIO $ getEnv "GIT_ANNEX_TMP_DIR"
+	gettmpdir = liftIO $ getEnv sshSocketDirEnv
 	usetmpdir tmpdir = liftIO $ catchMaybeIO $ do
 		let socktmp = tmpdir </> "ssh"
 		createDirectoryIfMissing True socktmp
diff --git a/CHANGELOG b/CHANGELOG
index c36aec282..e9b1c25b2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -19,6 +19,8 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
     uuid configured.
   * Support git remotes that need http basic auth to be accessed,
     using git credential to get the password.
+  * Display a warning when concurrency is enabled but ssh connection caching
+    is not enabled or won't work due to a crippled filesystem.
 
  -- Joey Hess <id@joeyh.name>  Wed, 01 Jan 2020 12:51:40 -0400
 
diff --git a/doc/bugs/2_ssh_connection_prompts_for_password.mdwn b/doc/bugs/2_ssh_connection_prompts_for_password.mdwn
index 7d718fbc9..633c8a3b6 100644
--- a/doc/bugs/2_ssh_connection_prompts_for_password.mdwn
+++ b/doc/bugs/2_ssh_connection_prompts_for_password.mdwn
@@ -63,3 +63,4 @@ get R042/R042-2013-08-16/R042-2013-08-16-CSC01a.ncs get R042/R042-2013-08-16/R04
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
 
+> warning added; [[done]] --[[Joey]] 
diff --git a/doc/bugs/2_ssh_connection_prompts_for_password/comment_4_43e5a90d6215b414f4e3357ea91af663._comment b/doc/bugs/2_ssh_connection_prompts_for_password/comment_4_43e5a90d6215b414f4e3357ea91af663._comment
new file mode 100644
index 000000000..7ae63772c
--- /dev/null
+++ b/doc/bugs/2_ssh_connection_prompts_for_password/comment_4_43e5a90d6215b414f4e3357ea91af663._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2020-01-23T15:51:46Z"
+ content="""
+I notice that debug output has no BatchMode=true in any ssh call. But
+the version of git-annex you show always runs ssh with that when
+-J is used, unless sshcaching is disabled.
+
+More evidence that sshcaching is disabled in your transcript is that when
+it does run ssh, it does not pass -S.
+
+I think the repository must be on a crippled filesystem, on which
+git-annex can't do ssh connection caching, because the filesystem
+does not support unix sockets. (Or it potentially could be crippled in some
+other way.) So it ignores the annex.sshcaching setting.
+You could work around this by setting the (undocumented)
+GIT_ANNEX_TMP_DIR to some temporary directory on a non-crippled filesystem.
+
+I'm going to add a warning message in this situation.
+"""]]
diff --git a/standalone/linux/skel/runshell b/standalone/linux/skel/runshell
index 0c3919919..7ef160135 100755
--- a/standalone/linux/skel/runshell
+++ b/standalone/linux/skel/runshell
@@ -212,8 +212,8 @@ case "$os" in
 		useproot=1
 		
 		# Store ssh connection caching sockets outside of sdcard.
-		GIT_ANNEX_TMP_DIR="$TMPDIR"
-		export GIT_ANNEX_TMP_DIR
+		GIT_ANNEX_SSH_SOCKET_DIR="$TMPDIR"
+		export GIT_ANNEX_SSH_SOCKET_DIR
 
 		GIT_ANNEX_STANDLONE_ENV="PATH GCONV_PATH MANPATH LOCPATH"
 		export GIT_ANNEX_STANDLONE_ENV

close as fixed in DAV library
diff --git a/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__.mdwn b/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__.mdwn
index 083f1351e..3c34b3f9b 100644
--- a/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__.mdwn
+++ b/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__.mdwn
@@ -42,3 +42,6 @@ apparently it is actually timing out on checking (I guess after chunk completion
 
 
 [[!meta author="yoh"]]
+
+> I see that the bug in the DAV library has been fixed (in 2018),
+> so hopefully nothing more needs to be done. [[done]] --[[Joey]]

support git remotes that need http basic auth
using git credential to get the password
One thing this doesn't do is wrap the password prompting inside the prompt
action. So with -J, the output can be a bit garbled.
diff --git a/Annex/Content.hs b/Annex/Content.hs
index 7c57cf504..961551366 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -780,13 +780,12 @@ saveState nocommit = doSideAction $ do
 
 {- Downloads content from any of a list of urls, displaying a progress
  - meter. -}
-downloadUrl :: Key -> MeterUpdate -> [Url.URLString] -> FilePath -> Annex Bool
-downloadUrl k p urls file = 
+downloadUrl :: Key -> MeterUpdate -> [Url.URLString] -> FilePath -> Url.UrlOptions -> Annex Bool
+downloadUrl k p urls file uo = 
 	-- Poll the file to handle configurations where an external
 	-- download command is used.
 	meteredFile file (Just p) k $
-		Url.withUrlOptions $ \uo -> 
-			anyM (\u -> Url.download p u file uo) urls
+		anyM (\u -> Url.download p u file uo) urls
 
 {- Copies a key's content, when present, to a temp file.
  - This is used to speed up some rsyncs. -}
diff --git a/Annex/Url.hs b/Annex/Url.hs
index bcc6a747f..08c3e3bf7 100644
--- a/Annex/Url.hs
+++ b/Annex/Url.hs
@@ -1,13 +1,14 @@
 {- Url downloading, with git-annex user agent and configured http
  - headers, security restrictions, etc.
  -
- - Copyright 2013-2019 Joey Hess <id@joeyh.name>
+ - Copyright 2013-2020 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
 
 module Annex.Url (
 	withUrlOptions,
+	withUrlOptionsPromptingCreds,
 	getUrlOptions,
 	getUserAgent,
 	ipAddressesUnlimited,
@@ -34,6 +35,7 @@ import qualified Utility.Url as U
 import Utility.IPAddress
 import Utility.HttpManagerRestricted
 import Utility.Metered
+import Git.Credential
 import qualified BuildInfo
 
 import Network.Socket
@@ -64,6 +66,7 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 			<*> pure urldownloader
 			<*> pure manager
 			<*> (annexAllowedUrlSchemes <$> Annex.getGitConfig)
+			<*> pure U.noBasicAuth
 	
 	headers = annexHttpHeadersCommand <$> Annex.getGitConfig >>= \case
 		Just cmd -> lines <$> liftIO (readProcess "sh" ["-c", cmd])
@@ -124,6 +127,21 @@ ipAddressesUnlimited =
 withUrlOptions :: (U.UrlOptions -> Annex a) -> Annex a
 withUrlOptions a = a =<< getUrlOptions
 
+-- When downloading an url, if authentication is needed, uses
+-- git-credential to prompt for username and password.
+withUrlOptionsPromptingCreds :: (U.UrlOptions -> Annex a) -> Annex a
+withUrlOptionsPromptingCreds a = do
+	g <- Annex.gitRepo
+	uo <- getUrlOptions
+	a $ uo
+		{ U.getBasicAuth = getBasicAuthFromCredential g
+		-- Can't download with curl and handle basic auth,
+		-- so avoid using curl.
+		, U.urlDownloader = case U.urlDownloader uo of
+			U.DownloadWithCurl _ -> U.DownloadWithConduit $ U.DownloadWithCurlRestricted mempty
+			v -> v
+		}
+
 checkBoth :: U.URLString -> Maybe Integer -> U.UrlOptions -> Annex Bool
 checkBoth url expected_size uo =
 	liftIO (U.checkBoth url expected_size uo) >>= \case
diff --git a/CHANGELOG b/CHANGELOG
index b80dcfe83..c36aec282 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -17,6 +17,8 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
     programs that use GETCONFIG/SETCONFIG are recommended to implement it.
   * init: Avoid an ugly error message when http remote has no git-annex
     uuid configured.
+  * Support git remotes that need http basic auth to be accessed,
+    using git credential to get the password.
 
  -- Joey Hess <id@joeyh.name>  Wed, 01 Jan 2020 12:51:40 -0400
 
diff --git a/Command/AddUrl.hs b/Command/AddUrl.hs
index 4fb03f6b0..f728eb175 100644
--- a/Command/AddUrl.hs
+++ b/Command/AddUrl.hs
@@ -261,7 +261,7 @@ downloadWeb addunlockedmatcher o url urlinfo file =
 	go =<< downloadWith' downloader urlkey webUUID url (AssociatedFile (Just (toRawFilePath file)))
   where
 	urlkey = addSizeUrlKey urlinfo $ Backend.URL.fromUrl url Nothing
-	downloader f p = downloadUrl urlkey p [url] f
+	downloader f p = Url.withUrlOptions $ downloadUrl urlkey p [url] f
 	go Nothing = return Nothing
 	-- If we downloaded a html file, try to use youtube-dl to
 	-- extract embedded media.
diff --git a/Git/Credential.hs b/Git/Credential.hs
index 5de95d1a3..9465d2796 100644
--- a/Git/Credential.hs
+++ b/Git/Credential.hs
@@ -1,6 +1,6 @@
 {- git credential interface
  -
- - Copyright 2019 Joey Hess <id@joeyh.name>
+ - Copyright 2019-2020 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -22,6 +22,23 @@ credentialUsername = M.lookup "username" . fromCredential
 credentialPassword :: Credential -> Maybe String
 credentialPassword = M.lookup "password" . fromCredential
 
+credentialBasicAuth :: Credential -> Maybe BasicAuth
+credentialBasicAuth cred = BasicAuth
+	<$> credentialUsername cred
+	<*> credentialPassword cred
+
+getBasicAuthFromCredential :: Repo -> GetBasicAuth
+getBasicAuthFromCredential r u = do
+	c <- getUrlCredential u r
+	case credentialBasicAuth c of
+		Just ba -> return $ Just (ba, signalsuccess c)
+		Nothing -> do
+			signalsuccess c False
+			return Nothing
+  where
+	signalsuccess c True = approveUrlCredential c r
+	signalsuccess c False = rejectUrlCredential c r
+
 -- | This may prompt the user for login information, or get cached login
 -- information.
 getUrlCredential :: URLString -> Repo -> IO Credential
diff --git a/Remote/External.hs b/Remote/External.hs
index cf3676cce..750e6ed77 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -747,7 +747,7 @@ checkUrlM external url =
 retrieveUrl :: Retriever
 retrieveUrl = fileRetriever $ \f k p -> do
 	us <- getWebUrls k
-	unlessM (downloadUrl k p us f) $
+	unlessM (withUrlOptions $ downloadUrl k p us f) $
 		giveup "failed to download content"
 
 checkKeyUrl :: Git.Repo -> CheckPresent
diff --git a/Remote/Git.hs b/Remote/Git.hs
index eb66215ab..ea07c0e4c 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -276,7 +276,7 @@ tryGitConfigRead autoinit r
 				warning $ "Unable to parse git config from " ++ configloc
 				return $ Left l
 
-	geturlconfig = Url.withUrlOptions $ \uo -> do
+	geturlconfig = Url.withUrlOptionsPromptingCreds $ \uo -> do
 		v <- withTmpFile "git-annex.tmp" $ \tmpfile h -> do
 			liftIO $ hClose h
 			let url = Git.repoLocation r ++ "/config"
@@ -382,7 +382,7 @@ inAnnex' repo rmt (State connpool duc _ _) key
 	checkhttp = do
 		showChecking repo
 		gc <- Annex.getGitConfig
-		ifM (Url.withUrlOptions $ \uo -> anyM (\u -> Url.checkBoth u (fromKey keySize key) uo) (keyUrls gc repo rmt key))
+		ifM (Url.withUrlOptionsPromptingCreds $ \uo -> anyM (\u -> Url.checkBoth u (fromKey keySize key) uo) (keyUrls gc repo rmt key))
 			( return True
 			, giveup "not found"
 			)
@@ -514,7 +514,8 @@ copyFromRemote'' :: Git.Repo -> Bool -> Remote -> State -> Key -> AssociatedFile
 copyFromRemote'' repo forcersync r st@(State connpool _ _ _) key file dest meterupdate
 	| Git.repoIsHttp repo = unVerified $ do
 		gc <- Annex.getGitConfig
-		Annex.Content.downloadUrl key meterupdate (keyUrls gc repo r key) dest
+		Url.withUrlOptionsPromptingCreds $
+			Annex.Content.downloadUrl key meterupdate (keyUrls gc repo r key) dest
 	| not $ Git.repoIsUrl repo = guardUsable repo (unVerified (return False)) $ do
 		params <- Ssh.rsyncParams r Download
 		u <- getUUID
diff --git a/Remote/GitLFS.hs b/Remote/GitLFS.hs
index f400d0dd9..2f91d1c77 100644
--- a/Remote/GitLFS.hs
+++ b/Remote/GitLFS.hs
@@ -37,6 +37,7 @@ import Crypto
 import Backend.Hash
 import Utility.Hash
 import Utility.SshHost
+import Utility.Url
 import Logs.Remote
 import Logs.RemoteState
 import qualified Utility.GitLFS as LFS
@@ -283,7 +284,7 @@ discoverLFSEndpoint tro h
 				if needauth (responseStatus resp)

(Diff truncated)
closed the wrong bug
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
index 89935343b..63c691cd6 100644
--- a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
@@ -145,6 +145,3 @@ git annex 7.20190819+git2-g908476a9b-1~ndall+1 and the same with bleeding edge 7
 [[!meta author=yoh]]
 [[!tag projects/dandi]]
 
-> [[fixed|done]] what I consider to be the core issue in this bug report. 
-> --[[Joey]]
-
diff --git a/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages.mdwn b/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages.mdwn
index 4c4fca407..57bb645ee 100644
--- a/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages.mdwn
+++ b/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages.mdwn
@@ -54,3 +54,7 @@ PS ignore "ignores --json-error-messages" part of the subject -- was detected in
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+> [[fixed|done]] what I consider to be the core issue in this bug report. 
+> --[[Joey]]
+

better error message when git config fails to parse remote config
Rather than leaking the name of the temp file, just say the config parse
failed, and where the config was downloaded from.
Not closing the bug report because two issues were reported in the same
bug report, because the universe wants me to continually re-read old
unclosed bug reports to waste my time determining what still needs to be
done.
diff --git a/Git/Config.hs b/Git/Config.hs
index 4b606641d..f88028623 100644
--- a/Git/Config.hs
+++ b/Git/Config.hs
@@ -1,6 +1,6 @@
 {- git repository configuration handling
  -
- - Copyright 2010-2019 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2020 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -14,6 +14,7 @@ import qualified Data.ByteString as S
 import qualified Data.ByteString.Char8 as S8
 import Data.Char
 import qualified System.FilePath.ByteString as P
+import Control.Concurrent.Async
 
 import Common
 import Git
@@ -184,19 +185,22 @@ coreBare = "core.bare"
 
 {- Runs a command to get the configuration of a repo,
  - and returns a repo populated with the configuration, as well as the raw
- - output of the command. -}
-fromPipe :: Repo -> String -> [CommandParam] -> IO (Either SomeException (Repo, S.ByteString))
+ - output and any standard output of the command. -}
+fromPipe :: Repo -> String -> [CommandParam] -> IO (Either SomeException (Repo, S.ByteString, S.ByteString))
 fromPipe r cmd params = try $
-	withHandle StdoutHandle createProcessSuccess p $ \h -> do
-		val <- S.hGetContents h
+	withOEHandles createProcessSuccess p $ \(hout, herr) -> do
+		geterr <- async $ S.hGetContents herr
+		getval <- async $ S.hGetContents hout
+		val <- wait getval
+		err <- wait geterr
 		r' <- store val r
-		return (r', val)
+		return (r', val, err)
   where
 	p = proc cmd $ toCommand params
 
 {- Reads git config from a specified file and returns the repo populated
  - with the configuration. -}
-fromFile :: Repo -> FilePath -> IO (Either SomeException (Repo, S.ByteString))
+fromFile :: Repo -> FilePath -> IO (Either SomeException (Repo, S.ByteString, S.ByteString))
 fromFile r f = fromPipe r "git"
 	[ Param "config"
 	, Param "--file"
diff --git a/Remote/GCrypt.hs b/Remote/GCrypt.hs
index d35ffc9b8..3f1907009 100644
--- a/Remote/GCrypt.hs
+++ b/Remote/GCrypt.hs
@@ -56,6 +56,7 @@ import Utility.Tmp
 import Logs.Remote
 import Utility.Gpg
 import Utility.SshHost
+import Utility.Tuple
 import Messages.Progress
 import Types.ProposedAccepted
 
@@ -471,7 +472,7 @@ getGCryptId :: Bool -> Git.Repo -> RemoteGitConfig -> Annex (Maybe Git.GCrypt.GC
 getGCryptId fast r gc
 	| Git.repoIsLocal r || Git.repoIsLocalUnknown r = extract <$>
 		liftIO (catchMaybeIO $ Git.Config.read r)
-	| not fast = extract . liftM fst <$> getM (eitherToMaybe <$>)
+	| not fast = extract . liftM fst3 <$> getM (eitherToMaybe <$>)
 		[ Ssh.onRemote NoConsumeStdin r (\f p -> liftIO (Git.Config.fromPipe r f p), return (Left $ error "configlist failed")) "configlist" [] []
 		, getConfigViaRsync r gc
 		]
@@ -480,7 +481,7 @@ getGCryptId fast r gc
 	extract Nothing = (Nothing, r)
 	extract (Just r') = (fromConfigValue <$> Git.Config.getMaybe coreGCryptId r', r')
 
-getConfigViaRsync :: Git.Repo -> RemoteGitConfig -> Annex (Either SomeException (Git.Repo, S.ByteString))
+getConfigViaRsync :: Git.Repo -> RemoteGitConfig -> Annex (Either SomeException (Git.Repo, S.ByteString, S.ByteString))
 getConfigViaRsync r gc = do
 	(rsynctransport, rsyncurl, _) <- rsyncTransport r gc
 	opts <- rsynctransport
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 05eaae1fc..eb66215ab 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -248,7 +248,7 @@ tryGitConfigRead autoinit r
 	| haveconfig r = return r -- already read
 	| Git.repoIsSsh r = storeUpdatedRemote $ do
 		v <- Ssh.onRemote NoConsumeStdin r
-			(pipedconfig autoinit, return (Left $ giveup "configlist failed"))
+			(pipedconfig autoinit (Git.repoDescribe r), return (Left $ giveup "configlist failed"))
 			"configlist" [] configlistfields
 		case v of
 			Right r'
@@ -263,23 +263,25 @@ tryGitConfigRead autoinit r
   where
 	haveconfig = not . M.null . Git.config
 
-	pipedconfig mustincludeuuuid cmd params = do
+	pipedconfig mustincludeuuuid configloc cmd params = do
 		v <- liftIO $ Git.Config.fromPipe r cmd params
 		case v of
-			Right (r', val) -> do
+			Right (r', val, _err) -> do
 				unless (isUUIDConfigured r' || S.null val || not mustincludeuuuid) $ do
 					warning $ "Failed to get annex.uuid configuration of repository " ++ Git.repoDescribe r
 					warning $ "Instead, got: " ++ show val
 					warning $ "This is unexpected; please check the network transport!"
 				return $ Right r'
-			Left l -> return $ Left l
+			Left l -> do
+				warning $ "Unable to parse git config from " ++ configloc
+				return $ Left l
 
 	geturlconfig = Url.withUrlOptions $ \uo -> do
 		v <- withTmpFile "git-annex.tmp" $ \tmpfile h -> do
 			liftIO $ hClose h
 			let url = Git.repoLocation r ++ "/config"
 			ifM (liftIO $ Url.downloadQuiet nullMeterUpdate url tmpfile uo)
-				( Just <$> pipedconfig False "git" [Param "config", Param "--null", Param "--list", Param "--file", File tmpfile]
+				( Just <$> pipedconfig False url "git" [Param "config", Param "--null", Param "--list", Param "--file", File tmpfile]
 				, return Nothing
 				)
 		case v of
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
index e01378722..89935343b 100644
--- a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
@@ -1,3 +1,5 @@
+[[!meta title="http remotes that require authentication are not yet supported"]]
+
 It is not a ground shaking issue, but probably would be best to handle it more gracefully.
 
 Initially mentioned while doing install using datalad. Account/permission is required to access this particular repo, ask Canadians for access if you don't have it yet Joey.  credentials I guess got asked for and cached by git upon initial invocation, so upon subsequent calls didn't ask for any:
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_5_495820d777c889f88466d725c11895f9._comment b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_5_495820d777c889f88466d725c11895f9._comment
new file mode 100644
index 000000000..a82a557c4
--- /dev/null
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_5_495820d777c889f88466d725c11895f9._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2020-01-22T17:04:09Z"
+ content="""
+Error message has been improved.
+"""]]

comment
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_3_3df07b6c8ef25ad96ff4f198931347ad._comment b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_3_3df07b6c8ef25ad96ff4f198931347ad._comment
new file mode 100644
index 000000000..7be8cac1f
--- /dev/null
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_3_3df07b6c8ef25ad96ff4f198931347ad._comment
@@ -0,0 +1,31 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2020-01-22T16:04:37Z"
+ content="""
+git-annex could use `git credential` if the config download fails with
+401 unauthorized and then retry with the credentials. (The git-lfs special
+remote already does this.) And it would also need to do the same thing
+when getting a key from the remote.
+
+But that would not help with the https://git.bic.mni.mcgill.ca example,
+apparently, because there's no 401, but a 302 redirect to a 200,
+that is indistingishable from a successful download. 
+
+Yeah, when git-annex expects a git config, if it doesn't parse as one,
+it could retry, asking for credentials.
+But that seems asking for trouble: what if it fails to parse for
+another reason, maybe the web server served up something other than the
+expected config, maybe a captive portal got in the way. There would be a
+username/password prompt that doesn't make sense to the user at all.
+
+And if this happens in a key download, git-annex certianly has no way to
+tell that what it downloaded is not intended as the content of a key,
+short of verifying the content, and failure to verify certainly doesn't
+justify prompting for a username/password.
+
+So, I am not comfortable with falling back to ask for credentials unless
+I've seen a http status code that indicates they are necessary.
+And IMHO gitlab's use of a 302 redirect to a login page is a bug in
+gitlab, and will need to be fixed there, or a better http server used.
+"""]]
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_4_051d22f0bd0eb601476cd3b74a9b98b8._comment b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_4_051d22f0bd0eb601476cd3b74a9b98b8._comment
new file mode 100644
index 000000000..956baeb59
--- /dev/null
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/comment_4_051d22f0bd0eb601476cd3b74a9b98b8._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""re: related: shouldn't git annex try external remotes to download config?"""
+ date="2020-01-22T16:31:16Z"
+ content="""
+No, the external special remote protocol is not aimed at downloading git
+config files. Anyway, this code path is never involved with using 
+special remotes; the uuid of a special remote is known and so there is no
+need to ever download a git config file to discover it.
+"""]]

avoid ugly error message
Http remotes that do expose a git config file, but are not initialized
resulted in an ugly and unncessary error message, now sqelched.
When git-annex-shell configlist is run w/o the autoinit field, it may
not generate a uuid for the repository. So in that case, it's not
unexpected for the config it does list to not include a UUID, and
dumping out the config in a warning message is not needed.
If configlist is asked to autoinit and we don't get back a config with a
UUID in it, that suggests some problem, and what we got back may not be
a config at all but some diagnostic message, so it does make sense to
output it then.
diff --git a/CHANGELOG b/CHANGELOG
index 4b58b9992..b80dcfe83 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -15,6 +15,8 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
     git annex initremote type=directory encryption=none --whatelse
   * Added LISTCONFIGS to external special remote protocol. Special remote
     programs that use GETCONFIG/SETCONFIG are recommended to implement it.
+  * init: Avoid an ugly error message when http remote has no git-annex
+    uuid configured.
 
  -- Joey Hess <id@joeyh.name>  Wed, 01 Jan 2020 12:51:40 -0400
 
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 0139df94b..05eaae1fc 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -248,7 +248,7 @@ tryGitConfigRead autoinit r
 	| haveconfig r = return r -- already read
 	| Git.repoIsSsh r = storeUpdatedRemote $ do
 		v <- Ssh.onRemote NoConsumeStdin r
-			(pipedconfig, return (Left $ giveup "configlist failed"))
+			(pipedconfig autoinit, return (Left $ giveup "configlist failed"))
 			"configlist" [] configlistfields
 		case v of
 			Right r'
@@ -263,11 +263,11 @@ tryGitConfigRead autoinit r
   where
 	haveconfig = not . M.null . Git.config
 
-	pipedconfig cmd params = do
+	pipedconfig mustincludeuuuid cmd params = do
 		v <- liftIO $ Git.Config.fromPipe r cmd params
 		case v of
 			Right (r', val) -> do
-				unless (isUUIDConfigured r' || S.null val) $ do
+				unless (isUUIDConfigured r' || S.null val || not mustincludeuuuid) $ do
 					warning $ "Failed to get annex.uuid configuration of repository " ++ Git.repoDescribe r
 					warning $ "Instead, got: " ++ show val
 					warning $ "This is unexpected; please check the network transport!"
@@ -279,7 +279,7 @@ tryGitConfigRead autoinit r
 			liftIO $ hClose h
 			let url = Git.repoLocation r ++ "/config"
 			ifM (liftIO $ Url.downloadQuiet nullMeterUpdate url tmpfile uo)
-				( Just <$> pipedconfig "git" [Param "config", Param "--null", Param "--list", Param "--file", File tmpfile]
+				( Just <$> pipedconfig False "git" [Param "config", Param "--null", Param "--list", Param "--file", File tmpfile]
 				, return Nothing
 				)
 		case v of
diff --git a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
index d93c3623a..e01378722 100644
--- a/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
+++ b/doc/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file.mdwn
@@ -143,4 +143,6 @@ git annex 7.20190819+git2-g908476a9b-1~ndall+1 and the same with bleeding edge 7
 [[!meta author=yoh]]
 [[!tag projects/dandi]]
 
+> [[fixed|done]] what I consider to be the core issue in this bug report. 
+> --[[Joey]]
 
diff --git a/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages/comment_1_a7ad806ceb76f13ff8fdcce62bc0ed8a._comment b/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages/comment_1_a7ad806ceb76f13ff8fdcce62bc0ed8a._comment
new file mode 100644
index 000000000..2cbbcac88
--- /dev/null
+++ b/doc/bugs/on_some_remotes_failing_to_detect_annex_spits_out_message_to_stderr_and_empty_lines_to_stderr__44___ignores_--json-error-messages/comment_1_a7ad806ceb76f13ff8fdcce62bc0ed8a._comment
@@ -0,0 +1,41 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-22T15:11:05Z"
+ content="""
+I've said this before, but I'll say it one more time: --json-error-messages
+is not a guarantee that every possible error message that may be output by
+git-annex in some exceptional circumstance will be formatted as json.
+
+In this case, while I happened to make it be captured in an unrelated
+change, there's actually no benefit to it being captured. If it were no
+longer captured tomorrow, I would not consider that a bug. This error
+message is not specific to a particular file in the repository, so if
+git-annex get outputs it, it doesn't help for the error message to be
+wrapped up in json. The actual purpose of --json-error messages is being
+able to correlate a failure to eg, get a particular file with an error
+message related to that action. Not in avoiding all possible stderr.
+
+----
+
+The extra newlines output to stdout are there because the `warning` action
+does not know if something may have been output to stdout earlier without a
+terminating newline, and it wants to avoid an ugly interleave of stdout and
+stderr. While state could be maintained to keep track of that, the end
+result would be git-annex would become some milliseconds slower, and it
+does not seem worth the complexity or minor speed hit to cater to the case
+where stderr is /dev/nulled. Note that this doesn't happen when using
+--json. Also, IIRC it's avoided when using concurrent output, which does
+pay the time/complexity overhead already to keep track of the state of the
+display to that extent. Anyway, I'm obviosuly not going to leave this bug
+report open for such a minor and tangential issue after the main issue in
+it is fixed, so it's kind of annoying to need to write this wall of text
+about it. May I suggest one bug report per distinct issue is a good way to
+avoid my current annoyed state?
+
+----
+
+Not wanting to sit down and write all this is why, the previous two or
+three times I opened this issue, I promptly closed the window rather than
+addressing any part of it.
+"""]]

New bug for fish-completion
diff --git a/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn b/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn
new file mode 100644
index 000000000..9c12d5a8f
--- /dev/null
+++ b/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn
@@ -0,0 +1,37 @@
+### Please describe the problem.
+
+[git-annex Makefile: install-completions](http://source.git-annex.branchable.com/?p=source.git;a=blob;f=Makefile;h=965f53e1fc4a8f6d69041eabaccd759268f6490f;hb=HEAD#l87)
+
+git-annex installs fish completions to the wrong directory. `$(SHAREDIR)/fish/completions` is the directory documented as being exclusive to completions which are shipped in the fish source code; third-party applications installing their own completions are intended to use `$(SHAREDIR)/fish/vendor_completions.d` instead.
+
+See [https://fishshell.com/docs/current/index.html#completion-path](https://fishshell.com/docs/current/index.html#completion-path)
+
+Note that this location can also be obtained in a similar manner to bash-completion:
+
+```
+$ pkg-config bash-completion --variable=completionsdir
+/usr/share/bash-completion/completions
+```
+
+```
+$ pkg-config fish --variable=completionsdir
+/usr/share/fish/vendor_completions.d
+```
+
+### What steps will reproduce the problem?
+
+Run "make install-completions", or install a linux distribution package of git-annex that builds with the current Makefile (Arch Linux or Debian will both show the same issue).
+
+### What version of git-annex are you using? On what operating system?
+
+Arch Linux
+
+git-annex 7.20191230-7
+
+### Please provide any additional information below.
+
+Apparently this is a very common mistake :/ so far I've seen many more projects do this wrong than do it right.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Not a user, just here to help improve cross-distro packaging. :)

Added a comment
diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment
new file mode 100644
index 000000000..337e648c0
--- /dev/null
+++ b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="dxld"
+ avatar="http://cdn.libravatar.org/avatar/742547a848e15c9f7fb381191c239141"
+ subject="comment 2"
+ date="2020-01-21T19:28:29Z"
+ content="""
+Honestly I feel like the (perceived) semantics of sync are broken by this behaviour. I would expect git-annex to do what it has to to make what I asked for happen.
+
+I agree that in general it's a good thing not to needlessly override git settings but for the sync command I really don't see any way that not merging can be considered sensible behaviour. To me as a user it just feels like I changed a setting completely unrelated to git-annex-sync and suddenly sync broke.
+
+Consider this: the git-annex-sync(1) man page never actually mentions that it will run git-merge. On the other hand git-pull(1) is very forthcoming with the fact that it's just a shorthand for `git fetch; git merge` so it's obvious to me that settings affecting merge will affect git-pull, not so for sync.
+
+I've been unable to sync my git-annex repos for a couple of months now because of this issue so firmly believe this is a serious usabiliy issue.
+
+At the very least we have a documentation issue here. Though I would still argue the behaviour is bonkers :)
+
+"""]]

comment
diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment
new file mode 100644
index 000000000..a5c94ec3a
--- /dev/null
+++ b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-21T18:31:22Z"
+ content="""
+I feel it's right for git-annex sync to honor git configs, so it's right
+for it to not merge origin/master. And, without that merge, it's right for
+it to fail to push master to origin. Since it does push synced/master, this
+does not prevent other clones of the repo, where git-annex sync is later
+ran, from getting the changes made by this sync.
+
+That leaves only this ugly thing:
+
+fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree.
+
+Which comes from Git.Branch.changed, but I'm not clear how the fast forward
+configuration would prevent either of those refs from existing.
+"""]]

diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn b/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn
new file mode 100644
index 000000000..0d0e488ef
--- /dev/null
+++ b/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn
@@ -0,0 +1,447 @@
+### Please describe the problem.
+
+Having globally configured git-merge to only allow fast forward merges breaks git-annex's sync command. 
+
+### What steps will reproduce the problem?
+
+Run the following script, beware it will change the global git config for `merge.ff`:
+
+```
+#!/bin/sh
+repro() {
+        rm -rf repo-a repo-b
+        mkdir repo-a repo-b
+        ( cd repo-a; git init; git annex init; git remote add repo-b ../repo-b )
+        ( cd repo-b; git init; git annex init; git remote add repo-a ../repo-a )
+
+        # Setup an initial commit a0 in repo-a
+        ( cd repo-a
+          touch a0
+          git add a0
+          git commit -m a0
+          git annex sync
+        )
+
+        # Pull a0 into repo-b and create commit 'b' on top of it
+        ( cd repo-b
+          git annex sync
+          touch b
+          git add b
+          git commit -m b
+        )
+
+        # Back in repo-a create a diverging commit 'a1' and try to sync
+        ( cd repo-a
+          touch a1
+          git add a1
+          git commit -m a1
+          git annex sync -d
+        )
+}
+
+# First try without merge.ff=only
+git config --global --unset merge.ff
+repro; rv=$?
+
+# Now with
+git config --global merge.ff only
+repro || echo "===== Breaks with merge.ff=only ====="
+[ $rv -eq 0 ] && echo "===== Works without merge.ff=only ====="
+```
+
+### What version of git-annex are you using? On what operating system?
+
+git-annex version 7.20190129-3 from Debian buster
+
+```
+$ git annex version
+git-annex version: 7.20190129
+build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
+operating system: linux x86_64
+supported repository versions: 5 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+```
+
+### Please provide any additional information below.
+
+A run of the repro script above with `sh -x`:
+
+[[!format sh """
++ git config --global --unset merge.ff
++ repro
++ rm -rf repo-a repo-b
++ mkdir repo-a repo-b
++ cd repo-a
++ git init
+Initialized empty Git repository in /tmp/repo-a/.git/
++ git annex init
+init  ok
+(recording state in git...)
++ git remote add repo-b ../repo-b
++ cd repo-b
++ git init
+Initialized empty Git repository in /tmp/repo-b/.git/
++ git annex init
+init  ok
+(recording state in git...)
++ git remote add repo-a ../repo-a
++ cd repo-a
++ touch a0
++ git add a0
++ git commit -m a0
+[master (root-commit) 6c82d47] a0
+ 1 file changed, 0 insertions(+), 0 deletions(-)
+ create mode 100644 a0
++ git annex sync
+commit 
+On branch master
+nothing to commit, working tree clean
+ok
+pull repo-b 
+remote: Enumerating objects: 4, done.
+remote: Counting objects: 100% (4/4), done.
+remote: Compressing objects: 100% (2/2), done.
+remote: Total 3 (delta 0), reused 0 (delta 0)
+Unpacking objects: 100% (3/3), done.
+From ../repo-b
+ * [new branch]      git-annex  -> repo-b/git-annex
+ok
+(merging repo-b/git-annex into git-annex...)
+(recording state in git...)
+push repo-b 
+Enumerating objects: 12, done.
+Counting objects: 100% (12/12), done.
+Delta compression using up to 8 threads
+Compressing objects: 100% (5/5), done.
+Writing objects: 100% (9/9), 805 bytes | 805.00 KiB/s, done.
+Total 9 (delta 1), reused 0 (delta 0)
+To ../repo-b
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+ok
++ cd repo-b
++ git annex sync
+commit 
+On branch master
+
+Initial commit
+
+nothing to commit
+ok
+fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree.
+Use '--' to separate paths from revisions, like this:
+'git <command> [<revision>...] -- [<file>...]'
+pull repo-a 
+From ../repo-a
+ * [new branch]      git-annex     -> repo-a/git-annex
+ * [new branch]      master        -> repo-a/master
+ * [new branch]      synced/master -> repo-a/synced/master
+
+
+Already up to date.
+ok
++ touch b
++ git add b
++ git commit -m b
+[master d3ecbbf] b
+ 1 file changed, 0 insertions(+), 0 deletions(-)
+ create mode 100644 b
++ cd repo-a
++ touch a1
++ git add a1
++ git commit -m a1
+[master 7c4054b] a1
+ 1 file changed, 0 insertions(+), 0 deletions(-)
+ create mode 100644 a1
++ git annex sync -d
+[2020-01-21 12:41:12.808810342] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
+[2020-01-21 12:41:12.810537111] process done ExitSuccess
+[2020-01-21 12:41:12.810623305] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
+[2020-01-21 12:41:12.812254921] process done ExitSuccess
+[2020-01-21 12:41:12.812457308] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..a2ceb59a5879cdf88bd33104c0ee8ac390b3f62e","--pretty=%H","-n1"]
+[2020-01-21 12:41:12.813939135] process done ExitSuccess
+[2020-01-21 12:41:12.814207234] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
+[2020-01-21 12:41:12.814533549] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
+[2020-01-21 12:41:12.820975511] read: git ["config","--null","--list"]
+[2020-01-21 12:41:12.822586611] process done ExitSuccess
+commit 
+[2020-01-21 12:41:12.823337737] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in dxld@Eli:/tmp/repo-a"]
+On branch master
+nothing to commit, working tree clean
+[2020-01-21 12:41:12.851712964] process done ExitFailure 1
+ok
+[2020-01-21 12:41:12.851809058] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
+[2020-01-21 12:41:12.852844432] process done ExitSuccess
+[2020-01-21 12:41:12.85289997] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
+[2020-01-21 12:41:12.853966619] process done ExitSuccess
+[2020-01-21 12:41:12.854022004] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"]
+[2020-01-21 12:41:12.85488393] process done ExitSuccess
+[2020-01-21 12:41:12.854935336] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"]
+[2020-01-21 12:41:12.85636602] process done ExitSuccess
+pull repo-b 
+[2020-01-21 12:41:12.85658479] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","repo-b"]
+remote: Enumerating objects: 3, done.
+remote: Counting objects: 100% (3/3), done.
+remote: Compressing objects: 100% (2/2), done.
+remote: Total 2 (delta 0), reused 0 (delta 0)
+Unpacking objects: 100% (2/2), done.
+From ../repo-b
+ * [new branch]      master     -> repo-b/master
+[2020-01-21 12:41:12.869425035] process done ExitSuccess
+[2020-01-21 12:41:12.86951948] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"]

(Diff truncated)
--whatelse is a better name than --describe-other-params
The use case is basically the user having forgotten, so --help would be
best, but it would be quite hard to include this in --help, since it may
even have to spin up an external special remote program.
I also considered --umm but typoed it the first time I tried it as
--uum, and while memorable, it's too cutesy. --whatelse is good because
it explicitly asks, what other params, besides the ones I've given?
diff --git a/CHANGELOG b/CHANGELOG
index 1654fbc1e..4b58b9992 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,8 +10,9 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
     when foo=yes is expected.
   * initremote, enableremote: Reject unknown configuration parameters
     provided to these commands.
-  * initremote: Added --describe-other-params option, to show additional
-    configuration parameters you might want to set.
+  * initremote: Added --whatelse option, to show additional
+    configuration parameters you might want to set. Eg:
+    git annex initremote type=directory encryption=none --whatelse
   * Added LISTCONFIGS to external special remote protocol. Special remote
     programs that use GETCONFIG/SETCONFIG are recommended to implement it.
 
diff --git a/Command/InitRemote.hs b/Command/InitRemote.hs
index b4c431743..d601e39c5 100644
--- a/Command/InitRemote.hs
+++ b/Command/InitRemote.hs
@@ -33,7 +33,7 @@ cmd = command "initremote" SectionSetup
 data InitRemoteOptions = InitRemoteOptions
 	{ cmdparams :: CmdParams
 	, sameas :: Maybe (DeferredParse UUID)
-	, describeOtherParams :: Bool
+	, whatElse :: Bool
 	}
 
 optParser :: CmdParamsDesc -> Parser InitRemoteOptions
@@ -41,8 +41,8 @@ optParser desc = InitRemoteOptions
 	<$> cmdParams desc
 	<*> optional parseSameasOption
 	<*> switch
-		( long "describe-other-params"
-		<> short 'p'
+		( long "whatelse"
+		<> short 'w'
 		<> help "describe other configuration parameters for a special remote"
 		)
 
@@ -74,7 +74,7 @@ start o (name:ws) = ifM (isJust <$> findExisting name)
 					(Logs.Remote.keyValToConfig Proposed ws)
 					<$> readRemoteLog
 				t <- either giveup return (findType c)
-				if describeOtherParams o
+				if whatElse o
 					then startingCustomOutput (ActionItemOther Nothing) $
 						describeOtherParamsFor c t
 					else starting "initremote" (ActionItemOther (Just name)) $
diff --git a/doc/git-annex-initremote.mdwn b/doc/git-annex-initremote.mdwn
index 592139013..ca204eac7 100644
--- a/doc/git-annex-initremote.mdwn
+++ b/doc/git-annex-initremote.mdwn
@@ -21,7 +21,7 @@ The remote's configuration is specified by the parameters passed
 to this command. Different types of special remotes need different
 configuration values, so consult the documentation of a special remote for
 details. The command will prompt for any required parameters you leave out;
-you can also pass --describe-other-params to see additional parameters.
+you can also pass --whatelse to see additional parameters.
 
 A few parameters that are supported by all special remotes are documented in
 the next section below.
@@ -38,14 +38,14 @@ want to use `git annex renameremote`.
 
 # OPTIONS
 
-* `--describe-other-params` / `-p`
+* `--whatelse` / `-w`
 
   Describe additional configuration parameters that you could specify.
 
   For example, if you know you want a S3 remote, but forget how to
   configure it:
 
-	git annex initremote mys3 type=S3 --describe-other-params
+	git annex initremote mys3 type=S3 --whatelse
 
 * `--fast`
 
@@ -88,7 +88,7 @@ want to use `git annex renameremote`.
   
   For details about this and other encrpytion settings, see
   <https://git-annex.branchable.com/encryption/>
-  or --describe-other-params
+  or --whatelse
 
 * `autoenable`
 
diff --git a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
index 493e7430f..bcc9dc9ff 100644
--- a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
+++ b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
@@ -10,8 +10,8 @@ and indeed I get asked for e.g. encryption to be explicitly specified (why not t
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
 
-> [[done]] as eg `git annex initremote type=rsync --describe-other-params`.
+> [[done]] as eg `git annex initremote type=rsync --whatelse`.
 > 
 > External special remotes that implement LISTCONFIGS can also be queried:
 > 
-> 	git annex inittype type=external externaltype=foo --describe-other-params
+> 	git annex inittype type=external externaltype=foo --whatelse

include passthrough params in --describe-other-params
diff --git a/Annex/SpecialRemote/Config.hs b/Annex/SpecialRemote/Config.hs
index d9d24c2c5..d4f8da22a 100644
--- a/Annex/SpecialRemote/Config.hs
+++ b/Annex/SpecialRemote/Config.hs
@@ -189,7 +189,7 @@ parseRemoteConfig c rpc =
   where
 	go l c' [] =
 		let (passover, leftovers) = partition
-			(remoteConfigRestPassthrough rpc . fst)
+			(maybe (const False) fst (remoteConfigRestPassthrough rpc) . fst)
 			(M.toList c')
 		    leftovers' = filter (notaccepted . fst) leftovers
 		in if not (null leftovers')
diff --git a/Command/InitRemote.hs b/Command/InitRemote.hs
index f82946c7b..b4c431743 100644
--- a/Command/InitRemote.hs
+++ b/Command/InitRemote.hs
@@ -114,22 +114,22 @@ cleanup u name c o = do
 describeOtherParamsFor :: RemoteConfig -> RemoteType -> CommandPerform
 describeOtherParamsFor c t = do
 	cp <- R.configParser t c
-	let l = mapMaybe mk $ filter notinconfig $ remoteConfigFieldParsers cp
-	liftIO $ forM_ l $ \(p, pd, vd) -> do
-		putStrLn p
-		putStrLn ("\t" ++ pd)
-		case vd of
-			Nothing -> return ()
-			Just vd' -> putStrLn $ "\t(" ++ vd' ++ ")"
+	let l = map mk (filter notinconfig $ remoteConfigFieldParsers cp)
+		++ map mk' (maybe [] snd (remoteConfigRestPassthrough cp))
+	liftIO $ forM_ l $ \(p, fd, vd) -> case fd of
+		HiddenField -> return ()
+		FieldDesc d -> do
+			putStrLn p
+			putStrLn ("\t" ++ d)
+			case vd of
+				Nothing -> return ()
+				Just (ValueDesc d') ->
+					putStrLn $ "\t(" ++ d' ++ ")"
 	next $ return True
   where
 	notinconfig fp = not (M.member (parserForField fp) c)
-	mk fp = case fieldDesc fp of
-		HiddenField -> Nothing
-		FieldDesc d -> Just
-			( fromProposedAccepted (parserForField fp)
-			, d
-			, case valueDesc fp of
-				Nothing -> Nothing
-				Just (ValueDesc vd) -> Just vd
-			)
+	mk fp = ( fromProposedAccepted (parserForField fp)
+		, fieldDesc fp
+		, valueDesc fp
+		)
+	mk' (k, v) = (k, v, Nothing)
diff --git a/Remote/External.hs b/Remote/External.hs
index 2ea671258..5d54f68f0 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -792,7 +792,10 @@ lenientRemoteConfigParser = addRemoteConfigParser specialRemoteConfigParsers $
 			, trueFalseParser readonlyField False
 				(FieldDesc "enable readonly mode")
 			]
-		, remoteConfigRestPassthrough = const True
+		, remoteConfigRestPassthrough = Just
+			( const True
+			, [("*", FieldDesc "all other parameters are passed to external special remote program")]
+			)
 		}
 
 {- When the remote supports LISTCONFIGS, only accept the ones it listed.
@@ -804,14 +807,14 @@ strictRemoteConfigParser external = listConfigs external >>= \case
 		let s = S.fromList (map fst l)
 		let listed f = S.member (fromProposedAccepted f) s
 		return $ lenientRemoteConfigParser
-			{ remoteConfigRestPassthrough = listed }
+			{ remoteConfigRestPassthrough = Just (listed, l) }
 
-listConfigs :: External -> Annex (Maybe [(Setting, Description)])
+listConfigs :: External -> Annex (Maybe [(Setting, FieldDesc)])
 listConfigs external = handleRequest external LISTCONFIGS Nothing (collect [])
   where
 	collect l req = case req of
 		CONFIG s d -> Just $ return $
-			GetNextMessage $ collect ((s, d) : l)
+			GetNextMessage $ collect ((s, FieldDesc d) : l)
 		CONFIGEND -> result (Just (reverse l))
 		UNSUPPORTED_REQUEST -> result Nothing
 		_ -> Nothing
diff --git a/Remote/Helper/Encryptable.hs b/Remote/Helper/Encryptable.hs
index 9b6723bf1..33ef848bb 100644
--- a/Remote/Helper/Encryptable.hs
+++ b/Remote/Helper/Encryptable.hs
@@ -78,7 +78,7 @@ encryptionConfigs = S.fromList (map parserForField encryptionConfigParsers)
 parseEncryptionConfig :: RemoteConfig -> Either String ParsedRemoteConfig
 parseEncryptionConfig c = parseRemoteConfig
 	(M.restrictKeys c encryptionConfigs)
-	(RemoteConfigParser encryptionConfigParsers (const False))
+	(RemoteConfigParser encryptionConfigParsers Nothing)
 
 encryptionFieldParser :: RemoteConfigFieldParser
 encryptionFieldParser = RemoteConfigFieldParser
diff --git a/Remote/S3.hs b/Remote/S3.hs
index 0e0284637..8e925a3f9 100644
--- a/Remote/S3.hs
+++ b/Remote/S3.hs
@@ -102,8 +102,13 @@ remote = specialRemoteType $ RemoteType
 			, optionalStringParser mungekeysField HiddenField
 			, optionalStringParser AWS.s3credsField HiddenField
 			]
-		, remoteConfigRestPassthrough = \f ->
-			isMetaHeader f || isArchiveMetaHeader f
+		, remoteConfigRestPassthrough = Just
+			( \f -> isMetaHeader f || isArchiveMetaHeader f
+			,
+				[ ("x-amz-meta-*", FieldDesc "http headers to add when storing on S3")
+				, ("x-archive-meta-*", FieldDesc "http headers to add when storing on Internet Archive")
+			  	]
+			)
 		}
 	, setup = s3Setup
 	, exportSupported = exportIsSupported
diff --git a/Types/RemoteConfig.hs b/Types/RemoteConfig.hs
index 1f6c935be..f0df89a10 100644
--- a/Types/RemoteConfig.hs
+++ b/Types/RemoteConfig.hs
@@ -53,11 +53,11 @@ newtype ValueDesc = ValueDesc String
 
 data RemoteConfigParser = RemoteConfigParser
 	{ remoteConfigFieldParsers :: [RemoteConfigFieldParser]
-	, remoteConfigRestPassthrough :: RemoteConfigField -> Bool
+	, remoteConfigRestPassthrough :: Maybe (RemoteConfigField -> Bool, [(String, FieldDesc)])
 	}
 
 mkRemoteConfigParser :: Monad m => [RemoteConfigFieldParser] -> RemoteConfig -> m RemoteConfigParser
-mkRemoteConfigParser l _ = pure (RemoteConfigParser l (const False))
+mkRemoteConfigParser l _ = pure (RemoteConfigParser l Nothing)
 
 addRemoteConfigParser :: [RemoteConfigFieldParser] -> RemoteConfigParser -> RemoteConfigParser
 addRemoteConfigParser l rpc = rpc
diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn
index 2a1bb5729..6221dd955 100644
--- a/doc/special_remotes/S3.mdwn
+++ b/doc/special_remotes/S3.mdwn
@@ -140,4 +140,9 @@ the S3 remote.
   then use the same bucket.
 
 * `x-amz-meta-*` are passed through as http headers when storing keys
-  in S3. see [the Internet Archive S3 interface documentation](https://archive.org/help/abouts3.txt) for example headers.
+  in S3.
+
+* `x-archive-meta-*` are passed through as http headers when storing keys
+  in the Internet Archive. See
+  [the Internet Archive S3 interface documentation](https://archive.org/help/abouts3.txt)
+  for example headers.
diff --git a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
index 3e24a01c7..493e7430f 100644
--- a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
+++ b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
@@ -9,3 +9,9 @@ and indeed I get asked for e.g. encryption to be explicitly specified (why not t
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+> [[done]] as eg `git annex initremote type=rsync --describe-other-params`.
+> 
+> External special remotes that implement LISTCONFIGS can also be queried:
+> 
+> 	git annex inittype type=external externaltype=foo --describe-other-params

initremote --describe-other-params
Does not yet include descriptions from external special remote programs.
diff --git a/Annex/SpecialRemote.hs b/Annex/SpecialRemote.hs
index 4a40127a9..759b4701b 100644
--- a/Annex/SpecialRemote.hs
+++ b/Annex/SpecialRemote.hs
@@ -72,7 +72,10 @@ findType config = maybe unspecified (specified . fromProposedAccepted) $
   where
 	unspecified = Left "Specify the type of remote with type="
 	specified s = case filter (findtype s) remoteTypes of
-		[] -> Left $ "Unknown remote type " ++ s
+		[] -> Left $ "Unknown remote type " ++ s 
+			++ " (pick from: "
+			++ intercalate " " (map typename remoteTypes)
+			++ ")"
 		(t:_) -> Right t
 	findtype s i = typename i == s
 
diff --git a/CHANGELOG b/CHANGELOG
index 84689b4e7..1654fbc1e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,10 +6,12 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
   * reinject --known: Fix bug that prevented it from working in a bare repo.
   * Support being used in a git repository that uses sha256 rather than sha1.
   * initremote, enableremote: Be stricter about rejecting invalid
-    configurations for remotes, particularly things like foo=true when
-    foo=yes is expected
-  * initremote, enableremote: Reject unknown parameters provided to these
-    commands.
+    configuration parameters for remotes, particularly things like foo=true
+    when foo=yes is expected.
+  * initremote, enableremote: Reject unknown configuration parameters
+    provided to these commands.
+  * initremote: Added --describe-other-params option, to show additional
+    configuration parameters you might want to set.
   * Added LISTCONFIGS to external special remote protocol. Special remote
     programs that use GETCONFIG/SETCONFIG are recommended to implement it.
 
diff --git a/CmdLine/Usage.hs b/CmdLine/Usage.hs
index 27619a04a..cdbb77623 100644
--- a/CmdLine/Usage.hs
+++ b/CmdLine/Usage.hs
@@ -96,8 +96,8 @@ paramItem :: String
 paramItem = "ITEM"
 paramTreeish :: String
 paramTreeish = "TREEISH"
-paramKeyValue :: String
-paramKeyValue = "K=V"
+paramParamValue :: String
+paramParamValue = "PARAM=VALUE"
 paramNothing :: String
 paramNothing = ""
 paramRepeating :: String -> String
diff --git a/Command/EnableRemote.hs b/Command/EnableRemote.hs
index 9d5fc4b10..1bd9c80b3 100644
--- a/Command/EnableRemote.hs
+++ b/Command/EnableRemote.hs
@@ -31,7 +31,7 @@ import qualified Data.Map as M
 cmd :: Command
 cmd = command "enableremote" SectionSetup
 	"enables git-annex to use a remote"
-	(paramPair paramName $ paramOptional $ paramRepeating paramKeyValue)
+	(paramPair paramName $ paramOptional $ paramRepeating paramParamValue)
 	(withParams seek)
 
 seek :: CmdParams -> CommandSeek
diff --git a/Command/InitRemote.hs b/Command/InitRemote.hs
index 4dc6acd9e..f82946c7b 100644
--- a/Command/InitRemote.hs
+++ b/Command/InitRemote.hs
@@ -16,6 +16,7 @@ import Annex.SpecialRemote
 import qualified Remote
 import qualified Logs.Remote
 import qualified Types.Remote as R
+import Types.RemoteConfig
 import Annex.UUID
 import Logs.UUID
 import Logs.Remote
@@ -26,18 +27,24 @@ import Config
 cmd :: Command
 cmd = command "initremote" SectionSetup
 	"creates a special (non-git) remote"
-	(paramPair paramName $ paramOptional $ paramRepeating paramKeyValue)
+	(paramPair paramName $ paramOptional $ paramRepeating paramParamValue)
 	(seek <$$> optParser)
 
 data InitRemoteOptions = InitRemoteOptions
 	{ cmdparams :: CmdParams
 	, sameas :: Maybe (DeferredParse UUID)
+	, describeOtherParams :: Bool
 	}
 
 optParser :: CmdParamsDesc -> Parser InitRemoteOptions
 optParser desc = InitRemoteOptions
 	<$> cmdParams desc
 	<*> optional parseSameasOption
+	<*> switch
+		( long "describe-other-params"
+		<> short 'p'
+		<> help "describe other configuration parameters for a special remote"
+		)
 
 parseSameasOption :: Parser (DeferredParse UUID)
 parseSameasOption = parseUUIDOption <$> strOption
@@ -67,8 +74,11 @@ start o (name:ws) = ifM (isJust <$> findExisting name)
 					(Logs.Remote.keyValToConfig Proposed ws)
 					<$> readRemoteLog
 				t <- either giveup return (findType c)
-				starting "initremote" (ActionItemOther (Just name)) $
-					perform t name c o
+				if describeOtherParams o
+					then startingCustomOutput (ActionItemOther Nothing) $
+						describeOtherParamsFor c t
+					else starting "initremote" (ActionItemOther (Just name)) $
+						perform t name c o
 			)
 	)
 
@@ -100,3 +110,26 @@ cleanup u name c o = do
 			setConfig (remoteConfig c "config-uuid") (fromUUID cu)
 			Logs.Remote.configSet cu c
 	return True
+
+describeOtherParamsFor :: RemoteConfig -> RemoteType -> CommandPerform
+describeOtherParamsFor c t = do
+	cp <- R.configParser t c
+	let l = mapMaybe mk $ filter notinconfig $ remoteConfigFieldParsers cp
+	liftIO $ forM_ l $ \(p, pd, vd) -> do
+		putStrLn p
+		putStrLn ("\t" ++ pd)
+		case vd of
+			Nothing -> return ()
+			Just vd' -> putStrLn $ "\t(" ++ vd' ++ ")"
+	next $ return True
+  where
+	notinconfig fp = not (M.member (parserForField fp) c)
+	mk fp = case fieldDesc fp of
+		HiddenField -> Nothing
+		FieldDesc d -> Just
+			( fromProposedAccepted (parserForField fp)
+			, d
+			, case valueDesc fp of
+				Nothing -> Nothing
+				Just (ValueDesc vd) -> Just vd
+			)
diff --git a/doc/git-annex-initremote.mdwn b/doc/git-annex-initremote.mdwn
index e20dec565..592139013 100644
--- a/doc/git-annex-initremote.mdwn
+++ b/doc/git-annex-initremote.mdwn
@@ -20,15 +20,12 @@ For a list and details, see <https://git-annex.branchable.com/special_remotes/>
 The remote's configuration is specified by the parameters passed
 to this command. Different types of special remotes need different
 configuration values, so consult the documentation of a special remote for
-details. The command will prompt for any required parameters you leave out.
+details. The command will prompt for any required parameters you leave out;
+you can also pass --describe-other-params to see additional parameters.
+
 A few parameters that are supported by all special remotes are documented in
 the next section below.
 
-To list all the configuration parameters that are accepted by a particular
-type of special remote, along with short descriptions, run eg:
-
-	git annex initremote --list-params-for=S3
-
 Once a special remote has been initialized once with this command,
 other clones of the repository can also be set up to access it using
 `git annex enableremote`.
@@ -41,10 +38,14 @@ want to use `git annex renameremote`.
 
 # OPTIONS
 
-* `--list-params-for=type`
+* `--describe-other-params` / `-p`
+
+  Describe additional configuration parameters that you could specify.
+
+  For example, if you know you want a S3 remote, but forget how to
+  configure it:
 
-  Lists all configuration parameters that are accepted for a given
-  type of special remote.
+	git annex initremote mys3 type=S3 --describe-other-params
 
 * `--fast`
 
@@ -87,6 +88,7 @@ want to use `git annex renameremote`.
   
   For details about this and other encrpytion settings, see
   <https://git-annex.branchable.com/encryption/>
+  or --describe-other-params
 
 * `autoenable`
 

diff --git a/doc/bugs/git_annex_copy_fails_without_error_message_on_macOS___40__but_works_on_Linux__41__.mdwn b/doc/bugs/git_annex_copy_fails_without_error_message_on_macOS___40__but_works_on_Linux__41__.mdwn
new file mode 100644
index 000000000..86ebd3357
--- /dev/null
+++ b/doc/bugs/git_annex_copy_fails_without_error_message_on_macOS___40__but_works_on_Linux__41__.mdwn
@@ -0,0 +1,19 @@
+I have a git repository with a webdav annex remote. I have used this remote to store a few large files, and transfer them to other computers.
+
+I have done this on three Linux computers on different networks, and it works beautifully! Truly a life-saver!
+
+However, I today tried setting up a macOS computer, and 
+
+    git annex copy --from=myremote
+
+fails with the simple error message "failed". It downloads about ten megabytes without issue before failing. It then continues with the next file and again downloads a bit (ca. 10 Mb), then fails. The JSON error message has an empty error message field, and the JSON progress messages only contain business-as-usual before the error. 
+
+On the computer that has this problem, I am running macOS 10.15.2 and git-annex 7.20191230. 
+
+I tried WiFi and wired network connections, which doesn't make a difference. The hard drive has enough space for the files. I tried manually selecting various annexed files, but all fail after 10 Mb.
+
+The same command worked on the linux boxes I set up a few days ago.
+
+The problem really is that I am getting no meaningful error message for triaging this problem.
+
+Any help would be appreciated!

diff --git a/doc/users/timothe.mdwn b/doc/users/timothe.mdwn
new file mode 100644
index 000000000..3ddbfa54f
--- /dev/null
+++ b/doc/users/timothe.mdwn
@@ -0,0 +1,5 @@
+Coucou Mijhito !
+
+Oui, c'est bien ici que le brainstorm se passe !
+
+Tim

done
diff --git a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters.mdwn b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters.mdwn
index dd95be5d6..493c8e42d 100644
--- a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters.mdwn
+++ b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters.mdwn
@@ -14,3 +14,5 @@ At least for the built in special remotes (not external) this should be possible
 
 [[!meta author=yoh]]
 [[!tag projects/dandi]]
+
+> [[done]] --[[Joey]] 
diff --git a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_4_b031ee12622b1ed28b6ffc8c037f7d30._comment b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_4_b031ee12622b1ed28b6ffc8c037f7d30._comment
new file mode 100644
index 000000000..526b38902
--- /dev/null
+++ b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_4_b031ee12622b1ed28b6ffc8c037f7d30._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2020-01-17T21:15:16Z"
+ content="""
+Added LISTCONFIGS to external special remote protocol, and once your
+special remotes implement it, initremote will notice if the user provides
+any setting with the wrong name.
+
+(external special remotes could already verify the values of settings using
+GETCONFIG at the INITREMOTE stage, and use INITREMOTE-FAILURE to inform the
+user of bad or missing values)
+"""]]

add LISTCONFIGS to external special remote protocol
Special remote programs that use GETCONFIG/SETCONFIG are recommended
to implement it.
The description is not yet used, but will be useful later when adding a way
to make initremote list all accepted configs.
configParser now takes a RemoteConfig parameter. Normally, that's not
needed, because configParser returns a parter, it does not parse it
itself. But, it's needed to look at externaltype and work out what
external remote program to run for LISTCONFIGS.
Note that, while externalUUID is changed to a Maybe UUID, checkExportSupported
used to use NoUUID. The code that now checks for Nothing used to behave
in some undefined way if the external program made requests that
triggered it.
Also, note that in externalSetup, once it generates external,
it parses the RemoteConfig strictly. That generates a
ParsedRemoteConfig, which is thrown away. The reason it's ok to throw
that away, is that, if the strict parse succeeded, the result must be
the same as the earlier, lenient parse.
initremote of an external special remote now runs the program three
times. First for LISTCONFIGS, then EXPORTSUPPORTED, and again
LISTCONFIGS+INITREMOTE. It would not be hard to eliminate at least
one of those, and it should be possible to only run the program once.
diff --git a/Assistant/DaemonStatus.hs b/Assistant/DaemonStatus.hs
index cbf323196..4f3c2effb 100644
--- a/Assistant/DaemonStatus.hs
+++ b/Assistant/DaemonStatus.hs
@@ -19,7 +19,6 @@ import Logs.Trust
 import Utility.TimeStamp
 import qualified Remote
 import qualified Types.Remote as Remote
-import Config
 import Config.DynamicConfig
 import Annex.SpecialRemote.Config
 
diff --git a/Assistant/WebApp/Configurators/AWS.hs b/Assistant/WebApp/Configurators/AWS.hs
index fd0a4b6de..92717bfe6 100644
--- a/Assistant/WebApp/Configurators/AWS.hs
+++ b/Assistant/WebApp/Configurators/AWS.hs
@@ -171,7 +171,7 @@ getEnableS3R uuid = do
 	isia <- case M.lookup uuid m of
 		Just c -> liftAnnex $ do
 			pc <- either mempty id . parseRemoteConfig c
-				<$> Remote.configParser S3.remote
+				<$> Remote.configParser S3.remote c
 			return $ S3.configIA pc
 		Nothing -> return False
 	if isia
diff --git a/Assistant/WebApp/Configurators/Edit.hs b/Assistant/WebApp/Configurators/Edit.hs
index cbb9ccb96..93a45ef7a 100644
--- a/Assistant/WebApp/Configurators/Edit.hs
+++ b/Assistant/WebApp/Configurators/Edit.hs
@@ -257,7 +257,7 @@ getRepoInfo (Just r) c = case fromProposedAccepted <$> M.lookup typeField c of
 	Just "S3" -> do
 #ifdef WITH_S3
 		pc <- liftAnnex $ either mempty id . parseRemoteConfig c
-			<$> Remote.configParser S3.remote
+			<$> Remote.configParser S3.remote c
 		if S3.configIA pc
 			then IA.getRepoInfo c
 			else AWS.getRepoInfo c
diff --git a/Assistant/WebApp/Configurators/WebDAV.hs b/Assistant/WebApp/Configurators/WebDAV.hs
index 2cdcdc0f2..37ff80455 100644
--- a/Assistant/WebApp/Configurators/WebDAV.hs
+++ b/Assistant/WebApp/Configurators/WebDAV.hs
@@ -63,7 +63,7 @@ postEnableWebDAVR uuid = do
 	mcreds <- liftAnnex $ do
 		dummycfg <- liftIO dummyRemoteGitConfig
 		pc <- either mempty id . parseRemoteConfig c
-			<$> configParser WebDAV.remote
+			<$> configParser WebDAV.remote c
 		getRemoteCredPairFor "webdav" pc dummycfg (WebDAV.davCreds uuid)
 	case mcreds of
 		Just creds -> webDAVConfigurator $ liftH $
diff --git a/CHANGELOG b/CHANGELOG
index 04d41e91f..84689b4e7 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,6 +10,8 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
     foo=yes is expected
   * initremote, enableremote: Reject unknown parameters provided to these
     commands.
+  * Added LISTCONFIGS to external special remote protocol. Special remote
+    programs that use GETCONFIG/SETCONFIG are recommended to implement it.
 
  -- Joey Hess <id@joeyh.name>  Wed, 01 Jan 2020 12:51:40 -0400
 
diff --git a/Remote/External.hs b/Remote/External.hs
index 16edec7f8..e63d78d1b 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -43,27 +43,19 @@ import Control.Concurrent.STM
 import Control.Concurrent.Async
 import System.Log.Logger (debugM)
 import qualified Data.Map as M
+import qualified Data.Set as S
 
 remote :: RemoteType
-remote = RemoteType
+remote = specialRemoteType $ RemoteType
 	{ typename = "external"
 	, enumerate = const (findSpecialRemotes "externaltype")
 	, generate = gen
-	, configParser = pure remoteConfigParser
+	, configParser = remoteConfigParser
 	, setup = externalSetup
 	, exportSupported = checkExportSupported
 	, importSupported = importUnsupported
 	}
 
-remoteConfigParser :: RemoteConfigParser
-remoteConfigParser = RemoteConfigParser
-	{ remoteConfigFieldParsers =
-		[ optionalStringParser externaltypeField
-		, trueFalseParser readonlyField False
-		]
-	, remoteConfigRestPassthrough = const True
-	}
-
 externaltypeField :: RemoteConfigField
 externaltypeField = Accepted "externaltype"
 
@@ -87,7 +79,7 @@ gen r u c gc rs
 			exportUnsupported
 			exportUnsupported
 	| otherwise = do
-		external <- newExternal externaltype u c gc (Just rs)
+		external <- newExternal externaltype (Just u) c (Just gc) (Just rs)
 		Annex.addCleanup (RemoteCleanup u) $ stopExternal external
 		cst <- getCost external r gc
 		avail <- getAvailability external r gc
@@ -170,7 +162,7 @@ gen r u c gc rs
 externalSetup :: SetupStage -> Maybe UUID -> Maybe CredPair -> RemoteConfig -> RemoteGitConfig -> Annex (RemoteConfig, UUID)
 externalSetup _ mu _ c gc = do
 	u <- maybe (liftIO genUUID) return mu
-	pc <- either giveup return $ parseRemoteConfig c remoteConfigParser
+	pc <- either giveup return $ parseRemoteConfig c lenientRemoteConfigParser
 	let externaltype = fromMaybe (giveup "Specify externaltype=") $
 		getRemoteConfigValue externaltypeField pc
 	(c', _encsetup) <- encryptionSetup c gc
@@ -180,8 +172,14 @@ externalSetup _ mu _ c gc = do
 			setConfig (remoteConfig (fromJust (lookupName c)) "readonly") (boolConfig True)
 			return c'
 		_ -> do
-			pc' <- either giveup return $ parseRemoteConfig c' remoteConfigParser
-			external <- newExternal externaltype u pc' gc Nothing
+			pc' <- either giveup return $ parseRemoteConfig c' lenientRemoteConfigParser
+			external <- newExternal externaltype (Just u) pc' (Just gc) Nothing
+			-- Now that we have an external, ask it to LISTCONFIGS, 
+			-- and re-parse the RemoteConfig strictly, so we can
+			-- error out if the user provided an unexpected config.
+			p <- strictRemoteConfigParser external
+			let p' = addRemoteConfigParser specialRemoteConfigParsers p
+			_ <- either giveup return $ parseRemoteConfig c' p'
 			handleRequest external INITREMOTE Nothing $ \resp -> case resp of
 				INITREMOTE_SUCCESS -> result ()
 				INITREMOTE_FAILURE errmsg -> Just $ giveup errmsg
@@ -201,7 +199,7 @@ checkExportSupported c gc = do
 	let externaltype = fromMaybe (giveup "Specify externaltype=") $
 		remoteAnnexExternalType gc <|> getRemoteConfigValue externaltypeField c
 	checkExportSupported' 
-		=<< newExternal externaltype NoUUID c gc Nothing
+		=<< newExternal externaltype Nothing c (Just gc) Nothing
 
 checkExportSupported' :: External -> Annex Bool
 checkExportSupported' external = go `catchNonAsync` (const (return False))
@@ -423,30 +421,36 @@ handleRequest' st external req mp responsehandler
 			. getRemoteConfigPassedThrough
 			<$> liftIO (atomically $ readTVar $ externalConfig st)
 		send $ VALUE value
-	handleRemoteRequest (SETCREDS setting login password) = do
-		let v = externalConfig st
-		c <- liftIO $ atomically $ readTVar v
-		let gc = externalGitConfig external
-		c' <- setRemoteCredPair' RemoteConfigValue id encryptionAlreadySetup c gc
-			(credstorage setting)
-			(Just (login, password))
-		void $ liftIO $ atomically $ swapTVar v c'
-	handleRemoteRequest (GETCREDS setting) = do
-		c <- liftIO $ atomically $ readTVar $ externalConfig st
-		let gc = externalGitConfig external
-		creds <- fromMaybe ("", "") <$> 
-			getRemoteCredPair c gc (credstorage setting)
-		send $ CREDS (fst creds) (snd creds)
-	handleRemoteRequest GETUUID = send $
-		VALUE $ fromUUID $ externalUUID external
+	handleRemoteRequest (SETCREDS setting login password) = case (externalUUID external, externalGitConfig external) of
+		(Just u, Just gc) -> do
+			let v = externalConfig st
+			c <- liftIO $ atomically $ readTVar v
+			c' <- setRemoteCredPair' RemoteConfigValue id encryptionAlreadySetup c gc
+				(credstorage setting u)
+				(Just (login, password))
+			void $ liftIO $ atomically $ swapTVar v c'
+		_ -> senderror "cannot send SETCREDS here"
+	handleRemoteRequest (GETCREDS setting) = case (externalUUID external, externalGitConfig external) of
+		(Just u, Just gc) -> do
+			c <- liftIO $ atomically $ readTVar $ externalConfig st
+			creds <- fromMaybe ("", "") <$> 
+				getRemoteCredPair c gc (credstorage setting u)
+			send $ CREDS (fst creds) (snd creds)
+		_ -> senderror "cannot send GETCREDS here"
+	handleRemoteRequest GETUUID = case externalUUID external of
+		Just u -> send $ VALUE $ fromUUID u
+		Nothing -> senderror "cannot send GETUUID here"
 	handleRemoteRequest GETGITDIR = 
 		send . VALUE . fromRawFilePath =<< fromRepo Git.localGitDir
-	handleRemoteRequest (SETWANTED expr) =
-		preferredContentSet (externalUUID external) expr
-	handleRemoteRequest GETWANTED = do
-		expr <- fromMaybe "" . M.lookup (externalUUID external)
-			<$> preferredContentMapRaw
-		send $ VALUE expr
+	handleRemoteRequest (SETWANTED expr) = case externalUUID external of
+		Just u -> preferredContentSet u expr
+		Nothing -> senderror "cannot send SETWANTED here"
+	handleRemoteRequest GETWANTED = case externalUUID external of
+		Just u -> do
+			expr <- fromMaybe "" . M.lookup u
+				<$> preferredContentMapRaw
+			send $ VALUE expr
+		Nothing -> senderror "cannot send GETWANTED here"
 	handleRemoteRequest (SETSTATE key state) =

(Diff truncated)
moving under datalad project, since I do not see explicit repronim-specific use case addressed
diff --git a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
index 8ad099701..3e24a01c7 100644
--- a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
+++ b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
@@ -8,4 +8,4 @@ manpage says
 and indeed I get asked for e.g. encryption to be explicitly specified (why not to default to none). But it would not even complain if I specify some bogus (mistyped etc) option.  I do not see any way (tried different --help's etc) to figure out what options any given special remote actually supports without going online to e.g. https://git-annex.branchable.com/special_remotes/rsync/ for `rsync` special remote.  It makes configuration of special remotes not a trivial matter for new users.  IMHO there should be some way (mentioned in man of initremote and enableremote) to get information about supported by the remote options, e.g. `git annex getremoteopts TYPE` or alike.  I am not sure how it should/would work for custom special remotes since I think protocol ATM doesn't support querying for a list of configuration options.  May be `GETCONFIG` interface command could be extended to return a list with of options with description? or some new command be added?
 
 [[!meta author=yoh]]
-[[!tag projects/repronim]]
+[[!tag projects/datalad]]

Added a comment
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_6_c4fe35a93ed32f7b14c5ffd5a4a8ec27._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_6_c4fe35a93ed32f7b14c5ffd5a4a8ec27._comment
new file mode 100644
index 000000000..80afc809c
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_6_c4fe35a93ed32f7b14c5ffd5a4a8ec27._comment
@@ -0,0 +1,77 @@
+[[!comment format=mdwn
+ username="hans"
+ avatar="http://cdn.libravatar.org/avatar/8bf4f81bbfce61274f5def013dde7083"
+ subject="comment 6"
+ date="2020-01-16T14:22:03Z"
+ content="""
+I have been trying `get` and `copy` and `whereis`. They all report failure and suggest making the remote `a3f8a46a-60fe-58e2-901b-2c093bcc22d -- smdata_encrypted_remote_wd_elements_small` available.
+
+The remote with this name is available -- at least, I can `sync` with it. However, in `.git/config` the `annex-uuid` is `97d51497-158f-54ef-baef-77a720c9d758`.
+
+as far as I can tell -- I'm shooting in the dark here -- that isn't the issue, because if I change the uuid in `.git/config` it still gives the same error, now with the changed uuid.
+
+So, let me review my mental model of this situation, which will hopefully reveal the gaping holes:
+
+normally, content tracking means each repo knows which other repos have copies of the file. In locked mode, as you said, the file is a symlink to the annexed object and if that target is missing the symlink is simply broken. In unlocked mode, the file is present at its correct location but if it is missing it is replaced with a text file with an annex object path as the contents.
+
+So, my content has somehow gone missing.
+
+Not knowing very much about git-annex's internals, my next question would be: how can I look for this content? I can't explore the files in the remote manually, since they are encrypted. This is what happens if I clone that bare repo again:
+
+```
+$mkdir test && cd test
+$git init
+$git remote add origin gcrypt::/media/hans/Elements/smdata_encrypted_remote_wd_elements_small
+$git annex init
+init  ok
+(recording state in git...)
+$git annex whereis demo_beheer.gpkg
+whereis demo_beheer.gpkg (0 copies) failed #ok, so I need to sync first
+$git annex sync
+commit 
+On branch master
+nothing to commit, working tree clean
+ok
+pull origin 
+gcrypt: Decrypting manifest
+gpg: Signature made do 16 jan 2020 14:08:44 CET
+gpg:                using RSA key xxxxxxxxxxxxxxxxxxxxxx
+gpg: Good signature from \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" [ultimate]
+From gcrypt::/media/hans/Elements/smdata_encrypted_remote_wd_elements_small
+ * [new branch]        synced/master    -> origin/synced/master
+ * [new branch]        synced/git-annex -> origin/synced/git-annex
+ * [new branch]        git-annex        -> origin/git-annex
+ok
+(merging origin/git-annex into git-annex...)
+(recording state in git...)
+push origin 
+gcrypt: Decrypting manifest
+gpg: Signature made do 16 jan 2020 14:08:44 CET
+gpg:                using RSA key xxxxxxxxxxxxx
+gpg: Good signature from \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" [ultimate]
+Enumerating objects: 8, done.
+Counting objects: 100% (8/8), done.
+Compressing objects: 100% (6/6), done.
+Total 8 (delta 1), reused 1 (delta 0)
+gcrypt: Encrypting to: --throw-keyids --default-recipient-self
+gcrypt: Requesting manifest signature
+gpg: using \"xxxxxxxxxxxxxxxxxx\" as default secret key for signing
+To gcrypt::/media/hans/Elements/smdata_encrypted_remote_wd_elements_small
+   10150f10..79a49878  git-annex -> synced/git-annex
+ok
+#try again
+$git annex whereis demo_beheer.gpkg
+whereis demo_beheer.gpkg (1 copy) 
+        a3f8a46a-60fe-58e2-901b-2c093bcc22d3 -- smdata_encrypted_remote_wd_elements_small
+ok
+```
+
+So: the `git-annex` branch is indicating that the content is available in this bare remote, but is that not true? Is there a way for me to determine (with or without git-annex) if the content is actually there or not?
+
+From the speed at which the `git annex get` command returns its error message I get the impression that it's not actually checking the remote, but determining from the local `git-annex` branch that the content is not in the remote. Is that correct? Why does it then suggest making that remote available? In that case, is there a way to figure out if and when the `git-annex` branch logs have diverged from reality, OR alternatively how to find out if there is content in that repository?
+
+What is also really bugging me is that I have the content of some files available in one of my clones, but I can't access them via git-annex. My files are of such a number and format that manually fishing them out of `.git/annex/objects` is not feasible (e.g. shapefiles which consist of about six different files).
+
+
+Finally, the most relevant git-annex command I've been able to find is `git annex unused`, which gives me some interesting information in the bare repository (1900+ unused objects), but `addunused` doesn't seem to  bring things back.
+"""]]

devblog
diff --git a/doc/devblog/day_617__remote_config_parsing_continued.mdwn b/doc/devblog/day_617__remote_config_parsing_continued.mdwn
new file mode 100644
index 000000000..48d41d0f8
--- /dev/null
+++ b/doc/devblog/day_617__remote_config_parsing_continued.mdwn
@@ -0,0 +1,20 @@
+This has been a big change, I'm now 3 days and a 3000 line diff in and I
+finally got all the remote configuration settings converted to the new
+up-front parsing.
+
+Seems like quite a lot of work, since the only user-visible improvement is
+these error messages:
+
+	# git annex initremote demo type=directory directory=../foo encryption=none foo=bar
+	initremote demo
+	git-annex: Unexpected fields: foo
+	
+	# git annex initremote demo type=directory directory=../foo encryption=none exporttree=true
+	initremote demo
+	git-annex: Bad value for exporttree (expected yes or no)
+
+But this involved paying down technical debt in a big code base, so of course it was
+expensive.
+
+Anyway, it should now be relatively easy to implement
+`git annex initremote --list-params-for=S3`

started on --list-params-for
diff --git a/Command/InitRemote.hs b/Command/InitRemote.hs
index 42818303f..4dc6acd9e 100644
--- a/Command/InitRemote.hs
+++ b/Command/InitRemote.hs
@@ -1,6 +1,6 @@
 {- git-annex command
  -
- - Copyright 2011-2019 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2020 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
diff --git a/doc/git-annex-enableremote.mdwn b/doc/git-annex-enableremote.mdwn
index d424f6056..ec5cefbbd 100644
--- a/doc/git-annex-enableremote.mdwn
+++ b/doc/git-annex-enableremote.mdwn
@@ -27,7 +27,8 @@ special remote.
   
 Some special remotes may need parameters to be specified every time they are
 enabled. For example, the directory special remote requires a directory=
-parameter every time.
+parameter every time. The command will prompt for any required parameters
+you leave out.
 
 This command can also be used to modify the configuration of an existing
 special remote, by specifying new values for parameters that are
diff --git a/doc/git-annex-initremote.mdwn b/doc/git-annex-initremote.mdwn
index 8340827f6..e20dec565 100644
--- a/doc/git-annex-initremote.mdwn
+++ b/doc/git-annex-initremote.mdwn
@@ -19,10 +19,16 @@ For a list and details, see <https://git-annex.branchable.com/special_remotes/>
  
 The remote's configuration is specified by the parameters passed
 to this command. Different types of special remotes need different
-configuration values. The command will prompt for parameters as needed. A
-few parameters that are supported by all special remotes are documented in
+configuration values, so consult the documentation of a special remote for
+details. The command will prompt for any required parameters you leave out.
+A few parameters that are supported by all special remotes are documented in
 the next section below.
 
+To list all the configuration parameters that are accepted by a particular
+type of special remote, along with short descriptions, run eg:
+
+	git annex initremote --list-params-for=S3
+
 Once a special remote has been initialized once with this command,
 other clones of the repository can also be set up to access it using
 `git annex enableremote`.
@@ -35,6 +41,11 @@ want to use `git annex renameremote`.
 
 # OPTIONS
 
+* `--list-params-for=type`
+
+  Lists all configuration parameters that are accepted for a given
+  type of special remote.
+
 * `--fast`
 
   When initializing a remote that uses encryption, a cryptographic key is
@@ -66,7 +77,7 @@ want to use `git annex renameremote`.
 
 * `encryption`
 
-  All special remotes support encryption. You will need to specify
+  Almost all special remotes support encryption. You will need to specify
   what encryption, if any, to use. 
 
   If you do not want any encryption, use `encryption=none`

comment
diff --git a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_3_f19ff768a3903f80dbaa378b74d2a7e3._comment b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_3_f19ff768a3903f80dbaa378b74d2a7e3._comment
new file mode 100644
index 000000000..45fa03522
--- /dev/null
+++ b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_3_f19ff768a3903f80dbaa378b74d2a7e3._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2020-01-15T17:52:07Z"
+ content="""
+Unknown fields will now result in an error message. And values like yes/no
+and true/false get parsed upfront.
+
+External special remotes currently still accept all fields, so work still
+needs to be done to extend the protocol to list acceptable fields.
+"""]]

removed
diff --git a/doc/forum/AOL_mail_on_android_not_working.mdwn b/doc/forum/AOL_mail_on_android_not_working.mdwn
deleted file mode 100644
index df1b9b75c..000000000
--- a/doc/forum/AOL_mail_on_android_not_working.mdwn
+++ /dev/null
@@ -1 +0,0 @@
-If you are a user of AOL mailing client then this thread is for you. Being a user I found many bugs by which it was insane for me to use and then I found some sort of solution by which you get rid of these bugs. Firstly check your internet connection if that is working properly then uninstall the app and re-install it. After then it does not work, check out all the setting of the AOL system. If after doing all these steps and yet your mail not working properly or any other services provided by AOL are not working properly then follow on <a href="https://emailsupports.net/how-to-fix-aol-mail-on-android-not-working/">AOL mail on android not working<a/>.  

diff --git a/doc/forum/AOL_mail_on_android_not_working.mdwn b/doc/forum/AOL_mail_on_android_not_working.mdwn
new file mode 100644
index 000000000..df1b9b75c
--- /dev/null
+++ b/doc/forum/AOL_mail_on_android_not_working.mdwn
@@ -0,0 +1 @@
+If you are a user of AOL mailing client then this thread is for you. Being a user I found many bugs by which it was insane for me to use and then I found some sort of solution by which you get rid of these bugs. Firstly check your internet connection if that is working properly then uninstall the app and re-install it. After then it does not work, check out all the setting of the AOL system. If after doing all these steps and yet your mail not working properly or any other services provided by AOL are not working properly then follow on <a href="https://emailsupports.net/how-to-fix-aol-mail-on-android-not-working/">AOL mail on android not working<a/>.  

removed
diff --git a/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn b/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn
deleted file mode 100644
index cbbba9a6a..000000000
--- a/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn
+++ /dev/null
@@ -1,2 +0,0 @@
-I had faced activation issue on Mac, I can't activate my iMessage & iCloud I got an error message- iMessage an Error Occurred During Activation Mac-
-https://www.appletechnicalsupportnumbers.com/blog/imessage-an-error-occurred-during-activation-mac/ and the provided URL from the apple support team its working properly.

removed
diff --git a/doc/forum/Is_this_a_canon_device_error__63__.mdwn b/doc/forum/Is_this_a_canon_device_error__63__.mdwn
deleted file mode 100644
index 955921c2f..000000000
--- a/doc/forum/Is_this_a_canon_device_error__63__.mdwn
+++ /dev/null
@@ -1 +0,0 @@
-I have a problem with my canon printer which I believe caused due to <a href=https://canonprintersupportnumbers.com/blog/canon-mx922-print-head/>canon mx922 print head</a>, I have searched various e-commerce site and I found a various website which sells printhead but in an expensive rate, can anyone link a site which provides a cheap rate.

removed
diff --git a/doc/forum/tv_tuner_problem.mdwn b/doc/forum/tv_tuner_problem.mdwn
deleted file mode 100644
index 30e17b34f..000000000
--- a/doc/forum/tv_tuner_problem.mdwn
+++ /dev/null
@@ -1,2 +0,0 @@
-I have been using windows 10 system. Recently installed tv tuner device in my system but after installation system was running too slow.I wanted to block <a href="https://errorcode0x.com/how-to-block-digital-tv-tuner-device-registration-application/">digital tv tuner device registration application</a> but could not be block.If anyone has any solution then suggest to me.
- 

diff --git a/doc/forum/tv_tuner_problem.mdwn b/doc/forum/tv_tuner_problem.mdwn
new file mode 100644
index 000000000..30e17b34f
--- /dev/null
+++ b/doc/forum/tv_tuner_problem.mdwn
@@ -0,0 +1,2 @@
+I have been using windows 10 system. Recently installed tv tuner device in my system but after installation system was running too slow.I wanted to block <a href="https://errorcode0x.com/how-to-block-digital-tv-tuner-device-registration-application/">digital tv tuner device registration application</a> but could not be block.If anyone has any solution then suggest to me.
+ 

diff --git a/doc/forum/Is_this_a_canon_device_error__63__.mdwn b/doc/forum/Is_this_a_canon_device_error__63__.mdwn
new file mode 100644
index 000000000..955921c2f
--- /dev/null
+++ b/doc/forum/Is_this_a_canon_device_error__63__.mdwn
@@ -0,0 +1 @@
+I have a problem with my canon printer which I believe caused due to <a href=https://canonprintersupportnumbers.com/blog/canon-mx922-print-head/>canon mx922 print head</a>, I have searched various e-commerce site and I found a various website which sells printhead but in an expensive rate, can anyone link a site which provides a cheap rate.

diff --git a/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn b/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn
new file mode 100644
index 000000000..cbbba9a6a
--- /dev/null
+++ b/doc/forum/Fix_Server_issue___38___Activation_issue_on_mac..mdwn
@@ -0,0 +1,2 @@
+I had faced activation issue on Mac, I can't activate my iMessage & iCloud I got an error message- iMessage an Error Occurred During Activation Mac-
+https://www.appletechnicalsupportnumbers.com/blog/imessage-an-error-occurred-during-activation-mac/ and the provided URL from the apple support team its working properly.

Added a comment: re: what am I doing wrong?
diff --git a/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment b/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment
new file mode 100644
index 000000000..163fbc662
--- /dev/null
+++ b/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="kyle"
+ avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
+ subject="re: what am I doing wrong?"
+ date="2020-01-14T03:19:19Z"
+ content="""
+I believe that should be `git annex config --set annex.addunlocked anything` (i.e. an \"annex.\" in front of the name).
+"""]]

Added a comment: what am I doing wrong?
diff --git a/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment b/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment
new file mode 100644
index 000000000..8892334d3
--- /dev/null
+++ b/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment
@@ -0,0 +1,83 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="what am I doing wrong?"
+ date="2020-01-13T20:05:38Z"
+ content="""
+I have tried to use this but I do not see it in effect:
+
+[[!format sh \"\"\"
+$> mkdir repo && cd repo && git init && git annex init && git annex config --set addunlocked anything && git show git-annex:config.log && touch 1 2 && git add 1 && git annex add 2 && git commit -m 'committing' && ls -l && git show
+Initialized empty Git repository in /tmp/repo/.git/
+init  (scanning for unlocked files...)
+ok
+(recording state in git...)
+addunlocked anything ok
+(recording state in git...)
+1578945668.466039639s addunlocked anything
+add 2 
+ok
+(recording state in git...)
+[master (root-commit) e428211] committing
+ 2 files changed, 1 insertion(+)
+ create mode 100644 1
+ create mode 120000 2
+total 4
+-rw------- 1 yoh yoh   0 Jan 13 15:01 1
+lrwxrwxrwx 1 yoh yoh 178 Jan 13 15:01 2 -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+commit e428211fe0c64e67cf45d8c92165c866db5ba75f (HEAD -> master)
+Author: Yaroslav Halchenko <debian@onerussian.com>
+Date:   Mon Jan 13 15:01:08 2020 -0500
+
+    committing
+
+diff --git a/1 b/1
+new file mode 100644
+index 0000000..e69de29
+diff --git a/2 b/2
+new file mode 120000
+index 0000000..ea46194
+--- /dev/null
++++ b/2
+@@ -0,0 +1 @@
++.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+
+\"\"\"]]
+
+so I have tried to say that \"anything\" (all files) should be added unlocked.  But it seems that neither file (`1` added via `git add` and `2` added via `git annex add`) were added unlocked.  
+
+<details>
+<summary>Here is some info on version/config: (click to expand)</summary> 
+
+
+[[!format sh \"\"\"
+(git-annex)lena:/tmp/repo[master]
+$> cat .git/config 
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = false
+	logallrefupdates = true
+[annex]
+	uuid = f220cc03-1510-4e23-acb5-b95723ecf9fc
+	version = 7
+[filter \"annex\"]
+	smudge = git-annex smudge -- %f
+	clean = git-annex smudge --clean -- %f
+(dev3) 1 17256.....................................:Mon 13 Jan 2020 03:03:30 PM EST:.
+(git-annex)lena:/tmp/repo[master]
+$> git annex version
+git-annex version: 7.20191230+git2-g2b9172e98-1~ndall+1
+build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
+operating system: linux x86_64
+supported repository versions: 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+local repository version: 7
+
+\"\"\"]]
+
+</details>
+"""]]

Added a comment: preferred way to automate population of the cache upon `get`
diff --git a/doc/tips/local_caching_of_annexed_files/comment_22_b047ea1c6b9247a42b560a111d4fdeae._comment b/doc/tips/local_caching_of_annexed_files/comment_22_b047ea1c6b9247a42b560a111d4fdeae._comment
new file mode 100644
index 000000000..7c24be4fe
--- /dev/null
+++ b/doc/tips/local_caching_of_annexed_files/comment_22_b047ea1c6b9247a42b560a111d4fdeae._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="preferred way to automate population of the cache upon `get`"
+ date="2020-01-13T19:01:07Z"
+ content="""
+```
+# Populating the cache
+For the cache to be used, you need to get file contents into it somehow. A simple way to do that is, in a git-annex repository that already contains the content of files:
+
+    git annex copy --to cache
+
+You could run that anytime after you get content. There are also ways to automate it, but getting some files into the cache manually is a good enough start.
+```
+
+Hi Joey.  What would be the preferred way you would advise to use (ideally with minimal manual configuration) to make it happen? I.e. whenever user(s) `get` some load, it gets automagically `annex copied` to the cache?
+"""]]

Revert spam
This reverts commit c4a33d15b54520ce68b876f0a7dae4497b61a1f2.
diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
deleted file mode 100644
index 3698c0866..000000000
--- a/doc/forum/router_problem.mdwn
+++ /dev/null
@@ -1 +0,0 @@
-As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?

Revert spam
This reverts commit a1261c5f364ed50621a519c18b047b2680f5c984.
diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
index 9cfb53650..3698c0866 100644
--- a/doc/forum/router_problem.mdwn
+++ b/doc/forum/router_problem.mdwn
@@ -1,4 +1 @@
 As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?
-
-
-

Revert spam
This reverts commit 1c5edbd823990e547258c964ba6513402a4daae6.
diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
index 1ccec4d25..9cfb53650 100644
--- a/doc/forum/router_problem.mdwn
+++ b/doc/forum/router_problem.mdwn
@@ -1 +1,4 @@
-As my router stopped working for some days I bought a new google router and wanted to set up. Can anyone how to access <a href=https://routerguide.org/google-wifi-setup/>google wifi login</a> page and set up the router?
+As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?
+
+
+

Revert spam
This reverts commit a746c210cc59cecae7e05d92063110055f74ac08.
diff --git a/doc/forum/printer_offline_problem.mdwn b/doc/forum/printer_offline_problem.mdwn
deleted file mode 100644
index 5f9167cec..000000000
--- a/doc/forum/printer_offline_problem.mdwn
+++ /dev/null
@@ -1,2 +0,0 @@
- Yesterday, I encountered a problem with my hp printer. It's working fine a few days ago but when I tried to print my essential papers, at that moment, I noticed my printer had given me a communication error, which is your hp printer offline in windows 10. After seeing this error I got a little bit afraid and think about how to fix the hp printer offline windows 10 error.
-<a href="https://hptechnicalsupportphonenumbersusa.com/fix-hp-printer-offline-windows-10/">hp printer offline windows 10</a>

Revert spam
This reverts commit fa7c78c31de7ef030e823bc525b1b482fae4d06d.
diff --git a/doc/forum/gmail_temporary_error_500.mdwn b/doc/forum/gmail_temporary_error_500.mdwn
deleted file mode 100644
index 563129e3c..000000000
--- a/doc/forum/gmail_temporary_error_500.mdwn
+++ /dev/null
@@ -1,5 +0,0 @@
-If you are a gmail user, then you must read this post. Nowadays in this digital world there are many types of viruses are found in this form of a attachment file, that can cause a severe damage on your account, and this is called gmail temporary error 500.
-
-If you have this type of error, well you know the consequences and if not then listen carefully. It will attack on your account by which your account will be blocked by gmail. And for that you will not able to get access your account for a while. So to get prevention you should check your internet first if that is okay, then you should turn off your internet security software. 
-
-Contact me if you have faced <a href="https://emailsupports.net/fix-gmail-temporary-error-500/">gmail temporary error 500<a/>.

devblog
diff --git a/doc/devblog/day_616__remote_config_parsing.mdwn b/doc/devblog/day_616__remote_config_parsing.mdwn
new file mode 100644
index 000000000..858b53d20
--- /dev/null
+++ b/doc/devblog/day_616__remote_config_parsing.mdwn
@@ -0,0 +1,9 @@
+I'm in the middle of a big change to internals. Remotes have buried inside
+them a string-based configuration, and those settings are only parsed when
+they're used, so bad configuration is often ignored rather than being
+detected when the user inputs it. The parsing is moving to happen upfront.
+
+This is something I could not have done when I first wrote git-annex,
+because the values that get parsed have many different types, so how can a
+single Remote data type contain those, whatever they are? Now I know how
+to use the Typeable class to do such things.

diff --git a/doc/forum/gmail_temporary_error_500.mdwn b/doc/forum/gmail_temporary_error_500.mdwn
new file mode 100644
index 000000000..563129e3c
--- /dev/null
+++ b/doc/forum/gmail_temporary_error_500.mdwn
@@ -0,0 +1,5 @@
+If you are a gmail user, then you must read this post. Nowadays in this digital world there are many types of viruses are found in this form of a attachment file, that can cause a severe damage on your account, and this is called gmail temporary error 500.
+
+If you have this type of error, well you know the consequences and if not then listen carefully. It will attack on your account by which your account will be blocked by gmail. And for that you will not able to get access your account for a while. So to get prevention you should check your internet first if that is okay, then you should turn off your internet security software. 
+
+Contact me if you have faced <a href="https://emailsupports.net/fix-gmail-temporary-error-500/">gmail temporary error 500<a/>.

diff --git a/doc/forum/printer_offline_problem.mdwn b/doc/forum/printer_offline_problem.mdwn
new file mode 100644
index 000000000..5f9167cec
--- /dev/null
+++ b/doc/forum/printer_offline_problem.mdwn
@@ -0,0 +1,2 @@
+ Yesterday, I encountered a problem with my hp printer. It's working fine a few days ago but when I tried to print my essential papers, at that moment, I noticed my printer had given me a communication error, which is your hp printer offline in windows 10. After seeing this error I got a little bit afraid and think about how to fix the hp printer offline windows 10 error.
+<a href="https://hptechnicalsupportphonenumbersusa.com/fix-hp-printer-offline-windows-10/">hp printer offline windows 10</a>

diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
index 9cfb53650..1ccec4d25 100644
--- a/doc/forum/router_problem.mdwn
+++ b/doc/forum/router_problem.mdwn
@@ -1,4 +1 @@
-As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?
-
-
-
+As my router stopped working for some days I bought a new google router and wanted to set up. Can anyone how to access <a href=https://routerguide.org/google-wifi-setup/>google wifi login</a> page and set up the router?

diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
index 3698c0866..9cfb53650 100644
--- a/doc/forum/router_problem.mdwn
+++ b/doc/forum/router_problem.mdwn
@@ -1 +1,4 @@
 As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?
+
+
+

diff --git a/doc/forum/router_problem.mdwn b/doc/forum/router_problem.mdwn
new file mode 100644
index 000000000..3698c0866
--- /dev/null
+++ b/doc/forum/router_problem.mdwn
@@ -0,0 +1 @@
+As my router stopped working for some days I bought a new google router and wanted to set up. I got the procedure from https://routerguide.org/google-wifi-setup/ but couldn't apply that. Can anyone how to access google wifi login page and set up the router?

add missing backtick
diff --git a/doc/tips/largefiles.mdwn b/doc/tips/largefiles.mdwn
index 7785daacd..b976615ba 100644
--- a/doc/tips/largefiles.mdwn
+++ b/doc/tips/largefiles.mdwn
@@ -18,7 +18,7 @@ and `git add`/`git commit -a` will add the large files to the annex and the
 small files to git.
 
 Other commands that use the annex.largefiles configuration include
-`git annex import`, git annex addurl`, `git annex importfeed`, and
+`git annex import`, `git annex addurl`, `git annex importfeed`, and
 the assistant.
 
 ## examples

be stricter about rejecting invalid configurations for remotes
This is a first step toward that goal, using the ProposedAccepted type
in RemoteConfig lets initremote/enableremote reject bad parameters that
were passed in a remote's configuration, while avoiding enableremote
rejecting bad parameters that have already been stored in remote.log
This does not eliminate every place where a remote config is parsed and a
default value is used if the parse false. But, I did fix several
things that expected foo=yes/no and so confusingly accepted foo=true but
treated it like foo=no. There are still some fields that are parsed with
yesNo but not not checked when initializing a remote, and there are other
fields that are parsed in other ways and not checked when initializing a
remote.
This also lays groundwork for rejecting unknown/typoed config keys.
diff --git a/Annex/Branch/Transitions.hs b/Annex/Branch/Transitions.hs
index a36091989..3041d54b3 100644
--- a/Annex/Branch/Transitions.hs
+++ b/Annex/Branch/Transitions.hs
@@ -22,6 +22,7 @@ import Types.TrustLevel
 import Types.UUID
 import Types.MetaData
 import Types.Remote
+import Types.ProposedAccepted
 import Annex.SpecialRemote.Config
 
 import qualified Data.Map as M
@@ -85,7 +86,7 @@ dropDead trustmap remoteconfigmap f content = case getLogVariety f of
 	trustmap' = trustmap `M.union`
 		M.map (const DeadTrusted) (M.filter sameasdead remoteconfigmap)
 	sameasdead cm =
-		case toUUID <$> M.lookup sameasUUIDField cm of
+		case toUUID . fromProposedAccepted <$> M.lookup sameasUUIDField cm of
 			Nothing -> False
 			Just u' -> M.lookup u' trustmap == Just DeadTrusted
 	minimizesameasdead u l
diff --git a/Annex/FileMatcher.hs b/Annex/FileMatcher.hs
index fe2c2e692..761f4397f 100644
--- a/Annex/FileMatcher.hs
+++ b/Annex/FileMatcher.hs
@@ -39,6 +39,7 @@ import Types.GitConfig
 import Config.GitConfig
 import Git.FilePath
 import Types.Remote (RemoteConfig)
+import Types.ProposedAccepted
 import Annex.CheckAttr
 import Git.CheckAttr (unspecifiedAttr)
 import qualified Git.Config
@@ -155,8 +156,8 @@ preferredContentKeylessTokens pcd =
 	, SimpleToken "inpreferreddir" (simply $ limitInDir preferreddir)
 	] ++ commonKeylessTokens LimitAnnexFiles
   where
-	preferreddir = fromMaybe "public" $
-		M.lookup "preferreddir" =<< (`M.lookup` configMap pcd) =<< repoUUID pcd
+	preferreddir = maybe "public" fromProposedAccepted $
+		M.lookup (Accepted "preferreddir") =<< (`M.lookup` configMap pcd) =<< repoUUID pcd
 
 preferredContentKeyedTokens :: PreferredContentData -> [ParseToken (MatchFiles Annex)]
 preferredContentKeyedTokens pcd =
diff --git a/Annex/SpecialRemote.hs b/Annex/SpecialRemote.hs
index 4d5cbb4a7..4a40127a9 100644
--- a/Annex/SpecialRemote.hs
+++ b/Annex/SpecialRemote.hs
@@ -17,6 +17,7 @@ import Annex.SpecialRemote.Config
 import Remote (remoteTypes)
 import Types.Remote (RemoteConfig, SetupStage(..), typename, setup)
 import Types.GitConfig
+import Types.ProposedAccepted
 import Config
 import Remote.List
 import Logs.Remote
@@ -49,10 +50,10 @@ newConfig
 	-- when sameas is used
 	-> RemoteConfig
 newConfig name sameas fromuser m = case sameas of
-	Nothing -> M.insert nameField name fromuser
+	Nothing -> M.insert nameField (Proposed name) fromuser
 	Just (Sameas u) -> addSameasInherited m $ M.fromList
-		[ (sameasNameField, name)
-		, (sameasUUIDField, fromUUID u)
+		[ (sameasNameField, Proposed name)
+		, (sameasUUIDField, Proposed (fromUUID u))
 		] `M.union` fromuser
 
 specialRemoteMap :: Annex (M.Map UUID RemoteName)
@@ -66,7 +67,8 @@ specialRemoteMap = do
 
 {- find the remote type -}
 findType :: RemoteConfig -> Either String RemoteType
-findType config = maybe unspecified specified $ M.lookup typeField config
+findType config = maybe unspecified (specified . fromProposedAccepted) $
+	M.lookup typeField config
   where
 	unspecified = Left "Specify the type of remote with type="
 	specified s = case filter (findtype s) remoteTypes of
@@ -94,7 +96,8 @@ autoEnable = do
 			_ -> return ()
   where
 	configured rc = fromMaybe False $
-		Git.Config.isTrueFalse =<< M.lookup autoEnableField rc
+		Git.Config.isTrueFalse . fromProposedAccepted
+			=<< M.lookup autoEnableField rc
 	canenable u = (/= DeadTrusted) <$> lookupTrust u
 	getenabledremotes = M.fromList
 		. map (\r -> (getcu r, r))
diff --git a/Annex/SpecialRemote/Config.hs b/Annex/SpecialRemote/Config.hs
index e09ae8ecc..a2f40b1ee 100644
--- a/Annex/SpecialRemote/Config.hs
+++ b/Annex/SpecialRemote/Config.hs
@@ -10,6 +10,7 @@ module Annex.SpecialRemote.Config where
 import Common
 import Types.Remote (RemoteConfigField, RemoteConfig)
 import Types.UUID
+import Types.ProposedAccepted
 
 import qualified Data.Map as M
 import qualified Data.Set as S
@@ -22,44 +23,54 @@ newtype ConfigFrom t = ConfigFrom t
 
 {- The name of a configured remote is stored in its config using this key. -}
 nameField :: RemoteConfigField
-nameField = "name"
+nameField = Accepted "name"
 
 {- The name of a sameas remote is stored using this key instead. 
  - This prevents old versions of git-annex getting confused. -}
 sameasNameField :: RemoteConfigField
-sameasNameField = "sameas-name"
+sameasNameField = Accepted "sameas-name"
 
 lookupName :: RemoteConfig -> Maybe String
-lookupName c = M.lookup nameField c <|> M.lookup sameasNameField c
+lookupName c = fmap fromProposedAccepted $
+	M.lookup nameField c <|> M.lookup sameasNameField c
 
 {- The uuid that a sameas remote is the same as is stored in this key. -}
 sameasUUIDField :: RemoteConfigField
-sameasUUIDField = "sameas-uuid"
+sameasUUIDField = Accepted "sameas-uuid"
 
 {- The type of a remote is stored in its config using this key. -}
 typeField :: RemoteConfigField
-typeField = "type"
+typeField = Accepted "type"
 
 autoEnableField :: RemoteConfigField
-autoEnableField = "autoenable"
+autoEnableField = Accepted "autoenable"
 
 encryptionField :: RemoteConfigField
-encryptionField = "encryption"
+encryptionField = Accepted "encryption"
 
 macField :: RemoteConfigField
-macField = "mac"
+macField = Accepted "mac"
 
 cipherField :: RemoteConfigField
-cipherField = "cipher"
+cipherField = Accepted "cipher"
 
 cipherkeysField :: RemoteConfigField
-cipherkeysField = "cipherkeys"
+cipherkeysField = Accepted "cipherkeys"
 
 pubkeysField :: RemoteConfigField
-pubkeysField = "pubkeys"
+pubkeysField = Accepted "pubkeys"
 
 chunksizeField :: RemoteConfigField
-chunksizeField = "chunksize"
+chunksizeField = Accepted "chunksize"
+
+embedCredsField :: RemoteConfigField
+embedCredsField = Accepted "embedcreds"
+
+exportTreeField :: RemoteConfigField
+exportTreeField = Accepted "exporttree"
+
+importTreeField :: RemoteConfigField
+importTreeField = Accepted "importtree"
 
 {- A remote with sameas-uuid set will inherit these values from the config
  - of that uuid. These values cannot be overridden in the remote's config. -}
@@ -92,7 +103,8 @@ addSameasInherited m c = case findSameasUUID c of
 			M.restrictKeys parentc sameasInherits
 
 findSameasUUID :: RemoteConfig -> Maybe (Sameas UUID)
-findSameasUUID c = Sameas . toUUID <$> M.lookup sameasUUIDField c
+findSameasUUID c = Sameas . toUUID . fromProposedAccepted
+	<$> M.lookup sameasUUIDField c
 
 {- Remove any fields inherited from a sameas-uuid. When storing a
  - RemoteConfig, those fields don't get stored, since they were already
@@ -108,4 +120,4 @@ findByRemoteConfig matching = map sameasuuid . filter (matching . snd) . M.toLis
   where
 	sameasuuid (u, c) = case M.lookup sameasUUIDField c of
 		Nothing -> (u, c, Nothing)
-		Just u' -> (toUUID u', c, Just (ConfigFrom u))
+		Just u' -> (toUUID (fromProposedAccepted u'), c, Just (ConfigFrom u))
diff --git a/Assistant/Gpg.hs b/Assistant/Gpg.hs
index 6215fba38..01226e064 100644
--- a/Assistant/Gpg.hs
+++ b/Assistant/Gpg.hs
@@ -11,6 +11,7 @@ import Utility.Gpg
 import Utility.UserInfo
 import Types.Remote (RemoteConfigField)
 import Annex.SpecialRemote.Config
+import Types.ProposedAccepted
 
 import qualified Data.Map as M
 import Control.Applicative
@@ -31,7 +32,7 @@ data EnableEncryption = HybridEncryption | SharedEncryption | NoEncryption
 	deriving (Eq)
 

(Diff truncated)
Added a comment: using hardlinks
diff --git a/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_4_51548724d72e2cc6cb2c6e8c3b82e3e4._comment b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_4_51548724d72e2cc6cb2c6e8c3b82e3e4._comment
new file mode 100644
index 000000000..b75702da3
--- /dev/null
+++ b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_4_51548724d72e2cc6cb2c6e8c3b82e3e4._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="satya.ortiz-gagne@a4c92de91eb4fd5ae8fc9893bb4fd674a19f2e59"
+ nickname="satya.ortiz-gagne"
+ avatar="http://cdn.libravatar.org/avatar/79c93025f174cd2aff98fbb952702c09"
+ subject="using hardlinks"
+ date="2020-01-10T16:11:47Z"
+ content="""
+Thanks for your help. Yes I believe that post-checkout hook could do the trick but I really like your idea of using a FUSE filesystem. Thanks a lot for sharing. I also believe this could be the base to progressively get the content of an indexed archive (like .zip) as it's getting needed.
+
+The worktree is a very interesting feature but I'm also using [DataLad 0.11.8](https://www.datalad.org/) which is unfortunately incompatible with it for the moment.
+
+As for my objective to not use locked files, I initially though that the script of a library I was using to preprocess some data was failing because of the fact the files were symlinks but I couldn't reproduce. Unfortunately, too many factor changed so I'm just going to assume I was doing something wrong. Still, it would sometimes be useful to work with unlocked files in the case I'm doing a multi-phases (with multi-commits) preprocessing of a big file. In that case, a phase would modify the file, trigger a copy by unlocking it and annex the modified file. I would be interested into skipping the copy to save a significant amount of time and space since the intermediate states of the file are only temporary. The checksums are still interesting to make sure the phase correctly executed. But that is very specific and will not happen too often so I'm fine with workarounds.
+"""]]

Added a comment: Summary; Application: shared thumbnails
diff --git a/doc/todo/does_not_preserve_timestamps/comment_11_eea115becce4ac79932d8cfa2903f20c._comment b/doc/todo/does_not_preserve_timestamps/comment_11_eea115becce4ac79932d8cfa2903f20c._comment
new file mode 100644
index 000000000..6ef3a9629
--- /dev/null
+++ b/doc/todo/does_not_preserve_timestamps/comment_11_eea115becce4ac79932d8cfa2903f20c._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="https://christian.amsuess.com/chrysn"
+ nickname="chrysn"
+ avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc"
+ subject="Summary; Application: shared thumbnails"
+ date="2020-01-10T08:41:18Z"
+ content="""
+There are two conflicting approaches to mtimes:
+
+* Treat them as local artifacts
+
+  This works great with Make, and generally with any software that works on \"is newer than\" properties.
+
+* Treat them as preservation-worthy file attributes
+
+  This is generally required by tools that compare time stamps by identity.
+
+Both approaches break tools that expect the other, and no single out-of-the-box choice will make all users happy. Tools like metastore, a bespoke solution like etckeeper's generated mkdir/chmod file or a git-annex solution like [[storing the full mtime at genmetadata time|bugs/file_modification_time_should_be_stored_in_exactly_one_metadata_field/]] with a (local or repository-wide) option to set the mtime at annex-get time would be convenient.
+
+One more application where this would be relevant is sharing generated thumbnails among clones of repositories (to eventually maybe even have them available when the full files are not present) following the [XDG specification on shared thumnail repositories](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html#SHARED). Not only does that design rely on the mtimes of the thumbnail and the file to match, it even encodes the mtime again inside the thumbnail, practically requiring all checkouts to not only have consistent mtimes between thumbnails and files, but identical ones.
+"""]]

comment
diff --git a/doc/todo/add_xxHash_backend/comment_2_945217badfaeba8672329d4f6ac4b8e2._comment b/doc/todo/add_xxHash_backend/comment_2_945217badfaeba8672329d4f6ac4b8e2._comment
new file mode 100644
index 000000000..1339435f8
--- /dev/null
+++ b/doc/todo/add_xxHash_backend/comment_2_945217badfaeba8672329d4f6ac4b8e2._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2020-01-09T20:52:25Z"
+ content="""
+Let alone blake3, which is 5-6 times as fast as blake2 while still
+apparently being a cryptographically secure hash.
+"""]]

notes on my recovery from a bad merge
diff --git a/doc/forum/Recovering_from_a_cross-merge.mdwn b/doc/forum/Recovering_from_a_cross-merge.mdwn
new file mode 100644
index 000000000..bde8138b4
--- /dev/null
+++ b/doc/forum/Recovering_from_a_cross-merge.mdwn
@@ -0,0 +1,80 @@
+Intro
+=====
+
+This experience report
+describes steps I've taken for recovering from a situation where
+an *unrelated* git-annex's remote was accidentally merged into a repository.
+
+It is posted to the forum for use by anyone who finds themselves in the same situation
+(especially myself…).
+
+The root cause of the issue was a copy-pasted `git remote add` gone wrong,
+and a subsequent `git annex sync`, that "contaminated" the rest of my remotes.
+That led to `git annex info` showing the union of all the repositories available to the two repositories,
+and `fsck --all` runs looking for files from any repository.
+
+It should go without saying, but here it is anyway:
+
+**Following these steps can eventualy lead to data loss**.
+
+The precautions I've taken are
+
+* knowing that two complete copies of the data sets exist,
+* having a filesystem level snapshot of a least one of those copies, and
+* not starting any file dropping until all remotes have completed fscks at the end.
+
+Identifying the last good state
+===============================
+
+By looking for the first occurrence of the UUID of one of the bad new remotes
+in `git log --patch git-annex`,
+I've identified the last good git-annex state before the merge.
+
+Tagging that as `git tag before-accidental-merging-with-other-server 83c1b945c2428cefa968aec587229f6a87649de6`.
+
+
+Removing potentially mergable information
+=========================================
+
+git-annex is eager to pull in updates lying around --
+while this is usually a good thing,
+here it incurs the danger of resurrecting the accident.
+
+On all remotes that were accessed since the accident,
+I've executed this to remove both the local synced/git-annex branch
+and any memory of cached remote branches:
+
+    $ git branch -D synced/git-annex
+    $ git branch -r | sed 's@remotes/@@' | xargs git branch -d -r
+
+and restore the git-annex branch:
+
+    $ git branch -f git-annex 83c1b945c2428cefa968aec587229f6a87649de6
+
+That proved to be insufficient --
+after I had first only done this,
+things looked good for a while and then after the first `git annex fsck --fast`,
+the remotes were back again.
+
+The only file large enough to contain the offending data in .git/annex was .git/annex/index,
+so I've removed that backed by [[internals]]' statement of it being safe to remove:
+
+    $ rm .git/annex/index
+
+(did that on all remotes; on bare ones it's `annex/index`, obviously).
+
+Verification
+============
+
+To ensure everyone is on the same page,
+I've run `git annex sync`;
+its speed already showed that now there's no information about a second repository being transferred.
+
+Subsequently, I've run `git annex fsck --all` in all locations.
+(That *did* show that I should previously have marked some keys as dead when they were migrated from SHA256E to SHA256,
+but that's beside the point here).
+
+Even after a sync following the above,
+no traces of the bad merge (be it in the form of a repository or of a file from there) have shown up any more.
+
+-- [[chrysn]]

comment
diff --git a/doc/todo/provide_windows_build_with_MagicMime/comment_5_4633d2a40bca1ba8fa93f7038569715f._comment b/doc/todo/provide_windows_build_with_MagicMime/comment_5_4633d2a40bca1ba8fa93f7038569715f._comment
new file mode 100644
index 000000000..b32a7200b
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime/comment_5_4633d2a40bca1ba8fa93f7038569715f._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2020-01-08T18:26:17Z"
+ content="""
+I guess that `git annex adjust --unlock` would avoid whatever the problem
+is with the symlinks.
+"""]]

response
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_5_df986910016ce53a6155edf96f4c312d._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_5_df986910016ce53a6155edf96f4c312d._comment
new file mode 100644
index 000000000..675ecc684
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_5_df986910016ce53a6155edf96f4c312d._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2020-01-08T18:24:43Z"
+ content="""
+`git-annex sync` does not, by default, download the content of annexed
+files. Use `git annex get`
+"""]]

Added a comment: git-annex in docker on Windows
diff --git a/doc/todo/provide_windows_build_with_MagicMime/comment_4_4727291eefa07a40da13dace75a2e255._comment b/doc/todo/provide_windows_build_with_MagicMime/comment_4_4727291eefa07a40da13dace75a2e255._comment
new file mode 100644
index 000000000..2f1b5b926
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime/comment_4_4727291eefa07a40da13dace75a2e255._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="git-annex in docker on Windows"
+ date="2020-01-08T16:16:31Z"
+ content="""
+Maybe, git-annex on Windows could be run in a Docker container, operating on Windows files via bind mounts?  A conda-forge package could be made that translates git-annex command invocations into 'docker run' commands.
+"""]]

fix a transcription error - to disable a remote, we set ignore to true, not false!
diff --git a/doc/tips/disabling_a_special_remote.mdwn b/doc/tips/disabling_a_special_remote.mdwn
index 3b80ef25c..c14b38370 100644
--- a/doc/tips/disabling_a_special_remote.mdwn
+++ b/doc/tips/disabling_a_special_remote.mdwn
@@ -9,7 +9,7 @@ simply to use the `remote.name.annex-ignore` configuration documented
 in the main [[git-annex]] manpage. For example, to disable the web
 remote, you would use:
 
-    git config remote.web.annex-ignore false
+    git config remote.web.annex-ignore true
 
 The result would be:
 

Added a comment: WSL 1/2 experience
diff --git a/doc/todo/provide_windows_build_with_MagicMime/comment_3_15509110c5900aee2cc977372da16f0b._comment b/doc/todo/provide_windows_build_with_MagicMime/comment_3_15509110c5900aee2cc977372da16f0b._comment
new file mode 100644
index 000000000..9921aa643
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime/comment_3_15509110c5900aee2cc977372da16f0b._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="michael.hanke@c60e12358aa3fc6060531bdead1f530ac4d582ec"
+ nickname="michael.hanke"
+ avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
+ subject="WSL 1/2 experience"
+ date="2020-01-08T15:20:04Z"
+ content="""
+I looked into WSL 1 and 2 in the context (and with the hope) of trying to avoid having windows-specific code and documentation in and for datalad. My conclusion was that neither of them is practical (maybe: yet), because with a unix-like default setup of a repository (lock, symlinks) no repository content was accessible from outside the WSL environment. I think the main motivation for running on Windows is using Windows apps. And that specifically is not possible/easily achievable for any random app.
+"""]]

Added a comment
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_4_0a9cec8f4e26c2e2365b54b29c91ee3d._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_4_0a9cec8f4e26c2e2365b54b29c91ee3d._comment
new file mode 100644
index 000000000..c5ebf224b
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_4_0a9cec8f4e26c2e2365b54b29c91ee3d._comment
@@ -0,0 +1,41 @@
+[[!comment format=mdwn
+ username="fasthans@87884c78197fe1a25ac25ae7fa1d930c147e9592"
+ avatar="http://cdn.libravatar.org/avatar/6d89ddec3053d8df7e57cf2f28c8cdcd"
+ subject="comment 4"
+ date="2020-01-08T12:08:23Z"
+ content="""
+Thanks for your attention, Joey.
+
+Hmm. `git annex fsck` reports 'ok' for all files. After that, running get/whereis I get:
+
+````
+git annex get demo_beheer.gpkg
+get demo_beheer.gpkg (not available) 
+  Try making some of these repositories available:
+        a3f8a46a-60fe-58e2-901b-2c093bcc22d3 -- smdata_encrypted_remote_wd_elements_small
+failed
+git-annex: get: 1 failed
+````
+
+this is the remote in question, which is accessible (it's on the same drive as the current repo)
+
+````
+git remote -v
+smdata_encrypted_remote_wd_elements_small       gcrypt::/media/hans/Elements/smdata_encrypted_remote_wd_elements_small/ (fetch)
+smdata_encrypted_remote_wd_elements_small       gcrypt::/media/hans/Elements/smdata_encrypted_remote_wd_elements_small/ (push)
+````
+
+However, I can run `git annex sync smdata_remote_wd_elements_small`, which does complete successfully, so why is it not accessible with `get` or `copy`?
+
+If I `lock` demo_beheer.gpkg, it turns into this symlink:
+
+````
+lrwxrwxrwx 1 hans hans 198 sep  2 07:32 demo_beheer.gpkg -> .git/annex/objects/px/Mg/SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg/SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg
+````
+
+of which the target indeed does not exist (the directory exists, not the file). However, git annex get still fails after locking (and a subsequent sync) and I am at a loss to know why.
+
+This wouldn't be due to the nature of the remote, or my URL for it, or something? Decryption works fine when syncing.
+
+
+"""]]

removed
diff --git a/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment b/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment
deleted file mode 100644
index dd63fa3e5..000000000
--- a/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="undral"
- avatar="http://cdn.libravatar.org/avatar/76e31a06ce1ed0759f84bbf29d68b49b"
- subject="hi"
- date="2020-01-08T00:17:13Z"
- content="""
-hi
-"""]]

Added a comment: hi
diff --git a/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment b/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment
new file mode 100644
index 000000000..dd63fa3e5
--- /dev/null
+++ b/doc/tips/storing_data_in_git-lfs/comment_1_ab3fca7db0907d7b5a5b3f3d2ad23cb9._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="undral"
+ avatar="http://cdn.libravatar.org/avatar/76e31a06ce1ed0759f84bbf29d68b49b"
+ subject="hi"
+ date="2020-01-08T00:17:13Z"
+ content="""
+hi
+"""]]

generalize docs so they will also work when git uses SHA256
diff --git a/doc/internals.mdwn b/doc/internals.mdwn
index 0745004c7..cd48a39f8 100644
--- a/doc/internals.mdwn
+++ b/doc/internals.mdwn
@@ -180,13 +180,13 @@ Tracks what trees have been exported to special remotes by
 Each line starts with a timestamp, then the uuid of the repository
 that exported to the special remote, followed by a colon (`:`) and
 the uuid of the special remote. Then, separated by a spaces,
-the sha1 of the tree that was exported, and optionally any number of
-subsequent sha1s, of trees that have started to be exported but whose
+the SHA of the tree that was exported, and optionally any number of
+subsequent SHAs, of trees that have started to be exported but whose
 export is not yet complete. 
 
 In order to record the beginning of the first export, where nothing
-has been exported yet, the sha1 of the exported tree can be
-the empty tree (4b825dc642cb6eb9a060e54bf8d69288fbee4904).
+has been exported yet, the SHA of the exported tree can be
+the empty tree (eg 4b825dc642cb6eb9a060e54bf8d69288fbee4904).
 
 For example:
 

response
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_3_119c181afe351a412e8a13c59175ec45._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_3_119c181afe351a412e8a13c59175ec45._comment
new file mode 100644
index 000000000..a08729f66
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_3_119c181afe351a412e8a13c59175ec45._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2020-01-07T19:31:14Z"
+ content="""
+Looking more closely at your list of files, your repository does not
+contain a copy of the current version of demo_beheer.gpkg, which is
+"SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg"
+
+There's evidence you tried to download that key from somewhere, but the
+download failed.
+
+So, it seems that your repository is not in any unsual state, you're just
+confused about how an unlocked file that is not present looks. Probably
+commands like these will be useful:
+
+	git annex get demo_beheer.gpkg
+	git annex whereis demo_beheer.gpkg
+"""]]

response
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_2_3e467d256ef7c2f4d469073c00cacbca._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_2_3e467d256ef7c2f4d469073c00cacbca._comment
new file mode 100644
index 000000000..f1ff0f671
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_2_3e467d256ef7c2f4d469073c00cacbca._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2020-01-07T19:24:31Z"
+ content="""
+"/annex/objects/SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg" is
+used when an annexed object in a v7 repository is unlocked.
+
+If you run `git annex lock` on it, it will be turned backed into a symlink to
+the .git/annex/objects file.
+
+Normally unlocked files have that pointer replaced with the file content when 
+it's available, and only when the file content is not available would you see
+that pointer. I guess you've done something to get your repository into this
+state where the content is present but the unlocked file is not populated with 
+it. It's likely that running `git annex fsck` on the file would fix that problem.
+"""]]

Added a comment: prior research
diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_1_e89aa61303d97bfd433e24173867723a._comment b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_1_e89aa61303d97bfd433e24173867723a._comment
new file mode 100644
index 000000000..e73ec845f
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects/comment_1_e89aa61303d97bfd433e24173867723a._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="fasthans@87884c78197fe1a25ac25ae7fa1d930c147e9592"
+ avatar="http://cdn.libravatar.org/avatar/6d89ddec3053d8df7e57cf2f28c8cdcd"
+ subject="prior research"
+ date="2020-01-07T19:07:17Z"
+ content="""
+I forgot to mention, all I've been able to find that seemed related was this post: https://git-annex.branchable.com/bugs/fix_git-annex_paths___47___objects___40__repository_not_available__41__/
+
+but I couldn't figure out if this was a similar problem to mine.
+"""]]

diff --git a/doc/forum/Paths_changed_in_.git__47__annex__47__objects.mdwn b/doc/forum/Paths_changed_in_.git__47__annex__47__objects.mdwn
new file mode 100644
index 000000000..4d0a88989
--- /dev/null
+++ b/doc/forum/Paths_changed_in_.git__47__annex__47__objects.mdwn
@@ -0,0 +1,31 @@
+Hello,
+
+Has anyone encountered this issue:
+
+I have a repository in version 7. It has various remotes and clones from my frantic attempts to recover my data without really knowing what I'm doing. Anyways, the files in the repository are text files with hashes in them:
+
+
+    cat demo_beheer.gpkg
+    /annex/objects/SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg
+
+
+But in `.git/annex/objects` there are two-letter directories. I can find this file in there, but its name is different than the above. All the .gpkg files under `.git/annex/objects` are intact (I can open them), and also appear to be versions of the same file (the `demo_beheer.gpkg` one I'm looking for).
+
+
+    find -name '*.gpkg' -type f
+    ./.git/annex/objects/Gz/v4/SHA256E-s204800--01120000361af90c29ee27a51ef7a6157bc413dc768d8ba495b7df8360c6dbfe.gpkg/SHA256E-s204800--01120000361af90c29ee27a51ef7a6157bc413dc768d8ba495b7df8360c6dbfe.gpkg
+    ./.git/annex/objects/mV/6j/SHA256E-s204800--222d8fe6975a07a6305b27a453c7db62df0518458d53252bee2f8bac16d1329c.gpkg/SHA256E-s204800--222d8fe6975a07a6305b27a453c7db62df0518458d53252bee2f8bac16d1329c.gpkg
+    ./.git/annex/objects/Q3/30/SHA256E-s204800--71d2d90cb98ea98806b6f9ae479ffae7d2d7f6b1fb6ea970c108ef0b7b0a52ec.gpkg/SHA256E-s204800--71d2d90cb98ea98806b6f9ae479ffae7d2d7f6b1fb6ea970c108ef0b7b0a52ec.gpkg
+    ./.git/annex/objects/X7/QJ/SHA256E-s204800--d3ce397eb2f1d5080641e15a8d28a5ebabf56ce03b756ed6ceb93fec0d390c72.gpkg/SHA256E-s204800--d3ce397eb2f1d5080641e15a8d28a5ebabf56ce03b756ed6ceb93fec0d390c72.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s3989504--78a7d01d5b7331cb867464c5787264292b39905533be58822a63c9f6d9ea8b3d.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s1929216--03c015bdd9ac6efadf5d855bed734fb20531938d93d70c075cfc47b5f3f3a64b.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s946176--49fe39ceb46d2518bfff4dfe2e1e83c043ad76f13871c4143a5fff68540d943c.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s204800--a518c074bc22f673f0c73191a01426fef0a7d8b262a17d2729a4a3ac51da40ce.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s51445760--a44651f781e5fce11bc498ba7fc30a0b79e5f7e282226852ef949f45888fb6eb.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s4091904--3eb3df304d9fab549dcc657198c88bfc300f8c11836ecc62ae68538a67e3d430.gpkg
+    ./.git/annex/transfer/failed/download/a3f8a46a-60fe-58e2-901b-2c093bcc22d3/SHA256E-s63684608--e5ff8eb805b96c7e231b7450514d101397a98a11f9320416d78084b9cad58e93.gpkg
+    ./demo_beheer.gpkg
+
+Is this a recognized result of some slip-up I've made? I can't remember what exactly I did to reach here other than try to clone the repo various times, and I might have messed up some paths because of struggling with the gcrypt url formats. I can of course provide more details if that would help. Alternatively, does someone have an idea of how I could recover these files simply, to start over?
+
+Help would be much appreciated.

plan
diff --git a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_2_afa8c10bd3b1df649c1f643430b300e9._comment b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_2_afa8c10bd3b1df649c1f643430b300e9._comment
new file mode 100644
index 000000000..47f70e8bb
--- /dev/null
+++ b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_2_afa8c10bd3b1df649c1f643430b300e9._comment
@@ -0,0 +1,34 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2020-01-07T17:59:35Z"
+ content="""
+I was thinking about implementing this today, but the shattered attack got
+in the way. Anyway, it seems like most of a plan:
+
+* Make RemoteConfig contain Old or New values. enableremote and initremote
+  set New values; Old values are anything read from git-annex:remote.log
+* When a RemoteConfig value fails to parse, it may make sense to use a
+  default instead when it's Old, and error out when it's New. This could
+  be used when parsing foo=yes/no to avoid treating foo=true the same as
+  foo=no, which some things do currently do 
+  (eg importtree, exporttree, embedcreds).
+* Add a Remote method that returns a list of all RemoteConfig fields it
+  uses. This is the one part I'm not sure about, because that violates DRY.
+  It would be nicer to have a parser that can also generate a list of the
+  fields it parses.
+* Before calling Remote setup, see if there is any New value in 
+  RemoteConfig whose field is not in the list. If so, error out.
+* For external special remotes, add a LISTCONFIG message. The program
+  reponds with a list of all the fields it may want to later GETCONFIG.
+  If the program responds with UNSUPPORTED-REQUEST then it needs to return
+  something that says any and all fields are allowed.
+* External special remotes are responsible for parsing the content of
+  GETCONFIG, as they do now, and can error out if there's a problem.
+
+Having a method return a list of fields will also allow
+implementing
+<https://git-annex.branchable.com/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type/>.
+It may be worthwhile to add, along with the field name, a human readable
+description of its value.
+"""]]

Added a comment
diff --git a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment
new file mode 100644
index 000000000..fb54f4802
--- /dev/null
+++ b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="spwhitton"
+ avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
+ subject="comment 2"
+ date="2020-01-07T12:29:47Z"
+ content="""
+Thank you for your reply.  Makes sense.  If that's the only way to do it then it might as well be a helper script rather than part of git-annex.
+
+Leaving this bug open because it would be good to have the limitation documented in git-annex-reinject(1).
+"""]]

comment
diff --git a/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment b/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment
new file mode 100644
index 000000000..c12a24cd9
--- /dev/null
+++ b/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T19:36:32Z"
+ content="""
+xxhash seems to fill a similar niche and is getting a lot more use from
+what I can see.
+
+Meow seems to claim a faster gb/s rate than xxhash does, but
+it's hard to tell if the benchmarks are really equivilant.
+"""]]

fix typo in close
diff --git a/doc/todo/addurl_improvements.mdwn b/doc/todo/addurl_improvements.mdwn
index 17ed129e8..35129f3d5 100644
--- a/doc/todo/addurl_improvements.mdwn
+++ b/doc/todo/addurl_improvements.mdwn
@@ -8,7 +8,7 @@ When an external special remote tells git-annex a fuller URL for a given file, g
 
 It would be better if, in the above log, the URL key was based on dx://file-FJZjVx001pB2BQPVKY4zX8kk/A4.assembly1-trinity.fasta , which would preserve the .fasta extension in the key and therefore in the symlink target.
 
-> [fixed|done]] --[[Joey]]
+> [[fixed|done]] --[[Joey]]
 
 Also, it would be good if the external special remote could return an etag
 for the URL, which would be a value guaranteed to change if the URL's

comment
diff --git a/doc/todo/add_xxHash_backend/comment_1_8993f493a8b43ffda9f8e0b6d5e186ac._comment b/doc/todo/add_xxHash_backend/comment_1_8993f493a8b43ffda9f8e0b6d5e186ac._comment
new file mode 100644
index 000000000..40e7e02f2
--- /dev/null
+++ b/doc/todo/add_xxHash_backend/comment_1_8993f493a8b43ffda9f8e0b6d5e186ac._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T19:30:43Z"
+ content="""
+I looked at xxHash recently. I can't seem to find benchmarks of it compared
+with other fast hashes like Blake2.
+"""]]

comment
diff --git a/doc/todo/universal_batch_mode/comment_3_c8232195fa3db718dc2b246d528e6c00._comment b/doc/todo/universal_batch_mode/comment_3_c8232195fa3db718dc2b246d528e6c00._comment
new file mode 100644
index 000000000..5f694569a
--- /dev/null
+++ b/doc/todo/universal_batch_mode/comment_3_c8232195fa3db718dc2b246d528e6c00._comment
@@ -0,0 +1,46 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2020-01-06T18:53:02Z"
+ content="""
+Thanks for digging that up.
+
+Hmm, if the goal was to check each command for such problems when adding
+--batch, it didn't stop `git-annex add --batch` from being added, despite
+indeed having such a buffering behavior. You can currently shoot your foot
+combining that with `git annex readpresentkey --batch`, the same way as you
+could with a hypothetical universal batch mode that let you run add
+followed by readpresentkey.
+
+I don't see a universal batch mode being really able to detect
+and avoid such problems either. How is it supposed to know that an add of
+"dir/" will amoung other things add the content of key FOO, which was not
+present before, and so a readpresentkey FOO should be delayed until after
+the add, and the add's buffer flushed. It would have to model the
+behavior of commands and insert barriers/flush points, and the modeling
+could necessarily not be that fine-grained, so it would need to flush
+the add buffer every time before readpresentkey. But there are surely
+ways to combine batch use of add and readpresentkey that you know won't be
+affected by the add buffering, and that would make those unncessarily slow.
+
+Anyway, looking at the implementation of --batch for different commands,
+sometimes it's trivial enough to wish it were generalized, but other times
+there is batch-specific behavior. `add --batch` errors out if --update
+is also used. `checkpresentkey --batch` outputs status codes rather than
+the command's normal behavior of exiting 1/0. So we need batch-specific
+implementations.
+
+---
+
+idea: What might be good is a mode that lets any batch-capable commands
+be combined together, not trying to support every possible command, and
+perhaps with some added commands that the user can use to flush buffers
+etc between operations as desired. Eg:
+
+	git annex batch <<EOF
+	add dir/
+	add whatever
+	flush
+	readpresentkey FOO
+	EOF
+"""]]

Added a comment
diff --git a/doc/todo/provide_windows_build_with_MagicMime/comment_2_8f2d350c853f8827e1d5f95897dc2bd8._comment b/doc/todo/provide_windows_build_with_MagicMime/comment_2_8f2d350c853f8827e1d5f95897dc2bd8._comment
new file mode 100644
index 000000000..e04f76ca2
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime/comment_2_8f2d350c853f8827e1d5f95897dc2bd8._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 2"
+ date="2020-01-06T19:27:23Z"
+ content="""
+FWIW:
+
+- I am not yet sure if WSL would ever become a panacea. So far in my attempts I kept running into issues which related to how you (user) expose Windows file system(s) inside WSL, and there might be no \"one setup which fits all the use cases\".
+
+- There seems to be efforts like https://github.com/nscaife/file-windows which seems to promise building Windows binaries of file and libmagic for Windows - may be it could be used here.
+
+- IMHO an ideal way for git-annex deployment on windows would have been a conda-forge build of it.  Unfortunately it would not facilitate solution here since I do not see [windows build for libmagic](https://anaconda.org/search?q=libmagic). (meanwhile [filed an issue](https://github.com/conda-forge/libmagic-feedstock/issues/11), if resolved - could give an additional motivation to approach [git-annex-feedstock/issues/15](https://github.com/conda-forge/git-annex-feedstock/issues/15)
+"""]]

close, redundant
diff --git a/doc/todo/annex.addunlocked_in_gitattributes.mdwn b/doc/todo/annex.addunlocked_in_gitattributes.mdwn
index 45e064e03..22ff605a5 100644
--- a/doc/todo/annex.addunlocked_in_gitattributes.mdwn
+++ b/doc/todo/annex.addunlocked_in_gitattributes.mdwn
@@ -9,3 +9,5 @@ would effectively build up a file match expression. So it might then follow
 that the git config should also be a file match expression, with "true"
 being the same as "anything" and "false" the same as "nothing" for
 back-compat. --[[Joey]]
+
+> This got accomplished by other means, [[done]] --[[Joey]]

comment and close
diff --git a/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn b/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn
index 2b36d8983..846ae0c4e 100644
--- a/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn
+++ b/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn
@@ -1 +1,3 @@
 `git diff` for annexed files, especially unlocked annexed files, is currently uninformative.  It would help if [[`git-annex-init`|git-annex-init]] configured a [git diff driver](https://git-scm.com/docs/gitattributes#_generating_diff_text) to diff the contents of the annexed files, rather than the pointer files.
+
+> [[wontfix|done]], see comment 
diff --git a/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment b/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment
new file mode 100644
index 000000000..70c2d59ea
--- /dev/null
+++ b/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T18:39:27Z"
+ content="""
+Normally annexed files are huge binary files. Line-by-line diff of such
+files is unlikely to be useful.
+
+So you would need some domain-specific diff for the kind of binary files
+you are storing in git-annex. If you have one, you can use
+[[git-annex-diffdriver]] to make git use it when diffing annexed files.
+
+Not seeing anything more I can do here, so I'm going to close this todo.
+"""]]

comments
diff --git a/doc/todo/documenting_sqlite_database_schemas/comment_1_cc94cb2467c2b261c0b79c64853103a5._comment b/doc/todo/documenting_sqlite_database_schemas/comment_1_cc94cb2467c2b261c0b79c64853103a5._comment
new file mode 100644
index 000000000..4003cda63
--- /dev/null
+++ b/doc/todo/documenting_sqlite_database_schemas/comment_1_cc94cb2467c2b261c0b79c64853103a5._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T18:07:00Z"
+ content="""
+There are not any situations where after losing the sqlite databases 
+git-annex can't recover the information that was stored in them by other
+means. I know because the v8 upgrade deletes all the old sqlite databases
+and then recovers the information by other means. So no future-proofing
+impact here.
+"""]]
diff --git a/doc/todo/documenting_sqlite_database_schemas/comment_2_06bc5d625d9a3e7522c7f6df53a7ac3d._comment b/doc/todo/documenting_sqlite_database_schemas/comment_2_06bc5d625d9a3e7522c7f6df53a7ac3d._comment
new file mode 100644
index 000000000..77b033984
--- /dev/null
+++ b/doc/todo/documenting_sqlite_database_schemas/comment_2_06bc5d625d9a3e7522c7f6df53a7ac3d._comment
@@ -0,0 +1,26 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2020-01-06T18:16:13Z"
+ content="""
+It's easy enough to dump the database and see its schema.
+
+	joey@darkstar:~/lib/big>sqlite3 .git/annex/keys/db
+	sqlite> .dump
+	CREATE TABLE IF NOT EXISTS "associated"("id" INTEGER PRIMARY KEY,"key" VARCHAR NOT NULL,"file" VARCHAR NOT NULL,CONSTRAINT "key_file_index" UNIQUE ("key","file"),CONSTRAINT "file_key_index" UNIQUE ("file","key"));
+	CREATE TABLE IF NOT EXISTS "content"("id" INTEGER PRIMARY KEY,"key" VARCHAR NOT NULL,"cache" VARCHAR NOT NULL,CONSTRAINT "key_cache_index" UNIQUE ("key","cache"));
+
+Or the fully typed schema can be looked up in the haskell code
+(Database/Keys/Sql.hs)
+
+I think that how the information in the databases relates to the state of the
+repository, and how it's updated from the git-annex branch etc is just as
+important as the schema. For example, if you wanted to use this database to
+query files using a key, you'd need to know this database only gets
+populated for unlocked files not locked files. And that the database may not
+reflect recent changes to the working tree, and there's a complicated process
+that can be used to update it to reflect any recent changes.
+
+That's rather deep into the implementation details to be documenting
+outside the code.
+"""]]

reinject --known: Fix bug that prevented it from working in a bare repo.
ifAnnexed in a bare repo passes to git cat-file :./filename , which it
refuses to do since the repo is bare.
Note that, reinject somefile someannexedfile in a bare repo silently does
nothing, because someannexedfile is never actually an annexed worktree
file, because the repo is bare.
diff --git a/CHANGELOG b/CHANGELOG
index 7a50cf28c..d94481da3 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,7 @@ git-annex (7.20191231) UNRELEASED; urgency=medium
   * add: --force-annex/--force-git options make it easier to override
     annex.largefiles configuration (and potentially safer as it avoids
     bugs like the smudge bug fixed in the last release).
+  * reinject --known: Fix bug that prevented it from working in a bare repo.
 
  -- Joey Hess <id@joeyh.name>  Wed, 01 Jan 2020 12:51:40 -0400
 
diff --git a/Command/Reinject.hs b/Command/Reinject.hs
index d33817deb..a73253ba1 100644
--- a/Command/Reinject.hs
+++ b/Command/Reinject.hs
@@ -13,6 +13,7 @@ import Annex.Content
 import Backend
 import Types.KeySource
 import Utility.Metered
+import qualified Git
 
 cmd :: Command
 cmd = command "reinject" SectionUtility 
@@ -65,8 +66,13 @@ startKnown src = notAnnexed src $
 				)
 
 notAnnexed :: FilePath -> CommandStart -> CommandStart
-notAnnexed src = ifAnnexed (toRawFilePath src) $
-	giveup $ "cannot used annexed file as src: " ++ src
+notAnnexed src a = 
+	ifM (fromRepo Git.repoIsLocalBare)
+		( a
+		, ifAnnexed (toRawFilePath src)
+			(giveup $ "cannot used annexed file as src: " ++ src)
+			a
+		)
 
 perform :: FilePath -> Key -> CommandPerform
 perform src key = ifM move
diff --git a/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn b/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn
index dc46ac71e..d4813403f 100644
--- a/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn
+++ b/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn
@@ -26,3 +26,5 @@
 Obviously this wasn't actually a file known to git-annex.  But I get the same error in a non-dummy bare repo I am trying to reinject.
 
 A workaround is to use `git worktree add` and run `git annex reinject` from there.
+
+> [[fixed|done]] --[[Joey]]

comment
diff --git a/doc/todo/S3_special_remote_support_for_DigitalOcean_Spaces/comment_1_12fd9dfb47e157e1d38c5e88a543498b._comment b/doc/todo/S3_special_remote_support_for_DigitalOcean_Spaces/comment_1_12fd9dfb47e157e1d38c5e88a543498b._comment
new file mode 100644
index 000000000..56718b48e
--- /dev/null
+++ b/doc/todo/S3_special_remote_support_for_DigitalOcean_Spaces/comment_1_12fd9dfb47e157e1d38c5e88a543498b._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T17:41:21Z"
+ content="""
+This is either a bug in <http://hackage.haskell.org/package/aws> or
+Digital Ocean's implementation of the S3 protocol. I don't know which,
+but I do know it has to be fixed in one of those two place, and not
+in git-annex.
+
+The fact that the aws library can't even parse an error message out of
+their response kind of says something.
+
+But the real problem seems to
+While they claim to support V2 signatures,
+my guess is there's a problem with their support for V2, since they're
+mostly dealing with V4. The aws library's support for V4 is experimental
+and [apparently buggy](https://github.com/aristidb/aws/issues/262), but
+as git-annex uses it, it will only use V2.
+"""]]

comment
diff --git a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment
new file mode 100644
index 000000000..3e25f3cad
--- /dev/null
+++ b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T17:11:58Z"
+ content="""
+I can't think of a reasonable way to implement this.
+
+It would need to hash and then look for a known SHA256E key that uses the
+hash. But the layout of the git-annex branch doesn't provide any way to do
+that, except for iterating over every filename in the branch. Which
+would be prohibitively slow when reinjecting many files. (N times git
+ls-tree -r) So it would need to build a data structure to map from SHA256
+to known SHA256E key. That can't be stored in memory, git-annex doesn't
+let the content of the repo cause it to use arbitrary amounts of memory
+(hopefully).
+
+All I can think of is to traverse the git-annex branch and build a sqlite
+database and then query that, but that would add quite a lot of setup
+overhead to the command.
+"""]]

update
Removed some encoding issues that I think are no longer a problem.
diff --git a/doc/todo/windows_support.mdwn b/doc/todo/windows_support.mdwn
index f636da9f5..da049b2f1 100644
--- a/doc/todo/windows_support.mdwn
+++ b/doc/todo/windows_support.mdwn
@@ -29,19 +29,8 @@ seems unreliable/broken on Windows.
   to avoid this, but only ended up with encoding crashes, or worse
   mojibake than this.
 
-* `md5FilePath` still uses the filesystem encoding, and so may produce the
-  wrong value on Windows. This would impact keys that contain problem characters
-  (probably coming from the filename extension), and might cause
-  interoperability problems when git-annex generates the hash directories of a
-  remote, for example a rsync remote.
-
-* `encodeW8` is used in Git.UnionMerge, and while I fixed the other calls to
-  encodeW8, which all involved ByteStrings reading from git and so can just
-  treat it as utf-8 on Windows (via `decodeBS`), in the union merge case,
-  the ByteString has no defined encoding. It may have been written on Unix
-  and contain keys with invalid unicode in them. On windows, the union
-  merge code should probably check if it's valid utf-8, and if not,
-  abort the merge.
+  > This may be use to windows actually using utf-16, but git-annex uses
+  > utf-8 for filename encoding when on windows.
 
 * If interoperating with a git-annex repository from a unix system, it's
   possible for a key to contain some invalid utf-8, which means its filename

comment
diff --git a/doc/todo/provide_windows_build_with_MagicMime/comment_1_11adb1623314e8a43c50704c5e6e62d4._comment b/doc/todo/provide_windows_build_with_MagicMime/comment_1_11adb1623314e8a43c50704c5e6e62d4._comment
new file mode 100644
index 000000000..a86e88b10
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime/comment_1_11adb1623314e8a43c50704c5e6e62d4._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T16:55:36Z"
+ content="""
+It's a C library and it's not vendored into the haskell library as is often
+the case, so it has be be linked in. I don't know what would be involved in
+getting it installed on windows and bundled with git-annex in a usable state,
+and I'm a bit afraid to find out.
+
+WSL can be used to run linux git-annex on windows, so that's a way already
+available to use MagicMime.
+
+(The entire future of the windows port is currently TBD, it seems likely
+that it will go away and WSL be used, aside from the current bugs in WSL
+that make sqlite crash.)
+"""]]

comment
diff --git a/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_1_63d48db769863cdfe411404e8c26a399._comment b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_1_63d48db769863cdfe411404e8c26a399._comment
new file mode 100644
index 000000000..a3e2d62db
--- /dev/null
+++ b/doc/todo/assure_correct_names___40__and_values__41___for_special_remotes_parameters/comment_1_63d48db769863cdfe411404e8c26a399._comment
@@ -0,0 +1,30 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2020-01-06T16:12:01Z"
+ content="""
+There's a subtle backwards compatibility issue here: The stored config of a
+special remote is used when enabling it, so if an older version of
+git-annex is used to enable a remote, there might be a setting that it does
+not know about, or a value it doesn't understand. If that caused it to fail
+to enable the remote it wouldn't be possible to use it, at least w/o
+changing/removing the config.
+
+For example, autoenable=true did not used to be a config setting, but older
+versions of git-annex can still use remotes that have that.
+
+Another example is chunk=. While older versions of git-annex don't
+understand that, and so won't use chunks when storing/retrieving, 
+the newer git-annex falls back to getting the unchunked object.
+So things stored by the old git-annex can be retrieved by the new,
+but not vice-versa.
+
+Another example is S3's storageclass=. Older git-annex doesn't understand
+it, so uses the default storage class, but that behavior is interoperable
+with the new behavior.
+
+So the stored config of a remote should not be checked
+everytime the remote is instantiated, but only the new settings passed
+to initremote/enableremote. That will complicate the API, since currently
+the old and new config are combined together by enableremote.
+"""]]

Added a comment: Would you accept a patch implementing an annex.autocommit gitattribute?
diff --git a/doc/todo/wishlist__58___disable_automatic_commits/comment_4_e021de3647a00aa0259a247441de50eb._comment b/doc/todo/wishlist__58___disable_automatic_commits/comment_4_e021de3647a00aa0259a247441de50eb._comment
new file mode 100644
index 000000000..56be81e35
--- /dev/null
+++ b/doc/todo/wishlist__58___disable_automatic_commits/comment_4_e021de3647a00aa0259a247441de50eb._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="AdamSpiers"
+ avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
+ subject="Would you accept a patch implementing an annex.autocommit gitattribute?"
+ date="2020-01-06T16:12:54Z"
+ content="""
+In a semi-mythical future where I find the time to learn Haskell, in principle would you consider accepting a patch implementing my above suggestion of a new `annex.autocommit` gitattribute which offers finer control of autocommit than the current binary toggle?  It would be very useful to me, and maybe I'm not the only one.
+"""]]

diff --git a/doc/bugs/assistant_doesn__39__t_notice_changes_to_gitignore_files.mdwn b/doc/bugs/assistant_doesn__39__t_notice_changes_to_gitignore_files.mdwn
new file mode 100644
index 000000000..805aaf00f
--- /dev/null
+++ b/doc/bugs/assistant_doesn__39__t_notice_changes_to_gitignore_files.mdwn
@@ -0,0 +1,41 @@
+### Please describe the problem.
+
+Once the assistant daemon is started, it won't notice updates to any gitignore files.  This is probably because it spawns a long-lived `git check-ignore -z --stdin --verbose --non-matching` process which only reads gitignore files on startup.
+
+### What steps will reproduce the problem?
+
+- Start the assistant
+- Add a pattern to a relevant `.gitignore` or `.git/info/exclude`
+- (Just to be safe) wait for the assistant to commit and sync that change
+- Create a file matching the new ignore pattern
+- Observe the assistant committing the file to the repository despite it matching the new ignore pattern
+
+### What version of git-annex are you using? On what operating system?
+
+7.20191230-gef6d1e327 on openSUSE Leap 15.1
+
+### Please provide any additional information below.
+
+I assume that the obvious fix would be to watch for changes to gitignore files, and restart the `check-ignore` process accordingly.
+
+A workaround is to restart the assistant.  I've tested this and it works fine.
+
+Unsurprisingly the daemon log just shows the normal behaviour so it's probably not very useful, but including anyway just in case:
+
+[[!format sh """
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+Everything up-to-date
+[2020-01-06 15:40:21.863988735] Committer: Committing changes to git
+(recording state in git...)
+[2020-01-06 15:40:22.024703938] Pusher: Syncing with peer1 
+[2020-01-06 15:40:23.025650786] Committer: Committing changes to git
+(recording state in git...)
+To tor-annex::<censored>.onion:5227
+   841db50..d47d59e  master -> synced/master
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Yes, worked great for many years, but struggling to get the assistant to a point I can use it.  Main blockers are [[assistant_sometimes_removes_and_re-adds_whole_file]] and [[todo/wishlist__58___disable_automatic_commits]].

a new TODO for having windows build with MagicMime
diff --git a/doc/todo/provide_windows_build_with_MagicMime.mdwn b/doc/todo/provide_windows_build_with_MagicMime.mdwn
new file mode 100644
index 000000000..2f511dba3
--- /dev/null
+++ b/doc/todo/provide_windows_build_with_MagicMime.mdwn
@@ -0,0 +1,7 @@
+I forgot already why windows build is without having MagicMime support added, and failed to find a reason in the archives.
+But [there is demand](https://github.com/datalad/datalad/pull/3956#pullrequestreview-335604805) from Windows users! ;)
+
+Without such functionality we cannot consistently (cross-platform) use git-annex for many DataLad datasets, where text files (well -- soon "non-binary files") are configured to be committed directly to git.
+
+[[!meta author=yoh]]
+[[!tag projects/datalad]]

devblog
diff --git a/doc/devblog/day_615__new_year.mdwn b/doc/devblog/day_615__new_year.mdwn
new file mode 100644
index 000000000..f36668383
--- /dev/null
+++ b/doc/devblog/day_615__new_year.mdwn
@@ -0,0 +1,21 @@
+The release of git-annex with all the ByteString optimisations went out
+earlier this week. The Windows autobuilder was down and I didn't try to get
+it building on Windows, so fixed that today, luckily all those changes only
+broke a few bits of Windows-specific code.
+
+Also today, I added git-annex add --force-annex/--force-git options.
+These do the same thing as -c annex.largefiles=anthing etc, but 
+are easier to type and may avoid some tricky git behavior in some edge
+cases.
+
+I'd kind of like to get back to v8 this month and perhaps release it.
+There's a `v8` branch now, which as well as the sqlite changes adds a new
+annex.dotfiles config setting, and cleans up the special cases around
+adding dotfiles. Anyone not using git-annex to manage large dotfiles
+(or files in dotdirs) won't be impacted, but those who do will need to
+enable annex.dotfiles and configure annex.largefiles to match the dotfiles
+they want annexed. There is a risk that someone who's in the habit of
+running `git annex add .dotfile` to add them to the annex will be surprised
+when the new version adds them to git because they've not done the necessary
+configuration. I'm still mulling over whether this is an acceptable risk
+to mostly de-uglify and de-special-case dotfiles.

comment
diff --git a/doc/install/Windows/comment_9_2687128ff7aa4b6f29e8b880b5d4d13d._comment b/doc/install/Windows/comment_9_2687128ff7aa4b6f29e8b880b5d4d13d._comment
new file mode 100644
index 000000000..00fad5d1c
--- /dev/null
+++ b/doc/install/Windows/comment_9_2687128ff7aa4b6f29e8b880b5d4d13d._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 9"""
+ date="2020-01-01T19:04:08Z"
+ content="""
+@michael.fsp autobuilder was down. There's a build now for the most recent
+release.
+"""]]

comments
diff --git a/doc/git-annex-unused/comment_11_5c1ea2e8ac475d8c0ddca2c38f968b95._comment b/doc/git-annex-unused/comment_11_5c1ea2e8ac475d8c0ddca2c38f968b95._comment
new file mode 100644
index 000000000..9c0c594fc
--- /dev/null
+++ b/doc/git-annex-unused/comment_11_5c1ea2e8ac475d8c0ddca2c38f968b95._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2020-01-01T18:54:04Z"
+ content="""
+Generally if git-annex unused does not identify some object as unused, that
+you expected would be unused, it's because it found a branch that still
+uses that object. It might be some old branch (or tag) that you made and
+forgot about, it might be a remote tracking branch, either for a remote
+that you forgot to sync the changes to, or for an old remote that
+got removed but in a way that left the tracking branch around.
+annex.used-refspec can also pull in other, non-branch refs.
+"""]]
diff --git a/doc/todo/git-annex-cat/comment_4_8708bf0aef0025e2ed1a8b9d2a7f0112._comment b/doc/todo/git-annex-cat/comment_4_8708bf0aef0025e2ed1a8b9d2a7f0112._comment
new file mode 100644
index 000000000..7c602eb53
--- /dev/null
+++ b/doc/todo/git-annex-cat/comment_4_8708bf0aef0025e2ed1a8b9d2a7f0112._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2020-01-01T18:44:37Z"
+ content="""
+@Ilya_Shlyakhter, I'd assume:
+
+* some remotes would write to the named pipe
+* some remotes would overwrite it with a file
+* some remotes would open it, try to seek around as they do non-sequential
+  recieves, and hang or something
+* some remotes would maybe open and write to it, but would no longer be
+  able to resume interrupted transfers, since they would I guess see its
+  size as 0
+"""]]

fix comment location
diff --git a/doc/git-annex-unused/comment_1_029c4bee59e48806c9f3d83579fdbb90/comment_1_c4d994706f69024788e6653d02a09517._comment b/doc/git-annex-unused/comment_2_c4d994706f69024788e6653d02a09517._comment
similarity index 63%
rename from doc/git-annex-unused/comment_1_029c4bee59e48806c9f3d83579fdbb90/comment_1_c4d994706f69024788e6653d02a09517._comment
rename to doc/git-annex-unused/comment_2_c4d994706f69024788e6653d02a09517._comment
index 81ce17ca7..4c06422d2 100644
--- a/doc/git-annex-unused/comment_1_029c4bee59e48806c9f3d83579fdbb90/comment_1_c4d994706f69024788e6653d02a09517._comment
+++ b/doc/git-annex-unused/comment_2_c4d994706f69024788e6653d02a09517._comment
@@ -3,6 +3,6 @@
  subject="""comment 1"""
  date="2016-01-11T16:30:18Z"
  content="""
-The `--from` option can be used with any remote, whether it's a normal git
+@mark The `--from` option can be used with any remote, whether it's a normal git
 remote or a special remote.
 """]]