diff --git a/doc/news/version_7.20190129.mdwn b/doc/news/version_7.20190129.mdwn
deleted file mode 100644
index 7c4087e72..000000000
--- a/doc/news/version_7.20190129.mdwn
+++ /dev/null
@@ -1,12 +0,0 @@
-git-annex 7.20190129 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * initremote S3: When configured with versioning=yes, either ask the user
-     to enable bucket versioning, or auto-enable it when built with aws-0.22.
-   * enableremote S3: Do not let versioning=yes be set on existing remote,
-     because when git-annex lacks S3 version IDs for files stored in
-     the bucket, deleting them would cause data loss.
-   * S3: Detect when version=yes but an exported file lacks a S3 version ID,
-     and refuse to delete it, to avoid data loss.
-   * S3: Send a Content-Type header when storing objects in S3,
-     so exports to public buckets can be linked to from web pages.
-     (When git-annex is built with MagicMime support.)"""]]
\ No newline at end of file
diff --git a/doc/news/version_7.20190615.mdwn b/doc/news/version_7.20190615.mdwn
new file mode 100644
index 000000000..ed2240702
--- /dev/null
+++ b/doc/news/version_7.20190615.mdwn
@@ -0,0 +1,35 @@
+git-annex 7.20190615 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Fixed bug that caused git-annex to fail to add a file when another
+     git-annex process cleaned up the temp directory it was using.
+   * Makefile: Added install-completions to install target.
+   * Added the ability to run one job per CPU (core), by setting
+     annex.jobs=cpus, or using option --jobs=cpus or -Jcpus.
+   * Honor preferred content of a special remote when exporting trees to it;
+     unwanted files are filtered out of the tree that is exported.
+   * Importing from a special remote honors its preferred content too;
+     unwanted files are not imported. But, some preferred content
+     expressions can't be checked before files are imported, and trying to
+     import with such an expression will fail.
+   * Don't try to import .git directories from special remotes, because
+     git does not support storing git repositories inside a git repository.
+   * Improve shape of commit tree when importing from unversioned special
+     remotes.
+   * init: When the repository already has a description, don't change it.
+   * describe: When run with no description parameter it used to set
+     the description to "", now it will error out.
+   * Android: Improve installation process when the user's login shell is not
+     bash.
+   * When a remote is configured to be readonly, don't allow changing
+     what's exported to it.
+   * Renamed annex.security.allowed-http-addresses to
+     annex.security.allowed-ip-addresses because it is not really specific
+     to the http protocol, also limiting eg, git-annex's use of ftp.
+     The old name for the config will still work.
+   * Add back support for ftp urls, which was disabled as part of the fix for
+     security hole CVE-2018-10857 (except for configurations which enabled curl
+     and bypassed public IP address restrictions). Now it will work
+     if allowed by annex.security.allowed-ip-addresses.
+   * Avoid a delay at startup when concurrency is enabled and there are
+     rsync or gcrypt special remotes, which was caused by git-annex
+     opening a ssh connection to the remote too early."""]]
\ No newline at end of file

diff --git a/doc/bugs/same_uuid_for_remotes_with_same_path_but_different_name.mdwn b/doc/bugs/same_uuid_for_remotes_with_same_path_but_different_name.mdwn
new file mode 100644
index 000000000..0e5fda943
--- /dev/null
+++ b/doc/bugs/same_uuid_for_remotes_with_same_path_but_different_name.mdwn
@@ -0,0 +1,188 @@
+### Please describe the problem.
+As a result of the steps below git-annex end up in a state with multiple remotes (pointing to a USB). The crux is that each has a different name, but same UUID and same path. 
+
+All, except one of the remotes is marked as dead. During sync git-annex tries to sync to all of them, and quite surprisingly appears to be successfull. 
+
+The core problem I guess is that git-annex mistakes one remote for another.
+
+I was just experimenting with git-annex, but I think this could happen for real. You put stuff on a usb, you loose the usb, you mark the usb remote as dead, time passes, you put stuff on a different usb and type the same remote name. Bang, it gets the same UUID.
+
+Not knowing much about git-annex it seems that it generates the UUID before checking for name clash and renaming?
+
+### What steps will reproduce the problem?
+Init ~/annex with CLI. (I had version 7, but probably bug on 5 too). Then add an example file, in my case "The Rust Programming Language.pdf". Finally mount usb drive at /mnt/usb.
+
+    - 1) Start git-annex-webapp.
+    - 2) Create an unencrypted usb remote using the webapp. In my case its name was "smallusb".
+    - 3) Sync files to the drive. Verify all is ok.
+    - 4) Imitate drive loss. Delete annex folder on usb drive: rm -rf /mnt/usb/annex
+    - 5) Imitate drive loss. Change to ~/annex and mark smallusb (or smallusb1, smallusb2) as dead.
+    - 6) Stop git-annex-webapp.
+
+Repeat the steps 1-6 at least 2 times in a loop :)
+git-annex appends a number to smallusb, hence the "(or smallusb1, smallusb2)".
+
+### What version of git-annex are you using? On what operating system?
+Arch Linux, version 7.20190504-gf98e97669 downloaded as a binary from git-annex website.
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+-- .git/annex/daemon.log
+[2019-06-14 06:02:25.999232228] main: starting assistant version 7.20190504-gf98e97669
+[2019-06-14 06:02:26.023575476] Cronner: You should enable consistency checking to protect your data. 
+[2019-06-14 06:02:26.058871795] read: git ["config","--null","--list"]
+[2019-06-14 06:02:26.064856922] process done ExitSuccess
+[2019-06-14 06:02:26.065157988] Tor hidden service not enabled
+(scanning...) [2019-06-14 06:02:26.136886408] Watcher: Performing startup scan
+(started...) 
+gpg: assuming signed data in '/tmp/git-annex.tmpxo2fXm/info'
+gpg: Signature made Wed 08 May 2019 01:26:23 AM MSK
+gpg:                using DSA key 40055C6AFD2D526B2961E78F5EE1DBA789C809CB
+gpg: /tmp/git-annex-gpg.tmp28lKKy/trustdb.gpg: trustdb created
+gpg: Good signature from "git-annex distribution signing key (for Joey Hess) <id@joeyh.name>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg:          There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 4005 5C6A FD2D 526B 2961  E78F 5EE1 DBA7 89C8 09CB
+gcrypt: Repository not found: /mnt/usb/annex7
+(recording state in git...)
+warning: no common commits
+From /mnt/usb/annex7
+ * [new branch]      git-annex  -> usb/git-annex
+(merging usb/git-annex into git-annex...)
+(recording state in git...)
+[2019-06-14 06:02:45.460756129] main: Syncing with usb 
+To /mnt/usb/annex7
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+[2019-06-14 06:02:45.731237811] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
+[2019-06-14 06:02:45.736487217] process done ExitSuccess
+[2019-06-14 06:02:45.736545194] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
+[2019-06-14 06:02:45.741412826] process done ExitSuccess
+[2019-06-14 06:02:45.741780286] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..c19630d0a5366076f7da95a2a5fa2e3cfc51445e","--pretty=%H","-n1"]
+[2019-06-14 06:02:45.74683984] process done ExitSuccess
+[2019-06-14 06:02:45.746899694] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..874014cc02ad2383e612a5a200691adfb279fab6","--pretty=%H","-n1"]
+[2019-06-14 06:02:45.751857523] process done ExitSuccess
+[2019-06-14 06:02:45.752059154] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
+[2019-06-14 06:02:45.755772389] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
+[2019-06-14 06:02:45.760705559] read: git ["config","--null","--list"]
+[2019-06-14 06:02:45.766315456] process done ExitSuccess
+[2019-06-14 06:02:45.766455838] read: git ["config","--null","--list"]
+[2019-06-14 06:02:45.771340045] process done ExitSuccess
+[2019-06-14 06:02:45.77170679] read: git ["config","--null","--list"]
+[2019-06-14 06:02:45.776625026] process done ExitSuccess
+[2019-06-14 06:02:45.776753592] read: git ["config","--null","--list"]
+[2019-06-14 06:02:45.782541882] process done ExitSuccess
+
+[2019-06-14 06:02:45.784051825] read: rsync ["--progress","--inplace","--perms",".git/annex/objects/XV/v2/SHA256E-s11122109--bbf1b9a6d9bc3be5515e3114aea5a7cf7a93ea406bbe53cf37eef81ba578b73f.pdf/SHA256E-s11122109--bbf1b9a6d9bc3be5515e3114aea5a7cf7a93ea406bbe53cf37eef81ba578b73f.pdf","../../../mnt/usb/annex7/annex/tmp/SHA256E-s11122109--bbf1b9a6d9bc3be5515e3114aea5a7cf7a93ea406bbe53cf37eef81ba578b73f.pdf"]
+SHA256E-s11122109--bbf1b9a6d9bc3be5515e3114aea5a7cf7a93ea406bbe53cf37eef81ba578b73f.pdf
+     11,122,109 100%  480.71MB/s    0:00:00 (xfr#1, to-chk=0/1)
+[2019-06-14 06:02:46.647226687] process done ExitSuccess
+(checksum...) [2019-06-14 06:02:46.732000864] chat: git ["--git-dir=../../../mnt/usb/annex7","--literal-pathspecs","--literal-pathspecs","cat-file","--batch"]
+[2019-06-14 06:02:46.732316108] chat: git ["--git-dir=../../../mnt/usb/annex7","--literal-pathspecs","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
+[2019-06-14 06:02:46.737658146] process done ExitSuccess
+[2019-06-14 06:02:46.737894862] process done ExitSuccess
+[2019-06-14 06:02:46.738498152] Transferrer: Uploaded The Rust ..guage.pdf
+[2019-06-14 06:02:46.739276009] Pusher: Syncing with usb, smallusb2, smallusb1, smallusb 
+(recording state in git...)
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at bb6dbcca16e11c47a129063b29fc26e1af0b6529 but expected c19630d0a5366076f7da95a2a5fa2e3cfc51445e        
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at bb6dbcca16e11c47a129063b29fc26e1af0b6529 but expected c19630d0a5366076f7da95a2a5fa2e3cfc51445e        
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at bb6dbcca16e11c47a129063b29fc26e1af0b6529 but expected c19630d0a5366076f7da95a2a5fa2e3cfc51445e        
+To /mnt/usb/annex7
+   c19630d..bb6dbcc  git-annex -> synced/git-annex
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+From /mnt/usb/annex7
+   874014c..bb6dbcc  synced/git-annex -> smallusb2/synced/git-annex
+From /mnt/usb/annex7
+   874014c..bb6dbcc  git-annex        -> smallusb1/git-annex
+   874014c..bb6dbcc  synced/git-annex -> smallusb1/synced/git-annex
+From /mnt/usb/annex7
+   874014c..bb6dbcc  git-annex        -> smallusb/git-annex
+   874014c..bb6dbcc  synced/git-annex -> smallusb/synced/git-annex
+Everything up-to-date
+Everything up-to-date
+Everything up-to-date
+[2019-06-14 06:03:26.105979618] Pusher: Syncing with smallusb3, smallusb2, smallusb1, smallusb 
+(recording state in git...)
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at 63d983955552f44ad51550dd017da9d3e089bb37 but expected bb6dbcca16e11c47a129063b29fc26e1af0b6529        
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at 63d983955552f44ad51550dd017da9d3e089bb37 but expected bb6dbcca16e11c47a129063b29fc26e1af0b6529        
+remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at 63d983955552f44ad51550dd017da9d3e089bb37 but expected bb6dbcca16e11c47a129063b29fc26e1af0b6529        
+To /mnt/usb/annex7
+   bb6dbcc..63d9839  git-annex -> synced/git-annex
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+To /mnt/usb/annex7
+ ! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
+error: failed to push some refs to '/mnt/usb/annex7'
+From /mnt/usb/annex7
+   c19630d..63d9839  git-annex        -> smallusb3/git-annex
+   bb6dbcc..63d9839  synced/git-annex -> smallusb3/synced/git-annex
+From /mnt/usb/annex7
+   bb6dbcc..63d9839  git-annex        -> smallusb2/git-annex
+   bb6dbcc..63d9839  synced/git-annex -> smallusb2/synced/git-annex
+From /mnt/usb/annex7
+   bb6dbcc..63d9839  git-annex        -> smallusb1/git-annex
+   bb6dbcc..63d9839  synced/git-annex -> smallusb1/synced/git-annex
+Everything up-to-date
+Everything up-to-date
+Everything up-to-date
+[2019-06-14 06:04:26.120035368] Pusher: Syncing with smallusb3, smallusb2, smallusb1, smallusb 
+Everything up-to-date
+Everything up-to-date
+Everything up-to-date
+Everything up-to-date
+
+
+-- .git/config
+
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = false
+	logallrefupdates = true
+[annex]
+	uuid = 2517eda3-c212-486e-a3a8-ffc085b842ba
+	version = 7
+	diskreserve = 1 megabyte
+	autoupgrade = ask
+	debug = true
+[filter "annex"]
+	smudge = git-annex smudge -- %f
+	clean = git-annex smudge --clean -- %f
+[remote "smallusb"]
+	url = /mnt/usb/annex7
+	annex-uuid = c52d9a9f-9362-4698-bb1b-fee40779de4c
+	fetch = +refs/heads/*:refs/remotes/smallusb/*
+[remote "smallusb1"]
+	url = /mnt/usb/annex7
+	annex-uuid = c52d9a9f-9362-4698-bb1b-fee40779de4c
+	fetch = +refs/heads/*:refs/remotes/smallusb1/*
+[remote "smallusb2"]
+	url = /mnt/usb/annex7
+	annex-uuid = c52d9a9f-9362-4698-bb1b-fee40779de4c
+	fetch = +refs/heads/*:refs/remotes/smallusb2/*
+[remote "smallusb3"]
+	url = /mnt/usb/annex7
+	annex-uuid = c52d9a9f-9362-4698-bb1b-fee40779de4c
+	fetch = +refs/heads/*:refs/remotes/smallusb3/*
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+I think this is genius software and just love the docs, the effort put into it shines through! :)

diff --git a/CHANGELOG b/CHANGELOG
index 1b82d5160..ec3ec44c2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -30,6 +30,9 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     security hole CVE-2018-10857 (except for configurations which enabled curl
     and bypassed public IP address restrictions). Now it will work
     if allowed by annex.security.allowed-ip-addresses.
+  * Avoid a delay at startup when concurrency is enabled and there are
+    rsync or gcrypt special remotes, which was caused by git-annex
+    opening a ssh connection to the remote too early.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/Remote/GCrypt.hs b/Remote/GCrypt.hs
index 38f7e5155..22a88cd6b 100644
--- a/Remote/GCrypt.hs
+++ b/Remote/GCrypt.hs
@@ -150,12 +150,12 @@ gen' r u c gc = do
 			{ displayProgress = False }
 		| otherwise = specialRemoteCfg c
 
-rsyncTransportToObjects :: Git.Repo -> RemoteGitConfig -> Annex ([CommandParam], String)
+rsyncTransportToObjects :: Git.Repo -> RemoteGitConfig -> Annex (Annex [CommandParam], String)
 rsyncTransportToObjects r gc = do
 	(rsynctransport, rsyncurl, _) <- rsyncTransport r gc
 	return (rsynctransport, rsyncurl ++ "/annex/objects")
 
-rsyncTransport :: Git.Repo -> RemoteGitConfig -> Annex ([CommandParam], String, AccessMethod)
+rsyncTransport :: Git.Repo -> RemoteGitConfig -> Annex (Annex [CommandParam], String, AccessMethod)
 rsyncTransport r gc
 	| "ssh://" `isPrefixOf` loc = sshtransport $ break (== '/') $ drop (length "ssh://") loc
 	| "//:" `isInfixOf` loc = othertransport
@@ -168,9 +168,10 @@ rsyncTransport r gc
 			then drop 3 path
 			else path
 		let sshhost = either error id (mkSshHost host)
-		opts <- sshOptions ConsumeStdin (sshhost, Nothing) gc []
-		return (rsyncShell $ Param "ssh" : opts, fromSshHost sshhost ++ ":" ++ rsyncpath, AccessShell)
-	othertransport = return ([], loc, AccessDirect)
+		let mkopts = rsyncShell . (Param "ssh" :) 
+			<$> sshOptions ConsumeStdin (sshhost, Nothing) gc []
+		return (mkopts, fromSshHost sshhost ++ ":" ++ rsyncpath, AccessShell)
+	othertransport = return (pure [], loc, AccessDirect)
 
 noCrypto :: Annex a
 noCrypto = giveup "cannot use gcrypt remote without encryption enabled"
@@ -263,14 +264,15 @@ setupRepo gcryptid r
 		dummycfg <- liftIO dummyRemoteGitConfig
 		(rsynctransport, rsyncurl, _) <- rsyncTransport r dummycfg
 		let tmpconfig = tmp </> "config"
-		void $ liftIO $ rsync $ rsynctransport ++
+		opts <- rsynctransport
+		void $ liftIO $ rsync $ opts ++
 			[ Param $ rsyncurl ++ "/config"
 			, Param tmpconfig
 			]
 		liftIO $ do
 			void $ Git.Config.changeFile tmpconfig coreGCryptId gcryptid
 			void $ Git.Config.changeFile tmpconfig denyNonFastForwards (Git.Config.boolConfig False)
-		ok <- liftIO $ rsync $ rsynctransport ++
+		ok <- liftIO $ rsync $ opts ++
 			[ Param "--recursive"
 			, Param $ tmp ++ "/"
 			, Param rsyncurl
@@ -456,9 +458,10 @@ getGCryptId fast r gc
 getConfigViaRsync :: Git.Repo -> RemoteGitConfig -> Annex (Either SomeException (Git.Repo, String))
 getConfigViaRsync r gc = do
 	(rsynctransport, rsyncurl, _) <- rsyncTransport r gc
+	opts <- rsynctransport
 	liftIO $ do
 		withTmpFile "tmpconfig" $ \tmpconfig _ -> do
-			void $ rsync $ rsynctransport ++
+			void $ rsync $ opts ++
 				[ Param $ rsyncurl ++ "/config"
 				, Param tmpconfig
 				]
diff --git a/Remote/Rsync.hs b/Remote/Rsync.hs
index f3aee924a..1845c5aeb 100644
--- a/Remote/Rsync.hs
+++ b/Remote/Rsync.hs
@@ -110,15 +110,18 @@ gen r u c gc = do
 		-- Rsync displays its own progress.
 		{ displayProgress = False }
 
-genRsyncOpts :: RemoteConfig -> RemoteGitConfig -> [CommandParam] -> RsyncUrl -> RsyncOpts
+genRsyncOpts :: RemoteConfig -> RemoteGitConfig -> Annex [CommandParam] -> RsyncUrl -> RsyncOpts
 genRsyncOpts c gc transport url = RsyncOpts
 	{ rsyncUrl = url
-	, rsyncOptions = transport ++ opts []
-	, rsyncUploadOptions = transport ++ opts (remoteAnnexRsyncUploadOptions gc)
-	, rsyncDownloadOptions = transport ++ opts (remoteAnnexRsyncDownloadOptions gc)
+	, rsyncOptions = appendtransport $ opts []
+	, rsyncUploadOptions = appendtransport $
+		opts (remoteAnnexRsyncUploadOptions gc)
+	, rsyncDownloadOptions = appendtransport $
+		opts (remoteAnnexRsyncDownloadOptions gc)
 	, rsyncShellEscape = (yesNo =<< M.lookup "shellescape" c) /= Just False
 	}
   where
+	appendtransport l = (++ l) <$> transport
 	opts specificopts = map Param $ filter safe $
 		remoteAnnexRsyncOptions gc ++ specificopts
 	safe opt
@@ -129,23 +132,23 @@ genRsyncOpts c gc transport url = RsyncOpts
 		| opt == "--delete-excluded" = False
 		| otherwise = True
 
-rsyncTransport :: RemoteGitConfig -> RsyncUrl -> Annex ([CommandParam], RsyncUrl)
+rsyncTransport :: RemoteGitConfig -> RsyncUrl -> Annex (Annex [CommandParam], RsyncUrl)
 rsyncTransport gc url
 	| rsyncUrlIsShell url =
-		(\rsh -> return (rsyncShell rsh, url)) =<<
+		(\transport -> return (rsyncShell <$> transport, url)) =<<
 		case fromNull ["ssh"] (remoteAnnexRsyncTransport gc) of
 			"ssh":sshopts -> do
 				let (port, sshopts') = sshReadPort sshopts
 				    userhost = either error id $ mkSshHost $ 
 				    	takeWhile (/= ':') url
-				(Param "ssh":) <$> sshOptions ConsumeStdin
+				return $ (Param "ssh":) <$> sshOptions ConsumeStdin
 					(userhost, port) gc
 					(map Param $ loginopt ++ sshopts')
-			"rsh":rshopts -> return $ map Param $ "rsh" :
+			"rsh":rshopts -> return $ pure $ map Param $ "rsh" :
 				loginopt ++ rshopts
 			rsh -> giveup $ "Unknown Rsync transport: "
 				++ unwords rsh
-	| otherwise = return ([], url)
+	| otherwise = return (pure [], url)
   where
 	login = case separate (=='@') url of
 		(_h, "") -> Nothing
@@ -232,9 +235,10 @@ remove o k = removeGeneric o includes
 removeGeneric :: RsyncOpts -> [String] -> Annex Bool
 removeGeneric o includes = do
 	ps <- sendParams
+	opts <- rsyncOptions o
 	withRsyncScratchDir $ \tmp -> liftIO $ do
 		{- Send an empty directory to rysnc to make it delete. -}
-		rsync $ rsyncOptions o ++ ps ++
+		rsync $ opts ++ ps ++
 			map (\s -> Param $ "--include=" ++ s) includes ++
 			[ Param "--exclude=*" -- exclude everything else
 			, Param "--quiet", Param "--delete", Param "--recursive"
@@ -249,14 +253,14 @@ checkKey r o k = do
 	checkPresentGeneric o (rsyncUrls o k)
 
 checkPresentGeneric :: RsyncOpts -> [RsyncUrl] -> Annex Bool
-checkPresentGeneric o rsyncurls =
+checkPresentGeneric o rsyncurls = do
+	opts <- rsyncOptions o
 	-- note: Does not currently differentiate between rsync failing
 	-- to connect, and the file not being present.
 	untilTrue rsyncurls $ \u -> 
 		liftIO $ catchBoolIO $ do
 			withQuietOutput createProcessSuccess $
-				proc "rsync" $ toCommand $
-					rsyncOptions o ++ [Param u]
+				proc "rsync" $ toCommand $ opts ++ [Param u]
 			return True
 
 storeExportM :: RsyncOpts -> FilePath -> Key -> ExportLocation -> MeterUpdate -> Annex Bool
@@ -341,13 +345,14 @@ showResumable a = ifM a
 rsyncRemote :: Direction -> RsyncOpts -> Maybe MeterUpdate -> [CommandParam] -> Annex Bool
 rsyncRemote direction o m params = do
 	showOutput -- make way for progress bar
+	opts <- mkopts
+	let ps = opts ++ Param "--progress" : params
 	case m of
 		Nothing -> liftIO $ rsync ps
 		Just meter -> do
 			oh <- mkOutputHandler
 			liftIO $ rsyncProgress oh meter ps
   where
-	ps = opts ++ Param "--progress" : params
-	opts
+	mkopts
 		| direction == Download = rsyncDownloadOptions o
 		| otherwise = rsyncUploadOptions o
diff --git a/Remote/Rsync/RsyncUrl.hs b/Remote/Rsync/RsyncUrl.hs
index eae712805..4c2f10843 100644
--- a/Remote/Rsync/RsyncUrl.hs
+++ b/Remote/Rsync/RsyncUrl.hs
@@ -25,9 +25,9 @@ type RsyncUrl = String
 
 data RsyncOpts = RsyncOpts
 	{ rsyncUrl :: RsyncUrl
-	, rsyncOptions :: [CommandParam]
-	, rsyncUploadOptions :: [CommandParam]
-	, rsyncDownloadOptions :: [CommandParam]
+	, rsyncOptions :: Annex [CommandParam]
+	, rsyncUploadOptions :: Annex [CommandParam]
+	, rsyncDownloadOptions :: Annex [CommandParam]
 	, rsyncShellEscape :: Bool
 }
 
diff --git a/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn b/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn
index ae9806ae3..f2c8b3b20 100644
--- a/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn

(Diff truncated)

diff --git a/doc/devblog/day_593__partial_success.mdwn b/doc/devblog/day_593__partial_success.mdwn
new file mode 100644
index 000000000..9a0335b68
--- /dev/null
+++ b/doc/devblog/day_593__partial_success.mdwn
@@ -0,0 +1,12 @@
+Finished the refactoring that I had started on Thursday. This was only
+a partial success, because it didn't result in the speedup to -J that I had
+hoped for. The slow start with -J turns out to not be caused by concurrency
+overhead at all, but a bug,
+[[bugs/rsync_and_gcrypt_special_remotes_make_-J_slow]].
+
+What was successful is that I got rid of the oldest implementation wart in
+git-annex, the implicitMessages state. And, I made progress toward
+separately parallizing checksum verification.
+
+This refactoring is still cooking in the `starting` branch and will be
+merged after the next release.

diff --git a/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn b/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn
new file mode 100644
index 000000000..ae9806ae3
--- /dev/null
+++ b/doc/bugs/rsync_and_gcrypt_special_remotes_make_-J_slow.mdwn
@@ -0,0 +1,10 @@
+When a rsync or gcrypt special remote is enabled, all git-annex commands
+with -J become slow, even those that don't access the remote.
+
+The problem is that Remote.Rsync.gen and Remote.Gcrypt.gen 
+call rsyncTransport which calls sshOptions. 
+When concurrency is enabled, that blocks for a ssh connection to be set up.
+Even though the ssh connection will often not be used.
+
+It should be possible for it not to block until the ssh connection is used.
+--[[Joey]]

diff --git a/doc/todo/parallel_possibilities.mdwn b/doc/todo/parallel_possibilities.mdwn
index bf9d2c72c..8ce72268e 100644
--- a/doc/todo/parallel_possibilities.mdwn
+++ b/doc/todo/parallel_possibilities.mdwn
@@ -21,7 +21,3 @@ are still some things that could be improved, tracked here:
   all that needs to be done is make checksum verification be done as the
   cleanup action. Currently, it's bundled into the same action that
   transfers content.
-
-* Using -J can sometimes lead to a slowdown while a rsync special remote
-  runs Remote.Rsync.rsyncTransport, which sets up a ssh connection to the
-  remote. This is done even when the remote is not used.

diff --git a/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn b/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn
index 734678adb..f5f3f5d9d 100644
--- a/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn
+++ b/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn
@@ -1 +1,14 @@
 If a repo is cloned using `git clone --single-branch --depth 1`, `git-annex-init` and `git-annex-sync` do not seem to correctly fetch the `git-annex` and `synced/git-annex` branches.  `git-annex-info` does not list remotes that were known at the cloned repo.
+
+> This is not a bug. You have cloned a repository without cloning the
+> git-annex branch, so as far as git-annex can *possibly* know, this is a
+> git repository in which git-annex has never been used before.
+> 
+> As soon as you fetch the git-annex branch from origin, git-annex will
+> know all the information that you expected it to know. So all you have to
+> do is: `git fetch origin git-annex`
+> 
+> There is no possible change I can make that will prevent or amelorate
+> this particular means of shooting yourself in the foot.
+> 
+> [[done]] --[[Joey]] 

diff --git a/doc/forum/status_of_v7.mdwn b/doc/forum/status_of_v7.mdwn
new file mode 100644
index 000000000..ae516d746
--- /dev/null
+++ b/doc/forum/status_of_v7.mdwn
@@ -0,0 +1,3 @@
+What is the current status of v7 repositories?  What has been people's experience in using them?
+
+`git-annex` still defaults to v5, so it sounds like v7 is still in beta?

diff --git a/Annex/Drop.hs b/Annex/Drop.hs
index 40b6add14..3fd301072 100644
--- a/Annex/Drop.hs
+++ b/Annex/Drop.hs
@@ -47,8 +47,8 @@ type Reason = String
  - In direct mode, all associated files are checked, and only if all
  - of them are unwanted are they dropped.
  -
- - The runner is used to run commands, and so can be either callCommand
- - or commandAction.
+ - The runner is used to run CommandStart sequentially, it's typically 
+ - callCommandAction.
  -}
 handleDropsFrom :: [UUID] -> [Remote] -> Reason -> Bool -> Key -> AssociatedFile -> [VerifiedCopy] -> (CommandStart -> CommandCleanup) -> Annex ()
 handleDropsFrom locs rs reason fromhere key afile preverified runner = do
diff --git a/CHANGELOG b/CHANGELOG
index 54004d731..12f0ee388 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -34,6 +34,10 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     in a separate queue than the main action queue. This can make some
     commands faster, because less time is spent on bookkeeping in
     between each file transfer.
+  * When a command like git-annex get skips over a lot of files
+    that it does not need to do anything with, the -J switch used to slow
+    it down significantly due to unncessary concurrenty overhead.
+    That slowdown has been fixed.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
index 3588d2ebd..8530c188c 100644
--- a/CmdLine/Action.hs
+++ b/CmdLine/Action.hs
@@ -1,4 +1,4 @@
-{- git-annex command-line actions
+{- git-annex command-line actions and concurrency
  -
  - Copyright 2010-2019 Joey Hess <id@joeyh.name>
  -
@@ -54,33 +54,70 @@ commandActions = mapM_ commandAction
  - This should only be run in the seek stage.
  -}
 commandAction :: CommandStart -> Annex ()
-commandAction a = Annex.getState Annex.concurrency >>= \case
-	NonConcurrent -> run
+commandAction start = Annex.getState Annex.concurrency >>= \case
+	NonConcurrent -> void $ includeCommandAction start
 	Concurrent n -> runconcurrent n
 	ConcurrentPerCpu -> runconcurrent =<< liftIO getNumProcessors
   where
-	run = void $ includeCommandAction a
-
 	runconcurrent n = do
 		tv <- Annex.getState Annex.workers
 		workerst <- waitWorkerSlot n (== PerformStage) tv
+		aid <- liftIO $ async $ snd <$> Annex.run workerst
+			(concurrentjob workerst)
+		liftIO $ atomically $ do
+			pool <- takeTMVar tv
+			let !pool' = addWorkerPool (ActiveWorker aid PerformStage) pool
+			putTMVar tv pool'
 		void $ liftIO $ forkIO $ do
-			aid <- async $ snd <$> Annex.run workerst
-				(inOwnConsoleRegion (Annex.output workerst) run)
-			atomically $ do
-				pool <- takeTMVar tv
-				let !pool' = addWorkerPool (ActiveWorker aid PerformStage) pool
-				putTMVar tv pool'
-			-- There won't usually be exceptions because the
-			-- async is running includeCommandAction, which
-			-- catches exceptions. Just in case, avoid
-			-- stalling by using the original workerst.
+			-- accountCommandAction will usually catch
+			-- exceptions. Just in case, fall back to the
+			-- original workerst.
 			workerst' <- either (const workerst) id
 				<$> waitCatch aid
 			atomically $ do
 				pool <- takeTMVar tv
 				let !pool' = deactivateWorker pool aid workerst'
 				putTMVar tv pool'
+	
+	concurrentjob workerst = start >>= \case
+		Nothing -> noop
+		Just (startmsg, perform) ->
+			concurrentjob' workerst startmsg perform
+	
+	concurrentjob' workerst startmsg perform = case mkActionItem startmsg of
+		OnlyActionOn k _ -> ensureOnlyActionOn k $
+			-- If another job performed the same action while we
+			-- waited, there may be nothing left to do, so re-run
+			-- the start stage to see if it still wants to do
+			-- something.
+			start >>= \case
+				Just (startmsg', perform') ->
+					case mkActionItem startmsg' of
+						OnlyActionOn k' _ | k' /= k ->
+							concurrentjob' workerst startmsg' perform'
+						_ -> mkjob workerst startmsg' perform'
+				Nothing -> noop
+		_ -> mkjob workerst startmsg perform
+	
+	mkjob workerst startmsg perform = 
+		inOwnConsoleRegion (Annex.output workerst) $
+			void $ accountCommandAction $
+				performconcurrent startmsg perform
+
+	-- Like callCommandAction, but the start stage has already run,
+	-- and the worker thread's stage is changed before starting the
+	-- cleanup action.
+	performconcurrent startmsg perform = do
+		showStartMessage startmsg
+		perform >>= \case
+			Just cleanup -> do
+				changeStageTo CleanupStage
+				r <- cleanup
+				implicitMessage (showEndResult r)
+				return r
+			Nothing -> do
+				implicitMessage (showEndResult False)
+				return False
 
 -- | Wait until there's an idle worker in the pool, remove it from the
 -- pool, and return its state.
@@ -138,16 +175,17 @@ finishCommandActions = do
 		swapTMVar tv UnallocatedWorkerPool
 	case pool of
 		UnallocatedWorkerPool -> noop
-		WorkerPool l -> forM_ (mapMaybe workerAsync l) $ \aid -> 
+		WorkerPool l -> forM_ (mapMaybe workerAsync l) $ \aid ->
 			liftIO (waitCatch aid) >>= \case
 				Left _ -> noop
 				Right st -> mergeState st
 
 {- Changes the current thread's stage in the worker pool.
  -
- - An idle worker with the desired stage is found in the pool
- - (waiting if necessary for one to become idle)
- - and the stages of it and the current thread are swapped.
+ - The pool needs to continue to contain the same number of worker threads
+ - for each stage. So, an idle worker with the desired stage is found in 
+ - the pool (waiting if necessary for one to become idle), and the stages
+ - of it and the current thread are swapped.
  -}
 changeStageTo :: WorkerStage -> Annex ()
 changeStageTo newstage = do
@@ -168,23 +206,26 @@ changeStageTo newstage = do
 
 {- Like commandAction, but without the concurrency. -}
 includeCommandAction :: CommandStart -> CommandCleanup
-includeCommandAction a = account =<< tryNonAsync (callCommandAction a)
-  where
-	account (Right True) = return True
-	account (Right False) = incerr
-	account (Left err) = case fromException err of
+includeCommandAction = accountCommandAction . callCommandAction
+
+accountCommandAction :: CommandCleanup -> CommandCleanup
+accountCommandAction a = tryNonAsync a >>= \case
+	Right True -> return True
+	Right False -> incerr
+	Left err -> case fromException err of
 		Just exitcode -> liftIO $ exitWith exitcode
 		Nothing -> do
 			toplevelWarning True (show err)
 			implicitMessage showEndFail
 			incerr
+  where
 	incerr = do
 		Annex.incError
 		return False
 
 {- Runs a single command action through the start, perform and cleanup
- - stages, without catching errors. Useful if one command wants to run
- - part of another command. -}
+ - stages, without catching errors and without incrementing error counter.
+ - Useful if one command wants to run part of another command. -}
 callCommandAction :: CommandStart -> CommandCleanup
 callCommandAction = fromMaybe True <$$> callCommandAction' 
 
@@ -203,9 +244,7 @@ callCommandActionQuiet start =
 			showStartMessage startmsg
 			perform >>= \case
 				Nothing -> return (Just False)
-				Just cleanup -> do
-					changeStageTo CleanupStage
-					Just <$> cleanup
+				Just cleanup -> Just <$> cleanup
 
 {- Do concurrent output when that has been requested. -}
 allowConcurrentOutput :: Annex a -> Annex a
@@ -253,22 +292,12 @@ allowConcurrentOutput a = do
 			liftIO $ setNumCapabilities n
 
 {- Ensures that only one thread processes a key at a time.
- - Other threads will block until it's done. -}
-{-
-onlyActionOn :: Key -> CommandStart -> CommandStart
-onlyActionOn k a = onlyActionOn' k run

(Diff truncated)

diff --git a/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn b/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn
new file mode 100644
index 000000000..734678adb
--- /dev/null
+++ b/doc/bugs/git-clone_--single-branch_confuses_git-annex-init_and_git-annex-sync.mdwn
@@ -0,0 +1 @@
+If a repo is cloned using `git clone --single-branch --depth 1`, `git-annex-init` and `git-annex-sync` do not seem to correctly fetch the `git-annex` and `synced/git-annex` branches.  `git-annex-info` does not list remotes that were known at the cloned repo.

diff --git a/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_2_89b9f0cabdfac72ce6b0b9461672705f._comment b/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_2_89b9f0cabdfac72ce6b0b9461672705f._comment
new file mode 100644
index 000000000..220860979
--- /dev/null
+++ b/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_2_89b9f0cabdfac72ce6b0b9461672705f._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="anthony@ad39673d230d75cbfd19d2757d754030049c7673"
+ nickname="anthony"
+ avatar="http://cdn.libravatar.org/avatar/05b48b72766177b3b0a6ff4afdb70790"
+ subject="comment 2"
+ date="2019-06-10T16:52:23Z"
+ content="""
+I worked around that one (I think I fully checked it out to Termux's $HOME, then moved it to sdcard. If I recall correctly, adjusted mode failed with more permission issues. But direct mode works... (I don't have the device in front of me, but it may be an actual SD card, not the fake sdcard. Which of course doesn't support permissions either.)
+
+I wonder if it'd be easy enough to do a LD_PRELOAD hack to \"fix\" chmod. 
+"""]]

diff --git a/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn b/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn
index b51f09ea7..d62be0401 100644
--- a/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn
+++ b/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn
@@ -2,6 +2,8 @@
 
 Attempting to use git-annex addurl on Android 9 (Pixel 3a) fails with "ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)"
 
+To be clear, this is the installed the current way, inside of Termux. Not the obsolete apk.
+
 [[!format sh """
 # If you can, paste a complete transcript of the problem occurring here.
 # If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

diff --git a/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn b/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn
new file mode 100644
index 000000000..b51f09ea7
--- /dev/null
+++ b/doc/bugs/Android__58___addurl__58___Network.BSD.getProtocolByName__58___does_not_exist___40__no_such_protocol_name__58___tcp__41__.mdwn
@@ -0,0 +1,32 @@
+### Please describe the problem.
+
+Attempting to use git-annex addurl on Android 9 (Pixel 3a) fails with "ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)"
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+$ git-annex addurl --file "Thermopen Mk4.pdf" "http://www.thermoworks.com/pdf/thermapen_mk4_operating_instructions_a.pdf"
+addurl http://www.thermoworks.com/pdf/thermapen_mk4_operating_instructions_a.pdf
+download failed: ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)
+failed
+git-annex: addurl: 1 failed
+
+$ git-annex version
+git-annex version: 7.20190508-g8b44548d0
+build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
+operating system: linux aarch64
+supported repository versions: 5 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+local repository version: 7
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Yep, I continue to use git-annex a bunch of places. And this bug isn't a huge deal, wget followed by add worked fine.
+

diff --git a/doc/bugs/indeterminite_preferred_content_state_for_duplicated_file/comment_1_36c2d9bb2de7365c898e886149fb4bcd._comment b/doc/bugs/indeterminite_preferred_content_state_for_duplicated_file/comment_1_36c2d9bb2de7365c898e886149fb4bcd._comment
new file mode 100644
index 000000000..cdf9e934d
--- /dev/null
+++ b/doc/bugs/indeterminite_preferred_content_state_for_duplicated_file/comment_1_36c2d9bb2de7365c898e886149fb4bcd._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="kirelagin@6d93475882c55a329fedae6be1971868a775ec7e"
+ nickname="kirelagin"
+ avatar="http://cdn.libravatar.org/avatar/325af9a946cb4337c6640f0e95044be1"
+ subject="Workaround?"
+ date="2019-06-08T13:03:49Z"
+ content="""
+Just got bitten by this. I guess it is not a very rare situation, so I was wondering, are there any known workarounds?
+"""]]

diff --git a/doc/bugs/exporttree_interaction_with_adjusted_branch.mdwn b/doc/bugs/exporttree_interaction_with_adjusted_branch.mdwn
new file mode 100644
index 000000000..e3f603e77
--- /dev/null
+++ b/doc/bugs/exporttree_interaction_with_adjusted_branch.mdwn
@@ -0,0 +1,9 @@
+When on an adjusted branch, git annex sync --content with an exportree
+remote that  should commit changes, merge back to master, and then when
+the remote's annex-tracking-branch is master, the updated master should be
+exported to the remote.
+
+I'm told there's a one sync lag in changes reaching the remote. Ie, it's
+exporting to the remote before merging adjusted/master into master, or
+something like that. Have not verified and it could be some other root
+cause than that. --[[Joey]]

diff --git a/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment b/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment
deleted file mode 100644
index c0cca8eb5..000000000
--- a/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment
+++ /dev/null
@@ -1,11 +0,0 @@
-[[!comment format=mdwn
- username="joer962500"
- avatar="http://cdn.libravatar.org/avatar/573a6c0ca60e11a8420ce772cd2e80a7"
- subject="asus tablet stuck loading screen"
- date="2019-05-31T07:24:27Z"
- content="""
-I want to share issues related to Asus tablet. Sometimes the issue may occur while loading. Recover the issue with just one click, follow this quick fix guide <a href=\"=https://errorcode0x.com/fix-asus-tablet-stuck-at-loading-screen/\">asus tablet stuck loading screen</a> and get our job done easily.
-
-
-
-"""]]

diff --git a/doc/devblog/day_592__refactoring_start_messages.mdwn b/doc/devblog/day_592__refactoring_start_messages.mdwn
new file mode 100644
index 000000000..c654326e8
--- /dev/null
+++ b/doc/devblog/day_592__refactoring_start_messages.mdwn
@@ -0,0 +1,4 @@
+I have a 2500 line patch on the `starting` branch that refactors how start
+messages get displayed. Prerequisite for faster parallel starts. This
+touched every single command, and quite a few needed non-trivial changes, 
+so it took all day to get it to even compile.

diff --git a/doc/todo/parallel_possibilities.mdwn b/doc/todo/parallel_possibilities.mdwn
index 0baa7484a..7895cbaba 100644
--- a/doc/todo/parallel_possibilities.mdwn
+++ b/doc/todo/parallel_possibilities.mdwn
@@ -22,6 +22,9 @@ are still some things that could be improved, tracked here:
   cleanup action. Currently, it's bundled into the same action that
   transfers content.
 
+* onlyActionOn collapses the cleanup action into the start action,
+  and so prevents use of the separate cleanup queue.
+
 * Don't parallelize start stage actions. They are supposed to run fast,
   and often a huge number of them don't print out anything. The overhead of
   bookkeeping for parallizing those swamps the benefit of parallelizing by

diff --git a/doc/devblog/day_591__superscalar_pipelining.mdwn b/doc/devblog/day_591__superscalar_pipelining.mdwn
new file mode 100644
index 000000000..f6bcc7f9d
--- /dev/null
+++ b/doc/devblog/day_591__superscalar_pipelining.mdwn
@@ -0,0 +1,17 @@
+A long day spent making CommandCleanup actions run in a separate job pool 
+than CommandPerform actions. I don't think this will speed anything up much
+yet, but it's useful groundwork. Now expensive things that are not
+the main action of a command can be moved into CommandCleanup and won't
+delay git-annex moving on to the next file. The main thing I want to move
+is checksum verification after a transfer. But there are probably other
+things I have not thought of.
+
+CommandCleanup was always not well distinguised from CommandPerform,
+and so there was little incentive to put things in it. Now that's changed.
+
+I also noticed that with -J, git-annex takes significantly longer than
+without to get started, when the first file it needs to process is quite a
+way down the ls-tree. This must be concurrency overhead. But, when
+CommandStart is skipping over a file that it doesn't need to process,
+there is no need to do that bookkeeping. Planning to take some time
+tomorrow to see if I can refactor CommandStart to avoid that overhead.

diff --git a/doc/todo/parallel_possibilities.mdwn b/doc/todo/parallel_possibilities.mdwn
index 8ce72268e..0baa7484a 100644
--- a/doc/todo/parallel_possibilities.mdwn
+++ b/doc/todo/parallel_possibilities.mdwn
@@ -21,3 +21,16 @@ are still some things that could be improved, tracked here:
   all that needs to be done is make checksum verification be done as the
   cleanup action. Currently, it's bundled into the same action that
   transfers content.
+
+* Don't parallelize start stage actions. They are supposed to run fast,
+  and often a huge number of them don't print out anything. The overhead of
+  bookkeeping for parallizing those swamps the benefit of parallelizing by
+  what seems to be a large degree. Compare `git annex get` in a directory
+  where the first several thousand files are already present with and
+  without -J.
+  
+  Only once the start stage has decided
+  something needs to be done should a job be started up.
+
+  This probably needs display of any output to be moved out of the start
+  stage, because no console region will be allocated for it.

diff --git a/Annex.hs b/Annex.hs
index 90c327eab..d80500cf5 100644
--- a/Annex.hs
+++ b/Annex.hs
@@ -142,7 +142,7 @@ data AnnexState = AnnexState
 	, tempurls :: M.Map Key URLString
 	, existinghooks :: M.Map Git.Hook.Hook Bool
 	, desktopnotify :: DesktopNotify
-	, workers :: WorkerPool AnnexState
+	, workers :: TMVar (WorkerPool AnnexState)
 	, activekeys :: TVar (M.Map Key ThreadId)
 	, activeremotes :: MVar (M.Map (Types.Remote.RemoteA Annex) Integer)
 	, keysdbhandle :: Maybe Keys.DbHandle
@@ -155,6 +155,7 @@ newState :: GitConfig -> Git.Repo -> IO AnnexState
 newState c r = do
 	emptyactiveremotes <- newMVar M.empty
 	emptyactivekeys <- newTVarIO M.empty
+	emptyworkerpool <- newTMVarIO UnallocatedWorkerPool
 	o <- newMessageState
 	sc <- newTMVarIO False
 	return $ AnnexState
@@ -199,7 +200,7 @@ newState c r = do
 		, tempurls = M.empty
 		, existinghooks = M.empty
 		, desktopnotify = mempty
-		, workers = UnallocatedWorkerPool
+		, workers = emptyworkerpool
 		, activekeys = emptyactivekeys
 		, activeremotes = emptyactiveremotes
 		, keysdbhandle = Nothing
diff --git a/Annex/Concurrent.hs b/Annex/Concurrent.hs
index b3f868805..61bb7d5b8 100644
--- a/Annex/Concurrent.hs
+++ b/Annex/Concurrent.hs
@@ -11,7 +11,6 @@ import Annex
 import Annex.Common
 import Annex.Action
 import qualified Annex.Queue
-import Types.WorkerPool
 
 import qualified Data.Map as M
 
@@ -43,9 +42,8 @@ dupState :: Annex AnnexState
 dupState = do
 	st <- Annex.getState id
 	return $ st
-		{ Annex.workers = UnallocatedWorkerPool
 		-- each thread has its own repoqueue
-		, Annex.repoqueue = Nothing
+		{ Annex.repoqueue = Nothing
 		-- avoid sharing eg, open file handles
 		, Annex.catfilehandles = M.empty
 		, Annex.checkattrhandle = Nothing
diff --git a/CHANGELOG b/CHANGELOG
index 1b82d5160..54004d731 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -30,6 +30,10 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     security hole CVE-2018-10857 (except for configurations which enabled curl
     and bypassed public IP address restrictions). Now it will work
     if allowed by annex.security.allowed-ip-addresses.
+  * When running multiple concurrent actions, the cleanup phase is run
+    in a separate queue than the main action queue. This can make some
+    commands faster, because less time is spent on bookkeeping in
+    between each file transfer.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
index 534c7ed3c..ecdea2460 100644
--- a/CmdLine/Action.hs
+++ b/CmdLine/Action.hs
@@ -24,7 +24,6 @@ import Control.Concurrent.Async
 import Control.Concurrent.STM
 import Control.Exception (throwIO)
 import GHC.Conc
-import Data.Either
 import qualified Data.Map.Strict as M
 import qualified System.Console.Regions as Regions
 
@@ -61,7 +60,9 @@ commandAction a = Annex.getState Annex.concurrency >>= \case
 	run = void $ includeCommandAction a
 
 	runconcurrent n = do
-		ws <- liftIO . drainTo (n-1) =<< Annex.getState Annex.workers
+		tv <- Annex.getState Annex.workers
+		ws <- liftIO $ drainTo (n-1) (== PerformStage) 
+			=<< atomically (takeTMVar tv)
 		(st, ws') <- case ws of
 			UnallocatedWorkerPool -> do
 				-- Generate the remote list now, to avoid
@@ -72,61 +73,99 @@ commandAction a = Annex.getState Annex.concurrency >>= \case
 				_ <- remoteList
 				st <- dupState
 				return (st, allocateWorkerPool st (n-1))
-			WorkerPool l -> findFreeSlot l
+			WorkerPool _ -> findFreeSlot (== PerformStage) ws
 		w <- liftIO $ async $ snd <$> Annex.run st
 			(inOwnConsoleRegion (Annex.output st) run)
-		Annex.changeState $ \s -> s
-			{ Annex.workers = addWorkerPool ws' (Right w) }
+		liftIO $ atomically $ putTMVar tv $
+			addWorkerPool (ActiveWorker w PerformStage) ws'
 
 commandActions :: [CommandStart] -> Annex ()
 commandActions = mapM_ commandAction
 
-{- Waits for any forked off command actions to finish.
+{- Waits for any worker threads to finish.
  -
- - Merge together the cleanup actions of all the AnnexStates used by
- - threads, into the current Annex's state, so they'll run at shutdown.
- -
- - Also merge together the errcounters of the AnnexStates.
+ - Merge the AnnexStates used by the threads back into the current Annex's
+ - state.
  -}
 finishCommandActions :: Annex ()
 finishCommandActions = do
-	ws <- Annex.getState Annex.workers
-	Annex.changeState $ \s -> s { Annex.workers = UnallocatedWorkerPool }
-	ws' <- liftIO $ drainTo 0 ws
-	forM_ (idleWorkers ws') mergeState
+	tv <- Annex.getState Annex.workers
+	let get = liftIO $ atomically $ takeTMVar tv
+	let put = liftIO . atomically . putTMVar tv
+	bracketOnError get put $ \ws -> do
+		ws' <- liftIO $ drainTo 0 (const True) ws
+		forM_ (idleWorkers ws') mergeState
+		put UnallocatedWorkerPool
 
 {- Wait for jobs from the WorkerPool to complete, until
- - the number of running jobs is not larger than the specified number.
+ - the number of running jobs of the desired stage
+ - is not larger than the specified number.
  -
  - If a job throws an exception, it is propigated, but first
  - all other jobs are waited for, to allow for a clean shutdown.
  -}
-drainTo :: Int -> WorkerPool t -> IO (WorkerPool t)
-drainTo _ UnallocatedWorkerPool = pure UnallocatedWorkerPool
-drainTo sz (WorkerPool l)
+drainTo :: Int -> (WorkerStage -> Bool) -> WorkerPool t -> IO (WorkerPool t)
+drainTo _ _ UnallocatedWorkerPool = pure UnallocatedWorkerPool
+drainTo sz wantstage (WorkerPool l)
 	| null as || sz >= length as = pure (WorkerPool l)
 	| otherwise = do
-		(done, ret) <- waitAnyCatch as
-		let as' = filter (/= done) as
+		(done, ret) <- waitAnyCatch (mapMaybe workerAsync as)
+		let (ActiveWorker _ donestage:[], as') =
+			partition (\w -> workerAsync w == Just done) as
 		case ret of
 			Left e -> do
-				void $ drainTo 0 $ WorkerPool $
-					map Left sts ++ map Right as'
+				void $ drainTo 0 (const True) $ WorkerPool $
+					sts ++ as' ++ otheras
 				throwIO e
 			Right st -> do
-				drainTo sz $ WorkerPool $
-					map Left (st:sts) ++ map Right as'
+				let w = IdleWorker st donestage
+				drainTo sz wantstage $ WorkerPool $
+					w : sts ++ as' ++ otheras
   where
-	(sts, as) = partitionEithers l
+	(sts, allas) = partition isidle l
+	(as, otheras) = partition (wantstage . workerStage) allas
+	isidle (IdleWorker _ _) = True
+	isidle (ActiveWorker _ _) = False
 
-findFreeSlot :: [Worker Annex.AnnexState] -> Annex (Annex.AnnexState, WorkerPool Annex.AnnexState)
-findFreeSlot = go []
+findFreeSlot :: (WorkerStage -> Bool) -> WorkerPool Annex.AnnexState -> Annex (Annex.AnnexState, WorkerPool Annex.AnnexState)
+findFreeSlot wantstage (WorkerPool l) = go [] l
   where
 	go c [] = do
 		st <- dupState
 		return (st, WorkerPool c)
-	go c (Left st:rest) = return (st, WorkerPool (c ++ rest))
+	go c ((IdleWorker st stage):rest) | wantstage stage = 
+		return (st, WorkerPool (c ++ rest))
 	go c (v:rest) = go (v:c) rest
+findFreeSlot _ UnallocatedWorkerPool = do
+	st <- dupState
+	return (st, UnallocatedWorkerPool)
+
+{- Changes the current thread's stage in the worker pool.
+ -
+ - An idle worker with the desired stage is found in the pool
+ - (waiting if necessary for one to become idle)
+ - and the stages of it and the current thread are swapped.
+ -}
+changeStageTo :: WorkerStage -> Annex ()
+changeStageTo newstage = Annex.getState Annex.concurrency >>= \case
+	NonConcurrent -> noop
+	Concurrent n -> go n
+	ConcurrentPerCpu -> go =<< liftIO getNumProcessors
+  where

(Diff truncated)

diff --git a/doc/bugs/annex_init_no_longer_generates_default_description/comment_1_eb415152db43c02e4d474dade4a1faca._comment b/doc/bugs/annex_init_no_longer_generates_default_description/comment_1_eb415152db43c02e4d474dade4a1faca._comment
new file mode 100644
index 000000000..f99e0ee5b
--- /dev/null
+++ b/doc/bugs/annex_init_no_longer_generates_default_description/comment_1_eb415152db43c02e4d474dade4a1faca._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="kyle"
+ avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3"
+ subject="uuid.log is also not created"
+ date="2019-06-05T15:18:30Z"
+ content="""
+As of that same commit (a14f6ce75), running `git annex init` in a fresh repository no longer creates uuid.log:
+
+```
+% cd $(mktemp -dt gx-XXXXXXX)            
+% git init
+Initialized empty Git repository in /tmp/gx-kpnGSjg/.git/
+% git annex init
+init  ok
+% git ls-tree -r git-annex  ;# no output
+```
+
+"""]]

diff --git a/doc/bugs/annex_init_no_longer_generates_default_description.mdwn b/doc/bugs/annex_init_no_longer_generates_default_description.mdwn
new file mode 100644
index 000000000..814ae263f
--- /dev/null
+++ b/doc/bugs/annex_init_no_longer_generates_default_description.mdwn
@@ -0,0 +1,96 @@
+### Please describe the problem.
+
+As of a14f6ce75 (fix repo description setting bugs, 2019-05-23), `git annex init` without a description sets the description to a blank string.  AFAICT from the description of that change and from the corresponding [bug report][0], that isn't an intentional change.
+
+[0]: https://git-annex.branchable.com/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description/
+### What steps will reproduce the problem?
+
+```
+cd $(mktemp -dt gx-XXXXXXX)
+git init
+git annex init
+git annex info --json | jq
+```
+
+On the parent of a14f6ce75, this shows
+
+```
+Initialized empty Git repository in /tmp/gx-BLmU7TJ/.git/
+init  ok
+(recording state in git...)
+{
+  "local annex size": "0 bytes",
+  "size of annexed files in working tree": "0 bytes",
+  "command": "info",
+  "untrusted repositories": [],
+  "success": true,
+  "semitrusted repositories": [
+    {
+      "here": false,
+      "uuid": "00000000-0000-0000-0000-000000000001",
+      "description": "web"
+    },
+    {
+      "here": false,
+      "uuid": "00000000-0000-0000-0000-000000000002",
+      "description": "bittorrent"
+    },
+    {
+      "here": true,
+      "uuid": "d1ef2ae8-9604-4c33-8c3c-181f81c4348f",
+      "description": "kyle@hylob:/tmp/gx-BLmU7TJ"
+    }
+  ],
+  "bloom filter size": "32 mebibytes (0% full)",
+  "repository mode": "indirect",
+  "backend usage": {},
+  "transfers in progress": [],
+  "local annex keys": 0,
+  "available local disk space": "301.33 gigabytes (+1 megabyte reserved)",
+  "annexed files in working tree": 0,
+  "file": null,
+  "trusted repositories": []
+}
+```
+
+On a14f6ce75, this shows
+
+```
+Initialized empty Git repository in /tmp/gx-pK345zF/.git/
+init  ok
+{
+  "local annex size": "0 bytes",
+  "size of annexed files in working tree": "0 bytes",
+  "command": "info",
+  "untrusted repositories": [],
+  "success": true,
+  "semitrusted repositories": [
+    {
+      "here": false,
+      "uuid": "00000000-0000-0000-0000-000000000001",
+      "description": "web"
+    },
+    {
+      "here": false,
+      "uuid": "00000000-0000-0000-0000-000000000002",
+      "description": "bittorrent"
+    },
+    {
+      "here": true,
+      "uuid": "00b8da69-41a5-4de8-9a6a-9ed991289f81",
+      "description": ""
+    }
+  ],
+  "bloom filter size": "32 mebibytes (0% full)",
+  "repository mode": "indirect",
+  "backend usage": {},
+  "transfers in progress": [],
+  "local annex keys": 0,
+  "available local disk space": "301.22 gigabytes (+1 megabyte reserved)",
+  "annexed files in working tree": 0,
+  "file": null,
+  "trusted repositories": []
+}
+```
+
+Note that the description for here is a blank string.

diff --git a/Types/Remote.hs b/Types/Remote.hs
index d16e7971c..282a38c0c 100644
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -232,10 +232,10 @@ data ExportActions a = ExportActions
 	, removeExport :: Key -> ExportLocation -> a Bool
 	-- Removes an exported directory. Typically the directory will be
 	-- empty, but it could possibly contain files or other directories,
-	-- and it's ok to delete those. If the remote does not use
-	-- directories, or automatically cleans up empty directories,
-	-- this can be Nothing. Should not fail if the directory was
-	-- already removed.
+	-- and it's ok to delete those (but not required to). 
+	-- If the remote does not use directories, or automatically cleans
+	-- up empty directories, this can be Nothing.
+	-- Should not fail if the directory was already removed.
 	, removeExportDirectory :: Maybe (ExportDirectory -> a Bool)
 	-- Checks if anything is exported to the remote at the specified
 	-- ExportLocation.
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 433aa4615..222308796 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -78,7 +78,8 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
   The directory will be in the form of a relative path, and may contain path
   separators, whitespace, and other special characters.  
   Typically the directory will be empty, but it could possibly contain
-  files or other directories, and it's ok to remove those.  
+  files or other directories, and it's ok to remove those, but not required
+  to do so.  
   * `REMOVEEXPORTDIRECTORY-SUCCESS`  
     Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully.
   * `REMOVEEXPORTDIRECTORY-FAILURE`  

diff --git a/Annex/Import.hs b/Annex/Import.hs
index 585deefa2..d70942acd 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -16,6 +16,7 @@ module Annex.Import (
 	downloadImport,
 	filterImportableContents,
 	makeImportMatcher,
+	listImportableContents,
 ) where
 
 import Annex.Common
@@ -54,6 +55,7 @@ import qualified Logs.ContentIdentifier as CIDLog
 import Control.Concurrent.STM
 import qualified Data.Map.Strict as M
 import qualified Data.Set as S
+import qualified System.FilePath.Posix as Posix
 
 {- Configures how to build an import tree. -}
 data ImportTreeConfig
@@ -480,3 +482,22 @@ filterImportableContents r matcher importable
 		<*> mapM (go dbhandle) (importableHistory ic)
 	
 	match dbhandle (loc, (_cid, sz)) = shouldImport dbhandle matcher loc sz
+	
+{- Gets the ImportableContents from the remote.
+ -
+ - Filters out any paths that include a ".git" component, because git does
+ - not allow storing ".git" in a git repository. While it is possible to
+ - write a git tree that contains that, git will complain and refuse to
+ - check it out.
+ -}
+listImportableContents :: Remote -> Annex (Maybe (ImportableContents (ContentIdentifier, ByteSize)))
+listImportableContents r = fmap removegitspecial
+	<$> Remote.listImportableContents (Remote.importActions r)
+  where
+	removegitspecial ic = ImportableContents
+		{ importableContents = 
+			filter (not . gitspecial . fst) (importableContents ic)
+		, importableHistory =
+			map removegitspecial (importableHistory ic)
+		}
+	gitspecial l = ".git" `elem` Posix.splitDirectories (fromImportLocation l)
diff --git a/CHANGELOG b/CHANGELOG
index 0cd1e0fab..1b82d5160 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -11,6 +11,8 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     unwanted files are not imported. But, some preferred content
     expressions can't be checked before files are imported, and trying to
     import with such an expression will fail.
+  * Don't try to import .git directories from special remotes, because
+    git does not support storing git repositories inside a git repository.
   * Improve shape of commit tree when importing from unversioned special
     remotes.
   * init: When the repository already has a description, don't change it.
diff --git a/Command/Import.hs b/Command/Import.hs
index f890f982b..c8659a1e9 100644
--- a/Command/Import.hs
+++ b/Command/Import.hs
@@ -291,7 +291,7 @@ seekRemote remote branch msubdir = do
 listContents :: Remote -> TVar (Maybe (ImportableContents (ContentIdentifier, Remote.ByteSize))) -> CommandStart
 listContents remote tvar = do
 	showStart' "list" (Just (Remote.name remote))
-	next $ Remote.listImportableContents (Remote.importActions remote) >>= \case
+	next $ listImportableContents remote >>= \case
 		Nothing -> giveup $ "Unable to list contents of " ++ Remote.name remote
 		Just importable -> do
 			importable' <- makeImportMatcher remote >>= \case
diff --git a/doc/bugs/import_tree_should_skip_.git.mdwn b/doc/bugs/import_tree_should_skip_.git.mdwn
index 589e52eb3..8ef9df8ee 100644
--- a/doc/bugs/import_tree_should_skip_.git.mdwn
+++ b/doc/bugs/import_tree_should_skip_.git.mdwn
@@ -19,6 +19,14 @@ The solution is certianly to
 1. filter out .git when importing
 2. avoid deleting .git when exporting
 
+   As long as the export tracking branch did not contain .git,
+   an export will harm no .git directories, because an exporttree
+   remote uses removeExportDirectoryWhenEmpty, not removeExportDirectory,
+   so will not delete non-empty directories. And other than deleting
+   directories, exporting only deletes files that are removed in the git
+   diff between two trees; if neither tree contained .git, the diff won't
+   contain it either, and so if importing skips .git, this will be ok.
+
 --[[Joey]]
 
 > [[fixed|done]] --[[Joey]]

diff --git a/Types/Remote.hs b/Types/Remote.hs
index 451d2d305..d16e7971c 100644
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -231,7 +231,7 @@ data ExportActions a = ExportActions
 	-- Removes an exported file (succeeds if the contents are not present)
 	, removeExport :: Key -> ExportLocation -> a Bool
 	-- Removes an exported directory. Typically the directory will be
-	-- empty, but it could possbly contain files or other directories,
+	-- empty, but it could possibly contain files or other directories,
 	-- and it's ok to delete those. If the remote does not use
 	-- directories, or automatically cleans up empty directories,
 	-- this can be Nothing. Should not fail if the directory was
diff --git a/doc/bugs/import_tree_should_skip_.git.mdwn b/doc/bugs/import_tree_should_skip_.git.mdwn
index d56a8c2bf..589e52eb3 100644
--- a/doc/bugs/import_tree_should_skip_.git.mdwn
+++ b/doc/bugs/import_tree_should_skip_.git.mdwn
@@ -20,3 +20,5 @@ The solution is certianly to
 2. avoid deleting .git when exporting
 
 --[[Joey]]
+
+> [[fixed|done]] --[[Joey]]

diff --git a/doc/todo/sqlite_database_improvements.mdwn b/doc/todo/sqlite_database_improvements.mdwn
index 0458c3f7c..2e8e10a49 100644
--- a/doc/todo/sqlite_database_improvements.mdwn
+++ b/doc/todo/sqlite_database_improvements.mdwn
@@ -28,13 +28,16 @@ process.
 	INSERT INTO associated VALUES(4,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a,'foo.ü');
 	INSERT INTO associated VALUES(5,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a','"foo.\56515\56508"');
 
+  And it seems likely that a query by filename would fail if the filename
+  was in the database but with a different encoding.
+
 * IKey could fail to round-trip as well, when a Key contains something
   (eg, a filename extension) that is not valid in the current locale,
   for similar reasons to SFilePath. Using BLOB would be better.
 
-  See [[!commit cf260d9a159050e2a7e70394fdd8db289c805ec3]] for how the
-  encoding problem was fixed for SFilePath. I reproduced a similar problem
-  by making a file `foo.ü` and running `git add` on  it in a unicode
+  See [[!commit cf260d9a159050e2a7e70394fdd8db289c805ec3]] for details
+  about the encoding problem for SFilePath. I reproduced a similar problem
+  for IKey by making a file `foo.ü` and running `git add` on  it in a unicode
   locale. Then with LANG=C, `git annex drop --force foo.ü` thinks
   it drops the content, but in fact the work tree file is left containing
   the dropped content. The database then contained:

diff --git a/doc/todo/sqlite_database_improvements.mdwn b/doc/todo/sqlite_database_improvements.mdwn
index d4c183dae..0458c3f7c 100644
--- a/doc/todo/sqlite_database_improvements.mdwn
+++ b/doc/todo/sqlite_database_improvements.mdwn
@@ -22,6 +22,171 @@ process.
   around with a hack that escapes FilePaths that contain unusual
   characters. It would be much better to use a BLOB.
 
+  Also, when LANG=C is sometimes used, the hack can result in duplicates with
+  different representations of the same filename, like this:
+
+	INSERT INTO associated VALUES(4,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a,'foo.ü');
+	INSERT INTO associated VALUES(5,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a','"foo.\56515\56508"');
+
 * IKey could fail to round-trip as well, when a Key contains something
   (eg, a filename extension) that is not valid in the current locale,
   for similar reasons to SFilePath. Using BLOB would be better.
+
+  See [[!commit cf260d9a159050e2a7e70394fdd8db289c805ec3]] for how the
+  encoding problem was fixed for SFilePath. I reproduced a similar problem
+  by making a file `foo.ü` and running `git add` on  it in a unicode
+  locale. Then with LANG=C, `git annex drop --force foo.ü` thinks
+  it drops the content, but in fact the work tree file is left containing
+  the dropped content. The database then contained:
+
+	INSERT INTO associated VALUES(8,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.ü','foo.ü');
+	INSERT INTO associated VALUES(9,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.��','"foo.\56515\56508"');
+
+> Investigated this in more detail, and I can't find a way to
+> solve the encoding problem other than changing the encoding
+> SKey, IKey, and SFilePath in a non-backwards-compatible way.
+> 
+> (Unless the encoding problem is related to persistent's use of Text
+> internally, and could then perhaps be avoided by avoiding that?)
+> 
+> The simplest and best final result would be use a ByteString
+> for all of them, and store a blob in sqlite. Attached patch
+> shows how to do that, but old git-annex won't be able to read
+> the updated databases, and won't know that it can't read them!
+> 
+> This seems to call for a flag day, throwing out the old database
+> contents and regenerating them from other data:
+> 
+> * Fsck (SKey)
+>   can't rebuild? Just drop and let incremental fscks re-do work
+> * ContentIdentifier (IKey)  
+>   rebuild with updateFromLog, would need to diff from empty tree to
+>   current git-annex branch, may be expensive to do!
+> * Export (IKey, SFilePath)  
+>   difficult to rebuild, what if in the middle of an interrupted
+>   export?
+>   
+>   updateExportTreeFromLog only updates two tables, not others
+>   
+>   Conceptually, this is the same as the repo being lost and another
+>   clone being used to update the export. The clone can only learn
+>   export state from the log. It's supposed to recover from such
+>   situations, the next time an export is run, so should be ok.
+>   But it might result in already exported files being re-uploaded,
+>   or other unncessary work.
+> Keys (IKey, SFilePath)
+>   rebuild with scanUnlockedFiles
+> 
+>   does that update the Content table with the InodeCache?
+>
+> But after such a transition, how to communicate to the old git-annex
+> that it can't use the databases any longer? Moving the databases
+> out of the way won't do; old git-annex will just recreate them and 
+> start with missing data!
+> 
+> And, what about users who really need to continue using an old git-annex
+> and get bitten by the flag day?
+> 
+> Should this instead be a annex.version bump from v7 to v8?
+> But v5 is also affected for ContentIdentifier and Export and Fsck.
+> Don't want v5.1.
+
+----
+
+[[!format patch """
+diff --git a/Database/Types.hs b/Database/Types.hs
+index f08cf4e9d..3e9c9e267 100644
+--- a/Database/Types.hs
++++ b/Database/Types.hs
+@@ -14,11 +14,12 @@ import Database.Persist.TH
+ import Database.Persist.Class hiding (Key)
+ import Database.Persist.Sql hiding (Key)
+ import Data.Maybe
+-import Data.Char
+ import qualified Data.ByteString as S
++import qualified Data.ByteString.Lazy as L
+ import qualified Data.Text as T
+ 
+ import Utility.PartialPrelude
++import Utility.FileSystemEncoding
+ import Key
+ import Utility.InodeCache
+ import Git.Types (Ref(..))
+@@ -37,23 +38,18 @@ fromSKey (SKey s) = fromMaybe (error $ "bad serialized Key " ++ s) (deserializeK
+ 
+ derivePersistField "SKey"
+ 
+--- A Key index. More efficient than SKey, but its Read instance does not
+--- work when it's used in any kind of complex data structure.
+-newtype IKey = IKey String
+-
+-instance Read IKey where
+-	readsPrec _ s = [(IKey s, "")]
+-
+-instance Show IKey where
+-	show (IKey s) = s
++-- A Key index. More efficient than SKey.
++newtype IKey = IKey S.ByteString
++	deriving (Eq, Show, PersistField, PersistFieldSql)
+ 
++-- FIXME: toStrict copies, not efficient
+ toIKey :: Key -> IKey
+-toIKey = IKey . serializeKey
++toIKey = IKey . L.toStrict . serializeKey'
+ 
+ fromIKey :: IKey -> Key
+-fromIKey (IKey s) = fromMaybe (error $ "bad serialized Key " ++ s) (deserializeKey s)
+-
+-derivePersistField "IKey"
++fromIKey (IKey b) = fromMaybe
++	(error $ "bad serialized Key " ++ show b) 
++	(deserializeKey' b)
+ 
+ -- A serialized InodeCache
+ newtype SInodeCache = I String
+@@ -67,39 +63,15 @@ fromSInodeCache (I s) = fromMaybe (error $ "bad serialized InodeCache " ++ s) (r
+ 
+ derivePersistField "SInodeCache"
+ 
+--- A serialized FilePath.
+---
+--- Not all unicode characters round-trip through sqlite. In particular,
+--- surrigate code points do not. So, escape the FilePath. But, only when
+--- it contains such characters.
+-newtype SFilePath = SFilePath String
+-
+--- Note that Read instance does not work when used in any kind of complex
+--- data structure.
+-instance Read SFilePath where
+-	readsPrec _ s = [(SFilePath s, "")]
+-
+-instance Show SFilePath where
+-	show (SFilePath s) = s
++-- A serialized FilePath. Stored as a ByteString to avoid encoding problems.
++newtype SFilePath = SFilePath S.ByteString
++	deriving (Eq, Show, PersistField, PersistFieldSql)
+ 
+ toSFilePath :: FilePath -> SFilePath
+-toSFilePath s@('"':_) = SFilePath (show s)
+-toSFilePath s
+-	| any needsescape s = SFilePath (show s)
+-	| otherwise = SFilePath s
+-  where
+-	needsescape c = case generalCategory c of
+-		Surrogate -> True
+-		PrivateUse -> True
+-		NotAssigned -> True
+-		_ -> False
++toSFilePath = SFilePath . encodeBS
+ 
+ fromSFilePath :: SFilePath -> FilePath
+-fromSFilePath (SFilePath s@('"':_)) =
+-	fromMaybe (error "bad serialized SFilePath " ++ s) (readish s)
+-fromSFilePath (SFilePath s) = s
+-
+-derivePersistField "SFilePath"
++fromSFilePath (SFilePath b) = decodeBS b
+ 
+ -- A serialized Ref
+ newtype SRef = SRef Ref
+"""]]

diff --git a/doc/thanks/list b/doc/thanks/list
index 0a1192a14..6b58c7273 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -58,3 +58,5 @@ Lp and Nick Daly,
 Walltime, 
 Caleb Allen, 
 TD, 
+Pedro Araújo, 
+

diff --git a/doc/bugs/Assistant_is_stuck_with___39__fuzztest__39__.mdwn b/doc/bugs/Assistant_is_stuck_with___39__fuzztest__39__.mdwn
new file mode 100644
index 000000000..76c6ad458
--- /dev/null
+++ b/doc/bugs/Assistant_is_stuck_with___39__fuzztest__39__.mdwn
@@ -0,0 +1,23 @@
+### Please describe the problem.
+
+I want to use Git Annex as a synchronization tool between external devices.  I did set up v7 repositories in two separate file systems placed on physically separate devices.  I created a systemd service that fires Git Annex Assistant in each of those file systems as soon as their corresponding devices are mounted.  The idea was to get automatic synchronization upon drag and drop of files to back up into either of the file systems.  What I noticed so far, is that things do not work well all the time.  There are several problems I encountered, but here I want to concentrate only on one.
+
+The first issue is that when doing some operations, at some point, Git Annex Assistant on either side stops synchronizing as if it would be stuck.  It's a race condition and cannot be easily reproduced.  Hence, I used `fuzztest` to prove that one can always get into that issue.  What I've noticed so far is that when this happens, it means that on either side there one of the Git Annex Assistant processes has spawned a child `git-annex transferkeys`, which is not present from the beginning but now is stuck there forever.  Most probably, this is the reason why synchronization stops.  That is during `fuzztest`, the files will keep accumulating but there will be no more automatic commits and/or pushes.
+
+### What steps will reproduce the problem?
+
+Just create two repos in v7 mode, make them remotes of each other, run assistants in each of them, and start `fuzztest` within each of them as well.
+
+### What version of git-annex are you using? On what operating system?
+
+[[!format sh """
+git-annex version: 7.20190322-gece57002c6
+build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
+operating system: linux x86_64
+supported repository versions: 5 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+local repository version: 7
+"""]]

diff --git a/doc/bugs/import_tree_should_skip_.git.mdwn b/doc/bugs/import_tree_should_skip_.git.mdwn
new file mode 100644
index 000000000..d56a8c2bf
--- /dev/null
+++ b/doc/bugs/import_tree_should_skip_.git.mdwn
@@ -0,0 +1,22 @@
+If the remote being imported from has a .git directory,
+the import of it fails, because git does not allow adding .git 
+to a repo. 
+
+But, git *does* allow creating a tree object containing .git, it just can't
+check it out. So what happens is that the remote tracking branch contains
+.git, but when git is asked to merge that into master, it skips checking
+out those files.
+
+An export back to the remote will then delete the .git directory from it.
+
+But note that there is *not* data loss! git-annex keeps all the files
+in its object store, and the remote tracking branch contains a tree with
+the .git in it, so if necessary it could be reconstructed. But with some
+difficulty.
+
+The solution is certianly to 
+
+1. filter out .git when importing
+2. avoid deleting .git when exporting
+
+--[[Joey]]

diff --git a/doc/forum/where_to_submit_pull_requests_for_git-annex__63__.mdwn b/doc/forum/where_to_submit_pull_requests_for_git-annex__63__.mdwn
new file mode 100644
index 000000000..61e41a5ba
--- /dev/null
+++ b/doc/forum/where_to_submit_pull_requests_for_git-annex__63__.mdwn
@@ -0,0 +1,4 @@
+Is there a place to submit pull requests for git-annex?
+
+I've made a small change to the docs, adding html anchors to enable hyperlinks to docs for specific config settings, on a branch here:
+<https://github.com/notestaff/git-annex/tree/is-doc-add-html-anchors-for-config-vars>

diff --git a/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment b/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment
new file mode 100644
index 000000000..c0cca8eb5
--- /dev/null
+++ b/doc/forum/git_annex_hanging_in_smudge_on_2k_file_on_rpi/comment_4_0043eedff90ebcea5bfa11f266256e92._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joer962500"
+ avatar="http://cdn.libravatar.org/avatar/573a6c0ca60e11a8420ce772cd2e80a7"
+ subject="asus tablet stuck loading screen"
+ date="2019-05-31T07:24:27Z"
+ content="""
+I want to share issues related to Asus tablet. Sometimes the issue may occur while loading. Recover the issue with just one click, follow this quick fix guide <a href=\"=https://errorcode0x.com/fix-asus-tablet-stuck-at-loading-screen/\">asus tablet stuck loading screen</a> and get our job done easily.
+
+
+
+"""]]

diff --git a/Utility/Url.hs b/Utility/Url.hs
index a15e51013..81787aaf0 100644
--- a/Utility/Url.hs
+++ b/Utility/Url.hs
@@ -55,6 +55,7 @@ import Control.Exception (throwIO)
 import Control.Monad.Trans.Resource
 import Network.HTTP.Conduit
 import Network.HTTP.Client
+import Network.HTTP.Simple (getResponseHeader)
 import Network.Socket
 import Network.BSD (getProtocolNumber)
 import Data.Either
@@ -208,19 +209,25 @@ getUrlInfo :: URLString -> UrlOptions -> IO UrlInfo
 getUrlInfo url uo = case parseURIRelaxed url of
 	Just u -> checkPolicy uo u dne warnError $
 		case (urlDownloader uo, parseUrlRequest (show u)) of
-			(DownloadWithConduit _, Just req) ->
-				existsconduit req
+			(DownloadWithConduit (DownloadWithCurlRestricted r), Just req) -> catchJust
+				-- When http redirects to a protocol which 
+				-- conduit does not support, it will throw
+				-- a StatusCodeException with found302
+				-- and a Response with the redir Location.
+				(matchStatusCodeException (== found302))
+				(existsconduit req)
+				(followredir r)
 					`catchNonAsync` (const $ return dne)
 			(DownloadWithConduit (DownloadWithCurlRestricted r), Nothing)
 				| isfileurl u -> existsfile u
-				| isftpurl u -> existscurlrestricted r u 21
+				| isftpurl u -> existscurlrestricted r u url ftpport
 					`catchNonAsync` (const $ return dne)
 				| otherwise -> do
 					unsupportedUrlScheme u warnError
 					return dne
 			(DownloadWithCurl _, _) 
 				| isfileurl u -> existsfile u
-				| otherwise -> existscurl u basecurlparams
+				| otherwise -> existscurl u (basecurlparams url)
 	Nothing -> return dne
   where
 	dne = UrlInfo False Nothing Nothing
@@ -229,10 +236,12 @@ getUrlInfo url uo = case parseURIRelaxed url of
 	isfileurl u = uriScheme u == "file:"
 	isftpurl u = uriScheme u == "ftp:"
 
-	basecurlparams = curlParams uo $
+	ftpport = 21
+
+	basecurlparams u = curlParams uo $
 		[ Param "-s"
 		, Param "--head"
-		, Param "-L", Param url
+		, Param "-L", Param u
 		, Param "-w", Param "%{http_code}"
 		]
 
@@ -278,8 +287,8 @@ getUrlInfo url uo = case parseURIRelaxed url of
 			_ | isftp && isJust len -> good
 			_ -> return dne
 	
-	existscurlrestricted r u defport = existscurl u 
-		=<< curlRestrictedParams r u defport basecurlparams
+	existscurlrestricted r u url defport = existscurl u 
+		=<< curlRestrictedParams r u defport (basecurlparams url)
 
 	existsfile u = do
 		let f = unEscapeString (uriPath u)
@@ -289,6 +298,22 @@ getUrlInfo url uo = case parseURIRelaxed url of
 				sz <- getFileSize' f stat
 				found (Just sz) Nothing
 			Nothing -> return dne
+#if MIN_VERSION_http_client(0,5,0)
+	followredir r (HttpExceptionRequest _ (StatusCodeException resp _)) = 
+		case headMaybe $ map decodeBS $ getResponseHeader hLocation resp of
+#else
+	followredir r (StatusCodeException _ respheaders _) =
+		case headMaybe $ map (decodeBS . snd) $ filter (\(h, _) -> h == hLocation) respheaders
+#endif
+			Just url' -> case parseURIRelaxed url' of
+				-- only follow http to ftp redirects;
+				-- http to file redirect would not be secure,
+				-- and http-conduit follows http to http.
+				Just u' | isftpurl u' ->
+					checkPolicy uo u' dne warnError $
+						existscurlrestricted r u' url' ftpport
+				_ -> return dne
+			Nothing -> return dne
 
 -- Parse eg: attachment; filename="fname.ext"
 -- per RFC 2616
@@ -327,16 +352,18 @@ download' noerror meterupdate url file uo =
 	go = case parseURIRelaxed url of
 		Just u -> checkPolicy uo u False dlfailed $
 			case (urlDownloader uo, parseUrlRequest (show u)) of
-				(DownloadWithConduit _, Just req) ->
-					downloadconduit req
+				(DownloadWithConduit (DownloadWithCurlRestricted r), Just req) -> catchJust
+					(matchStatusCodeException (== found302))
+					(downloadconduit req)
+					(followredir r)
 				(DownloadWithConduit (DownloadWithCurlRestricted r), Nothing)
 					| isfileurl u -> downloadfile u
-					| isftpurl u -> downloadcurlrestricted r u 21
+					| isftpurl u -> downloadcurlrestricted r u url ftpport
 						`catchNonAsync` (dlfailed . show)
 					| otherwise -> unsupportedUrlScheme u dlfailed
 				(DownloadWithCurl _, _)
 					| isfileurl u -> downloadfile u
-					| otherwise -> downloadcurl basecurlparams
+					| otherwise -> downloadcurl url basecurlparams
 		Nothing -> do
 			liftIO $ debugM "url" url
 			dlfailed "invalid url"
@@ -344,6 +371,8 @@ download' noerror meterupdate url file uo =
 	isfileurl u = uriScheme u == "file:"
 	isftpurl u = uriScheme u == "ftp:"
 
+	ftpport = 21
+
 	downloadconduit req = catchMaybeIO (getFileSize file) >>= \case
 		Nothing -> runResourceT $ do
 			liftIO $ debugM "url" (show req')
@@ -434,15 +463,15 @@ download' noerror meterupdate url file uo =
 		, Param "-C", Param "-"
 		]
 
-	downloadcurl curlparams = do
+	downloadcurl rawurl curlparams = do
 		-- curl does not create destination file
 		-- if the url happens to be empty, so pre-create.
 		unlessM (doesFileExist file) $
 			writeFile file ""
-		boolSystem "curl" (curlparams ++ [Param "-o", File file, File url])
+		boolSystem "curl" (curlparams ++ [Param "-o", File file, File rawurl])
 
-	downloadcurlrestricted r u defport =
-		downloadcurl =<< curlRestrictedParams r u defport basecurlparams
+	downloadcurlrestricted r u rawurl defport =
+		downloadcurl rawurl =<< curlRestrictedParams r u defport basecurlparams
 
 	downloadfile u = do
 		let src = unEscapeString (uriPath u)
@@ -450,6 +479,20 @@ download' noerror meterupdate url file uo =
 			L.writeFile file
 		return True
 
+#if MIN_VERSION_http_client(0,5,0)
+	followredir r ex@(HttpExceptionRequest _ (StatusCodeException resp _)) = 
+		case headMaybe $ map decodeBS $ getResponseHeader hLocation resp of
+#else
+	followredir r ex@(StatusCodeException _ respheaders _) =
+		case headMaybe $ map (decodeBS . snd) $ filter (\(h, _) -> h == hLocation) respheaders
+#endif
+			Just url' -> case parseURIRelaxed url' of
+				Just u' | isftpurl u' ->
+					checkPolicy uo u' False dlfailed $
+						downloadcurlrestricted r u' url' ftpport
+				_ -> throwIO ex
+			Nothing -> throwIO ex
+
 {- Sinks a Response's body to a file. The file can either be opened in
  - WriteMode or AppendMode. Updates the meter as data is received.
  -
diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn
index ff2d364ba..cf97926fd 100644
--- a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn
@@ -66,3 +66,5 @@ local repository version: 5
 Originally [reported in DataLad #3321](https://github.com/datalad/datalad/issues/3321) with workarounds to force `curl` downloads
 
 [[!meta author=yoh]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_7_11f4424d3b9fb6c54fa510320affac73._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_7_11f4424d3b9fb6c54fa510320affac73._comment
new file mode 100644
index 000000000..44434ee0b
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_7_11f4424d3b9fb6c54fa510320affac73._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2019-05-30T20:03:28Z"
+ content="""
+All fixed now.
+"""]]

diff --git a/Annex/Url.hs b/Annex/Url.hs
index 9f792a5ce..91407ee16 100644
--- a/Annex/Url.hs
+++ b/Annex/Url.hs
@@ -59,7 +59,8 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 			-- it from accessing specific IP addresses.
 			curlopts <- map Param . annexWebOptions <$> Annex.getGitConfig
 			let urldownloader = if null curlopts
-				then U.DownloadWithConduit
+				then U.DownloadWithConduit $
+					U.DownloadWithCurlRestricted mempty
 				else U.DownloadWithCurl curlopts
 			manager <- liftIO $ U.newManager U.managerSettings
 			return (urldownloader, manager)
@@ -78,7 +79,7 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 			let connectionrestricted = addrConnectionRestricted 
 				("Configuration of annex.security.allowed-ip-addresses does not allow accessing address " ++)
 			let r = Restriction
-				{ addressRestriction = \addr ->
+				{ checkAddressRestriction = \addr ->
 					if isallowed (addrAddress addr)
 						then Nothing
 						else Just (connectionrestricted addr)
@@ -90,7 +91,9 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 				Just ProxyRestricted -> toplevelWarning True
 					"http proxy settings not used due to annex.security.allowed-ip-addresses configuration"
 			manager <- liftIO $ U.newManager settings
-			return (U.DownloadWithConduit, manager)
+			let urldownloader = U.DownloadWithConduit $
+				U.DownloadWithCurlRestricted r
+			return (urldownloader, manager)
 
 ipAddressesUnlimited :: Annex Bool
 ipAddressesUnlimited = 
diff --git a/CHANGELOG b/CHANGELOG
index 833a232cb..0cd1e0fab 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -24,6 +24,10 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     annex.security.allowed-ip-addresses because it is not really specific
     to the http protocol, also limiting eg, git-annex's use of ftp.
     The old name for the config will still work.
+  * Add back support for ftp urls, which was disabled as part of the fix for
+    security hole CVE-2018-10857 (except for configurations which enabled curl
+    and bypassed public IP address restrictions). Now it will work
+    if allowed by annex.security.allowed-ip-addresses.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/Utility/HttpManagerRestricted.hs b/Utility/HttpManagerRestricted.hs
index 00611b7b5..ac867aba5 100644
--- a/Utility/HttpManagerRestricted.hs
+++ b/Utility/HttpManagerRestricted.hs
@@ -30,11 +30,39 @@ import qualified Data.ByteString.UTF8 as BU
 import Data.Default
 import Data.Typeable
 import Control.Applicative
+#if MIN_VERSION_base(4,9,0)
+import qualified Data.Semigroup as Sem
+#endif
+import Data.Monoid
+import Prelude
 
 data Restriction = Restriction
-	{ addressRestriction :: AddrInfo -> Maybe ConnectionRestricted
+	{ checkAddressRestriction :: AddrInfo -> Maybe ConnectionRestricted
+	}
+
+appendRestrictions :: Restriction -> Restriction -> Restriction
+appendRestrictions a b = Restriction
+	{ checkAddressRestriction = \addr ->
+		checkAddressRestriction a addr <|> checkAddressRestriction b addr
 	}
 
+-- | mempty does not restrict HTTP connections in any way
+instance Monoid Restriction where
+	mempty = Restriction
+		{ checkAddressRestriction = \_ -> Nothing
+		}
+#if MIN_VERSION_base(4,11,0)
+#elif MIN_VERSION_base(4,9,0)
+	mappend = (Sem.<>)
+#else
+	mappend = appendRestrictions
+#endif
+
+#if MIN_VERSION_base(4,9,0)
+instance Sem.Semigroup Restriction where
+	(<>) = appendRestrictions
+#endif
+
 -- | An exception used to indicate that the connection was restricted.
 data ConnectionRestricted = ConnectionRestricted String
 	deriving (Show, Typeable)
@@ -117,7 +145,7 @@ restrictProxy cfg base = do
 			return $ proxy $ f $ dummyreq https
 	
 	mkproxy Nothing = (noProxy, Nothing)
-	mkproxy (Just proxyaddr) = case addressRestriction cfg proxyaddr of
+	mkproxy (Just proxyaddr) = case checkAddressRestriction cfg proxyaddr of
 		Nothing -> (addrtoproxy (addrAddress proxyaddr), Nothing)
 		Just _ -> (noProxy, Just ProxyRestricted)
 	
@@ -200,7 +228,7 @@ getConnection cfg tls = do
 			close
 			(\sock -> NC.connectFromSocket context sock connparams)
 	  where
-		tryToConnect addr = case addressRestriction cfg addr of
+		tryToConnect addr = case checkAddressRestriction cfg addr of
 			Nothing -> bracketOnError
 				(socket (addrFamily addr) (addrSocketType addr) (addrProtocol addr))
 				close
diff --git a/Utility/IPAddress.hs b/Utility/IPAddress.hs
index 64b8688f1..cfe9873ca 100644
--- a/Utility/IPAddress.hs
+++ b/Utility/IPAddress.hs
@@ -12,9 +12,22 @@ import Utility.Exception
 import Network.Socket
 import Data.Word
 import Data.Memory.Endian
+import Data.List
 import Control.Applicative
+import Text.Printf
 import Prelude
 
+extractIPAddress :: SockAddr -> Maybe String
+extractIPAddress (SockAddrInet _ ipv4) =
+	let (a,b,c,d) = hostAddressToTuple ipv4
+	in Just $ intercalate "." [show a, show b, show c, show d]
+extractIPAddress (SockAddrInet6 _ _ ipv6 _) =
+	let (a,b,c,d,e,f,g,h) = hostAddress6ToTuple ipv6
+	in Just $ intercalate ":" [s a, s b, s c, s d, s e, s f, s g, s h]
+  where
+	s = printf "%x"
+extractIPAddress _ = Nothing
+
 {- Check if an IP address is a loopback address; connecting to it
  - may connect back to the local host. -}
 isLoopbackAddress :: SockAddr -> Bool
diff --git a/Utility/Url.hs b/Utility/Url.hs
index a9c46b432..a15e51013 100644
--- a/Utility/Url.hs
+++ b/Utility/Url.hs
@@ -1,6 +1,6 @@
 {- Url downloading.
  -
- - Copyright 2011-2018 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2019 Joey Hess <id@joeyh.name>
  -
  - License: BSD-2-clause
  -}
@@ -19,6 +19,7 @@ module Utility.Url (
 	mkScheme,
 	allowedScheme,
 	UrlDownloader(..),
+	NonHttpUrlDownloader(..),
 	UrlOptions(..),
 	defUrlOptions,
 	mkUrlOptions,
@@ -40,18 +41,25 @@ module Utility.Url (
 import Common
 import Utility.Metered
 import Utility.HttpManagerRestricted
+import Utility.IPAddress
 
 import Network.URI
 import Network.HTTP.Types
+import qualified Network.Connection as NC
 import qualified Data.CaseInsensitive as CI
 import qualified Data.ByteString as B
 import qualified Data.ByteString.UTF8 as B8
 import qualified Data.ByteString.Lazy as L
 import qualified Data.Set as S
+import Control.Exception (throwIO)
 import Control.Monad.Trans.Resource
 import Network.HTTP.Conduit
 import Network.HTTP.Client
+import Network.Socket
+import Network.BSD (getProtocolNumber)
+import Data.Either
 import Data.Conduit
+import Text.Read
 import System.Log.Logger
 
 #if ! MIN_VERSION_http_client(0,5,0)
@@ -92,14 +100,17 @@ data UrlOptions = UrlOptions
 	}
 
 data UrlDownloader
-	= DownloadWithConduit
+	= DownloadWithConduit NonHttpUrlDownloader
 	| DownloadWithCurl [CommandParam]
 
+data NonHttpUrlDownloader
+	= DownloadWithCurlRestricted Restriction
+
 defUrlOptions :: IO UrlOptions
 defUrlOptions = UrlOptions
 	<$> pure Nothing
 	<*> pure []

(Diff truncated)

diff --git a/Annex/Url.hs b/Annex/Url.hs
index 89618fd76..9f792a5ce 100644
--- a/Annex/Url.hs
+++ b/Annex/Url.hs
@@ -11,7 +11,7 @@ module Annex.Url (
 	withUrlOptions,
 	getUrlOptions,
 	getUserAgent,
-	httpAddressesUnlimited,
+	ipAddressesUnlimited,
 ) where
 
 import Annex.Common
@@ -52,7 +52,7 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 		Just cmd -> lines <$> liftIO (readProcess "sh" ["-c", cmd])
 		Nothing -> annexHttpHeaders <$> Annex.getGitConfig
 	
-	checkallowedaddr = words . annexAllowedHttpAddresses <$> Annex.getGitConfig >>= \case
+	checkallowedaddr = words . annexAllowedIPAddresses <$> Annex.getGitConfig >>= \case
 		["all"] -> do
 			-- Only allow curl when all are allowed,
 			-- as its interface does not allow preventing
@@ -76,7 +76,7 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 				| isPrivateAddress addr = False
 				| otherwise = True
 			let connectionrestricted = addrConnectionRestricted 
-				("Configuration of annex.security.allowed-http-addresses does not allow accessing address " ++)
+				("Configuration of annex.security.allowed-ip-addresses does not allow accessing address " ++)
 			let r = Restriction
 				{ addressRestriction = \addr ->
 					if isallowed (addrAddress addr)
@@ -88,13 +88,13 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
 			case pr of
 				Nothing -> return ()
 				Just ProxyRestricted -> toplevelWarning True
-					"http proxy settings not used due to annex.security.allowed-http-addresses configuration"
+					"http proxy settings not used due to annex.security.allowed-ip-addresses configuration"
 			manager <- liftIO $ U.newManager settings
 			return (U.DownloadWithConduit, manager)
 
-httpAddressesUnlimited :: Annex Bool
-httpAddressesUnlimited = 
-	("all" == ) . annexAllowedHttpAddresses <$> Annex.getGitConfig
+ipAddressesUnlimited :: Annex Bool
+ipAddressesUnlimited = 
+	("all" == ) . annexAllowedIPAddresses <$> Annex.getGitConfig
 
 withUrlOptions :: (U.UrlOptions -> Annex a) -> Annex a
 withUrlOptions a = a =<< getUrlOptions
diff --git a/Annex/YoutubeDl.hs b/Annex/YoutubeDl.hs
index c80d9bc34..9855a391a 100644
--- a/Annex/YoutubeDl.hs
+++ b/Annex/YoutubeDl.hs
@@ -31,13 +31,13 @@ import Control.Concurrent.Async
 -- localhost or a private address. So, it's only allowed to download
 -- content if the user has allowed access to all addresses.
 youtubeDlAllowed :: Annex Bool
-youtubeDlAllowed = httpAddressesUnlimited
+youtubeDlAllowed = ipAddressesUnlimited
 
 youtubeDlNotAllowedMessage :: String
 youtubeDlNotAllowedMessage = unwords
 	[ "This url is supported by youtube-dl, but"
 	, "youtube-dl could potentially access any address, and the"
-	, "configuration of annex.security.allowed-http-addresses"
+	, "configuration of annex.security.allowed-ip-addresses"
 	, "does not allow that. Not using youtube-dl."
 	]
 
@@ -55,7 +55,7 @@ youtubeDlNotAllowedMessage = unwords
 -- (Note that we can't use --output to specifiy the file to download to,
 -- due to <https://github.com/rg3/youtube-dl/issues/14864>)
 youtubeDl :: URLString -> FilePath -> Annex (Either String (Maybe FilePath))
-youtubeDl url workdir = ifM httpAddressesUnlimited
+youtubeDl url workdir = ifM ipAddressesUnlimited
 	( withUrlOptions $ youtubeDl' url workdir
 	, return $ Left youtubeDlNotAllowedMessage
 	)
diff --git a/CHANGELOG b/CHANGELOG
index 59ff62518..833a232cb 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -20,6 +20,10 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     bash.
   * When a remote is configured to be readonly, don't allow changing
     what's exported to it.
+  * Renamed annex.security.allowed-http-addresses to
+    annex.security.allowed-ip-addresses because it is not really specific
+    to the http protocol, also limiting eg, git-annex's use of ftp.
+    The old name for the config will still work.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs
index dfcd010cb..0bc72d402 100644
--- a/Types/GitConfig.hs
+++ b/Types/GitConfig.hs
@@ -98,7 +98,7 @@ data GitConfig = GitConfig
 	, annexRetry :: Maybe Integer
 	, annexRetryDelay :: Maybe Seconds
 	, annexAllowedUrlSchemes :: S.Set Scheme
-	, annexAllowedHttpAddresses :: String
+	, annexAllowedIPAddresses :: String
 	, annexAllowUnverifiedDownloads :: Bool
 	, annexMaxExtensionLength :: Maybe Int
 	, annexJobs :: Concurrency
@@ -172,8 +172,10 @@ extractGitConfig r = GitConfig
 	, annexAllowedUrlSchemes = S.fromList $ map mkScheme $
 		maybe ["http", "https", "ftp"] words $
 			getmaybe (annex "security.allowed-url-schemes")
-	, annexAllowedHttpAddresses = fromMaybe "" $
-		getmaybe (annex "security.allowed-http-addresses")
+	, annexAllowedIPAddresses = fromMaybe "" $
+		getmaybe (annex "security.allowed-ip-addresses")
+			<|>
+		getmaybe (annex "security.allowed-http-addresses") -- old name
 	, annexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $
 		getmaybe (annex "security.allow-unverified-downloads")
 	, annexMaxExtensionLength = getmayberead (annex "maxextensionlength")
diff --git a/doc/git-annex-addurl.mdwn b/doc/git-annex-addurl.mdwn
index 9854f1789..ce7ad04eb 100644
--- a/doc/git-annex-addurl.mdwn
+++ b/doc/git-annex-addurl.mdwn
@@ -13,7 +13,7 @@ Downloads each url to its own file, which is added to the annex.
 When `youtube-dl` is installed, it can be used to check for a video
 embedded in  a web page at the url, and that is added to the annex instead.
 (However, this is disabled by default as it can be a security risk. 
-See the documentation of annex.security.allowed-http-addresses
+See the documentation of annex.security.allowed-ip-addresses
 in [[git-annex]](1) for details.)
 
 Urls to torrent files (including magnet links) will cause the content of
diff --git a/doc/git-annex-importfeed.mdwn b/doc/git-annex-importfeed.mdwn
index d70b02cd9..6db005791 100644
--- a/doc/git-annex-importfeed.mdwn
+++ b/doc/git-annex-importfeed.mdwn
@@ -16,7 +16,7 @@ them.
 When `youtube-dl` is installed, it can be used to download links in the feed.
 This allows importing e.g., YouTube playlists.
 (However, this is disabled by default as it can be a security risk. 
-See the documentation of annex.security.allowed-http-addresses
+See the documentation of annex.security.allowed-ip-addresses
 in [[git-annex]](1) for details.)
 
 To make the import process add metadata to the imported files from the feed,
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 250a55a52..212c78985 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1426,7 +1426,7 @@ Here are all the supported configuration settings.
   Or to make curl use your ~/.netrc file, set it to "--netrc".
 
   Setting this option makes git-annex use curl, but only
-  when annex.security.allowed-http-addresses is configured in a
+  when annex.security.allowed-ip-addresses is configured in a
   specific way. See its documentation.
 
 * `annex.youtube-dl-options`
@@ -1469,10 +1469,11 @@ Here are all the supported configuration settings.
   Some special remotes support their own domain-specific URL
   schemes; those are not affected by this configuration setting.
 
-* `annex.security.allowed-http-addresses`
+* `annex.security.allowed-ip-addresses`
 
-  By default, git-annex only makes HTTP connections to public IP addresses;
-  it will refuse to use HTTP servers on localhost or on a private network.
+  By default, git-annex only makes connections to public IP addresses;
+  it will refuse to use HTTP and other servers on localhost or on a
+  private network.
 
   This setting can override that behavior, allowing access to particular
   IP addresses. For example "127.0.0.1 ::1" allows access to localhost
@@ -1480,13 +1481,19 @@ Here are all the supported configuration settings.
   
   Think very carefully before changing this; there are security
   implications. Anyone who can get a commit into your git-annex repository
-  could `git annex addurl` an url on a private http server, possibly
+  could `git annex addurl` an url on a private server, possibly
   causing it to be downloaded into your repository and transferred to
   other remotes, exposing its content.
 
   Note that, since the interfaces of curl and youtube-dl do not allow
   these IP address restrictions to be enforced, curl and youtube-dl will
-  never be used unless annex.security.allowed-http-addresses=all.
+  never be used unless annex.security.allowed-ip-addresses=all.
+
+* `annex.security.allowed-http-addresses`
+
+  Old name for annex.security.allowed-ip-addresses.
+  If set, this is treated the same as having
+  annex.security.allowed-ip-addresses set.
 
 * `annex.security.allow-unverified-downloads`
 
diff --git a/doc/news/security_fix_release.mdwn b/doc/news/security_fix_release.mdwn
index 10eb7bd45..63ce14853 100644
--- a/doc/news/security_fix_release.mdwn
+++ b/doc/news/security_fix_release.mdwn
@@ -21,7 +21,7 @@ security fixes:

(Diff truncated)

diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment
index ad1fa967c..53324a29e 100644
--- a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment
@@ -4,6 +4,17 @@
  date="2019-05-30T15:59:57Z"
  content="""
 Yet another problem: A ftp server might have both IPv4 and IPv6 addresses,
-and only one might work, so it seems git-annex will need to run curl more
-than once.
+and only one might work. So git-annex would need to run curl more
+than once if substituting in IP.
+
+Hmm.. curl has a --resolve that could be used instead of git-annex
+replacing the server hostname with its IP. This allows passing both ipv6
+and ipv4 addresses:
+
+	curl --resolve *:80:[::1],127.0.0.1 http://google.com/
+
+And it does also work for ftp and other protocols, but the protocol port
+number has to be included and can't be wildcarded. In particular, if 
+the ftp url is on a nonstandard port, that port has to be included,
+otherwise port 21.
 """]]

diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_4_b15837921956ad166cf53a5dd23cb7dd._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_4_b15837921956ad166cf53a5dd23cb7dd._comment
new file mode 100644
index 000000000..f08a5344f
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_4_b15837921956ad166cf53a5dd23cb7dd._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""ftp bounce attack"""
+ date="2019-05-30T15:53:26Z"
+ content="""
+In a PASV ftp connection, the server provides to the client an IP address
+and port to connect to. That is exploited by a ftp bounce attack.
+(Which I last thought about in like 1998? Why are we still using these bad
+old protocols?)
+
+So it seems git-annex can't rely on checking the ftp server IP is not
+local, because the non-local ftp server could use that to get the client to
+connect to a local ftp server. After content from that gets added to the
+git annex, we're back to CVE-2018-10857.
+
+curl defaults to PASV of course (active FTP is unlikely to work on the
+"modern" non-p2p internet). Seems curl does have a --ftp-skip-pasv-ip
+that makes it ignore whatever IP address the FTP server might present and
+just continue to use the same server IP.
+"""]]
diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment
new file mode 100644
index 000000000..ad1fa967c
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_5_2a66b890e3c60d1102acb7cef0b86308._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2019-05-30T15:59:57Z"
+ content="""
+Yet another problem: A ftp server might have both IPv4 and IPv6 addresses,
+and only one might work, so it seems git-annex will need to run curl more
+than once.
+"""]]

diff --git a/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_4_a5d5b908c4b9e91aff5eeb7cdb760ee7._comment b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_4_a5d5b908c4b9e91aff5eeb7cdb760ee7._comment
new file mode 100644
index 000000000..8cc8e8b87
--- /dev/null
+++ b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_4_a5d5b908c4b9e91aff5eeb7cdb760ee7._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="emanuele.olivetti@47d88ed185b03191e25329caa6fabc2efb3118b2"
+ nickname="emanuele.olivetti"
+ avatar="http://cdn.libravatar.org/avatar/f51cc5c6c3a0eb28faa6491c3cbcfcce"
+ subject="comment 4"
+ date="2019-05-29T17:36:06Z"
+ content="""
+Thank you for the great work done with `git-annex`!
+
+I am not much familiar with how the Debian maintainers decide to address - or not - a bug/patch like this one. But if dropping a line to support the [related Debian Bug Report](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928882) or subscribing to it could help, I invite everyone to do it. In my case, I have a 6TB RAID1 ARM Debian box meant to be used with `git-annex` but without it...
+"""]]

diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_3_921d2493f51ecb61b711773238c9f7e7._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_3_921d2493f51ecb61b711773238c9f7e7._comment
new file mode 100644
index 000000000..e4114dd15
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_3_921d2493f51ecb61b711773238c9f7e7._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2019-05-29T14:22:57Z"
+ content="""
+The netrc(5) file does use the hostname of the ftp server, so if git-annex
+swapped in an IP address it resolved it would not match the netrc file. But
+curl only reads the file when --netrc is used.
+
+If annex.web-options is set to --netrc (or anything), and
+annex.security.allowed-http-addresses=all, git-annex uses curl already
+and the security measures are disabled.
+
+So, git-annex could replace the ftp server hostname with the IP address
+when not so configured, and nothing that currently works would break, and
+this problem would be solved without needing any new configuration.
+"""]]

diff --git a/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_3_9117c76ad8f74c1f11eda2ecf7144402._comment b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_3_9117c76ad8f74c1f11eda2ecf7144402._comment
new file mode 100644
index 000000000..a5dd38de9
--- /dev/null
+++ b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_3_9117c76ad8f74c1f11eda2ecf7144402._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2019-05-29T13:58:43Z"
+ content="""
+Thanks for tracking down the ghc bug. The git-annex standalone build is
+built on Debian using its toolchain. So once that bug gets fixed, I will
+just need to make sure the builder is up-to-date to close this.
+"""]]

diff --git a/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server/comment_1_db2b9e26bc01abbc3312731b95cf770f._comment b/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server/comment_1_db2b9e26bc01abbc3312731b95cf770f._comment
new file mode 100644
index 000000000..1b18333c9
--- /dev/null
+++ b/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server/comment_1_db2b9e26bc01abbc3312731b95cf770f._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-29T13:19:08Z"
+ content="""
+Behavior change due to a security hole being fixed. See the discussion
+here:
+<http://git-annex.branchable.com/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/>
+
+While that bug was specific to redirects, as I note in the comment it's a
+more general reversion, but necessary due to the security fix. Perhaps
+I'll find a fix though.
+
+The workaround is this which makes git-annex use curl: 
+
+	git config annex.web-options -n
+	git config annex.security.allowed-http-addresses all
+"""]]

diff --git a/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_2_3dfc1d4c0fcd6b9af1205ad0f43bb1a7._comment b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_2_3dfc1d4c0fcd6b9af1205ad0f43bb1a7._comment
new file mode 100644
index 000000000..da997df73
--- /dev/null
+++ b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_2_3dfc1d4c0fcd6b9af1205ad0f43bb1a7._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="emanuele.olivetti@47d88ed185b03191e25329caa6fabc2efb3118b2"
+ nickname="emanuele.olivetti"
+ avatar="http://cdn.libravatar.org/avatar/f51cc5c6c3a0eb28faa6491c3cbcfcce"
+ subject="Misconfiguration in GHC"
+ date="2019-05-28T20:38:07Z"
+ content="""
+Apparently, the issue described above is due to a misconfiguration of `ghc`, as reported in this recent [Debian Bug Report (#928882)](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928882) which was triggered by [another bug report (#915333)](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915333) from users reporting `git-annex` not working in Debian on arm5te.
+
+In essence, this misconfiguration causes **all** Debian packages compiled with ghc not to work on ARM architectures below arm6. I hope that the proposed patch will be accepted soon.
+"""]]

diff --git a/doc/devblog/day_590__toward_importing_from_externals.mdwn b/doc/devblog/day_590__toward_importing_from_externals.mdwn
new file mode 100644
index 000000000..414038c27
--- /dev/null
+++ b/doc/devblog/day_590__toward_importing_from_externals.mdwn
@@ -0,0 +1,11 @@
+I've added an
+[[external_special_remote_protocol/export_and_import_appendix]]
+to the [[external_special_remote_protocol]] which documents how
+the protocol might be extended to allow for importing from external special
+remotes.
+
+Feel this needs more thought. It's complicated by there already
+being an interface that only supports export, and import needing all
+the same operations, but with more checks that the content has not been
+modified behind git-annex's back. Unifying them at the protocol level would
+be possible, but perhaps more confusing.

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 247ebd542..433aa4615 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -236,7 +236,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     Indicates that the directory was empty, but could not be
     removed for some reason.
 
-## protocol example
+### protocol example
 
 The protocol starts off with VERSION etc as usual, and
 then git-annex asks the special remote if it supports

diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 284b03cda..ddaad9ae6 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -156,6 +156,12 @@ The following requests *must* all be supported by the special remote.
   * `REMOVE-FAILURE Key ErrorMsg`  
     Indicates that the key was unable to be removed from the remote.
 
+Special remotes can optionally support tree exports and imports,
+which makes the [[git-annex-export]] and [[git-annex-import]] commands
+work with them. See the [[export_and_import_appendix]] for
+additional requests that git-annex will make when using special remotes in
+this way.
+
 The following requests can optionally be supported. If not supported,
 the special remote can reply with `UNSUPPORTED-REQUEST`.
 
@@ -230,74 +236,6 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
     Gives the value of an info field.
   * `INFOEND`  
     Indicates the end of the response block.
-* `EXPORTSUPPORTED`  
-  Used to check if a special remote supports exports.
-  Note that this request may be made before or after `PREPARE`.
-  * `EXPORTSUPPORTED-SUCCESS`  
-    Indicates that it makes sense to use this special remote as an export.
-  * `EXPORTSUPPORTED-FAILURE`  
-    Indicates that it does not make sense to use this special remote as an
-    export.
-* `EXPORT Name`  
-  Comes immediately before each of the following export-related requests, 
-  specifying the name of the exported file. It will be in the form
-  of a relative path, and may contain path separators, whitespace,
-  and other special characters.  
-  No response is made to this message.
-* `TRANSFEREXPORT STORE|RETRIEVE Key File`  
-  Requests the transfer of a File on local disk to or from the previously 
-  provided Name on the special remote.  
-  Note that it's important that, while a file is being stored,
-  CHECKPRESENTEXPORT not indicate it's present until all the data has
-  been transferred.  
-  While the transfer is running, the remote can send any number of
-  `PROGRESS` messages. Once the transfer is complete, it finishes by
-  sending one of these replies:
-  * `TRANSFER-SUCCESS STORE|RETRIEVE Key`  
-    Indicates the transfer completed successfully.
-  * `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`  
-    Indicates the transfer failed.
-* `CHECKPRESENTEXPORT Key`  
-  Requests the remote to check if the previously provided Name is present
-  in it.  
-  * `CHECKPRESENT-SUCCESS Key`  
-    Indicates that a content has been positively verified to be present in the
-    remote.
-  * `CHECKPRESENT-FAILURE Key`  
-    Indicates that a contents has been positively verified to not be present in the
-    remote.
-  * `CHECKPRESENT-UNKNOWN Key ErrorMsg`  
-    Indicates that it is not currently possible to verify if content is
-    present in the remote. (Perhaps the remote cannot be contacted.)
-* `REMOVEEXPORT Key`  
-  Requests the remote to remove content stored by `TRANSFEREXPORT`
-  with the previously provided Name.  
-  * `REMOVE-SUCCESS Key`  
-    Indicates the content has been removed from the remote. May be returned when
-    the content was already not present.
-  * `REMOVE-FAILURE Key ErrorMsg`  
-    Indicates that the content was unable to be removed from the remote.
-* `REMOVEEXPORTDIRECTORY Directory`  
-  Requests the remote remove an exported directory.  
-  If the remote does not use directories, or REMOVEEXPORT cleans up
-  directories that are empty, this does not need to be implemented.  
-  The directory will be in the form of a relative path, and may contain path
-  separators, whitespace, and other special characters.  
-  Typically the directory will be empty, but it could possibly contain
-  files or other directories, and it's ok to remove those.  
-  * `REMOVEEXPORTDIRECTORY-SUCCESS`  
-    Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully.
-  * `REMOVEEXPORTDIRECTORY-FAILURE`  
-    Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
-    Should not be returned if the directory did not exist.
-* `RENAMEEXPORT Key NewName`  
-  Requests the remote rename a file stored on it from the previously
-  provided Name to the NewName. Remotes that support exports but not
-  renaming do not need to implement this.  
-  * `RENAMEEXPORT-SUCCESS Key`  
-    Indicates that a `RENAMEEXPORT` was done successfully.
-  * `RENAMEEXPORT-FAILURE Key`  
-    Indicates that a `RENAMEEXPORT` failed for whatever reason.
 
 More optional requests may be added, without changing the protocol version,
 so if an unknown request is seen, don't crash, just reply with
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
new file mode 100644
index 000000000..247ebd542
--- /dev/null
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -0,0 +1,303 @@
+This is an appendix to the [[external_special_remote_protocol]].
+
+Some special remotes interface to a key/value datastore using keys that are
+eg hashes, and it won't make sense for them to implement any of this.
+
+When a special remote interfaces with something that looks like a directory
+of files, the simple export interface can be implemented to allow the
+[[git-annex-export]] command to be used to store file trees on the special
+remote.
+
+If other tools can write files to the special remote too, the import/export
+interface can be implemented. This allows for both [[git-annex-import]] and 
+[[git-annex-export]] to be used with the special remote.
+
+[[!toc]]
+
+## simple export interface
+
+git-annex will use this when the special remote is initialized with
+exporttree=yes but without importtree=yes and indicates that it supports
+exports.
+
+These are requests git-annex sends to the special remote program.
+Once the special remote has finished performing a request, it
+should send one of the listed replies. Or, if it does not support
+a request, it can reply with `UNSUPPORTED-REQUEST`.
+
+* `EXPORTSUPPORTED`  
+  Used to check if a special remote supports exports.
+  Note that this request may be made before or after `PREPARE`.
+  * `EXPORTSUPPORTED-SUCCESS`  
+    Indicates that it makes sense to export to this special remote.
+  * `EXPORTSUPPORTED-FAILURE`  
+    Indicates that it does not make sense to export to this special remote.
+* `EXPORT Name`  
+  Comes immediately before each of the following requests, 
+  specifying the name of the exported file. It will be in the form
+  of a relative path, and may contain path separators, whitespace,
+  and other special characters.  
+  No response is made to this message.
+* `TRANSFEREXPORT STORE|RETRIEVE Key File`  
+  Requests the transfer of a File on local disk to or from the previously 
+  provided Name on the special remote.  
+  Note that it's important that, while a file is being stored,
+  `CHECKPRESENTEXPORT` not indicate it's present until all the data has
+  been transferred.  
+  While the transfer is running, the remote can send any number of
+  `PROGRESS` messages. Once the transfer is complete, it finishes by
+  sending one of these replies:
+  * `TRANSFER-SUCCESS STORE|RETRIEVE Key`  
+    Indicates the transfer completed successfully.
+  * `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`  
+    Indicates the transfer failed.
+* `CHECKPRESENTEXPORT Key`  
+  Requests the remote to check if the previously provided Name is present
+  in it.  
+  * `CHECKPRESENT-SUCCESS Key`  
+    Indicates that a content has been positively verified to be present in the
+    remote.
+  * `CHECKPRESENT-FAILURE Key`  
+    Indicates that a contents has been positively verified to not be present in the
+    remote.
+  * `CHECKPRESENT-UNKNOWN Key ErrorMsg`  
+    Indicates that it is not currently possible to verify if content is
+    present in the remote. (Perhaps the remote cannot be contacted.)
+* `REMOVEEXPORT Key`  
+  Requests the remote to remove content stored by `TRANSFEREXPORT`
+  with the previously provided Name.  
+  * `REMOVE-SUCCESS Key`  
+    Indicates the content has been removed from the remote. May be returned when
+    the content was already not present.
+  * `REMOVE-FAILURE Key ErrorMsg`  
+    Indicates that the content was unable to be removed from the remote.
+* `REMOVEEXPORTDIRECTORY Directory`  
+  Requests the remote remove an exported directory.  
+  If the remote does not use directories, or `REMOVEEXPORT` cleans up
+  directories that are empty, this does not need to be implemented.  
+  The directory will be in the form of a relative path, and may contain path
+  separators, whitespace, and other special characters.  
+  Typically the directory will be empty, but it could possibly contain
+  files or other directories, and it's ok to remove those.  
+  * `REMOVEEXPORTDIRECTORY-SUCCESS`  
+    Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully.
+  * `REMOVEEXPORTDIRECTORY-FAILURE`  
+    Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
+    Should not be returned if the directory did not exist.
+* `RENAMEEXPORT Key NewName`  
+  Requests the remote rename a file stored on it from the previously
+  provided Name to the NewName. Remotes that support exports but not
+  renaming do not need to implement this.  
+  * `RENAMEEXPORT-SUCCESS Key`  
+    Indicates that a `RENAMEEXPORT` was done successfully.
+  * `RENAMEEXPORT-FAILURE Key`  
+    Indicates that a `RENAMEEXPORT` failed for whatever reason.
+
+## import/export interface
+
+(This part is a draft, not implemented yet.)
+
+git-annex will use this interface when the special remote is
+initialized with both exporttree=yes and importtree=yes and indicates
+that it supports both imports and exports.

(Diff truncated)

diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index c3b22a019..284b03cda 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -94,14 +94,12 @@ Now git-annex will send its next request.
 Once git-annex is done with the special remote, it will close its stdin.
 The special remote program can then exit.
 
-## git-annex request messages
+## git-annex request messages and replies
 
 These are messages git-annex sends to the special remote program.
-None of these messages require an immediate reply. The special
-remote can send any messages it likes while handling the requests.
 
 Once the special remote has finished performing the request, it should
-send one of the corresponding replies listed in the next section.
+send one of the listed replies.
 
 The following requests *must* all be supported by the special remote.
 
@@ -112,65 +110,134 @@ The following requests *must* all be supported by the special remote.
   different repositories, or as the configuration of a remote is changed.
   (Both `git annex initremote` and `git-annex enableremote` run this.)
   So any one-time setup tasks should be done idempotently.
+  * `INITREMOTE-SUCCESS`  
+    Indicates the INITREMOTE succeeded and the remote is ready to use.
+  * `INITREMOTE-FAILURE ErrorMsg`  
+    Indicates that INITREMOTE failed.
 * `PREPARE`  
   Tells the remote that it's time to prepare itself to be used.  
   Only a few requests for details about the remote can come before this.
-  Those include EXTENSIONS, INITREMOTE, EXPORTSUPPORTED, and may later
-  include others.
+  Those include EXTENSIONS, INITREMOTE, and EXPORTSUPPORTED, but others
+  may be added later.
+  * `PREPARE-SUCCESS`  
+    Sent as a response to PREPARE once the special remote is ready for use.
+  * `PREPARE-FAILURE ErrorMsg`  
+    Sent as a response to PREPARE if the special remote cannot be used.
 * `TRANSFER STORE|RETRIEVE Key File`  
   Requests the transfer of a key. For STORE, the File is the file to upload;
   for RETRIEVE the File is where to store the download.  
   Note that the File should not influence the filename used on the remote.  
   Note that in some cases, the File may contain whitespace.  
-  Note that it's important that, while a Key is being stored, CHECKPRESENT
+  It's important that, while a Key is being stored, `CHECKPRESENT`
   not indicate it's present until all the data has been transferred.  
+  While the transfer is running, the remote can send any number of
+  `PROGRESS` messages. Once the transfer is done, it finishes by sending
+  one of these replies:
+  * `TRANSFER-SUCCESS STORE|RETRIEVE Key`  
+    Indicates the transfer completed successfully.
+  * `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`  
+    Indicates the transfer failed.
 * `CHECKPRESENT Key`  
   Requests the remote to check if a key is present in it.
+  * `CHECKPRESENT-SUCCESS Key`  
+    Indicates that a key has been positively verified to be present in the
+    remote.
+  * `CHECKPRESENT-FAILURE Key`  
+    Indicates that a key has been positively verified to not be present in the
+    remote.
+  * `CHECKPRESENT-UNKNOWN Key ErrorMsg`  
+    Indicates that it is not currently possible to verify if the key is
+    present in the remote. (Perhaps the remote cannot be contacted.)
 * `REMOVE Key`  
   Requests the remote to remove a key's contents.
+  * `REMOVE-SUCCESS Key`  
+    Indicates the key has been removed from the remote. May be returned if
+    the remote didn't have the key at the point removal was requested.
+  * `REMOVE-FAILURE Key ErrorMsg`  
+    Indicates that the key was unable to be removed from the remote.
 
-The following requests can optionally be supported. If not handled,
-replying with `UNSUPPORTED-REQUEST` is acceptable.
+The following requests can optionally be supported. If not supported,
+the special remote can reply with `UNSUPPORTED-REQUEST`.
 
 * `EXTENSIONS List`  
   Sent to indicate protocol extensions which git-annex is capable
   of using. The list is a space-delimited list of protocol extension
   keywords. The remote can reply to this with its own EXTENSIONS list.
+  * `EXTENSIONS List`  
+    Sent in response to a EXTENSIONS request, the List could be used to indicate
+    protocol extensions that the special remote uses, but there are currently
+    no such extensions.
 * `GETCOST`  
   Requests the remote to return a use cost. Higher costs are more expensive.
   (See Config/Cost.hs for some standard costs.)
+  * `COST Int`  
+    Indicates the cost of the remote.
 * `GETAVAILABILITY`
-  Requests the remote to send back an `AVAILABILITY` reply.
+  Asks the remote if it is locally or globally available.
+  (Ie stored in the cloud vs on a local disk.)  
   If the remote replies with `UNSUPPORTED-REQUEST`, its availability
   is assumed to be global. So, only remotes that are only reachable
   locally need to worry about implementing this.
+  * `AVAILABILITY GLOBAL|LOCAL`  
+    Indicates if the remote is globally or only locally available.
 * `CLAIMURL Url`  
   Asks the remote if it wishes to claim responsibility for downloading
-  an url. If so, the remote should send back an `CLAIMURL-SUCCESS` reply.
-  If not, it can send `CLAIMURL-FAILURE`.
+  an url.
+  * `CLAIMURL-SUCCESS`  
+    Indicates that the CLAIMURL url will be handled by this remote.
+  * `CLAIMURL-FAILURE`  
+    Indicates that the CLAIMURL url wil not be handled by this remote.
 * `CHECKURL Url`  
   Asks the remote to check if the url's content can currently be downloaded
-  (without downloading it). The remote replies with one of `CHECKURL-FAILURE`,
-  `CHECKURL-CONTENTS`, or `CHECKURL-MULTI`.
+  (without downloading it).
+  * `CHECKURL-CONTENTS Size|UNKNOWN Filename`  
+    Indicates that the requested url has been verified to exist.  
+    The Size is the size in bytes, or use "UNKNOWN" if the size could not be
+    determined.  
+    The Filename can be empty (in which case a default is used),
+    or can specify a filename that is suggested to be used for this url.
+  * `CHECKURL-MULTI Url1 Size1|UNKNOWN Filename1 Url2 Size2|UNKNOWN Filename2 ...`  
+    Indicates that the requested url has been verified to exist,
+    and contains multiple files, which can each be accessed using
+    their own url.  Each triplet of url, size, and filename should be listed,
+    one after the other.
+    Note that since a list is returned, neither the Url nor the Filename
+    can contain spaces.
+  * `CHECKURL-FAILURE`  
+    Indicates that the requested url could not be accessed.
 * `WHEREIS Key`
   Asks the remote to provide additional information about ways to access
   the content of a key stored in it, such as eg, public urls.
-  This will be displayed to the user by eg, `git annex whereis`. The remote
-  replies with `WHEREIS-SUCCESS` or `WHEREIS-FAILURE`.  
+  This will be displayed to the user by eg, `git annex whereis`.
   Note that users expect `git annex whereis` to run fast, without eg,
   network access.  
-  This is not needed when `SETURIPRESENT` is used, since such uris are
-  automatically displayed by `git annex whereis`.  
+  * `WHEREIS-SUCCESS String`  
+    Indicates a location of a key. Typically an url, the string can
+    be anything that it makes sense to display to the user about content
+    stored in the special remote.
+  * `WHEREIS-FAILURE`  
+    Indicates that no location is known for a key.
+    This is not needed when `SETURIPRESENT` is used, since such uris are
+    automatically displayed by `git annex whereis`.  
 * `GETINFO`  
   Requests the remote to send some information describing its
-  configuration, for display by `git annex info`.  
-  Reply with a series of `INFOFIELD` each followed by `INFOVALUE`,
-  and concluded with `INFOEND`.
+  configuration, for display by `git annex info`. A block of responses
+  can be made to this, which must always end with `INFOEND`.
+  * `INFOFIELD Name`  
+    Gives the name of an info field. The name can be anything you want to
+    be displayed to the user. Must be immediately followed by `INFOVALUE`.
+  * `INFOVALUE Value`  
+    Gives the value of an info field.
+  * `INFOEND`  
+    Indicates the end of the response block.
 * `EXPORTSUPPORTED`  
-  Used to check if a special remote supports exports. The remote
-  responds with either `EXPORTSUPPORTED-SUCCESS` or
-  `EXPORTSUPPORTED-FAILURE`. Note that this request may be made before
-  or after `PREPARE`.
+  Used to check if a special remote supports exports.
+  Note that this request may be made before or after `PREPARE`.
+  * `EXPORTSUPPORTED-SUCCESS`  
+    Indicates that it makes sense to use this special remote as an export.
+  * `EXPORTSUPPORTED-FAILURE`  
+    Indicates that it does not make sense to use this special remote as an
+    export.
 * `EXPORT Name`  
   Comes immediately before each of the following export-related requests, 
   specifying the name of the exported file. It will be in the form
@@ -183,21 +250,33 @@ replying with `UNSUPPORTED-REQUEST` is acceptable.
   Note that it's important that, while a file is being stored,
   CHECKPRESENTEXPORT not indicate it's present until all the data has
   been transferred.  
-  The remote responds with either `TRANSFER-SUCCESS` or
-  `TRANSFER-FAILURE`, and a remote where exports do not make sense
-  may always fail.
+  While the transfer is running, the remote can send any number of
+  `PROGRESS` messages. Once the transfer is complete, it finishes by
+  sending one of these replies:
+  * `TRANSFER-SUCCESS STORE|RETRIEVE Key`  
+    Indicates the transfer completed successfully.
+  * `TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg`  
+    Indicates the transfer failed.
 * `CHECKPRESENTEXPORT Key`  
   Requests the remote to check if the previously provided Name is present
   in it.  
-  The remote responds with `CHECKPRESENT-SUCCESS`, `CHECKPRESENT-FAILURE`,
-  or `CHECKPRESENT-UNKNOWN`.
+  * `CHECKPRESENT-SUCCESS Key`  
+    Indicates that a content has been positively verified to be present in the
+    remote.
+  * `CHECKPRESENT-FAILURE Key`  

(Diff truncated)

diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment
new file mode 100644
index 000000000..74978945f
--- /dev/null
+++ b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="readonly special remotes]"
+ date="2019-05-28T17:25:19Z"
+ content="""
+\"Setting readonly=true for a special remote prevents using the external special remote program at all\" -- that's the part I didn't realize.  If I understand correctly:
+
+1. for all remotes, setting `readonly=true` \"prevents git-annex from making changes to a remote... this both prevents git-annex sync from pushing changes, and prevents storing or removing files from read-only remote.\"
+2. additionally, for special remotes, this \"prevents using the external special remote program at all\", and instead causes `git-annex` to be [downloading the content of a file using a regular http connection, with no authentication](https://git-annex.branchable.com/design/external_special_remote_protocol/#index9h2)?
+
+If so, it would be best to add a separate config setting for (2).  I'm not sure, though, why that case needs special handling.  If a key's contents is available at a regular `http` URL, shouldn't the key be recorded as present in the built-in web special remote, rather than the user-defined special remote?
+
+The external special remote here is one I wrote myself, for accessing data on [DNAnexus](https://www.dnanexus.com/).
+"""]]

diff --git a/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_1_1143917decff204a3965959d1cfaa8f4._comment b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_1_1143917decff204a3965959d1cfaa8f4._comment
new file mode 100644
index 000000000..ed696deb7
--- /dev/null
+++ b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__/comment_1_1143917decff204a3965959d1cfaa8f4._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="emanuele.olivetti@47d88ed185b03191e25329caa6fabc2efb3118b2"
+ nickname="emanuele.olivetti"
+ avatar="http://cdn.libravatar.org/avatar/f51cc5c6c3a0eb28faa6491c3cbcfcce"
+ subject="git-annex not working on arm at all?"
+ date="2019-05-28T17:23:57Z"
+ content="""
+Further attempts to get a working git-annex on arm (kirkwood), all failed:
+
+1. I tried to compile git-annex from source, on Debian stretch on my ARM machine - armel, kirkwood subarchitecture, it's a [QNAP Turbo Station](https://www.debian.org/releases/stable/armel/ch02s01.html.en) - following the instruction in [fromsource](http://git-annex.branchable.com/install/fromsource/). Make failed with [this error](https://gist.github.com/emanuele/54d0fb1e8905cb48da0cdf9371679462). Additional comments: I used `sudo apt-get build-dep git-annex` to install dependencies but libghc-microlens was missing and I had to install it separately. Make required a large amount of RAM: >500Mb or resident memory and >1Gb of virtual memory, which is more than what's available on such hardware. So make failed but re-running it multiple times worked, until the bug above.
+2. I installed git-annex from the official package of Debian testing (buster), via `apt`. The installation went well. Unfortunately, the executable does not work, issuing the same error mentioned before:
+
+```
+    # which git-annex
+    /usr/bin/git-annex
+    # git annex
+    error: git-annex died of signal 4
+    # git-annex
+    Illegal instruction
+```
+
+Thanks in advance for any help.
+
+"""]]

diff --git a/doc/bugs/Support_manual_configuration_of_tor_hidden_service.mdwn b/doc/bugs/Support_manual_configuration_of_tor_hidden_service.mdwn
new file mode 100644
index 000000000..59d863c38
--- /dev/null
+++ b/doc/bugs/Support_manual_configuration_of_tor_hidden_service.mdwn
@@ -0,0 +1,23 @@
+### Please describe the problem.
+I am running NixOS, so the automatic configuration of the tor hidden service does not work because system state is not mutable. Could you provide a way to configure the tor hidden service manually, perhaps printing out the configuration instead of installing it?
+
+### What steps will reproduce the problem?
+N/A
+
+### What version of git-annex are you using? On what operating system?
+7.2, NixOS
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+Yes, I love it.
+
+

diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 9dec284b9..c3b22a019 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -114,7 +114,9 @@ The following requests *must* all be supported by the special remote.
   So any one-time setup tasks should be done idempotently.
 * `PREPARE`  
   Tells the remote that it's time to prepare itself to be used.  
-  Only EXTENSIONS and INITREMOTE or EXPORTSUPPORTED can come before this.
+  Only a few requests for details about the remote can come before this.
+  Those include EXTENSIONS, INITREMOTE, EXPORTSUPPORTED, and may later
+  include others.
 * `TRANSFER STORE|RETRIEVE Key File`  
   Requests the transfer of a key. For STORE, the File is the file to upload;
   for RETRIEVE the File is where to store the download.  

diff --git a/doc/special_remotes/external.mdwn b/doc/special_remotes/external.mdwn
index 63a7b929b..7b89944bc 100644
--- a/doc/special_remotes/external.mdwn
+++ b/doc/special_remotes/external.mdwn
@@ -21,6 +21,13 @@ It's not hard!
 * If you build a new special remote, please add it to the list
   of [[special_remotes]].
 
+## libraries
+
+For Python, there is a [library](https://github.com/Lykos153/AnnexRemote)
+that takes care of all the protocol details.
+
+## examples
+
 Here's an example of using an external special remote to add torrent
 support to git-annex: [[external/git-annex-remote-torrent]]
 
@@ -28,6 +35,4 @@ Here's a simple shell script example, which can easily be adapted
 to run whatever commands you need. Or better, re-written in some better
 language of your choice.
 
-For Python, there is a [library](https://github.com/Lykos153/AnnexRemote) that takes care of all the protocol details.
-
 [[!inline pages="special_remotes/external/example.sh" feeds=no]]

diff --git a/Remote/Directory.hs b/Remote/Directory.hs
index 72ac3c693..5cfb70254 100644
--- a/Remote/Directory.hs
+++ b/Remote/Directory.hs
@@ -81,6 +81,8 @@ gen r u c gc = do
 				, retrieveExportWithContentIdentifier = retrieveExportWithContentIdentifierM dir
 				, storeExportWithContentIdentifier = storeExportWithContentIdentifierM dir
 				, removeExportWithContentIdentifier = removeExportWithContentIdentifierM dir
+				-- Not needed because removeExportWithContentIdentifier
+				-- auto-removes empty directories.
 				, removeExportDirectoryWhenEmpty = Nothing
 				, checkPresentExportWithContentIdentifier = checkPresentExportWithContentIdentifierM dir
 				}
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 36bf5cb92..9dec284b9 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -198,8 +198,8 @@ replying with `UNSUPPORTED-REQUEST` is acceptable.
   respond with `REMOVE-SUCCESS`.
 * `REMOVEEXPORTDIRECTORY Directory`  
   Requests the remote remove an exported directory.  
-  If the remote does not use directories, or automatically cleans up
-  empty directories, this does not need to be implemented.  
+  If the remote does not use directories, or REMOVEEXPORT cleans up
+  directories that are empty, this does not need to be implemented.  
   The directory will be in the form of a relative path, and may contain path
   separators, whitespace, and other special characters.  
   Typically the directory will be empty, but it could possibly contain
diff --git a/doc/design/importing_trees_from_special_remotes.mdwn b/doc/design/importing_trees_from_special_remotes.mdwn
index ca5def33e..5e1603c26 100644
--- a/doc/design/importing_trees_from_special_remotes.mdwn
+++ b/doc/design/importing_trees_from_special_remotes.mdwn
@@ -281,7 +281,9 @@ the race.
 
 removeExportDirectoryWhenEmpty is used instead of removeExportDirectory.
 It should only remove empty directories, and succeeds if there are files
-in the directory.
+in the directory. This is needed because removeExportDirectory removes
+non-empty directories, but when importing the stuff in those directories is
+stuff that ought to be imported.
 
 checkPresentExportWithContentIdentifier is used instead of
 checkPresentExport. It should verify that one of the provided

diff --git a/CHANGELOG b/CHANGELOG
index 430825bf1..59ff62518 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -18,6 +18,8 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     the description to "", now it will error out.
   * Android: Improve installation process when the user's login shell is not
     bash.
+  * When a remote is configured to be readonly, don't allow changing
+    what's exported to it.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/Remote/Helper/ReadOnly.hs b/Remote/Helper/ReadOnly.hs
index d98278616..add92aa1e 100644
--- a/Remote/Helper/ReadOnly.hs
+++ b/Remote/Helper/ReadOnly.hs
@@ -1,6 +1,6 @@
 {- Adds readonly support to remotes.
  -
- - Copyright 2013, 2015 Joey Hess <id@joeyh.name>
+ - Copyright 2013-2019 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -28,6 +28,17 @@ adjustReadOnly r
 		{ storeKey = readonlyStoreKey
 		, removeKey = readonlyRemoveKey
 		, repairRepo = Nothing
+		, exportActions = exportActions r
+			{ storeExport = readonlyStoreExport
+			, removeExport = readonlyRemoveExport
+			, removeExportDirectory = Just readonlyRemoveExportDirectory
+			, renameExport = readonlyRenameExport
+			}
+		, importActions = importActions r
+			{ storeExportWithContentIdentifier = readonlyStoreExportWithContentIdentifiera
+			, removeExportWithContentIdentifier = readonlyRemoveExportWithContentIdentifier
+			, removeExportDirectoryWhenEmpty = Just readonlyRemoveExportDirectory
+			}
 		}
 	| otherwise = r
 
@@ -40,7 +51,30 @@ readonlyRemoveKey _ = readonlyFail
 readonlyStorer :: Storer
 readonlyStorer _ _ _ = readonlyFail
 
+readonlyStoreExport :: FilePath -> Key -> ExportLocation -> MeterUpdate -> Annex Bool
+readonlyStoreExport _ _ _ _ = readonlyFail
+
+readonlyRemoveExport :: Key -> ExportLocation -> Annex Bool
+readonlyRemoveExport _ _ = readonlyFail
+
+readonlyRemoveExportDirectory :: ExportDirectory -> Annex Bool
+readonlyRemoveExportDirectory _ = readonlyFail
+
+readonlyRenameExport :: Key -> ExportLocation -> ExportLocation -> Annex (Maybe Bool)
+readonlyRenameExport _ _ _ = return Nothing
+
+readonlyStoreExportWithContentIdentifier :: FilePath -> Key -> ExportLocation -> [ContentIdentifier] -> MeterUpdate -> Annex (Maybe ContentIdentifier)
+readonlyStoreExportWithContentIdentifier _ _ _ _ _ = do
+	readonlyWarning
+	return Nothing
+
+removeExportWithContentIdentifier :: Key -> ExportLocation -> [ContentIdentifier] -> Annex Bool
+removeExportWithContentIdentifier _ _ _ = readonlyFail
+
 readonlyFail :: Annex Bool
 readonlyFail = do
-	warning "this remote is readonly"
+	readonlyWarning
 	return False
+
+readonlyWarning :: Annex ()
+readonylWarning = warning "this remote is readonly"
diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp
deleted file mode 100644
index 0f0043862..000000000
Binary files a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp and /dev/null differ

diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment
new file mode 100644
index 000000000..8c1fe491d
--- /dev/null
+++ b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment
@@ -0,0 +1,23 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-28T14:29:45Z"
+ content="""
+The error message is git-annex trying to access a "dx://" url, which is not
+allowed by the default configuration. (Also which git-annex would have no
+idea how to access even if it were allowed.)
+
+Setting readonly=true for a special remote prevents using the external 
+special remote program at all, which would presumably do something that
+worked better if it ran.
+
+This might be a bug in the special remote you're using, if its
+documentation suggests it will work in readonly mode and it only registers
+these "dx://" urls that can't work in readonly mode.
+
+Did you set remote.name.annex-readonly manually, or did you run
+git annex enableremote with readonly=true? Only the latter is documented to
+disable use of the special remote program, which could be considered a
+documentation bug in git-annex. That's the only potential git-annex bug I'm
+seeing here.
+"""]]

diff --git a/doc/bugs/smudge_errors_on_os-x/comment_3_ee9dd8af508516ee8a1622efbd10642e._comment b/doc/bugs/smudge_errors_on_os-x/comment_3_ee9dd8af508516ee8a1622efbd10642e._comment
new file mode 100644
index 000000000..1bcbab308
--- /dev/null
+++ b/doc/bugs/smudge_errors_on_os-x/comment_3_ee9dd8af508516ee8a1622efbd10642e._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="ndj"
+ avatar="http://cdn.libravatar.org/avatar/0e1938953fb670f94a4e65b81a3cc58d"
+ subject="comment 3"
+ date="2019-05-28T03:36:52Z"
+ content="""
+I did more digging and I it looks like the issue is that I have a very old version of git (2.2.1) installed via brew that was taking precedence over the system git (2.11.0). Taking that out of the mix made everything work.
+
+Also, by just using `runshell`, it worked as well. I may switch to using that method of adding git-annex to my path, rather than just manually tacking it on the end.
+
+This is on MacOS 10.12.6 and git annex version 7.20190220-g1812ec5da.
+
+Thank you both for your help.
+"""]]

diff --git a/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__.mdwn b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__.mdwn
new file mode 100644
index 000000000..70caaacd0
--- /dev/null
+++ b/doc/bugs/Linux_standalone_on_armv5tel__58___error__58_____34__git-annex_died_of_signal_4__34__.mdwn
@@ -0,0 +1,15 @@
+The current Linux standalone for ARM, used on armv5tel with Debian stretch, does not work:
+
+    $ cd git-annex.linux/
+    $ ./runshell 
+    $ git annex
+    error: git-annex died of signal 4
+    $ git-annex
+    Illegal instruction
+
+
+I downloaded the arm tarball, unpacked and did as above. Same if do as above as root or use the alternative way of modifying the PATH, as suggested in the installation instructions. The CPU is Feroceon 88FR131 rev 1 (v5l).
+
+Is there a way of getting a working pre-built git-annex on such a system?
+
+Thanks in advance!

diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn b/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn
new file mode 100644
index 000000000..7230b5c5e
--- /dev/null
+++ b/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn
@@ -0,0 +1,10 @@
+In an external special remote, if I set annex-readonly=true, I get
+
+[[!format sh """
+ git annex --verbose --debug checkpresentkey MD5E-s18932--663e2e38c829d7b3fff12fce3a6fdb6d.fasta 380286ac-2e8f-4285-94da-406eca323411  
+...
+(checking dnanexus...) Configuration does not allow accessing dx://file-BXF0vYQ0QyBF509G9J12g927/contaminants.clip_db.fasta
+"""]]
+
+Removing the annex-readonly setting lets checkpresentkey work.   But isn't checkpresentkey a read-only operation?
+

diff --git a/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server.mdwn b/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server.mdwn
new file mode 100644
index 000000000..8e8b1bca7
--- /dev/null
+++ b/doc/bugs/__34__Unsupported_url_scheme__34___message_when_trying_to_cconnect_to_ftp_server.mdwn
@@ -0,0 +1,52 @@
+I am trying to use git-annex as part of a datalad install on an Ubuntu 18.04 VirtualBox VM to set up a repository linking to some datasets at a public ftp server.  When I use git-annex version 7.20190219 (installed with apt-get install git-annex-standalone from NeuroDebian) to add the links, I get the following error:
+
+emmet@emmet-VirtualBox:~/conp-dataset/projects/1KGP-22May2019$ git annex addurl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz –file=1KGP_chr6-vcf.gz
+addurl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz Unsupported url scheme ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
+
+download failed: Unsupported url scheme ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
+failed
+git-annex: addurl: 1 failed
+
+
+
+However when I use git-annex v 6.20180227 (installed from the apt repository that came with Ubuntu 18.04) the link is generated correctly:
+
+emmet@emmet-VirtualBox:~/conp-dataset/projects/1KGP-22May2019$ git annex addurl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz --file=1KGP_chr6-vcf.gz
+addurl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
+ALL.chr6.phase3_shapeit2_mvncall_integrated_ 100%[===========================================================================================>] 953.14M  9.81MB/s    in 93s    
+2019-05-22 16:10:44 URL: ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr6.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz [999436830] -> "/home/emmet/conp-dataset/projects/1KGP-22May2019/.git/annex/tmp/URL-s999436830--ftp&c%%ftp.1000genomes.ebi.ac.uk-dda0592051d00d19f6c947a6b3aae6a5" [1]
+(to 1KGP_chr6-vcf.gz) ok
+(recording state in git...)
+
+
+
+Similarly, after publishing the correctly formed links generated with 6.20180227 to github, trying to download them on another Ubuntu VirtualBox VM with git-annex 7.20190219 installed causes the following error:
+
+emmet@emmet-VirtualBox:~/conp-dataset/projects/1KGP-22May2019$ git annex get 1KGP_chr2-vcf.gz
+get 1KGP_chr2-vcf.gz (from web...)
+download failed: Unsupported url scheme ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr2.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
+
+  Unable to access these remotes: web
+
+  Try making some of these repositories available:
+      00000000-0000-0000-0000-000000000001 -- web
+       2cc9a182-813f-4740-8157-c7a560560238 -- emmet@emmet-VirtualBox:~/conp-dataset/projects/1KGP-22May2019
+
+  (Note that these git remotes have annex-ignore set: origin)
+failed
+git-annex: get: 1 failed
+
+
+
+but downloading them on that same VM with git-annex v 6.20180227 installed instead works:
+
+emmet@emmet-VirtualBox:~/conp-dataset/projects/1KGP-22May2019$ git annex get 1KGP_chr2-vcf.gz
+get 1KGP_chr2-vcf.gz (from web...)
+ALL.chr2.phase3_sha 100%[===================>]   1.22G  6.99MB/s    in 3m 11s 
+2019-05-22 16:25:44 URL: ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr2.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz [1312735578] -> "/home/emmet/conp-dataset/projects/1KGP-22May2019/.git/annex/tmp/MD5E-s1312735578--774370621affaf125e6994089f975ed7.gz" [1]
+(checksum...) ok
+(recording state in git...)
+
+
+Would you know why this is happening?  And how I can avoid this failure mode in the more up-to-date version of git-annex?
+ 

diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_2_860458173cb4d745e8c9833b804398d8._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_2_860458173cb4d745e8c9833b804398d8._comment
new file mode 100644
index 000000000..b286f624c
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_2_860458173cb4d745e8c9833b804398d8._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="configuration variable"
+ date="2019-05-23T19:31:46Z"
+ content="""
+I think then we could add in DataLad ability to record in the repository's configuration custom git annex configuration options to be passed to git annex calls (for that repo) and this way mitigate it where needed
+"""]]

diff --git a/doc/design/roadmap.mdwn b/doc/design/roadmap.mdwn
index 8e43ca269..67ac62bfd 100644
--- a/doc/design/roadmap.mdwn
+++ b/doc/design/roadmap.mdwn
@@ -18,9 +18,8 @@ Speed improvements, including:
   encode data stored in sqlite.
 
 Improvements to tree export support, including supporting export to more
-external special remotes, exporting only preferred content, more
+external special remotes, more
 efficient renames, and improving support for exporting non-annexed files.
-<https://git-annex.branchable.com/todo/export_preferred_content/>
 <https://git-annex.branchable.com/todo/export_paired_rename_innefficenctcy/>
 
 Improve support for branches where annexed files without content locally

diff --git a/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_1_2d68f1f5019621bccfd59be5066fdcbe._comment b/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_1_2d68f1f5019621bccfd59be5066fdcbe._comment
new file mode 100644
index 000000000..fb7121c56
--- /dev/null
+++ b/doc/bugs/git-annex__58___.git__47__annex__47__keys.tmp__47__db__58___setFileMode__58___permission_denied___40__Operation_not_permitted__41__/comment_1_2d68f1f5019621bccfd59be5066fdcbe._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T18:06:39Z"
+ content="""
+I remeber that git message, it's trying to remove the world read bit
+from .git/config and the hack that is /sdcard does not support that.
+
+On my phone (Android 8.1), neither git nor git-annex has the problem.
+/sdcard still doesn't really support unix file permissions, but chmod on 
+it no longer fails, just silently leaves wrong permissions.
+
+Given that there are plenty of places in git-annex that set file modes,
+I'd rather not complicate it by ignoring a failure to do so.
+It would not be a problem to fix this one specific case of it, which
+is in Database/Init.hs's callto setFileMode, but I don't know if fixing
+that won't just expose lots of other problems.
+"""]]

diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp
new file mode 100644
index 000000000..0f0043862
Binary files /dev/null and b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/.comment_1_5184c1b87e249f203541f610278ed08e._comment.swp differ
diff --git a/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_1_5184c1b87e249f203541f610278ed08e._comment b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_1_5184c1b87e249f203541f610278ed08e._comment
new file mode 100644
index 000000000..fe831bfe0
--- /dev/null
+++ b/doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported/comment_1_5184c1b87e249f203541f610278ed08e._comment
@@ -0,0 +1,32 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T17:39:36Z"
+ content="""
+The old behavior changed due to a security fix in commit
+[[!commit b54b2cdc0ef1373fc200c0d28fded3c04fd57212]].
+
+When git-annex followed a http redirect with curl, the http server
+could provide a ftp url that causes curl to access localhost or the lan.
+Which basically violates annex.security.allowed-http-addresses.
+(That is aimed at http, but accessing a ftp server would have similar
+security problems.)
+
+(For the same reason, non-redirect ftp urls don't work unless using 
+curl is enabled.)
+
+To avoid that attack while following the ftp redirect, git-annex would need
+to parse out the ftp server's hostname, resolve it and I suppose substitute
+the IP address back into the ftp url before passing it to curl. (To avoid
+DNS tricks.)
+
+I don't know if replacing a ftp hostname with an IP address would always
+work; there may be something that cares about the specific ftp hostname
+to eg decide what password to give the ftp server? Been a while since I FTPed.
+
+----
+
+Maybe a new configuration item is a better fix?
+`git config annex.security.allowed-ftp-addresses all` could enable
+use of curl as needed for ftp, without forcing use of curl for http.
+"""]]

diff --git a/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment b/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment
new file mode 100644
index 000000000..7bf3de4cc
--- /dev/null
+++ b/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T17:37:33Z"
+ content="""
+Can you be more specific please?
+"""]]

diff --git a/doc/todo/checksum_verification_on_transfer/comment_5_30add692347382aaa718b9b3ee7f116d._comment b/doc/todo/checksum_verification_on_transfer/comment_5_30add692347382aaa718b9b3ee7f116d._comment
new file mode 100644
index 000000000..27c061edd
--- /dev/null
+++ b/doc/todo/checksum_verification_on_transfer/comment_5_30add692347382aaa718b9b3ee7f116d._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2019-05-23T17:34:40Z"
+ content="""
+It seems better to keep checksums specific to a given protocol at the level
+of that protocol. What would be the benefit of pushing that up to
+git-annex's APIs?
+"""]]

diff --git a/doc/bugs/adb_special_remote_fetches_but_does_not_add_all_files/comment_1_31142cacafa0c975cd33a76c22f60bf3._comment b/doc/bugs/adb_special_remote_fetches_but_does_not_add_all_files/comment_1_31142cacafa0c975cd33a76c22f60bf3._comment
new file mode 100644
index 000000000..acc84d41f
--- /dev/null
+++ b/doc/bugs/adb_special_remote_fetches_but_does_not_add_all_files/comment_1_31142cacafa0c975cd33a76c22f60bf3._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T17:21:53Z"
+ content="""
+"Failed to import some files from remotename. Re-run command to resume import."
+
+AFAICS if it displayed that, it would not go on to add *any* files
+to the remote's tracking branch. Only when it's been able to import all
+the files it found on the remote does a commit get made.
+
+I have seen the adb pull of a file fail once or twice (out of several
+hundred imports so far), and adb did display an error message, and I
+assume git-annex also then said the import of that file failed.
+
+The other way it could fail to download a file is if the content seems to
+have changed since file list for the import was built. In that case,
+there would be no error message, just "import foo failed" rather than 
+"import foo ok"
+
+Can you paste a transcript of the problem?
+"""]]

diff --git a/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_2_96312bab546412b1eecfa74aec1d7fee._comment b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_2_96312bab546412b1eecfa74aec1d7fee._comment
new file mode 100644
index 000000000..14598c58c
--- /dev/null
+++ b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_2_96312bab546412b1eecfa74aec1d7fee._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="comment 2"
+ date="2019-05-23T17:07:40Z"
+ content="""
+I'll try to reproduce this and re-run under `strace`.  Maybe also, take another look at [[todo/add_tests_under_concurrency]]?  I think that re-running concurrent operations on a small repo many times should test all relevant operation interleavings, and bring out the concurrency-specific bugs.
+
+With large repos, concurrency is the only way to make operations run in sane time.  So fixing this would help a lot.
+
+"""]]

diff --git a/CHANGELOG b/CHANGELOG
index 9718bf95b..430825bf1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -16,6 +16,8 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
   * init: When the repository already has a description, don't change it.
   * describe: When run with no description parameter it used to set
     the description to "", now it will error out.
+  * Android: Improve installation process when the user's login shell is not
+    bash.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
index 5cea6440d..f4f5e661e 100644
--- a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
+++ b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
@@ -29,3 +29,7 @@ git: 'annex' is not a git command. See 'git --help'.
 Yes, been using it for many years and couldn't live without it.
 
 [[!meta title="termux install adds git-annex only to bash path, not zsh etc"]]
+
+> made it detect when the login shell is not bash, and rather than add to
+> .profile, print out a message letting the user know what they need to
+> add to their shell's path [[done]]
diff --git a/standalone/linux/skel/runshell b/standalone/linux/skel/runshell
index 32849e3f8..0c3919919 100755
--- a/standalone/linux/skel/runshell
+++ b/standalone/linux/skel/runshell
@@ -185,7 +185,7 @@ case "$os" in
 	# Make this bundle work well on Android.
 	Android)
 		if [ -e "$base/git" ]; then
-			echo "Running on Android.. Adding git-annex to PATH for you, and tuning for optimal behavior." >&2
+			echo "Running on Android.. Tuning for optimal behavior." >&2
 			# The bundled git does not work well on sdcard, so delete
 			# it and use termux's git which works better.
 			cd "$base"
@@ -197,8 +197,13 @@ case "$os" in
 			termux-fix-shebang bin/* runshell git-annex git-annex-shell git-annex-webapp 
 			cd "$orig"
 			# Save the poor Android user the typing.
-			if ! [ -e "$HOME/.profile" ] || ! grep -q "$base" "$HOME/.profile"; then
-				echo 'PATH=$PATH:'"$base" >> $HOME/.profile
+			if echo "$SHELL" | grep -q '/bash'; then
+				if ! [ -e "$HOME/.profile" ] || ! grep -q "$base" "$HOME/.profile"; then
+					echo "Adding git-annex to PATH for you, in $HOME/.profile"
+					echo 'PATH=$PATH:'"$base" >> $HOME/.profile
+				fi
+			else
+				echo "To use git-annex, you will need to add $base to your shell's PATH."
 			fi
 		fi
 		

diff --git a/Annex/Init.hs b/Annex/Init.hs
index 88389c138..cb7f8905e 100644
--- a/Annex/Init.hs
+++ b/Annex/Init.hs
@@ -1,6 +1,6 @@
 {- git-annex repository initialization
  -
- - Copyright 2011-2017 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2019 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -52,6 +52,8 @@ import System.Posix.User
 import qualified Utility.LockFile.Posix as Posix
 #endif
 
+import qualified Data.Map as M
+
 checkCanInitialize :: Annex a -> Annex a
 checkCanInitialize a = inRepo (noAnnexFileContent . Git.repoWorkTree) >>= \case
 	Nothing -> a
@@ -88,7 +90,10 @@ initialize mdescription mversion = checkCanInitialize $ do
 	initSharedClone sharedclone
 
 	u <- getUUID
-	describeUUID u =<< genDescription mdescription
+	{- Avoid overwriting existing description with a default
+	 - description. -}
+	whenM (pure (isJust mdescription) <||> not . M.member u <$> uuidDescMap) $
+		describeUUID u =<< genDescription mdescription
 
 -- Everything except for uuid setup, shared clone setup, and initial
 -- description.
diff --git a/CHANGELOG b/CHANGELOG
index 943c62cf1..9718bf95b 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -13,6 +13,9 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     import with such an expression will fail.
   * Improve shape of commit tree when importing from unversioned special
     remotes.
+  * init: When the repository already has a description, don't change it.
+  * describe: When run with no description parameter it used to set
+    the description to "", now it will error out.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/Command/Describe.hs b/Command/Describe.hs
index 2135482da..9e1533438 100644
--- a/Command/Describe.hs
+++ b/Command/Describe.hs
@@ -21,7 +21,7 @@ seek :: CmdParams -> CommandSeek
 seek = withWords (commandAction . start)
 
 start :: [String] -> CommandStart
-start (name:description) = do
+start (name:description) | not (null description) = do
 	showStart' "describe" (Just name)
 	u <- Remote.nameToUUID name
 	next $ perform u $ unwords description
diff --git a/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn
index 52885b300..6687d61cc 100644
--- a/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn
+++ b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn
@@ -18,3 +18,5 @@ upgrade supported from repository versions: 0 1 2 3 4 5 6
 local repository version: 5
 (master_env_v135_py36) 13:47  [r2] $ uname -a
 Linux ip-172-31-90-58.ec2.internal 4.14.114-105.126.amzn2.x86_64 #1 SMP Tue May 7 02:26:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description/comment_1_7facb84de8c8148e8f12ba2c4bd5c74b._comment b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description/comment_1_7facb84de8c8148e8f12ba2c4bd5c74b._comment
new file mode 100644
index 000000000..1dd9222c9
--- /dev/null
+++ b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description/comment_1_7facb84de8c8148e8f12ba2c4bd5c74b._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T16:40:50Z"
+ content="""
+Yikes, I was surprised to see this behavior. fixed for `init`.
+
+For `describe`, I see that when it's run with no parameter for the
+description, it sets the description to "". I suppose that's what you
+meant.
+
+I suppose something could conceivably depend on the current `git annex
+describe remote` behavior, but I checked and it's nowhere documented that
+describe with no description is allowed -- even the --help doesn't say that
+luckily. So I have also disallowed that.
+"""]]

diff --git a/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_1_b38f13a4c28ad8ce26aef72e7d6fcb11._comment b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_1_b38f13a4c28ad8ce26aef72e7d6fcb11._comment
new file mode 100644
index 000000000..e9b4702eb
--- /dev/null
+++ b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add/comment_1_b38f13a4c28ad8ce26aef72e7d6fcb11._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-05-23T16:18:23Z"
+ content="""
+This may be somehow related to the other bug in that it might also have to
+do with the same temp file, but it does not seem to be the same problem.
+
+Why would rename() be failing with a permissions problem due to a temp
+directory cleanup?
+
+It would be helpful to see a strace of the failed `rename()` call, because
+I can't immadiately see where a temp file with that name would be exposed
+to `rename()` in a way that could crash like that.
+"""]]

diff --git a/doc/bugs/smudge_errors_on_os-x/comment_2_fe5919747887d3afda8bf2b77a3ab944._comment b/doc/bugs/smudge_errors_on_os-x/comment_2_fe5919747887d3afda8bf2b77a3ab944._comment
new file mode 100644
index 000000000..77cff33ae
--- /dev/null
+++ b/doc/bugs/smudge_errors_on_os-x/comment_2_fe5919747887d3afda8bf2b77a3ab944._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2019-05-23T16:09:38Z"
+ content="""
+Could it be that git is not finding git-annex in PATH to run it, or that
+the binary in PATH is somehow broken?
+
+I'm leaning toward the problem being something like that, because if git
+was able to run git-annex and git-annex somehow failed, it would display an
+error message. It seems as if git-annex is not being run.
+"""]]

diff --git a/doc/bugs/smudge_errors_on_os-x/comment_1_71a3a522b80617d7afc0f4af7f547c67._comment b/doc/bugs/smudge_errors_on_os-x/comment_1_71a3a522b80617d7afc0f4af7f547c67._comment
new file mode 100644
index 000000000..9d6cf19af
--- /dev/null
+++ b/doc/bugs/smudge_errors_on_os-x/comment_1_71a3a522b80617d7afc0f4af7f547c67._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="andrew"
+ avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
+ subject="comment 1"
+ date="2019-05-21T23:33:57Z"
+ content="""
+I am not able to reproduce this bug on MacOS 10.12.6, using git-annex 7.20190508-g06bc46eaa with the git-annex bundled git version 2.18.0. Also, I don't have \"ci\" options on my git so I did `git commit -m 'init' --allow-empty`.
+
+What is your output of `git-annex version`? What version of `git` and `MacOS` are you using?
+
+—Andrew
+"""]]

diff --git a/doc/devblog/day_589__wrapping_up_import_export_preferred_content.mdwn b/doc/devblog/day_589__wrapping_up_import_export_preferred_content.mdwn
new file mode 100644
index 000000000..e9df5c70c
--- /dev/null
+++ b/doc/devblog/day_589__wrapping_up_import_export_preferred_content.mdwn
@@ -0,0 +1,7 @@
+Kind of surprised it all came together so well today, especially because I
+noticed another big problem with the design, but I was able to work around
+that and import/export with preferred content works great.
+
+I did end up limiting import to supported a subset of preferred
+content expressions. Downloading content that it doesn't yet know if it
+wants to import seemed too surprising and potentially very expensive.

diff --git a/doc/tips/android_sync_with_adb.mdwn b/doc/tips/android_sync_with_adb.mdwn
index cf2070f5b..c6b3f0f0f 100644
--- a/doc/tips/android_sync_with_adb.mdwn
+++ b/doc/tips/android_sync_with_adb.mdwn
@@ -19,14 +19,13 @@ already.
 
 Then, in that repository, set up an adb special remote:
 
-	initremote android type=adb androiddirectory=/sdcard/DCIM encryption=none exporttree=yes importtree=yes
+	initremote android type=adb androiddirectory=/sdcard encryption=none exporttree=yes importtree=yes
 
-The above example imports files from the /sdcard/DCIM directory of the
-Android device, so it will only import photos and videos, not other files.
-If you wanted to import everything, you could instead use 
-"androiddirectory=/sdcard".
+The above example syncs with the /sdcard directory of the
+Android device. That can be a lot of files, so you may want a more
+limited directory. See the sample workflows below for some more examples.
 
-Next, configure how file trees imported from it will be merged into your
+Next, configure how trees of files imported from it will be merged into your
 git repository.
 
         git config remote.android.annex-tracking-branch master:android
@@ -36,6 +35,14 @@ of the master branch, and makes all its files appear to be inside a
 subdirectory named `android`. If you want its files to not be in a
 subdirectory, set it to "master" instead.
 
+Finally, you may want to configure a preferred content expression for the
+remote. That will limit both what is exported to it, and what is imported
+from it. If you want to fully sync all files, you don't need to do this.
+
+For example, to limit the files that get imported and exported to sound files:
+
+	git annex wanted android 'include=*.mp3 or include=*.ogg'
+
 ## syncing with it
 
 	git annex sync --content android
@@ -57,21 +64,32 @@ these to only import from or export to the android device:
 
 ### photos
 
-The examples above showed how to import photos from your Android device
-into an android subdirectory. If you don't want to keep old photos on your
-Android device, you can simply `git mv` the files out of the android
-directory, and the next sync with the phone will delete them from the
-Android device:
+Set up the remote to use the /sdcard/DCIM directory where the phone's
+camera stores them.
+
+	initremote android type=adb androiddirectory=/sdcard/DCIM encryption=none exporttree=yes importtree=yes
+       
+The annex-tracking-branch can be the same as before, to limit
+the files that are synced to those in an android directory:
+
+	git config remote.android.annex-tracking-branch master:android
+
+If you don't want to keep old photos on your Android device, you can simply
+`git mv` the files from the android directory to another directory, and
+the next sync with the phone will delete them from the Android device:
 
 	git mv android/* .
 	git annex sync --content
 
 ### music and podcasts
 
-Set up the remote to use the /sdcard/Music directory:
+You could set up the remote to use the /sdcard/Music directory.
+But, I sometimes download music to other locations, and perhaps you do too.
+Let's instead limit the remote to mp3 and ogg files:
+
+	initremote android type=adb androiddirectory=/sdcard encryption=none exporttree=yes importtree=yes
+	git annex wanted android 'include=*.mp3 or include=*.ogg'
 
-	initremote android type=adb androiddirectory=/sdcard/Music encryption=none exporttree=yes importtree=yes
-       
 The annex-tracking-branch can be the same as before, to limit
 the files that are synced to those in an android directory:
 	
@@ -82,14 +100,17 @@ And then do an initial sync:
 	git annex sync --content android
 
 Now, you can copy music and podcasts you want to listen 
-to over to the Android device, by first copying them to the Android
+to over to the Android device, by first copying them to the android
 directory of your git-annex repo:
 
 	cp -a podcasts/LibreLounge/Episode_14__Secure_Scuttlebutt_with_Joey_Hess.ogg android/
 	git annex add android
 	git annex sync --content android
 
-Once you're done with listening to something on the Android device,
-you can simply delete on the device, and the next time git-annex syncs, it
+That will also import any new sound files from the Android device into 
+your git-annex repo.
+
+Once you're done with listening to something on the Android device, you can
+simply delete it from the device, and the next time git-annex syncs, it
 will get removed from the android directory. Or, you can delete it from the
 android directory and the next sync will delete it from the Android device.

diff --git a/Annex/Import.hs b/Annex/Import.hs
index d268ad6dd..585deefa2 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -13,7 +13,9 @@ module Annex.Import (
 	ImportCommitConfig(..),
 	buildImportCommit,
 	buildImportTrees,
-	downloadImport
+	downloadImport,
+	filterImportableContents,
+	makeImportMatcher,
 ) where
 
 import Annex.Common
@@ -41,6 +43,10 @@ import Messages.Progress
 import Utility.DataUnits
 import Logs.Export
 import Logs.Location
+import Logs.PreferredContent
+import Types.FileMatcher
+import Annex.FileMatcher
+import Utility.Matcher (isEmpty)
 import qualified Database.Export as Export
 import qualified Database.ContentIdentifier as CIDDb
 import qualified Logs.ContentIdentifier as CIDLog
@@ -192,7 +198,7 @@ buildImportCommit' remote importcommitconfig mtrackingcommit imported@(History t
 					| otherwise -> do
 						let oldimportedtrees = mapHistory historyCommitTree oldimported
 						mknewcommits oldhc oldimportedtrees imported
-			ti' <- addBackNonPreferredContent remote ti
+			ti' <- addBackExportExcluded remote ti
 			Just <$> makeRemoteTrackingBranchMergeCommit'
 				trackingcommit importedcommit ti'
 	  where
@@ -399,11 +405,11 @@ importKey (ContentIdentifier cid) size = stubKey
  -- special remote).
  --
  -- That presents a problem: Merging the imported tree would result
- -- in deletion of the non-preferred content. To avoid that happening,
- -- this adds the non-preferred content back to the imported tree.
+ -- in deletion of the files that were excluded from export.
+ -- To avoid that happening, this adds them back to the imported tree.
  --}
-addBackNonPreferredContent :: Remote -> Sha -> Annex Sha
-addBackNonPreferredContent remote importtree =
+addBackExportExcluded :: Remote -> Sha -> Annex Sha
+addBackExportExcluded remote importtree =
 	getExportExcluded (Remote.uuid remote) >>= \case
 		[] -> return importtree
 		excludedlist -> inRepo $
@@ -417,3 +423,60 @@ addBackNonPreferredContent remote importtree =
 				(\imported _excluded -> imported)
 				[]
 				importtree
+
+{- Match the preferred content of the remote at import time.
+ -
+ - Only keyless tokens are supported, because the keys are not known
+ - until an imported file is downloaded, which is too late to bother
+ - excluding it from an import.
+ -}
+makeImportMatcher :: Remote -> Annex (Either String (FileMatcher Annex))
+makeImportMatcher r = load preferredContentKeylessTokens >>= \case
+	Nothing -> return $ Right matchAll
+	Just (Right v) -> return $ Right v
+	Just (Left err) -> load preferredContentTokens >>= \case
+		Just (Left err') -> return $ Left err'
+		_ -> return $ Left $
+			"The preferred content expression contains terms that cannot be checked when importing: " ++ err
+  where
+	load t = M.lookup (Remote.uuid r) . fst <$> preferredRequiredMapsLoad' t
+
+wantImport :: FileMatcher Annex -> ImportLocation -> ByteSize -> Annex Bool
+wantImport matcher loc sz = checkMatcher' matcher mi mempty
+  where
+	mi = MatchingInfo $ ProvidedInfo
+		{ providedFilePath = Right $ fromImportLocation loc
+		, providedKey = unavail "key"
+		, providedFileSize = Right sz
+		, providedMimeType = unavail "mime"
+		, providedMimeEncoding = unavail "mime"
+		}
+	-- This should never run, as long as the FileMatcher was generated
+	-- using the preferredContentKeylessTokens.
+	unavail v = Left $ error $ "Internal error: unavailable " ++ v
+
+{- If a file is not preferred content, but it was previously exported or
+ - imported to the remote, not importing it would result in a remote
+ - tracking branch that, when merged, would delete the file.
+ -
+ - To avoid that problem, such files are included in the import.
+ - The next export will remove them from the remote.
+ -}
+shouldImport :: Export.ExportHandle -> FileMatcher Annex -> ImportLocation -> ByteSize -> Annex Bool
+shouldImport dbhandle matcher loc sz = 
+	wantImport matcher loc sz
+		<||>
+	liftIO (not . null <$> Export.getExportTreeKey dbhandle loc)
+
+filterImportableContents :: Remote -> FileMatcher Annex -> ImportableContents (ContentIdentifier, ByteSize) -> Annex (ImportableContents (ContentIdentifier, ByteSize))
+filterImportableContents r matcher importable
+	| isEmpty matcher = return importable
+	| otherwise = do
+		dbhandle <- Export.openDb (Remote.uuid r)
+		go dbhandle importable
+  where
+	go dbhandle ic = ImportableContents
+		<$> filterM (match dbhandle) (importableContents ic)
+		<*> mapM (go dbhandle) (importableHistory ic)
+	
+	match dbhandle (loc, (_cid, sz)) = shouldImport dbhandle matcher loc sz
diff --git a/CHANGELOG b/CHANGELOG
index 0e78fb27b..943c62cf1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -7,6 +7,10 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
     annex.jobs=cpus, or using option --jobs=cpus or -Jcpus.
   * Honor preferred content of a special remote when exporting trees to it;
     unwanted files are filtered out of the tree that is exported.
+  * Importing from a special remote honors its preferred content too;
+    unwanted files are not imported. But, some preferred content
+    expressions can't be checked before files are imported, and trying to
+    import with such an expression will fail.
   * Improve shape of commit tree when importing from unversioned special
     remotes.
 
diff --git a/Command/Import.hs b/Command/Import.hs
index b108535f7..f890f982b 100644
--- a/Command/Import.hs
+++ b/Command/Import.hs
@@ -293,9 +293,13 @@ listContents remote tvar = do
 	showStart' "list" (Just (Remote.name remote))
 	next $ Remote.listImportableContents (Remote.importActions remote) >>= \case
 		Nothing -> giveup $ "Unable to list contents of " ++ Remote.name remote
-		Just importable -> next $ do
-			liftIO $ atomically $ writeTVar tvar (Just importable)
-			return True
+		Just importable -> do
+			importable' <- makeImportMatcher remote >>= \case
+				Right matcher -> filterImportableContents remote matcher importable
+				Left err -> giveup $ "Cannot import from " ++ Remote.name remote ++ " because of a problem with its configuration: " ++ err
+			next $ do
+				liftIO $ atomically $ writeTVar tvar (Just importable')
+				return True
 
 commitRemote :: Remote -> Branch -> RemoteTrackingBranch -> Maybe Sha -> ImportTreeConfig -> ImportCommitConfig -> ImportableContents Key -> CommandStart
 commitRemote remote branch tb trackingcommit importtreeconfig importcommitconfig importable = do
diff --git a/Logs/PreferredContent.hs b/Logs/PreferredContent.hs
index 16ffef129..57bbeb6bd 100644
--- a/Logs/PreferredContent.hs
+++ b/Logs/PreferredContent.hs
@@ -20,6 +20,7 @@ module Logs.PreferredContent (
 	setStandardGroup,
 	defaultStandardGroup,
 	preferredRequiredMapsLoad,
+	preferredRequiredMapsLoad',
 	prop_standardGroups_parse,
 ) where
 
@@ -71,24 +72,37 @@ requiredContentMap = maybe (snd <$> preferredRequiredMapsLoad preferredContentTo
 
 preferredRequiredMapsLoad :: (PreferredContentData -> [ParseToken (MatchFiles Annex)]) -> Annex (FileMatcherMap Annex, FileMatcherMap Annex)
 preferredRequiredMapsLoad mktokens = do
+	(pc, rc) <- preferredRequiredMapsLoad' mktokens
+	let pc' = handleunknown pc
+	let rc' = handleunknown rc
+	Annex.changeState $ \s -> s
+		{ Annex.preferredcontentmap = Just pc'
+		, Annex.requiredcontentmap = Just rc'
+		}
+	return (pc', rc')
+  where
+	handleunknown = M.mapWithKey $ \u -> 
+		fromRight (unknownMatcher u)
+
+preferredRequiredMapsLoad' :: (PreferredContentData -> [ParseToken (MatchFiles Annex)]) -> Annex (M.Map UUID (Either String (FileMatcher Annex)), M.Map UUID (Either String (FileMatcher Annex)))
+preferredRequiredMapsLoad' mktokens = do
 	groupmap <- groupMap
 	configmap <- readRemoteLog
 	let genmap l gm = 
-		let mk u = fromRight (unknownMatcher u) .
-			makeMatcher groupmap configmap gm u mktokens
+		let mk u = makeMatcher groupmap configmap gm u mktokens
 		in simpleMap
 			. parseLogOldWithUUID (\u -> mk u . decodeBS <$> A.takeByteString)
 			<$> Annex.Branch.get l
 	pc <- genmap preferredContentLog =<< groupPreferredContentMapRaw
 	rc <- genmap requiredContentLog M.empty
-	-- Required content is implicitly also preferred content, so
-	-- combine.
-	let m = M.unionWith combineMatchers pc rc
-	Annex.changeState $ \s -> s
-		{ Annex.preferredcontentmap = Just m
-		, Annex.requiredcontentmap = Just rc
-		}
-	return (m, rc)
+	-- Required content is implicitly also preferred content, so combine.
+	let pc' = M.unionWith combiner pc rc
+	return (pc', rc)

(Diff truncated)

diff --git a/doc/todo/export_preferred_content.mdwn b/doc/todo/export_preferred_content.mdwn
index f7fb911fd..cba1e4361 100644
--- a/doc/todo/export_preferred_content.mdwn
+++ b/doc/todo/export_preferred_content.mdwn
@@ -2,6 +2,8 @@
 which is generally what the user wants.
 But, in some situations, the user may want to export a subset of files,
 in a way that can be well expressed by a preferred content expression.
+ 
+> started work on this in the `preferred` branch. --[[Joey]]
 
 For example, they may want to export .mp3 files but not the .wav
 files used to produce those.
@@ -53,7 +55,7 @@ TODO
 > if directory Music is excluded from an android remote, importing from
 > it should exclude that directory.
 
-----
+## import after limited export
 
 > Problem: If a tree is exported with eg, no .wav files, and then an import
 > is made from the remote, and necessarily lacks .wav files, the remote
@@ -86,7 +88,7 @@ TODO
 > 
 > > done
 
----
+## matching preferred content expressions on import
 
 > Matching a preferred content expression at import time before the content
 > is downloaded means that the imported key may not yet be known. (Only
@@ -149,16 +151,21 @@ TODO
 > OR download the content
 > from the remote, generate a key from it, and re-match the preferred
 > content expression. That avoids any surprises and supports all
-> expressions at the expense of an unnessary download. As long as the ContentIdentifier to
-> Key mapping gets updated, it will only download a given file unncessarily
-> one time.
+> expressions at the expense of an unnessary download. As long as the
+> ContentIdentifier to Key mapping gets updated, it will only download
+> a given file unncessarily one time.
 > 
 > Which approach is better? Note that almost all of the standard groups
 > do depend on the key. But it seems very likely that most actual
 > uses of this feature would involve the name or size of a file that's
 > being imported, and nothing else.
 > 
-> > started work on this in the `preferred` branch. --[[Joey]]
+> Imagine an import of all non-mp3s from the phone, and the phone has
+> a 20 gb mp3 collection. Downloading them all just to check the preferred
+> content expression would be an enormous amount of unnecessary work.
+> If the user started with `exclude=*.mp3`, they'd expect it to be fast,
+> and if they changed to `exclude=*.mp3 or metadata=tag=podcast`
+> and it did all that extra work, that would be surprising.
 
 ## different preferred content for export and import?
 
@@ -172,5 +179,61 @@ annex sync --content would not work well.
 
 Better example: Make the phone want all content that is in the laptop
 group, so all files on my laptop export to the phone but not others that I
-have archived. But want to import all files from the phone, which is not in
-the laptop group, so need a separate expression for import.
+have archived. But want to import all files from the phone except for mp3s.
+
+One way would be new preferred content terminals that match when importing
+and exporting:
+
+	(exporting and copies=laptop:1) or (importing and exclude=*.mp3)
+
+This needs preferred content expressions for import to be able to
+support things like copies= that need to know the key, as discussed above.
+
+If the import preferred content expression is limited to not include those
+terms, the above example can't be used at import time. Unless it can be
+simplified before being checked for those terms:
+
+	(false and copies=laptop:1) or (true and exclude=*.mp3)
+
+	false or (true and exclude=*.mp3)
+
+Or there could be separate expressions for import and export.
+And this kind of makes sense; the normal preferred content expression
+controls what is stored on a remote, so affects export, and the import
+preferred content expression is only used to fine-tune which files get
+imported from it.
+
+Problem: Suppose a tree is exported to a special remote, and the tree includes some
+mp3 files. And then an import is done, excluding mp3 files. That will
+create a remote tracking branch that lacks the mp3 files; merging it will
+delete them from master. That could be surprising! This is the inverse of
+the "import after limited export" problem, but it seems it 
+can't be solved in a similar way.
+
+That problem might be a fatal blow to this idea of separate expressions
+for import and export. But it's worse than that! -- 
+
+Problem: Suppose a tree is exported to a special remote and the tree
+includes mp3 files. Then the remote's preferred content is set to exclude
+mp3 files. Then on import from the remote, a tree will be constructed that
+lacks the mp3 files that were exported before; merging it will delete them
+from master.
+
+That could be avoided by making the import notice that the preferred
+content expression has changed, and so throw away the old remote tracking
+branch history and import a commit with no parent, avoiding the deletion.
+But, if some other file got deleted from the special remote after the
+export, the import would then not delete it.
+
+Alternatively, when a preferred content expression doesn't match a file at
+import, could check if the same file was present in the last export. (With
+same or different content.) If so, assume the preferred content has changed
+and that the user does not want to delete this file, so keep it in the
+import anyway (using the content that was last exported to it). 
+(The state does not currently differentiate between the last export
+and the last import, so the file would keep being included in
+imports until an export was made that removed it.)
+
+OR, don't match preferred content expressions on import at all; download
+everything, and let the user delete unwanted imports locally. Does avoid
+all these complications.

diff --git a/Annex/Import.hs b/Annex/Import.hs
index 006bd0299..2e152a0f7 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -163,40 +163,44 @@ buildImportCommit remote importtreeconfig importcommitconfig importable =
 buildImportCommit' :: ImportCommitConfig -> Maybe Sha -> History Sha -> Annex (Maybe Sha)
 buildImportCommit' importcommitconfig mtrackingcommit imported@(History ti _) =
 	case mtrackingcommit of
-		Nothing -> Just <$> mkcommits imported
+		Nothing -> Just <$> mkcommitsunconnected imported
 		Just trackingcommit -> do
 			-- Get history of tracking branch to at most
 			-- one more level deep than what was imported,
 			-- so we'll have enough history to compare,
 			-- but not spend too much time getting it.
-			let maxdepth = succ (historyDepth imported)
+			let maxdepth = succ importeddepth
 			inRepo (getHistoryToDepth maxdepth trackingcommit)
 				>>= go trackingcommit
   where
-	go _ Nothing = Just <$> mkcommits imported
+	go _ Nothing = Just <$> mkcommitsunconnected imported
 	go trackingcommit (Just h)
 		-- If the tracking branch head is a merge commit
 		-- with a tree that matches the head of the history,
 		-- and one side of the merge matches the history,
 		-- nothing new needs to be committed.
-		| t == ti && any (sametodepth imported) (S.toList s) = return Nothing
+		| t == ti && any sametodepth (S.toList s) = return Nothing
 		-- If the tracking branch matches the history,
 		-- nothing new needs to be committed.
 		-- (This is unlikely to happen.)
-		| sametodepth imported h' = return Nothing
+		| sametodepth h' = return Nothing
 		| otherwise = do
 			importedcommit <- case getRemoteTrackingBranchImportHistory h of
-				Nothing ->
-					mkcommits imported
-				Just oldimported@(History oldhc _) -> do
-					let oldimportedtrees = mapHistory historyCommitTree oldimported
-					mknewcommits oldhc oldimportedtrees imported
+				Nothing -> mkcommitsunconnected imported
+				Just oldimported@(History oldhc _)
+					| importeddepth == 1 ->
+						mkcommitconnected imported oldimported
+					| otherwise -> do
+						let oldimportedtrees = mapHistory historyCommitTree oldimported
+						mknewcommits oldhc oldimportedtrees imported
 			Just <$> makeRemoteTrackingBranchMergeCommit'
 				trackingcommit importedcommit ti
 	  where
 		h'@(History t s) = mapHistory historyCommitTree h
 
-	sametodepth a b = a == truncateHistoryToDepth (historyDepth a) b
+	importeddepth = historyDepth imported
+
+	sametodepth b = imported == truncateHistoryToDepth importeddepth b
 
 	mkcommit parents tree = inRepo $ Git.Branch.commitTree
 		(importCommitMode importcommitconfig)
@@ -204,8 +208,16 @@ buildImportCommit' importcommitconfig mtrackingcommit imported@(History ti _) =
 		parents
 		tree
 
-	mkcommits (History importedtree hs) = do
-		parents <- mapM mkcommits (S.toList hs)
+	-- Start a new history of import commits, not connected to any
+	-- prior import commits.
+	mkcommitsunconnected (History importedtree hs) = do
+		parents <- mapM mkcommitsunconnected (S.toList hs)
+		mkcommit parents importedtree
+
+	-- Commit the new history connected with the old history.
+	-- Used when the import is not versioned, so the history depth is 1.
+	mkcommitconnected (History importedtree _) (History oldhc _) = do
+		let parents = [historyCommit oldhc]
 		mkcommit parents importedtree
 
 	-- Reuse the commits from the old imported History when possible.
diff --git a/CHANGELOG b/CHANGELOG
index dbcab637e..bb47c6909 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,8 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
   * Makefile: Added install-completions to install target.
   * Added the ability to run one job per CPU (core), by setting
     annex.jobs=cpus, or using option --jobs=cpus or -Jcpus.
+  * Improve shape of commit tree when importing from unversioned special
+    remotes.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/doc/todo/import_from_special_remote_large_git_log.mdwn b/doc/todo/import_from_special_remote_large_git_log.mdwn
index f1b43221f..594936715 100644
--- a/doc/todo/import_from_special_remote_large_git_log.mdwn
+++ b/doc/todo/import_from_special_remote_large_git_log.mdwn
@@ -18,3 +18,5 @@ and adds more commits on top of those, so this is mostly not a problem
 there. If the old and new imported histories are disjoint, a commit or
 commits will be made with no parent, but that seems acceptable; it's an
 edge case and it's replicating the information from the remote.
+
+> [[done]] --[[Joey]]

diff --git a/doc/bugs/surprising_import_tree_merge_result.mdwn b/doc/bugs/surprising_import_tree_merge_result.mdwn
index 7831eb2a5..3d2f37f3c 100644
--- a/doc/bugs/surprising_import_tree_merge_result.mdwn
+++ b/doc/bugs/surprising_import_tree_merge_result.mdwn
@@ -13,9 +13,3 @@ raw/photos/something
 This is quite surprising. It seems somehow when it modified the old
 tree, it rooted the new at the location the old had been moved to in
 master.
-
-----
-
-Also, commit cd55a5be377020b7261c3a737b5e32a01e22f4b8 had no parent, so
-git diff --stat shows it as adding every file that was in the tree before
-as well as the imported files, which seems wrong at least visually.
diff --git a/doc/todo/import_from_special_remote_large_git_log.mdwn b/doc/todo/import_from_special_remote_large_git_log.mdwn
new file mode 100644
index 000000000..f1b43221f
--- /dev/null
+++ b/doc/todo/import_from_special_remote_large_git_log.mdwn
@@ -0,0 +1,20 @@
+After import from a special remote and merge, git log --stat shows
+a large diff, because every file that got imported from the special 
+remote is put on a commit with no parent, so the diff shows it as if those
+files are newly added.
+
+(Of course when using S3 with versioning, the commit tree does have
+parents, but still a root commit with no parent.)
+
+This does not seem avoidable for the initial import, even if the remote was
+populated by an earlier export, that starts a new, disconnected history for
+reasons explained in the import tree design. 
+
+Subsequent imports though could fix it, by setting the parent of the new
+import to the previous import.
+
+For versioned imports, it reuses commits from the old imported history,
+and adds more commits on top of those, so this is mostly not a problem
+there. If the old and new imported histories are disjoint, a commit or
+commits will be made with no parent, but that seems acceptable; it's an
+edge case and it's replicating the information from the remote.

diff --git a/doc/devblog/day_588__export_preferred_content.mdwn b/doc/devblog/day_588__export_preferred_content.mdwn
new file mode 100644
index 000000000..fd2cf1d6c
--- /dev/null
+++ b/doc/devblog/day_588__export_preferred_content.mdwn
@@ -0,0 +1,8 @@
+Made `git annex export --to remote` honor the preferred content of the
+remote. In a nice bit of code reuse, `adjustTree` was just what I needed to
+filter unwanted content out of the exported tree.
+
+Then a hard problem: When a tree is exported with some non-preferred
+content filtered out, importing from the remote generates a tree that is
+lacking those files, but merging that tree would delete the files
+from the working tree. Solving that took the rest of the day.

diff --git a/Annex/AdjustedBranch.hs b/Annex/AdjustedBranch.hs
index a06bc9cd6..9cc449b0d 100644
--- a/Annex/AdjustedBranch.hs
+++ b/Annex/AdjustedBranch.hs
@@ -265,7 +265,12 @@ adjustCommit adj basis = do
 adjustTree :: Adjustment -> BasisBranch -> Annex Sha
 adjustTree adj (BasisBranch basis) = do
 	let toadj = adjustTreeItem adj
-	treesha <- Git.Tree.adjustTree toadj [] [] basis =<< Annex.gitRepo
+	treesha <- Git.Tree.adjustTree
+		toadj 
+		[] 
+		(\_old new -> new)
+		[]
+		basis =<< Annex.gitRepo
 	return treesha
 
 type CommitsPrevented = Git.LockFile.LockHandle
@@ -544,6 +549,7 @@ reverseAdjustedTree basis adj csha = do
 	treesha <- Git.Tree.adjustTree
 		(propchanges changes)
 		adds'
+		(\_old new -> new)
 		(map Git.DiffTree.file removes)
 		basis
 		=<< Annex.gitRepo
diff --git a/Annex/Import.hs b/Annex/Import.hs
index 006bd0299..e8a7baddb 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -103,7 +103,7 @@ buildImportCommit remote importtreeconfig importcommitconfig importable =
 	go trackingcommit = do
 		imported@(History finaltree _) <-
 			buildImportTrees basetree subdir importable
-		buildImportCommit' importcommitconfig trackingcommit imported >>= \case
+		buildImportCommit' remote importcommitconfig trackingcommit imported >>= \case
 			Just finalcommit -> do
 				updatestate finaltree
 				return (Just finalcommit)
@@ -160,8 +160,8 @@ buildImportCommit remote importtreeconfig importcommitconfig importable =
 			Export.runExportDiffUpdater updater db oldtree finaltree
 		Export.closeDb db
 
-buildImportCommit' :: ImportCommitConfig -> Maybe Sha -> History Sha -> Annex (Maybe Sha)
-buildImportCommit' importcommitconfig mtrackingcommit imported@(History ti _) =
+buildImportCommit' :: Remote -> ImportCommitConfig -> Maybe Sha -> History Sha -> Annex (Maybe Sha)
+buildImportCommit' remote importcommitconfig mtrackingcommit imported@(History ti _) =
 	case mtrackingcommit of
 		Nothing -> Just <$> mkcommits imported
 		Just trackingcommit -> do
@@ -176,7 +176,6 @@ buildImportCommit' importcommitconfig mtrackingcommit imported@(History ti _) =
 	go _ Nothing = Just <$> mkcommits imported
 	go trackingcommit (Just h)
 		-- If the tracking branch head is a merge commit
-		-- with a tree that matches the head of the history,
 		-- and one side of the merge matches the history,
 		-- nothing new needs to be committed.
 		| t == ti && any (sametodepth imported) (S.toList s) = return Nothing
@@ -191,8 +190,9 @@ buildImportCommit' importcommitconfig mtrackingcommit imported@(History ti _) =
 				Just oldimported@(History oldhc _) -> do
 					let oldimportedtrees = mapHistory historyCommitTree oldimported
 					mknewcommits oldhc oldimportedtrees imported
+			ti' <- addBackNonPreferredContent remote ti
 			Just <$> makeRemoteTrackingBranchMergeCommit'
-				trackingcommit importedcommit ti
+				trackingcommit importedcommit ti'
 	  where
 		h'@(History t s) = mapHistory historyCommitTree h
 
@@ -380,3 +380,30 @@ importKey (ContentIdentifier cid) size = stubKey
 	, keyVariety = OtherKey "CID"
 	, keySize = Just size
 	}
+
+{-- Export omits non-preferred content from the tree stored on the
+ -- remote. So the import will normally have that content
+ -- omitted (unless something else added files with the same names to the
+ -- special remote).
+ --
+ -- That presents a problem: Merging the imported tree would result
+ -- in deletion of the non-preferred content. To avoid that happening,
+ -- this adds the non-preferred content back to the imported tree.
+ --}
+addBackNonPreferredContent :: Remote -> Sha -> Annex Sha
+addBackNonPreferredContent remote importtree =
+	getExportExcluded (Remote.uuid remote) >>= \case
+		[] -> return importtree
+		-- TODO: does this overwrite newly imported files
+		-- with excluded files? CHECK
+		excludedlist -> inRepo $
+			adjustTree
+				-- don't remove any
+				(pure . Just)
+				excludedlist
+				-- if something was imported with the same
+				-- name as a file that was previously
+				-- excluded from import, use what was imported
+				(\imported _excluded -> imported)
+				[]
+				importtree
diff --git a/Annex/Locations.hs b/Annex/Locations.hs
index 295809c7e..0dfd1387c 100644
--- a/Annex/Locations.hs
+++ b/Annex/Locations.hs
@@ -50,6 +50,7 @@ module Annex.Locations (
 	gitAnnexExportDbDir,
 	gitAnnexExportLock,
 	gitAnnexExportUpdateLock,
+	gitAnnexExportExcludeLog,
 	gitAnnexContentIdentifierDbDir,
 	gitAnnexContentIdentifierLock,
 	gitAnnexScheduleState,
@@ -361,6 +362,11 @@ gitAnnexExportLock u r = gitAnnexExportDbDir u r ++ ".lck"
 gitAnnexExportUpdateLock :: UUID -> Git.Repo -> FilePath
 gitAnnexExportUpdateLock u r = gitAnnexExportDbDir u r ++ ".upl"
 
+{- Log file used to keep track of files that were in the tree exported to a
+ - remote, but were excluded by its preferred content settings. -}
+gitAnnexExportExcludeLog :: UUID -> Git.Repo -> FilePath
+gitAnnexExportExcludeLog u r = gitAnnexDir r </> "export.ex" </> fromUUID u
+
 {- Directory containing database used to record remote content ids.
  -
  - (This used to be "cid", but a problem with the database caused it to
diff --git a/Command/Export.hs b/Command/Export.hs
index 64faa9a5d..192c3157d 100644
--- a/Command/Export.hs
+++ b/Command/Export.hs
@@ -442,19 +442,29 @@ removeEmptyDirectories r db loc ks
 -- expression.
 newtype PreferredFiltered t = PreferredFiltered t
 
+-- | Filters the tree to files that are preferred content of the remote.
+--
+-- A log is written with files that were filtered out, so they can be added
+-- back in when importing from the remote.
 filterPreferredContent :: Remote -> Git.Ref -> Annex (PreferredFiltered Git.Ref)
-filterPreferredContent r tree = do
+filterPreferredContent r tree = logExportExcluded (uuid r) $ \logwriter -> do
 	m <- preferredContentMap
 	case M.lookup (uuid r) m of
-		Just matcher | not (isEmpty matcher) -> 
-			PreferredFiltered <$> go matcher
+		Just matcher | not (isEmpty matcher) -> do
+			PreferredFiltered <$> go matcher logwriter
 		_ -> return (PreferredFiltered tree)
   where
-	go matcher = do
+	go matcher logwriter = do
 		g <- Annex.gitRepo
-		Git.Tree.adjustTree (checkmatcher matcher) [] [] tree g
+		Git.Tree.adjustTree
+			(checkmatcher matcher logwriter)
+			[]
+			(\_old new -> new)
+			[]
+			tree
+			g
 	
-	checkmatcher matcher ti@(Git.Tree.TreeItem topf _ sha) =
+	checkmatcher matcher logwriter ti@(Git.Tree.TreeItem topf _ sha) =
 		catKey sha >>= \case
 			Just k -> do
 				-- Match filename relative to the
@@ -464,7 +474,9 @@ filterPreferredContent r tree = do
 				let mi = MatchingKey k af
 				ifM (checkMatcher' matcher mi mempty)
 					( return (Just ti)
-					, return Nothing
+					, do
+						() <- liftIO $ logwriter ti
+						return Nothing
 					)
 			-- Always export non-annexed files.
 			Nothing -> return (Just ti)
diff --git a/Git/LsTree.hs b/Git/LsTree.hs
index 0fdb35a36..8ca805402 100644
--- a/Git/LsTree.hs
+++ b/Git/LsTree.hs
@@ -15,6 +15,7 @@ module Git.LsTree (
 	lsTreeParams,
 	lsTreeFiles,
 	parseLsTree,
+	formatLsTree,
 ) where
 
 import Common
@@ -90,3 +91,12 @@ parseLsTree l = TreeItem
 	!f = drop 1 past_s
 	!smode = fst $ Prelude.head $ readOct m
 	!sfile = asTopFilePath $ Git.Filename.decode f
+
+{- Inverse of parseLsTree -}
+formatLsTree :: TreeItem -> String
+formatLsTree ti = unwords
+	[ showOct (mode ti) ""
+	, typeobj ti
+	, fromRef (sha ti)
+	, getTopFilePath (file ti)
+	]

(Diff truncated)

diff --git a/CHANGELOG b/CHANGELOG
index dbcab637e..e9342d813 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,8 @@ git-annex (7.20190508) UNRELEASED; urgency=medium
   * Makefile: Added install-completions to install target.
   * Added the ability to run one job per CPU (core), by setting
     annex.jobs=cpus, or using option --jobs=cpus or -Jcpus.
+  * Honor preferred content of a special remote when exporting trees to it;
+    unwanted files are filtered out of the tree that is exported.
 
  -- Joey Hess <id@joeyh.name>  Mon, 06 May 2019 13:52:02 -0400
 
diff --git a/doc/git-annex-export.mdwn b/doc/git-annex-export.mdwn
index 36d28d93e..fa92efb7e 100644
--- a/doc/git-annex-export.mdwn
+++ b/doc/git-annex-export.mdwn
@@ -23,6 +23,9 @@ The treeish to export can be the name of a git branch, or a tag, or any
 other treeish accepted by git, including eg master:subdir to only export a
 subdirectory from a branch.
 
+When the remote has a preferred content setting, the treeish is filtered
+through it, excluding files it does not want from being exported to it.
+
 Repeated exports are done efficiently, by diffing the old and new tree,
 and transferring only the changed files, and renaming files as necessary.
 
@@ -65,6 +68,8 @@ let an export overwrite the modified file; then `git annex import`
 will create a sequence of commits that includes the modified file,
 so the overwritten modification is not lost.)
 
+# PREFERRED 
+
 # OPTIONS
 
 * `--to=remote`
@@ -132,6 +137,8 @@ export`, it will detect the export conflict, and resolve it.
 
 [[git-annex-sync]](1)
 
+[[git-annex-preferred-content]](1)
+
 # HISTORY
 
 The `export` command was introduced in git-annex version 6.20170925.
diff --git a/doc/git-annex-preferred-content.mdwn b/doc/git-annex-preferred-content.mdwn
index e81bd0239..d9209a7a9 100644
--- a/doc/git-annex-preferred-content.mdwn
+++ b/doc/git-annex-preferred-content.mdwn
@@ -37,13 +37,17 @@ elsewhere to allow removing it).
   Match files to include, or exclude.
 
   While --include=glob and --exclude=glob match files relative to the current
-  directory, preferred content expressions always match files relative to the
-  top of the git repository. 
+  directory, preferred content expressions match files relative to the
+  top of the git repository.
 
   For example, suppose you put files into `archive` directories
   when you're done with them. Then you could configure your laptop to prefer
   to not retain those files, like this: `exclude=*/archive/*`
 
+  When a subdirectory is being exported to a special remote (see
+  [[git-annex-export]](1)), these match relative to the top of the
+  subdirectory.
+
 * `copies=number`
 
   Matches only files that git-annex believes to have the specified number
diff --git a/doc/todo/export_preferred_content.mdwn b/doc/todo/export_preferred_content.mdwn
index a60023089..cc8f01a33 100644
--- a/doc/todo/export_preferred_content.mdwn
+++ b/doc/todo/export_preferred_content.mdwn
@@ -11,7 +11,9 @@ been listened to.
 
 It seems doable to make `git annex export` honor whatever
 preferred content settings have been configured for the remote.
-(And `git annex sync --content` too.)
+(And `git annex sync --content` too.)  
+
+> done
 
 Problem: A preferred content expression include=subdir/foo or
 exclude=subdir/bar matches relative to the top of the repository.
@@ -24,6 +26,8 @@ It may be better to add a note that preferred content expressions include=
 exclude= etc match relative to the top of the exported tree when exporting
 a subtree.
 
+> done
+
 ----
 
 > `git annex import` of a tree from a special remote would also be
@@ -60,10 +64,23 @@ a subtree.
 > version of the branch for preferred content and location tracking.
 > (And use of `git-annex forget` could break it.)
 > 
-> It seems easier to instead record the tree of excluded files somewhere,
-> Logs.Export already records the whole exported tree in the git-annex
-> branch, so extend it to also record the tree of excluded files.
-> Complication: Export conflicts.
+> Logs.Export already records the tree that the user chose to export
+> into the git-annex branch. Should excluded files be present in that
+> tree or not? If not, the diff from that tree to the remote tracking
+> branch will be the excluded files. Another good reason to do that
+> is that if the preferred content settings change, the next export
+> will pick up on the change, since the exported tree differs from the tree
+> to be exported.
+> 
+> So, plan: Make export of a tree filter that tree through the preferred
+> content of the remote, and use the new tree as the tree that really
+> gets exported, recording it in the git-annex branch. But the remote
+> tracking branch will point to the commit that the user chose to export.
+> > (done)
+> 
+> The import then just needs to diff between the last exported tree and
+> and the remote tracking branch to get the files that were excluded,
+> and add them back into the imported tree.
 
 ---
 

diff --git a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_8_c15caba920b49cebf88ebd36caa053b7._comment b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_8_c15caba920b49cebf88ebd36caa053b7._comment
new file mode 100644
index 000000000..65b5546ed
--- /dev/null
+++ b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_8_c15caba920b49cebf88ebd36caa053b7._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
+ nickname="branchable"
+ avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
+ subject="comment 8"
+ date="2019-05-20T15:23:45Z"
+ content="""
+I understand, and of course it's your prerogative to take that position.  While there are many UNIX shells out there, I'm pretty sure that bash and zsh are the two which combined take the lion share of the market.
+
+If you're worried about the impact of adding `cp .profile .zprofile` to the install, a reasonable alternative would be to document that the install relies on setting up `$PATH` via `.profile` - this confusion only arose in the first place because that mechanism is somewhat hidden rather than taking place directly in the install script.
+"""]]

diff --git a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
index be717d6d2..5cea6440d 100644
--- a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
+++ b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH.mdwn
@@ -27,3 +27,5 @@ git: 'annex' is not a git command. See 'git --help'.
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 Yes, been using it for many years and couldn't live without it.
+
+[[!meta title="termux install adds git-annex only to bash path, not zsh etc"]]

diff --git a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_7_f9b09f1a720ab73f6a089e8c65f7c402._comment b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_7_f9b09f1a720ab73f6a089e8c65f7c402._comment
new file mode 100644
index 000000000..abfb836e4
--- /dev/null
+++ b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_7_f9b09f1a720ab73f6a089e8c65f7c402._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2019-05-20T13:52:37Z"
+ content="""
+There are a lot of shells, and AFAIK no single place an environment
+variable can be put that works with all of them.
+
+I'm inclined to say that working with termux's default shell is all this
+needs to do, and if the user installs some other shell, it's up to them to
+configure it.
+"""]]

diff --git a/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_4_d8cec7fae740e82c5da1da47037e2da6._comment b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_4_d8cec7fae740e82c5da1da47037e2da6._comment
new file mode 100644
index 000000000..2c9ce281a
--- /dev/null
+++ b/doc/bugs/Android_install_doesn__39__t_permanently_add_to___36__PATH/comment_4_d8cec7fae740e82c5da1da47037e2da6._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
+ nickname="branchable"
+ avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
+ subject=".profile doesn't take effect"
+ date="2019-05-18T19:46:09Z"
+ content="""
+Ah, interesting.  Very odd that the install script doesn't do that directly.  I do have the same contents in my `~/.profile`, but Termux happily ignores it, so `$PATH` is unaltered.  Does it really work for you?  What version of Termux do you have?
+"""]]
diff --git a/doc/bugs/Show_current_config_of_special_remote.mdwn b/doc/bugs/Show_current_config_of_special_remote.mdwn
new file mode 100644
index 000000000..aaa36680b
--- /dev/null
+++ b/doc/bugs/Show_current_config_of_special_remote.mdwn
@@ -0,0 +1,9 @@
+### Please describe the problem.
+There appears to be no way of getting the current configuration of special remotes (e.g. url, keyid). Maybe there should be git-annex-showremote (or similar).
+
+### What steps will reproduce the problem?
+Configure the special remote (e.g. webdav), and then forget what settings you specified ;)
+
+### What version of git-annex are you using? On what operating system?
+7.20190129-3 (Debian)
+
diff --git a/doc/bugs/Show_current_config_of_special_remote/comment_1_3886d242006fbc6928685bfeaaf93d70._comment b/doc/bugs/Show_current_config_of_special_remote/comment_1_3886d242006fbc6928685bfeaaf93d70._comment
new file mode 100644
index 000000000..2539e907d
--- /dev/null
+++ b/doc/bugs/Show_current_config_of_special_remote/comment_1_3886d242006fbc6928685bfeaaf93d70._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="derobert"
+ avatar="http://cdn.libravatar.org/avatar/05b48b72766177b3b0a6ff4afdb70790"
+ subject="comment 1"
+ date="2019-05-16T08:12:45Z"
+ content="""
+Did you try `git-annex info «remote»` (where «remote» could be the name or the UUID).
+"""]]
diff --git a/doc/bugs/Show_current_config_of_special_remote/comment_2_bf5f08276435a95f0c3822ef99655f67._comment b/doc/bugs/Show_current_config_of_special_remote/comment_2_bf5f08276435a95f0c3822ef99655f67._comment
new file mode 100644
index 000000000..a8e791fe6
--- /dev/null
+++ b/doc/bugs/Show_current_config_of_special_remote/comment_2_bf5f08276435a95f0c3822ef99655f67._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="aragilar"
+ avatar="http://cdn.libravatar.org/avatar/7390f7c0dd07d43ca827a5446b65c102"
+ subject="comment 2"
+ date="2019-05-16T08:26:28Z"
+ content="""
+Weird, I tried that on the same remote on a different system, and it gave what I wanted... I'll have to work out why that didn't work the first time. Thanks @derobert!
+"""]]
diff --git a/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn
new file mode 100644
index 000000000..52885b300
--- /dev/null
+++ b/doc/bugs/git-annex-init_with_no_args_overwrites_existing_repo_description.mdwn
@@ -0,0 +1,20 @@
+### Please describe the problem.
+The man page for [[`git-annex-init`|git-annex-init]] suggests that it is idempotent.  But, when run with no args, it will overwrite an existing repo description with an automatically-generated one.  Same for [[`git-annex-describe`|git-annex-describe]].
+
+### What version of git-annex are you using? On what operating system?
+(master_env_v135_py36) 13:44  [r2] $ git annex version
+git-annex version: 7.20190507-g0c2cc3d
+build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.4.2 http-client-0.5.14 persistent-sq\
+lite-2.9.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_\
+224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLA\
+KE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP\
+256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
+operating system: linux x86_64
+supported repository versions: 5 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+local repository version: 5
+(master_env_v135_py36) 13:47  [r2] $ uname -a
+Linux ip-172-31-90-58.ec2.internal 4.14.114-105.126.amzn2.x86_64 #1 SMP Tue May 7 02:26:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
diff --git a/doc/bugs/still_seeing_errors_with_parallel_git-annex-add.mdwn b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add.mdwn
new file mode 100644
index 000000000..282cb558d
--- /dev/null
+++ b/doc/bugs/still_seeing_errors_with_parallel_git-annex-add.mdwn
@@ -0,0 +1,17 @@
+It seems that the [[issue with parallel git-annex-add|bugs/withOtherTmp_file_escapes]] is not yet quite fixed; I'm seeing it in the git-annex version 7.20190507-g0c2cc3d .  Repeating the git-annex-add command succeeds in adding the files that failed the first time.
+
+[[!format sh """
+add benchmarks/rabies200/analysis-FK7BF7Q0761xKv6V3z3gzZgK/benchmark_variants/v1.23.0_spades-3.13.0-trinity-metaspades-01/inputs-local.json
+git-annex: .git/annex/othertmp/inge11793-2931: rename: permission denied (Permission denied)
+failed
+add benchmarks/rabies200/analysis-FK7BKGj0761VgPb22550gfBX/benchmark_variants/v1.23.0_spades-3.13.0-trinity-metaspades-01/inputs-local.json
+git-annex: .git/annex/othertmp/inge11793-3330: rename: permission denied (Permission denied)
+failed
+git-annex: add: 2 failed
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+git-annex has become an essential tool for organizing my bioinformatics data analyses.  It scales well to large repos.  It lets me keep track of analyses without
+locking myself into any specific cloud platform.  It lets me quickly free up space on demand by moving data to cheaper storage, with the assurance that nothing will get lost.
+Thanks @joeyh for all your work on it.
diff --git a/doc/forum/git-annex__58___openFile__58___resource_busy___40__file_is_locked__41__/comment_3_4462c0640e11e20a8655743ecc78eaaa._comment b/doc/forum/git-annex__58___openFile__58___resource_busy___40__file_is_locked__41__/comment_3_4462c0640e11e20a8655743ecc78eaaa._comment
new file mode 100644
index 000000000..ae97acc29
--- /dev/null
+++ b/doc/forum/git-annex__58___openFile__58___resource_busy___40__file_is_locked__41__/comment_3_4462c0640e11e20a8655743ecc78eaaa._comment
@@ -0,0 +1,76 @@
+[[!comment format=mdwn
+ username="ply"
+ avatar="http://cdn.libravatar.org/avatar/1270501a59ed4a4042366b00295fe236"
+ subject="comment 3"
+ date="2019-05-17T20:55:48Z"
+ content="""
+Hello, I have similar problem, but with exFAT on macOS 10.13.6. Output is not stripped, it's no filename there:
+
+```
+$ git annex info --fast --debug               
+[2019-05-17 22:42:09.997466] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
+[2019-05-17 22:42:12.914831] process done ExitSuccess
+[2019-05-17 22:42:12.915017] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+[2019-05-17 22:42:12.953171] process done ExitSuccess
+[2019-05-17 22:42:12.953815] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..a0409bc558f0ed6679bf591e80f9782f1cd81da3\",\"--pretty=%H\",\"-n1\"]
+[2019-05-17 22:42:12.995305] process done ExitSuccess
+[2019-05-17 22:42:12.996023] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
+[2019-05-17 22:42:13.014492] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-05-17 22:42:13.029075] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+[2019-05-17 22:42:13.043862] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"]
+[2019-05-17 22:42:13.050909] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"diff-index\",\"--raw\",\"-z\",\"-r\",\"--no-renames\",\"-l0\",\"--cached\",\"refs/heads/git-annex\",\"--\"]
+[2019-05-17 22:42:14.066638] process done ExitSuccess
+[2019-05-17 22:42:14.08207] process done ExitSuccess
+[2019-05-17 22:42:14.090912] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"]
+[2019-05-17 22:42:15.852868] process done ExitSuccess
+(recording state in git...)
+[2019-05-17 22:42:15.868392] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"]
+[2019-05-17 22:42:17.763573] process done ExitSuccess
+[2019-05-17 22:42:17.763746] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"20ab70e8f50d62cc8d9657eaaff71cce88dfaa21\",\"--no-gpg-sign\",\"-p\",\"refs/heads/git-annex\"]
+[2019-05-17 22:42:17.800472] process done ExitSuccess
+[2019-05-17 22:42:17.800747] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"e0259808f7fbed2ce7cf0d8080dabb8d208c0da1\"]
+[2019-05-17 22:42:17.867585] process done ExitSuccess
+
+git-annex: openFile: resource busy (file is locked)
+failed
+[2019-05-17 22:42:17.870299] process done ExitSuccess
+[2019-05-17 22:42:17.875024] process done ExitSuccess
+[2019-05-17 22:42:17.886388] process done ExitSuccess
+```
+
+I tried `sudo dtruss -f git annex info --fast --debug`, and this is an excerpt from the terminal:
+
+```
+82503/0x1b0045:  write(0x2, \"[2019-05-17 22:29:55.91659] process done ExitSuccess\n\0\", 0x35)		 = 53 0
+82503/0x1b0045:  mkdir(\".git/annex\0\", 0x1FF, 0x0)		 = -1 Err#17
+82503/0x1b0045:  stat64(\".git/annex\0\", 0x4200038E40, 0x0)		 = 0 0
+82503/0x1b0045:  getpid(0x0, 0x0, 0x0)		 = 82503 0
+82503/0x1b0045:  open(\".git/annex/index.lck82503-2.tmp\0\", 0x20A06, 0x180)		 = 13 0
+82503/0x1b0045:  fstat64(0xD, 0x420004D050, 0x0)		 = 0 0
+82503/0x1b0045:  stat64(\".git/annex\0\", 0x420004D1D0, 0x0)		 = 0 0
+82503/0x1b0045:  mkdir(\".git/annex\0\", 0x1FF, 0x0)		 = -1 Err#17
+82503/0x1b0045:  stat64(\".git/annex\0\", 0x420004D3A0, 0x0)		 = 0 0
+82503/0x1b0045:  open(\".git/annex/index.lck82503-3.tmp\0\", 0x20A06, 0x180)		 = 21 0
+82503/0x1b0045:  fstat64(0x15, 0x420004D580, 0x0)		 = 0 0
+82503/0x1b0045:  close(0xC)		 = 0 0
+82503/0x1b0045:  poll(0x7FFEEC5CC810, 0x1, 0x0)		 = 1 0
+82503/0x1b0045:  write(0x1, \"\n\0\", 0x1)		 = 1 0
+82503/0x1b0045:  poll(0x7FFEEC5CC810, 0x1, 0x0)		 = 1 0
+82503/0x1b0045:  write(0x2, \"git-annex: openFile: resource busy (file is locked)\n\0\", 0x34)		 = 52 0
+```
+
+Removing `.git/annex/*.lck*` hadn't helped.
+
+I use git-annex version 7.20190322 from Homebrew:
+
+```
+build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.2.2 http-client-0.5.14 persistent-sqlite-2.6.4 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
+operating system: darwin x86_64
+supported repository versions: 5 7
+upgrade supported from repository versions: 0 1 2 3 4 5 6
+local repository version: 7
+```
+"""]]