diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_4_379ff2e840514a3b4f4a426e2aae4a9a._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_4_379ff2e840514a3b4f4a426e2aae4a9a._comment
new file mode 100644
index 000000000..95c51e5c9
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_4_379ff2e840514a3b4f4a426e2aae4a9a._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="git add or git commit does not trigger assistant, but git rm does"
+ date="2019-11-20T01:37:41Z"
+ content="""
+More diagnostics on this one..
+
+I create a file that is below the size threshold for annex, but I do have a gitlab upstream.
+
+1) echo \"foo\" > foo  -  Assistant (or webapp) does not fire
+2) git add foo -  Assistant/webapp does not fire
+3) git commit -am \"random testing\"  - assistant does not fire
+4) git annex sync - foo ends up in gitlab
+
+I remove a file
+
+1) git rm foo  - Assistant fires and file is removed from gitlab
+
+Is this the expected behavior?  If so, why does a git rm trigger annex assistant/webapp, but a git add does not? 
+
+
+"""]]

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_3_1556ebbb2a8db4aa859e3852160df044._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_3_1556ebbb2a8db4aa859e3852160df044._comment
new file mode 100644
index 000000000..65d4bf485
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_3_1556ebbb2a8db4aa859e3852160df044._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="git add or git commit does not trigger assistant, but git rm does"
+ date="2019-11-20T01:37:15Z"
+ content="""
+More diagnostics on this one..
+
+I create a file that is below the size threshold for annex, but I do have a gitlab upstream.
+
+1) echo \"foo\" > foo  -  Assistant (or webapp) does not fire
+2) git add foo -  Assistant/webapp does not fire
+3) git commit -am \"random testing\"  - assistant does not fire
+4) git annex sync - foo ends up in gitlab
+
+I remove a file
+
+1) git rm foo  - Assistant fires and file is removed from gitlab
+
+Is this the expected behavior?  If so, why does a git rm trigger annex assistant/webapp, but a git add does not? 
+
+
+"""]]

diff --git a/doc/todo/remove_legacy_import_directory_interface.mdwn b/doc/todo/remove_legacy_import_directory_interface.mdwn
new file mode 100644
index 000000000..d8f8a316c
--- /dev/null
+++ b/doc/todo/remove_legacy_import_directory_interface.mdwn
@@ -0,0 +1,47 @@
+The old `git annex import /dir` interface should be removed, in favor of
+the new import from special remote interface, which can be used with a
+directory special remote to do the same kind of operation.
+
+There have always been complaints about the old interface being surprising
+and/or not doing quite what some users want.
+Tried to find a principled way to address some of that with the "duplicate"
+options, but users just complain they're confusing (which they certianly
+are) and don't quite do what they want.
+
+The fundamental mistake that the old interface made is it conflated
+copying content into the repository, deleting content from the directory, 
+and updating the working tree. The new interface decouples all 3,
+only doing the first, and updating a tracking branch. The user is then free
+to merge the tracking branch as-is, or otherwise modify before merging.
+There are some options to manipulate the tracking branch in commonly
+wanted ways, which just boil down do git branch manipulation. Less common
+desires can be handled using all of git's facilities. As for deleting from
+the directory, that's an export of a branch, which can just be an empty
+branch if they want to delete everything, or again they can use all of git
+to construct the branch with the changes they desire.
+
+So while it's not been used as much as the old interface, I think the new
+interface will be much more flexible to the varied needs of users. What's
+less clear is if it can well support every way that the old interface can
+be used.
+
+Of course the first pain point is that the user has to set up a directory
+special remote. Which may be annoying if they are importing from a variety
+of different directories ad-hoc.
+
+Another likely pain point is ad-hoc importing of individual files or
+files matched by wildcard. The new interface is much more about importing
+whole trees, perhaps configured by preferred content settings
+
+One approach would be to make the old interface be implemented using the
+new interface, and paper over the cracks, by eg setting up a directory
+special remote automatically.
+
+Or, the old interface could warn when used, linking to some documentation
+about how to accomplish the same tasks with the new interface.
+
+Either could be done incrementally, eg start with the most common import
+cases, convert to the new interface, and keep others using the old
+interface.
+
+--[[Joey]]

diff --git a/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_3_ee247d9316df963fd584df2199b80fe2._comment b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_3_ee247d9316df963fd584df2199b80fe2._comment
new file mode 100644
index 000000000..d272eb88a
--- /dev/null
+++ b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_3_ee247d9316df963fd584df2199b80fe2._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="using hardlinks"
+ date="2019-11-19T17:51:08Z"
+ content="""
+I again don't have a full answer, but maybe you could customize git's [post-checkout hook](https://git-scm.com/docs/githooks#_post_checkout)?  (You'd need to still call the hook that git-annex installs.)
+
+Also, I've thrown together a [[FUSE filesystem|https://github.com/broadinstitute/viral-ngs/blob/is-dx-benchmarks/tools/git-annex-remotes/git-annex-on-demand.py]] that fetches git-annexed files on demand, maybe that could be adapted.  It only works with locked files and symlinks though.
+
+What is the reason you don't want to use locked files?   You can have different [[worktrees|tips/Using_git-worktree_with_annex]] with symlinks of locked files pointing into the same annex.
+"""]]

diff --git a/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_2_f2d436822490e74544bf58a4f1c9ee79._comment b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_2_f2d436822490e74544bf58a4f1c9ee79._comment
new file mode 100644
index 000000000..c4c86551d
--- /dev/null
+++ b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_2_f2d436822490e74544bf58a4f1c9ee79._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="import/export"
+ date="2019-11-19T17:39:47Z"
+ content="""
+Makes sense.  It's certainly better if import/export did complementary things.  Maybe, move the old `git-annex-import` functionality to a new command called `git-annex-ingest`?
+
+\"everything not involving importing from a special remote should be deprecated\" -- i.e. to ingest a directory you'd first create a directory special remote for it, and then `git-annex-import` from that?
+"""]]

diff --git a/Command/Import.hs b/Command/Import.hs
index ca4448808..0488ef4cb 100644
--- a/Command/Import.hs
+++ b/Command/Import.hs
@@ -39,7 +39,7 @@ cmd :: Command
 cmd = notBareRepo $
 	withGlobalOptions [jobsOption, jsonOptions, fileMatchingOptions] $
 		command "import" SectionCommon 
-			"import files from elsewhere into the repository"
+			"add a tree of files to the repository"
 			(paramPaths ++ "|BRANCH[:SUBDIR]")
 			(seek <$$> optParser)
 
diff --git a/doc/git-annex-import.mdwn b/doc/git-annex-import.mdwn
index 89a30e82c..1b7239f09 100644
--- a/doc/git-annex-import.mdwn
+++ b/doc/git-annex-import.mdwn
@@ -1,16 +1,16 @@
 # NAME
 
-git-annex import - add files from a non-versioned directory or a special remote
+git-annex import - add a tree of files to the repository
 
 # SYNOPSIS
 
-git annex import `[path ...]` | git annex import --from remote branch[:subdir]
+git annex import --from remote branch[:subdir] | `[path ...]`
 
 # DESCRIPTION
 
-This command is a way to import files from elsewhere into your git-annex
-repository. It can import files from a directory into your repository, 
-or it can import files from a git-annex special remote.
+This command is a way to import a tree of files from elsewhere into your
+git-annex repository. It can import files from a git-annex special remote,
+or from a directory.
 
 # IMPORTING FROM A SPECIAL REMOTE
 
@@ -30,7 +30,8 @@ remote.
 
 You can only import from special remotes that were configured with
 `importtree=yes` when set up with [[git-annex-initremote]](1). Only some
-kinds of special remotes will let you configure them this way.
+kinds of special remotes will let you configure them this way. A perhaps
+non-exhastive list is the directory, s3, and adb special remotes.
 
 To import from a special remote, you must specify the name of a branch.
 A corresponding remote tracking branch will be updated by `git annex
@@ -88,6 +89,10 @@ things that depend on the key. Preferred content expressions containing
 When run with a path, `git annex import` moves files from somewhere outside
 the git working copy, and adds them to the annex.
 
+This is a legacy interface. It is still supported, but please consider
+switching to importing from a directory special remote instead, using the
+interface documented above.
+
 Individual files to import can be specified. If a directory is specified,
 the entire directory is imported.
   
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 43f656048..b191e3a5b 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -146,10 +146,9 @@ subdirectories).
   
   See [[git-annex-rmurl]](1) for details.
 
-* `import [path ...]`
+* `import --from remote branch[:subdir] | [path ...]`
 
-  Add files from a non-version-controlled directory or a 
-  special remote into the annex.
+  Add a tree of files to the repository.
   
   See [[git-annex-import]](1) for details.
 

diff --git a/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_1_4eb794daaeef843b104bd480e11f7b42._comment b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_1_4eb794daaeef843b104bd480e11f7b42._comment
new file mode 100644
index 000000000..7e1005036
--- /dev/null
+++ b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default/comment_1_4eb794daaeef843b104bd480e11f7b42._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-19T17:09:00Z"
+ content="""
+My general feeling about git-annex import is that everything not involving
+importing from a special remote should be deprecated and eventually
+removed.
+
+The --duplicate option probably does what you want, but if the interface is
+going to be changed, such as making that the default, I'd rather the
+interface change move toward the goal of deprecating the old mode.
+
+The fundamental mistake that the legacy interface made is it conflated
+copying content into the repository, dropping content from the directory, 
+and updating the working tree. The new interface decouples all 3,
+only doing the first, and updating a tracking branch, which the user is then
+free to merge as-is, or otherwise modify before merging. Dropping requires
+an export of a new tree, which is the main pain point in emulating
+the old interface, but you happen to not want to drop the content from the
+directory, so that pain point shouldn't affect you.
+"""]]

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_f6196f814ed388d276947ffcc92ff37c._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_f6196f814ed388d276947ffcc92ff37c._comment
new file mode 100644
index 000000000..456146b11
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_f6196f814ed388d276947ffcc92ff37c._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="comment 2"
+ date="2019-11-19T17:18:18Z"
+ content="""
+bump?  Any thoughts?
+"""]]

diff --git a/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_1_d385d0fffdd6ac18f38828f805e4daff._comment b/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_1_d385d0fffdd6ac18f38828f805e4daff._comment
new file mode 100644
index 000000000..a34534e6f
--- /dev/null
+++ b/doc/bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place/comment_1_d385d0fffdd6ac18f38828f805e4daff._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-19T17:12:41Z"
+ content="""
+I think that you can accomplish what you want by making the directory
+you're importing from be a directory special remote with exporttree=yes
+importtree=yes and use the new `git annex import master --from remote`
+
+If that does not do what you want, I'd prefer to look at making it be able
+to do so. I hope to eventually remove the legacy git-annex import from
+directory, since we have this new more general interface.
+"""]]

diff --git a/doc/todo/add_import_--to_command/comment_4_4a30627ac78b32911604c3377b958cd0._comment b/doc/todo/add_import_--to_command/comment_4_4a30627ac78b32911604c3377b958cd0._comment
new file mode 100644
index 000000000..20d03ae2c
--- /dev/null
+++ b/doc/todo/add_import_--to_command/comment_4_4a30627ac78b32911604c3377b958cd0._comment
@@ -0,0 +1,40 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2019-11-19T16:58:06Z"
+ content="""
+Since git-annex does now support importtree from directory special remotes,
+you can almost get what you said you want by:
+
+	git annex initremote usb-drive type=directory directory=/mnt/usb-drive/myfiles \
+		exporttree=yes importtree=yes encryption=none
+
+Then `git annex import master --from usb-drive` will import the files
+into a usb-drive/master branch that you can merge. And you can run it
+repeatedly to import new and changed files from the directory.
+
+So then you have the files sitting in a special remote like you wanted.
+Namely the directory special remote on the USB drive. Only problem is that
+importing the files does also copy them into the git-annex repo. So you'd
+have to drop the files again, assuming you had disk space for them all
+to begin with.
+
+I wonder, if it were possible to import the files without add their content
+to the repo you ran the import from, leaving them on the special remote,
+would that meet your use case? That seems like something it would be
+possible to add.
+
+It would still probably have to copy the file into the local repo, in order
+to hash it, and then just delete the content from the local repo. Of course
+when the file is in a directory on the local system, that's not strictly
+necessary; it could do the hashing of the file in place. But that would
+need an extension to the special remote API to hash a file.
+
+But like I said in my other comment, I'd just clone my git-annex repo onto the
+drive and add the files to the repo there. Avoids all this complication.
+You'd need to provide a good justification for why you can't do that for
+me to pursue this any further.
+
+(As far as adding a --to switch to import, [[transitive_transfers]]
+discusses this kind of thing, and some issues with implementing that.)
+"""]]
diff --git a/doc/todo/git-annex-export_--from_option/comment_1_10f107aa0094d5ee4886878f5b1aaf06._comment b/doc/todo/git-annex-export_--from_option/comment_1_10f107aa0094d5ee4886878f5b1aaf06._comment
new file mode 100644
index 000000000..2f536eacd
--- /dev/null
+++ b/doc/todo/git-annex-export_--from_option/comment_1_10f107aa0094d5ee4886878f5b1aaf06._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-19T16:52:00Z"
+ content="""
+`git annex export --from` would be basically the same as
+[[transitive_transfers]] and the comments there detail
+the problems with trying to support that.
+
+What you can do is download the files onto the computer that is connected
+to the phone, export them to the phone, and then drop the files from the
+computer.
+"""]]

diff --git a/doc/index.mdwn b/doc/index.mdwn
new file mode 100644
index 000000000..77b61da9d
--- /dev/null
+++ b/doc/index.mdwn
@@ -0,0 +1,40 @@
+[[!inline raw=yes pages="summary"]]
+
+[[!sidebar content="""
+[[!inline feeds=no template=bare pages=sidebar]]
+"""]]
+
+<table>
+<tr>
+<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/key_concepts]]</td>
+<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/the_details]]</td>
+<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/other_stuff]]</td>
+</tr>
+</table>
+
+<table>
+<tr>
+<td width="50%" valign="top">[[!inline feeds=no template=bare pages=use_case/bob]]</td>
+<td width="50%" valign="top">[[!inline feeds=no template=bare pages=use_case/alice]]</td>
+</tr>
+</table>
+
+If that describes you, or if you're some from column A and some from column
+B, then git-annex may be the tool you've been looking for to expand from
+keeping all your small important files in git, to managing your large
+files with git.
+
+<table>
+<tr>
+<td width="50%" valign="top">[[!inline feeds=no template=bare pages=footer/column_a]]</td>
+<td width="50%" valign="top">[[!inline feeds=no template=bare pages=footer/column_b]]</td>
+</tr>
+</table>
+
+----
+
+git-annex is [[Free Software|license]], written in [Haskell](http://www.haskell.org/).
+You can [[contribute]]!
+
+git-annex's wiki is powered by [Ikiwiki](http://ikiwiki.info/) and
+hosted by [Branchable](http://branchable.com/).

diff --git a/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_2_e4bfea21f664a0cb1aa8ec18ee88fc50._comment b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_2_e4bfea21f664a0cb1aa8ec18ee88fc50._comment
new file mode 100644
index 000000000..359626b21
--- /dev/null
+++ b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_2_e4bfea21f664a0cb1aa8ec18ee88fc50._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="satya.ortiz-gagne@a4c92de91eb4fd5ae8fc9893bb4fd674a19f2e59"
+ nickname="satya.ortiz-gagne"
+ avatar="http://cdn.libravatar.org/avatar/79c93025f174cd2aff98fbb952702c09"
+ subject="Re: using hardlinks"
+ date="2019-11-18T20:46:22Z"
+ content="""
+Thanks for your comment. I've looked into [local caching of annexed files](https://git-annex.branchable.com/tips/local_caching_of_annexed_files) and most of it can be found in the scenario [described in the test gist](https://gist.github.com/satyaog/b08a6e5d1eee75217ba823d38b84fb8b).
+
+The two settings `annex.thin` and `annex.hardlink` are also set in the two git-annex repositories of the test. Thanks for letting me know about the caveats. Based on the tests that I've executed, it would seam that [`git-annex unlock`](https://git-annex.branchable.com/git-annex-unlock/) now copies the file to avoid the mentioned issue as I noticed different inodes? I understand that this prevents unwanted lost of data while using git-annex but I would actually like to have a hardlink instead of a copy. I'm wondering if it's possible.
+"""]]

diff --git a/doc/devblog/day_608__easier_git-lfs_setup.mdwn b/doc/devblog/day_608__easier_git-lfs_setup.mdwn
new file mode 100644
index 000000000..4c1f6da54
--- /dev/null
+++ b/doc/devblog/day_608__easier_git-lfs_setup.mdwn
@@ -0,0 +1,25 @@
+The git-lfs support I added to git-annex had one small problem: People
+expect to be able to clone a git repo and get right to using it, but after
+cloning a git-annex repo that's on a server that uses git-lfs, there
+was an extra `git annex enableremote` step to be able to use it as a git-lfs
+special remote. And, you ended up with a "origin" git remote and a git-lfs
+special remote with some other name.
+
+Now, it's this simple to set up a git-lfs repo on eg, github:
+
+	git annex initremote github type=git-lfs encryption=none url=https://github.com/joeyh/lfstest
+	git annex sync github
+	git annex copy --to github ...
+
+And then for others to clone and use it is even simpler:
+
+	git clone https://github.com/joeyh/lfstest
+	cd lfstest
+	git annex get
+
+The only gotcha is that git-annex has to know the url that's used for
+the remote. Cloning any other url any other way (eg http instead of https)
+will result in git-annex not using it. This is a consequence of git-lfs
+not having any equivilant of a git-annex repository UUID, so git-annex
+can't probe for the UUID and has to compare urls. This can be worked
+around using `initremote --sameas` to tell git-annex about other urls.

diff --git a/CHANGELOG b/CHANGELOG
index 7fc2c7712..ab28d365a 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -7,6 +7,7 @@ git-annex (7.20191115) UNRELEASED; urgency=medium
     an url. initremote --sameas can be used to add additional urls.
   * git-lfs: When there's a git remote with an url that's known to be
     used for git-lfs, automatically enable the special remote.
+  * sync, assistant: Pull and push from git-lfs remotes.
 
  -- Joey Hess <id@joeyh.name>  Fri, 15 Nov 2019 11:57:19 -0400
 
diff --git a/Remote/List.hs b/Remote/List.hs
index 5f3016b25..3e7ca9fa7 100644
--- a/Remote/List.hs
+++ b/Remote/List.hs
@@ -128,4 +128,8 @@ updateRemote remote = do
 {- Checks if a remote is syncable using git. -}
 gitSyncableRemote :: Remote -> Bool
 gitSyncableRemote r = remotetype r `elem`
-	[ Remote.Git.remote, Remote.GCrypt.remote, Remote.P2P.remote ]
+	[ Remote.Git.remote
+	, Remote.GCrypt.remote
+	, Remote.P2P.remote
+	, Remote.GitLFS.remote
+	]
diff --git a/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn b/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn
index 13df9dcfd..ebefc2c59 100644
--- a/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn
+++ b/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn
@@ -1,6 +1,5 @@
 git annex sync with a git-lfs special remote does not pull or push.
 It should.
-
-(Does gcrypt have the same problem, also being a special remote that's a
-git repo?)
 --[[Joey]]
+
+> [[fixed|done]] --[[Joey]]

diff --git a/Annex/SpecialRemote.hs b/Annex/SpecialRemote.hs
index 828eb6e77..37e0e2129 100644
--- a/Annex/SpecialRemote.hs
+++ b/Annex/SpecialRemote.hs
@@ -34,21 +34,9 @@ findExisting name = do
 	t <- trustMap
 	headMaybe
 		. sortBy (comparing $ \(u, _, _) -> Down $ M.lookup u t)
-		. findByName name
+		. findByRemoteConfig (\c -> lookupName c == Just name)
 		<$> Logs.Remote.readRemoteLog
 
-findByName :: RemoteName ->  M.Map UUID RemoteConfig -> [(UUID, RemoteConfig, Maybe (ConfigFrom UUID))]
-findByName n = map sameasuuid . filter (matching . snd) . M.toList
-  where
-	matching c = case lookupName c of
-		Nothing -> False
-		Just n'
-			| n' == n -> True
-			| otherwise -> False
-	sameasuuid (u, c) = case M.lookup sameasUUIDField c of
-		Nothing -> (u, c, Nothing)
-		Just u' -> (toUUID u', c, Just (ConfigFrom u))
-
 newConfig
 	:: RemoteName
 	-> Maybe (Sameas UUID)
diff --git a/Annex/SpecialRemote/Config.hs b/Annex/SpecialRemote/Config.hs
index 73688569c..e09ae8ecc 100644
--- a/Annex/SpecialRemote/Config.hs
+++ b/Annex/SpecialRemote/Config.hs
@@ -101,3 +101,11 @@ removeSameasInherited :: RemoteConfig -> RemoteConfig
 removeSameasInherited c = case M.lookup sameasUUIDField c of
 	Nothing -> c
 	Just _ -> M.withoutKeys c sameasInherits
+
+{- Finds remote uuids with matching RemoteConfig. -}
+findByRemoteConfig :: (RemoteConfig -> Bool) -> M.Map UUID RemoteConfig -> [(UUID, RemoteConfig, Maybe (ConfigFrom UUID))]
+findByRemoteConfig matching = map sameasuuid . filter (matching . snd) . M.toList
+  where
+	sameasuuid (u, c) = case M.lookup sameasUUIDField c of
+		Nothing -> (u, c, Nothing)
+		Just u' -> (toUUID u', c, Just (ConfigFrom u))
diff --git a/CHANGELOG b/CHANGELOG
index 182448d54..7fc2c7712 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,11 @@ git-annex (7.20191115) UNRELEASED; urgency=medium
 
   * Stop displaying rsync progress, and use git-annex's own progress display
     for local-to-local repo transfers.
+  * git-lfs: The url provided to initremote/enableremote will now be
+    stored in the git-annex branch, allowing enableremote to be used without
+    an url. initremote --sameas can be used to add additional urls.
+  * git-lfs: When there's a git remote with an url that's known to be
+    used for git-lfs, automatically enable the special remote.
 
  -- Joey Hess <id@joeyh.name>  Fri, 15 Nov 2019 11:57:19 -0400
 
diff --git a/Git/Config.hs b/Git/Config.hs
index f250af73d..9ebd4bd0f 100644
--- a/Git/Config.hs
+++ b/Git/Config.hs
@@ -94,6 +94,14 @@ store s repo = do
 		, fullconfig = M.unionWith (++) c (fullconfig repo)
 		}
 
+{- Stores a single config setting in a Repo, returning the new version of
+ - the Repo. Config settings can be updated incrementally. -}
+store' :: String -> String -> Repo -> Repo
+store' k v repo = repo
+	{ config = M.singleton k v `M.union` config repo
+	, fullconfig = M.unionWith (++) (M.singleton k [v]) (fullconfig repo)
+	}
+
 {- Updates the location of a repo, based on its configuration.
  -
  - Git.Construct makes LocalUknown repos, of which only a directory is
diff --git a/Git/Remote.hs b/Git/Remote.hs
index 9b05a86fb..fa336013e 100644
--- a/Git/Remote.hs
+++ b/Git/Remote.hs
@@ -51,6 +51,7 @@ makeLegalName s = case filter legal $ replace "/" "_" s of
 	legal c = isAlphaNum c
 	
 data RemoteLocation = RemoteUrl String | RemotePath FilePath
+	deriving (Eq)
 
 remoteLocationIsUrl :: RemoteLocation -> Bool
 remoteLocationIsUrl (RemoteUrl _) = True
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 059a1673d..6e1b31f74 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -143,7 +143,9 @@ configRead autoinit r = do
 		(True, _, _)
 			| remoteAnnexCheckUUID gc -> tryGitConfigRead autoinit r
 			| otherwise -> return r
-		(False, _, NoUUID) -> tryGitConfigRead autoinit r
+		(False, _, NoUUID) -> configSpecialGitRemotes r >>= \case
+			Nothing -> tryGitConfigRead autoinit r
+			Just r' -> return r'
 		_ -> return r
 
 gen :: Git.Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex (Maybe Remote)
@@ -231,7 +233,7 @@ repoAvail r
 tryGitConfigRead :: Bool -> Git.Repo -> Annex Git.Repo
 tryGitConfigRead autoinit r 
 	| haveconfig r = return r -- already read
-	| Git.repoIsSsh r = store $ do
+	| Git.repoIsSsh r = storeUpdatedRemote $ do
 		v <- Ssh.onRemote NoConsumeStdin r
 			(pipedconfig, return (Left $ giveup "configlist failed"))
 			"configlist" [] configlistfields
@@ -240,10 +242,10 @@ tryGitConfigRead autoinit r
 				| haveconfig r' -> return r'
 				| otherwise -> configlist_failed
 			Left _ -> configlist_failed
-	| Git.repoIsHttp r = store geturlconfig
+	| Git.repoIsHttp r = storeUpdatedRemote geturlconfig
 	| Git.GCrypt.isEncrypted r = handlegcrypt =<< getConfigMaybe (remoteConfig r "uuid")
 	| Git.repoIsUrl r = return r
-	| otherwise = store $ liftIO $ 
+	| otherwise = storeUpdatedRemote $ liftIO $ 
 		readlocalannexconfig `catchNonAsync` (const $ return r)
   where
 	haveconfig = not . M.null . Git.config
@@ -278,18 +280,6 @@ tryGitConfigRead autoinit r
 				set_ignore "not usable by git-annex" False
 				return r
 
-	store = observe $ \r' -> do
-		l <- Annex.getGitRemotes
-		let rs = exchange l r'
-		Annex.changeState $ \s -> s { Annex.gitremotes = Just rs }
-
-	exchange [] _ = []
-	exchange (old:ls) new
-		| Git.remoteName old == Git.remoteName new =
-			new : exchange ls new
-		| otherwise =
-			old : exchange ls new
-
 	{- Is this remote just not available, or does
 	 - it not have git-annex-shell?
 	 - Find out by trying to fetch from the remote. -}
@@ -319,7 +309,7 @@ tryGitConfigRead autoinit r
 		g <- gitRepo
 		case Git.GCrypt.remoteRepoId g (Git.remoteName r) of
 			Nothing -> return r
-			Just v -> store $ liftIO $ setUUID r $
+			Just v -> storeUpdatedRemote $ liftIO $ setUUID r $
 				genUUIDInNameSpace gCryptNameSpace v
 
 	{- The local repo may not yet be initialized, so try to initialize
@@ -337,6 +327,31 @@ tryGitConfigRead autoinit r
 		then [(Fields.autoInit, "1")]
 		else []
 
+{- Handles special remotes that can be enabled by the presence of
+ - regular git remotes.
+ -
+ - When a remote repo is found to be such a special remote, its
+ - UUID is cached in the git config, and the repo returned with
+ - the UUID set.
+ -}
+configSpecialGitRemotes :: Git.Repo -> Annex (Maybe Git.Repo)
+configSpecialGitRemotes r = Remote.GitLFS.configKnownUrl r >>= \case
+	Nothing -> return Nothing
+	Just r' -> Just <$> storeUpdatedRemote (return r')
+
+storeUpdatedRemote :: Annex Git.Repo -> Annex Git.Repo
+storeUpdatedRemote = observe $ \r' -> do
+	l <- Annex.getGitRemotes
+	let rs = exchange l r'
+	Annex.changeState $ \s -> s { Annex.gitremotes = Just rs }
+  where
+	exchange [] _ = []
+	exchange (old:ls) new
+		| Git.remoteName old == Git.remoteName new =
+			new : exchange ls new
+		| otherwise =
+			old : exchange ls new
+
 {- Checks if a given remote has the content for a key in its annex. -}
 inAnnex :: Remote -> State -> Key -> Annex Bool
 inAnnex rmt st key = do
diff --git a/Remote/GitLFS.hs b/Remote/GitLFS.hs
index d62978819..01f76a5a8 100644
--- a/Remote/GitLFS.hs
+++ b/Remote/GitLFS.hs
@@ -5,7 +5,7 @@
  - Licensed under the GNU AGPL version 3 or higher.
  -}
 
-module Remote.GitLFS (remote, gen) where
+module Remote.GitLFS (remote, gen, configKnownUrl) where
 
 import Annex.Common
 import Types.Remote

(Diff truncated)

diff --git a/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_1_b59a72142e8f752d7b68a63a6ac1bbfe._comment b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_1_b59a72142e8f752d7b68a63a6ac1bbfe._comment
new file mode 100644
index 000000000..d341a02d3
--- /dev/null
+++ b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks/comment_1_b59a72142e8f752d7b68a63a6ac1bbfe._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="using hardlinks"
+ date="2019-11-18T19:21:18Z"
+ content="""
+I don't have a full answer, but [[tips/local_caching_of_annexed_files]] might have relevant info.
+
+There is also the `annex.thin` setting; but check some [caveats](https://git-annex.branchable.com/bugs/annex.thin_can_cause_corrupt___40__not_just_missing__41___data/) related to it.
+"""]]

diff --git a/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks.mdwn b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks.mdwn
new file mode 100644
index 000000000..403e2cafa
--- /dev/null
+++ b/doc/forum/How_to_prevent_copies_on_a_single_device_and_use_only_hardlinks.mdwn
@@ -0,0 +1,13 @@
+Hi,
+
+Thank you very much for this software. I'm working in a research institute and we are very interested into using git-annex with DataLad to manage our datasets.
+
+We aim to provide a datasets repository accessible through the local network on a single file system. Some of our datasets are multi TB with a few millions of files. It will be managed by a few people but the primary users, the researchers, will only have read access. We would like to use hardlinks everywhere to avoid infrequent reading errors related to symlinks and save space when we want to propose different versions of the datasets with slight changes. The file system will be backed-up so we don't really need multi copies of the same files on a single file system.
+
+We seam to be able to achieve this using the `direct` mode in git-annex version 5 but it seams that the `unlock` mode in version 7 does copies instead of hardlinks. I'm wondering how we could achieve the same behaviour as in version 5. I believe I've read in the doc that there's a maximum of 2 hardlinks for a single file but I can't remember where or see if that is still the case. If that is still the case, I couldn't find if there is a setting to set or remove this maximum.
+
+We've tested with git-annex local version 5 / build 7.20190819, local version 7 / build 7.20190819 and local version 7 / build 7.20191106. [Here is a gist](https://gist.github.com/satyaog/b08a6e5d1eee75217ba823d38b84fb8b) containing test scripts for each setup. The `.annex-cache` part can be ignored for this topic. I've used Miniconda3-4.3.30 on Ubuntu 18.04.2 LTS to setup the environments.
+
+Thank you,
+
+Satya

diff --git a/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn b/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn
new file mode 100644
index 000000000..13df9dcfd
--- /dev/null
+++ b/doc/todo/sync_git-lfs_special_remote_should_sync_git_too.mdwn
@@ -0,0 +1,6 @@
+git annex sync with a git-lfs special remote does not pull or push.
+It should.
+
+(Does gcrypt have the same problem, also being a special remote that's a
+git repo?)
+--[[Joey]]

diff --git a/doc/special_remotes/git-lfs.mdwn b/doc/special_remotes/git-lfs.mdwn
index e48a76cf4..00e0d2940 100644
--- a/doc/special_remotes/git-lfs.mdwn
+++ b/doc/special_remotes/git-lfs.mdwn
@@ -10,13 +10,6 @@ the git-lfs special remote:
 
 * `url` - Required. The url to the git-lfs repository to use.
   Can be either a ssh url (scp-style is also accepted) or a http url.
-  But currently, a http url accesses the git-lfs repository without
-  authentication. To authenticate, you will need to use a ssh url.
-
-  This parameter needs to be specified in the initial `git annex
-  initremote` but also each time you `git annex enableremote`
-  an existing git-lfs special remote. It's fine to use different urls
-  at different times as long as they point to the same git-lfs repository.
 
 * `encryption` - One of "none", "hybrid", "shared", or "pubkey".
   Required. See [[encryption]]. Also see the encryption notes below.

diff --git a/doc/index.mdwn b/doc/index.mdwn
deleted file mode 100644
index 77b61da9d..000000000
--- a/doc/index.mdwn
+++ /dev/null
@@ -1,40 +0,0 @@
-[[!inline raw=yes pages="summary"]]
-
-[[!sidebar content="""
-[[!inline feeds=no template=bare pages=sidebar]]
-"""]]
-
-<table>
-<tr>
-<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/key_concepts]]</td>
-<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/the_details]]</td>
-<td width="33%" valign="top">[[!inline feeds=no template=bare pages=links/other_stuff]]</td>
-</tr>
-</table>
-
-<table>
-<tr>
-<td width="50%" valign="top">[[!inline feeds=no template=bare pages=use_case/bob]]</td>
-<td width="50%" valign="top">[[!inline feeds=no template=bare pages=use_case/alice]]</td>
-</tr>
-</table>
-
-If that describes you, or if you're some from column A and some from column
-B, then git-annex may be the tool you've been looking for to expand from
-keeping all your small important files in git, to managing your large
-files with git.
-
-<table>
-<tr>
-<td width="50%" valign="top">[[!inline feeds=no template=bare pages=footer/column_a]]</td>
-<td width="50%" valign="top">[[!inline feeds=no template=bare pages=footer/column_b]]</td>
-</tr>
-</table>
-
-----
-
-git-annex is [[Free Software|license]], written in [Haskell](http://www.haskell.org/).
-You can [[contribute]]!
-
-git-annex's wiki is powered by [Ikiwiki](http://ikiwiki.info/) and
-hosted by [Branchable](http://branchable.com/).

diff --git a/doc/todo/git-annex-export_--from_option.mdwn b/doc/todo/git-annex-export_--from_option.mdwn
new file mode 100644
index 000000000..c9d07ee9d
--- /dev/null
+++ b/doc/todo/git-annex-export_--from_option.mdwn
@@ -0,0 +1,7 @@
+I just wanted to have a way to manage data copying / syncing between a fileserver and my android phone. So I pushed some files on my fileserver into a git remote and added the files with the annex subcommands then cloned the git tree from my workstation which is connected to my smartphone.
+
+Now I followed the documentation about the special remote adb and created that remote with the initremote command. When I then export I get (not available) failed errors.
+
+Which is caused by the fact that I didn't have checked out the files on my workstation. I don't need the files on this pc so it would be stupid to checkout partially huge files there or in other words I don't need the files at that place, I don't get why the export command not has a --from option where it can get the files?
+
+Is there a reason that does not exist and if so what would be a way to do sending files to the android device without ssh-ing into my server?

diff --git a/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default.mdwn b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default.mdwn
new file mode 100644
index 000000000..4fc5706e2
--- /dev/null
+++ b/doc/todo/change_git-annex-import_not_to_delete_original_files_by_default.mdwn
@@ -0,0 +1,5 @@
+[[git-annex-import]] by default deletes the original files.  Keeping them by default would be better. "import" in many other tools (e.g. the bioinformatics tool [Geneious](https://www.geneious.com/)) means a non-destructive import.  The short description of `git-annex-import` on its man page says it "adds" files to the repo, which does not suggest erasure.  When I first used `git-annex-import`, I was surprised by the default behavior, and others may be too.  Also, the command has now been "overloaded" for importing from a special remote, and in that mode the originals are not erased; giving the import-from-dir mode the same default would be more consistent.  In general, erasing data by default seems dangerous: what if it was being imported into a temporary or untrusted repo?
+
+Changing the default would also let one [[repeatedly re-import a directory while keeping original files in place|bugs/impossible__40____63____41___to_continuously_re-import_a_directory_while_keeping_original_files_in_place]].
+
+I realize this would be a breaking change for some workflows; warning of it [[like git does|todo/warn_of_breaking_changes_same_way_git_does]] would mitigate the breakage.

diff --git a/doc/news/version_7.20191009.mdwn b/doc/news/version_7.20191009.mdwn
deleted file mode 100644
index c9647bf87..000000000
--- a/doc/news/version_7.20191009.mdwn
+++ /dev/null
@@ -1,29 +0,0 @@
-git-annex 7.20191009 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Fix bug in handling of annex.largefiles that use largerthan/smallerthan.
-     When adding a modified file, it incorrectly used the file size of the
-     old version of the file, not the current size.
-   * Added --mimetype and --mimeencoding file matching options.
-   * Added --unlocked and --locked file matching options.
-   * Added adjust --lock, to enter an adjusted branch where files are locked.
-   * git-lfs: Added support for http basic auth.
-   * git-lfs: Only do endpoint discovery once when concurrency is enabled.
-   * fsck --incremental/--more: Fix bug that prevented the incremental fsck
-     information from being updated every 5 minutes as it was supposed to be;
-     it was only updated after 1000 files were checked, which may be more
-     files that are possible to fsck in a given fsck time window.
-     Thanks to Peter Simons for help with analysis of this bug.
-   * Test: Use more robust directory removal when built with directory-1.2.7.
-   * Close sqlite databases more robustly.
-   * remotedaemon: Don't list --stop in help since it's not supported.
-   * enable-tor: Run kdesu with -c option.
-   * enable-tor: Use pkexec to run command as root when gksu and kdesu are not
-     available.
-   * When dropping an unlocked file, preserve its mtime, which avoids
-     git status unncessarily running the clean filter on the file.
-   * uninit: Remove several git hooks that git-annex init sets up.
-   * uninit: Remove the smudge and clean filters that git-annex init sets up.
-   * Work around git cat-file --batch's odd stripping of carriage return
-     from the end of the line (some windows infection), avoiding crashing
-     when the repo contains a filename ending in a carriage return.
-   * git-annex-standalone.rpm: Fix the git-annex-shell symlink."""]]
\ No newline at end of file
diff --git a/doc/news/version_7.20191114.mdwn b/doc/news/version_7.20191114.mdwn
new file mode 100644
index 000000000..6dfb3401e
--- /dev/null
+++ b/doc/news/version_7.20191114.mdwn
@@ -0,0 +1,10 @@
+git-annex 7.20191114 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Added annex.allowsign option.
+   * Make --json-error-messages capture more errors,
+     particularly url download errors.
+   * Fix a crash (STM deadlock) when -J is used with multiple files
+     that point to the same key.
+   * linuxstandalone: Fix a regression that broke git-remote-https.
+   * OSX git-annex.app: Fix a problem that prevented using the bundled
+     git-remote-https, git-remote-http, and git-shell."""]]
\ No newline at end of file

diff --git a/Build/OSXMkLibs.hs b/Build/OSXMkLibs.hs
index 17af6592b..8a11c88ee 100644
--- a/Build/OSXMkLibs.hs
+++ b/Build/OSXMkLibs.hs
@@ -50,8 +50,12 @@ installLibs appbase replacement_libs libmap = do
 		let symdest = appbase </> shortlib
 		-- This is a hack; libraries need to be in the same
 		-- directory as the program, so also link them into the
-		-- extra directory.
-		let symdestextra = appbase </> "extra" </> shortlib
+		-- extra and git-core directories so programs in those will
+		-- find them.
+		let symdestextra = 
+			[ appbase </> "extra" </> shortlib
+			, appbase </> "git-core" </> shortlib
+			]
 		ifM (doesFileExist dest)
 			( return Nothing
 			, do
@@ -59,9 +63,11 @@ installLibs appbase replacement_libs libmap = do
 				putStrLn $ "installing " ++ pathlib ++ " as " ++ shortlib
 				unlessM (boolSystem "cp" [File pathlib, File dest]
 					<&&> boolSystem "chmod" [Param "644", File dest]
-					<&&> boolSystem "ln" [Param "-s", File fulllib, File symdest]
-					<&&> boolSystem "ln" [Param "-s", File (".." </> fulllib), File symdestextra]) $
+					<&&> boolSystem "ln" [Param "-s", File fulllib, File symdest]) $
 					error "library install failed"
+				forM_ symdestextra $ \d ->
+					unlessM (boolSystem "ln" [Param "-s", File (".." </> fulllib), File d]) $
+						error "library linking failed"
 				return $ Just appbase
 			)
 	return (catMaybes libs, replacement_libs', libmap')
diff --git a/CHANGELOG b/CHANGELOG
index e775f6737..716bcb983 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,8 +6,8 @@ git-annex (7.20191107) UNRELEASED; urgency=medium
   * Fix a crash (STM deadlock) when -J is used with multiple files
     that point to the same key.
   * linuxstandalone: Fix a regression that broke git-remote-https.
-  * OSX git-annex.app: Fix a regression that broke git-remote-https,
-    git-remote-http, and git-shell.
+  * OSX git-annex.app: Fix a problem that prevented using the bundled
+    git-remote-https, git-remote-http, and git-shell.
 
  -- Joey Hess <id@joeyh.name>  Mon, 11 Nov 2019 15:59:47 -0400
 
diff --git a/doc/bugs/OSX_dmg_git-core_binaries_do_not_link.mdwn b/doc/bugs/OSX_dmg_git-core_binaries_do_not_link.mdwn
new file mode 100644
index 000000000..d976be1fd
--- /dev/null
+++ b/doc/bugs/OSX_dmg_git-core_binaries_do_not_link.mdwn
@@ -0,0 +1,12 @@
+The OSX .dmg contains a few binaries in git-core like git-remote-http.
+They have been adjusted by otool to link to libraries in the same directory
+as the binary. However, the libraries are not located in the git-core
+directory, but in its parent directory, and so the git-core binaries don't
+link.
+
+I don't think this is a new regression, but not entirely sure.
+
+Seems that OSXMkLibs could symlink ../lib into git-core.
+--[[Joey]] 
+
+> [[fixed|done]] --[[Joey]]

diff --git a/Build/Standalone.hs b/Build/Standalone.hs
index 6f1dab369..e02e70a66 100644
--- a/Build/Standalone.hs
+++ b/Build/Standalone.hs
@@ -56,16 +56,24 @@ installGitLibs topdir = do
 		if issymlink
 			then do
 				-- many git-core files may symlink to eg
-				-- ../../git. The link targets are put
-				-- into a subdirectory so all links to 
-				-- .../git get the same binary.
+				-- ../../bin/git, which is located outside
+				-- the git-core directory. The target of
+				-- such links is installed into a bin
+				-- directory, and the links repointed to it.
+				--
+				-- Other git-core files symlink to a file
+				-- beside them in the directory. Those
+				-- links can be copied as-is.
 				linktarget <- readSymbolicLink f
-				let linktarget' = gitcoredestdir </> "bin" </> takeFileName linktarget
-				createDirectoryIfMissing True (takeDirectory linktarget')
-				L.readFile f >>= L.writeFile linktarget'
-				nukeFile destf
-				rellinktarget <- relPathDirToFile (takeDirectory destf) linktarget'
-				createSymbolicLink rellinktarget destf
+				if takeFileName linktarget == linktarget
+					then cp f destf
+					else do
+						let linktarget' = gitcoredestdir </> "bin" </> takeFileName linktarget
+						createDirectoryIfMissing True (takeDirectory linktarget')
+						L.readFile f >>= L.writeFile linktarget'
+						nukeFile destf
+						rellinktarget <- relPathDirToFile (takeDirectory destf) linktarget'
+						createSymbolicLink rellinktarget destf
 			else cp f destf
 	
 	-- install git's template files
diff --git a/CHANGELOG b/CHANGELOG
index c6a628a6e..f2d4fe15a 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ git-annex (7.20191107) UNRELEASED; urgency=medium
     particularly url download errors.
   * Fix a crash (STM deadlock) when -J is used with multiple files
     that point to the same key.
+  * linuxstandalone: Fix a regression that broke git-remote-https.
 
  -- Joey Hess <id@joeyh.name>  Mon, 11 Nov 2019 15:59:47 -0400
 
diff --git a/Makefile b/Makefile
index 58b3f586c..944437e79 100644
--- a/Makefile
+++ b/Makefile
@@ -166,6 +166,7 @@ linuxstandalone:
 	cp /usr/share/file/magic.mgc "$(LINUXSTANDALONE_DEST)/magic"
 	cp /usr/share/i18n -a "$(LINUXSTANDALONE_DEST)"
 	
+	read me
 	./Build/LinuxMkLibs "$(LINUXSTANDALONE_DEST)"
 	
 	$(MAKE) install-mans DESTDIR="$(LINUXSTANDALONE_DEST)"
diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
index acdcb73e8..9ec6964e7 100644
--- a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
+++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
@@ -75,3 +75,5 @@ Unfortunately in datalad we had no test testing cloning over https, so I added s
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time/comment_1_1735409e62ce82f7ba7258b0167fda06._comment b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time/comment_1_1735409e62ce82f7ba7258b0167fda06._comment
new file mode 100644
index 000000000..1c80ed5ec
--- /dev/null
+++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time/comment_1_1735409e62ce82f7ba7258b0167fda06._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-14T18:14:35Z"
+ content="""
+It will either be caused by 5463f97ca216cd261f7a1da08aa8a62cef415a71 or by
+a new version of git reorging files (or both).
+"""]]

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment
deleted file mode 100644
index b24289907..000000000
--- a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment
+++ /dev/null
@@ -1,11 +0,0 @@
-[[!comment format=mdwn
- username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
- nickname="git-annex"
- avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
- subject="Sometimes it sorta works"
- date="2019-11-14T17:24:29Z"
- content="""
-So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab.  However, sometimes it works.  For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo.  When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't.  
-
-
-"""]]

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment
new file mode 100644
index 000000000..b24289907
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_2_a77d982399c91c2513ae74fb346243f0._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="Sometimes it sorta works"
+ date="2019-11-14T17:24:29Z"
+ content="""
+So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab.  However, sometimes it works.  For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo.  When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't.  
+
+
+"""]]

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment
new file mode 100644
index 000000000..71e8b6fe6
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo/comment_1_b61139816f55dbc4f7eac22207f243c0._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="Sometimes it sorta works"
+ date="2019-11-14T17:24:11Z"
+ content="""
+So, sometimes (often), it says it synched in the webapp, but no changes show up in any branch on gitlab.  However, sometimes it works.  For example, sometimes after killing everything off (including any ssh sessions) and restarting the webapp, a file I added to the repo locally as a test will show up in the gitlab repo.  When I then move that file, the webapp/assistant wakes up, syncs (says it was successful), and the \"delete\" is processed in the gitlab repo (the old file is deleted), but the \"add\" (the file showing up with a new name) doesn't.  
+
+
+"""]]

diff --git a/Annex/Concurrent.hs b/Annex/Concurrent.hs
index 4626a9294..1ff8e0c73 100644
--- a/Annex/Concurrent.hs
+++ b/Annex/Concurrent.hs
@@ -90,10 +90,20 @@ enteringStage newstage a = Annex.getState Annex.workers >>= \case
 	Nothing -> a
 	Just tv -> do
 		mytid <- liftIO myThreadId
-		let set = changeStageTo mytid tv newstage
-		let restore = maybe noop (void . changeStageTo mytid tv)
+		let set = changeStageTo mytid tv (const newstage)
+		let restore = maybe noop (void . changeStageTo mytid tv . const)
 		bracket set restore (const a)
 
+{- Transition the current thread to the initial stage.
+ - This is done once the thread is ready to begin work.
+ -}
+enteringInitialStage :: Annex ()
+enteringInitialStage = Annex.getState Annex.workers >>= \case
+	Nothing -> noop
+	Just tv -> do
+		mytid <- liftIO myThreadId
+		void $ changeStageTo mytid tv initialStage
+
 {- This needs to leave the WorkerPool with the same number of
  - idle and active threads, and with the same number of threads for each
  - WorkerStage. So, all it can do is swap the WorkerStage of our thread's
@@ -110,14 +120,15 @@ enteringStage newstage a = Annex.getState Annex.workers >>= \case
  - in the pool than spareVals. That does not prevent other threads that call
  - this from using them though, so it's fine.
  -}
-changeStageTo :: ThreadId -> TMVar (WorkerPool AnnexState) -> WorkerStage -> Annex (Maybe WorkerStage)
-changeStageTo mytid tv newstage = liftIO $
+changeStageTo :: ThreadId -> TMVar (WorkerPool AnnexState) -> (UsedStages -> WorkerStage) -> Annex (Maybe WorkerStage)
+changeStageTo mytid tv getnewstage = liftIO $
 	replaceidle >>= maybe
 		(return Nothing)
 		(either waitidle (return . Just))
   where
 	replaceidle = atomically $ do
 		pool <- takeTMVar tv
+		let newstage = getnewstage (usedStages pool)
 		let notchanging = do
 			putTMVar tv pool
 			return Nothing
@@ -128,7 +139,7 @@ changeStageTo mytid tv newstage = liftIO $
 						Nothing -> do
 							putTMVar tv $
 								addWorkerPool (IdleWorker oldstage) pool'
-							return $ Just $ Left (myaid, oldstage)
+							return $ Just $ Left (myaid, newstage, oldstage)
 						Just pool'' -> do
 							-- optimisation
 							putTMVar tv $
@@ -139,27 +150,26 @@ changeStageTo mytid tv newstage = liftIO $
 				_ -> notchanging
 			else notchanging
 	
-	waitidle (myaid, oldstage) = atomically $ do
+	waitidle (myaid, newstage, oldstage) = atomically $ do
 		pool <- waitIdleWorkerSlot newstage =<< takeTMVar tv
 		putTMVar tv $ addWorkerPool (ActiveWorker myaid newstage) pool
 		return (Just oldstage)
 
--- | Waits until there's an idle worker in the worker pool
--- for its initial stage, removes it from the pool, and returns its state.
+-- | Waits until there's an idle StartStage worker in the worker pool,
+-- removes it from the pool, and returns its state.
 --
 -- If the worker pool is not already allocated, returns Nothing.
-waitInitialWorkerSlot :: TMVar (WorkerPool Annex.AnnexState) -> STM (Maybe (Annex.AnnexState, WorkerStage))
-waitInitialWorkerSlot tv = do
+waitStartWorkerSlot :: TMVar (WorkerPool Annex.AnnexState) -> STM (Maybe (Annex.AnnexState, WorkerStage))
+waitStartWorkerSlot tv = do
 	pool <- takeTMVar tv
-	let stage = initialStage (usedStages pool)
-	st <- go stage pool
-	return $ Just (st, stage)
+	st <- go pool
+	return $ Just (st, StartStage)
   where
-	go wantstage pool = case spareVals pool of
+	go pool = case spareVals pool of
 		[] -> retry
 		(v:vs) -> do
 			let pool' = pool { spareVals = vs }
-			putTMVar tv =<< waitIdleWorkerSlot wantstage pool'
+			putTMVar tv =<< waitIdleWorkerSlot StartStage pool'
 			return v
 
 waitIdleWorkerSlot :: WorkerStage -> WorkerPool Annex.AnnexState -> STM (WorkerPool Annex.AnnexState)
diff --git a/CHANGELOG b/CHANGELOG
index 03d8e7b11..c6a628a6e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,8 @@ git-annex (7.20191107) UNRELEASED; urgency=medium
   * Added annex.allowsign option.
   * Make --json-error-messages capture more errors,
     particularly url download errors.
+  * Fix a crash (STM deadlock) when -J is used with multiple files
+    that point to the same key.
 
  -- Joey Hess <id@joeyh.name>  Mon, 11 Nov 2019 15:59:47 -0400
 
diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
index 87298a95f..67a7618e4 100644
--- a/CmdLine/Action.hs
+++ b/CmdLine/Action.hs
@@ -63,7 +63,7 @@ commandAction start = Annex.getState Annex.concurrency >>= \case
 	runconcurrent = Annex.getState Annex.workers >>= \case
 		Nothing -> runnonconcurrent
 		Just tv -> 
-			liftIO (atomically (waitInitialWorkerSlot tv)) >>=
+			liftIO (atomically (waitStartWorkerSlot tv)) >>=
 				maybe runnonconcurrent (runconcurrent' tv)
 	runconcurrent' tv (workerst, workerstage) = do
 		aid <- liftIO $ async $ snd <$> Annex.run workerst
@@ -99,12 +99,13 @@ commandAction start = Annex.getState Annex.concurrency >>= \case
 					case mkActionItem startmsg' of
 						OnlyActionOn k' _ | k' /= k ->
 							concurrentjob' workerst startmsg' perform'
-						_ -> mkjob workerst startmsg' perform'
+						_ -> beginjob workerst startmsg' perform'
 				Nothing -> noop
-		_ -> mkjob workerst startmsg perform
+		_ -> beginjob workerst startmsg perform
 	
-	mkjob workerst startmsg perform = 
-		inOwnConsoleRegion (Annex.output workerst) $
+	beginjob workerst startmsg perform =
+		inOwnConsoleRegion (Annex.output workerst) $ do
+			enteringInitialStage
 			void $ accountCommandAction startmsg $
 				performconcurrent startmsg perform
 
diff --git a/Types/WorkerPool.hs b/Types/WorkerPool.hs
index 178c30166..8a6816313 100644
--- a/Types/WorkerPool.hs
+++ b/Types/WorkerPool.hs
@@ -40,7 +40,12 @@ instance Show (Worker t) where
 	show (ActiveWorker _ s) = "ActiveWorker " ++ show s
 
 data WorkerStage
-	= PerformStage
+	= StartStage
+	-- ^ All threads start in this stage, and then transition away from
+	-- it to the initialStage when they begin doing work. This should
+	-- never be included in UsedStages, because transition from some
+	-- other stage back to this one could result in a deadlock.
+	| PerformStage
 	-- ^ Running a CommandPerform action.
 	| CleanupStage
 	-- ^ Running a CommandCleanup action.
@@ -102,12 +107,13 @@ workerAsync (ActiveWorker aid _) = Just aid
 allocateWorkerPool :: t -> Int -> UsedStages -> WorkerPool t
 allocateWorkerPool t n u = WorkerPool
 	{ usedStages = u
-	, workerList = take totalthreads $ map IdleWorker stages
+	, workerList = map IdleWorker $
+		take totalthreads $ concat $ repeat stages
 	, spareVals = replicate totalthreads t
 	}
   where
-	stages = concat $ repeat $ S.toList $ stageSet u
-	totalthreads = n * S.size (stageSet u)
+	stages = StartStage : S.toList (stageSet u)
+	totalthreads = n * length stages
 
 addWorkerPool :: Worker t -> WorkerPool t -> WorkerPool t
 addWorkerPool w pool = pool { workerList = w : workerList pool }
diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn
index 837cfecb1..1aa191ed8 100644
--- a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn
@@ -69,3 +69,5 @@ I felt like it is an old issue but failed to find a trace of it upon a quick loo
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_e33df0ec76069deb069b94c944d62c76._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_e33df0ec76069deb069b94c944d62c76._comment
new file mode 100644
index 000000000..7f348fc9f
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_e33df0ec76069deb069b94c944d62c76._comment
@@ -0,0 +1,69 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2019-11-14T15:20:13Z"
+ content="""
+Added tracing of changes to the WorkerPool.
+
+	joey@darkstar:/tmp/dst>git annex get -J1 1 2 --json
+	("initial pool",WorkerPool UsedStages {initialStage = TransferStage, stageSet = fromList [TransferStage,VerifyStage]} [IdleWorker TransferStage,IdleWorker VerifyStage] 2)
+	("starting worker",WorkerPool UsedStages {initialStage = TransferStage, stageSet = fromList [TransferStage,VerifyStage]} [ActiveWorker TransferStage,IdleWorker VerifyStage] 1)
+
+Transfer starts for file 1
+
+	(("change stage from",TransferStage,"to",VerifyStage),WorkerPool UsedStages {initialStage = TransferStage, stageSet = fromList [TransferStage,VerifyStage]} [IdleWorker TransferStage,ActiveWorker VerifyStage] 1)

(Diff truncated)

diff --git a/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn
new file mode 100644
index 000000000..c7f821b2b
--- /dev/null
+++ b/doc/forum/Issues_with_webapp___47___assistant_and_gitlab_as_a_metadata_only_repo.mdwn
@@ -0,0 +1,63 @@
+Greetings,
+
+Two more issues have come up.  
+
+1) webapp (and assistant) don't always seem to inherit ssh-agent keys (I've setup a passwordless key for gitlab).  It usually takes multiple cycles of killing off ssh and annex-related daemons before a new instance will not fail auth with gitlab - is there a more solid way of doing this?
+
+2) When I do get auth, the assistant / webui will notice file changes (i.e moving a small file that doesn't match annex.largefiles, but the changes don't get committed to origin (gitlab - metadata only).  The log file is below - thougts?
+
+[2019-11-13 20:24:33.378362] main: starting assistant version 7.20191106
+[2019-11-13 20:24:33.439501] TransferScanner: Syncing with origin
+(scanning...) [2019-11-13 20:24:33.529954] Watcher: Performing startup scan
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+fatal: Pathspec 'workflow/cc-archive-exif/LICENSE' is in submodule 'workflow/cc-archive-exif'
+
+git cat-file EOF: user error
+
+fd:39: hFlush: resource vanished (Broken pipe)
+
+fd:39: hFlush: resource vanished (Broken pipe)
+(started...)
+[2019-11-13 20:24:34.481957] Committer: Committing changes to git
+(recording state in git...)
+> GitLab: Disallowed command
+ControlSocket .git/annex/ssh/git@gitlab.com already exists, disabling multiplexing
+[2019-11-13 20:24:35.42995] Pusher: Syncing with origin
+Everything up-to-date
+> GitLab: Disallowed command
+Everything up-to-date
+> GitLab: Disallowed command
+> GitLab: Disallowed command
+> GitLab: Disallowed command
+> GitLab: Disallowed command
+> GitLab: Disallowed command
+
+fd:39: hFlush: resource vanished (Broken pipe)
+
+fd:39: hFlush: resource vanished (Broken pipe)
+[2019-11-13 20:26:09.256217] main: Syncing with origin
+Everything up-to-date
+> GitLab: Disallowed command
+
+fd:39: hFlush: resource vanished (Broken pipe)
+[2019-11-13 20:27:31.922901] Committer: Committing changes to git
+(recording state in git...)
+[2019-11-13 20:27:31.956249] Pusher: Syncing with origin
+Everything up-to-date
+
+fd:39: hFlush: resource vanished (Broken pipe)
+[2019-11-13 20:28:35.846741] Committer: Committing changes to git
+(recording state in git...)
+[2019-11-13 20:28:35.87987] Pusher: Syncing with origin
+Everything up-to-date
+> GitLab: Disallowed command
+> GitLab: Disallowed command

diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
index 153a7e2c8..acdcb73e8 100644
--- a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
+++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
@@ -67,6 +67,8 @@ git-credential-cache@  git-http-fetch@                git-remote-tor-annex@  rsy
 
 """]]
 
+</details>
+
 so may be that is related.
 
 Unfortunately in datalad we had no test testing cloning over https, so I added such integration test in https://github.com/datalad/datalad/pull/3867 to at least detect such regressions in the future before hitting the userland

diff --git a/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
new file mode 100644
index 000000000..153a7e2c8
--- /dev/null
+++ b/doc/bugs/regression__58___standalone_build_is_deficient_on_linux_after_7.20190819+git2-g908476a9b-1__126__ndall+1_some_time.mdwn
@@ -0,0 +1,75 @@
+### Please describe the problem.
+
+There were a few changes introduced since then to Makefile (I will not guess which one broke it) which resulted in git within git-annex-standalone of neurodebian to be unable to clone from https://:
+
+
+[[!format sh """
+$> /usr/lib/git-annex.linux/git clone https://github.com/afni/afni_ci_test_data.git                            
+Cloning into 'afni_ci_test_data'...
+fatal: unable to find remote helper for 'https'
+
+"""]]
+
+<details>
+<summary>diff between list of files in 7.20190819+git60-gcdb679818 and 7.20191017+git2-g7b13db551 package builds shows many git-* missing</summary>
+[[!format sh """
+lena:/tmp
+$> ls 7.2019*/usr/lib/git-annex.linux/exe/                                         
+7.20190819/usr/lib/git-annex.linux/exe/:
+cp@                            git-diff-index@          git-mktag@                git-sh-i18n--envsubst@
+curl@                          git-diff-tree@           git-mktree@               git-shell@
+git@                           git-difftool@            git-multi-pack-index@     git-shortlog@
+git-add@                       git-fast-export@         git-mv@                   git-show@
+git-am@                        git-fast-import@         git-name-rev@             git-show-branch@
+git-annex@                     git-fetch@               git-notes@                git-show-index@
+git-annex-shell@               git-fetch-pack@          git-pack-objects@         git-show-ref@
+git-annotate@                  git-fmt-merge-msg@       git-pack-redundant@       git-stage@
+git-apply@                     git-for-each-ref@        git-pack-refs@            git-status@
+git-archive@                   git-format-patch@        git-patch-id@             git-stripspace@
+git-bisect--helper@            git-fsck@                git-prune@                git-submodule--helper@
+git-blame@                     git-fsck-objects@        git-prune-packed@         git-symbolic-ref@
+git-branch@                    git-gc@                  git-pull@                 git-tag@
+git-bundle@                    git-get-tar-commit-id@   git-push@                 git-unpack-file@
+git-cat-file@                  git-grep@                git-range-diff@           git-unpack-objects@
+git-check-attr@                git-hash-object@         git-read-tree@            git-update-index@
+git-check-ignore@              git-help@                git-rebase@               git-update-ref@
+git-check-mailmap@             git-http-backend@        git-rebase--interactive@  git-update-server-info@
+git-check-ref-format@          git-http-fetch@          git-receive-pack@         git-upload-archive@
+git-checkout@                  git-http-push@           git-reflog@               git-upload-pack@
+git-checkout-index@            git-imap-send@           git-remote@               git-var@
+git-cherry@                    git-index-pack@          git-remote-ext@           git-verify-commit@
+git-cherry-pick@               git-init@                git-remote-fd@            git-verify-pack@
+git-clean@                     git-init-db@             git-remote-ftp@           git-verify-tag@
+git-clone@                     git-interpret-trailers@  git-remote-ftps@          git-whatchanged@
+git-column@                    git-log@                 git-remote-http@          git-worktree@
+git-commit@                    git-ls-files@            git-remote-https@         git-write-tree@
+git-commit-graph@              git-ls-remote@           git-remote-testsvn@       localedef@
+git-commit-tree@               git-ls-tree@             git-remote-tor-annex@     lsof@
+git-config@                    git-mailinfo@            git-repack@               rsync@
+git-count-objects@             git-mailsplit@           git-replace@              sh@
+git-credential@                git-merge@               git-rerere@               ssh@
+git-credential-cache@          git-merge-base@          git-reset@                ssh-keygen@
+git-credential-cache--daemon@  git-merge-file@          git-rev-list@             tar@
+git-credential-store@          git-merge-index@         git-rev-parse@            uname@
+git-daemon@                    git-merge-ours@          git-revert@               xargs@
+git-describe@                  git-merge-recursive@     git-rm@
+git-diff@                      git-merge-subtree@       git-send-pack@
+git-diff-files@                git-merge-tree@          git-serve@
+
+7.20191017/usr/lib/git-annex.linux/exe/:
+cp@                    git-credential-cache--daemon@  git-http-push@         git-sh-i18n--envsubst@  sh@
+curl@                  git-credential-store@          git-imap-send@         git-shell@              ssh@
+git@                   git-daemon@                    git-receive-pack@      git-upload-pack@        ssh-keygen@
+git-annex@             git-fast-import@               git-remote-http@       localedef@              tar@
+git-annex-shell@       git-http-backend@              git-remote-testsvn@    lsof@                   uname@
+git-credential-cache@  git-http-fetch@                git-remote-tor-annex@  rsync@                  xargs@
+
+
+"""]]
+
+so may be that is related.
+
+Unfortunately in datalad we had no test testing cloning over https, so I added such integration test in https://github.com/datalad/datalad/pull/3867 to at least detect such regressions in the future before hitting the userland
+
+[[!meta author=yoh]]
+[[!tag projects/datalad]]

diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment
new file mode 100644
index 000000000..0e4a37c0e
--- /dev/null
+++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_2_ff0dab8dc544f7aa8e4ebfbec8f2a081._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="git-annex@17927e6dc041ab425c14217a97a685adf3ecf44f"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/66e5c6e044d726597ce5a0ad68f86fe4"
+ subject="comment 2"
+ date="2019-11-14T00:38:56Z"
+ content="""
+Thank's Joey - that's what I needed. 
+"""]]

diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment
new file mode 100644
index 000000000..f8d18b1cf
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_6_880c962d5ca77c494c984e7f74725265._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="catch-all deadlock breaker"
+ date="2019-11-13T22:33:59Z"
+ content="""
+Not sure if feasible, but maybe a [[catch-all deadlock breaker|todo/more_extensive_retries_to_mask_transient_failures]] could be implemented to mask this and other deadlocks?
+
+The moon landings software [[had something|https://www.ibiblio.org/apollo/hrst/archive/1033.pdf]] [[like this|https://history.nasa.gov/computers/Ch2-6.html]], and it worked [[pretty well|https://www.wsj.com/articles/apollo-11-had-a-hidden-hero-software-11563153001]]...
+"""]]

diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_4_eecaa7f7b0279c56902c90ed58d1444f._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_4_eecaa7f7b0279c56902c90ed58d1444f._comment
new file mode 100644
index 000000000..4c1ad5521
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_4_eecaa7f7b0279c56902c90ed58d1444f._comment
@@ -0,0 +1,83 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2019-11-13T21:22:07Z"
+ content="""
+Simplified version of patch above, that converts ensureOnlyActionOn to not use
+STM at all, and is significantly simpler.
+
+With this patch, the test case still STM deadlocks. So this seems to be
+proof that the actual problem is not in ensureOnlyActionOn.
+
+	diff --git a/Annex.hs b/Annex.hs
+	index 9eb4c5f39..9baf7755a 100644
+	--- a/Annex.hs
+	+++ b/Annex.hs
+	@@ -143,7 +143,7 @@ data AnnexState = AnnexState
+	 	, existinghooks :: M.Map Git.Hook.Hook Bool
+	 	, desktopnotify :: DesktopNotify
+	 	, workers :: Maybe (TMVar (WorkerPool AnnexState))
+	-	, activekeys :: TVar (M.Map Key ThreadId)
+	+	, activekeys :: MVar (M.Map Key (ThreadId, MVar ()))
+	 	, activeremotes :: MVar (M.Map (Types.Remote.RemoteA Annex) Integer)
+	 	, keysdbhandle :: Maybe Keys.DbHandle
+	 	, cachedcurrentbranch :: (Maybe (Maybe Git.Branch, Maybe Adjustment))
+	@@ -154,7 +154,7 @@ data AnnexState = AnnexState
+	 newState :: GitConfig -> Git.Repo -> IO AnnexState
+	 newState c r = do
+	 	emptyactiveremotes <- newMVar M.empty
+	-	emptyactivekeys <- newTVarIO M.empty
+	+	emptyactivekeys <- newMVar M.empty
+	 	o <- newMessageState
+	 	sc <- newTMVarIO False
+	 	return $ AnnexState
+	diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
+	index 87298a95f..a8c2bd205 100644
+	--- a/CmdLine/Action.hs
+	+++ b/CmdLine/Action.hs
+	@@ -22,7 +22,7 @@ import Remote.List
+	 import Control.Concurrent
+	 import Control.Concurrent.Async
+	 import Control.Concurrent.STM
+	-import GHC.Conc
+	+import GHC.Conc (getNumProcessors)
+	 import qualified Data.Map.Strict as M
+	 import qualified System.Console.Regions as Regions
+	 
+	@@ -267,17 +267,22 @@ ensureOnlyActionOn k a = debugLocks $
+	 	go (Concurrent _) = goconcurrent
+	 	go ConcurrentPerCpu = goconcurrent
+	 	goconcurrent = do
+	-		tv <- Annex.getState Annex.activekeys
+	-		bracket (setup tv) id (const a)
+	-	setup tv = liftIO $ do
+	+		mv <- Annex.getState Annex.activekeys
+	+		bracketIO (setup mv) id (const a)
+	+	setup mv =  do
+	 		mytid <- myThreadId
+	-		atomically $ do
+	-			m <- readTVar tv
+	-			case M.lookup k m of
+	-				Just tid
+	-					| tid /= mytid -> retry
+	-					| otherwise -> return $ return ()
+	-				Nothing -> do
+	-					writeTVar tv $! M.insert k mytid m
+	-					return $ liftIO $ atomically $
+	-						modifyTVar tv $ M.delete k
+	+		m <- takeMVar mv
+	+		let ready sem = do
+	+			putMVar mv $! M.insert k (mytid, sem) m
+	+			return $ do
+	+				modifyMVar_ mv $ pure . M.delete k
+	+				putMVar sem ()
+	+		case M.lookup k m of
+	+			Nothing -> ready =<< newEmptyMVar
+	+			Just (tid, sem)
+	+				| tid /= mytid -> do
+	+					takeMVar sem
+	+					ready sem
+	+				| otherwise -> do
+	+					putMVar mv m
+	+					return noop
+"""]]
diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_5_052acf169f9c2b8b0233adddabb02559._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_5_052acf169f9c2b8b0233adddabb02559._comment
new file mode 100644
index 000000000..4d3cc32da
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_5_052acf169f9c2b8b0233adddabb02559._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2019-11-13T21:42:58Z"
+ content="""
+finishCommandActions is reaching the retry case, and STM deadlocks there.
+The WorkerPool is getting into a state where allIdle is False, and is not
+leaving it, perhaps due to an earlier STM deadlock. (There seem to be two
+different ones.)
+
+Also, I notice with --json-error-messages:
+
+	{"command":"get","note":"from origin...\nchecksum...","success":false,"key":"SHA256E-s524288--07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541","error-messages":["git-annex: thread blocked indefinitely in an STM transaction"],"file":"1"}
+
+So the thread that actually gets to run on the key is somehow reaching a
+STM deadlock.
+
+Which made me wonder if that thread deadlocks on enteringStage.
+And it seems so. If Command.Get is changed to use commandStages
+rather than transferStages, the test case succeeds.
+
+Like finishCommandActions, enteringStage has a STM retry if it needs to
+wait for something to happen to the WorkerPool. So again it looks like
+the WorkerPool is getting screwed up.
+"""]]

diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_1_10fa70e1e5e08e12ab7c1570881b38f8._comment b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_1_10fa70e1e5e08e12ab7c1570881b38f8._comment
new file mode 100644
index 000000000..d9d17fd26
--- /dev/null
+++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common/comment_1_10fa70e1e5e08e12ab7c1570881b38f8._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-13T19:58:20Z"
+ content="""
+Recent versions of git-annex have a git-lfs special
+remote, and gitlab does support git-lfs, so git-annex
+can store data on it. That does not answer your question, just wanted to
+mention it.
+
+For the assistant to quickly notice changes, it needs to be able to talk to
+a git-annex-shell on the remote's server. As a fallback, it does poll
+remotes periodically that don't have git-annex on them.  That's done once
+per hour, and also when network connection changes are detected.
+
+If you need to poll more frequently, you can install a cron job that does a
+git pull.
+"""]]

diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment
new file mode 100644
index 000000000..f2b950348
--- /dev/null
+++ b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2019-11-13T19:29:34Z"
+ content="""
+--debug might provide some clue in its http dump.
+
+The ParseError comes from attoparsec. Seems likely that aeson/aws is what's
+using it there, and that it is failing to parse something from S3.
+
+Of course, the malloc error suggests a low-level memory problem, probably
+from C code. I don't think git-annex contains anything like that, so it
+must be from a dependency.
+
+The S3 signature being wrong again points to the aws library, or something
+lower level. And then the following double free is another low-level memory
+problem.
+
+So there's a pattern, and it seems to extend across linux and OSX.
+
+Kind of wondering if something in the library stack is somehow failing to
+be concurrency safe. If two http requests end up using the same memory,
+it would kind of explain all of this.
+"""]]
diff --git a/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment b/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment
new file mode 100644
index 000000000..140dc91de
--- /dev/null
+++ b/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2019-11-13T19:37:16Z"
+ content="""
+The signal 11 is very significant. It points to a problem in a lower-level
+library (or ghc runtime), or perhaps a bad memory problem. git-annex does
+not itself contain any code that can segfault, afaik.
+
+Almost certianly the same as the other bug.
+"""]]

diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
index aa58a1df9..a1177cbf1 100644
--- a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
+++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
@@ -18,26 +18,3 @@ Thx
 Christopher
 
 
-
-
-
-Greetings,
-
-I'm setting up git-annex as the store for a darktable environment.  I have to assume that all of the backup and client machines can not necessarily see each-other.  Let's call them:
-client1
-client2
-backup
-usb (incr. backup - in case a client can't reach the gdrive)
-gitlab (upstream - do I need to group this as well - if so, what)
-gdrive (again, labeled as a backup so it will take everything)
-
-They all can reach both a gitlab repo (I know they don't support annex anymore) and a special remote for the large files (gdrive via the rclone remote to allow me to use a team drive)
-
-So either client1 or client2 will create or update content.  git-annex assistant on the client will sync that to gitlab and gdrive (I assume).
-
-The question is, will git-annex assistant on backup and the other client check to see if there have been updates to the gitlab upstream?  If so, how often?  If not, should I git-annex schedule a git-annex-sync on each client and backup to force it?  
-
-Thx
-Christopher
-
-

diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
index a1177cbf1..aa58a1df9 100644
--- a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
+++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
@@ -18,3 +18,26 @@ Thx
 Christopher
 
 
+
+
+
+Greetings,
+
+I'm setting up git-annex as the store for a darktable environment.  I have to assume that all of the backup and client machines can not necessarily see each-other.  Let's call them:
+client1
+client2
+backup
+usb (incr. backup - in case a client can't reach the gdrive)
+gitlab (upstream - do I need to group this as well - if so, what)
+gdrive (again, labeled as a backup so it will take everything)
+
+They all can reach both a gitlab repo (I know they don't support annex anymore) and a special remote for the large files (gdrive via the rclone remote to allow me to use a team drive)
+
+So either client1 or client2 will create or update content.  git-annex assistant on the client will sync that to gitlab and gdrive (I assume).
+
+The question is, will git-annex assistant on backup and the other client check to see if there have been updates to the gitlab upstream?  If so, how often?  If not, should I git-annex schedule a git-annex-sync on each client and backup to force it?  
+
+Thx
+Christopher
+
+

diff --git a/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
new file mode 100644
index 000000000..a1177cbf1
--- /dev/null
+++ b/doc/forum/git-annex_clients_with_only_a_gitLab_repo_and_GDrive_special_remote_in_common.mdwn
@@ -0,0 +1,20 @@
+Greetings,
+
+I'm setting up git-annex as the store for a darktable environment.  I have to assume that all of the backup and client machines can not necessarily see each-other.  Let's call them:
+client1
+client2
+backup
+usb (incr. backup - in case a client can't reach the gdrive)
+gitlab (upstream - do I need to group this as well - if so, what)
+gdrive (again, labeled as a backup so it will take everything)
+
+They all can reach both a gitlab repo (I know they don't support annex anymore) and a special remote for the large files (gdrive via the rclone remote to allow me to use a team drive)
+
+So either client1 or client2 will create or update content.  git-annex assistant on the client will sync that to gitlab and gdrive (I assume).
+
+The question is, will git-annex assistant on backup and the other client check to see if there have been updates to the gitlab upstream?  If so, how often?  If not, should I git-annex schedule a git-annex-sync on each client and backup to force it?  
+
+Thx
+Christopher
+
+

diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_1_46420a92dbd8655af9b16349da24d0fc._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_1_46420a92dbd8655af9b16349da24d0fc._comment
new file mode 100644
index 000000000..77b039cc6
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_1_46420a92dbd8655af9b16349da24d0fc._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2019-11-13T16:34:34Z"
+ content="""
+Reproduced.
+
+After building git-annex with the DebugLocks flag, I got this:
+
+	debugLocks, called at ./Annex/Transfer.hs:248:18 in main:Annex.Transfer
+	debugLocks, called at ./CmdLine/Action.hs:263:26 in main:CmdLine.Action
+
+Which points to pickRemote and ensureOnlyActionOn. But pickRemote
+does no STM actions when there's only 1 remote, so it must really be
+the latter.
+
+Also, I notice that when 5 files to get are provided, it crashes, but with
+less than 5, it succeeds. 
+Even this trivial case crashes: `git annex get -J1 1 2`
+"""]]
diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_2_b1f4dc12ad00a4aa73e5bdc7c0a8f489._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_2_b1f4dc12ad00a4aa73e5bdc7c0a8f489._comment
new file mode 100644
index 000000000..0721cf378
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_2_b1f4dc12ad00a4aa73e5bdc7c0a8f489._comment
@@ -0,0 +1,83 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2019-11-13T17:07:29Z"
+ content="""
+Ok, I see the bug. ensureOnlyActionOn does a STM 
+retry if it finds in the activekeys map some other thread
+is operating on the same key.
+But, there is no running STM transaction what will update
+the map. So, STM detects that the retry would deadlock.
+
+It's not really a deadlock, because once the other thread finishes,
+it will update the map to remove itself. But STM can't know that.
+The solution will be to not use STM for waiting on the other thread.
+
+Hmm, I tried the obvious approach, using a MVar semaphore to wait for the
+thread, but that just resulted in more STM and MVar deadlocks. 
+
+I don't understand why after puzzling over it for two hours. I did
+instrument all calls to atomically, and it looks, unfortunately, like
+the one in finishCommandActions is deadlocking. If the problem extends
+beyond ensureOnlyActionOn it may be much more complicated.
+
+Patch that does not work and I don't know why.
+
+	diff --git a/CmdLine/Action.hs b/CmdLine/Action.hs
+	index 87298a95f..bf4bdd589 100644
+	--- a/CmdLine/Action.hs
+	+++ b/CmdLine/Action.hs
+	@@ -268,16 +268,30 @@ ensureOnlyActionOn k a = debugLocks $
+	 	go ConcurrentPerCpu = goconcurrent
+	 	goconcurrent = do
+	 		tv <- Annex.getState Annex.activekeys
+	-		bracket (setup tv) id (const a)
+	-	setup tv = liftIO $ do
+	+		bracketIO (setup tv) id (const a)
+	+	setup tv = do
+	+		mysem <- newEmptyMVar
+	 		mytid <- myThreadId
+	-		atomically $ do
+	+		finishsetup <- atomically $ do
+	 			m <- readTVar tv
+	 			case M.lookup k m of
+	-				Just tid
+	-					| tid /= mytid -> retry
+	-					| otherwise -> return $ return ()
+	+				Just (tid, theirsem)
+	+					| tid /= mytid -> return $ do
+	+						-- wait for the other
+	+						-- thread to finish, and
+	+						-- retry (STM retry would
+	+						-- deadlock)
+	+						readMVar theirsem
+	+						setup tv
+	+					| otherwise -> return $
+	+						-- same thread, so no
+	+						-- blocking
+	+						return $ return ()
+	 				Nothing -> do
+	-					writeTVar tv $! M.insert k mytid m
+	-					return $ liftIO $ atomically $
+	-						modifyTVar tv $ M.delete k
+	+					writeTVar tv $! M.insert k (mytid, mysem) m
+	+					return $ return $ do
+	+						atomically $ modifyTVar tv $
+	+							M.delete k
+	+						-- indicate finished
+	+						putMVar mysem ()
+	+		finishsetup
+	diff --git a/Annex.hs b/Annex.hs
+	index 9eb4c5f39..936399ae7 100644
+	--- a/Annex.hs
+	+++ b/Annex.hs
+	@@ -143,7 +143,7 @@ data AnnexState = AnnexState
+	 	, existinghooks :: M.Map Git.Hook.Hook Bool
+	 	, desktopnotify :: DesktopNotify
+	 	, workers :: Maybe (TMVar (WorkerPool AnnexState))
+	-	, activekeys :: TVar (M.Map Key ThreadId)
+	+	, activekeys :: TVar (M.Map Key (ThreadId, MVar ()))
+	 	, activeremotes :: MVar (M.Map (Types.Remote.RemoteA Annex) Integer)
+	 	, keysdbhandle :: Maybe Keys.DbHandle
+	 	, cachedcurrentbranch :: (Maybe (Maybe Git.Branch, Maybe Adjustment))
+"""]]
diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_3_d8ba7cc5a860e9ccaab32c637cc2a7cd._comment b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_3_d8ba7cc5a860e9ccaab32c637cc2a7cd._comment
new file mode 100644
index 000000000..2880a4632
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction/comment_3_d8ba7cc5a860e9ccaab32c637cc2a7cd._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2019-11-13T19:07:49Z"
+ content="""
+Tried going back to c04b2af3e1a8316e7cf640046ad0aa68826650ed,
+which is before the separation of perform and cleanup stages.
+The same code was in onlyActionOn back then. And the test case does not
+crash.
+
+So, that gives a good commit to start a bisection. Which will probably
+find the bug was introduced in the separation of perform and cleanup stages,
+because that added a lot of STM complexity.
+
+(Have to cherry-pick 018b5b81736a321f3eb9762a2afb7124e19dbdf9
+onto those old commits to make them build with current libraries.)
+"""]]

diff --git a/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_5_28aabb525a1a487eaecdfe591cc7108b._comment b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_5_28aabb525a1a487eaecdfe591cc7108b._comment
new file mode 100644
index 000000000..329cd3c40
--- /dev/null
+++ b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_5_28aabb525a1a487eaecdfe591cc7108b._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 5"
+ date="2019-11-13T04:41:56Z"
+ content="""
+> If the repository being accessed over globus uses .git/annex/objects/ locations, it sounds to me like it's a git-annex repo, being accessed over a protocol other than ssh. 
+
+That is correct.
+
+> A special remote that accesses remote annex objects could be created, and --sameas used to make the special remote have the same uuid as the (remote) git-annex repo.
+
+That is correct too. The question is either it should be a dedicated `git-annex-remote-globus-gitannex` special remote which would need to probably use the same functionality of a `git-annex-remote-globus` for actual authentication and interaction with globus (with difference largely in paths to assume) or just an option to the `git-annex-remote-globus`...?
+"""]]

diff --git a/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn
new file mode 100644
index 000000000..837cfecb1
--- /dev/null
+++ b/doc/bugs/parallel_get_to_the_files_for_the_same_key_would_fail_with__thread_blocked_indefinitely_in_an_STM_transaction.mdwn
@@ -0,0 +1,71 @@
+Originally was trying to reproduce [datalad/issues/3653](https://github.com/datalad/datalad/issues/3653) assuming that multiple files pointed to the same key.
+It was not the case, and my attempt revealed another bug - annex inability to "obtain" files in parallel when multiple of them point to the same key:
+
+<details>
+<summary>setup of original repo(click to expand)</summary>
+
+[[!format sh """
+/tmp > mkdir src; (cd src; git init; git annex init; dd if=/dev/zero of=1 count=1024 bs=1024; for f in {2..10}; do cp 1 $f; done ; git annex add *; git commit -m added; )
+Initialized empty Git repository in /tmp/src/.git/
+init  (scanning for unlocked files...)
+ok
+(recording state in git...)
+1024+0 records in
+1024+0 records out
+1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00106651 s, 983 MB/s
+add 1 
+ok                                
+add 10 
+ok                                
+add 2 
+ok                                
+add 3 
+ok                                
+add 4 
+ok                                
+add 5 
+ok                                
+add 6 
+ok                                
+add 7 
+ok                                
+add 8 
+ok                                
+add 9 
+ok                                
+(recording state in git...)
+[master (root-commit) 63b1163] added
+ 10 files changed, 10 insertions(+)
+ create mode 120000 1
+ create mode 120000 10
+ create mode 120000 2
+ create mode 120000 3
+ create mode 120000 4
+ create mode 120000 5
+ create mode 120000 6
+ create mode 120000 7
+ create mode 120000 8
+ create mode 120000 9
+"""]]
+</details>
+
+And that is what happens then when we try to get the same key in parallel:
+[[!format sh """
+
+/tmp > git clone src dst; (cd dst; git annex get -J 5 *; )
+Cloning into 'dst'...
+done.
+(merging origin/git-annex into git-annex...)
+(recording state in git...)
+(scanning for unlocked files...)
+get 2 (from origin...) (checksum...) 
+git-annex: thread blocked indefinitely in an STM transaction
+failed
+git-annex: thread blocked indefinitely in an MVar operation
+
+"""]]
+
+I felt like it is an old issue but failed to find a trace of it upon a quick lookup
+
+[[!meta author=yoh]]
+[[!tag projects/datalad]]

diff --git a/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_4_3bea0473d4805b1ef56b955ba30166e8._comment b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_4_3bea0473d4805b1ef56b955ba30166e8._comment
new file mode 100644
index 000000000..63877c943
--- /dev/null
+++ b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_4_3bea0473d4805b1ef56b955ba30166e8._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2019-11-12T20:34:23Z"
+ content="""
+If the repository being accessed over globus uses .git/annex/objects/ 
+locations, it sounds to me like it's a git-annex repo, being accessed over
+a protocol other than ssh. A special remote that accesses remote annex
+objects could be created, and --sameas used to make the special remote have
+the same uuid as the (remote) git-annex repo.
+"""]]

diff --git a/doc/bugs/cygwin.mdwn b/doc/bugs/cygwin.mdwn
new file mode 100644
index 000000000..4d0bf07a6
--- /dev/null
+++ b/doc/bugs/cygwin.mdwn
@@ -0,0 +1,406 @@
+Cygwin do not work with git-annex windows installed version
+
+
+### What steps will reproduce the problem?
+* Install git-annex windows version
+* Try run git annex test under cygwin, and got 65 test failed out of 101.
+* Try run git annex test under git bash windows and got 101 test passed.
+* NOTE: git-lfs windows installed version working fine under cygwin and git bash windows.
+
+
+### What version of git-annex are you using? On what operating system?
+git-annex version: 7.20191106-ge486fd5e0
+build flags: Assistant Webapp Pairing S3 WebDAV TorrentParser Feeds Testsuite
+dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
+operating system: mingw32 x86_64
+supported repository versions: 7
+upgrade supported from repository versions: 2 3 4 5 6
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+Cygwin ~/git..t_tools/wget/cache (test)
+(506)$ git-annex test
+Tests
+  QuickCheck
+    prop_encode_decode_roundtrip:                         OK (0.04s)
+      +++ OK, passed 1000 tests.
+    prop_encode_c_decode_c_roundtrip:                     OK (0.03s)
+      +++ OK, passed 1000 tests.
+    prop_isomorphic_key_encode:                           OK
+      +++ OK, passed 1000 tests.
+    prop_isomorphic_shellEscape:                          OK (0.02s)
+      +++ OK, passed 1000 tests.
+    prop_isomorphic_shellEscape_multiword:                OK (0.70s)
+      +++ OK, passed 1000 tests.
+    prop_isomorphic_configEscape:                         OK (0.02s)
+      +++ OK, passed 1000 tests.
+    prop_parse_show_Config:                               OK (0.04s)
+      +++ OK, passed 1000 tests.
+    prop_upFrom_basics:                                   OK (0.02s)
+      +++ OK, passed 1000 tests.
+    prop_relPathDirToFile_basics:                         OK (0.03s)
+      +++ OK, passed 1000 tests.
+    prop_relPathDirToFile_regressionTest:                 OK
+      +++ OK, passed 1 test.
+    prop_cost_sane:                                       OK
+      +++ OK, passed 1 test.
+    prop_matcher_sane:                                    OK
+      +++ OK, passed 1 test.
+    prop_HmacSha1WithCipher_sane:                         OK
+      +++ OK, passed 1 test.
+    prop_VectorClock_sane:                                OK
+      +++ OK, passed 1 test.
+    prop_addMapLog_sane:                                  OK
+      +++ OK, passed 1 test.
+    prop_verifiable_sane:                                 OK (0.07s)
+      +++ OK, passed 1000 tests.
+    prop_segment_regressionTest:                          OK
+      +++ OK, passed 1 test.
+    prop_read_write_transferinfo:                         OK (0.04s)
+      +++ OK, passed 1000 tests.
+    prop_read_show_inodecache:                            OK (0.02s)
+      +++ OK, passed 1000 tests.
+    prop_parse_build_presence_log:                        OK (1.27s)
+      +++ OK, passed 1000 tests.
+    prop_parse_build_contentidentifier_log:               OK (1.23s)
+      +++ OK, passed 1000 tests.
+    prop_read_show_TrustLevel:                            OK
+      +++ OK, passed 1 test.
+    prop_parse_build_TrustLevelLog:                       OK
+      +++ OK, passed 1 test.
+    prop_hashes_stable:                                   OK
+      +++ OK, passed 1 test.
+    prop_mac_stable:                                      OK
+      +++ OK, passed 1 test.
+    prop_schedule_roundtrips:                             OK (0.01s)
+      +++ OK, passed 1000 tests.
+    prop_past_sane:                                       OK
+      +++ OK, passed 1 test.
+    prop_duration_roundtrips:                             OK
+      +++ OK, passed 1000 tests.
+    prop_metadata_sane:                                   OK (0.86s)
+      +++ OK, passed 1000 tests.
+    prop_metadata_serialize:                              OK (0.84s)
+      +++ OK, passed 1000 tests.
+    prop_branchView_legal:                                OK (0.77s)
+      +++ OK, passed 1000 tests.
+    prop_viewPath_roundtrips:                             OK (0.03s)
+      +++ OK, passed 1000 tests.
+    prop_view_roundtrips:                                 OK (0.52s)
+      +++ OK, passed 1000 tests.
+    prop_viewedFile_rountrips:                            OK (0.02s)
+      +++ OK, passed 1000 tests.
+    prop_b64_roundtrips:                                  OK
+      +++ OK, passed 1000 tests.
+    prop_standardGroups_parse:                            OK
+      +++ OK, passed 1 test.
+  Unit Tests v7 adjusted unlocked branch
+    add dup:                                              Init Tests
+  init: init test repo
+  Detected a filesystem without fifo support.
+
+  Disabling ssh connection caching.
+
+  Detected a crippled filesystem.
+
+  Disabling core.symlinks.
+(scanning for unlocked files...)
+
+  Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
+ not found .
+git-annex.exe: pre-commit: 1 failed
+
+  Failed to enter adjusted branch!
+ok
+(recording state in git...)
+ not found .
+git-annex.exe: pre-commit: 1 failed
+FAIL (6.92s)
+    .\\Test\\Framework.hs:469:
+    git commit failed
+  add:  add foo
+ok
+(recording state in git...)
+add sha1foo
+ok
+(recording state in git...)
+ not found .
+git-annex.exe: pre-commit: 1 failed
+FAIL (8.10s)
+    Test.hs:303:
+    git commit failed
+
+2 out of 2 tests failed (15.02s)
+FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    add extras:                                           FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    export_import:                                        FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    export_import_subdir:                                 FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    shared clone:                                         FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    log:                                                  FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    import:                                               FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    reinject:                                             FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    unannex (no copy):                                    FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    unannex (with copy):                                  FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    drop (no remote):                                     FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    drop (with remote):                                   FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    drop (untrusted remote):                              FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework
+    get:                                                  FAIL
+      Exception: init tests failed! cannot continue
+      CallStack (from HasCallStack):
+        error, called at .\\Test\\Framework.hs:427:33 in main:Test.Framework

(Diff truncated)

diff --git a/doc/security/comment_3_0cc44e032fc94f4d374fb297ca9a8f2d._comment b/doc/security/comment_3_0cc44e032fc94f4d374fb297ca9a8f2d._comment
new file mode 100644
index 000000000..c94ab3895
--- /dev/null
+++ b/doc/security/comment_3_0cc44e032fc94f4d374fb297ca9a8f2d._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="Another use case -- http_proxy"
+ date="2019-11-12T17:51:25Z"
+ content="""
+On some institutional servers they mandate for all http traffic to go through proxy.  In our case `http_proxy` looked like `http://ptn07-e0:3128`. 
+`datalad install` worked out but an attempt to `datalad get` a bunch of files resulted in massive list of errors, and `annex-ignore` being set for \"origin\" which is otherwise available.
+We managed to fish out that warning about security schemes and http_proxy being ignored for that reason in one of the subsequent attempts (after unsetting annex-ignore).
+Upon attempts to set that variable we realized that we had to provide IP  instead of `http_proxy` full value or just a name (`ptn07-e0`), netmasked address (e.g. 10.0.0.1/24) also didn't work.  That makes it really inflexible. Actual IP could change, without http_proxy being changed.  Even the value of http_proxy could change system wide.  It would be painful to require our users to adjust for that every time, and it is infeasible to demand sysadmins to somehow tune up their configuration across HPC (we have no direct connection to them).  If we could at least whitelist private network -- that would provide some remedy.  Regular expression, though indeed not a really security-friendly solution, could have also provided remedy.
+Have we missed some already existing way to make our lives easy on that system?
+"""]]

diff --git a/Annex/Content.hs b/Annex/Content.hs
index 3b41784a5..43fc3238c 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -776,7 +776,7 @@ downloadUrl k p urls file =
 	-- download command is used.
 	meteredFile file (Just p) k $
 		Url.withUrlOptions $ \uo -> 
-			liftIO $ anyM (\u -> Url.download p u file uo) urls
+			anyM (\u -> Url.download p u file uo) urls
 
 {- Copies a key's content, when present, to a temp file.
  - This is used to speed up some rsyncs. -}
diff --git a/Annex/Url.hs b/Annex/Url.hs
index b1f970a6a..bcc6a747f 100644
--- a/Annex/Url.hs
+++ b/Annex/Url.hs
@@ -1,24 +1,39 @@
 {- Url downloading, with git-annex user agent and configured http
  - headers, security restrictions, etc.
  -
- - Copyright 2013-2018 Joey Hess <id@joeyh.name>
+ - Copyright 2013-2019 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
 
 module Annex.Url (
-	module U,
 	withUrlOptions,
 	getUrlOptions,
 	getUserAgent,
 	ipAddressesUnlimited,
+	checkBoth,
+	download,
+	exists,
+	getUrlInfo,
+	U.downloadQuiet,
+	U.URLString,
+	U.UrlOptions(..),
+	U.UrlInfo(..),
+	U.sinkResponseFile,
+	U.matchStatusCodeException,
+	U.downloadConduit,
+	U.downloadPartial,
+	U.parseURIRelaxed,
+	U.allowedScheme,
+	U.assumeUrlExists,
 ) where
 
 import Annex.Common
 import qualified Annex
-import Utility.Url as U
+import qualified Utility.Url as U
 import Utility.IPAddress
 import Utility.HttpManagerRestricted
+import Utility.Metered
 import qualified BuildInfo
 
 import Network.Socket
@@ -43,7 +58,7 @@ getUrlOptions = Annex.getState Annex.urloptions >>= \case
   where
 	mk = do
 		(urldownloader, manager) <- checkallowedaddr
-		mkUrlOptions
+		U.mkUrlOptions
 			<$> (Just <$> getUserAgent)
 			<*> headers
 			<*> pure urldownloader
@@ -108,3 +123,27 @@ ipAddressesUnlimited =
 
 withUrlOptions :: (U.UrlOptions -> Annex a) -> Annex a
 withUrlOptions a = a =<< getUrlOptions
+
+checkBoth :: U.URLString -> Maybe Integer -> U.UrlOptions -> Annex Bool
+checkBoth url expected_size uo =
+	liftIO (U.checkBoth url expected_size uo) >>= \case
+		Right r -> return r
+		Left err -> warning err >> return False
+
+download :: MeterUpdate -> U.URLString -> FilePath -> U.UrlOptions -> Annex Bool
+download meterupdate url file uo =
+	liftIO (U.download meterupdate url file uo) >>= \case
+		Right () -> return True
+		Left err -> warning err >> return False
+
+exists :: U.URLString -> U.UrlOptions -> Annex Bool
+exists url uo = liftIO (U.exists url uo) >>= \case
+	Right b -> return b
+	Left err -> warning err >> return False
+
+getUrlInfo :: U.URLString -> U.UrlOptions -> Annex U.UrlInfo
+getUrlInfo url uo = liftIO (U.getUrlInfo url uo) >>= \case
+	Right i -> return i
+	Left err -> do
+		warning err
+		return $ U.UrlInfo False Nothing Nothing
diff --git a/Annex/YoutubeDl.hs b/Annex/YoutubeDl.hs
index 9855a391a..64ca3fbf4 100644
--- a/Annex/YoutubeDl.hs
+++ b/Annex/YoutubeDl.hs
@@ -18,7 +18,6 @@ import Annex.Common
 import qualified Annex
 import Annex.Content
 import Annex.Url
-import Utility.Url (URLString)
 import Utility.DiskFree
 import Utility.HtmlDetect
 import Utility.Process.Transcript
diff --git a/Assistant/Restart.hs b/Assistant/Restart.hs
index 1660c1317..ef8477ead 100644
--- a/Assistant/Restart.hs
+++ b/Assistant/Restart.hs
@@ -95,7 +95,9 @@ newAssistantUrl repo = do
  - warp-tls listens to http, in order to show an error page, so this works.
  -}
 assistantListening :: URLString -> IO Bool
-assistantListening url = catchBoolIO $ exists url' =<< defUrlOptions
+assistantListening url = catchBoolIO $ do
+	uo <- defUrlOptions
+	(== Right True) <$> exists url' uo
   where
 	url' = case parseURI url of
 		Nothing -> url
diff --git a/Assistant/Upgrade.hs b/Assistant/Upgrade.hs
index a8920bb9c..e46ac86ce 100644
--- a/Assistant/Upgrade.hs
+++ b/Assistant/Upgrade.hs
@@ -40,9 +40,10 @@ import Utility.Metered
 import qualified Utility.Lsof as Lsof
 import qualified BuildInfo
 import qualified Utility.Url as Url
-import qualified Annex.Url as Url
+import qualified Annex.Url as Url hiding (download)
 import Utility.Tuple
 
+import Data.Either
 import qualified Data.Map as M
 
 {- Upgrade without interaction in the webapp. -}
@@ -323,8 +324,8 @@ downloadDistributionInfo = do
 	liftIO $ withTmpDir "git-annex.tmp" $ \tmpdir -> do
 		let infof = tmpdir </> "info"
 		let sigf = infof ++ ".sig"
-		ifM (Url.download nullMeterUpdate distributionInfoUrl infof uo
-			<&&> Url.download nullMeterUpdate distributionInfoSigUrl sigf uo
+		ifM (isRight <$> Url.download nullMeterUpdate distributionInfoUrl infof uo
+			<&&> (isRight <$> Url.download nullMeterUpdate distributionInfoSigUrl sigf uo)
 			<&&> verifyDistributionSig gpgcmd sigf)
 			( parseInfoFile <$> readFileStrict infof
 			, return Nothing
diff --git a/Assistant/WebApp/Configurators/IA.hs b/Assistant/WebApp/Configurators/IA.hs
index 84d609761..04feb965b 100644
--- a/Assistant/WebApp/Configurators/IA.hs
+++ b/Assistant/WebApp/Configurators/IA.hs
@@ -192,7 +192,7 @@ escapeHeader = escapeURIString (\c -> isUnescapedInURI c && c /= ' ')
 getRepoInfo :: RemoteConfig -> Widget
 getRepoInfo c = do
 	uo <- liftAnnex Url.getUrlOptions
-	exists <- liftIO $ catchDefaultIO False $ Url.exists url uo
+	exists <- liftAnnex $ catchDefaultIO False $ Url.exists url uo
 	[whamlet|
 <a href="#{url}">
   Internet Archive item
diff --git a/CHANGELOG b/CHANGELOG
index f59d029ee..03d8e7b11 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,6 +1,8 @@
 git-annex (7.20191107) UNRELEASED; urgency=medium
 
   * Added annex.allowsign option.
+  * Make --json-error-messages capture more errors,
+    particularly url download errors.
 
  -- Joey Hess <id@joeyh.name>  Mon, 11 Nov 2019 15:59:47 -0400
 
diff --git a/Command/AddUrl.hs b/Command/AddUrl.hs
index 2c363148a..aafa76491 100644
--- a/Command/AddUrl.hs
+++ b/Command/AddUrl.hs
@@ -197,8 +197,7 @@ startWeb o urlstring = go $ fromMaybe bad $ parseURI urlstring
 		pathmax <- liftIO $ fileNameLengthLimit "."
 		urlinfo <- if relaxedOption (downloadOptions o)
 			then pure Url.assumeUrlExists
-			else Url.withUrlOptions $
-				liftIO . Url.getUrlInfo urlstring
+			else Url.withUrlOptions $ Url.getUrlInfo urlstring
 		file <- adjustFile o <$> case fileOption (downloadOptions o) of
 			Just f -> pure f
 			Nothing -> case Url.urlSuggestedFile urlinfo of
diff --git a/Command/ImportFeed.hs b/Command/ImportFeed.hs
index 6eda7b84b..2eca65864 100644
--- a/Command/ImportFeed.hs
+++ b/Command/ImportFeed.hs
@@ -146,13 +146,12 @@ findDownloads u f = catMaybes $ map mk (feedItems f)
 downloadFeed :: URLString -> Annex (Maybe String)
 downloadFeed url
 	| Url.parseURIRelaxed url == Nothing = giveup "invalid feed url"
-	| otherwise = Url.withUrlOptions $ \uo ->

(Diff truncated)

diff --git a/Annex/AdjustedBranch.hs b/Annex/AdjustedBranch.hs
index 131f1dd70..bd4341163 100644
--- a/Annex/AdjustedBranch.hs
+++ b/Annex/AdjustedBranch.hs
@@ -59,6 +59,7 @@ import Utility.Tmp.Dir
 import Utility.CopyFile
 import qualified Database.Keys
 import Config
+import Config.CommitMode
 
 import qualified Data.Map as M
 
@@ -224,8 +225,9 @@ adjustToCrippledFileSystem :: Annex ()
 adjustToCrippledFileSystem = do
 	warning "Entering an adjusted branch where files are unlocked as this filesystem does not support locked files."
 	checkVersionSupported
-	whenM (isNothing <$> inRepo Git.Branch.current) $
-		void $ inRepo $ Git.Branch.commitCommand Git.Branch.AutomaticCommit
+	whenM (isNothing <$> inRepo Git.Branch.current) $ do
+		cmode <-  implicitCommitMode
+		void $ inRepo $ Git.Branch.commitCommand cmode
 			[ Param "--quiet"
 			, Param "--allow-empty"
 			, Param "-m"
@@ -310,12 +312,16 @@ commitAdjustedTree' :: Sha -> BasisBranch -> [Ref] -> Annex Sha
 commitAdjustedTree' treesha (BasisBranch basis) parents =
 	go =<< catCommit basis
   where
-	go Nothing = inRepo mkcommit
-	go (Just basiscommit) = inRepo $ commitWithMetaData
-		(commitAuthorMetaData basiscommit)
-		(commitCommitterMetaData basiscommit)
-		mkcommit
-	mkcommit = Git.Branch.commitTree Git.Branch.AutomaticCommit
+	go Nothing = do
+		cmode <- implicitCommitMode
+		inRepo $ mkcommit cmode
+	go (Just basiscommit) = do
+		cmode <- implicitCommitMode
+		inRepo $ commitWithMetaData
+			(commitAuthorMetaData basiscommit)
+			(commitCommitterMetaData basiscommit)
+			(mkcommit cmode)
+	mkcommit cmode = Git.Branch.commitTree cmode
 		adjustedBranchCommitMessage parents treesha
 
 {- This message should never be changed. -}
@@ -444,7 +450,8 @@ mergeToAdjustedBranch tomerge (origbranch, adj) mergeconfig canresolvemerge comm
 	reparent adjtree adjmergecommit (Just currentcommit) = do
 		if (commitTree currentcommit /= adjtree)
 			then do
-				c <- inRepo $ Git.Branch.commitTree Git.Branch.AutomaticCommit
+				cmode <- implicitCommitMode
+				c <- inRepo $ Git.Branch.commitTree cmode
 					("Merged " ++ fromRef tomerge) [adjmergecommit]
 					(commitTree currentcommit)
 				inRepo $ Git.Branch.update "updating adjusted branch" currbranch c
@@ -534,12 +541,14 @@ reverseAdjustedCommit commitparent adj (csha, basiscommit) origbranch
 	| length (commitParent basiscommit) > 1 = return $
 		Left $ "unable to propigate merge commit " ++ show csha ++ " back to " ++ show origbranch
 	| otherwise = do
+		cmode <- implicitCommitMode
 		treesha <- reverseAdjustedTree commitparent adj csha
 		revadjcommit <- inRepo $ commitWithMetaData
 			(commitAuthorMetaData basiscommit)
 			(commitCommitterMetaData basiscommit) $
-				Git.Branch.commitTree Git.Branch.AutomaticCommit
-					(commitMessage basiscommit) [commitparent] treesha
+				Git.Branch.commitTree cmode
+					(commitMessage basiscommit)
+					[commitparent] treesha
 		return (Right revadjcommit)
 
 {- Adjusts the tree of the basis, changing only the files that the
diff --git a/Annex/Branch.hs b/Annex/Branch.hs
index b033c059c..d520b0bff 100644
--- a/Annex/Branch.hs
+++ b/Annex/Branch.hs
@@ -70,6 +70,7 @@ import Annex.Branch.Transitions
 import qualified Annex
 import Annex.Hook
 import Utility.Directory.Stream
+import Config.CommitMode
 
 {- Name of the branch that is used to store git-annex's information. -}
 name :: Git.Ref
@@ -109,8 +110,9 @@ getBranch = maybe (hasOrigin >>= go >>= use) return =<< branchsha
 			[Param "branch", Param $ fromRef name, Param $ fromRef originname]
 		fromMaybe (error $ "failed to create " ++ fromRef name)
 			<$> branchsha
-	go False = withIndex' True $
-		inRepo $ Git.Branch.commitAlways Git.Branch.AutomaticCommit "branch created" fullname []
+	go False = withIndex' True $ do
+		cmode <- implicitCommitMode
+		inRepo $ Git.Branch.commitAlways cmode "branch created" fullname []
 	use sha = do
 		setIndexSha sha
 		return sha
@@ -317,7 +319,8 @@ commitIndex jl branchref message parents = do
 commitIndex' :: JournalLocked -> Git.Ref -> String -> String -> Integer -> [Git.Ref] -> Annex ()
 commitIndex' jl branchref message basemessage retrynum parents = do
 	updateIndex jl branchref
-	committedref <- inRepo $ Git.Branch.commitAlways Git.Branch.AutomaticCommit message fullname parents
+	cmode <- implicitCommitMode
+	committedref <- inRepo $ Git.Branch.commitAlways cmode message fullname parents
 	setIndexSha committedref
 	parentrefs <- commitparents <$> catObject committedref
 	when (racedetected branchref parentrefs) $
@@ -551,7 +554,8 @@ performTransitionsLocked jl ts neednewlocalbranch transitionedrefs = do
 		Annex.Queue.flush
 		if neednewlocalbranch
 			then do
-				committedref <- inRepo $ Git.Branch.commitAlways Git.Branch.AutomaticCommit message fullname transitionedrefs
+				cmode <- implicitCommitMode
+				committedref <- inRepo $ Git.Branch.commitAlways cmode message fullname transitionedrefs
 				setIndexSha committedref
 			else do
 				ref <- getBranch
@@ -657,9 +661,10 @@ rememberTreeish treeish graftpoint = lockJournal $ \jl -> do
 	origtree <- fromMaybe (giveup "unable to determine git-annex branch tree") <$>
 		inRepo (Git.Ref.tree branchref)
 	addedt <- inRepo $ Git.Tree.graftTree treeish graftpoint origtree
-	c <- inRepo $ Git.Branch.commitTree Git.Branch.AutomaticCommit
+	cmode <- implicitCommitMode
+	c <- inRepo $ Git.Branch.commitTree cmode
 		"graft" [branchref] addedt
-	c' <- inRepo $ Git.Branch.commitTree Git.Branch.AutomaticCommit
+	c' <- inRepo $ Git.Branch.commitTree cmode
 		"graft cleanup" [c] origtree
 	inRepo $ Git.Branch.update' fullname c'
 	-- The tree in c' is the same as the tree in branchref,
diff --git a/Annex/RemoteTrackingBranch.hs b/Annex/RemoteTrackingBranch.hs
index 02f0bb01b..6c1d642f5 100644
--- a/Annex/RemoteTrackingBranch.hs
+++ b/Annex/RemoteTrackingBranch.hs
@@ -22,6 +22,7 @@ import qualified Git.Ref
 import qualified Git.Branch
 import Git.History
 import qualified Types.Remote as Remote
+import Config.CommitMode
 
 import qualified Data.Set as S
 
@@ -72,9 +73,10 @@ makeRemoteTrackingBranchMergeCommit tb commitsha =
 				_ -> return commitsha
 
 makeRemoteTrackingBranchMergeCommit' :: Sha -> Sha -> Sha -> Annex Sha
-makeRemoteTrackingBranchMergeCommit' commitsha importedhistory treesha =
+makeRemoteTrackingBranchMergeCommit' commitsha importedhistory treesha = do
+	cmode <- implicitCommitMode
 	inRepo $ Git.Branch.commitTree
-			Git.Branch.AutomaticCommit
+			cmode
 			"remote tracking branch"
 			[commitsha, importedhistory]
 			treesha
diff --git a/Annex/View.hs b/Annex/View.hs
index a136b6b84..da72436a5 100644
--- a/Annex/View.hs
+++ b/Annex/View.hs
@@ -30,6 +30,7 @@ import Logs.View
 import Utility.Glob
 import Types.Command
 import CmdLine.Action
+import Config.CommitMode
 
 import qualified Data.Text as T
 import qualified Data.ByteString as B
@@ -418,7 +419,8 @@ withViewIndex a = do
 genViewBranch :: View -> Annex Git.Branch
 genViewBranch view = withViewIndex $ do
 	let branch = branchView view
-	void $ inRepo $ Git.Branch.commit Git.Branch.AutomaticCommit True (fromRef branch) branch []
+	cmode <- implicitCommitMode
+	void $ inRepo $ Git.Branch.commit cmode True (fromRef branch) branch []
 	return branch
 
 withCurrentView :: (View -> Annex a) -> Annex a
diff --git a/Assistant/MakeRepo.hs b/Assistant/MakeRepo.hs
index 372f21663..fd768ee9f 100644
--- a/Assistant/MakeRepo.hs
+++ b/Assistant/MakeRepo.hs
@@ -22,6 +22,7 @@ import Logs.PreferredContent
 import qualified Annex.Branch
 import Utility.Process.Transcript
 import Config
+import Config.CommitMode
 
 {- Makes a new git repository. Or, if a git repository already
  - exists, returns False. -}
@@ -53,8 +54,9 @@ initRepo True primary_assistant_repo dir desc mgroup = inDir dir $ do
 	initRepo' desc mgroup
 	{- Initialize the master branch, so things that expect
 	 - to have it will work, before any files are added. -}
-	unlessM (Git.Config.isBare <$> gitRepo) $
-		void $ inRepo $ Git.Branch.commitCommand Git.Branch.AutomaticCommit
+	unlessM (Git.Config.isBare <$> gitRepo) $ do
+		cmode <- implicitCommitMode
+		void $ inRepo $ Git.Branch.commitCommand cmode
 			[ Param "--quiet"

(Diff truncated)

diff --git a/doc/git-annex-addurl.mdwn b/doc/git-annex-addurl.mdwn
index ce7ad04eb..224d23078 100644
--- a/doc/git-annex-addurl.mdwn
+++ b/doc/git-annex-addurl.mdwn
@@ -16,8 +16,10 @@ embedded in  a web page at the url, and that is added to the annex instead.
 See the documentation of annex.security.allowed-ip-addresses
 in [[git-annex]](1) for details.)
 
-Urls to torrent files (including magnet links) will cause the content of
-the torrent to be downloaded, using `aria2c`.
+Special remotes can add other special handling of particular urls. For
+example, the bittorrent special remotes makes urls to torrent files
+(including magnet links) download the content of the torrent,
+using `aria2c`.
 
 Normally the filename is based on the full url, so will look like
 "www.example.com_dir_subdir_bigfile". In some cases, addurl is able to

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index c27239d8b..b841f079d 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -164,13 +164,15 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
   * `END`  
     Indicates the end of a block of responses.
 * `LOCATION Name`  
-  Comes before each of the following requests, 
+  Comes before each of the following requests (except for
+  REMOVEEXPORTDIRECTORYWHENEMPTY), 
   specifying the name of the file on the remote. It will be in the
   form of a relative path, and may contain path separators,
   whitespace, and other special characters.  
   No response is made to this message.
 * `EXPECTED ContentIdentifier`  
-  Comes before each of the following requests, specifying the
+  Comes before each of the following requests (except
+  for REMOVEEXPORTDIRECTORYWHENEMPTY), specifying the
   ContentIdentifier that is expected to be present on the remote.
 * `NOTHINGEXPECTED`  
   If no ContentIdentifier is expected to be present, this is sent
@@ -224,7 +226,9 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
     Indicates the content has been removed from the remote. May be returned when
     the content was already not present.
   * `REMOVE-FAILURE Key ErrorMsg`  
-    Indicates that the content was unable to be removed from the remote.
+    Indicates that the content was unable to be removed from the remote,
+    either because of an access problem, or because it did not match
+    the ContentIdentifier.
 * `REMOVEEXPORTDIRECTORYWHENEMPTY Directory`  
   Requests the remote remove an exported directory, so long as it's empty.  
   If the remote does not use directories, or `REMOVEEXPORTEXPECTED`
@@ -294,11 +298,9 @@ of the older versions.)
 Next git-annex asks for the content of a file to be retrieved.
 
 	LOCATION foo
-	EXPECTED 100 48511528411921470	
+	EXPECTED 100 48511528411921470
 	RETRIEVEEXPORTEXPECTED tmpfile
 
-But, the file "foo" has been modified and so the
-ContentIdentifier no longer matches.
+If the requested version no longer exists, the response should be:
 
 	RETRIEVE-FAILURE content has changed
-

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index eac26458d..c27239d8b 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -179,7 +179,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
   Retrieves the content of a file from the special remote
   to the File on local disk. Must take care to only retrieve
   content that has the ContentIdentifier specified by 
-  by `EXPECTED`.  
+  `EXPECTED`.  
   While the transfer is running, the remote can send any number of
   `PROGRESS` messages. Once the transfer is complete, it finishes by
   sending one of these replies:

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index a93e707c7..eac26458d 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -148,7 +148,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
   remote. A block of responses
   can be made to this, which must always end with `END`.
   * `CONTENT Size Name`  
-    An file stored in the special remote. The Size is its size
+    A file stored in the special remote. The Size is its size
     in bytes. The Name is the name of the file on the remote,
     in the form of a relative path, and may contain path separators,
     whitespace, and other special characters.  

diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 222308796..a93e707c7 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -118,7 +118,7 @@ file. This is called a ContentIdentifier. A good ContentIdentifier needs to:
 
 It's up to the implementor of a external special remote program what
 to use for their ContentIdentifier, but not meeting those criteria
-will leas to unhappy users, and it's better not to implement this interface if
+will lead to unhappy users, and it's better not to implement this interface if
 you can't do it well.
 
 ### protocol messages

diff --git a/doc/todo/documenting_sqlite_database_schemas.mdwn b/doc/todo/documenting_sqlite_database_schemas.mdwn
new file mode 100644
index 000000000..5a67c1921
--- /dev/null
+++ b/doc/todo/documenting_sqlite_database_schemas.mdwn
@@ -0,0 +1,3 @@
+If a spec of the [[sqlite database schemas|todo/sqlite_database_improvements]] could be added to the [[internals]] docs, this would open some possibilities for third-party tools based on this info.  E.g. one could write some sqlite3 queries to get aggregate info on the number (and total size?) of keys present in specific combinations of repos.  It would of course be understood that this is internal info subject to frequent change.
+
+Also, if [[Sometimes the databases are used for data that has not yet been committed to git|devblog/day_607__v8_is_done]], this would improve [[future_proofing]].

diff --git a/doc/devblog/day_607__v8_is_done/comment_3_7d3cad862be54dc5a39059f8b82834a6._comment b/doc/devblog/day_607__v8_is_done/comment_3_7d3cad862be54dc5a39059f8b82834a6._comment
new file mode 100644
index 000000000..0231d92ef
--- /dev/null
+++ b/doc/devblog/day_607__v8_is_done/comment_3_7d3cad862be54dc5a39059f8b82834a6._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="Re: Symlinks for not-present unlocked files in v8"
+ date="2019-11-10T23:03:42Z"
+ content="""
+Adding symlinks for not-present unlocked files in v8 would be great, but if it would hold up the release with SQL db fixes, making that release first would be better.  Some query operations -- like finding all unlocked files, or all files present in one remote but not the other -- currently take a while on large repos; if the SQL refactor could solve that, that by itself would be a major step forward.
+"""]]

diff --git a/doc/devblog/day_607__v8_is_done/comment_2_802311f91808a13473aa99578a429497._comment b/doc/devblog/day_607__v8_is_done/comment_2_802311f91808a13473aa99578a429497._comment
new file mode 100644
index 000000000..6da2674ba
--- /dev/null
+++ b/doc/devblog/day_607__v8_is_done/comment_2_802311f91808a13473aa99578a429497._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="gb@4a49bb1afcf3d183bba8f07297b0395808768c6c"
+ nickname="gb"
+ avatar="http://cdn.libravatar.org/avatar/51ed40932fbf8748c70b31fd7446d40e"
+ subject="Symlinks for not-present unlocked files in v8?"
+ date="2019-11-10T14:51:59Z"
+ content="""
+Hi Joey, do you think it would be possible to squeeze a solution for the missing symlinks for not-present unlocked files in v8?
+
+Symlinks for missing unlocked files are the last thing missing from V5 for those of us simulating direct mode with adjusted branches and `annex.thin`. For reference <https://git-annex.branchable.com/devblog/day_601__v7_default/> and <https://git-annex.branchable.com/todo/symlinks_for_not-present_unlocked_files/>
+
+"""]]

diff --git a/doc/todo/split_off_clean__47__smudge_filter__63__/comment_7_9817fd2cbb46c97124cd35a7a2a9c4bd._comment b/doc/todo/split_off_clean__47__smudge_filter__63__/comment_7_9817fd2cbb46c97124cd35a7a2a9c4bd._comment
new file mode 100644
index 000000000..1b05c62ba
--- /dev/null
+++ b/doc/todo/split_off_clean__47__smudge_filter__63__/comment_7_9817fd2cbb46c97124cd35a7a2a9c4bd._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="limiting clean/smudge filter to unlocked files"
+ date="2019-11-08T19:48:11Z"
+ content="""
+\"Each new invokation of git-annex has to re-open databases, start up git cat-file to query from, link the executable, read git config, etc. That takes a few hundred milliseconds.\" -- this is somewhat more of an issue now that all `git add/checkout` operations call the clean/smudge filter, even when there are no unlocked files.  One option is to [[only configure the filters for unlocked files|todo/only_pass_unlocked_files_through_the_clean__47__smudge_filter]] when only a few files are unlocked.
+"""]]

diff --git a/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_3_43107f1bec1ad141af2e97f715a70fe9._comment b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_3_43107f1bec1ad141af2e97f715a70fe9._comment
new file mode 100644
index 000000000..a9b7c3ebb
--- /dev/null
+++ b/doc/todo/globus_special_remote_as_a___34__transport__34___layer/comment_3_43107f1bec1ad141af2e97f715a70fe9._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="RFC: how would it work for regular git remote + special remote"
+ date="2019-11-08T19:19:18Z"
+ content="""
+Added recently `--sameas` functionality provides support at the \"UUID logistics\" level, and examples in the comments exercise it for two external remotes (rsync + directory) with the same layout of annex objects.
+The original use case I am pursuing is for a regular git repository (e.g. non-bare) with \"git repository\" layout of the store (i.e. under `.git/annex/objects/`) use a special remote primarily as a transport mechanism. In our case it will be `globus`.  I really doubt it would work \"out of the box\" since AFAIK any special remote has only two possible ideas about layout of objects: its regular \"special remote layout\" (e.g. a flat list of keys or with some hash directories) or exported (such as a file tree). Only in case of `git` special remote layout would be the same, but otherwise special remote layout would be different, and \"export\" wouldn't really be the one desired (especially for placing files to the remote).
+So it seems that the only way to accomplish my mission would be to implement in the `globus` custom special remote the support of additional layout by parametrizing special remote upon initremote with e.g. `layout=local`, which would lookup location for the key in the local repository (under `.git/annex/objects`), and use it as the path for the key on the remote.
+
+Is that a correct idea Joey?  or you see a better way?
+"""]]

diff --git a/doc/forum/lets_discuss_git_add_behavior/comment_35_0423806de951f97f156b5edea85522cc._comment b/doc/forum/lets_discuss_git_add_behavior/comment_35_0423806de951f97f156b5edea85522cc._comment
new file mode 100644
index 000000000..5b0c18a5e
--- /dev/null
+++ b/doc/forum/lets_discuss_git_add_behavior/comment_35_0423806de951f97f156b5edea85522cc._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="http://templeofcrom.duckdns.org/"
+ nickname="Karl"
+ avatar="http://cdn.libravatar.org/avatar/336975995d2c8652aa98284987d5987e90e1b4d137da415af18a8e04c29edbc3"
+ subject="A warning in the docs about earlier v7 revisions would be nice"
+ date="2019-11-08T04:59:43Z"
+ content="""
+I started using git-annex for the first time today and ran head-first into this bug, so I'm glad to see a course correction here.  I just wish the update had been done a month earlier, as the version packaged with Fedora 30 has the 'git add' override behavior which cost me a few hours in figuring out how to get files out of the annex and into a git object.  The git-annex-add wiki page really could use a couple warnings to say that 'git add' may be overridden and that in earlier v7 revisions you are basically required to use annex.largefiles with a strict filter in order to make normal use of 'git add'.
+"""]]

diff --git a/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment b/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment
deleted file mode 100644
index 1def2047d..000000000
--- a/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment
+++ /dev/null
@@ -1,10 +0,0 @@
-[[!comment format=mdwn
- username="pigmonkey"
- avatar="http://cdn.libravatar.org/avatar/560cedfcec1b75e8ac2e98a10615d770"
- subject="Consider deprecation warnings in the web app"
- date="2019-11-07T18:39:02Z"
- content="""
-If rebuilding the database is an operation that will take some time, it might be nice to have deprecation warnings in the web app before the assistant eventually autoupgrades repos.
-
-I use the assistant/web app to manage about 10 repos, some of which are on the larger size both in terms of disk space and number of files. I suspect that if the assistant kicked off an upgrade with a database rebuild on all of these at once, it would have a noticeable performance impact on my machine. If, after v8 is merged but before the assistant autoupgrades, the web app displayed a message like \"This repository is using a version that will soon be upgraded, click here to learn more about v8 and consider upgrading\", it would give folks (who don't read the devblog or release notes) a heads up and give them a chance to manually upgrade repositories one by one.
-"""]]

diff --git a/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment b/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment
new file mode 100644
index 000000000..1def2047d
--- /dev/null
+++ b/doc/devblog/day_607__v8_is_done/comment_2_b8080cc794dcdf7c51668610d53ed2c1._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="pigmonkey"
+ avatar="http://cdn.libravatar.org/avatar/560cedfcec1b75e8ac2e98a10615d770"
+ subject="Consider deprecation warnings in the web app"
+ date="2019-11-07T18:39:02Z"
+ content="""
+If rebuilding the database is an operation that will take some time, it might be nice to have deprecation warnings in the web app before the assistant eventually autoupgrades repos.
+
+I use the assistant/web app to manage about 10 repos, some of which are on the larger size both in terms of disk space and number of files. I suspect that if the assistant kicked off an upgrade with a database rebuild on all of these at once, it would have a noticeable performance impact on my machine. If, after v8 is merged but before the assistant autoupgrades, the web app displayed a message like \"This repository is using a version that will soon be upgraded, click here to learn more about v8 and consider upgrading\", it would give folks (who don't read the devblog or release notes) a heads up and give them a chance to manually upgrade repositories one by one.
+"""]]

diff --git a/doc/devblog/day_607__v8_is_done/comment_1_7326382d8ff23873c2fe0d6acd984454._comment b/doc/devblog/day_607__v8_is_done/comment_1_7326382d8ff23873c2fe0d6acd984454._comment
new file mode 100644
index 000000000..97e6a5cf8
--- /dev/null
+++ b/doc/devblog/day_607__v8_is_done/comment_1_7326382d8ff23873c2fe0d6acd984454._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="pigmonkey"
+ avatar="http://cdn.libravatar.org/avatar/560cedfcec1b75e8ac2e98a10615d770"
+ subject="Consider deprecation warnings in the web app"
+ date="2019-11-07T18:38:28Z"
+ content="""
+If rebuilding the database is an operation that will take some time, it might be nice to have deprecation warnings in the web app before the assistant eventually autoupgrades repos.
+
+I use the assistant/web app to manage about 10 repos, some of which are on the larger size both in terms of disk space and number of files. I suspect that if the assistant kicked off an upgrade with a database rebuild on all of these at once, it would have a noticeable performance impact on my machine. If, after v8 is merged but before the assistant autoupgrades, the web app displayed a message like \"This repository is using a version that will soon be upgraded, click here to learn more about v8 and consider upgrading\", it would give folks (who don't read the devblog or release notes) a heads up and give them a chance to manually upgrade repositories one by one.
+"""]]

diff --git a/doc/devblog/day_607__v8_is_done.mdwn b/doc/devblog/day_607__v8_is_done.mdwn
new file mode 100644
index 000000000..eead30e0c
--- /dev/null
+++ b/doc/devblog/day_607__v8_is_done.mdwn
@@ -0,0 +1,40 @@
+Spent the past two weeks on the [[todo/sqlite_database_improvements]]
+which will be git-annex v8.
+
+That cleaned up a significant amount of technical debt. I had made some bad
+choices about encoding sqlite data early on, and the persistent library
+turns out to make a dubious choice about how String is stored, that
+prevents some unicode surrigate code points from roundtripping sometimes.
+On top of those problems, there were some missing indexes. And then to
+resolve the `git add` mess, I had to write a raw SQL query that used LIKE,
+which was super ugly, slow, and not indexed.
+
+Really good to get all that resolved. And I have microbenchmarks that are
+good too; 10-25% speedup across the board for database operations.
+
+The tricky thing was that, due to the encoding problem, both filenames and
+keys stored in the old sqlite databases can't be trusted to be valid. This
+ruled out a database migration because it could leave a repo with bad old
+data in it. Instead, the old databases have to be thrown away, and the
+upgrade has to somehow build new databases that contain all the necessary
+data. Seems a tall order, but luckily git-annex is a distributed system and
+so the databases are used as a local fast cache for information that can be
+looked up more slowly from git. Well, mostly. Sometimes the databases are
+used for data that has not yet been committed to git, or that is local to a
+single repo.
+
+So I had to find solutions to a lot of hairly problems. In a couple cases,
+the solutions involve git-annex doing more work after the upgrade for a
+while, until it is able to fully regenerate the data that was stored in the
+old databases.
+
+One nice thing about this approach is that, if I ever need to change the
+sqlite databases again, I can reuse the same code to delete the old and
+regnerate the new, rather than writing migration code specific to a
+given database change.
+
+Anyway, v8 is all ready to merge, but I'm inclined to sit on it for a month or
+two, to avoid upgrade fatigue. Also I find more ways to improve the
+database schema. Perhaps it would be worth it to do some normalization,
+and/or move everything into a single large database rather than the current
+smattering of unnormalized databases?

diff --git a/doc/todo/sqlite_database_improvements.mdwn b/doc/todo/sqlite_database_improvements.mdwn
index c462b1621..d6647b693 100644
--- a/doc/todo/sqlite_database_improvements.mdwn
+++ b/doc/todo/sqlite_database_improvements.mdwn
@@ -1,9 +1,14 @@
+The `sqlite` branch changes the databases, updating annex.version to 8.
+The branch is ready to merge, but it might be deferred until Q1 2020
+to avoid user upgrade fatigue. Discussion below is about the motivation for
+these changes.
+
+----
+
 Collection of non-ideal things about git-annex's use of sqlite databases.
 Would be good to improve these sometime, but it would need a migration
 process.
 
-The `sqlite` branch changes the databases, updating annex.version to 8.
-This todo documents the state of that branch.
 
 * Database.Keys.SQL.isInodeKnown has some really ugly SQL LIKE queries. 
   Probably an index would not speed them up. They're only needed when

diff --git a/doc/news/version_7.20190912.mdwn b/doc/news/version_7.20190912.mdwn
deleted file mode 100644
index 5e2beb26f..000000000
--- a/doc/news/version_7.20190912.mdwn
+++ /dev/null
@@ -1,62 +0,0 @@
-News for git-annex 7.20190912:
-
-This version of git-annex uses repository version 7 for all repositories.
-Existing v5 repositories will be automatically upgraded by default.
-
-You can prevent this, by runing: git config annex.autoupgraderepository false
-
-A v7 repository can can have some files locked while other files are
-unlocked, and all git and git-annex commands can be used on both locked and
-unlocked files. It's a good idea to make sure that all users of the
-repository have upgraded git-annex and upgraded their repositories
-to the new version before starting to use that feature, since old
-versions of git-annex will ignore the new unlocked files.
-
-The behavior of some commands changes in an upgraded repository:
-
-* `git add` will add files to the annex, rather than adding them directly
-  to the git repository. To cause some files to be added directly
-  to git, you can configure `annex.largefiles`. For example:
-  git config annex.largefiles "largerthan=100kb and not (include=*.c or include=*.h)"
-
-* `git annex unlock` and `git annex lock` change how the pointer to
-  the annexed content is stored in git. If you commit the change,
-  that will impact all clones of the repository.
-
-git-annex 7.20190912 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Default to v7 for new repositories.
-   * Automatically upgrade v5 repositories to v7.
-   * Automatically convert direct mode repositories to v7 with adjusted
-     unlocked branches and set annex.thin.
-   * Added annex.autoupgraderepository configuration that can be set to false
-     to prevent any automatic repository upgrades.
-   * Refuse to upgrade direct mode repositories when git is older than 2.22,
-     which fixed a memory leak that could cause an OOM during the upgrade.
-   * Removed support for git versions older than 2.1.
-   * assistant: When creating a new repository, no longer use direct
-     mode, instead use v7 adjusted branches with annex.thin.
-   * init: When run on a crippled filesystem with --version=5,
-     will error out, since version 7 is needed for adjusted unlocked branch.
-   * direct: This command always errors out as direct mode is no longer
-     supported.
-   * indirect: This command has become a deprecated noop.
-   * proxy: This command is deprecated because it was only needed in direct
-     mode. (But it continues to work.)
-   * info: Removed the "repository mode" from its output (including the
-     --json output) since with the removal of direct mode, there is no
-     repository mode.
-   * info: When file matching options are specified when getting
-     info of something other than a directory, they won't have any effect,
-     so error out to avoid confusion.
-   * info: Display trust level when getting info on a uuid, same as a remote.
-   * When upgrading a direct mode repo to v7 with adjusted unlocked branches,
-     fix a bug that prevented annex.thin from taking effect for the files
-     in working tree.
-   * Avoid making a commit when upgrading from direct mode to v7.
-   * init: Catch more exceptions when testing locking.
-   * init: Fix a reversion that broke initialization on systems that
-     need to use pid locking.
-   * A git-annex-standalone rpm is now distributed along with the standalone
-     tarball.
-   * Added standalone/rpm/rpmbuild-from-standalone-tarball script."""]]
diff --git a/doc/news/version_7.20191106.mdwn b/doc/news/version_7.20191106.mdwn
new file mode 100644
index 000000000..e51c99a3d
--- /dev/null
+++ b/doc/news/version_7.20191106.mdwn
@@ -0,0 +1,5 @@
+git-annex 7.20191106 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * init: Fix bug that lost modifications to unlocked files when init is
+     re-ran in an already initialized repo.
+   * benchmark: Add --databases to benchmark sqlite databases."""]]
\ No newline at end of file

diff --git a/doc/news/version_7.20191024/comment_2_209826312977675475affe8f669dfa73._comment b/doc/news/version_7.20191024/comment_2_209826312977675475affe8f669dfa73._comment
new file mode 100644
index 000000000..1cdeae256
--- /dev/null
+++ b/doc/news/version_7.20191024/comment_2_209826312977675475affe8f669dfa73._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="nivek-retsof@97a405d1563af7581f6c8d1f7bae67b0ce896721"
+ nickname="nivek-retsof"
+ avatar="http://cdn.libravatar.org/avatar/3aec9a88897e105952a4ed38007fb224"
+ subject="Thanks!"
+ date="2019-11-05T22:18:49Z"
+ content="""
+This behavior makes more sense to me. I like keeping my text files in git and binaries in git-annex, and treating them separately is no great mental burden.
+
+Thanks for your great work. Git-annex has become one of the best parts of my computing experience. It is a great relief knowing that my files are in a robust distributed and redundant collection that is easy to maintain.
+"""]]

diff --git a/doc/thanks/list b/doc/thanks/list
index 850d665d1..00670e948 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -69,3 +69,4 @@ Marius Konitzer,
 Ryan Rix, 
 Svenne Krap, 
 Jelmer Vernooij, 
+Rian McGuire, 

diff --git a/doc/todo/sqlite_database_improvements.mdwn b/doc/todo/sqlite_database_improvements.mdwn
index 62bd6e862..c462b1621 100644
--- a/doc/todo/sqlite_database_improvements.mdwn
+++ b/doc/todo/sqlite_database_improvements.mdwn
@@ -2,23 +2,22 @@ Collection of non-ideal things about git-annex's use of sqlite databases.
 Would be good to improve these sometime, but it would need a migration
 process.
 
-* Database.Keys.SQL.isInodeKnown seems likely to get very slow
-  when there are a lot of unlocked annexed files. It needs
-  an index in the database, eg "InodeIndex cache"
+The `sqlite` branch changes the databases, updating annex.version to 8.
+This todo documents the state of that branch.
 
-  It also has to do some really ugly SQL LIKE queries. Probably an index
-  would not speed them up. They're only needed when git-annex detects
-  inodes are not stable, eg on fat or probably windows. A better database
+* Database.Keys.SQL.isInodeKnown has some really ugly SQL LIKE queries. 
+  Probably an index would not speed them up. They're only needed when
+  git-annex detects inodes are not stable, eg on fat or probably windows.
+  A better database
   schema should be able to eliminate the need for those LIKE queries.
   Eg, store the size and allowable mtimes in a separate table that is
   queried when necessary.
 
-* Database.Export.getExportedKey would be faster if there was an index
-  in the database, eg "ExportedIndex file key". This only affects
-  the speed of `git annex export`, which is probably swamped by the actual
-  upload of the data to the remote.
+  Fixed.
 
-* There may be other selects elsewhere that are not indexed.
+* Several selects were not able to use indexes, so would be slow.
+
+  Fixed by adding indexes.
 
 * Database.Types has some suboptimal encodings for Key and InodeCache.
   They are both slow due to being implemented using String
@@ -26,6 +25,8 @@ process.
   and the VARCHARs they generate are longer than necessary
   since they look like eg `SKey "whatever"` and `I "whatever"`
 
+  Fixed.
+
 * SFilePath is stored efficiently, and has to be a String anyway,
   (until ByteStringFilePath is used)
   but since it's stored as a VARCHAR, which sqlite interprets using the
@@ -45,6 +46,8 @@ process.
   And it seems likely that a query by filename would fail if the filename
   was in the database but with a different encoding.
 
+  Fixed by converting to blob.
+
 * IKey could fail to round-trip as well, when a Key contains something
   (eg, a filename extension) that is not valid in the current locale,
   for similar reasons to SFilePath. Using BLOB would be better.
@@ -59,6 +62,14 @@ process.
 	INSERT INTO associated VALUES(8,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.ü','foo.ü');
 	INSERT INTO associated VALUES(9,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.��','"foo.\56515\56508"');
 
+  Fixed by converting to blob.
+
+----
+
+remaining todo:
+
+* migration
+
 > Investigated this in more detail, and I can't find a way to
 > solve the encoding problem other than changing the encoding
 > SKey, IKey, and SFilePath in a non-backwards-compatible way.
@@ -99,116 +110,10 @@ process.
 >   situations, the next time an export is run, so should be ok.
 >   But it might result in already exported files being re-uploaded,
 >   or other unncessary work.
-> Keys (IKey, SFilePath)
->   rebuild with scanUnlockedFiles
-> 
->   does that update the Content table with the InodeCache?
->
-> But after such a transition, how to communicate to the old git-annex
-> that it can't use the databases any longer? Moving the databases
-> out of the way won't do; old git-annex will just recreate them and 
-> start with missing data!
-> 
-> And, what about users who use a mix of old and new git-annex versions?
+> Keys (IKey, SFilePath, SInodeCache)
+>   Use scanUnlockedFiles to repopulate the Associated table.
 > 
-> Seems this needs an annex.version bump from v7 to v8.
-
-----
-
-[[!format patch """
-diff --git a/Database/Types.hs b/Database/Types.hs
-index f08cf4e9d..3e9c9e267 100644
---- a/Database/Types.hs
-+++ b/Database/Types.hs
-@@ -14,11 +14,12 @@ import Database.Persist.TH
- import Database.Persist.Class hiding (Key)
- import Database.Persist.Sql hiding (Key)
- import Data.Maybe
--import Data.Char
- import qualified Data.ByteString as S
-+import qualified Data.ByteString.Lazy as L
- import qualified Data.Text as T
- 
- import Utility.PartialPrelude
-+import Utility.FileSystemEncoding
- import Key
- import Utility.InodeCache
- import Git.Types (Ref(..))
-@@ -37,23 +38,18 @@ fromSKey (SKey s) = fromMaybe (error $ "bad serialized Key " ++ s) (deserializeK
- 
- derivePersistField "SKey"
- 
---- A Key index. More efficient than SKey, but its Read instance does not
---- work when it's used in any kind of complex data structure.
--newtype IKey = IKey String
--
--instance Read IKey where
--	readsPrec _ s = [(IKey s, "")]
--
--instance Show IKey where
--	show (IKey s) = s
-+-- A Key index. More efficient than SKey.
-+newtype IKey = IKey S.ByteString
-+	deriving (Eq, Show, PersistField, PersistFieldSql)
- 
-+-- FIXME: toStrict copies, not efficient
- toIKey :: Key -> IKey
--toIKey = IKey . serializeKey
-+toIKey = IKey . L.toStrict . serializeKey'
- 
- fromIKey :: IKey -> Key
--fromIKey (IKey s) = fromMaybe (error $ "bad serialized Key " ++ s) (deserializeKey s)
--
--derivePersistField "IKey"
-+fromIKey (IKey b) = fromMaybe
-+	(error $ "bad serialized Key " ++ show b) 
-+	(deserializeKey' b)
- 
- -- A serialized InodeCache
- newtype SInodeCache = I String
-@@ -67,39 +63,15 @@ fromSInodeCache (I s) = fromMaybe (error $ "bad serialized InodeCache " ++ s) (r
- 
- derivePersistField "SInodeCache"
- 
---- A serialized FilePath.
----
---- Not all unicode characters round-trip through sqlite. In particular,
---- surrigate code points do not. So, escape the FilePath. But, only when
---- it contains such characters.
--newtype SFilePath = SFilePath String
--
---- Note that Read instance does not work when used in any kind of complex
---- data structure.
--instance Read SFilePath where
--	readsPrec _ s = [(SFilePath s, "")]
--
--instance Show SFilePath where
--	show (SFilePath s) = s
-+-- A serialized FilePath. Stored as a ByteString to avoid encoding problems.
-+newtype SFilePath = SFilePath S.ByteString
-+	deriving (Eq, Show, PersistField, PersistFieldSql)
- 
- toSFilePath :: FilePath -> SFilePath
--toSFilePath s@('"':_) = SFilePath (show s)
--toSFilePath s
--	| any needsescape s = SFilePath (show s)
--	| otherwise = SFilePath s
--  where
--	needsescape c = case generalCategory c of
--		Surrogate -> True
--		PrivateUse -> True
--		NotAssigned -> True
--		_ -> False
-+toSFilePath = SFilePath . encodeBS
- 
- fromSFilePath :: SFilePath -> FilePath
--fromSFilePath (SFilePath s@('"':_)) =
--	fromMaybe (error "bad serialized SFilePath " ++ s) (readish s)
--fromSFilePath (SFilePath s) = s
--
--derivePersistField "SFilePath"
-+fromSFilePath (SFilePath b) = decodeBS b
- 
- -- A serialized Ref
- newtype SRef = SRef Ref
-"""]]
+>   But that does not repopulate the Content table. Doing so needs
+    to iterate over the unlocked files, filter out any that are modified,
+    and record the InodeCaches of the unmodified ones. Seems that it would
+    have to use git's index to know which files are modified.

diff --git a/doc/bugs/git-lfs_remote_URL_is_not_recorded__63__.mdwn b/doc/bugs/git-lfs_remote_URL_is_not_recorded__63__.mdwn
index c607cf7f8..04784716b 100644
--- a/doc/bugs/git-lfs_remote_URL_is_not_recorded__63__.mdwn
+++ b/doc/bugs/git-lfs_remote_URL_is_not_recorded__63__.mdwn
@@ -116,4 +116,4 @@ Shouldn't that URL be recorded in remote.log? (similarly to `type=git` remotes)
 [[!meta author=yoh]]
 [[!tag projects/dandi]]
 
-> [[done]; see my comment --[[Joey]]
+> [[done]]; see my comment --[[Joey]]

diff --git a/doc/projects/datalad.mdwn b/doc/projects/datalad.mdwn
index 9ab203776..6680e3f86 100644
--- a/doc/projects/datalad.mdwn
+++ b/doc/projects/datalad.mdwn
@@ -4,20 +4,27 @@ TODOs for DataLad
 [[!inline pages="todo/* and !todo/done and !link(todo/done) and
 (author(yoh) or author(mih) or author(ben) or author(yarikoptic) or author(kyle))" sort=mtime feeds=no actions=yes archive=yes show=0]]
 
-Done
-----
+
+<details>
+<summary>Done</summary>
 
 [[!inline pages="todo/* and !todo/done and link(todo/done) and
 (author(yoh) or author(mih) or author(ben) or author(yarikoptic) or author(kyle))" feeds=no actions=yes archive=yes show=0]]
 
+</details>
+
 My bugs
 =======
 
 [[!inline pages="bugs/* and !bugs/done and !link(bugs/done) and
 (author(yoh) or author(mih) or author(ben) or author(yarikoptic) or author(kyle))" sort=mtime feeds=no actions=yes archive=yes show=0  template=buglist]]
 
-Fixed
------
+
+
+<details>
+<summary>Fixed</summary>
 
 [[!inline pages="bugs/* and !bugs/done and link(bugs/done) and
 (author(yoh) or author(mih) or author(ben) or author(yarikoptic) or author(kyle))" feeds=no actions=yes archive=yes show=0  template=buglist]]
+
+</details>

diff --git a/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_6_2d78e647a46cfbb002cf66e6318043ab._comment b/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_6_2d78e647a46cfbb002cf66e6318043ab._comment
new file mode 100644
index 000000000..1fe3d6531
--- /dev/null
+++ b/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_6_2d78e647a46cfbb002cf66e6318043ab._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="comment 6"
+ date="2019-11-04T16:40:52Z"
+ content="""
+Great that it's working.
+
+Not sure if relevant to your case, but one other way to have multiple checkouts of one repo is [[tips/Using_git-worktree_with_annex]].  Each checkout must be of a different branch though.  On the plus side, they all share one annex, so there is only one copy of each annexed file (for locked files).
+"""]]

diff --git a/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_5_d8c402765b3a5896db91605f43149001._comment b/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_5_d8c402765b3a5896db91605f43149001._comment
new file mode 100644
index 000000000..93343ce50
--- /dev/null
+++ b/doc/forum/Is_it_possible_adding_files_to_git_annex_bypassing___40__some_parts_of__41___the_git_annex_tools/comment_5_d8c402765b3a5896db91605f43149001._comment
@@ -0,0 +1,38 @@
+[[!comment format=mdwn
+ username="irieger"
+ avatar="http://cdn.libravatar.org/avatar/83622f3c2fedea51e69f9a3986f95d73"
+ subject="comment 5"
+ date="2019-11-04T08:30:07Z"
+ content="""
+Ok, thanks.
+
+My second question was formulated not quite well. Of course I meant multiple checkouts of this one repository. That in the end is my use case that I have multiple checksum verified copies of all my images/movie clips in one go.
+
+Did quite a bit of testing over the past weeks and stuff seams working well so far.
+
+Here is a sample bash script I used to experiment that shows the workflow if anyone is interested. (Just an example, seems to do basically the same as git annex add $FILE as it is):
+
+    #!/bin/bash
+    
+    FILE=$1
+    echo \"Adding file '$FILE'\"
+    
+    HASH=$(sha256sum \"$FILE\" | sed 's/[^a-fA-F0-9].*//')
+    SIZE=$(stat --printf=\"%s\" \"$FILE\")
+    EXT=$(echo \"${FILE##*.}\" | tr '[:upper:]' '[:lower:]')
+    KEY=\"SHA256E-s${SIZE}--${HASH}.${EXT}\"
+    OBJECT=$(git annex examinekey $KEY --format='.git/annex/objects/${hashdirmixed}${key}/${key}')
+    
+    echo \"    ext:    $EXT\"
+    echo \"    sha256: $HASH\"
+    echo \"    size:   $SIZE\"
+    echo \"    KEY:    $KEY\"
+    echo \"    OBJECT: $OBJECT\"
+    
+    git -c annex.verify=false -c annex.alwayscommit=false annex setkey \"$KEY\" \"$FILE\"
+    ln -s \"$OBJECT\" \"$FILE\"
+    git add \"$FILE\"
+
+Kind regards,
+Ingmar
+"""]]

diff --git a/doc/forum/Specifying_private_key_for_ssh_remote/comment_1_94d38f788ab5bc34da9ba56c2fb729b5._comment b/doc/forum/Specifying_private_key_for_ssh_remote/comment_1_94d38f788ab5bc34da9ba56c2fb729b5._comment
new file mode 100644
index 000000000..2df2ea4b3
--- /dev/null
+++ b/doc/forum/Specifying_private_key_for_ssh_remote/comment_1_94d38f788ab5bc34da9ba56c2fb729b5._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="hobbes@b2cacef69071743c3a831e60511062f7e014e52f"
+ nickname="hobbes"
+ avatar="http://cdn.libravatar.org/avatar/44b70169c4d862b3619812c360aa8f1e"
+ subject="Ssh config"
+ date="2019-11-03T23:49:05Z"
+ content="""
+I would look into `man ssh_config. You can set up hosts and specific identities for those hosts.
+"""]]

diff --git a/doc/todo/git-annex-sync_handling_of_linked_worktrees.mdwn b/doc/todo/git-annex-sync_handling_of_linked_worktrees.mdwn
new file mode 100644
index 000000000..544254508
--- /dev/null
+++ b/doc/todo/git-annex-sync_handling_of_linked_worktrees.mdwn
@@ -0,0 +1 @@
+When using [[linked worktrees|tips/Using_git-worktree_with_annex]], the main tree is currently handled differently from the linked trees: "if there is change in the tree then syncing doesn't update git worktrees and their indices, but updates the checked out branches. This is different to the handling of the main working directory as it's either got updated or left behind with its branch if there is a conflict."   Is there a reason for this?  Could linked worktrees be treated same as main one?

diff --git a/doc/bugs/2_ssh_connection_prompts_for_password/comment_3_f621e615a745910b4e28f91b27132c0e._comment b/doc/bugs/2_ssh_connection_prompts_for_password/comment_3_f621e615a745910b4e28f91b27132c0e._comment
new file mode 100644
index 000000000..4745d9990
--- /dev/null
+++ b/doc/bugs/2_ssh_connection_prompts_for_password/comment_3_f621e615a745910b4e28f91b27132c0e._comment
@@ -0,0 +1,43 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="the issue persists"
+ date="2019-11-01T18:12:27Z"
+ content="""
+Ran into the same problem again, and it is not clear to me either connection caching is enabled or not (and why?):
+
+[[!format sh \"\"\"
+[d31548v@discovery7 bids]$ git -c annex.sshcaching=true annex --debug get -J2 --from=origin sub-sid000005
+[2019-11-01 14:10:56.178577] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"]
+[2019-11-01 14:10:56.475956] process done ExitSuccess
+[2019-11-01 14:10:56.47622] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+[2019-11-01 14:10:56.836271] process done ExitSuccess
+[2019-11-01 14:10:56.865928] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..8a694d5c54eb81b1e5c5446fa63bdcd13daa34b3\",\"--pretty=%H\",\"-n1\"]
+[2019-11-01 14:10:57.229787] process done ExitSuccess
+[2019-11-01 14:10:57.234655] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-11-01 14:10:57.23592] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+[2019-11-01 14:10:57.546203] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"]
+[2019-11-01 14:10:57.780246] process done ExitSuccess
+[2019-11-01 14:10:57.780454] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"]
+[2019-11-01 14:10:58.097345] process done ExitSuccess
+[2019-11-01 14:10:58.09754] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"sub-sid000005\"]
+[2019-11-01 14:10:58.298181] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-11-01 14:10:58.29998] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+[2019-11-01 14:10:58.305022] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-11-01 14:10:58.306024] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+[2019-11-01 14:10:58.62005] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-11-01 14:10:58.621714] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+[2019-11-01 14:10:58.632596] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"]
+[2019-11-01 14:10:58.6338] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
+get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude2.nii.gz get sub-sid000005/ses-actions1/fmap/sub-sid000005_ses-actions1_acq-25mm_magnitude1.nii.gz (from origin...) (from origin...) 
+[2019-11-01 14:10:59.489719] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
+
+[2019-11-01 14:10:59.553029] chat: ssh [\"yohtest@rolando.cns.dartmouth.edu\",\"-T\",\"git-annex-shell 'p2pstdio' '/inbox/BIDS/Haxby/Sam/1021_actions' '--debug' 'fd3f7af9-cf7d-4d7e-8efd-30e6bedf838d' --uuid d839134c-3afe-4456-920a-e280ce0fdf2a\"]
+yohtest@rolando.cns.dartmouth.edu's password: yohtest@rolando.cns.dartmouth.edu's password: 
+
+
+[d31548v@discovery7 bids]$ git annex version
+git-annex version: 7.20191024-g6dc2272
+\"\"\"]]
+Could you hint me on what/where to dig?
+"""]]

diff --git a/doc/forum/Alternative_to_Tor_for_remote_pairing__63__/comment_1_39417a8bd5e6834041d759ea3666d894._comment b/doc/forum/Alternative_to_Tor_for_remote_pairing__63__/comment_1_39417a8bd5e6834041d759ea3666d894._comment
new file mode 100644
index 000000000..6e2d1bdb2
--- /dev/null
+++ b/doc/forum/Alternative_to_Tor_for_remote_pairing__63__/comment_1_39417a8bd5e6834041d759ea3666d894._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="plasmastrike@66c1cae4abae92d03faf344111c446a1908981ff"
+ nickname="plasmastrike"
+ avatar="http://cdn.libravatar.org/avatar/2859d0b6fe78b2252c6fbf46ce02e85b"
+ subject="Yggdrasill"
+ date="2019-11-01T13:34:36Z"
+ content="""
+You could use Yggdrasil and a Yggdrasil ipv6 address. It would allow you to ssh even if the ssh port is normally blocked. Think of Yggdrasil as dynamic reconfiguring vpn so all traffic goes through one port and encrypted.
+
+https://yggdrasil-network.github.io/
+https://yggdrasil-network.github.io/2018/07/15/remote-access.html
+
+You would still need ssh, rsync or an alternative to move the files but you will get past blocked ports
+"""]]

diff --git a/doc/bugs/Make_webapp_port_configurable..sh b/doc/bugs/Make_webapp_port_configurable..sh
new file mode 100644
index 000000000..fa9ee10f0
--- /dev/null
+++ b/doc/bugs/Make_webapp_port_configurable..sh
@@ -0,0 +1,41 @@
+### Please describe the problem.
+This is more of a feature request then a bug. 
+
+It would be nice and more intuitive if the webapp --listen parameter accepted a port specifier too allowing configuration of the port. 
+
+For my workflow, I thought I would contain all of git annexes dependencies inside a docker image since I'm quite comfortable with docker(and emerge on gentoo took a long time and finally failed). With an unconfigurable dynamic port though, it makes running the webapp subcommand in docker not really viable since I don't want to use docker run's port range mapping feature which will lock all those ports.
+
+### What steps will reproduce the problem?
+Dockerfile:
+ARG DEBIAN_TAG=buster-slim
+FROM debian:${DEBIAN_TAG}
+
+RUN set -ex \
+  && apt-get update \
+  && apt-get install -y \
+     git \
+     git-annex
+
+docker build -t git-annex:test .
+docker run --rm -it git-annex:test git annex webapp --listen 0.0.0.0:8888
+
+### What version of git-annex are you using? On what operating system?
+
+git-annex version: 7.20190129
+build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+git annex seems awesome with the little bit of testing I've done. It seems like the perfect tool for what I want to accomplish. Thanks!
+

diff --git a/doc/bugs/Make_webapp_port_configurable..pl b/doc/bugs/Make_webapp_port_configurable..pl
new file mode 100644
index 000000000..fa9ee10f0
--- /dev/null
+++ b/doc/bugs/Make_webapp_port_configurable..pl
@@ -0,0 +1,41 @@
+### Please describe the problem.
+This is more of a feature request then a bug. 
+
+It would be nice and more intuitive if the webapp --listen parameter accepted a port specifier too allowing configuration of the port. 
+
+For my workflow, I thought I would contain all of git annexes dependencies inside a docker image since I'm quite comfortable with docker(and emerge on gentoo took a long time and finally failed). With an unconfigurable dynamic port though, it makes running the webapp subcommand in docker not really viable since I don't want to use docker run's port range mapping feature which will lock all those ports.
+
+### What steps will reproduce the problem?
+Dockerfile:
+ARG DEBIAN_TAG=buster-slim
+FROM debian:${DEBIAN_TAG}
+
+RUN set -ex \
+  && apt-get update \
+  && apt-get install -y \
+     git \
+     git-annex
+
+docker build -t git-annex:test .
+docker run --rm -it git-annex:test git annex webapp --listen 0.0.0.0:8888
+
+### What version of git-annex are you using? On what operating system?
+
+git-annex version: 7.20190129
+build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
+dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
+
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+git annex seems awesome with the little bit of testing I've done. It seems like the perfect tool for what I want to accomplish. Thanks!
+

diff --git a/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_8_bca507381436a8282645a931aa39fb20._comment b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_8_bca507381436a8282645a931aa39fb20._comment
new file mode 100644
index 000000000..c1feb76f8
--- /dev/null
+++ b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_8_bca507381436a8282645a931aa39fb20._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="annexing previously non-annexed files, and compatibility with v5"
+ date="2019-10-31T16:19:39Z"
+ content="""
+\"it would be better if files already in git were not annexed even if they match annex.largefiles\" -- actually not sure: what if a file was in git but gets modified to something _much_ larger?  If you have \"a few files just a bit larger\" than your `annex.largefiles` setting, maybe just increase that setting?  You could also set `annex.gitaddtoannex=false` to prevent `git add` from annexing previously non-annexed files, and use `git annex add` to annex new files.
+
+Also, if goal is compatibility with v5, you can lock files after annexing.  If [[todo/auto-lock_files_after_one_edit]] gets implemented at some point, that could also help this case.
+"""]]

diff --git a/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_7_e349403ef92a96a08195a7dd1dbf9561._comment b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_7_e349403ef92a96a08195a7dd1dbf9561._comment
new file mode 100644
index 000000000..815383787
--- /dev/null
+++ b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_7_e349403ef92a96a08195a7dd1dbf9561._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Ilya_Shlyakhter"
+ avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
+ subject="not annexing files already stored in git"
+ date="2019-10-31T16:03:41Z"
+ content="""
+\"a few files just a bit larger than that are commited to git\" -- if it's just a few, maybe set `annex.largefiles=nothing` explicitly for each of them in .gitattributes?  (Also then set `annex.largefiles=(largerthan=100kb)` in .gitattributes rather than in git config, since that overrides .gitattributes).  But in general it would be better if files already in git were not annexed even if they match `annex.largefiles`.
+"""]]

diff --git a/doc/todo/warn_of_breaking_changes_same_way_git_does.mdwn b/doc/todo/warn_of_breaking_changes_same_way_git_does.mdwn
index d0871534d..c2e0427ce 100644
--- a/doc/todo/warn_of_breaking_changes_same_way_git_does.mdwn
+++ b/doc/todo/warn_of_breaking_changes_same_way_git_does.mdwn
@@ -1 +1,7 @@
 When git introduces a breaking change, it prints a warning describing the change and saying "to turn off this warning set advice.warnaboutthis=false".  This adds the hassle of setting the config (once, globally), but the reduced confusion might be worth it.
+
+git also uses this mechanism to start warning of upcoming breaking changes in future versions, before these versions become current.
+
+Another way to signal breaking changes can be to use some form of [semantic versioning](https://semver.org/) for git-annex versions, bumping the major version number to indicate incompatible changes.  [[7.20190912|news/version_7.20190912]] would then be 8.20190912 .  This would help when specifying [package dependencies](https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#package-match-specifications).
+
+Of course, best is to avoid breaking changes, as @joeyh ultimately [[managed to do with v7|news/version_7.20191024]].  But they'll happen, and some people miss the [[current scheme for announcing them|forum/lets_discuss_git_add_behavior/#comment-727918fd9abda5445891f2022ac225c8]], so low-cost ways to extend the scheme might help.

diff --git a/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_6_e5d4ba2bb9f78380dfa94446cb1f4b54._comment b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_6_e5d4ba2bb9f78380dfa94446cb1f4b54._comment
new file mode 100644
index 000000000..ef262a945
--- /dev/null
+++ b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_6_e5d4ba2bb9f78380dfa94446cb1f4b54._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="bremner"
+ avatar="http://cdn.libravatar.org/avatar/22c2d800db6a7699139df604a67cb221"
+ subject="plz don't annex already committed files"
+ date="2019-10-31T13:50:09Z"
+ content="""
+I am suffering a bit at the moment from git-annex 7 annexing committed files and replacing them with pseudo symlinks. This breaks the file for v5 clients. What's the best way to disable this? I have annex.largefiles=100k, but a few files just a bit larger than that are commited to git.
+"""]]

diff --git a/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn b/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn
index b16e3c32a..82c8bae21 100644
--- a/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn
+++ b/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn
@@ -19,4 +19,6 @@ the reason for the failure overall is that `datalad` external special remote is
 
 If user convenience was something to strive for here, it should technically be possible for git annex to "sense" all available external special remotes and inquire them (without enabling) either they would support a given url, and then report to the user a more informative error message ("URL ... is not supported by git-annex.  You can initialize one of the available external special remotes which support that url: datalad, somebodies-s3, ...".  It could even check if the special remote is configured but not enabled and then replace "initialize" with "enable" in the message above).
 
+[[!meta author=yoh]]
+[[!tag projects/datalad]]
 

diff --git a/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn b/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn
new file mode 100644
index 000000000..b16e3c32a
--- /dev/null
+++ b/doc/bugs/error_message_in_addurl_is_not_channeled_into_json_record_with_--json-error-messages.mdwn
@@ -0,0 +1,22 @@
+### Please describe the problem.
+
+Original report: https://github.com/datalad/datalad/issues/3825
+
+
+[[!format sh """
+$> git annex addurl --json --json-error-messages s3://hcp-openaccess/HCP_1200/100206/.xdlm/100206_3T_Diffusion_unproc.json                                       
+Configuration does not allow accessing s3://hcp-openaccess/HCP_1200/100206/.xdlm/100206_3T_Diffusion_unproc.json
+download failed: Configuration does not allow accessing s3://hcp-openaccess/HCP_1200/100206/.xdlm/100206_3T_Diffusion_unproc.json
+{"command":"addurl","success":false,"error-messages":[],"file":null}
+git-annex: addurl: 1 failed
+
+$> git annex version | head
+git-annex version: 7.20191009-1~ndall+1
+...
+"""]]
+
+the reason for the failure overall is that `datalad` external special remote is not enabled.  So the error message might also be considered misleading -- it is not about "allowing" but rather being "not supported" since AFAIK git-annex doesn't support s3:// urls (see [an existing issue](https://git-annex.branchable.com/todo/Natively_support_s3__58____47____47___urls___40__for_addurl__44___get__44___etc__41__/))
+
+If user convenience was something to strive for here, it should technically be possible for git annex to "sense" all available external special remotes and inquire them (without enabling) either they would support a given url, and then report to the user a more informative error message ("URL ... is not supported by git-annex.  You can initialize one of the available external special remotes which support that url: datalad, somebodies-s3, ...".  It could even check if the special remote is configured but not enabled and then replace "initialize" with "enable" in the message above).
+
+

diff --git a/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_5_380240e562358efb25a8acb65b5d4cc7._comment b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_5_380240e562358efb25a8acb65b5d4cc7._comment
new file mode 100644
index 000000000..d338e184b
--- /dev/null
+++ b/doc/forum/Exactly_what_does_a_v5_to_v7_upgrade_entail__63__/comment_5_380240e562358efb25a8acb65b5d4cc7._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="ghen1"
+ avatar="http://cdn.libravatar.org/avatar/efd0e92b6198291138f0cd7aedbf86b6"
+ subject="comment 5"
+ date="2019-10-30T22:51:27Z"
+ content="""
+I just wanted to report that I've successfully upgraded my repos without any problems. I appreciate everyone's help.
+"""]]

diff --git a/doc/forum/lets_discuss_git_add_behavior/comment_34_404f07cb863eb7e097df0e5def592356._comment b/doc/forum/lets_discuss_git_add_behavior/comment_34_404f07cb863eb7e097df0e5def592356._comment
new file mode 100644
index 000000000..a96e37be3
--- /dev/null
+++ b/doc/forum/lets_discuss_git_add_behavior/comment_34_404f07cb863eb7e097df0e5def592356._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="Dwk"
+ avatar="http://cdn.libravatar.org/avatar/65fade4f1582ef3f00e9ad6ae27dae56"
+ subject="comment 34"
+ date="2019-10-29T21:04:42Z"
+ content="""
+@joey
+
+First, thanks a lot for this solution, which seems to strike a good balance between various use-cases.
+
+Second, I apologize for concluding my comment (which was intended as constructive…) on a too-strongly-worded disclaimer about how I'd felt when upgrading. I thought it would give you grounds for discarding my opinion (if you felt it wasn't well-founded) as that of a someone unthinkingly annoyed at seeing one's workflow changed. I certainly didn't intend to push you around in any way.
+"""]]

diff --git a/doc/forum/lets_discuss_git_add_behavior/comment_33_96fa22163951037d1807d25c961602a8._comment b/doc/forum/lets_discuss_git_add_behavior/comment_33_96fa22163951037d1807d25c961602a8._comment
new file mode 100644
index 000000000..8d6e54407
--- /dev/null
+++ b/doc/forum/lets_discuss_git_add_behavior/comment_33_96fa22163951037d1807d25c961602a8._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="strmd"
+ avatar="http://cdn.libravatar.org/avatar/035707b9756129bbdea6b36a7f7b38d3"
+ subject="comment 33"
+ date="2019-10-29T20:01:01Z"
+ content="""
+@joey,
+
+Thanks for your reply. I certainly appreciate you giving your perspective and rationale, and you bring up several good points. I never intended for my comment to be hostile or harsh, and I'm sorry if it struck you that way.
+
+I do believe you acted in the best of intentions, and going back through the devblog and documentation one can follow along and see how this change came about. So the bit about it being casually communicated, as I wrote, is in one sense absolutely false.
+
+And yet, to me, a somewhat recent user (who still find aspects of git-annex and its inner workings hard to grasp at times), it appeared to happen without warning and came pretty much out of nowhere. As you note, there are several reasons for that, and I'm obviously responsible for making sure I understand the tools I use.
+
+As you have touched upon on your personal blog, there are probably some lessons to learn here. I hope we can turn it into something constructive. And mostly, I hope we as a community can avoid putting you in this regrettable position again.
+
+Finally, the solution you have come up with since my post, seems very, very good and should alleviate most, if not all, concerns. 
+
+"""]]

diff --git a/CHANGELOG b/CHANGELOG
index 4b19b7689..22c69752e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,9 @@
+git-annex (7.20191025) UNRELEASED; urgency=medium
+
+  * benchmark: Add --databases to benchmark sqlite databases.
+
+ -- Joey Hess <id@joeyh.name>  Tue, 29 Oct 2019 15:13:03 -0400
+
 git-annex (7.20191024) upstream; urgency=medium
 
   * Changed git add/git commit -a default behavior back to what it was
diff --git a/Command/Benchmark.hs b/Command/Benchmark.hs
index 95418c1c0..7ecbf338d 100644
--- a/Command/Benchmark.hs
+++ b/Command/Benchmark.hs
@@ -11,10 +11,11 @@ module Command.Benchmark where
 
 import Command
 import Types.Benchmark
-
 #ifdef WITH_BENCHMARK
+import Database.Benchmark
+
 import Criterion.Main
-import Criterion.Main.Options (parseWith, Mode)
+import Criterion.Main.Options (parseWith)
 #endif
 
 cmd :: BenchmarkGenerator -> Command
@@ -23,20 +24,26 @@ cmd generator = command "benchmark" SectionTesting
 	paramNothing
 	(seek generator <$$> optParser)
 
-#ifndef WITH_BENCHMARK
-type Mode = ()
-#endif
-
-data BenchmarkOptions = BenchmarkOptions CmdParams Mode
+data BenchmarkOptions 
+	= BenchmarkOptions CmdParams CriterionMode
+	| BenchmarkDatabases CriterionMode
 
 optParser :: CmdParamsDesc -> Parser BenchmarkOptions
-optParser desc = BenchmarkOptions
-	<$> cmdParams desc
+optParser desc = benchmarkoptions <|> benchmarkdatabases
+  where
+	benchmarkoptions = BenchmarkOptions
+		<$> cmdParams desc
+		<*> criterionopts
+	benchmarkdatabases = BenchmarkDatabases
+		<$> criterionopts
+		<* flag' () 
+			( long "databases"
+			<> help "benchmark sqlite databases"
+			)
 #ifdef WITH_BENCHMARK
-	-- parse criterion's options
-	<*> parseWith defaultConfig
+	criterionopts = parseWith defaultConfig
 #else
-	<*> pure ()
+	criterionopts = pure ()
 #endif
 
 seek :: BenchmarkGenerator -> BenchmarkOptions -> CommandSeek
@@ -44,6 +51,7 @@ seek :: BenchmarkGenerator -> BenchmarkOptions -> CommandSeek
 seek generator (BenchmarkOptions ps mode) = do
 	runner <- generator ps
 	liftIO $ runMode mode [ bench (unwords ps) $ nfIO runner ]
+seek _ (BenchmarkDatabases mode) = benchmarkDbs mode
 #else
 seek _ _ = giveup "git-annex is not built with benchmarking support"
 #endif
diff --git a/Database/Benchmark.hs b/Database/Benchmark.hs
new file mode 100644
index 000000000..7f0929757
--- /dev/null
+++ b/Database/Benchmark.hs
@@ -0,0 +1,120 @@
+{- git-annex database benchmarks
+ -
+ - Copyright 2016-2019 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+{-# LANGUAGE CPP #-}
+{-# LANGUAGE OverloadedStrings #-}
+
+module Database.Benchmark (benchmarkDbs) where
+
+import Annex.Common
+import Types.Benchmark
+#ifdef WITH_BENCHMARK
+import qualified Database.Keys.SQL as SQL
+import qualified Database.Queue as H
+import Database.Init
+import Database.Types
+import Utility.Tmp.Dir
+import Git.FilePath
+import Types.Key
+
+import Criterion.Main
+import Control.Monad.IO.Class (liftIO)
+import qualified Data.ByteString.Char8 as B8
+import System.Random
+#endif
+
+benchmarkDbs :: CriterionMode -> Annex ()
+#ifdef WITH_BENCHMARK
+benchmarkDbs mode = withTmpDirIn "." "benchmark" $ \tmpdir -> do
+	-- benchmark different sizes of databases
+	dbs <- mapM (benchDb tmpdir)
+		[ 1000
+		, 10000
+		-- , 100000
+		]
+	liftIO $ runMode mode
+		[ bgroup "keys database" $ flip concatMap dbs $ \db ->
+			[ getAssociatedFilesHitBench db
+			, getAssociatedFilesMissBench db
+			, getAssociatedKeyHitBench db
+			, getAssociatedKeyMissBench db
+			, addAssociatedFileOldBench db
+			, addAssociatedFileNewBench db
+			]
+		]
+#else
+benchmarkDbs _ = error "not built with criterion, cannot benchmark"
+#endif
+
+#ifdef WITH_BENCHMARK
+
+getAssociatedFilesHitBench :: BenchDb -> Benchmark
+getAssociatedFilesHitBench (BenchDb h num) = bench ("getAssociatedFiles from " ++ show num ++ " (hit)") $ nfIO $ do
+	n <- getStdRandom (randomR (1,num))
+	SQL.getAssociatedFiles (toIKey (keyN n)) (SQL.ReadHandle h)
+
+getAssociatedFilesMissBench :: BenchDb -> Benchmark
+getAssociatedFilesMissBench (BenchDb h num) = bench ("getAssociatedFiles from " ++ show num ++ " (miss)") $ nfIO $
+	SQL.getAssociatedFiles (toIKey keyMiss) (SQL.ReadHandle h)
+
+getAssociatedKeyHitBench :: BenchDb -> Benchmark
+getAssociatedKeyHitBench (BenchDb h num) = bench ("getAssociatedKey from " ++ show num ++ " (hit)") $ nfIO $ do
+	n <- getStdRandom (randomR (1,num))
+	-- fromIKey because this ends up being used to get a Key
+	map fromIKey <$> SQL.getAssociatedKey (fileN n) (SQL.ReadHandle h)
+
+getAssociatedKeyMissBench :: BenchDb -> Benchmark
+getAssociatedKeyMissBench (BenchDb h num) = bench ("getAssociatedKey from " ++ show num ++ " (miss)") $ nfIO $
+	-- fromIKey because this ends up being used to get a Key
+	map fromIKey <$> SQL.getAssociatedKey fileMiss (SQL.ReadHandle h)
+
+addAssociatedFileOldBench :: BenchDb -> Benchmark
+addAssociatedFileOldBench (BenchDb h num) = bench ("addAssociatedFile to " ++ show num ++ " (old)") $ nfIO $ do
+	n <- getStdRandom (randomR (1,num))
+	SQL.addAssociatedFile (toIKey (keyN n)) (fileN n) (SQL.WriteHandle h)
+	H.flushDbQueue h
+
+addAssociatedFileNewBench :: BenchDb -> Benchmark
+addAssociatedFileNewBench (BenchDb h num) = bench ("addAssociatedFile to " ++ show num ++ " (new)") $ nfIO $ do
+	n <- getStdRandom (randomR (1,num))
+	SQL.addAssociatedFile (toIKey (keyN n)) (fileN (n+1)) (SQL.WriteHandle h)
+	H.flushDbQueue h
+
+populateAssociatedFiles :: H.DbQueue -> Int -> IO ()
+populateAssociatedFiles h num = do
+	forM_ [1..num] $ \n ->
+		SQL.addAssociatedFile (toIKey (keyN n)) (fileN n) (SQL.WriteHandle h)
+	H.flushDbQueue h
+
+keyN :: Int -> Key
+keyN n = stubKey
+	{ keyName = B8.pack $ "key" ++ show n
+	, keyVariety = OtherKey "BENCH"
+	}
+
+fileN :: Int -> TopFilePath
+fileN n = asTopFilePath ("file" ++ show n)
+
+keyMiss :: Key
+keyMiss = keyN 0 -- 0 is never stored
+
+fileMiss :: TopFilePath
+fileMiss = fileN 0 -- 0 is never stored
+
+data BenchDb = BenchDb H.DbQueue Int
+
+benchDb :: FilePath -> Int -> Annex BenchDb
+benchDb tmpdir num = do
+	liftIO $ putStrLn $ "setting up database with " ++ show num
+	initDb db SQL.createTables
+	h <- liftIO $ H.openDbQueue H.MultiWriter db SQL.containedTable
+	liftIO $ populateAssociatedFiles h num
+	return (BenchDb h num)
+  where

(Diff truncated)

diff --git a/doc/todo/lockdown_hooks/comment_3_f57299385578914f5bca05f06b76cd46._comment b/doc/todo/lockdown_hooks/comment_3_f57299385578914f5bca05f06b76cd46._comment
new file mode 100644
index 000000000..906166d17
--- /dev/null
+++ b/doc/todo/lockdown_hooks/comment_3_f57299385578914f5bca05f06b76cd46._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="ACLs keep fighting back"
+ date="2019-10-29T18:03:57Z"
+ content="""
+I am getting back to tackle a datalad/git-annex usecase on one of such (nfs4 ACL) usecases, so if hooks are still the way to go, I will be ready to test ;-)
+"""]]

diff --git a/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment b/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment
deleted file mode 100644
index dc7213b68..000000000
--- a/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment
+++ /dev/null
@@ -1,7 +0,0 @@
-[[!comment format=mdwn
- username="Simon"
- subject="What is a recent enough version?"
- date="2019-10-29T15:51:54Z"
- content="""
-It seems Ubuntu ships with git-annex version: 7.20190129 and that doesn't work.
-"""]]

diff --git a/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment b/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment
new file mode 100644
index 000000000..dc7213b68
--- /dev/null
+++ b/doc/tips/storing_data_in_git-lfs/comment_1_6f79fbe153bb928b00da6cf57a16d567._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="Simon"
+ subject="What is a recent enough version?"
+ date="2019-10-29T15:51:54Z"
+ content="""
+It seems Ubuntu ships with git-annex version: 7.20190129 and that doesn't work.
+"""]]