Recent changes to this wiki:

update
diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_23_cedee281d3f6f97499e70dacd0560f70._comment b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_23_cedee281d3f6f97499e70dacd0560f70._comment
new file mode 100644
index 0000000000..285a3ed8cb
--- /dev/null
+++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_23_cedee281d3f6f97499e70dacd0560f70._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 23"""
+ date="2022-01-21T17:04:11Z"
+ content="""
+v9 is implemented, and enables filter-process. Not yet the default, but
+will be eventually.
+"""]]

enable filter.annex.process in v9
This has tradeoffs, but is generally a win, and users who it causes git add to
slow down unacceptably for can just disable it again.
It needed to happen in an upgrade, since there are git-annex versions
that do not support it, and using such an old version with a v8
repository with filter.annex.process set will cause bad behavior.
By enabling it in v9, it's guaranteed that any git-annex version that
can use the repository does support it. Although, this is not a perfect
protection against problems, since an old git-annex version, if it's
used with a v9 repository, will cause git add to try to run
git-annex filter-process, which will fail. But at least, the user is
unlikely to have an old git-annex in path if they are using a v9
repository, since it won't work in that repository.
Sponsored-by: Dartmouth College's Datalad project
diff --git a/Annex/Version.hs b/Annex/Version.hs
index 944dc14c6f..7f604fefc9 100644
--- a/Annex/Version.hs
+++ b/Annex/Version.hs
@@ -57,6 +57,11 @@ setVersion (RepoVersion v) = setConfig versionField (show v)
 removeVersion :: Annex ()
 removeVersion = unsetConfig versionField
 
+versionSupportsFilterProcess :: Maybe RepoVersion -> Bool
+versionSupportsFilterProcess (Just v) 
+	| v >= RepoVersion 9 = True
+versionSupportsFilterProcess _ = False
+
 versionNeedsWritableContentFiles :: Maybe RepoVersion -> Bool
 versionNeedsWritableContentFiles (Just v) 
 	| v >= RepoVersion 10 = False
diff --git a/CHANGELOG b/CHANGELOG
index b5893ecce9..8ff21c3bb9 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -9,6 +9,11 @@ git-annex (10.20220121) UNRELEASED; urgency=medium
     upgrade, in order to allow time for any old git-annex processes that
     are not aware of the locking change to finish. Or git-annex upgrade
     can be used to upgrade to v10 immediately.
+  * In v9, set filter.annex.process. This makes git add/checkout faster when
+    there are a lot of unlocked annexed files or non-annexed files, but can
+    also makes git add of large files to the annex somewhat slower.
+    If this tradeoff does not work for your use case, you can still unset
+    filter.annex.process.
   * export: When a non-annexed symlink is in the tree to be exported, skip it.
   * import: When the previously exported tree contained a non-annexed symlink,
     preserve it in the imported tree so it does not get deleted.
diff --git a/Config/Smudge.hs b/Config/Smudge.hs
index e124125b65..3a33bcfda3 100644
--- a/Config/Smudge.hs
+++ b/Config/Smudge.hs
@@ -1,6 +1,6 @@
 {- Git smudge filter configuration
  -
- - Copyright 2011-2019 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -16,6 +16,7 @@ import qualified Git.Command
 import Git.Types
 import Config
 import Utility.Directory.Create
+import Annex.Version
 
 import qualified System.FilePath.ByteString as P
 
@@ -32,6 +33,8 @@ configureSmudgeFilter = unlessM (fromRepo Git.repoIsLocalBare) $ do
 
 	setConfig (ConfigKey "filter.annex.smudge") "git-annex smudge -- %f"
 	setConfig (ConfigKey "filter.annex.clean") "git-annex smudge --clean -- %f"
+	whenM (versionSupportsFilterProcess <$> getVersion)
+		configureSmudgeFilterProcess
 	lf <- Annex.fromRepo Git.attributesLocal
 	gf <- Annex.fromRepo Git.attributes
 	lfs <- readattr lf
@@ -43,6 +46,10 @@ configureSmudgeFilter = unlessM (fromRepo Git.repoIsLocalBare) $ do
   where
 	readattr = liftIO . catchDefaultIO "" . readFileStrict . fromRawFilePath
 
+configureSmudgeFilterProcess :: Annex ()
+configureSmudgeFilterProcess =
+	setConfig (ConfigKey "filter.annex.process") "git-annex filter-process"
+
 stdattr :: [String]
 stdattr =
 	[ "* filter=annex"
diff --git a/Upgrade/V8.hs b/Upgrade/V8.hs
index f10333cbc9..57f4951dbb 100644
--- a/Upgrade/V8.hs
+++ b/Upgrade/V8.hs
@@ -9,10 +9,13 @@ module Upgrade.V8 where
 
 import Annex.Common
 import Types.Upgrade
+import Config.Smudge
 
 upgrade :: Bool -> Annex UpgradeResult
 upgrade automatic = do
 	unless automatic $
 		showAction "v8 to v9"
 
+	configureSmudgeFilterProcess
+
 	return UpgradeSuccess
diff --git a/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn b/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
index 3ff8d873c9..a8e4fee4e5 100644
--- a/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
+++ b/doc/todo/git_smudge_clean_interface_suboptiomal.mdwn
@@ -17,7 +17,8 @@ to git. git-lfs uses it that way.
 The first problem with the interface was that it ran a command once per
 file. This was later fixed by extending it to support long-running filter
 processes, which git-lfs uses. git-annex can also use that interface,
-when `git-annex filter-process` is enabled, but it does not by default.
+when `git-annex filter-process` is enabled. That is the case in v9
+repositories and above.
 
 A second problem with the interface, which affects git-lfs AFAIK, is that
 git buffers the output of the smudge filter in memory before updating the
@@ -81,12 +82,12 @@ And here's the consequences of git-annex's workarounds:
 * It doesn't use the long-running filter process interface by default, 
   so `git add` of a lot of files runs `git-annex smudge --clean` once per file,
   which is slower than it could be. Using `git-annex add` avoids this problem.
-  So does enabling `git-annex filter-process`.
+  So does enabling `git-annex filter-process`, which is default in v9.
 
 * After a git-annex get/drop or a git checkout or pull that affects a lot
   of files, the clean filter gets run once per file, which is again, slower
   than ideal. Enabling `git-annex filter-process` can speed this up
-  in some cases.
+  in some cases, and is default in v9.
 
 * When `git-annex filter-process` is enabled, it cannot use the trick
   described above that `git-annex smudge --clean` uses to avoid git
diff --git a/doc/todo/incremental_hashing_for_add.mdwn b/doc/todo/incremental_hashing_for_add.mdwn
index 3708780964..fae818593a 100644
--- a/doc/todo/incremental_hashing_for_add.mdwn
+++ b/doc/todo/incremental_hashing_for_add.mdwn
@@ -1,7 +1,7 @@
-When `git-annex filter-process` is enabled, `git add` pipes the content of
-files into it, but that's thrown away, and the file is read again by git-annex
-to generate a hash. It would improve performance to hash the content
-provided via the pipe.
+When `git-annex filter-process` is enabled (v9 and above), `git add` pipes
+the content of files into it, but that's thrown away, and the file is read
+again by git-annex to generate a hash. It would improve performance to hash
+the content provided via the pipe.
 
 When filter-process is not enabled, `git-annex smudge --clean` reads
 the file to hash it, then reads it a second time to copy it into
diff --git a/doc/todo/v9_changes.mdwn b/doc/todo/v9_changes.mdwn
index 87ac7b39f7..4b242359f1 100644
--- a/doc/todo/v9_changes.mdwn
+++ b/doc/todo/v9_changes.mdwn
@@ -18,3 +18,5 @@ could change and if it does, these things could be included.
   seem worth it.
 
   May want to implement [[incremental_hashing_for_add]] first.
+
+[[done]] --[[Joey]]

close
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn
index 5508e37979..b0146468f3 100644
--- a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn
@@ -29,3 +29,4 @@ I guess I need to annex fsck tons of repositories now :-/
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
 
+> [[fixed|done]] in git-annex v10. --[[Joey]]
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_13_43a09de767ed9060dd91d6eca5e0b3a3._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_13_43a09de767ed9060dd91d6eca5e0b3a3._comment
new file mode 100644
index 0000000000..5c5d08ccf7
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_13_43a09de767ed9060dd91d6eca5e0b3a3._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 13"""
+ date="2022-01-21T16:59:49Z"
+ content="""
+v10 is implemented, so upgrade your affected repositories to it to get the
+fix.
+"""]]

update, v9 upgrade not yet automatic
and v9 does not have full upgrade locking
diff --git a/doc/upgrades.mdwn b/doc/upgrades.mdwn
index 794c7464bc..836f922b85 100644
--- a/doc/upgrades.mdwn
+++ b/doc/upgrades.mdwn
@@ -61,11 +61,11 @@ it very unlikely that such a process is still running.
 
 ## v8 -> v9 (git-annex version 10.x)
 
-v8 repositories are automatically upgraded to v9.
+v8 repositories are not yet automatically upgraded to v9.
 
-Starting in v9, upgrades to a repository are prevented while another
-git-annex command is running. That is needed for the v10 repository
-upgrade to be performed safely.
+v9 is a stepping stone to the v10 upgrade. By adding this intermediate
+version, old versions of git-annex that only support v8 will not be able to
+start new processes in the repository after the v9 upgrade.
 
 ## v7 -> v8 (git-annex version 8.x)
 

v10 upgrade locking
The v10 upgrade should almost be safe now. What remains to be done is
notice when the v10 upgrade has occurred, while holding the shared lock,
and switch to using v10 lock files.
Sponsored-by: Dartmouth College's Datalad project
diff --git a/Annex/Content.hs b/Annex/Content.hs
index dafc94c503..4895b9b182 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -188,9 +188,8 @@ winLocker _ _ Nothing = (return Nothing, Nothing)
  - the content is not present, because the content file is not always
  - the file that is locked. -}
 lockContentUsing :: ContentLocker -> Key -> Annex a -> Annex a -> Annex a
-lockContentUsing contentlocker key fallback a = do
+lockContentUsing contentlocker key fallback a = withContentLockFile key $ \mlockfile -> do
 	contentfile <- calcRepo (gitAnnexLocation key)
-	mlockfile <- contentLockFile key
 	let (locker, sharedtoexclusive) = contentlocker contentfile mlockfile
 	bracket
 		(lock locker mlockfile)
diff --git a/Annex/Content/Presence.hs b/Annex/Content/Presence.hs
index df57716220..67c430efbc 100644
--- a/Annex/Content/Presence.hs
+++ b/Annex/Content/Presence.hs
@@ -1,6 +1,6 @@
 {- git-annex object content presence
  -
- - Copyright 2010-2021 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -17,14 +17,16 @@ module Annex.Content.Presence (
 	isUnmodified,
 	isUnmodified',
 	isUnmodifiedCheap,
-	contentLockFile,
+	withContentLockFile,
 ) where
 
 import Annex.Content.Presence.LowLevel
 import Annex.Common
 import qualified Annex
 import Annex.LockPool
+import Annex.LockFile
 import Annex.Version
+import Types.RepoVersion
 import qualified Database.Keys
 import Annex.InodeSentinal
 import Utility.InodeCache
@@ -77,7 +79,7 @@ inAnnexSafe key = inAnnex' (fromMaybe True) (Just False) go key
 	is_unlocked = Just True
 	is_missing = Just False
 
-	go contentfile = flip checklock contentfile =<< contentLockFile key
+	go contentfile = withContentLockFile key $ flip checklock contentfile
 
 #ifndef mingw32_HOST_OS
 	checklock Nothing contentfile = checkOr is_missing contentfile
@@ -116,20 +118,36 @@ inAnnexSafe key = inAnnex' (fromMaybe True) (Just False) go key
 			)
 #endif
 
-contentLockFile :: Key -> Annex (Maybe RawFilePath)
+{- Runs an action with the lock file to use to lock a key's content.
+ - When the content file itself should be locked, runs the action with
+ - Nothing.
+ -
+ - In v9 and below, while the action is running, a shared lock is held of the
+ - gitAnnexContentLockLock. That prevents the v10 upgrade, which changes how
+ - content locking works, from running at the same time as content is locked
+ - using the old method.
+ -}
+withContentLockFile :: Key -> (Maybe RawFilePath -> Annex a) -> Annex a
+withContentLockFile k a = do
+	v <- getVersion
+	let go = contentLockFile k v >>= a
+	if versionNeedsWritableContentFiles v
+		then withSharedLock gitAnnexContentLockLock go
+		else go
+
+contentLockFile :: Key -> Maybe RepoVersion -> Annex (Maybe RawFilePath)
 #ifndef mingw32_HOST_OS
 {- Older versions of git-annex locked content files themselves, but newer
  - versions use a separate lock file, to better support repos shared
  - amoung users in eg a group. -}
-contentLockFile key = ifM (versionNeedsWritableContentFiles <$> getVersion)
-	( pure Nothing
-	, Just <$> calcRepo (gitAnnexContentLock key)
-	)
+contentLockFile key v
+	| versionNeedsWritableContentFiles v = pure Nothing
+	| otherwise = Just <$> calcRepo (gitAnnexContentLock key)
 #else
 {- Windows always has to use a separate lock file from the content, since
  - locking the actual content file would interfere with the user's
  - use of it. -}
-contentLockFile key = Just <$> calcRepo (gitAnnexContentLock key)
+contentLockFile key _ = Just <$> calcRepo (gitAnnexContentLock key)
 #endif
 
 {- Performs an action, passing it the location to use for a key's content. -}
diff --git a/Annex/Locations.hs b/Annex/Locations.hs
index f782efb927..3a3933ea4c 100644
--- a/Annex/Locations.hs
+++ b/Annex/Locations.hs
@@ -20,6 +20,7 @@ module Annex.Locations (
 	gitAnnexLink,
 	gitAnnexLinkCanonical,
 	gitAnnexContentLock,
+	gitAnnexContentLockLock,
 	gitAnnexInodeSentinal,
 	gitAnnexInodeSentinalCache,
 	annexLocationsBare,
@@ -239,6 +240,11 @@ gitAnnexContentLock key r config = do
 	loc <- gitAnnexLocation key r config
 	return $ loc <> ".lck"
 
+{- Lock that is held when taking the gitAnnexContentLock to support the v10
+ - upgrade. -}
+gitAnnexContentLockLock :: Git.Repo -> RawFilePath
+gitAnnexContentLockLock r = gitAnnexDir r P.</> "content.lck"
+
 gitAnnexInodeSentinal :: Git.Repo -> RawFilePath
 gitAnnexInodeSentinal r = gitAnnexDir r P.</> "sentinal"
 
diff --git a/Upgrade/V8.hs b/Upgrade/V8.hs
index 0f3280689d..f10333cbc9 100644
--- a/Upgrade/V8.hs
+++ b/Upgrade/V8.hs
@@ -9,7 +9,6 @@ module Upgrade.V8 where
 
 import Annex.Common
 import Types.Upgrade
-import Utility.Daemon
 
 upgrade :: Bool -> Annex UpgradeResult
 upgrade automatic = do
diff --git a/Upgrade/V9.hs b/Upgrade/V9.hs
index 43822ebbd9..029cd74fd5 100644
--- a/Upgrade/V9.hs
+++ b/Upgrade/V9.hs
@@ -11,6 +11,8 @@ import Annex.Common
 import Types.Upgrade
 import Annex.Content
 import Annex.Perms
+import Annex.LockFile
+import Annex.Version
 import Git.ConfigTypes
 import Types.RepoVersion
 import Logs.Upgrade
@@ -21,12 +23,12 @@ import Data.Time.Clock.POSIX
 upgrade :: Bool -> Annex UpgradeResult
 upgrade automatic
 	| automatic = do
-		-- For automatic upgrade, wait until a year after the v9
-		-- upgrade. This is to give time for any old processes
-		-- that were running before the v9 upgrade to finish.
-		-- Such old processes lock content using the old method,
-		-- and it is not safe for such to still be running after
-		-- this upgrade.
+		{- For automatic upgrade, wait until a year after the v9
+		 - upgrade. This is to give time for any old processes
+		 - that were running before the v9 upgrade to finish.
+		 - Such old processes lock content using the old method,
+		 - and it is not safe for such to still be running after
+		 - this upgrade. -}
 		timeOfUpgrade (RepoVersion 9) >>= \case
 			Nothing -> performUpgrade automatic
 			Just t -> do
@@ -37,9 +39,9 @@ upgrade automatic
 						performUpgrade automatic
 	| otherwise = performUpgrade automatic
   where
-	-- Skip upgrade when git-annex assistant (or watch) is running,
-	-- because these are long-running daemons that could conceivably
-	-- run for an entire year.
+	{- Skip upgrade when git-annex assistant (or watch) is running,
+	 - because these are long-running daemons that could conceivably
+	 - run for an entire year and so predate the v9 upgrade. -}
 	checkassistantrunning a = do
 		pidfile <- fromRepo gitAnnexPidFile
 		liftIO (checkDaemon (fromRawFilePath pidfile)) >>= \case
@@ -50,19 +52,28 @@ performUpgrade :: Bool -> Annex UpgradeResult
 performUpgrade automatic = do
 	unless automatic $
 		showAction "v9 to v10"
+	
+	{- Take a lock to ensure that there are no other git-annex
+	 - processes running that are using the old content locking method. -}
+	withExclusiveLock gitAnnexContentLockLock $ do
+		{- When core.sharedRepository is set, object files
+		 - used to have their write bits set. That can now be
+		 - removed, if the user the upgrade is running as has
+		 - permission to remove it.
+		 - (Otherwise, a later fsck will fix up the permissions.) -}
+		withShared $ \sr -> case sr of
+			GroupShared -> removewrite sr
+			AllShared -> removewrite sr
+			_ -> return ()
 
-	{- When core.sharedRepository is set, object files
-	 - used to have their write bits set. That can now be removed,
-	 - if the user the upgrade is running as has permission to remove
-	 - it. (Otherwise, a later fsck will fix up the permissions.) -}

(Diff truncated)
diff --git a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn
new file mode 100644
index 0000000000..664a808bfa
--- /dev/null
+++ b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn
@@ -0,0 +1,132 @@
+Under very particular conditions a `git annex drop --force --all` fails in a bare repository on windows with:
+
+```
+git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
+```
+
+(`<BASEDIR>` and `<REPODIR>` are put by me to improve readability here and below).
+
+The key reported as "access denied" is in fact present and has the "read-only" flag set.
+
+What makes this strange is that I can reliably trigger this error, or avoid it, with small differences in the command sequence -- differences that, to me, should not impact the ability to drop a key.
+
+Below I am posting two complete command sequences (and the diff between them) for triggering and not triggering the error.
+I edited the logs to make all matching paths uniform, for readability again. The logs are produced by the same unittest under the same runtime conditions. Both logs start by initializing a git repo in a previously non-existing directory. In both cases a `type=directory` special remote is initialized, but once with `exporttree=yes` and once without. The only other difference is that in the `exporttree=yes` case a custom tree is written to the repository (without creating a commit or advancing a branch), and a "fake" `export.log` record is created that claims that this custom tree was exported from "here" to the initialized directory special remote (this change is committed to the git-annex branch). Afterwards the sequence is identical again.
+
+The failure occurs in the simple case, *without* the fake-export, and only on windows. Both scenarios work on linux and macos.
+
+I am using git annex version 8.20211117-gc3af94eff and git version 2.34.1.windows.1 on an up-to-date win10 box. But I am also seeing this behavior on a windows appveyor CI runner (windows 10.0.17763) with 2.33.1.windows.1.
+
+### Please provide any additional information below.
+
+[[!format sh """
+
+# drop-failure
+
+[DBG] Initialize empty Git repository at '<BASEDIR>\<REPODIR>' ['--bare']
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-C', '<BASEDIR>\\<REPODIR>', 'init', '--bare'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'config', 'annex.private', 'true'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'init'] (cwd=<BASEDIR>\<REPODIR>)
+[Level 11] stderr|   Detected a filesystem without fifo support.
+[Level 11] stderr|   Disabling ssh connection caching.
+[Level 11] stderr|   Detected a crippled filesystem.
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 1
+fatal: CommandError(CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "annex" "drop" "--force" "--all" "-c" "annex.dotfiles=true"' failed with exitcode 1 under <BASEDIR>\<REPODIR> [out: 'drop XDLRA--refs
+failed
+drop XDLRA--refs
+failed'] [err: 'git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
+git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
+drop: 2 failed'])
+
+# drop-success / exporttree=yes
+
+[DBG] Initialize empty Git repository at '<BASEDIR>\<REPODIR>' ['--bare']
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-C', '<BASEDIR>\\<REPODIR>', 'init', '--bare'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'config', 'annex.private', 'true'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'init'] (cwd=<BASEDIR>\<REPODIR>)
+[Level 11] stderr|   Detected a filesystem without fifo support.
+[Level 11] stderr|   Disabling ssh connection caching.
+[Level 11] stderr|   Detected a crippled filesystem.
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', 'exporttree=yes', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', 'a4725c5396b9e1abbb67ad1531b30ba5902ccf02', '.datalad/dotgit/refs'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', '6e67546efb8b9f1a0ded02f07221586c6a93f292', '.datalad/dotgit/repo.zip'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'read-tree', 'git-annex'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '100644', '8db4da90bf1b3cd4e9d0b6d079532696e71c1998', 'export.log'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'commit-tree', '-m', 'Fake export', '-p', 'git-annex', '0c8af6e758e0d66d3d0d041c7f3f73f79f0dcab9'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'update-ref', 'refs/heads/git-annex', '9dbd3377a4cca0cded93986a83bb025a0ac15b66'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DATALAD-ANNEX] drop XDLRA--refs ok
+
+"""]]
+
+And here is the diff between the two sequences  (failure(-) vs success(+)
+
+[[!format diff """
+
+--- annex_remote.txt    2022-01-19 21:25:57.025990250 +0100
++++ export_remote.txt   2022-01-19 21:29:57.262464761 +0100
+@@ -8,17 +8,23 @@
+ [Level 11] stderr|   Detected a crippled filesystem.
+ [DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+ [DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
+-[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', 'exporttree=yes', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+ [DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', 'a4725c5396b9e1abbb67ad1531b30ba5902ccf02', '.datalad/dotgit/refs'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', '6e67546efb8b9f1a0ded02f07221586c6a93f292', '.datalad/dotgit/repo.zip'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'read-tree', 'git-annex'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '100644', '8db4da90bf1b3cd4e9d0b6d079532696e71c1998', 'export.log'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', 'commit-tree', '-m', 'Fake export', '-p', 'git-annex', '0c8af6e758e0d66d3d0d041c7f3f73f79f0dcab9'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'update-ref', 'refs/heads/git-annex', '9dbd3377a4cca0cded93986a83bb025a0ac15b66'] (cwd=<BASEDIR>\<REPODIR>)
+ [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
+ [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+ [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+ [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+ [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+-[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 1
+-fatal: CommandError(CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "annex" "drop" "--force" "--all" "-c" "annex.dotfiles=true"' failed with exitcode 1 under <BASEDIR>\<REPODIR> [out: 'drop XDLRA--refs
+-failed
+-drop XDLRA--refs
+-failed'] [err: 'git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
+-git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
+-drop: 2 failed'])
++[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
++[DATALAD-ANNEX] drop XDLRA--refs ok
+
+
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+This observation comes from an effort to implement a git-remote-helper that uses git-annex to enable fetch/push to and from any location that can be reached by any special remote. The fact that git-annex can be used to build things like this is just crazy cool! Thx much!
+
+[[!tag projects/datalad]]

update on status
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment
index 2ddaccd98e..6e11790e16 100644
--- a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment
@@ -5,22 +5,23 @@
  content="""
 This is going to need two repository version bumps:
 
-v9: Add the upgrade lock file, and all git-annex processes take a shared
-lock to avoid the repository being upgraded out from under them. 
-Upgrade is skipped when assistant is running.
+v9:  Add the upgrade lock file, and all git-annex processes take a shared
+lock to avoid the repository being upgraded out from under them.
 
 v10: Skipped until the upgrade lock file is of a certain age. Take upgrade
-lock before increasing annex.version.
-In v10, stop locking content files and lock separate lock files.
+lock before upgrading.
 
-This way, an old version of git-annex cannot be used in the v9 repository,
-and so the v10 upgrade only needs to worry about any git-annex processes
-that were started in v8.
+In v10, stop locking content files and lock separate lock files.
 
-The age could be eg 1 month, which assumes that no old git-annex process
-like `git-annex move --to remote` is still running after that long.
+The age could be eg 1 month, which assumes that no pre-v9 git-annex
+process like `git-annex move --to remote` is still running after that long.
 Of course, that is still an assumption, but it can be pushed out as long as
-it takes to feel comfortable with it. Maybe 1 year?
+it takes to feel comfortable with it. Maybe 1 year? The only disadvantage
+really is that any v11 upgrade would also get deferred.
+
+Since the assistant can possibly run for longer than a year without
+restarting, the v10 upgrade would need to be skipped when the assistant is
+running.
 
 `git-annex upgrade --version=10` could be available to speed up that
 upgrade. The user would be responsible for making sure there are no such
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_12_fc10e403f35f47a6c4464f3f68d01eca._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_12_fc10e403f35f47a6c4464f3f68d01eca._comment
new file mode 100644
index 0000000000..4da9359828
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_12_fc10e403f35f47a6c4464f3f68d01eca._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 12"""
+ date="2022-01-19T19:55:10Z"
+ content="""
+What kind of locking is needed for the v10 upgrade? Everything else is in
+place now, but the actual locking is TBD.
+
+My plan above calls for a way to detect if another git-annex process
+(v9 or above) is running in the repo. That will be hard to implement
+though.. 
+
+It might cause problems with annex.pidlock, if every git-annex
+process starts taking a shared lock, because pidlock does not support
+shared locks, so only 1 git-annex process will be able to run, perhaps in
+situations where multiples can run now even with pidlocking because no
+locking is needed.
+
+Also, the existing locking machinery runs in the Annex monad, but such a
+lock needs to be implemented in Annex.hs itself, so that would be a
+recursive dep. And, it would add overhead to every git-annex process.
+(A small amount.)
+
+Alternatively, there could be a top-level lock file that is held shared
+whenever locking content files. And the v10 upgrade takes an exclusive
+lock. But this seems to fail when a v9 process is running -- if it blocks
+on the shared lock for the v10 upgrade, it would still go on the lock in v9
+mode in the now v10 repository.
+"""]]

split upgrade into v9 and v10
v10 will run 1 year after the upgrade to v9, to give time for any v8
processes to die. Until that point, the v10 upgrade will be tried by
every process but deferred, so added support for deferring upgrades.
The upgrade prevention lock file that will be used by v10 is not yet
implemented, so it does not yet defer.
Sponsored-by: Dartmouth College's Datalad project
diff --git a/Annex/Version.hs b/Annex/Version.hs
index 27a4cb832b..c8c47f84f2 100644
--- a/Annex/Version.hs
+++ b/Annex/Version.hs
@@ -19,19 +19,19 @@ import qualified Annex
 import qualified Data.Map as M
 
 defaultVersion :: RepoVersion
-defaultVersion = RepoVersion 8
+defaultVersion = RepoVersion 10
 
 latestVersion :: RepoVersion
-latestVersion = RepoVersion 9
+latestVersion = RepoVersion 10
 
 supportedVersions :: [RepoVersion]
-supportedVersions = map RepoVersion [8, 9]
+supportedVersions = map RepoVersion [8, 9, 10]
 
 upgradeableVersions :: [RepoVersion]
 #ifndef mingw32_HOST_OS
-upgradeableVersions = map RepoVersion [0..8]
+upgradeableVersions = map RepoVersion [0..10]
 #else
-upgradeableVersions = map RepoVersion [2..8]
+upgradeableVersions = map RepoVersion [2..10]
 #endif
 
 autoUpgradeableVersions :: M.Map RepoVersion RepoVersion
@@ -41,6 +41,8 @@ autoUpgradeableVersions = M.fromList
 	, (RepoVersion 5, latestVersion)
 	, (RepoVersion 6, latestVersion)
 	, (RepoVersion 7, latestVersion)
+	, (RepoVersion 8, latestVersion) 
+	, (RepoVersion 9, latestVersion) 
 	]
 
 versionField :: ConfigKey
@@ -57,5 +59,5 @@ removeVersion = unsetConfig versionField
 
 versionNeedsWritableContentFiles :: Maybe RepoVersion -> Bool
 versionNeedsWritableContentFiles (Just v) 
-	| v >= RepoVersion 9 = False
+	| v >= RepoVersion 10 = False
 versionNeedsWritableContentFiles _ = True
diff --git a/Types/Upgrade.hs b/Types/Upgrade.hs
new file mode 100644
index 0000000000..0249437443
--- /dev/null
+++ b/Types/Upgrade.hs
@@ -0,0 +1,10 @@
+{- git-annex upgrade types
+ -
+ - Copyright 2022 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+module Types.Upgrade where
+
+data UpgradeResult = UpgradeSuccess | UpgradeFailed | UpgradeDeferred
diff --git a/Upgrade.hs b/Upgrade.hs
index d87a7b0302..c8a7c7a748 100644
--- a/Upgrade.hs
+++ b/Upgrade.hs
@@ -1,6 +1,6 @@
 {- git-annex upgrade support
  -
- - Copyright 2010-2020 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -10,6 +10,7 @@
 module Upgrade where
 
 import Annex.Common
+import Types.Upgrade
 import qualified Annex
 import qualified Git
 import Config
@@ -27,6 +28,7 @@ import qualified Upgrade.V5
 import qualified Upgrade.V6
 import qualified Upgrade.V7
 import qualified Upgrade.V8
+import qualified Upgrade.V9
 
 import qualified Data.Map as M
 
@@ -63,25 +65,27 @@ needsUpgrade v
 
 upgrade :: Bool -> RepoVersion -> Annex Bool
 upgrade automatic destversion = do
-	upgraded <- go =<< getVersion
-	when upgraded
-		postupgrade
+	(upgraded, newversion) <- go =<< getVersion
+	when upgraded $
+		postupgrade newversion
 	return upgraded
   where
 	go (Just v)
-		| v >= destversion = return True
+		| v >= destversion = return (True, Just v)
 		| otherwise = ifM upgradingRemote
 			( upgraderemote
-			, ifM (up v)
-				( go (Just (RepoVersion (fromRepoVersion v + 1)))
-				, return False
-				)
+			, up v >>= \case
+				UpgradeSuccess -> go (Just (incrversion v) )
+				UpgradeFailed -> return (False, Just v)
+				UpgradeDeferred -> return (True, Just v)
 			)
-	go _ = return True
+	go Nothing = return (True, Nothing)
 
-	postupgrade = ifM upgradingRemote
+	incrversion v = RepoVersion (fromRepoVersion v + 1)
+
+	postupgrade newversion = ifM upgradingRemote
 		( reloadConfig
-		, setVersion destversion
+		, maybe noop setVersion newversion
 		)
 
 #ifndef mingw32_HOST_OS
@@ -98,7 +102,8 @@ upgrade automatic destversion = do
 	up (RepoVersion 6) = Upgrade.V6.upgrade automatic
 	up (RepoVersion 7) = Upgrade.V7.upgrade automatic
 	up (RepoVersion 8) = Upgrade.V8.upgrade automatic
-	up _ = return True
+	up (RepoVersion 9) = Upgrade.V9.upgrade automatic
+	up _ = return UpgradeDeferred
 
 	-- Upgrade local remotes by running git-annex upgrade in them.
 	-- This avoids complicating the upgrade code by needing to handle
@@ -111,8 +116,8 @@ upgrade automatic destversion = do
 			]
 			(\p -> p { cwd = Just rp })
 			(\_ _ _ pid -> waitForProcess pid >>= return . \case
-				ExitSuccess -> True
-				_ -> False
+				ExitSuccess -> (True, Nothing)
+				_ -> (False, Nothing)
 			)
 
 upgradingRemote :: Annex Bool
diff --git a/Upgrade/V0.hs b/Upgrade/V0.hs
index ca8601e224..3e6d9e6202 100644
--- a/Upgrade/V0.hs
+++ b/Upgrade/V0.hs
@@ -8,10 +8,11 @@
 module Upgrade.V0 where
 
 import Annex.Common
+import Types.Upgrade
 import Annex.Content
 import qualified Upgrade.V1
 
-upgrade :: Annex Bool
+upgrade :: Annex UpgradeResult
 upgrade = do
 	showAction "v0 to v1"
 
diff --git a/Upgrade/V1.hs b/Upgrade/V1.hs
index bb2abd743d..107db03615 100644
--- a/Upgrade/V1.hs
+++ b/Upgrade/V1.hs
@@ -17,6 +17,7 @@ import qualified Data.ByteString.Lazy as L
 import qualified System.FilePath.ByteString as P
 
 import Annex.Common
+import Types.Upgrade
 import Annex.Content
 import Annex.Link
 import Annex.Perms
@@ -53,7 +54,7 @@ import qualified Upgrade.V2
 -- Something similar to the migrate subcommand could be used,
 -- and users could then run that at their leisure.
 
-upgrade :: Annex Bool
+upgrade :: Annex UpgradeResult
 upgrade = do
 	showAction "v1 to v2"
 	
diff --git a/Upgrade/V2.hs b/Upgrade/V2.hs
index 6ba538634a..13bd191cb2 100644
--- a/Upgrade/V2.hs
+++ b/Upgrade/V2.hs
@@ -8,6 +8,7 @@
 module Upgrade.V2 where
 
 import Annex.Common
+import Types.Upgrade
 import qualified Git
 import qualified Git.Command
 import qualified Git.Ref
@@ -38,7 +39,7 @@ olddir g

(Diff truncated)
commeent
diff --git a/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment b/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment
new file mode 100644
index 0000000000..bc6b5fbdd7
--- /dev/null
+++ b/doc/tips/using_nested_git_repositories/comment_2_63f30b652c0cbdb0acf6745891f9f09e._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-19T15:51:17Z"
+ content="""
+I agree, submodules are the usual way to nest git repositories, and will
+more or less just work with git-annex.
+
+I think that the author of this tip is wanting to version control the
+contents of `.git` itself. Eg, to version control `.git/config` and 
+`.git/hooks/`.
+
+One problem with this approach is that when the outer repository has
+"dotgit/annex/objects/` files added to it, running `git-annex drop` inside
+the nested git repository will drop the content, but the outer repository
+will still contain a copy too. You would have to use `git-annex unused`
+to eventually clean up those copies. And it stores 2 copies of every
+annexed file to use it this way.
+"""]]

close
diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
index 1bfd4b6859..758406251a 100644
--- a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
+++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
@@ -18,4 +18,5 @@ conda linux nodep (standalone) build 8.20211012-geb95ed486 and then with another
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
 
-[[!tag moreinfo]]
+> [[done]] provisionally, followup if further testing shows otherwise
+> --[[Joey]]

Added a comment: Now it's working normally again
diff --git a/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment
new file mode 100644
index 0000000000..fc65820127
--- /dev/null
+++ b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update/comment_1_68429f9e44ce24250c608da8ffffe643._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="ainohzoa"
+ avatar="http://cdn.libravatar.org/avatar/0d6a2dbd95f6c4f410cc41d32beaebe9"
+ subject="Now it's working normally again"
+ date="2022-01-14T21:10:13Z"
+ content="""
+After running a `git status` today (the first time running it was slow as usual) the problem seems to have gone away. Perhaps it just needed to rebuild some kind of cache or something. I believe that I had waited for a `git status` command to completely finish before, so maybe it takes several tries to resolve this performance problem? Or possibly I'm misremembering and I never actually waited for the `git status` command to fully finish. I'm baffled by this, but happy that it's back to normal. I'll post an update if the problem comes back. 
+"""]]

Added a comment: nested git repositories are git submodules
diff --git a/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment b/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment
new file mode 100644
index 0000000000..92c56b1916
--- /dev/null
+++ b/doc/tips/using_nested_git_repositories/comment_1_cd7f673de1c39d2f37d9958357d5e72d._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="https://christian.amsuess.com/chrysn"
+ nickname="chrysn"
+ avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc"
+ subject="nested git repositories are git submodules"
+ date="2022-01-14T13:02:36Z"
+ content="""
+Using nested git repositories is well possible; if they are checked in they are called submodules, otherwise they just sit there unadded.
+
+Apart from some [[odd quirx you never run into in normal operation|bugs/creating dot-git-as-symlink workaround drops worktree configuration from submodules]], submodules work fine also with git-annex.
+"""]]

diff --git a/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn
new file mode 100644
index 0000000000..37ba9a94f0
--- /dev/null
+++ b/doc/forum/Please_diagnose__58_____96__git_status__96___is_slow_after_update.mdwn
@@ -0,0 +1,74 @@
+Hi, if anyone can suggest some steps to take to diagnose a sudden performance problem in my git-annex repo, I'd appreciate it.
+
+I have been using git annex for several years on this same repo and performance has never been a problem.
+I recently did a OS update of all my packages (arch linux), which updated git annex, and now performance is really bad whenever I run `git status` or `git commit` in the repo (performance is fine in git repos without git annex). For an example of how bad the performance is, I first noticed the problem when trying to make a commit, and I waited 20 minutes before cancelling with Ctrl+C. The update took me from package version "8.20210310-16" to "8.20210803-63". After noticing the problem I updated again to package version "8.20210803-81", since I was hoping there might be a fix in the most recent version, but that didn't resolve it. Those versions might be specific to arch linux, so if that's the case and anybody needs the true versions let me know and I can try to figure out which git-annex commit they are at.
+
+To try to get a sense of what is going on, I started by running `GIT_TRACE_PERFORMANCE=1 git commit`. There seems to be one command that git is running that has a non-trivial time difference between other commands, and its
+
+    18:14:31.968248 trace.c:487             performance: 247.576906378 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)'
+
+The time difference between this command and the next is over 1 second, and it seems that `git commit` is running this type of command many times. I posted a sample of the full output below.
+
+When searching the web for that git command, I came across this conversation: https://git-annex.branchable.com/todo/speed_up_git_annex_sync_--content_--all/
+
+Could that be related to my issue?
+
+What should I do next to figure out what the problem is?
+ 
+I posted the output of `git annex version` and `git annex info` below in case that is helpful.
+
+Thanks!
+
+----------
+
+`git annex version`:
+
+    git-annex version: 8.20210803-g9cae7c5bb
+    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
+    dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-9.0.2 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+    operating system: linux x86_64
+    supported repository versions: 8
+    upgrade supported from repository versions: 0 1 2 3 4 5 6 7
+    local repository version: 8
+
+
+`git annex info` (omitting repository info):
+
+    available local disk space: 146.18 gigabytes (+1 megabyte reserved)
+    local annex keys: 2120
+    local annex size: 2.59 gigabytes
+    annexed files in working tree: 1945
+    size of annexed files in working tree: 2.55 gigabytes
+    bloom filter size: 32 mebibytes (0.4% full)
+    backend usage:
+            SHA256E: 1945
+
+
+sample of the output of `GIT_TRACE_PERFORMANCE=1 git commit`:
+
+    18:14:31.949865 trace.c:487             performance: 0.031945775 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' --buffer
+    18:14:31.950372 trace.c:487             performance: 0.031692879 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch=%(objectname) %(objecttype) %(objectsize)' --buffer
+    18:14:31.963858 trace.c:487             performance: 0.007070777 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs hash-object --stdin-paths --no-filters
+    18:14:31.967006 trace.c:487             performance: 247.578879398 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
+    18:14:31.968248 trace.c:487             performance: 247.576906378 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)'
+    18:14:33.217226 trace.c:487             performance: 0.000427304 s: git command: git config --null --list
+    18:14:33.244772 read-cache.c:2398       performance: 0.011644939 s:  read cache .git/index
+    18:14:33.245219 read-cache.c:2398       performance: 0.001136355 s:  read cache .git/index
+    18:14:33.263160 trace.c:487             performance: 0.001024382 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs show-ref --hash refs/annex/last-index
+    18:14:33.274091 read-cache.c:2398       performance: 0.001037856 s:  read cache .git/index
+    18:14:33.284793 unpack-trees.c:1802     performance: 0.000024725 s:    traverse_trees
+    18:14:33.284872 unpack-trees.c:418      performance: 0.000001050 s:    check_updates
+    18:14:33.284917 unpack-trees.c:1892     performance: 0.000349269 s:   unpack_trees
+    18:14:33.289148 diff-lib.c:629          performance: 0.004585392 s:  diff-index
+    18:14:33.289220 trace.c:487             performance: 0.017392572 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs -c filter.annex.smudge= -c filter.annex.clean= -c diff.external= diff d62ecb841c32ff0d62df4f8ca7cb0567616cf415 --staged --raw -z --no-abbrev -G/annex/objects/ --no-renames --ignore-submodules=all --no-ext-diff
+    18:14:33.294517 trace.c:487             performance: 0.025457813 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' --buffer
+    18:14:33.297215 trace.c:487             performance: 0.024423869 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch=%(objectname) %(objecttype) %(objectsize)' --buffer
+    18:14:33.303299 trace.c:487             performance: 0.000746262 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs hash-object --stdin-paths --no-filters
+    18:14:33.305448 trace.c:487             performance: 0.076031969 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch
+    18:14:33.308580 trace.c:487             performance: 0.075975445 s: git command: git --git-dir=.git --work-tree=. --literal-pathspecs cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)'
+    18:14:34.537129 trace.c:487             performance: 0.000424618 s: git command: git config --null --list
+    18:14:34.570720 read-cache.c:2398       performance: 0.008743858 s:  read cache .git/index
+    18:14:34.573279 read-cache.c:2398       performance: 0.001075431 s:  read cache .git/index
+    

Added a comment
diff --git a/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment b/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment
new file mode 100644
index 0000000000..d0a162c77a
--- /dev/null
+++ b/doc/forum/Storing_git_repos_in_git-annex/comment_6_4a5ddb1980b9ba0378531a8d4fb93523._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lukey"
+ avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
+ subject="comment 6"
+ date="2022-01-13T18:19:31Z"
+ content="""
+The main problem is that the git developers stubbornly refuse to support nested git repos, so it's impossible for git-annex to support it. However I found a good workaround, see [[tips/using_nested_git_repositories/]].
+"""]]

Added a comment
diff --git a/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment b/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment
new file mode 100644
index 0000000000..260ea540e1
--- /dev/null
+++ b/doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lukey"
+ avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
+ subject="comment 2"
+ date="2022-01-13T18:10:22Z"
+ content="""
+I found an even better workaround, see [[tips/using_nested_git_repositories/]]
+"""]]

diff --git a/doc/tips/using_nested_git_repositories.mdwn b/doc/tips/using_nested_git_repositories.mdwn
new file mode 100644
index 0000000000..6ebd9e64c8
--- /dev/null
+++ b/doc/tips/using_nested_git_repositories.mdwn
@@ -0,0 +1,7 @@
+Using nested git repositories in git is not possible and thus this also applies to git-annex. However, here is a good workaround that I found:
+
+Rename the `.git` directory of the nested repo to `dotgit` (or similar), `git annex add` it and then create a symbolic link from `.git` to `dotgit`. It's important that the link is created only after the nested repo has been `git annex add`'ed. Also, the link needs to be created manually on each clone. Finally you'll need to hide the `dotgit` directory from the nested repo itself by adding `/dotgit` to `dotgit/info/exclude`.
+
+    mv nested/.git nested/dotgit; echo "/dotgit" >>nested/dotgit/info/exclude
+    git annex add nested; git commit -m "add nested"
+    cd nested; ln -s dotgit .git # needs to be done on every clone

diff --git a/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn b/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn
new file mode 100644
index 0000000000..188efeaf2e
--- /dev/null
+++ b/doc/bugs/Cloned_from_.git_under_Windows__44___get_fails.mdwn
@@ -0,0 +1,292 @@
+### Please describe the problem.
+
+Under Windows (not tested in Unix), if an annex is cloned from the .git directory, git annex get fails and is different to cloning from workdir
+
+### What steps will reproduce the problem?
+
+[[!format sh """
+shaddy@COMPUTER-W10 U:\Temp
+> git annex version
+git-annex version: 8.20211118-g2e2d35869
+build flags: Assistant Webapp Pairing TorrentParser MagicMime Feeds Testsuite S3 WebDAV
+dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+operating system: mingw32 x86_64
+supported repository versions: 8
+upgrade supported from repository versions: 2 3 4 5 6 7
+
+shaddy@COMPUTER-W10 U:\Temp
+> dir
+ Volume in drive U has no label.
+ Volume Serial Number is D684-6493
+
+ Directory of U:\Temp
+
+13/01/2022  01:42 PM    <DIR>          .
+13/01/2022  01:42 PM    <DIR>          ..
+10/04/2020  07:36 PM        14,572,000 vc_redist.x64.exe
+               1 File(s)     14,572,000 bytes
+               2 Dir(s)  21,238,497,280 bytes free
+
+shaddy@COMPUTER-W10 U:\Temp
+> git init --separate-git-dir separategitmaster.git separategitmaster
+Initialized empty Git repository in U:/Temp/separategitmaster.git/
+
+shaddy@COMPUTER-W10 U:\Temp
+> cd separategitmaster
+
+shaddy@COMPUTER-W10 U:\Temp\separategitmaster
+> git annex init master
+init master
+  Detected a filesystem without fifo support.
+
+  Disabling ssh connection caching.
+
+  Detected a crippled filesystem.
+
+  Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
+
+Switched to branch 'adjusted/master(unlocked)'
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separategitmaster
+> copy ..\vc_redist.x64.exe .
+        1 file(s) copied.
+
+shaddy@COMPUTER-W10 U:\Temp\separategitmaster
+> git annex add vc_redist.x64.exe
+add vc_redist.x64.exe
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separategitmaster
+> git commit -m vc_redist.x64.exe
+[adjusted/master(unlocked) a32d7ba] vc_redist.x64.exe
+ 1 file changed, 1 insertion(+)
+ create mode 100755 vc_redist.x64.exe
+
+shaddy@COMPUTER-W10 U:\Temp\separategitmaster
+> cd ..
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitslave
+> REM note, cloning from workdir, not .git
+
+shaddy@COMPUTER-W10 U:\Temp
+> git clone separategitmaster separategitslave
+Cloning into 'separategitslave'...
+done.
+
+shaddy@COMPUTER-W10 U:\Temp
+> cd separategitslave
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> dir
+ Volume in drive U has no label.
+ Volume Serial Number is D684-6493
+
+ Directory of U:\Temp\separategitslave
+
+13/01/2022  01:45 PM    <DIR>          .
+13/01/2022  01:45 PM    <DIR>          ..
+13/01/2022  01:45 PM               108 vc_redist.x64.exe
+               1 File(s)            108 bytes
+               2 Dir(s)  21,209,219,072 bytes free
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> git config user.name "Shaddy Baddah"
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> git annex init slave
+init slave
+  Detected a filesystem without fifo support.
+
+  Disabling ssh connection caching.
+
+  Detected a crippled filesystem.
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> git annex sync
+commit
+On branch adjusted/master(unlocked)
+Your branch is up to date with 'origin/adjusted/master(unlocked)'.
+
+nothing to commit, working tree clean
+ok
+pull origin
+ok
+push origin
+Enumerating objects: 9, done.
+Counting objects: 100% (9/9), done.
+Delta compression using up to 2 threads
+Compressing objects: 100% (5/5), done.
+Writing objects: 100% (6/6), 718 bytes | 359.00 KiB/s, done.
+Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
+To U:/Temp/separategitmaster
+ * [new branch]      master -> synced/master
+ * [new branch]      git-annex -> synced/git-annex
+ok
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> git annex get vc_redist.x64.exe
+get vc_redist.x64.exe (from origin...)
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> REM All ok. But now lets clone from the .git dir
+
+shaddy@COMPUTER-W10 U:\Temp\separategitslave
+> cd ..
+
+shaddy@COMPUTER-W10 U:\Temp
+> git init --separate-git-dir separatetake2gitmaster.git separatetake2gitmaster
+Initialized empty Git repository in U:/Temp/separatetake2gitmaster.git/
+
+shaddy@COMPUTER-W10 U:\Temp
+> cd separatetake2gitmaster
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> git annex init master
+init master
+  Detected a filesystem without fifo support.
+
+  Disabling ssh connection caching.
+
+  Detected a crippled filesystem.
+
+  Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
+
+Switched to branch 'adjusted/master(unlocked)'
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> copy ..\vc_redist.x64.exe .
+        1 file(s) copied.
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> git annex add vc_redist.x64.exe
+add vc_redist.x64.exe
+ok
+(recording state in git...)
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> git commit -m vc_redist.x64.exe
+git-annex.exe: .\vc_redist.x64.exe: DeleteFile "\\\\?\\U:\\Temp\\separatetake2gitmaster\\vc_redist.x64.exe": permission denied (The process cannot access the file because it is being used by another process.)
+error: external filter 'git-annex smudge --clean -- %f' failed 1
+error: external filter 'git-annex smudge --clean -- %f' failed
+[adjusted/master(unlocked) 50bb6c7] vc_redist.x64.exe
+ 1 file changed, 1 insertion(+)
+ create mode 100755 vc_redist.x64.exe
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> REM unfortunate Windows gotcha, but I believe it doesnt effect the scenario
+
+shaddy@COMPUTER-W10 U:\Temp\separatetake2gitmaster
+> dir
+ Volume in drive U has no label.
+ Volume Serial Number is D684-6493
+
+ Directory of U:\Temp\separatetake2gitmaster

(Diff truncated)
liking this solution pretty well
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment
new file mode 100644
index 0000000000..2ddaccd98e
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_11_e9af4784ffb84b49b6faaa21aa7c8efc._comment
@@ -0,0 +1,28 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2022-01-12T19:40:35Z"
+ content="""
+This is going to need two repository version bumps:
+
+v9: Add the upgrade lock file, and all git-annex processes take a shared
+lock to avoid the repository being upgraded out from under them. 
+Upgrade is skipped when assistant is running.
+
+v10: Skipped until the upgrade lock file is of a certain age. Take upgrade
+lock before increasing annex.version.
+In v10, stop locking content files and lock separate lock files.
+
+This way, an old version of git-annex cannot be used in the v9 repository,
+and so the v10 upgrade only needs to worry about any git-annex processes
+that were started in v8.
+
+The age could be eg 1 month, which assumes that no old git-annex process
+like `git-annex move --to remote` is still running after that long.
+Of course, that is still an assumption, but it can be pushed out as long as
+it takes to feel comfortable with it. Maybe 1 year?
+
+`git-annex upgrade --version=10` could be available to speed up that
+upgrade. The user would be responsible for making sure there are no such
+old git-annex processes running, so that might need --force.
+"""]]

Added a comment
diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment
new file mode 100644
index 0000000000..2023ad7858
--- /dev/null
+++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_3_ab590b41983d84c8fe7d5dc7276689b8._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 3"
+ date="2022-01-12T19:33:15Z"
+ content="""
+Still checking but so far looks good with `8.20211231+git69-ga55fc567c-1~ndall+1` -- thank you!
+"""]]

so close
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment
new file mode 100644
index 0000000000..95809e9d9b
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_10_2e61ccc41839b8da70ad26d31e40ad74._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2022-01-12T19:14:10Z"
+ content="""
+This would almost work:
+
+Continue taking a shared lock of the content
+file when locking to prevent dropping. That does not need write access,
+only an exclusive lock does, so the content file can have its write bit
+removed. Also lock the new lock file, with a shared lock to prevent
+dropping, or an exclusive when dropping.
+
+The old git-annex version, when dropping, will fail to exclusively lock the
+content file, either because it's not writable, or because of a shared
+lock intended to prevent dropping. So a git-annex drop that was in progress
+may start to fail, but it will not lose any data.
+
+Problem: The old git-annex version, when locking to prevent dropping
+(eg git-annex move --from remote),
+will take the shared lock of the content file. If the new git-annex version
+is locking to drop, it will also take the shared lock of the content file,
+followed by the exlusive lock of the new lock file. So the old git-annex will
+not be able to prevent the new git-annex from dropping.
+"""]]

complication
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment
new file mode 100644
index 0000000000..f72e43dfe8
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_9_f74642e1dfc4e533b9582e5a13a16898._comment
@@ -0,0 +1,41 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 9"""
+ date="2022-01-12T16:29:40Z"
+ content="""
+While this is mostly implemented in the branch, the upgrade to it has a
+serious danger point.
+
+If a long-running git-annex process, eg a large drop, is running during the
+upgrade, then it will keep on using the current locking method. Meanwhile,
+other processes run after the upgrade will use the new locking method. So
+this could cause data loss: Old git-annex locks a file to drop it, and at
+the same time new git-annex-shell is used to lock the same file, to prevent
+it from being dropped.
+
+Avoiding that seems to require a way to make sure there are no running
+git-annex processes when performing a repository upgrade. (Which would be
+nice in general but has somehow not been necessary until now.)
+
+One way to do it would be to have a shared lock file that all git-annex
+processes hold while they are running. And upgrade takes an exclusive lock.
+That locking would need to be implemented first and somehow be known
+that any git-annex process that is running is using it, before performing
+the repository upgrade. Ideally without taking years in between to wait
+for all git-annex binaries to be upgraded.
+
+I suppose that git-annex v9 could ship with that added lock file, and not
+upgrade the repository to v9 immediately. Instead, stat the lock file, and
+only when its ctime is sufficiently old that it seems safe to assume any
+running git-annex process would be using it, do the repository upgrade. Eg
+after 3 months or so or perhaps when the ctime is older than the last
+reboot. But this would not avoid problems if an older git-annex
+version was also used in the same repository as the new version.
+
+Or git-annex upgrade could hunt for other running git-annex processes that
+are using the repository and refuse to perform the v9 upgrade. But that is
+hard because a processes's cwd is not necessarily inside the repository
+it's using (eg a remote). It would have to look for git child processes
+of git-annex processes that are using the repository, such as git cat-file.
+Also network filesystems would be a problem.
+"""]]

diff --git a/doc/bugs/git-annex-shell_fails_to_run_true.mdwn b/doc/bugs/git-annex-shell_fails_to_run_true.mdwn
index bbace3f830..230a94a340 100644
--- a/doc/bugs/git-annex-shell_fails_to_run_true.mdwn
+++ b/doc/bugs/git-annex-shell_fails_to_run_true.mdwn
@@ -162,3 +162,5 @@ Key part above:
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 I'm a long time git-annex user (early adopter, even?) and git-annex is generally serving my uses very well, both as a backup and archival system. The encryption features, however, are a bit more obscure than my tolerance level right now. Thankfully, there is the [[special_remotes/borg/]] I'm thinking of using to replace all this complexity in the future... 
+
+-- [[done]], problem was with a configuration in the gcrypt remote sandbox. -- [[anarcat]]

Added a comment: indeed.
diff --git a/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment b/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment
new file mode 100644
index 0000000000..a9d422e054
--- /dev/null
+++ b/doc/bugs/git-annex-shell_fails_to_run_true/comment_2_5be34b5ffaee39013d24daf83a8dd437._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="anarcat"
+ avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
+ subject="indeed."
+ date="2022-01-12T15:56:25Z"
+ content="""
+the problem was that i had a `~/git-shell-commands/rsync` overriding the rsync command to try to restrict *that*, since i'm using a gcrypt remote... it was clunky and badly designed and, ultimately, insecure. i replaced it with `rrsync /home/anarcat/offsite/` but even that is not the best either, as the script says it \"assumes someone will not subvert the rsync protocol\" which hardly seems reassuring.
+
+i also had to change my remotes to follow the \"chroot\" of sorts:
+
+    git config remote.offsite-annex.annex-rsyncurl anarcat@remote-annex:Videos.annex/
+    git remote set-url offsite-git gcrypt::rsync://anarcat@remote-annex:Videos.git/
+
+Running `git fetch` by hand convinced me the problem wasn't git-annex related... From there I reread the git-annex-shell and git-shell commands and noticed that bit about the obscure `git-shell-commands` directory, and found that jem. I even had an error in `journalctl -f` that confirm the problem, something i overlooked as well...
+
+so this indeed seems like a problem on my end. sorry for the delay and the noise!
+"""]]

v9-locking branch still wip
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment
index a1b85d21ee..de67df2e7c 100644
--- a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment
@@ -3,7 +3,7 @@
  subject="""comment 8"""
  date="2022-01-11T18:48:15Z"
  content="""
-This is implemented in the `v9-locking` branch,
+This is being implemented in the `v9-locking` branch,
 which will only get merged when v9 happens, whenever that is.
 See [[todo/v9_changes]].
 """]]
diff --git a/doc/todo/v9_changes.mdwn b/doc/todo/v9_changes.mdwn
index b4aa21b4d0..87ac7b39f7 100644
--- a/doc/todo/v9_changes.mdwn
+++ b/doc/todo/v9_changes.mdwn
@@ -12,7 +12,7 @@ could change and if it does, these things could be included.
 
   Note that windows already uses a separate lock file.
 
-  This is implemented in git branch `v9-locking`.
+  This is being implemented in git branch `v9-locking`.
 
 * Possibly enable `git-annex filter-process` by default. If the tradeoffs
   seem worth it.

Added a comment: Thanks!
diff --git a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment
new file mode 100644
index 0000000000..ed38f6ac40
--- /dev/null
+++ b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_2_fac648fb7550f0c3c433ce423e3ea7f6._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="mih"
+ avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
+ subject="Thanks!"
+ date="2022-01-11T18:59:36Z"
+ content="""
+That was the missing piece for me. Thank you!
+"""]]

v9-locking branch
diff --git a/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment
new file mode 100644
index 0000000000..a1b85d21ee
--- /dev/null
+++ b/doc/bugs/shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__/comment_8_c3d447a8f4e63f6a28a02ccb4eed1ad4._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 8"""
+ date="2022-01-11T18:48:15Z"
+ content="""
+This is implemented in the `v9-locking` branch,
+which will only get merged when v9 happens, whenever that is.
+See [[todo/v9_changes]].
+"""]]
diff --git a/doc/todo/v9_changes.mdwn b/doc/todo/v9_changes.mdwn
index 23b735da95..b4aa21b4d0 100644
--- a/doc/todo/v9_changes.mdwn
+++ b/doc/todo/v9_changes.mdwn
@@ -12,6 +12,8 @@ could change and if it does, these things could be included.
 
   Note that windows already uses a separate lock file.
 
+  This is implemented in git branch `v9-locking`.
+
 * Possibly enable `git-annex filter-process` by default. If the tradeoffs
   seem worth it.
 

followup and close
diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__.mdwn b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__.mdwn
index bb3dc89f72..1d422b2316 100644
--- a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__.mdwn
+++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__.mdwn
@@ -58,3 +58,5 @@ I am reporting this behavior now, because it has worsened since [I last looked i
 Thanks for git-annex!
 
 [[!tag projects/datalad]]
+
+> [[fixed|done]] (enable filter.annex.process) --[[Joey]]
diff --git a/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_22_1c001d885bd4d7f60b3260d494406e5c._comment b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_22_1c001d885bd4d7f60b3260d494406e5c._comment
new file mode 100644
index 0000000000..fc6853dd50
--- /dev/null
+++ b/doc/bugs/Windows__58___substantial_per-file_cost_for___96__add__96__/comment_22_1c001d885bd4d7f60b3260d494406e5c._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 22"""
+ date="2022-01-11T17:24:49Z"
+ content="""
+Enabling filter.annex.process in new repositories would unfortunately
+break using that repository with an older version of git-annex before that
+was added to it. So it seems that `git-annex init` cannot do it for
+annex.version 8 and will need to wait for v9.
+
+Of course, if you are sure users of a repository will not be using an old
+version of git-annex, it's fine to enable it.
+
+I'm going to close this bug now, since [[todo/v9_changes]] is open to get it
+enabled by default in v9.
+"""]]

comment
diff --git a/doc/tips/using_signed_git_commits/comment_4_5ad0596373cc363ec9bc0b69efbac03c._comment b/doc/tips/using_signed_git_commits/comment_4_5ad0596373cc363ec9bc0b69efbac03c._comment
new file mode 100644
index 0000000000..7ac72d40d7
--- /dev/null
+++ b/doc/tips/using_signed_git_commits/comment_4_5ad0596373cc363ec9bc0b69efbac03c._comment
@@ -0,0 +1,21 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2022-01-11T17:02:59Z"
+ content="""
+Are we still concerned about this? Well, git has a workaround for SHA1's
+insecurity and will eventually change hashes. There are plenty of other
+reasons to want to sign git commits, certianly.
+
+The webapp bypasses gpg signing because it commits automatically and
+potentially frequently, and depending on how gpg handles password
+prompting, that could flood the user with repeated password prompts.
+But you can change this default with the `annex.allowsign` configuration.
+
+(Commits to the git-annex branch are also not signed by default, for similar
+reasons. Also, the risks of SHA1 collisions involving the git-annex branch
+seem small to nonexistant, since that branch only records bookeeping
+information git-annex cares about, and a small amount of configuration.
+git-annex does not use data from that branch in any way that would let
+an untrusted person who modified the branch do anything malicious.)
+"""]]

Avoid crashing when run in a bare git repo that somehow contains an index file
Do not populate the keys database with associated files,
because a bare repo has no working tree, and so it does not make sense to
populate it.
Queries of associated files in the keys database always return empty lists
in a bare repo, even if it's somehow populated. One way it could be
populated is if a user converts a non-bare repo to a bare repo.
Note that Git.Config.isBare does a string comparison, so this is not free!
But, that string comparison is very small compared to a sqlite query.
Sponsored-by: Erik Bjäreholt on Patreon
diff --git a/CHANGELOG b/CHANGELOG
index 16cd4f0098..88f7ce5a5e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,6 +10,8 @@ git-annex (8.20211232) UNRELEASED; urgency=medium
     a retry from failing again. 
     (reversion introduced in version 8.20210903)
   * adb: Added ignorefinderror configuration parameter.
+  * Avoid crashing when run in a bare git repo that somehow contains an
+    index file.
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Jan 2022 14:01:14 -0400
 
diff --git a/Database/Keys.hs b/Database/Keys.hs
index 386c3245b1..a342274f5e 100644
--- a/Database/Keys.hs
+++ b/Database/Keys.hs
@@ -49,6 +49,7 @@ import Git.Sha
 import Git.CatFile
 import Git.Branch (writeTreeQuiet, update')
 import qualified Git.Ref
+import qualified Git.Config
 import Config.Smudge
 import qualified Utility.RawFilePath as R
 
@@ -154,11 +155,23 @@ addAssociatedFile k f = runWriterIO $ SQL.addAssociatedFile k f
 {- Note that the files returned were once associated with the key, but
  - some of them may not be any longer. -}
 getAssociatedFiles :: Key -> Annex [TopFilePath]
-getAssociatedFiles = runReaderIO . SQL.getAssociatedFiles
+getAssociatedFiles k = emptyWhenBare $ runReaderIO $ SQL.getAssociatedFiles k
+
+{- Queries for associated files never return anything when in a bare
+ - repository, since without a work tree there can be no associated files. 
+ -
+ - Normally the keys database is not even populated with associated files 
+ - in a bare repository, but it might happen if a non-bare repo got
+ - converted to bare. -}
+emptyWhenBare :: Annex [a] -> Annex [a]
+emptyWhenBare a = ifM (Git.Config.isBare <$> gitRepo)
+	( return []
+	, a
+	)
 
 {- Include a known associated file along with any recorded in the database. -}
 getAssociatedFilesIncluding :: AssociatedFile -> Key -> Annex [RawFilePath]
-getAssociatedFilesIncluding afile k = do
+getAssociatedFilesIncluding afile k = emptyWhenBare $ do
 	g <- Annex.gitRepo
 	l <- map (`fromTopFilePath` g) <$> getAssociatedFiles k
 	return $ case afile of
@@ -168,7 +181,7 @@ getAssociatedFilesIncluding afile k = do
 {- Gets any keys that are on record as having a particular associated file.
  - (Should be one or none but the database doesn't enforce that.) -}
 getAssociatedKey :: TopFilePath -> Annex [Key]
-getAssociatedKey = runReaderIO . SQL.getAssociatedKey
+getAssociatedKey f = emptyWhenBare $ runReaderIO $ SQL.getAssociatedKey f
 
 removeAssociatedFile :: Key -> TopFilePath -> Annex ()
 removeAssociatedFile k = runWriterIO . SQL.removeAssociatedFile k
@@ -233,7 +246,7 @@ isInodeKnown i s = or <$> runReaderIO ((:[]) <$$> SQL.isInodeKnown i s)
  - is an associated file.
  -}
 reconcileStaged :: H.DbQueue -> Annex ()
-reconcileStaged qh = do
+reconcileStaged qh = unlessM (Git.Config.isBare <$> gitRepo) $ do
 	gitindex <- inRepo currentIndexFile
 	indexcache <- fromRawFilePath <$> fromRepo gitAnnexKeysDbIndexCache
 	withTSDelta (liftIO . genInodeCache gitindex) >>= \case
diff --git a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn
index 9cbb3926ff..1c861fd5aa 100644
--- a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn
+++ b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn
@@ -55,3 +55,4 @@ AFAICS this creates the same records as if I would have done this in a regular w
 
 Is this a bug, or am I doing something wrong? Thanks in advance for your time!
 
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_1_22fb9214e6e31d04b0ffe3266d26ea69._comment b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_1_22fb9214e6e31d04b0ffe3266d26ea69._comment
new file mode 100644
index 0000000000..082b7b7332
--- /dev/null
+++ b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all/comment_1_22fb9214e6e31d04b0ffe3266d26ea69._comment
@@ -0,0 +1,28 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-11T16:25:50Z"
+ content="""
+`git update-index` creates an index file. A repo with an index file is no
+longer a bare repo to a certain extent, at least it is very unusual for a
+repo to be both bare and contain an index file, since after all an index
+file is a record of the files in the working tree. This seems to be the root
+of the confusion.
+
+I was able to reproduce this by simply copying `.git/index` from a non-bare
+repo into the bare repo and then running `git-annex drop --all`.
+
+Removing the index file was not sufficient to fix it. It turned out I also 
+needed to delete `annex/keydb*`. Then things returned to normal.
+
+So, it seems that the keys database is getting populated in a bare repo
+when there's an index file, and once the keys database is populated, it
+runs code paths that will not work in a bare repo, because that database
+contains paths (taken from the index) that it treats as being present in a
+nonexistant working tree. I've fixed it to both avoid populating the keys
+database, and ignore a populated keys database in this situation.
+
+But.. My suggestion is, if you need to do this kind of thing in a bare repo,
+set `GIT_INDEX_FILE` to some other file. That's how git-annex makes similar
+tree objects.
+"""]]

moreinfo
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_7_247db3d1c03b207296e97975a3554756._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_7_247db3d1c03b207296e97975a3554756._comment
new file mode 100644
index 0000000000..438e5cd155
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_7_247db3d1c03b207296e97975a3554756._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2022-01-07T16:34:42Z"
+ content="""
+This seems to be a reversion caused by incremental verification. In
+P2P.Annex it does incremental verification, and so it notices that the temp
+file is corrupted. So, the transfer fails, and the temp file is left
+behind.
+
+So I suppose the fix is that, whenever doing incremental verification,
+also delete the temp file when it fails.
+"""]]
diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
index 917748b90b..1bfd4b6859 100644
--- a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
+++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist.mdwn
@@ -17,3 +17,5 @@ conda linux nodep (standalone) build 8.20211012-geb95ed486 and then with another
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+[[!tag moreinfo]]
diff --git a/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_2_92dec3e0c31ffbef1a70ea3038ba6ca1._comment b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_2_92dec3e0c31ffbef1a70ea3038ba6ca1._comment
new file mode 100644
index 0000000000..ec448e5cc1
--- /dev/null
+++ b/doc/bugs/get_-J5___58___pidlock__58___getFileStatus__58___does_not_exist/comment_2_92dec3e0c31ffbef1a70ea3038ba6ca1._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-11T16:16:51Z"
+ content="""
+Ping, can you try to reproduce this with a recent build?
+"""]]
diff --git a/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism.mdwn b/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism.mdwn
index 13c8315ddf..e2fa117913 100644
--- a/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism.mdwn
+++ b/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism.mdwn
@@ -11,3 +11,5 @@ For reference, here is a PR in DataLad [https://github.com/datalad/datalad/pull/
 
 [[!meta author=yoh]]
 [[!tag projects/datalad]]
+
+[[!tag moreinfo]]
diff --git a/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism/comment_2_9e2c1d58f0d91dc55524d2e164b89b97._comment b/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism/comment_2_9e2c1d58f0d91dc55524d2e164b89b97._comment
new file mode 100644
index 0000000000..d5f479a8b0
--- /dev/null
+++ b/doc/todo/more_flexible___40____34__external__34____63____41___credentials_mechanism/comment_2_9e2c1d58f0d91dc55524d2e164b89b97._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-11T16:19:09Z"
+ content="""
+> That also relates to software relying on git-annex, such as datalad, as it somehow needs to know/discover that underlying call to git-annex apparently needs some credentials to be passed via env vars
+
+Since `git-annex initremote` displays an error message when it needs
+creds, does datalad not forward that error message to the user?
+
+There seems to be a significant disconnect between what you're describing
+and what I've seen of and know of creds usage in git-annex. Gonna need more
+information.
+"""]]

diff --git a/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn
new file mode 100644
index 0000000000..9cbb3926ff
--- /dev/null
+++ b/doc/bugs/Repo_manipulation_breaks_git-annex_drop_--all.mdwn
@@ -0,0 +1,57 @@
+I am working with a bare repository to transfer two keys from a custom backend to and from a special remote. This seems to be working fine.
+
+In order to be able to make use of export remotes (exporttree=yes), I need to be able to specific a tree to be exported. For technical reasons, I want to keep using a bare repository, and use a `hash-object`, `update-index`, and `write-tree` manually in order to create a tree. The Python code snippet that does this looks like this:
+
+```
+    for key, prefix, fname in (
+            # the prefixes are constant hashdir-lower
+            (RepoAnnexGitRemote.refs_key, 'a11/1c8', '.datalad/dotgit/refs'),
+            (RepoAnnexGitRemote.repo_export_key, '6b2/c13',
+             '.datalad/dotgit/repo-export.zip')):
+        # create a blob for the annex link
+        out = repo._git_runner.run(
+            ['git', 'hash-object', '-w', '--stdin'],
+            stdin=bytes(
+                f'../../.git/annex/objects/{prefix}/{key}/{key}', 'utf-8'),
+            protocol=StdOutCapture)
+        linkhash = out['stdout'].strip()
+        # place link into a tree
+        out = repo._git_runner.run(
+            ['git', 'update-index', '--add', '--cacheinfo', '120000',
+             linkhash, fname],
+            protocol=StdOutCapture)
+    # write the complete tree, and return ID
+    out = repo._git_runner.run(
+        ['git', 'write-tree'],
+        protocol=StdOutCapture)
+    exporttree = out['stdout'].strip()
+ ```
+
+It essentially creates the two blobs for the annex links, puts them together in a tree, and writes it to the repo.
+
+However, after this code ran, git-annex is not longer operating properly in the bare repo:
+
+```
+% git annex drop --all
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+fatal: relative path syntax can't be used outside working tree
+git-annex: fd:21: Data.ByteString.hGetLine: end of file
+```
+
+(fatal error messages are from cat-file batch calls inside)
+
+When I comment this code out, everything goes back to normal. It seems to makes no difference whether I follow the problematic code up with a `commit-tree` and `update-ref` to actually have the mainline branch point to a commit with that tree. It also seems to make no difference, when I explicitly `setpresentkey <key> <here> 0`.
+
+AFAICS this creates the same records as if I would have done this in a regular worktree using high-level git-annex tooling. Other git-annex commands like `fsck` seem to be working fine. If a create a branch with that tree, also `findref` seems to be working properly.
+
+Is this a bug, or am I doing something wrong? Thanks in advance for your time!
+

adb: Added ignorefinderror configuration parameter
On a phone with Calyxos, adb find in /sdcard complains:
find: ./Android/data/com.android.providers.downloads.ui: Permission denied
But otherwise works, so this option makes import and export work ok, except
for that one app's data.
Sponsored-by: Graham Spencer
diff --git a/CHANGELOG b/CHANGELOG
index 79d655a6b4..16cd4f0098 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -9,6 +9,7 @@ git-annex (8.20211232) UNRELEASED; urgency=medium
     by deleting the temporary file when it fails to verify. This prevents
     a retry from failing again. 
     (reversion introduced in version 8.20210903)
+  * adb: Added ignorefinderror configuration parameter.
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Jan 2022 14:01:14 -0400
 
diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index 580a640fc5..2001727b11 100644
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -43,6 +43,8 @@ remote = specialRemoteType $ RemoteType
 			(FieldDesc "location on the Android device where the files are stored")
 		, optionalStringParser androidserialField
 			(FieldDesc "sometimes needed to specify which Android device to use")
+		, yesNoParser ignorefinderrorField (Just False)
+			(FieldDesc "ignore adb find errors")
 		]
 	, setup = adbSetup
 	, exportSupported = exportIsSupported
@@ -56,6 +58,10 @@ androiddirectoryField = Accepted "androiddirectory"
 androidserialField :: RemoteConfigField
 androidserialField = Accepted "androidserial"
 
+ignorefinderrorField :: RemoteConfigField
+ignorefinderrorField = Accepted "ignorefinderror"
+
+
 gen :: Git.Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex (Maybe Remote)
 gen r u rc gc rs = do
 	c <- parsedRemoteConfig remote rc
@@ -83,7 +89,7 @@ gen r u rc gc rs = do
 			, renameExport = renameExportM serial adir
 			}
 		, importActions = ImportActions
-			{ listImportableContents = listImportableContentsM serial adir
+			{ listImportableContents = listImportableContentsM serial adir c
 			, importKey = Nothing
 			, retrieveExportWithContentIdentifier = retrieveExportWithContentIdentifierM serial adir
 			, storeExportWithContentIdentifier = storeExportWithContentIdentifierM serial adir
@@ -289,13 +295,13 @@ renameExportM serial adir _k old new = do
 		, File newloc
 		]
 
-listImportableContentsM :: AndroidSerial -> AndroidPath -> Annex (Maybe (ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)))
-listImportableContentsM serial adir = adbfind >>= \case
+listImportableContentsM :: AndroidSerial -> AndroidPath -> ParsedRemoteConfig -> Annex (Maybe (ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)))
+listImportableContentsM serial adir c = adbfind >>= \case
 	Just ls -> return $ Just $ ImportableContentsComplete $ 
 		ImportableContents (mapMaybe mk ls) []
 	Nothing -> giveup "adb find failed"
   where
-	adbfind = adbShell serial
+	adbfind = adbShell' serial
 		[ Param "find"
 		-- trailing slash is needed, or android's find command
 		-- won't recurse into the directory
@@ -304,7 +310,10 @@ listImportableContentsM serial adir = adbfind >>= \case
 		, Param "-exec", Param "stat"
 		, Param "-c", Param statformat
 		, Param "{}", Param "+"
-		]
+		] 
+		(if ignorefinderror then "|| true" else "")
+
+	ignorefinderror = fromMaybe False (getRemoteConfigValue ignorefinderrorField c)
 
 	statformat = adbStatFormat ++ "\t%n"
 
@@ -390,8 +399,11 @@ enumerateAdbConnected = checkAdbInPath [] $ liftIO $
 --
 -- Any stdout from the command is returned, separated into lines.
 adbShell :: AndroidSerial -> [CommandParam] -> Annex (Maybe [String])
-adbShell serial cmd = adbShellRaw serial $
-	unwords $ map shellEscape (toCommand cmd)
+adbShell serial cmd = adbShell' serial cmd ""
+
+adbShell' :: AndroidSerial -> [CommandParam] -> String -> Annex (Maybe [String])
+adbShell' serial cmd extra = adbShellRaw serial $
+	(unwords $ map shellEscape (toCommand cmd)) ++ extra
 
 adbShellBool :: AndroidSerial -> [CommandParam] -> Annex Bool
 adbShellBool serial cmd =
diff --git a/doc/special_remotes/adb.mdwn b/doc/special_remotes/adb.mdwn
index 4eed62daf3..44f0f79e90 100644
--- a/doc/special_remotes/adb.mdwn
+++ b/doc/special_remotes/adb.mdwn
@@ -32,6 +32,10 @@ the adb remote.
   by [[git-annex-import]]. When set in combination with exporttree,
   this lets files be imported from it, and changes exported back to it.
 
+* `ignorefinderror` - Set to "yes" to ignore errors when running "adb find"
+  on the Android device. This can be useful eg, if some subdirectories are
+  not readable, to let it import the other things that are readable.
+
 * `encryption` - One of "none", "hybrid", "shared", or "pubkey".
   See [[encryption]].
 

removed
diff --git a/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment b/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment
deleted file mode 100644
index 7e880ee07a..0000000000
--- a/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment
+++ /dev/null
@@ -1,62 +0,0 @@
-[[!comment format=mdwn
- username="lell"
- avatar="http://cdn.libravatar.org/avatar/4c4138a71d069e290240a3a12367fabe"
- subject="Propagation is different between &quot;annex adjust&quot; and &quot;annex sync&quot;"
- date="2022-01-07T10:21:46Z"
- content="""
-Adjusted branches are important to my data science project, because my programs cannot deal with the read-only symlinks to annex'ed files.
-
-But I find this command confusing, especially that
-
-1. Calling on an unlocked adjusted branch, \"git annex adjust --unlock\" propagates commits back to the master branch differently than \"git annex sync --no-push --no-pull --no-content\" does. 
-
-2. I can't find a way to \"un-adjust\" a branch without resorting to lower-level git commands.
-
-## Problem 1:
-
-Say I have done `git annex adjust --unlock` and then have done more commits. The history now looks like this: 
-
-    * My new commit 2  (HEAD -> adjusted/master(unlocked))
-    * My new commit 1  (HEAD -> adjusted/master(unlocked))
-    * git-annex adjusted branch
-    * Last old commit (master, basis/adjusted/master(unlocked))
-    * Previous commits
-
-If I execute now `git annex adjust --unlock` again, the commits are propagated back to the original branch, 
-but my HEAD is still on the original adjusted branch. So both the master branch and my adjusted branch grow over time which clutters the history and is confusing.
-
-    *     My new commit 2  (master)
-    | *   My new commit 2  (HEAD -> adjusted/master(unlocked))
-    * |   My new commit 1
-    | *   My new commit 1
-    | *   git-annex adjusted branch
-    |/
-    * Last old commit (basis/adjusted/master(unlocked))
-    * Previous commits
-
-On the other hand, if I do `git annex sync --no-push --no-pull --no-content`, the commits are propagated back to master, and a new adjusted branch is created on top of that:
-
-    * git-annex adjusted branch (HEAD -> adjusted/master(unlocked))
-    *     My new commit 2  (master, basis/adjusted/master(unlocked))
-    *     My new commit 1
-    | *   My new commit 2  [abandoned old adjusted branch]
-    | *   My new commit 1
-    | *   git-annex adjusted branch
-    |/
-    * Last old commit
-    * Previous commits
-
-This behaviour makes much more sense to me! Why does it take the modified `sync` command to do this? Why is this not done as well when re-calling `annex adjust --unlock`? The sync command seems a counter-intuitive place to do this, using the `adjust` command would be far more intuitive for me and I think also for other users.
-
-
-## Problem 2
-
-I see no easy way of \"un-adjusting\" an adjusted branch. Currently I do
-
-    git sync --no-push --no-pull --no-content
-    git checkout master
-    git branch -D \"adjusted/master(unlock)\" 
-    git branch -D \"refs/basis/adjusted/master(unlock)\"
-
-That's a lot of text for the inverse operation of `git annex adjust --unlock` and also I have to take care myself to not forget and loose commits I did on the adjusted branch. Did I miss an easier way? If not, I think it would be a great addition. 
-"""]]

Added a comment
diff --git a/doc/git-annex-adjust/comment_7_a9c88b1d83d5e874f8f8063fcdd9e624._comment b/doc/git-annex-adjust/comment_7_a9c88b1d83d5e874f8f8063fcdd9e624._comment
new file mode 100644
index 0000000000..d2896f45fd
--- /dev/null
+++ b/doc/git-annex-adjust/comment_7_a9c88b1d83d5e874f8f8063fcdd9e624._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="lell"
+ avatar="http://cdn.libravatar.org/avatar/4c4138a71d069e290240a3a12367fabe"
+ subject="comment 7"
+ date="2022-01-10T08:33:46Z"
+ content="""
+Thanks for the help on where to put the question, joey. I have deleted and [moved](https://git-annex.branchable.com/forum/Propagate_changes_and_remove_adjusted_branch/?updated) my previous post. 
+"""]]

diff --git a/doc/forum/Propagate_changes_and_remove_adjusted_branch.mdwn b/doc/forum/Propagate_changes_and_remove_adjusted_branch.mdwn
new file mode 100644
index 0000000000..f4bd813e58
--- /dev/null
+++ b/doc/forum/Propagate_changes_and_remove_adjusted_branch.mdwn
@@ -0,0 +1,55 @@
+*This post is moved from the git-annex-adjust manpage, thanks @joey for the hint on where to put it* 
+
+Adjusted branches are important to my data science project, because my programs cannot deal with the read-only symlinks to annex'ed files.
+
+But I find this command confusing, especially that
+
+1. Calling on an unlocked adjusted branch, `git annex adjust --unlock` propagates commits back to the master branch differently than `git annex sync --no-push --no-pull --no-content` does.
+
+2. I can't find a way to "un-adjust" a branch without resorting to lower-level git commands.
+
+## Problem 1:
+
+Say I have done `git annex adjust --unlock` and then have done more commits. The history now looks like this:
+
+    * My new commit 2  (HEAD -> adjusted/master(unlocked))
+    * My new commit 1  (HEAD -> adjusted/master(unlocked))
+    * git-annex adjusted branch
+    * Last old commit (master, basis/adjusted/master(unlocked))
+    * Previous commits
+
+If I execute now `git annex adjust --unlock` again, the commits are propagated back to the original branch, but my HEAD is still on the original adjusted branch. So both the master branch and my adjusted branch grow over time which clutters the history and is confusing.
+
+    *     My new commit 2  (master)
+    | *   My new commit 2  (HEAD -> adjusted/master(unlocked))
+    * |   My new commit 1
+    | *   My new commit 1
+    | *   git-annex adjusted branch
+    |/
+    * Last old commit (basis/adjusted/master(unlocked))
+    * Previous commits
+
+On the other hand, if I do `git annex sync --no-push --no-pull --no-content`, the commits are propagated back to master, and a new adjusted branch is created on top of that:
+
+    * git-annex adjusted branch (HEAD -> adjusted/master(unlocked))
+    *     My new commit 2  (master, basis/adjusted/master(unlocked))
+    *     My new commit 1
+    | *   My new commit 2  [abandoned old adjusted branch]
+    | *   My new commit 1
+    | *   git-annex adjusted branch
+    |/
+    * Last old commit
+    * Previous commits
+
+This behaviour makes much more sense to me! Why does it take the modified sync command to do this? Why is this not done as well when re-calling `git annex adjust --unlock`? The sync command seems a counter-intuitive place to do this, using the adjust command would be far more intuitive for me and I think also for other users.
+
+## Problem 2
+
+I see no easy way of "un-adjusting" an adjusted branch. Currently I do
+
+    git sync --no-push --no-pull --no-content
+    git checkout master
+    git branch -D "adjusted/master(unlock)" 
+    git branch -D "refs/basis/adjusted/master(unlock)"
+
+That's a lot of text for the inverse operation of `git annex adjust --unlock` and also I have to take care myself to not forget and loose commits I did on the adjusted branch. Did I miss an easier way? If not, I think it would be a great addition.

Added a comment
diff --git a/doc/special_remotes/gcrypt/comment_3_2b22c692fb98be1a964430b5b1a329a9._comment b/doc/special_remotes/gcrypt/comment_3_2b22c692fb98be1a964430b5b1a329a9._comment
new file mode 100644
index 0000000000..b26cc75a2e
--- /dev/null
+++ b/doc/special_remotes/gcrypt/comment_3_2b22c692fb98be1a964430b5b1a329a9._comment
@@ -0,0 +1,53 @@
+[[!comment format=mdwn
+ username="aaron"
+ avatar="http://cdn.libravatar.org/avatar/8a07e2f7af4bbf1bfcb48bbc53e00747"
+ subject="comment 3"
+ date="2022-01-09T08:36:34Z"
+ content="""
+@tomdhunt, Are you saying that the difference is the rsync remote only contains the files and the actual history stuff from git isn't tracked in it while the git-remote-gcrypt one also tracks history because it is a bare git repo?
+
+Additionally, I just started trying out the grcrypt version on rsync.net and it seems to use a slightly different initialization when compared to the others.  I've made some progress, but I am still not quite able to make it work, it seems that I'm having issues initializing the bare remote when I do it via the terminal.  If I don't try to create a bare and push it the first commit completely fails, I seem to be able to make more progress by creating a bare, pushing it, and then adding it (but it still fails).  This is what I have got to so far:
+
+```bash
+user@localhost:$ sudo chown <rsync.net user>:<rsync.net user> -R
+user@localhost:$ git init --bare shared=group test_repo.git
+user@localhost:$ sudo rsync -vrSP test_repo.git <rsync.net user>@<server>:annex
+user@localhost:$ git annex initremote \"<some_userful_name>\" type=gcrypt gitrepo=<rsync.net user>@<server>:annex chunk=1MiB keyid=<key_id> encryption=shared mac=HMACSHA512 autoenable=true
+```
+
+The error message that I get:
+
+```bash
+user@localhost:$ git annex sync
+commit 
+On branch master
+
+Initial commit
+
+nothing to commit (create/copy files and use \"git add\" to track)
+ok
+pull <rsync.net server> 
+gcrypt: Decrypting manifest
+gpg: selecting card failed: No such device
+gpg: Signature made Sun 09 Jan 2022 08:26:18 AM GMT
+gpg:                using EDDSA key <key>
+gpg: Good signature from \"<key comment>\" [ultimate]
+
+merge: refs/remotes/<remote name>/master - not something we can merge
+
+merge: refs/remotes/<remote name>/synced/master - not something we can merge
+failed
+sync: 1 failed
+
+```
+
+It also looks like this method fails to add `gcrypt-participants = <key>` and `gcrypt-signingkey = <key>` to the `.git/config` file like webapp does.
+
+Furthermore, when I use the `git annex webapp` to generate the repo, it does something that seems to be even more different (and successfully creates the bare repo by itself), specifically the URL looks something like this:
+
+```bash
+url = gcrypt::<rsync.net user>@git-annex-.<country>.2D<server_subname?>.2E<server_name>.2E<server_domain>-<rsync.net user>_22_annex:annex/
+```
+
+It seems to be encoding some of the characters to make a URL?  Is there another web API that we can interact with?
+"""]]

Added a comment
diff --git a/doc/special_remotes/gcrypt/comment_2_08245fd6716ee3ed5ac98fe95e7d5ca1._comment b/doc/special_remotes/gcrypt/comment_2_08245fd6716ee3ed5ac98fe95e7d5ca1._comment
new file mode 100644
index 0000000000..425f696e59
--- /dev/null
+++ b/doc/special_remotes/gcrypt/comment_2_08245fd6716ee3ed5ac98fe95e7d5ca1._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="tomdhunt"
+ avatar="http://cdn.libravatar.org/avatar/02694633d0fb05bb89f025cf779218a3"
+ subject="comment 2"
+ date="2022-01-09T05:45:35Z"
+ content="""
+The gcrypt special remote is for use with the git-remote-gcrypt protocol for making encrypted git remotes. The rsync special remote can also encrypt the files it stores, but it's not related to git-remote-gcrypt; it just puts files in a plain directory tree using rsync.
+
+If you want to keep a repository remote (not a special remote) on your rsync.net host, and want it to be encrypted, then you can use git-remote-gcrypt and use this special remote so that the data is all together. If you're not using git-remote-gcrypt, then the rsync special remote is what you want.
+"""]]

Added a comment: How does the gcrypt type compare to the rsync special remote
diff --git a/doc/special_remotes/gcrypt/comment_1_16a7b4978a58284b5ab3c58f2102e9d0._comment b/doc/special_remotes/gcrypt/comment_1_16a7b4978a58284b5ab3c58f2102e9d0._comment
new file mode 100644
index 0000000000..b0daadff39
--- /dev/null
+++ b/doc/special_remotes/gcrypt/comment_1_16a7b4978a58284b5ab3c58f2102e9d0._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="aaron"
+ avatar="http://cdn.libravatar.org/avatar/8a07e2f7af4bbf1bfcb48bbc53e00747"
+ subject="How does the gcrypt type compare to the rsync special remote"
+ date="2022-01-09T05:31:34Z"
+ content="""
+For systems (such as rsync.net) that allow us to use both rsync and git-annex, which is the better option?  I noticed that the default for rsync.net was this type (gcrypt), does it provide additional capabilities or perform slightly better than the [rsync version](https://git-annex.branchable.com/special_remotes/rsync/)?
+"""]]

Added a comment: Not auto-signing commits with webapp (and possible assistant)
diff --git a/doc/tips/using_signed_git_commits/comment_3_3edce6f128e96e1156601c6136118427._comment b/doc/tips/using_signed_git_commits/comment_3_3edce6f128e96e1156601c6136118427._comment
new file mode 100644
index 0000000000..5144396baf
--- /dev/null
+++ b/doc/tips/using_signed_git_commits/comment_3_3edce6f128e96e1156601c6136118427._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="aaron"
+ avatar="http://cdn.libravatar.org/avatar/8a07e2f7af4bbf1bfcb48bbc53e00747"
+ subject="Not auto-signing commits with webapp (and possible assistant)"
+ date="2022-01-09T04:29:26Z"
+ content="""
+Are we still concerned about this (and does this still apply if we are using encrypted remotes?)?  I do have my normal git stuff configured to use GPG signing because I like the concept, but noticed that when running `git annex webapp` the automated commits seemed to bypass my normal global git config and not add a signature.
+"""]]

Added a comment
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_6_aafa4a00a9579e54c3080586c1b7bb3e._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_6_aafa4a00a9579e54c3080586c1b7bb3e._comment
new file mode 100644
index 0000000000..f5136565b2
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_6_aafa4a00a9579e54c3080586c1b7bb3e._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="amerlyq"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 6"
+ date="2022-01-08T14:32:11Z"
+ content="""
+Now, if Android is varying the mtime it reports for files [...]
+
+> I tried, using a directory special remote, touching a file in the remote after having already imported it once.
+
+Hm, I think I will enable debug logging for awhile, and will try to catch more info for my heisenbug.
+It may take weeks though, so simply know that no activity in this issue does not mean I had abandoned it.
+I will explicitly state so, if it will ever be a case.
+
+> On the merge commits, importing creates one, and exporting creates one. So sync creates two.
+> Also, if you export and then merge the remote tracking branch (a fast-forward merge), and then export again, it makes another merge commit. 
+
+Yes, and I hoped for a fast and dirty fix -- check diff before merge -- and if it's empty -- don't do that useless merge commit.
+It will unblock my primary workflow to start using ADB in full, as I stop fearing to trash my history on all my remotes (as I mentioned \"rebase\" won't help due to how \"git annex sync\" works).
+But maybe on empty commits still better to print something into debug logs or in warnings -- so the original bug still could be tracked and I continued searching for root cause.
+
+> See 1503b86a14865ce300ebb9c4d96315eeb254d0b8 (and subsequent 2bd0e07ed83db39907f0c824854d68c1a8ba77ac and a32f31235a67d572d989ad9e344efe11d78774a5 where this was introduced. This stuff makes my head hurt, and getting it wrong leads to broken merges from the remote tracking branch...
+
+I skimmed through those diffs, and I may say my head huts too :)
+And I will need to look more into surrounding code to understand them in full.
+Still I will return to them again after some debug logs were collected.
+
+Until then -- is it possible to do what I mentioned above -- \"check diff before merge -- and don't merge if it's empty\" ?
+"""]]

Added a comment
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_6_77416807482416c552922d39a0319a65._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_6_77416807482416c552922d39a0319a65._comment
new file mode 100644
index 0000000000..152645c26d
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_6_77416807482416c552922d39a0319a65._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="amerlyq"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 6"
+ date="2022-01-08T14:15:55Z"
+ content="""
+> open to being argued out of my current position
+
+Ok, let's continue :)
+
+> same problem:
+
+> * checking the files into a git repository not using git-annex, and pulling from that repository.
+
+> * running git-annex add and using git-annex get to transfer over a ssh connection.
+
+> * [not supporting] workflow with adb or some other type of remote [is not a bug]
+
+I distinguish ADB from all other types of remotes -- because it's the actual *source* of new files -- not yet processed by user manually.
+And what you mentioned above -- are scenarios occuring on full-fledged work system, not on half-baked android phone.
+
+When you sync Laptop with PC -- you must add files either on Laptop or on PC into git-annex first.
+Therefore you have an opportunity to do something with files first e.g. sort them into the folders by date,
+before adding them into git and losing that mtime information (which at that point is still useful, but not necessary).
+
+When you sync PC with any \"backup\" remote -- they are pushed/pulled *after* files were added to git-annex.
+I.e. none of them adds new files, which user never seen before -- and process only \"existing\" ones.
+
+But when you use ADB (or maybe Directory too -- however I still don't have a usecase for that) -- new files are added to ADB
+directly, avoiding user intervention. Because it's a pain to sort them on the phone immeditely without proper tools and scripts.
+And one of the purposes of using git-annex here -- is to fetch them to PC to sort properly on big screen.
+But fetching them without \"pull -a\" looses the necessary information.
+It's not that big of a problem for DCIM folder, as files contain dates in filenames, but it's an issue for Downloads (and separate folders of each chat app).
+
+Therefore yes, ADB is different, it's involved into different workflow, and therefore deserves different treatment.
+
+---
+
+> when you git-annex add a file, the mtime of the file (now a symlink) should also be unchanged
+Ok, that's a different original reason. Agreed.
+
+Still, it has a nice consequence of preserving mtime for files already present on PC.
+
+And it allows me to scan whole filesystem and dump metadata into a separate file (e.g. \"find -printf \"%T@ %P\n\"),
+to preserve the information \"when I first seen/downloaded that file\" for the future.
+
+And it's very important information (at least for me), because it's easier to remember and link related things occuring in a similar timespan,
+than to sort files by types and then fruitlessly trying to link those fragmented and sparse datasets inside my head after several months or years.
+"""]]

fix handling of corrupted data received from git remote
Recover from corrupted content being received from a git remote due eg to a
wire error, by deleting the temporary file when it fails to verify. This
prevents a retry from failing again.
Reversion introduced in version 8.20210903, when incremental verification
was added.
Only the git remote seems to be affected, although it is certianly
possible that other remotes could later have the same issue. This only
affects things passed to getViaTmp that return (False, UnVerified) due to
verification failing. As far as getViaTmp can tell, that could just as well
mean that the transfer failed in a way that would resume, so it cannot
delete the temp file itself. Remote.Git and P2P.Annex use getViaTmp internally,
while other remotes do not, which is why only it seems affected.
A better fix perhaps would be to improve the types of the callback
passed to getViaTmp, so that some other value could be used to indicate
the state where the transfer succeeded but verification failed.
Sponsored-by: Boyd Stephen Smith Jr.
diff --git a/Annex/Content.hs b/Annex/Content.hs
index e48e9d6d32..68d21506f1 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -1,6 +1,6 @@
 {- git-annex file content managing
  -
- - Copyright 2010-2021 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -19,6 +19,7 @@ module Annex.Content (
 	RetrievalSecurityPolicy(..),
 	getViaTmp,
 	getViaTmpFromDisk,
+	verificationOfContentFailed,
 	checkDiskSpaceToGet,
 	checkSecureHashes,
 	prepTmp,
@@ -222,7 +223,6 @@ getViaTmpFromDisk rsp v key af action = checkallowed $ do
 	tmpfile <- prepTmp key
 	resuming <- liftIO $ R.doesPathExist tmpfile
 	(ok, verification) <- action tmpfile
-	liftIO $ print ok
 	-- When the temp file already had content, we don't know if
 	-- that content is good or not, so only trust if it the action
 	-- Verified it in passing. Otherwise, force verification even
@@ -236,16 +236,7 @@ getViaTmpFromDisk rsp v key af action = checkallowed $ do
 		then ifM (verifyKeyContentPostRetrieval rsp v verification' key tmpfile)
 			( pruneTmpWorkDirBefore tmpfile (moveAnnex key af)
 			, do
-				warning "verification of content failed"
-				-- The bad content is not retained, because
-				-- a retry should not try to resume from it
-				-- since it's apparently corrupted.
-				-- Also, the bad content could be any data,
-				-- including perhaps the content of another
-				-- file than the one that was requested,
-				-- and so it's best not to keep it on disk.
-				pruneTmpWorkDirBefore tmpfile
-					(liftIO . removeWhenExistsWith R.removeLink)
+				verificationOfContentFailed tmpfile
 				return False
 			)
 		-- On transfer failure, the tmp file is left behind, in case
@@ -264,6 +255,24 @@ getViaTmpFromDisk rsp v key af action = checkallowed $ do
 				)
 			)
 
+{- When the content of a file that was successfully transferred from a remote
+ - fails to verify, use this to display a message so the user knows why it
+ - failed, and to clean up the corrupted content.
+ -
+ - The bad content is not retained, because the transfer of it succeeded.
+ - So it's not incomplete and a resume using it will not work. While
+ - some protocols like rsync could recover such a bad content file,
+ - they are assumed to not write out bad data to a file in the first place.
+ - Most protocols, including the P2P protocol, pick up downloads where they
+ - left off, and so if the bad content were not deleted, repeated downloads
+ - would continue to fail.
+ -}
+verificationOfContentFailed :: RawFilePath -> Annex ()
+verificationOfContentFailed tmpfile = do
+	warning "Verification of content failed"
+	pruneTmpWorkDirBefore tmpfile
+		(liftIO . removeWhenExistsWith R.removeLink)
+
 {- Checks if there is enough free disk space to download a key
  - to its temp file.
  -
diff --git a/CHANGELOG b/CHANGELOG
index c6c35d37ba..79d655a6b4 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,10 +5,10 @@ git-annex (8.20211232) UNRELEASED; urgency=medium
     preserve it in the imported tree so it does not get deleted.
   * enableremote, renameremote: Better handling of the unusual case where
     multiple special remotes have been initialized with the same name.
-  * Recover from over the wire errors when downloading from remotes,
-    by deleting the object file when verification of it fails. This allows
-    the next attempt at a download to succeed, rather than using the same
-    content and failing again.
+  * Recover from corrupted content being received from a git remote,
+    by deleting the temporary file when it fails to verify. This prevents
+    a retry from failing again. 
+    (reversion introduced in version 8.20210903)
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Jan 2022 14:01:14 -0400
 
diff --git a/P2P/Annex.hs b/P2P/Annex.hs
index a2641d38da..08177bec6e 100644
--- a/P2P/Annex.hs
+++ b/P2P/Annex.hs
@@ -1,6 +1,6 @@
 {- P2P protocol, Annex implementation
  -
- - Copyright 2016-2021 Joey Hess <id@joeyh.name>
+ - Copyright 2016-2022 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -206,7 +206,9 @@ runLocal runst runner a = case a of
 							| rightsize -> liftIO (finalizeIncrementalVerifier iv) >>= \case
 								Nothing -> return (True, UnVerified)
 								Just True -> return (True, Verified)
-								Just False -> return (False, UnVerified)
+								Just False -> do
+									verificationOfContentFailed (toRawFilePath dest)
+									return (False, UnVerified)
 							| otherwise -> return (False, UnVerified)
 						Nothing -> return (rightsize, UnVerified)
 					Right (Just Invalid) | l == 0 ->
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 99ccc60330..45e79ee348 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -30,6 +30,7 @@ import Logs.Presence
 import Annex.Transfer
 import Annex.CopyFile
 import Annex.Verify
+import Annex.Content (verificationOfContentFailed)
 import Annex.UUID
 import qualified Annex.Content
 import qualified Annex.BranchState
@@ -706,7 +707,9 @@ mkFileCopier remotewanthardlink (State _ _ copycowtried _ _) = do
 			ifM (liftIO (catchBoolIO (linker src dest)))
 				( ifM check
 					( return (True, Verified)
-					, return (False, UnVerified)
+					, do
+						verificationOfContentFailed (toRawFilePath dest)
+						return (False, UnVerified)
 					)
 				, copier src dest k p check verifyconfig
 				)
@@ -717,7 +720,9 @@ mkFileCopier remotewanthardlink (State _ _ copycowtried _ _) = do
 		fileCopier copycowtried src dest p iv >>= \case
 			Copied -> ifM check
 				( finishVerifyKeyContentIncrementally iv
-				, return (False, UnVerified)
+				, do
+					verificationOfContentFailed (toRawFilePath dest)
+					return (False, UnVerified)
 				)
 			CopiedCoW -> unVerified check
 
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol.mdwn b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol.mdwn
index 2b8eb58278..cb0a2b663b 100644
--- a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol.mdwn
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol.mdwn
@@ -115,3 +115,6 @@ Both servers are debian.
 I've been using git annex for years now, some periods more intensively than others. The only data I've lost I can only blame on my own stupidity, even after a terminal disk failure I managed to recover almost everything with a reinject. I'm still using it entirely cli though, no assistants or gui for me.
 
 [[!meta title="corrupted tmp file can prevent getting an object"]]
+
+> [[fixed|done]] (the related issue that I discovered in comment #1
+> specifically) --[[Joey]]

comment
diff --git a/Annex/Content.hs b/Annex/Content.hs
index 58f1244070..e48e9d6d32 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -222,6 +222,7 @@ getViaTmpFromDisk rsp v key af action = checkallowed $ do
 	tmpfile <- prepTmp key
 	resuming <- liftIO $ R.doesPathExist tmpfile
 	(ok, verification) <- action tmpfile
+	liftIO $ print ok
 	-- When the temp file already had content, we don't know if
 	-- that content is good or not, so only trust if it the action
 	-- Verified it in passing. Otherwise, force verification even
diff --git a/CHANGELOG b/CHANGELOG
index b8c375d323..c6c35d37ba 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,10 @@ git-annex (8.20211232) UNRELEASED; urgency=medium
     preserve it in the imported tree so it does not get deleted.
   * enableremote, renameremote: Better handling of the unusual case where
     multiple special remotes have been initialized with the same name.
+  * Recover from over the wire errors when downloading from remotes,
+    by deleting the object file when verification of it fails. This allows
+    the next attempt at a download to succeed, rather than using the same
+    content and failing again.
 
  -- Joey Hess <id@joeyh.name>  Mon, 03 Jan 2022 14:01:14 -0400
 
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_6_5ed6591954aeafe6c99ae152f4f4ad67._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_6_5ed6591954aeafe6c99ae152f4f4ad67._comment
new file mode 100644
index 0000000000..08dc1ba5e3
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_6_5ed6591954aeafe6c99ae152f4f4ad67._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2022-01-07T16:12:20Z"
+ content="""
+Current thinking on deleting corrupted tmp files: If a download succeeds,
+and verification then fails, the whole file content has been downloaded,
+and is corrupt. So it would be ok to always delete it then, as far as p2p
+transfers goes.
+
+For other remotes, the same is often true. The only exceptions are like
+rsync and bittorrent, which can recover from corruption on retry. But,
+I don't think either rsync or bittorrent will usually write corrupt data
+to a file anyway. They would catch over-the-wire corruption with rolling
+checksums etc. So, it seems like a verification should never fail after
+a successful rsync or bittorrent download. Unless the disk corrupted the
+data in the meantime. Which is an unlikely situation, and not one that it's
+really necessary for git-annex to recover from with optimal efficiency.
+
+... Oh interesting.. It already is supposed to do that, in
+getViaTmpFromDisk. It seems, what is happening is the transfer fails
+when all the file content is present, and so it never gets to the point of
+verifying it, let alone deleting it.
+"""]]

comment
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_5_244a5803178b4e44f3ec4460c5daf258._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_5_244a5803178b4e44f3ec4460c5daf258._comment
new file mode 100644
index 0000000000..38f5cf2239
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_5_244a5803178b4e44f3ec4460c5daf258._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2022-01-07T16:06:52Z"
+ content="""
+This bug report is still open because I am still open to being argued out
+of my current position. But such an argument would have to engage with the
+things I discussed.
+"""]]

comment
diff --git a/doc/git-annex-adjust/comment_6_f00c27078cce5ad75fcf0cec1033751b._comment b/doc/git-annex-adjust/comment_6_f00c27078cce5ad75fcf0cec1033751b._comment
new file mode 100644
index 0000000000..dcb26546ae
--- /dev/null
+++ b/doc/git-annex-adjust/comment_6_f00c27078cce5ad75fcf0cec1033751b._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2022-01-07T16:03:13Z"
+ content="""
+@lell this is a man page, it's really not a place to file bug reports
+or discuss in detail complicated scenarios. 
+
+Please use [[bugs]] and [[forum]] for such things.
+
+Comments are open on this page
+in case someone has a suggestion to improve this documentation, not for
+general discussion about the feature.
+"""]]

Added a comment
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_4_9d5cbd2479a07d778bacfdfe0aaaa524._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_4_9d5cbd2479a07d778bacfdfe0aaaa524._comment
new file mode 100644
index 0000000000..6fc38031b2
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_4_9d5cbd2479a07d778bacfdfe0aaaa524._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="amerlyq"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 4"
+ date="2022-01-07T12:52:35Z"
+ content="""
+Basical consclusion is: you won't ever add \"pull -a\" (despite android's \"nosuid noexec\"), correct ?
+"""]]

Added a comment: Propagation is different between "annex adjust" and "annex sync"
diff --git a/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment b/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment
new file mode 100644
index 0000000000..7e880ee07a
--- /dev/null
+++ b/doc/git-annex-adjust/comment_5_4c16fdc3f3eb5f2f062c7aa50b10afef._comment
@@ -0,0 +1,62 @@
+[[!comment format=mdwn
+ username="lell"
+ avatar="http://cdn.libravatar.org/avatar/4c4138a71d069e290240a3a12367fabe"
+ subject="Propagation is different between &quot;annex adjust&quot; and &quot;annex sync&quot;"
+ date="2022-01-07T10:21:46Z"
+ content="""
+Adjusted branches are important to my data science project, because my programs cannot deal with the read-only symlinks to annex'ed files.
+
+But I find this command confusing, especially that
+
+1. Calling on an unlocked adjusted branch, \"git annex adjust --unlock\" propagates commits back to the master branch differently than \"git annex sync --no-push --no-pull --no-content\" does. 
+
+2. I can't find a way to \"un-adjust\" a branch without resorting to lower-level git commands.
+
+## Problem 1:
+
+Say I have done `git annex adjust --unlock` and then have done more commits. The history now looks like this: 
+
+    * My new commit 2  (HEAD -> adjusted/master(unlocked))
+    * My new commit 1  (HEAD -> adjusted/master(unlocked))
+    * git-annex adjusted branch
+    * Last old commit (master, basis/adjusted/master(unlocked))
+    * Previous commits
+
+If I execute now `git annex adjust --unlock` again, the commits are propagated back to the original branch, 
+but my HEAD is still on the original adjusted branch. So both the master branch and my adjusted branch grow over time which clutters the history and is confusing.
+
+    *     My new commit 2  (master)
+    | *   My new commit 2  (HEAD -> adjusted/master(unlocked))
+    * |   My new commit 1
+    | *   My new commit 1
+    | *   git-annex adjusted branch
+    |/
+    * Last old commit (basis/adjusted/master(unlocked))
+    * Previous commits
+
+On the other hand, if I do `git annex sync --no-push --no-pull --no-content`, the commits are propagated back to master, and a new adjusted branch is created on top of that:
+
+    * git-annex adjusted branch (HEAD -> adjusted/master(unlocked))
+    *     My new commit 2  (master, basis/adjusted/master(unlocked))
+    *     My new commit 1
+    | *   My new commit 2  [abandoned old adjusted branch]
+    | *   My new commit 1
+    | *   git-annex adjusted branch
+    |/
+    * Last old commit
+    * Previous commits
+
+This behaviour makes much more sense to me! Why does it take the modified `sync` command to do this? Why is this not done as well when re-calling `annex adjust --unlock`? The sync command seems a counter-intuitive place to do this, using the `adjust` command would be far more intuitive for me and I think also for other users.
+
+
+## Problem 2
+
+I see no easy way of \"un-adjusting\" an adjusted branch. Currently I do
+
+    git sync --no-push --no-pull --no-content
+    git checkout master
+    git branch -D \"adjusted/master(unlock)\" 
+    git branch -D \"refs/basis/adjusted/master(unlock)\"
+
+That's a lot of text for the inverse operation of `git annex adjust --unlock` and also I have to take care myself to not forget and loose commits I did on the adjusted branch. Did I miss an easier way? If not, I think it would be a great addition. 
+"""]]

comment
diff --git a/doc/bugs/describe_what_git-annex_actually_is/comment_2_1f6ec7cedf56863b261bad0e3f3c80de._comment b/doc/bugs/describe_what_git-annex_actually_is/comment_2_1f6ec7cedf56863b261bad0e3f3c80de._comment
new file mode 100644
index 0000000000..70272d5e84
--- /dev/null
+++ b/doc/bugs/describe_what_git-annex_actually_is/comment_2_1f6ec7cedf56863b261bad0e3f3c80de._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-06T16:23:15Z"
+ content="""
+Thanks, these are good suggestions. I've made some improvements to the
+description.
+"""]]

improve description
Sponsored-by: Jack Hill on Patreon
diff --git a/README b/README
index 3ff0a9bfb0..8d9c559f0b 100644
--- a/README
+++ b/README
@@ -1,6 +1,7 @@
-git-annex allows managing files with git, without checking the file
-contents into git. While that may seem paradoxical, it is useful when
-dealing with files larger than git can currently easily handle, whether due
-to limitations in memory, checksumming time, or disk space.
+git-annex allows managing large files with git, without storing the file
+contents in git. It can sync, backup, and archive your data, offline
+and online. Checksums and encryption keep your data safe and secure. Bring
+the power and distributed nature of git to bear on your large files with
+git-annex.
 
 For documentation, see doc/ or <https://git-annex.branchable.com/>
diff --git a/debian/control b/debian/control
index d1d3697fd3..f999940249 100644
--- a/debian/control
+++ b/debian/control
@@ -124,14 +124,15 @@ Suggests:
 	uftp,
 Breaks: datalad (< 0.12.3~)
 Description: manage files with git, without checking their contents into git
- git-annex allows managing files with git, without checking the file
- contents into git. While that may seem paradoxical, it is useful when
- dealing with files larger than git can currently easily handle, whether due
- to limitations in memory, time, or disk space.
+ git-annex allows managing large files with git, without storing the file 
+ contents in git. It can sync, backup, and archive your data, offline
+ and online. Checksums and encryption keep your data safe and secure. Bring
+ the power and distributed nature of git to bear on your large files with
+ git-annex.
  .
  It can store large files in many places, from local hard drives, to a
  large number of cloud storage services, including S3, WebDAV,
- and rsync, with a dozen cloud storage providers usable via plugins.
+ and rsync, with dozens of cloud storage providers usable via plugins.
  Files can be stored encrypted with gpg, so that the cloud storage
  provider cannot see your data. git-annex keeps track of where each file
  is stored, so it knows how many copies are available, and has many
diff --git a/doc/bugs/describe_what_git-annex_actually_is.mdwn b/doc/bugs/describe_what_git-annex_actually_is.mdwn
index 237cf2584a..feee65a0e5 100644
--- a/doc/bugs/describe_what_git-annex_actually_is.mdwn
+++ b/doc/bugs/describe_what_git-annex_actually_is.mdwn
@@ -10,3 +10,5 @@ The second sentence does not really help to explain the tool, for me it sounds a
 > It enables managing files larger than git can easily handle, whether due to limitations in memory, time, or disk space.
 
 Still not optimal, but feels a little more precise
+
+> [[done]] --[[Joey]]
diff --git a/doc/summary.mdwn b/doc/summary.mdwn
index be7b2cc2c1..a5e5400d3b 100644
--- a/doc/summary.mdwn
+++ b/doc/summary.mdwn
@@ -1,7 +1,8 @@
-git-annex allows managing files with git, without checking the file
-contents into git. While that may seem paradoxical, it is useful when
-dealing with files larger than git can currently easily handle, whether due
-to limitations in memory, time, or disk space.
+git-annex allows managing large files with git, without storing the file
+contents in git. It can sync, backup, and archive your data, offline
+and online. Checksums and encryption keep your data safe and secure. Bring
+the power and distributed nature of git to bear on your large files with
+git-annex.
 
 [[!img assistant/thumbnail.png link=assistant align=right]]
 git-annex is designed for git users who love the command line.

Added a comment: reply
diff --git a/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_2_9f5d0e8d0c6eff548318b54797cefd09._comment b/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_2_9f5d0e8d0c6eff548318b54797cefd09._comment
new file mode 100644
index 0000000000..850d80e8c3
--- /dev/null
+++ b/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_2_9f5d0e8d0c6eff548318b54797cefd09._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="arekmenner@f0f5403c5cbc803fedaf860e3a6f839a41075e3b"
+ nickname="arekmenner"
+ avatar="http://cdn.libravatar.org/avatar/ec065e4a23146814e887fdcd16a1ee44"
+ subject="reply"
+ date="2022-01-06T15:13:15Z"
+ content="""
+Very much so. I'll try to start this whole computer from a clean setup and we'll see where we get from there.
+
+My guess at what happened was it was a combination of interleaved inits/enables in desperation. In particular, any time a \"get\" failed it asked me to \"enableremote\" which I tried several times.
+"""]]

comment
diff --git a/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_1_708da81b0678576a240ce7a388806e07._comment b/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_1_708da81b0678576a240ce7a388806e07._comment
new file mode 100644
index 0000000000..7e9d5713f5
--- /dev/null
+++ b/doc/forum/Windows_-_gpg_fails_with_shared_encryption/comment_1_708da81b0678576a240ce7a388806e07._comment
@@ -0,0 +1,27 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-05T19:44:24Z"
+ content="""
+The shared encryption key is what git-annex passes to gpg as
+"C:\\Users\\REDACTED\\AppData\\Local\\Temp\\gpg8159".
+
+From the other forum thread,
+[[Failing_to_get_files_from_rclone_special_remote]], 
+you had many special remotes that somehow existed with
+the same name.
+
+There's a big question mark over how you even got into that state.
+initremote does not let two special remotes with the same name be created
+in the same repository so normally it would be rather hard to accomplish
+once, let alone multiple times. Without understanding more, it's hard for
+me to guess what has happened.
+
+It also might be some problem with your gpg on windows. 
+googling for "gpg: decryption failed: Bad session key" seems to find that
+the problem is split between the wrong encryption key and some kind of gpg
+bugs.
+
+If you still have the data, would deleting that special remote and starting
+over with setting up a new one (with a different name!) be an option?
+"""]]

remove incorrect speculation
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment
index a0a5da33ef..7edf7a61f0 100644
--- a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment
@@ -3,10 +3,7 @@
  subject="""comment 4"""
  date="2022-01-05T19:24:50Z"
  content="""
-Thanks for verifying. If decryption is failing, I'd guess you somehow
-picked the wrong one to enable. The encryption information is included in
-the remote's configuration, which could otherwise be close enough that it's
-able to get files from the remote, but then fail to decrypt them.
+Thanks for verifying.
 
 Also FWIW, I'm pushing some changes that improve `git-annex enableremote`
 in this situation, so it does not choose a random remote to enable.

description of another noob question
diff --git a/doc/forum/Windows_-_gpg_fails_with_shared_encryption.mdwn b/doc/forum/Windows_-_gpg_fails_with_shared_encryption.mdwn
new file mode 100644
index 0000000000..fea1ffe9d6
--- /dev/null
+++ b/doc/forum/Windows_-_gpg_fails_with_shared_encryption.mdwn
@@ -0,0 +1,82 @@
+I have an rclone special remote that I can get my Windows computer to see, but it seems to be messing up actually getting the file:
+
+By the looks of it, either the temp file that GPG uses doesn't get made, or it gets made but with incorrect contents:
+
+```
+$ git annex get --debug 'Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif'
+[2022-01-05 14:22:52.4414076] (Utility.Process) process [8196] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
+[2022-01-05 14:22:52.4594467] (Utility.Process) process [8196] done ExitSuccess
+[2022-01-05 14:22:52.4615933] (Utility.Process) process [16468] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/main"]
+[2022-01-05 14:22:52.4934873] (Utility.Process) process [16468] done ExitSuccess
+[2022-01-05 14:22:52.4974156] (Utility.Process) process [16656] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--","Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif"]
+[2022-01-05 14:22:52.5004149] (Utility.Process) process [3996] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
+[2022-01-05 14:22:52.5035735] (Utility.Process) process [17168] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
+[2022-01-05 14:22:52.5243346] (Utility.Process) process [17136] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
+get Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif [2022-01-05 14:22:52.5593587] (Utility.Process) process [22208] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
+[2022-01-05 14:22:52.5862972] (Utility.Process) process [22208] done ExitSuccess
+[2022-01-05 14:22:52.5902901] (Utility.Process) process [18248] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
+[2022-01-05 14:22:52.610443] (Utility.Process) process [18248] done ExitSuccess
+[2022-01-05 14:22:52.6143883] (Utility.Process) process [19620] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..3671f1ffc7c0744725c504a01fc8a67fa4539a4a","--pretty=%H","-n1"]
+[2022-01-05 14:22:52.6462441] (Utility.Process) process [19620] done ExitSuccess
+[2022-01-05 14:22:52.6482446] (Utility.Process) process [20716] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..3cf61e8a81a8ee97dbfbdf0a86323380e0bc942a","--pretty=%H","-n1"]
+[2022-01-05 14:22:52.6822859] (Utility.Process) process [20716] done ExitSuccess
+[2022-01-05 14:22:52.6910388] (Utility.Process) process [16484] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","hash-object","-w","--stdin-paths","--no-filters"]
+[2022-01-05 14:22:52.7008252] (Utility.Process) process [6724] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
+[2022-01-05 14:22:52.7066723] (Utility.Process) process [12884] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-index","-z","--index-info"]
+[2022-01-05 14:22:52.7106744] (Utility.Process) process [17688] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex","--"]
+[2022-01-05 14:22:52.7624402] (Utility.Process) process [17688] done ExitSuccess
+[2022-01-05 14:22:52.7678167] (Utility.Process) process [12884] done ExitSuccess
+(from pcloud_annex...)
+[2022-01-05 14:22:52.8412254] (Utility.Process) process [18812] chat: bash ["C:\\Users\\REDACTED\\Documents\\Mine\\programs\\bin\\git-annex-remote-rclone"]
+[2022-01-05 14:22:52.9153659] (Annex.ExternalAddonProcess) bash[1] --> VERSION 1
+[2022-01-05 14:22:52.9155845] (Annex.ExternalAddonProcess) bash[1] <-- EXTENSIONS INFO GETGITREMOTENAME ASYNC
+[2022-01-05 14:22:52.9155845] (Annex.ExternalAddonProcess) bash[1] --> UNSUPPORTED-REQUEST
+[2022-01-05 14:22:52.9155845] (Annex.ExternalAddonProcess) bash[1] <-- PREPARE
+[2022-01-05 14:22:52.9155845] (Annex.ExternalAddonProcess) bash[1] --> GETCONFIG prefix
+[2022-01-05 14:22:52.9155845] (Annex.ExternalAddonProcess) bash[1] <-- VALUE test_annex
+[2022-01-05 14:22:53.0371658] (Annex.ExternalAddonProcess) bash[1] --> GETCONFIG target
+[2022-01-05 14:22:53.0371658] (Annex.ExternalAddonProcess) bash[1] <-- VALUE pcloud_test_annex
+[2022-01-05 14:22:53.1413009] (Annex.ExternalAddonProcess) bash[1] --> GETCONFIG rclone_layout
+[2022-01-05 14:22:53.1413009] (Annex.ExternalAddonProcess) bash[1] <-- VALUE lower
+[2022-01-05 14:22:53.2805175] (Annex.ExternalAddonProcess) bash[1] --> PREPARE-SUCCESS
+[2022-01-05 14:22:53.2805175] (Annex.ExternalAddonProcess) bash[1] <-- TRANSFER RETRIEVE GPGHMACSHA512--ceb25ebd86f4098b366419fbed8fc388e421cc38429ce2da09688b1ba4575333cea33b1920455c2d4b4dfc28a1d7027d91be028f059322b10d1d9a0731adbedb .git\annex\tmp\GPGHMACSHA512--ceb25ebd86f4098b366419fbed8fc388e421cc38429ce2da09688b1ba4575333cea33b1920455c2d4b4dfc28a1d7027d91be028f059322b10d1d9a0731adbedb
+[2022-01-05 14:22:53.2805175] (Annex.ExternalAddonProcess) bash[1] --> DIRHASH-LOWER GPGHMACSHA512--ceb25ebd86f4098b366419fbed8fc388e421cc38429ce2da09688b1ba4575333cea33b1920455c2d4b4dfc28a1d7027d91be028f059322b10d1d9a0731adbedb
+[2022-01-05 14:22:53.2805175] (Annex.ExternalAddonProcess) bash[1] <-- VALUE 34d\c20\
+[2022-01-05 14:23:01.8937568] (Annex.ExternalAddonProcess) bash[1] --> TRANSFER-SUCCESS RETRIEVE GPGHMACSHA512--ceb25ebd86f4098b366419fbed8fc388e421cc38429ce2da09688b1ba4575333cea33b1920455c2d4b4dfc28a1d7027d91be028f059322b10d1d9a0731adbedb
+[2022-01-05 14:23:01.9017536] (Utility.Process) process [5260] chat: gpg ["--quiet","--trust-model","always","--batch","--passphrase-file","C:\\Users\\REDACTED\\AppData\\Local\\Temp\\gpg8159","--decrypt"]
+gpg: decryption failed: Bad session key
+[2022-01-05 14:23:02.2915591] (Utility.Process) process [5260] done ExitFailure 2
+[2022-01-05 14:23:02.2915591] (Annex.ExternalAddonProcess) bash[1] <-- TRANSFER RETRIEVE GPGHMACSHA512--952a317030848a0bcc83728f8fdba124b24763f32395b48481c1ac4f1645067be126ec1711008a14cfd62920284cc34e4a0f12d0617826c402b632dafc31578e .git\annex\tmp\GPGHMACSHA512--952a317030848a0bcc83728f8fdba124b24763f32395b48481c1ac4f1645067be126ec1711008a14cfd62920284cc34e4a0f12d0617826c402b632dafc31578e
+[2022-01-05 14:23:02.2925835] (Annex.ExternalAddonProcess) bash[1] --> DIRHASH-LOWER GPGHMACSHA512--952a317030848a0bcc83728f8fdba124b24763f32395b48481c1ac4f1645067be126ec1711008a14cfd62920284cc34e4a0f12d0617826c402b632dafc31578e
+[2022-01-05 14:23:02.2925835] (Annex.ExternalAddonProcess) bash[1] <-- VALUE da7\9eb\
+2022/01/05 14:23:03 ERROR : : error reading source directory: directory not found
+2022/01/05 14:23:03 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
+2022/01/05 14:23:03 ERROR : : error reading source directory: directory not found
+2022/01/05 14:23:03 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
+2022/01/05 14:23:04 ERROR : : error reading source directory: directory not found
+2022/01/05 14:23:04 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
+2022/01/05 14:23:04 Failed to copy: directory not found
+[2022-01-05 14:23:04.1235177] (Annex.ExternalAddonProcess) bash[1] --> TRANSFER-FAILURE RETRIEVE GPGHMACSHA512--952a317030848a0bcc83728f8fdba124b24763f32395b48481c1ac4f1645067be126ec1711008a14cfd62920284cc34e4a0f12d0617826c402b632dafc31578e
+
+  user error (gpg ["--quiet","--trust-model","always","--batch","--passphrase-file","C:\\Users\\REDACTED\\AppData\\Local\\Temp\\gpg8159","--decrypt"] exited 2)
+
+  Unable to access these remotes: pcloud_annex
+
+  Maybe add some of these git remotes (git remote add ...):
+        bab5704a-562e-43eb-8bbb-71f181f043e8 -- BandName@computer-a:~/Splice/Christmas Steve Vocals
+
+  (Note that these git remotes have annex-ignore set: origin)
+failed
+[2022-01-05 14:23:04.1457308] (Utility.Process) process [17136] done ExitSuccess
+[2022-01-05 14:23:04.1457308] (Utility.Process) process [17168] done ExitSuccess
+[2022-01-05 14:23:04.1466597] (Utility.Process) process [3996] done ExitSuccess
+[2022-01-05 14:23:04.1466597] (Utility.Process) process [16656] done ExitSuccess
+[2022-01-05 14:23:04.1576581] (Utility.Process) process [18812] done ExitSuccess
+[2022-01-05 14:23:04.1740489] (Utility.Process) process [6724] done ExitSuccess
+[2022-01-05 14:23:04.188721] (Utility.Process) process [16484] done ExitSuccess
+get: 1 failed
+```
+
+I don't know how to proceed. As I understand it, shared encryption shares a key between the git repos. What file stores the shared encryption key so I could see if it's the same in both places?
+
+Is there anything else obviously dumb I'm doing?

comment
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment
new file mode 100644
index 0000000000..a0a5da33ef
--- /dev/null
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_4_1360ee3e3b711990f59e0320e7892cd2._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2022-01-05T19:24:50Z"
+ content="""
+Thanks for verifying. If decryption is failing, I'd guess you somehow
+picked the wrong one to enable. The encryption information is included in
+the remote's configuration, which could otherwise be close enough that it's
+able to get files from the remote, but then fail to decrypt them.
+
+Also FWIW, I'm pushing some changes that improve `git-annex enableremote`
+in this situation, so it does not choose a random remote to enable.
+"""]]

Added a comment: response
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_3_58282be918139a3154938850793a80fb._comment b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_3_58282be918139a3154938850793a80fb._comment
new file mode 100644
index 0000000000..f27c19a9fc
--- /dev/null
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_3_58282be918139a3154938850793a80fb._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="arekmenner@f0f5403c5cbc803fedaf860e3a6f839a41075e3b"
+ nickname="arekmenner"
+ avatar="http://cdn.libravatar.org/avatar/ec065e4a23146814e887fdcd16a1ee44"
+ subject="response"
+ date="2022-01-05T19:18:09Z"
+ content="""
+This is all completely correct and includes enough explanation and internals that I think I'm unlikely to get into this situation again.
+
+Thanks so much for your time! Now decryption is failing so I'll make another thread
+"""]]

comment
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_2_cfb6be6c940350dbed6cc9e98ee1d633._comment b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_2_cfb6be6c940350dbed6cc9e98ee1d633._comment
new file mode 100644
index 0000000000..c7076a7813
--- /dev/null
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_2_cfb6be6c940350dbed6cc9e98ee1d633._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-05T17:53:35Z"
+ content="""
+Easier option, with a newish version of git-annex:
+
+	git-annex renameremote e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d pcloud_annex
+	git-annex enableremote pcloud_annex
+"""]]

comment
diff --git a/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_2_a5587ebc2a406d52855868cdeb39b83e._comment b/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_2_a5587ebc2a406d52855868cdeb39b83e._comment
new file mode 100644
index 0000000000..60180fddb2
--- /dev/null
+++ b/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_2_a5587ebc2a406d52855868cdeb39b83e._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2022-01-05T16:59:08Z"
+ content="""
+Well, this could be anything that it happens to do in some situation to
+actually drop the file. That's what happens in the interval there.
+
+Are the files unlocked? It might be running git to update the index,
+which can take a long time.
+
+Is annex.secure-erase-command set? Such a command can take a long time.
+
+Otherwise, strace ...
+"""]]

comment
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_4e9e55d54d9afb2dbb48320156757f11._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_4e9e55d54d9afb2dbb48320156757f11._comment
new file mode 100644
index 0000000000..6dbe5a96ec
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_4e9e55d54d9afb2dbb48320156757f11._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2022-01-05T16:32:50Z"
+ content="""
+> Now, if Android is varying the mtime it reports for files [...]
+
+I do not think this idea of mine can cause it. I tried, using a directory
+special remote, touching a file in the remote after having already imported
+it once. This resulted in git-annex sync importing the same file again, but
+since the content was the same it built the same tree it had before, and
+noticed this and avoided making an empty commit.
+
+On the merge commits, importing creates one, and exporting creates one.
+So sync creates two. Also, if you export and then merge the remote tracking
+branch (a fast-forward merge), and then export again, 
+it makes another merge commit. So any number can be stacked up that way.
+
+See [[!commit 1503b86a14865ce300ebb9c4d96315eeb254d0b8]]
+(and subsequent [[!commit 2bd0e07ed83db39907f0c824854d68c1a8ba77ac]]
+and [[!commit a32f31235a67d572d989ad9e344efe11d78774a5]] where this was
+introduced. This stuff makes my head hurt, and getting it wrong leads to
+broken merges from the remote tracking branch...
+"""]]

comment
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_1_4a762c369e4049bdedecca4a32242f18._comment b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_1_4a762c369e4049bdedecca4a32242f18._comment
new file mode 100644
index 0000000000..4febd83e71
--- /dev/null
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote/comment_1_4a762c369e4049bdedecca4a32242f18._comment
@@ -0,0 +1,30 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-05T16:19:05Z"
+ content="""
+So you have several remotes with the same name, probably all but
+this one you want to use were false starts or created by accident.
+
+In this unusual situation, `git-annex enableremote` picks one more or less at
+random.
+
+(Normally this is prevented by `git-annex initremote`, which won't use a
+name that already exists, but initializing remotes with the same name in
+different clones and merging is one way it could happen.)
+
+What you can do instead is specify the uuid or description of the remote.
+Eg:
+
+	git-annex enableremote e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d
+
+The other thing you can do if you want is to mark all the other
+uuids with this name as dead, and then tell git-annex to forget about dead
+remotes. Doing this will leave only the one that you did not mark as dead,
+so enableremote by name will do what you want.
+
+	git-annex dead ebca6ee8-1baa-4a47-ae85-0a5b65db73f6
+	git-annex dead 69acaaf2-e58b-463d-a578-e2418df5c6b0
+	git-annex dead 2e771318-83eb-4beb-85f4-567896b0e835
+	git-annex forget --drop-dead
+"""]]

comment
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_3_cf555d2f0f93248a913014f30ce513b2._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_3_cf555d2f0f93248a913014f30ce513b2._comment
new file mode 100644
index 0000000000..4feb134ac8
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_3_cf555d2f0f93248a913014f30ce513b2._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2022-01-05T16:07:09Z"
+ content="""
+But you would have the same problem with that workflow if you were
+checking the files into a git repository not using git-annex, 
+and pulling from that repository. 
+
+For that metter, you would have the same problem with that workflow if
+you were running `git-annex add` and using `git-annex get`
+to transfer over a ssh connection.
+
+The mere fact that it might be possible to support that particular workflow
+with adb or some other type of remote does not mean that it's a bug for
+git-annex not to support it. Or that it's a good idea to complicate
+git-annex to partially support it.
+
+> And when you annex files -- git-annex copies file own mtime to its symlink
+
+That's entirely different. When you run `git add` on a file, the file's
+mtime is left unchanged. It follows that, when you `git-annex add` a file,
+the mtime of the file (now a symlink) should also be unchanged. That's
+entirely unrelated to remotes.
+"""]]

retrospectively add from NEWS
diff --git a/doc/news/version_8.20211028.mdwn b/doc/news/version_8.20211028.mdwn
index 8ac3155554..10ab327351 100644
--- a/doc/news/version_8.20211028.mdwn
+++ b/doc/news/version_8.20211028.mdwn
@@ -27,4 +27,9 @@ git-annex 8.20211028 released with [[!toggle text="these changes"]]
     likely to actually be stored in the remote.
   * Avoid crashing tilde expansion on user who does not exist.
   * test: Put gpg temp home directory in system temp directory,
-    not filesystem being tested."""]]
\ No newline at end of file
+    not filesystem being tested."""]]
+
+News for 8.20211028: 
+
+This version of git-annex removes support for communicating with git-annex
+remotes that have version 6.20180312 or older installed.

Added a comment
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_d500b9ae8e3f57a2f8dcd2314c0022dd._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_d500b9ae8e3f57a2f8dcd2314c0022dd._comment
new file mode 100644
index 0000000000..edb23a1ee5
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_4_d500b9ae8e3f57a2f8dcd2314c0022dd._comment
@@ -0,0 +1,73 @@
+[[!comment format=mdwn
+ username="amerlyq"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 4"
+ date="2022-01-05T15:46:07Z"
+ content="""
+> I do not thik your CID analysis is correct.
+
+Yes, sorry, you are right. You only use \"find -exec stat\" to get remote CID.
+
+> Please show me how to get more than one \"import from foo\" commit that is empty, starting from a new remote.
+
+Hm, it really looks like *heisenbug* as I can't reproduce it stably and immediately.
+Usually issue arises after some indeterminate time had passed. 
+Dunno what action could trigger it, as all of them look unrelated and improbable: phone reboot, system reboot, cold file cache, NTP skew...
+
+When I wrap everything into the script which completes in 10 sec -- there is no problem.
+But after several days -- import creates new empty commit and export refuses to overwrite existing file.
+
+I read something kind of similar months ago: https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1631854.html
+
+---
+
+Let's focus on empty \"remote tracking branch\" merge commits for now.
+At least until I find how to reproduce empty \"import from foo\" too.
+
+Can we avoid new empty merge commits if \"import\" was done first?
+
+    #!/usr/bin/env bash
+    set -o errexit -o errtrace -o noclobber -o noglob -o nounset -o pipefail
+
+    adb shell \"rm -rf /storage/0000-0000/ttt\"
+
+    cd /tmp
+    test -d ttt && find ttt -not -writable -exec chmod +w -- {} +
+    rm -rf ttt
+    mkdir ttt
+
+    cd ttt
+    git init .
+    git config --local init.defaultBranch main
+    git config --local user.name annex
+    git config --local user.email test
+
+    git annex init test
+    git annex config --set annex.largefiles 'not mimetype=text/*'
+
+    # git commit --allow-empty -m init
+    echo hello > README.rst
+    git add README.rst
+    git commit -m init
+
+    mkdir new old
+    for f in old/1 old/2 new/3 new/4; do (head -c100 /dev/zero;echo $f) > $f; done
+    git annex add
+    git commit -m annex
+
+    git annex initremote android type=adb androiddirectory=/storage/0000-0000/ttt encryption=none exporttree=yes importtree=yes
+    git config --local remote.android.annex-sync false
+
+    echo
+    git config remote.android.annex-tracking-branch main
+    for _ in {0..6}; do git annex sync --no-commit --content android; done
+
+    echo
+    find . -name .git -prune -o -xtype f -exec stat -c '%s %Y %i %n' {} +
+    echo
+    adb shell \"find /storage/0000-0000/ttt -type f -exec stat -c '%s %Y %i %n' {} +\"
+
+    echo
+    git -c color.ui=no --no-pager log --topo-order --oneline --patch --decorate --graph main
+
+"""]]

update
diff --git a/doc/thanks/list b/doc/thanks/list
index 9a57d462c5..cd7ed9f3ef 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -122,3 +122,4 @@ Nick,
 Nicholas Golder-Manning, 
 Troels Henriksen, 
 Max Thoursie, 
+Thomas Haller, 

adding a noob question
diff --git a/doc/forum/Failing_to_get_files_from_rclone_special_remote.mdwn b/doc/forum/Failing_to_get_files_from_rclone_special_remote.mdwn
new file mode 100644
index 0000000000..ee0bd72553
--- /dev/null
+++ b/doc/forum/Failing_to_get_files_from_rclone_special_remote.mdwn
@@ -0,0 +1,104 @@
+Hi! Thanks for your time, and please let me know if this is an rclone special remote-specific question that I should be asking elsewhere.
+
+I very likely may be _fundamentally_ misconfiguring something since I'm a git-annex beginner (although a git veteran)
+
+I'm trying to make a system to sync Ableton projects and their corresponding samples between my band's computers.
+
+So far I have Computer A with a repo and git annex with the local file contents.
+
+From here, I've pushed the repository itself to github and the contents to an rclone special remote hosted on pCloud.
+
+```
+% git annex list
+here
+|github
+||web
+|||bittorrent
+||||pcloud_test_annex
+|||||
+X___X Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif
+X___X Samples/Recorded/1-Audio 0004 [2021-12-08 092222].aif
+X___X Samples/Recorded/1-Audio 0005 [2021-12-08 092402].aif
+...
+
+
+% git remote -v
+github	git@github.com:branch-pcloud.git (fetch)
+github	git@github.com:branch-pcloud.git (push)
+pcloud_test_annex
+```
+
+On computer B, I tried to clone the github repo and pull the pCloud data down, but I can't manage to get it to work.
+
+```
+$ git annex get 'Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif'
+get Samples/Recorded/1-Audio 0001 [2021-12-08 093649].aif (not available)
+  Maybe enable some of these special remotes (git annex initremote ...):
+        e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d -- pcloud_test_annex
+  Maybe add some of these git remotes (git remote add ...):
+        bab5704a-562e-43eb-8bbb-71f181f043e8 -- bl@compy_A:~/test_project
+
+  (Note that these git remotes have annex-ignore set: origin)
+failed
+get: 1 failed
+```
+
+Okay, well github (origin) won't have it, so I'll try to enable the special remote:
+
+```
+$ git annex enableremote pcloud_test_annex
+enableremote pcloud_test_annex ok
+(recording state in git...)
+```
+
+But it doesn't help. I get the same error afterwards.
+
+Here's something that's probably bad: on computer B, there's a whole lot of git annex remotes:
+
+```
+$ git annex info
+trusted repositories: 0
+semitrusted repositories: 8
+        00000000-0000-0000-0000-000000000001 -- web
+        00000000-0000-0000-0000-000000000002 -- bittorrent
+        2e771318-83eb-4beb-85f4-567896b0e835 -- [pcloud_test_annex]
+        32c6a55a-3879-4a67-beb9-d7e4d1624cc4 -- bl@compy_B:~/Music\mine\test_project [here]
+        69acaaf2-e58b-463d-a578-e2418df5c6b0 -- pcloud_test_annex
+        bab5704a-562e-43eb-8bbb-71f181f043e8 -- bl@compy_A:~/test_project
+        e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d -- pcloud_test_annex
+        ebca6ee8-1baa-4a47-ae85-0a5b65db73f6 -- pcloud_test_annex
+untrusted repositories: 0
+transfers in progress: none
+available local disk space: 152.39 gigabytes (+1 megabyte reserved)
+local annex keys: 0
+local annex size: 0 bytes
+annexed files in working tree: 18
+size of annexed files in working tree: 688.91 megabytes
+bloom filter size: 32 mebibytes (0% full)
+backend usage:
+        SHA256E: 18
+```
+
+After some investigation, one of these is actually the right one (I think)!
+
+```
+$ git annex info e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d
+uuid: e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d
+description: pcloud_test_annex
+trust: semitrusted
+remote annex keys: 18
+remote annex size: 688.91 megabytes
+```
+
+But I can't figure out how to configure computer B's repository to access it:
+
+```
+$ git annex info e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d
+uuid: e7e3e2b7-40e7-4b10-83f6-bb5425f33d6d
+description: pcloud_test_annex
+trust: semitrusted
+remote annex keys: 18
+remote annex size: 688.91 megabytes
+```
+
+How can I get other computers to pull the contents from pCloud (rclone)?

fix close link
diff --git a/doc/forum/export_tracking_not_enabled_when_branch_is_set.mdwn b/doc/forum/export_tracking_not_enabled_when_branch_is_set.mdwn
index 5180a39707..2618dc3926 100644
--- a/doc/forum/export_tracking_not_enabled_when_branch_is_set.mdwn
+++ b/doc/forum/export_tracking_not_enabled_when_branch_is_set.mdwn
@@ -31,4 +31,4 @@ this seems to be a bug, but i wanted to make sure i wasn't doing something wrong
 
 [[!tag forumbug]]
 
-> [[done]] --[[Joey]]
+> [[bugs/done]] --[[Joey]]

comment
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_3_a9b3e07f180ab97f5025d7ce9ca58c05._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_3_a9b3e07f180ab97f5025d7ce9ca58c05._comment
new file mode 100644
index 0000000000..2bb507eabb
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_3_a9b3e07f180ab97f5025d7ce9ca58c05._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2022-01-04T15:33:25Z"
+ content="""
+Please show me how to get more than one "import from foo"
+commit that is empty, starting from a new remote.
+
+I have only managed to get one, maximum, as I said.
+
+I do not thik your CID analysis is correct. The CID is
+calculated on the android side and has nothing to do with
+the timestamps of anything in the git repo.
+
+Now, if Android is varying the mtime it reports for files, so resulting in
+a changed CID, it may be that would cause an import of content it had already 
+imported before, and maybe that leads to the unncessary commit.
+"""]]

close
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn
index a30eeb33ad..7059e00c06 100644
--- a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn
@@ -42,3 +42,5 @@ Digging into it further with dfly people over IRC, it's been discovered:
 Has anyone tried git-annex with ghc 8.10.7 on a different OS?
 
 I tried to figure out why `.git/annex/index` was not being created, but I had some trouble figuring out which line of code is supposed to create it. If someone could point me there I could do some further debugging to see why it isn't created.
+
+> [[done]] --[[Joey]]

comment
diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_3_7346adcffb3d9056dc880dcf1e2ab03f._comment b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_3_7346adcffb3d9056dc880dcf1e2ab03f._comment
new file mode 100644
index 0000000000..c1f930d9d1
--- /dev/null
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_3_7346adcffb3d9056dc880dcf1e2ab03f._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2022-01-04T15:24:33Z"
+ content="""
+The only special remotes that could probably even support symlinks
+in theory are adb and directory, and those only sometimes.
+
+Supporting them would complicate the special remote API, and I feel
+unncessarily so.
+"""]]

comment
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_5_c667e4d4899925a5484c94256044f6ee._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_5_c667e4d4899925a5484c94256044f6ee._comment
new file mode 100644
index 0000000000..260d2f191a
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_5_c667e4d4899925a5484c94256044f6ee._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2022-01-04T15:22:47Z"
+ content="""
+Ah, I see. :)
+
+I am going to leave this bug open, since the corrupted tmp file problem I
+found is a real problem, though not actually your problem.
+"""]]

Added a comment
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_2_5000af944e3a7917a97b65fb7d1c5c3e._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_2_5000af944e3a7917a97b65fb7d1c5c3e._comment
new file mode 100644
index 0000000000..45e1f3c187
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_2_5000af944e3a7917a97b65fb7d1c5c3e._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="amerlyq+annex@12121d70a602f094228689a0a24d348d478a8af1"
+ nickname="amerlyq+annex"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 2"
+ date="2022-01-04T13:41:03Z"
+ content="""
+> I guess you mean commits like \"import from foo\", or perhaps \"remote tracking branch\". Not adb-specific at all.
+
+Yes. 
+
+* First \"import from foo\" commit is OK, but second was empty -- and false-positive, because there were no changes on android.
+* Consequently, first \"remote tracking branch\" is OK to merge histories, but all next are empty -- and useless, because there is no changes in \"android/main\" branch.
+---
+> The \"remote tracking branch\" commits are merge commits, so cannot be avoided, even though they don't make any apparent changes.
+
+Can't we simply... don't do them? Skip? Revert? Fast-forward?
+
+I don't mind this history hell inside git-annex branch, but cluttering \"main\" branch... is something else.
+After first merge of unrelated histories -- we don't really need another merge commit unless new changes where introduced during import, do we?
+---
+> Of course, you can rebase out any of these commits if you want to.
+
+Theoretically yes, practically no. If I run \"git annex sync\" -- this history will be propagated to all my connected in that moment devices -- PC, laptop, NAS, remote VPS, and maybe HDD.
+Rebasing and cleaning up them after that -- impractical.
+
+---
+> It mostly does manage to avoid making the \"import from\" commits when there is no difference from the previous commit.
+> Subsequent imports, when there are no changes to the tree, do not make a new commit.
+
+It seems this does not work as described.
+
+You depend on CID (size, modificiation time, and inode) when doing comparisons.
+
+1) Most likely this issue is related to https://git-annex.branchable.com/bugs/adb_pull_does_not_preserve_timestamp/ ,
+that results in a different mtime on remote and on PC when you \"import\" new files.
+
+Then each time you do \"import\" -- git-annex detect changes by mtime and creates commit, but it's empty because content hadn't actually changed.
+
+2) Alternative *heisenbug* is the time skew between what android \"sdcard\" virtual filesystem reports and what it actually transfers. Probability is too low, though :)
+
+---
+P.S. I'm not sure if it's a separate bug or related to \"mtime\" too (scenario (1) or even (2)), but:
+
+* if I create a new repo and do \"export\" multiple times in a row into empty folder -- it works OK.
+* But on the next day when I'm trying to \"export\" again -- it reports error \"unsafe to overwrite file //failed\"
+
+But I had no changes in either repo or in android folder. And it worked multiple times in a row the day before!
+"""]]

Added a comment
diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_2_f5db61aa7efb7a916d0ffebd563cc2c7._comment b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_2_f5db61aa7efb7a916d0ffebd563cc2c7._comment
new file mode 100644
index 0000000000..d042b41f10
--- /dev/null
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_2_f5db61aa7efb7a916d0ffebd563cc2c7._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="amerlyq+annex@12121d70a602f094228689a0a24d348d478a8af1"
+ nickname="amerlyq+annex"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 2"
+ date="2022-01-04T13:10:13Z"
+ content="""
+Probably skipping symlinks for both import/export is for the best.
+
+My personal expectations (pipe dream?) are to export symlinks as real files -- but only compare during import and warn if different.
+
+Because if I see file/symlink in some repo -- it's placed there for a reason, and after sync I expect to find all the expected files in their folders (even if through multiple copies). 
+Silently ignoring of symlinks forces me to remember to check that whatever I want to sync does not contain them.
+Otherwise I may be in a very delicate situation, if files needed for work were not synced to phone and I have no access to PC at that moment ;)
+
+On the other hand, I remember literally months I spent debugging and configuring samba, and reading their history of symlinks implementations.
+It contains so much frustrating corner cases, that banning symlinks completely isn't a bad idea, actually.
+"""]]

Added a comment
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_2_2d49693c5f8d826f9a4b1cf2f8dca2c5._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_2_2d49693c5f8d826f9a4b1cf2f8dca2c5._comment
new file mode 100644
index 0000000000..0513d95fca
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_2_2d49693c5f8d826f9a4b1cf2f8dca2c5._comment
@@ -0,0 +1,39 @@
+[[!comment format=mdwn
+ username="amerlyq+annex@12121d70a602f094228689a0a24d348d478a8af1"
+ nickname="amerlyq+annex"
+ avatar="http://cdn.libravatar.org/avatar/3d63c9f436b45570d45bd003e468cbd3"
+ subject="comment 2"
+ date="2022-01-04T12:55:14Z"
+ content="""
+> Git does not preserve timestamps. So this is kind of out of scope.
+
+Yes, git does not, but PC filesystem *does*.
+And when you annex files -- git-annex copies file own mtime to its symlink.
+Therefore you can *sort* and *seek* files based on the date of your first *acquaintance* (downloading) instead of the date of batch-sync.
+
+> pulling in random permissions for annex object files could lead to arbitrary breakage (imagine a suid executable for example). 
+> So it would need to clear the permissions.
+
+I don't see a reason to sync any native executables with Android phone for usual mediafile-centered workflow with usual /storage location.
+Moreover, suid is of no concern for years due to virtual filesystem used in Android, which simply does not support suid or exec anyway:
+
+    /data/media on /storage/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=20MB)
+    /mnt/media_rw/0000-0000 on /storage/0000-0000 type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,mask=6)
+
+> The complexity does not seem worth trying to support a thing that is generally out of scope.
+
+Consider next workflow:
+
+* android folder /Downloads (or even whole /emulated) is actively filled by many apps -- and used as \"source\" for git-annex
+* once a week I use \"adb import\" to move everything present in folder/android to PC and sort it on PC -- using comfortable tools 
+* usually things downloaded on the same day are somehow related to each other -- therefore sorting them much easier if presorted by date first, and then by filetype
+* some files are moved to \"archive of misc references/2022-01-XX\" -- because they are not important enough to spend time thinking about them further, but good to backtrack when needed
+* other important files are usually moved to respective contexts -- work/home/projects/leisure/etc. -- by keeping them inside their \"2022-01-XX\" folder to easily refer to other tasks I did on that day in those contexts.
+* after that some contexts (which have separate git-annex) are \"exported\" back to android to have immediate access -- e.g. filtered repos for \"leisure\", \"books\" -- which contain only most important and most recent entries.
+
+If adb does not preserve timestamps -- you lose \"date\" information, and core of the workflow crumbles to dust.
+And you can't use (name, size, mtime) as kind of UID for sync too.
+
+To support this workflow I used rsync+adbfs, but it's a pain due to  how you can't simply and efficiently rsync from phone into annex symlinks and back and be done with it.
+Moreover it's a separate different step, which could be eliminated if \"git annex sync\" properly worked with adb remote.
+"""]]

Added a comment: well, yes
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_4_1e286c1506fc542df85c6158a8ec3d1c._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_4_1e286c1506fc542df85c6158a8ec3d1c._comment
new file mode 100644
index 0000000000..3d948e5205
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_4_1e286c1506fc542df85c6158a8ec3d1c._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="tim@5431dd39464df207b7d46d3cf1bc74c82123ac68"
+ nickname="tim"
+ avatar="http://cdn.libravatar.org/avatar/967c2815f001d24b5f9f33300c03ba6f"
+ subject="well, yes"
+ date="2022-01-04T11:28:33Z"
+ content="""
+Well, yes, it does, if you start discarding disks after you've done a git annex pull on a temp usb repo and then again to the new server... Hence, my own stupidity.
+"""]]

Added a comment
diff --git a/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_1_5c92b3e7c2fc0e8a8e943690c1806322._comment b/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_1_5c92b3e7c2fc0e8a8e943690c1806322._comment
new file mode 100644
index 0000000000..95318f69c3
--- /dev/null
+++ b/doc/todo/speed_up_drop_on___34__heavy__34___repo/comment_1_5c92b3e7c2fc0e8a8e943690c1806322._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 1"
+ date="2022-01-03T20:03:38Z"
+ content="""
+FWIW: Re-running with -J3 right away made git-annex 200% CPU busy while --debug was showing
+
+```
+[2022-01-03 14:59:56.323517] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID \"00000000-0000-0000-0000-000000000001\"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = \"c7e4ea78ccd5486983b9185b22cc67fa1f96df84630ae23e6bb2b63f3726f0d9.h5\", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 147972754129, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = \"SHA256E-s147972754129--c7e4ea78ccd5486983b9185b22cc67fa1f96df84630ae23e6bb2b63f3726f0d9.h5\"}))))
+[2022-01-03 14:59:56.347374] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID \"00000000-0000-0000-0000-000000000001\"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = \"c658b317bf9bf35a8dc69eaff18899c7db5336d5b230ba8fbad88bd702344f6c.h5\", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 79593126729, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = \"SHA256E-s79593126729--c658b317bf9bf35a8dc69eaff18899c7db5336d5b230ba8fbad88bd702344f6c.h5\"}))))
+```
+
+-J2 makes it 100% busy .  -J1 - only 2% CPU and in D state.
+"""]]

initial report on slow drop
diff --git a/doc/todo/speed_up_drop_on___34__heavy__34___repo.mdwn b/doc/todo/speed_up_drop_on___34__heavy__34___repo.mdwn
new file mode 100644
index 0000000000..0944b862ef
--- /dev/null
+++ b/doc/todo/speed_up_drop_on___34__heavy__34___repo.mdwn
@@ -0,0 +1,38 @@
+On [this monster repo](https://github.com/dandisets/000108.git) with only 2290 files in worktree covering 168.57TB, it spends many seconds for each key drop
+
+Here running serially with `--debug` I see
+
+```
+[2022-01-03 13:07:53.229582] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID "00000000-0000-0000-0000-000000000001"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = "32376dfb94d171b1b51c0829d4071fee14be0bbde6f270d2c102312851fbdfb3.h5", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 45210151184, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = "SHA256E-s45210151184--32376dfb94d171b1b51c0829d4071fee14be0bbde6f270d2c102312851fbdfb3.h5"}))))
+[2022-01-03 13:08:02.026986] (Annex.Branch) read 381/7a1/SHA256E-s45210151184--32376dfb94d171b1b51c0829d4071fee14be0bbde6f270d2c102312851fbdfb3.h5.log
+[2022-01-03 13:08:02.058449] (Annex.Branch) set 381/7a1/SHA256E-s45210151184--32376dfb94d171b1b51c0829d4071fee14be0bbde6f270d2c102312851fbdfb3.h5.log
+```
+
+so took 8 seconds to come out from "Dropping" to the next one... and I think time it takes grows with every next file, here is a sample for some one after a few:
+
+```
+[2022-01-03 13:10:12.39011] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID "00000000-0000-0000-0000-000000000001"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = "ee77e5c090543ab2128961cec827d1d07aa955ad60aa6843331ffdfc5fa2eeaf.h5", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 97264091411, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = "SHA256E-s97264091411--ee77e5c090543ab2128961cec827d1d07aa955ad60aa6843331ffdfc5fa2eeaf.h5"}))))
+[2022-01-03 13:10:37.962407] (Annex.Branch) read 47a/7fa/SHA256E-s97264091411--ee77e5c090543ab2128961cec827d1d07aa955ad60aa6843331ffdfc5fa2eeaf.h5.log
+[2022-01-03 13:10:37.982222] (Annex.Branch) set 47a/7fa/SHA256E-s97264091411--ee77e5c090543ab2128961cec827d1d07aa955ad60aa6843331ffdfc5fa2eeaf.h5.log
+
+```
+which is 25 seconds... then 40 ... although some go quickly
+
+```
+[2022-01-03 13:12:02.61183] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID "00000000-0000-0000-0000-000000000001"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = "3a7397803d7af7ffaf323bd9b34f06e2e2b743c0687d911cd04352e6851e193f.h5", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 18094051444, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = "SHA256E-s18094051444--3a7397803d7af7ffaf323bd9b34f06e2e2b743c0687d911cd04352e6851e193f.h5"}))))
+[2022-01-03 13:12:04.213675] (Annex.Branch) read 4a1/f2c/SHA256E-s18094051444--3a7397803d7af7ffaf323bd9b34f06e2e2b743c0687d911cd04352e6851e193f.h5.log
+[2022-01-03 13:12:04.231074] (Annex.Branch) set 4a1/f2c/SHA256E-s18094051444--3a7397803d7af7ffaf323bd9b34f06e2e2b743c0687d911cd04352e6851e193f.h5.log
+```
+
+well -- filesystem (BTRFS) was busy doing its maintenance meanwhile (had to free up space), but still -- quite long wait times.  I interrupted that `drop`, waited for filesystem to settle down, repeated for some files -- still quite a long time, e.g.
+
+```
+[2022-01-03 14:58:09.411661] (Command.Drop) Dropping from here proof: Just (SafeDropProof (NumCopies 1) (MinCopies 1) [RecentlyVerifiedCopy UUID "00000000-0000-0000-0000-000000000001"] (Just (ContentRemovalLock (MkKey {keyData = Key {keyName = "3b0d2deef0223146e6edec86b7b96521349def252e9e26c25b46fc18067350ad.h5", keyVariety = SHA2Key (HashSize 256) (HasExt True), keySize = Just 25859843418, keyMtime = Nothing, keyChunkSize = Nothing, keyChunkNum = Nothing}, keySerialization = "SHA256E-s25859843418--3b0d2deef0223146e6edec86b7b96521349def252e9e26c25b46fc18067350ad.h5"}))))
+[2022-01-03 14:58:18.857103] (Annex.Branch) read 4e4/a95/SHA256E-s25859843418--3b0d2deef0223146e6edec86b7b96521349def252e9e26c25b46fc18067350ad.h5.log
+```
+
+NB not sure if related, prior long run with `-J3` was also getting CPU busy which made little sense to me for `drop`.  I will monitor CPU consumption in the next attempt.
+
+git annex 8.20211028-g1c76278
+
+[here is a full log](http://www.onerussian.com/tmp/long-drop-20220103-1.log).

Added a comment
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_5_27a017a942f805e9b0216e77d4890c86._comment b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_5_27a017a942f805e9b0216e77d4890c86._comment
new file mode 100644
index 0000000000..3b9278918a
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_5_27a017a942f805e9b0216e77d4890c86._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="falsifian"
+ avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae"
+ subject="comment 5"
+ date="2022-01-03T19:06:11Z"
+ content="""
+This turned out to a bug in DragonFlyBSD. It is [now fixed](https://gitweb.dragonflybsd.org/dragonfly.git/commit/8202da125af7f265528f7844bdc98d544a320bd1). Thanks for your help finding the problem!
+"""]]

avoid exporting non-annexed symlinks
So that importing does not replace them with plain files.
This works similarly to how the previous handling of submodules and
matchers did, except that annexed symlinks still get exported as plain
files of course, it's only non-annexed symlinks that it does not make sense
to export.
When symlinks have previously been exported, updating the export will
unexport them after upgrading to this commit.
Sponsored-by: Kevin Mueller on Patreon
diff --git a/CHANGELOG b/CHANGELOG
index a1c1032eef..235b49aacc 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,11 @@
+git-annex (8.20211232) UNRELEASED; urgency=medium
+
+  * export: When a non-annexed symlink is in the tree to be exported, skip it.
+  * import: When the previously exported tree contained a non-annexed symlink,
+    preserve it in the imported tree so it does not get deleted.
+
+ -- Joey Hess <id@joeyh.name>  Mon, 03 Jan 2022 14:01:14 -0400
+
 git-annex (8.20211231) upstream; urgency=medium
 
   * Improved support for using git-annex in a read-only repository,
diff --git a/Command/Export.hs b/Command/Export.hs
index bb848f42f1..6efbaf1002 100644
--- a/Command/Export.hs
+++ b/Command/Export.hs
@@ -479,7 +479,8 @@ removeEmptyDirectories r db loc ks
 newtype ExportFiltered t = ExportFiltered t
 
 -- | Filters the tree to annexed files that are preferred content of the
--- remote, and also including non-annexed files, but not submodules.
+-- remote, and also including non-annexed files, but not submodules or
+-- non-annexed symlinks.
 --
 -- A log is written with tree items that were filtered out, so they can
 -- be added back in when importing from the remote.
@@ -505,17 +506,28 @@ filterExport r tree = logExportExcluded (uuid r) $ \logwriter -> do
 		case toTreeItemType mode of
 			-- Don't export submodule entries.
 			Just TreeSubmodule -> excluded
-			_ -> case mmatcher of
-				Nothing -> return (Just ti)
-				Just matcher -> catKey sha >>= \case
-					Just k -> checkmatcher matcher k
-					-- Always export non-annexed files.
-					Nothing -> return (Just ti)
+			Just TreeSymlink -> checkkey True
+			_ -> checkkey False
 	  where
 		excluded = do
 			() <- liftIO $ logwriter ti
 			return Nothing
 
+		checkkey issymlink =
+			case mmatcher of
+				Nothing
+					| issymlink -> catKey sha >>= \case
+						Just _ -> return (Just ti)
+						Nothing
+							| issymlink -> excluded
+							| otherwise -> return (Just ti)
+					| otherwise -> return (Just ti)
+				Just matcher -> catKey sha >>= \case
+					Just k -> checkmatcher matcher k
+					Nothing
+						| issymlink -> excluded
+						| otherwise -> return (Just ti)
+
 		checkmatcher matcher k = do
 			let mi = MatchingInfo $ ProvidedInfo
 				{ providedFilePath = Just $
diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
index e6bf632ddc..5e663589fb 100644
--- a/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
@@ -26,3 +26,5 @@ Don't do anything if they are the same.
 Replace symlink by actual file if they differ.
 
 [[!meta title="export then import replaces non-annex symlinks by plain text files"]]
+
+> [[fixed|done]] by skipping syminks. --[[Joey]]

comment
diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index 060366a9fb..580a640fc5 100644
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -202,6 +202,7 @@ retrieve' serial src dest =
 		showOutput -- make way for adb pull output
 		liftIO $ boolSystem "adb" $ mkAdbCommand serial
 			[ Param "pull"
+			, Param "-a"
 			, File $ fromAndroidPath src
 			, File dest
 			]
diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_1_b695826d1f948249b496c055b4d745df._comment b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_1_b695826d1f948249b496c055b4d745df._comment
new file mode 100644
index 0000000000..49c8924aab
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp/comment_1_b695826d1f948249b496c055b4d745df._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-03T17:48:38Z"
+ content="""
+Git does not preserve timestamps. So this is kind of out of scope.
+
+While git-annex will sometimes preserve timestamps when it's easy to do so,
+there are no guarantees. Most special remotes do not preserve any
+timestamps in either direction.
+
+It might be that `adb pull -a` would fix this. But that also preserves
+the permissions, and pulling in random permissions for annex object files
+could lead to arbitrary breakage (imagine a suid executable for example).
+So it would need to clear the permissions. The complexity does not seem
+worth trying to support a thing that is generally out of scope.
+"""]]

comment and retitle
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn b/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn
index de0d91281c..5beee2397e 100644
--- a/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn
@@ -1,5 +1,7 @@
 ### Please describe the problem.
 
+[[!meta title="import tree creates empty commits"]]
+
 Adb special remote on both "import" and "sync" commands creates empty commits in "main" branch,
 even if files hadn't changed  in either remote or local repo.
 
diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_1_522c68d019e5db592e140db9501435ba._comment b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_1_522c68d019e5db592e140db9501435ba._comment
new file mode 100644
index 0000000000..6ab85944bb
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync/comment_1_522c68d019e5db592e140db9501435ba._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-03T17:32:34Z"
+ content="""
+I guess you mean commits like "import from foo",
+or perhaps "remote tracking branch". Not adb-specific at all.
+
+The "remote tracking branch" commits are merge commits, so cannot be
+avoided, even though they don't make any apparent changes.
+
+It mostly does manage to avoid making the "import from" commits
+when there is no difference from the previous commit.
+The only case I know of where it does not is in the initial import,
+if the branch was exported first, and then imported. I see an empty 
+"import from" then. Subsequent imports, when there are no changes to the
+tree, do not make a new commit.
+
+Of course, you can rebase out any of these commits if you want to.
+"""]]

comment and retitle
diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
index 3a45b95eb0..e6bf632ddc 100644
--- a/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
@@ -24,3 +24,5 @@ Cavet: user won't be able to *intentionally* replace symlinks by plain files on
 But on "import" compare its size against followed symlink file size, and then its content.
 Don't do anything if they are the same. 
 Replace symlink by actual file if they differ.
+
+[[!meta title="export then import replaces non-annex symlinks by plain text files"]]
diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_1_bcc12b77c5069769ab04e448b23b3d85._comment b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_1_bcc12b77c5069769ab04e448b23b3d85._comment
new file mode 100644
index 0000000000..cd422c2272
--- /dev/null
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files/comment_1_bcc12b77c5069769ab04e448b23b3d85._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2022-01-03T17:27:03Z"
+ content="""
+This is not adb specific. No special remotes support preserving symlinks
+(except for git-annex object links). Reproduced using a directory special
+remote.
+
+Probably export/import should somehow skip symlinks.
+"""]]

comment
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_3_4c359c12d5c61351484dfcb2f5547222._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_3_4c359c12d5c61351484dfcb2f5547222._comment
new file mode 100644
index 0000000000..3ac6afe830
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_3_4c359c12d5c61351484dfcb2f5547222._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2022-01-03T17:25:03Z"
+ content="""
+Err, unless you have `annex.thin` set, once you `git-annex add` a file,
+it should not be possible for it be overwritten in a way that loses the
+annexed copy of it.
+"""]]

comment
diff --git a/doc/bugs/get__58___no_passphrase_ask_for_the_2nd_ssh_invocation/comment_11_f818dadfa541520ac7ad2409b9174776._comment b/doc/bugs/get__58___no_passphrase_ask_for_the_2nd_ssh_invocation/comment_11_f818dadfa541520ac7ad2409b9174776._comment
new file mode 100644
index 0000000000..0dc8ba08e8
--- /dev/null
+++ b/doc/bugs/get__58___no_passphrase_ask_for_the_2nd_ssh_invocation/comment_11_f818dadfa541520ac7ad2409b9174776._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2022-01-03T16:22:05Z"
+ content="""
+Ok, you've ruled out my theory about -T, and stdin is being connected up
+through ssh correctly it looks like.
+
+One other thing to try would be to enable ssh verbose debugging,
+by running: `git config annex.ssh-options "-v -v -v"`  
+And perhaps that will give a clue.
+
+But I think the signs are pointing to you needing to change something
+about your ssh installation on Windows. Perhaps a different version of ssh
+for Windows would support the GUI password prompt. Normally, ssh comes
+from Git for Windows, which bundles a ssh, and I think that one does
+support the GUI prompt. Do you have some other ssh build installed?
+"""]]

Added a comment: Found the issue
diff --git a/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_2_53970994816f1de9816e3496fdf0ef71._comment b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_2_53970994816f1de9816e3496fdf0ef71._comment
new file mode 100644
index 0000000000..2fdb5f6c4c
--- /dev/null
+++ b/doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_2_53970994816f1de9816e3496fdf0ef71._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="tim@5431dd39464df207b7d46d3cf1bc74c82123ac68"
+ nickname="tim"
+ avatar="http://cdn.libravatar.org/avatar/967c2815f001d24b5f9f33300c03ba6f"
+ subject="Found the issue"
+ date="2022-01-03T08:25:05Z"
+ content="""
+Hi Joey,
+
+Thank you for having a look at this, but we can put this one under \"my own stupidity\". The files that give these errors are apparently only status files which get rewritten every time you play a recording, so they continuously get rewritten, get \"git annex add\"ed with a cron job, but don't always get pushed before they get overwritten, so yeah, it's not unexpected that some of these may get lost at some point.
+Still no real data lost though, but errors like these worry me enough to file a bug report.
+
+regards,
+Tim
+"""]]

diff --git a/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
new file mode 100644
index 0000000000..3a45b95eb0
--- /dev/null
+++ b/doc/bugs/adb_replaces_symlinks_by_plain_text_files.mdwn
@@ -0,0 +1,26 @@
+### Please describe the problem.
+Using adb special remote import/export corrupts all symlinks in main repo (by replacing them by plain text files).
+
+### What steps will reproduce the problem?
+Primary annex repo contains an absolute symlink to another annex repo (or whatever).
+Exporting it through adb replaces it by plain text file (as android FS abstraction does not support symlinks).
+Importing from adb copies that plain text file and deletes symlink already present in the repo,
+treating it as a new file modified by user.
+
+### What version of git-annex are you using? On what operating system?
+git-annex version: 8.20211118-g23ee48898
+
+
+### Please provide any additional information below.
+
+Possible solutions:
+
+1) if PC repo contains symlink -- continue to export it as a plain text file (less changes).
+But on "import" compare its size with symlink size, and then compare content.
+Don't do anything if they are the same. 
+Cavet: user won't be able to *intentionally* replace symlinks by plain files on remote. Too esoteric usecase to consider.
+
+2) resolve symlink on PC and export actual file to android (probably most preferred workflow?).
+But on "import" compare its size against followed symlink file size, and then its content.
+Don't do anything if they are the same. 
+Replace symlink by actual file if they differ.

diff --git a/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn b/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn
new file mode 100644
index 0000000000..de0d91281c
--- /dev/null
+++ b/doc/bugs/adb_creates_empty_commits_on_import__47__sync.mdwn
@@ -0,0 +1,39 @@
+### Please describe the problem.
+
+Adb special remote on both "import" and "sync" commands creates empty commits in "main" branch,
+even if files hadn't changed  in either remote or local repo.
+
+This clutters history of "main" branch (which is mostly manually curated),
+and hides commits actually introducing added/deleted files amongst the clutter.
+
+Moreover, this behavior differs from expectations: "sync" with proper remote never created empty commits in "main" branch.
+
+## What steps will reproduce the problem?
+
+Setup adb import/export, like here: 
+[[https://git-annex.branchable.com/forum/Mixed_content_repos_with_import_and_export/#comment-764ac971faf756140055333649ffb94c]].
+
+Repetition of "git annex sync --content" introduces new empty commits without proper reason.
+
+### What version of git-annex are you using? On what operating system?
+git-annex version: 8.20211118-g23ee48898
+
+
+### Please provide any additional information below.
+
+Example of "main" branch history cluttered by empty commits:
+
+[[!format txt """
+*   2022-01-01 bb23200 remote tracking branch                                       (android/main) [annex]
+|\
+* | 2022-01-01 42b8111 remote tracking branch                                       (HEAD -> main, synced/main) [annex]
+|\|
+| * 2022-01-01 223f5f7 import from android                                          [annex]
+* | 2022-01-01 f5b5f0a remote tracking branch                                       [annex]
+|\|
+* | 2022-01-01 2a6177e remote tracking branch                                       [annex]
+|\|
+| * 2022-01-01 7ef4649 import from android                                          [annex]
+"""]]
+
+

diff --git a/doc/bugs/adb_pull_does_not_preserve_timestamp.mdwn b/doc/bugs/adb_pull_does_not_preserve_timestamp.mdwn
new file mode 100644
index 0000000000..4b7e821479
--- /dev/null
+++ b/doc/bugs/adb_pull_does_not_preserve_timestamp.mdwn
@@ -0,0 +1,44 @@
+### Please describe the problem.
+Adb special remote does not preserve remote timestamp on "import".
+Therefore files downloaded several days ago lose their real mtime when transferred to PC,
+and from that moment onward files on android will have mtime different from stored in annex/symlinks.
+
+But all files already present in PC repo -- during "export" correctly propagate their mtime to android.
+
+### What steps will reproduce the problem?
+
+[[!format sh """
+git annex initremote android type=adb androiddirectory=/storage/0000-0000/books encryption=none exporttree=yes importtree=yes
+git config remote.android.annex-sync false
+git config remote.android.annex-tracking-branch main
+git annex export main --to android
+# ADD: download some "file" through android into "books"
+git annex sync --content android
+stat -c%y "file"
+Modify: 2022-01-01 20:51:11.487603199 +0200
+"""]]
+
+### What version of git-annex are you using? On what operating system?
+git-annex version: 8.20211118-g23ee48898
+ArchLinux 5.15.12-arch1-1
+
+### Please provide any additional information below.
+Most likely can be fixed by:
+
+[[!format diff """
+--- i/Remote/Adb.hs
++++ w/Remote/Adb.hs
+@@ -202,6 +202,7 @@ retrieve' serial src dest =
+                showOutput -- make way for adb pull output
+                liftIO $ boolSystem "adb" $ mkAdbCommand serial
+                        [ Param "pull"
++                      , Param "-a"
+                        , File $ fromAndroidPath src
+                        , File dest
+                        ]
+"""]]
+
+### Have you had any luck using git-annex before?
+
+Nice work fixing earlier notorious bug, it made the ADB special remote usable for me at least in PC -> android direction! :)  
+[[https://git-annex.branchable.com/forum/Mixed_content_repos_with_import_and_export/#comment-764ac971faf756140055333649ffb94c]]

add news item for git-annex 8.20211231
diff --git a/doc/news/version_8.20210903.mdwn b/doc/news/version_8.20210903.mdwn
deleted file mode 100644
index 8faa2ef22b..0000000000
--- a/doc/news/version_8.20210903.mdwn
+++ /dev/null
@@ -1,32 +0,0 @@
-git-annex 8.20210903 released with [[!toggle text="these changes"]]
-[[!toggleable text="""  * Deal with clock skew, both forwards and backwards, when logging
-    information to the git-annex branch.
-  * GIT\_ANNEX\_VECTOR\_CLOCK can now be set to a fixed value (eg 1)
-    rather than needing to be advanced each time a new change is made.
-  * Misuse of GIT\_ANNEX\_VECTOR\_CLOCK will no longer confuse git-annex.
-  * add: When adding a dotfile, avoid treating its name as an extension.
-  * rsync special remote: Stop displaying rsync progress, and use
-    git-annex's own progress display.
-  * Many special remotes now checksum content while it is being retrieved,
-    instead of in a separate pass at the end. This is supported for all
-    special remotes on Linux (except for bittorrent), and for many
-    on other OS's (except for adb, external, gcrypt, hook, and rsync).
-  * unused: Skip the refs/annex/last-index ref that git-annex recently
-    started creating.
-  * Fix test suite failure on Windows.
-  * New --batch-keys option added to these commands:
-    get, drop, move, copy, whereis
-  * Added annex.youtube-dl-command config. This can be used to run some
-    forks of youtube-dl.
-  * Run cp -a with --no-preserve=xattr, to avoid problems with copied
-    xattrs, including them breaking permissions setting on some NFS
-    servers.
-  * add, import: Detect when xattrs or perhaps ACLs prevent removing
-    write permissions from an annexed file, and fail with an informative
-    message.
-  * Fix support for readonly git remotes.
-    (Reversion in version 8.20210621)
-  * When downloading urls fail, explain which urls failed for which
-    reasons.
-  * web: Avoid displaying a warning when downloading one url failed
-    but another url later succeeded."""]]
\ No newline at end of file
diff --git a/doc/news/version_8.20211231.mdwn b/doc/news/version_8.20211231.mdwn
new file mode 100644
index 0000000000..8699568c76
--- /dev/null
+++ b/doc/news/version_8.20211231.mdwn
@@ -0,0 +1,22 @@
+git-annex 8.20211231 released with [[!toggle text="these changes"]]
+[[!toggleable text="""  * Improved support for using git-annex in a read-only repository,
+    git-annex branch information from remotes that cannot be merged into
+    the git-annex branch will now not crash it, but will be merged in
+    memory.
+  * addurl, youtube-dl: When --check-raw prevents downloading an url,
+    still continue with any downloads that come after it, rather than
+    erroring out.
+  * Fix locking problems when annex.pidlock is set and concurrency is
+    enabled eg with -J.
+  * Improve error message display when autoinit fails due to eg, a
+    permissions problem.
+  * export: Avoid unncessarily re-exporting non-annexed files that were
+    already exported.
+  * Improve git command queue flushing so that eg, addurl of several
+    large files that take time to download will update the index for each
+    file, rather than deferring the index updates to the end.
+  * sync: Better error message when unable to export to a remote because
+    remote.name.annex-tracking-branch is configured to a ref that does not
+    exist.
+  * Fix build with ghc 9.0.1
+  * Fix build with old versions of feed library."""]]
\ No newline at end of file

comment
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_4_59e2ea4d59d5f4a8472cc8c945b8a419._comment b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_4_59e2ea4d59d5f4a8472cc8c945b8a419._comment
new file mode 100644
index 0000000000..75d4e2c10c
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_4_59e2ea4d59d5f4a8472cc8c945b8a419._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2021-12-31T19:00:10Z"
+ content="""
+Upstream bug report: <https://github.com/haskell/process/issues/223>
+"""]]

Added a comment
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_3_641473e43c8da7e7be02658cf8d3bd3e._comment b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_3_641473e43c8da7e7be02658cf8d3bd3e._comment
new file mode 100644
index 0000000000..0e689741e4
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_3_641473e43c8da7e7be02658cf8d3bd3e._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="falsifian"
+ avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae"
+ subject="comment 3"
+ date="2021-12-31T16:03:33Z"
+ content="""
+Looks like you are right. Thanks for the help! On DragonFlyBSD with ghc 8.10.7 I see:
+
+```
+copter-dfly tmp $ runghc foo.hs
+testing:
+testing: bar
+```
+
+I guess it is time to look at the `System.Process` module.
+"""]]

comment
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_2_eeff3cb51df849b783446e238de002a8._comment b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_2_eeff3cb51df849b783446e238de002a8._comment
new file mode 100644
index 0000000000..da0d68bd57
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_2_eeff3cb51df849b783446e238de002a8._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2021-12-31T15:54:25Z"
+ content="""
+On Linux this problem does not occur. 
+
+I built git-annex with ghc-8.10.7 by changing the
+stack.yaml to use "resolver: lts-18.20" and running
+"stack build".
+"""]]

comment
diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_1_70b807c69be263fd53c578fe6ec84561._comment b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_1_70b807c69be263fd53c578fe6ec84561._comment
new file mode 100644
index 0000000000..2481394326
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD/comment_1_70b807c69be263fd53c578fe6ec84561._comment
@@ -0,0 +1,26 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2021-12-31T15:05:40Z"
+ content="""
+It seems that somehow it is failing to set environment variables. Eg,
+`GIT_INDEX_FILE` is what makes git use `.git/annex/index` and if it was not
+set it would behave as you describe.
+
+This strongly suggests that there is a bug in the compiler/libraries.
+
+Here is a small test program:
+
+	import System.Environment
+	import System.Process
+	
+	main = do
+		let p = (proc "sh" ["-c", "echo testing: $TEST"])
+		let p' = p { env = Just [("TEST", "foo")] } 
+		withCreateProcess p' $ \_ _ _ pid -> waitForProcess pid
+		setEnv "TEST" "bar"
+		withCreateProcess p $ \_ _ _ pid -> waitForProcess pid
+
+If run with "runghc foo.hs", that should output "testing: foo" and then 
+"testing: bar".
+"""]]

diff --git a/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn
new file mode 100644
index 0000000000..a30eeb33ad
--- /dev/null
+++ b/doc/bugs/unusable_with_ghc_8.10.7_on_DragonFlyBSD.mdwn
@@ -0,0 +1,44 @@
+On DragonflyBSD with git-annex built with ghc 8.10.7, a simple `git annex add` puts strange entries in the index, as shown:
+
+```
+copter-dfly tmp $ mkdir d
+copter-dfly tmp $ cd d
+copter-dfly d $ git init
+Initialized empty Git repository in /home/falsifian/tmp/d/.git/
+copter-dfly d $ git annex init
+init  ok
+(recording state in git...)
+copter-dfly d $ echo test>f
+copter-dfly d $ git annex add f
+add f
+ok
+(recording state in git...)
+copter-dfly d $ git status
+On branch master
+
+No commits yet
+
+Changes to be committed:
+  (use "git rm --cached <file>..." to unstage)
+        new file:   30d/f3a/SHA256-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2.log
+        new file:   f
+        new file:   uuid.log
+
+Changes not staged for commit:
+  (use "git add/rm <file>..." to update what will be committed)
+  (use "git restore <file>..." to discard changes in working directory)
+        deleted:    30d/f3a/SHA256-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2.log
+        deleted:    uuid.log
+
+copter-dfly d $
+```
+
+Digging into it further with dfly people over IRC, it's been discovered:
+
+* The bug does not appear with ghc 8.10.5, with git-annex 20210903 or 20210223.
+* But it consistently happens with ghc 8.10.7, with all the versions I tried.
+* I also noticed that `git annex init` does not create the file `.git/annex/index` when the bug is present.
+
+Has anyone tried git-annex with ghc 8.10.7 on a different OS?
+
+I tried to figure out why `.git/annex/index` was not being created, but I had some trouble figuring out which line of code is supposed to create it. If someone could point me there I could do some further debugging to see why it isn't created.

comments
diff --git a/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_10_d2242b526c7cce1084bbb63ef2bfe06a._comment b/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_10_d2242b526c7cce1084bbb63ef2bfe06a._comment
new file mode 100644
index 0000000000..2ff530eb74
--- /dev/null
+++ b/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_10_d2242b526c7cce1084bbb63ef2bfe06a._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2021-12-30T16:43:10Z"
+ content="""
+@yoh I'm also planning to extend git-annex's test suite with checks that
+some of the basic queries work in readonly repos.
+"""]]
diff --git a/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_9_49cda6f52fa4fb527b392f6ee6bae078._comment b/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_9_49cda6f52fa4fb527b392f6ee6bae078._comment
new file mode 100644
index 0000000000..9b6c23c4ab
--- /dev/null
+++ b/doc/bugs/merge-annex-branches__61__false_-_automate_and_extend/comment_9_49cda6f52fa4fb527b392f6ee6bae078._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 9"""
+ date="2021-12-30T16:34:03Z"
+ content="""
+@jkniiv that may somehow be windows-specific, if opening a sqlite database
+on windows in order to only query it needs write access.
+
+I did try this on linux with an unlocked file and it did not crash, but the
+file was already present in the database. I also tried with the keysdb
+deleted, and it still worked. I'm sure that there are situations though
+where the database is not updated where it will try to populate it before
+querying it, which will fail when it cannot write.
+
+It would be worth filing new bug reports for such issues. Although
+when write access is unavoidable, I might just have to close them.
+"""]]