Recent changes to this wiki:

investigate using youtube-dl. not pretty..
diff --git a/doc/todo/switch_from_quvi_to_youtube-dl.mdwn b/doc/todo/switch_from_quvi_to_youtube-dl.mdwn
new file mode 100644
index 000000000..31c1a73f0
--- /dev/null
+++ b/doc/todo/switch_from_quvi_to_youtube-dl.mdwn
@@ -0,0 +1,41 @@
+quvi does not seem maintained (last upstream release in 2013)
+and it supports many fewer videos than youtube-dl does.
+
+The difficulty with using youtube-dl is it, by design, does not
+provide a way to probe if it supports an url, other than running it
+and seeing if it finds a video at the url. This would make `git annex
+addurl` significantly slower if it ran youtube-dl to probe every url.
+
+It is possible to use youtube-dl to download arbitrary non-video files;
+it stores the file to disk just as wget or curl. But, that's well outside
+its intended use case, and so it does not feel like a good idea to make
+git-annex depend on using youtube-dl to download generic urls.
+(Also, youtube-dl has bugs with downloading non-video 
+urls, see for example http://bugs.debian.org/874321)
+
+So, switching to youtube-dl would probably need a new switch, like `git
+annex addurl --rip` that enables using it.
+
+Currently `git annex importfeed` automatically tests for video urls with
+quvi; it would also need to support `--rip`.
+
+Both of those changes would need changes to user's workflows and cron jobs.
+git-annex could keep supporting quvi for some time, and warn when it uses
+quvi, to help with the transition.
+
+Another gotcha is playlists. youtube-dl downloads playlists automatically.
+But, git-annex needs to record an url that downloads a single file so that
+`git annex get` works right. So, playlists will need to be disabled when
+git-annex runs youtube-dl. But, `--no-playlist` does not always disable
+playlists. Best option seems to be `--playlist-items 0` which works for
+non-playlists, and downloads only 1 item from playlists (hopefully a fairly
+stable item, but who knows..).
+
+Another gotcha is that youtube-dl's -o option does not fully determine the
+filename it downloads to. Sometims it will tack on an additional extension
+(seen with youtube videos where it added a ".mkv").
+And --get-filename does not report the actual filename when that happens.
+Only way I can find to avoid this wart is output to stdout with "-o -",
+but that would prevent resuming. Seems it would have to be run in a temp
+dir and the file moved out to the git-annex location when done, which would
+prevent stuff that operates on incomplete downloads from working.

ah, I got it
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment
index 170b21764..5645cb675 100644
--- a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment
@@ -5,4 +5,13 @@
  content="""
 I don't understand what you mean with the find command. Are you talking
 about the "unknown" values in the json?
+
+Oh, I suppose you mean particularly that the bytesize is unknown.
+
+Well, this would make `find` output differ depending on whether the key is
+present or not, in cases where it would otherwise be the same. And it would
+change machine-consumable output in a way that for all I know would break
+stuff.
+
+So, changing that seems like a bad idea.
 """]]

huh?
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment
new file mode 100644
index 000000000..170b21764
--- /dev/null
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_5_8c33fc01dd1de2abcd3782ff89b437de._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2017-11-20T18:18:07Z"
+ content="""
+I don't understand what you mean with the find command. Are you talking
+about the "unknown" values in the json?
+"""]]

diff --git a/doc/bugs/yesod-default_appears_no_longer_necessary.mdwn b/doc/bugs/yesod-default_appears_no_longer_necessary.mdwn
new file mode 100644
index 000000000..8c5e79528
--- /dev/null
+++ b/doc/bugs/yesod-default_appears_no_longer_necessary.mdwn
@@ -0,0 +1,21 @@
+### Please describe the problem.
+
+The description for yesod-default is:
+
+> Since version 1.2 of Yesod, this functionality is provided by the yesod package. This package is no longer needed.
+
+git-annex depends on yesod-default >=1.2.0 and yesod >=1.2.6 (and many other yesod-*>1.2).
+
+Thus git-annex should be able to drop yesod-default from the cabal file.
+
+### What steps will reproduce the problem?
+
+Just build from source.
+
+### What version of git-annex are you using? On what operating system?
+
+6.20171109, source tarball.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Sure, this is not really a "problem" anyway.

diff --git a/doc/bugs/GPG_subkeys.mdwn b/doc/bugs/GPG_subkeys.mdwn
index 66ce12cad..d3eedf4b9 100644
--- a/doc/bugs/GPG_subkeys.mdwn
+++ b/doc/bugs/GPG_subkeys.mdwn
@@ -7,7 +7,7 @@ The issue is that git-annex consistently tries to encrypt with the master key, w
 
 I'm submitting a bug report but I should say that I'm not sure this is the intended (or documented) way to use GPG subkeys and it may not be a git-annex concern. The [Debian Wiki](https://wiki.debian.org/Subkeys) does say that it won't work but the [password store](https://lists.zx2c4.com/pipermail/password-store/2014-September/001131.html) project has implemented it.
 
-Side note: this is not working in grcypt either.
+Side note: this is not working in gcrypt either.
 
 ### What steps will reproduce the problem?
 

diff --git a/doc/bugs/GPG_subkeys.mdwn b/doc/bugs/GPG_subkeys.mdwn
new file mode 100644
index 000000000..66ce12cad
--- /dev/null
+++ b/doc/bugs/GPG_subkeys.mdwn
@@ -0,0 +1,30 @@
+### Please describe the problem.
+Despite the very fast, much appreciated, and excellent response to my initial question [here](http://git-annex.branchable.com/encryption/#comment-cb9c20121e570f80fcd799c1619ced69), I still can't get encryption working with subkeys. 
+
+I have a few different machines, each with its own subkey and each configured with the master key stripped out. I'm hoping to be able to encrypt and decrypt from the same special remotes with each of them.
+
+The issue is that git-annex consistently tries to encrypt with the master key, which is not available, and so encryption and the `initremote` fail.
+
+I'm submitting a bug report but I should say that I'm not sure this is the intended (or documented) way to use GPG subkeys and it may not be a git-annex concern. The [Debian Wiki](https://wiki.debian.org/Subkeys) does say that it won't work but the [password store](https://lists.zx2c4.com/pipermail/password-store/2014-September/001131.html) project has implemented it.
+
+Side note: this is not working in grcypt either.
+
+### What steps will reproduce the problem?
+
+I've tried the following in a few different combinations, with both short and long KEYIDs, and with or without the appended '!' that's supposed to tell gpg which key to use for encryption:  
+`git annex initremote cloud type=S3 encryption=hybrid keyid=EFEFEFE host=storage.googleapis.com port=80`
+
+In each case I get the same error:  
+`gpg: decryption failed: No secret key`  
+`git-annex: user error (/usr/local/bin/gpg ["--quiet","--trust-model","always","--decrypt"] exited 2)`  
+`failed` 
+
+### What version of git-annex are you using? On what operating system?
+`git-annex version: 6.20171109`  
+OS X 10.10.5
+
+### Please provide any additional information below.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+I love git annex. Thank you for your work on it. 
+

diff --git a/doc/forum/Adding_a_URL_from_wayback_machine.mdwn b/doc/forum/Adding_a_URL_from_wayback_machine.mdwn
index 49eeef470..d019e255b 100644
--- a/doc/forum/Adding_a_URL_from_wayback_machine.mdwn
+++ b/doc/forum/Adding_a_URL_from_wayback_machine.mdwn
@@ -59,6 +59,6 @@ This is what I've got so far:
                 urls.append(r.geturl())
                 break
     assert(len(files) == len(urls))
-    p = Popen(['git-annex', 'addurl', '--batch', '--with-files', '--relzed'], stdin=PIPE, stdout=stdout)
+    p = Popen(['git-annex', 'addurl', '--batch', '--with-files', '--relaxed'], stdin=PIPE, stdout=stdout)
     p.stdin.writelines(str.encode(f + ' ' + u) for f, u in zip(urls, files))
     p.communicate()

diff --git a/doc/forum/Adding_a_URL_from_wayback_machine.mdwn b/doc/forum/Adding_a_URL_from_wayback_machine.mdwn
new file mode 100644
index 000000000..49eeef470
--- /dev/null
+++ b/doc/forum/Adding_a_URL_from_wayback_machine.mdwn
@@ -0,0 +1,64 @@
+Is there a nice way to add a URL from the Wayback machine?
+I don't want to add HTML documents but rather PDFs etc.
+The problem is that web.archive.org doesn't send a Content-Length header:
+
+    $ curl -I http://web.archive.org/web/20171117120847/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg
+    HTTP/1.1 200 OK
+    Server: Tengine/2.1.0
+    Date: Fri, 17 Nov 2017 12:09:35 GMT
+    Content-Type: audio/ogg
+    Connection: keep-alive
+    X-Archive-Orig-content-length: 29784324
+    X-Archive-Orig-accept-ranges: bytes
+    X-Archive-Orig-server: Apache/2.4.29 (Debian)
+    X-Archive-Orig-last-modified: Wed, 11 Feb 2015 01:31:47 GMT
+    X-Archive-Orig-connection: close
+    X-Archive-Orig-etag: "1c67904-50ec5f783257c"
+    X-Archive-Orig-date: Fri, 17 Nov 2017 12:08:49 GMT
+    Cache-Control: max-age=1800
+    X-Archive-Guessed-Content-Type: audio/ogg
+    Memento-Datetime: Fri, 17 Nov 2017 12:08:47 GMT
+    Link: <https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="original", <http://web.archive.org/web/timemap/link/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="timemap"; type="application/link-format", <http://web.archive.org/web/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="timegate", <http://web.archive.org/web/20140426045733/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="first memento"; datetime="Sat, 26 Apr 2014 04:57:33 GMT", <http://web.archive.org/web/20170822174608/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="prev memento"; datetime="Tue, 22 Aug 2017 17:46:08 GMT", <http://web.archive.org/web/20171117120847/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="memento"; datetime="Fri, 17 Nov 2017 12:08:47 GMT", <http://web.archive.org/web/20171117120847/https://downloads.kitenet.net/videos/git-annex/git-annex_views_demo.ogg>; rel="last memento"; datetime="Fri, 17 Nov 2017 12:08:47 GMT"
+    Content-Security-Policy: default-src 'self' 'unsafe-eval' 'unsafe-inline' data: archive.org web.archive.org analytics.archive.org
+    X-App-Server: wwwb-app39
+    X-ts: ----
+    X-Archive-Playback: 0
+    X-location: All
+    X-Page-Cache: MISS
+
+Therefore, I'd have to pass the `--relaxed` flag when calling `addurl`.
+Is there maybe a way to tell git-annex to download the file before checking its size?
+Or is there a way to use the Wayback machine via the S3 special remote?
+
+Eventually, I'd like to have a script that automatically saves all web content in my repo
+into the Wayback machine and adds the new URLs for redundancy.
+
+This is what I've got so far:
+
+    #! /usr/bin/env python3
+    
+    from subprocess import check_output, Popen, PIPE
+    import json
+    from urllib import request
+    from sys import stdout
+    
+    output = check_output(['git-annex', 'find', '--in', 'web', '--json'])
+    files = [json.loads(line)['file'] for line in output.split(b'\n')[:-1]]
+    p = Popen(['git-annex', 'whereis', '--batch', '--json'], stdin=PIPE, stdout=PIPE)
+    p.stdin.writelines(str.encode(f + '\n') for f in files)
+    out, err = p.communicate()
+    urls = []
+    for line in out.split(b'\n')[:-1]:
+        for loc in json.loads(line)['untrusted']:
+            if loc['uuid'] == '00000000-0000-0000-0000-000000000001':
+                url = loc['urls'][0]
+                import urllib.request
+                print('GET https://web.archive.org/save/' + url)
+                r = request.urlopen('https://web.archive.org/save/' + url)
+                assert(r.getcode() == 200)
+                urls.append(r.geturl())
+                break
+    assert(len(files) == len(urls))
+    p = Popen(['git-annex', 'addurl', '--batch', '--with-files', '--relzed'], stdin=PIPE, stdout=stdout)
+    p.stdin.writelines(str.encode(f + ' ' + u) for f, u in zip(urls, files))
+    p.communicate()

Added a comment: thanks, what about 'find' with the same issue?
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_4_bc55ede32beee2a74b943132e6377005._comment b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_4_bc55ede32beee2a74b943132e6377005._comment
new file mode 100644
index 000000000..0f36e4396
--- /dev/null
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_4_bc55ede32beee2a74b943132e6377005._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="thanks, what about 'find' with the same issue?"
+ date="2017-11-16T19:11:51Z"
+ content="""
+Shoudn't the same apply to \"find\" output?
+
+[[!format sh \"\"\"
+$> git annex find --not --in localhost --json Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4
+{\"bytesize\":\"unknown\",\"mtime\":\"unknown\",\"keyname\":\"quvi:https://www.youtube.com/watch,63v,618t6znEOEDVo\",\"backend\":\"URL\",\"key\":\"URL--quvi:https://www.youtube.com/watch,63v,618t6znEOEDVo\",\"humansize\":\"unknown\",\"hashdirmixed\":\"kq/PM/\",\"file\":\"Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4\",\"hashdirlower\":\"8ba/bf9/\"}
+
+$> sudo dpkg -i /tmp/git-annex-standalone_6.20171114+gitg5e6c3ba30-1\~ndall+1_amd64.deb 
+[sudo] password for yoh: 
+(Reading database ... 706557 files and directories currently installed.)
+Preparing to unpack .../git-annex-standalone_6.20171114+gitg5e6c3ba30-1~ndall+1_amd64.deb ...
+Unpacking git-annex-standalone (6.20171114+gitg5e6c3ba30-1~ndall+1) over (6.20171018+gitgbb20b1ed3-1~ndall+1) ...
+Setting up git-annex-standalone (6.20171114+gitg5e6c3ba30-1~ndall+1) ...
+...
+
+$> git annex find --not --in localhost --json Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4
+{\"bytesize\":\"unknown\",\"mtime\":\"unknown\",\"keyname\":\"quvi:https://www.youtube.com/watch,63v,618t6znEOEDVo\",\"backend\":\"URL\",\"key\":\"URL--quvi:https://www.youtube.com/watch,63v,618t6znEOEDVo\",\"humansize\":\"unknown\",\"hashdirmixed\":\"kq/PM/\",\"file\":\"Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4\",\"hashdirlower\":\"8ba/bf9/\"}
+
+$> ls -lL Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4
+-r-------- 1 yoh yoh 64354577 Mar 22  2014 Brainhack2012__Share_your_tools__But_fear_the_wombat_.mp4
+
+
+\"\"\"]]
+"""]]

diff --git a/doc/todo/Add_a_way_to_mark_exporttree_remotes_dead.mdwn b/doc/todo/Add_a_way_to_mark_exporttree_remotes_dead.mdwn
new file mode 100644
index 000000000..eba133133
--- /dev/null
+++ b/doc/todo/Add_a_way_to_mark_exporttree_remotes_dead.mdwn
@@ -0,0 +1,8 @@
+There is currently no way to get rid of an exporttree remote, because the trust level `untrusted` is enforced.
+
+
+    $ git annex dead exp
+    dead exp
+      This remote's trust level is overridden to untrusted.
+    ok
+    (recording state in git...)

Display progress meter when uploading a key without size information
Getting the size by statting the content file.
This commit was supported by the NSF-funded DataLad project.
diff --git a/CHANGELOG b/CHANGELOG
index 21aaa2650..72d8d775f 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,10 @@
+git-annex (6.20171110) UNRELEASED; urgency=medium
+
+  * Display progress meter when uploading a key without size information,
+    getting the size by statting the content file.
+
+ -- Joey Hess <id@joeyh.name>  Tue, 14 Nov 2017 16:14:20 -0400
+
 git-annex (6.20171109) unstable; urgency=medium
 
   * Fix export of subdir of a branch.
diff --git a/Command/Export.hs b/Command/Export.hs
index dfa452956..4e1880c13 100644
--- a/Command/Export.hs
+++ b/Command/Export.hs
@@ -215,20 +215,20 @@ performExport r ea db ek af contentsha loc = do
 	let storer = storeExport ea
 	sent <- case ek of
 		AnnexKey k -> ifM (inAnnex k)
-			( metered Nothing k $ \m -> do
-				let rollback = void $
-					performUnexport r ea db [ek] loc
-				notifyTransfer Upload af $
-					upload (uuid r) k af noRetry $ \pm -> do
-						let m' = combineMeterUpdate pm m
-						sendAnnex k rollback
-							(\f -> storer f k loc m')
+			( notifyTransfer Upload af $
+				upload (uuid r) k af noRetry $ \pm -> do
+					let rollback = void $
+						performUnexport r ea db [ek] loc
+					sendAnnex k rollback $ \f ->
+						metered Nothing k (return $ Just f) $ \m -> do
+							let m' = combineMeterUpdate pm m
+							storer f k loc m'
 			, do
 				showNote "not available"
 				return False
 			)
 		-- Sending a non-annexed file.
-		GitKey sha1k -> metered Nothing sha1k $ \m ->
+		GitKey sha1k -> metered Nothing sha1k (return Nothing) $ \m ->
 			withTmpFile "export" $ \tmp h -> do
 				b <- catObject contentsha
 				liftIO $ L.hPut h b
diff --git a/Messages/Progress.hs b/Messages/Progress.hs
index 3c263c05c..61486d78d 100644
--- a/Messages/Progress.hs
+++ b/Messages/Progress.hs
@@ -24,12 +24,18 @@ import qualified System.Console.Concurrent as Console
 #endif
 
 {- Shows a progress meter while performing a transfer of a key.
- - The action is passed a callback to use to update the meter. -}
-metered :: Maybe MeterUpdate -> Key -> (MeterUpdate -> Annex a) -> Annex a
-metered othermeter key a = withMessageState $ go (keySize key)
+ - The action is passed a callback to use to update the meter.
+ -
+ - When the key's size is not known, the srcfile is statted to get the size.
+ - This allows uploads of keys without size to still have progress
+ - displayed.
+ --}
+metered :: Maybe MeterUpdate -> Key -> Annex (Maybe FilePath) -> (MeterUpdate -> Annex a) -> Annex a
+metered othermeter key getsrcfile a = withMessageState $ \st ->
+	flip go st =<< getsz
   where
 	go _ (MessageState { outputType = QuietOutput }) = nometer
-	go (msize) (MessageState { outputType = NormalOutput, concurrentOutputEnabled = False }) = do
+	go msize (MessageState { outputType = NormalOutput, concurrentOutputEnabled = False }) = do
 		showOutput
 		meter <- liftIO $ mkMeter msize bandwidthMeter $ 
 			displayMeterHandle stdout
@@ -38,7 +44,7 @@ metered othermeter key a = withMessageState $ go (keySize key)
 		r <- a (combinemeter m)
 		liftIO $ clearMeterHandle meter stdout
 		return r
-	go (msize) (MessageState { outputType = NormalOutput, concurrentOutputEnabled = True }) =
+	go msize (MessageState { outputType = NormalOutput, concurrentOutputEnabled = True }) =
 #if WITH_CONCURRENTOUTPUT
 		withProgressRegion $ \r -> do
 			meter <- liftIO $ mkMeter msize bandwidthMeter $ \_ s ->
@@ -61,14 +67,22 @@ metered othermeter key a = withMessageState $ go (keySize key)
 	combinemeter m = case othermeter of
 		Nothing -> m
 		Just om -> combineMeterUpdate m om
+	
+	getsz = case keySize key of
+		Just sz -> return (Just sz)
+		Nothing -> do
+			srcfile <- getsrcfile
+			case srcfile of
+				Nothing -> return Nothing
+				Just f -> catchMaybeIO $ liftIO $ getFileSize f
 
 {- Use when the command's own progress output is preferred.
  - The command's output will be suppressed and git-annex's progress meter
  - used for concurrent output, and json progress. -}
-commandMetered :: Maybe MeterUpdate -> Key -> (MeterUpdate -> Annex a) -> Annex a
-commandMetered combinemeterupdate key a = 
+commandMetered :: Maybe MeterUpdate -> Key -> Annex (Maybe FilePath) -> (MeterUpdate -> Annex a) -> Annex a
+commandMetered combinemeterupdate key getsrcfile a = 
 	withMessageState $ \s -> if needOutputMeter s
-		then metered combinemeterupdate key a
+		then metered combinemeterupdate key getsrcfile a
 		else a (fromMaybe nullMeterUpdate combinemeterupdate)
 
 {- Poll file size to display meter, but only when concurrent output or
@@ -76,7 +90,7 @@ commandMetered combinemeterupdate key a =
 meteredFile :: FilePath -> Maybe MeterUpdate -> Key -> Annex a -> Annex a
 meteredFile file combinemeterupdate key a = 
 	withMessageState $ \s -> if needOutputMeter s
-		then metered combinemeterupdate key $ \p ->
+		then metered combinemeterupdate key (return Nothing) $ \p ->
 			watchFileSize file p a
 		else a
 
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 30307d037..da2ecee57 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -435,7 +435,7 @@ copyFromRemote :: Remote -> Key -> AssociatedFile -> FilePath -> MeterUpdate ->
 copyFromRemote r key file dest p
 	| Git.repoIsHttp (repo r) = unVerified $
 		Annex.Content.downloadUrl key p (keyUrls r key) dest
-	| otherwise = commandMetered (Just p) key $
+	| otherwise = commandMetered (Just p) key (return Nothing) $
 		copyFromRemote' r key file dest
 
 copyFromRemote' :: Remote -> Key -> AssociatedFile -> FilePath -> MeterUpdate -> Annex (Bool, Verification)
@@ -546,27 +546,24 @@ copyFromRemoteCheap _ _ _ _ = return False
 
 {- Tries to copy a key's content to a remote's annex. -}
 copyToRemote :: Remote -> Key -> AssociatedFile -> MeterUpdate -> Annex Bool
-copyToRemote r key file meterupdate = 
-	commandMetered (Just meterupdate) key $
-		copyToRemote' r key file
-
-copyToRemote' :: Remote -> Key -> AssociatedFile -> MeterUpdate -> Annex Bool
-copyToRemote' r key file meterupdate
+copyToRemote r key file meterupdate
 	| not $ Git.repoIsUrl (repo r) =
 		guardUsable (repo r) (return False) $ commitOnCleanup r $
 			copylocal =<< Annex.Content.prepSendAnnex key
 	| Git.repoIsSsh (repo r) = commitOnCleanup r $
-		Annex.Content.sendAnnex key noop $ \object -> do
-			-- This is too broad really, but recvkey normally
-			-- verifies content anyway, so avoid complicating
-			-- it with a local sendAnnex check and rollback.
-			unlocked <- isDirect <||> versionSupportsUnlockedPointers
-			Ssh.rsyncHelper (Just meterupdate)
-				=<< Ssh.rsyncParamsRemote unlocked r Upload key object file
+		Annex.Content.sendAnnex key noop $ \object ->
+			withmeter object $ \p -> do
+				-- This is too broad really, but recvkey normally
+				-- verifies content anyway, so avoid complicating
+				-- it with a local sendAnnex check and rollback.
+				unlocked <- isDirect <||> versionSupportsUnlockedPointers
+				Ssh.rsyncHelper (Just p)
+					=<< Ssh.rsyncParamsRemote unlocked r Upload key object file
 	| otherwise = giveup "copying to non-ssh repo not supported"
   where
+	withmeter object = commandMetered (Just meterupdate) key (return $ Just object)
 	copylocal Nothing = return False
-	copylocal (Just (object, checksuccess)) = do
+	copylocal (Just (object, checksuccess)) = withmeter object $ \p -> do
 		-- The checksuccess action is going to be run in
 		-- the remote's Annex, but it needs access to the local
 		-- Annex monad's state.
@@ -581,11 +578,11 @@ copyToRemote' r key file meterupdate
 				ensureInitialized
 				copier <- mkCopier hardlink params
 				let verify = Annex.Content.RemoteVerify r
-				runTransfer (Transfer Download u key) file forwardRetry $ \p ->
-					let p' = combineMeterUpdate meterupdate p
+				runTransfer (Transfer Download u key) file forwardRetry $ \p' ->
+					let p'' = combineMeterUpdate p p'
 					in Annex.Content.saveState True `after`
 						Annex.Content.getViaTmp verify key
-							(\dest -> copier object dest p' (liftIO checksuccessio))
+							(\dest -> copier object dest p'' (liftIO checksuccessio))
 			)
 
 fsckOnRemote :: Git.Repo -> [CommandParam] -> Annex (IO Bool)
diff --git a/Remote/Helper/Special.hs b/Remote/Helper/Special.hs
index ae654d517..f7e9759a4 100644
--- a/Remote/Helper/Special.hs
+++ b/Remote/Helper/Special.hs
@@ -187,7 +187,7 @@ specialRemote' cfg c preparestorer prepareretriever prepareremover preparecheckp
 		go (Just storer) = preparecheckpresent k $ safely . go' storer
 		go Nothing = return False
 		go' storer (Just checker) = sendAnnex k rollback $ \src ->
-			displayprogress p k $ \p' ->
+			displayprogress p k (Just src) $ \p' ->
 				storeChunks (uuid baser) chunkconfig enck k src p'
 					(storechunk enc storer)
 					checker

(Diff truncated)
Added a comment
diff --git a/doc/forum/help__44___a_bunch_of_files_disappeared/comment_3_4b700ccba2b40fdc1d172a3dcbc441e5._comment b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_3_4b700ccba2b40fdc1d172a3dcbc441e5._comment
new file mode 100644
index 000000000..c5b402ec9
--- /dev/null
+++ b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_3_4b700ccba2b40fdc1d172a3dcbc441e5._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="astrophoenix"
+ avatar="http://cdn.libravatar.org/avatar/cb0b7982b6877c58c3860d8ad5fb3148"
+ subject="comment 3"
+ date="2017-11-14T19:45:41Z"
+ content="""
+I'm nearly 100% certain I did a drop --force by mistake. I'm pulling the files from backups. 
+
+thank you!
+"""]]

thoughts
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_1_8e8dd79385b523502b39247135385bc4._comment b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_1_8e8dd79385b523502b39247135385bc4._comment
new file mode 100644
index 000000000..631155c20
--- /dev/null
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_1_8e8dd79385b523502b39247135385bc4._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-14T19:19:30Z"
+ content="""
+It indeed would be possible for `copy --to` to check the actual file
+size when the key does not have a known size, and use that for progress.
+I don't know how hard it would be.
+
+Note that, even if that were done, there's no guarantee that a given remote
+will update progress information, and if it doesn't, --json-progress
+won't result in any. So your code certianly needs to handle that case.
+"""]]
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_2_85c2ca68a4611e684c147fc005470fab._comment b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_2_85c2ca68a4611e684c147fc005470fab._comment
new file mode 100644
index 000000000..29c124845
--- /dev/null
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file/comment_2_85c2ca68a4611e684c147fc005470fab._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-14T19:29:21Z"
+ content="""
+Messages.Progress.metered is what looks at the keySize, and when it's
+not known, displays no meter. So it would need an additional Maybe FilePath
+that's the file being uploaded, to look at when the keySize is not known.
+
+That does not seem too hard a change to make; I'm not convinced the extra
+complexity is worth it, since this would only add progress for uploads,
+and not for downloads.
+
+There is something to be said for consistency;
+and if some transfers of a key have progress and others do not,
+it seems the user might get confused, while if nothing does, the user
+can conclude that git-annex is not able to provide progress for a key
+that does not contain a size, and if they don't like that, avoid
+things that generate such keys.
+"""]]

initial whining about 2 ssh prompts
diff --git a/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file.mdwn b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file.mdwn
new file mode 100644
index 000000000..c1cc87816
--- /dev/null
+++ b/doc/bugs/annex_copy_might_not_report_percent-progress_when_it_has_actual_key_file.mdwn
@@ -0,0 +1,23 @@
+### Please describe the problem.
+
+git-annex does not report percent-progress in the json-progress output if size is not listed in the key.  But the file is available so git annex could easily report the %s.  We will need to workaround in datalad atm where we assumed that percents are always reported
+
+[[!format sh """
+$> git annex copy --to=localhost --json --json-progress Why_is_git_annex_awesome__This_is_why_.webm 
+{"byte-progress":32768,"action":{"command":"copy","note":"to localhost...","key":"URL--quvi:https://www.youtube.com/watch,63v,614qCZFW_uGU0","file":"Why_is_git_annex_awesome__This_is_why_.webm"}}
+{"command":"copy","note":"to localhost...","success":true,"key":"URL--quvi:https://www.youtube.com/watch,63v,614qCZFW_uGU0","file":"Why_is_git_annex_awesome__This_is_why_.webm"}
+
+$> du -scmL Why_is_git_annex_awesome__This_is_why_.webm
+6	Why_is_git_annex_awesome__This_is_why_.webm
+6	total
+
+$> git annex version
+git-annex version: 6.20171018+gitgbb20b1ed3-1~ndall+1
+...
+
+$> ls -l Why_is_git_annex_awesome__This_is_why_.webm
+lrwxrwxrwx 1 yoh yoh 150 Nov  3 09:02 Why_is_git_annex_awesome__This_is_why_.webm -> ../../.git/annex/objects/8f/XP/URL--quvi&chttps&c%%www.youtube.com%watch,63v,614qCZFW_uGU0/URL--quvi&chttps&c%%www.youtube.com%watch,63v,614qCZFW_uGU0
+
+"""]]
+
+

response
diff --git a/doc/forum/help__44___a_bunch_of_files_disappeared/comment_2_7675ff78cf1b5f58618c5c57da8591b5._comment b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_2_7675ff78cf1b5f58618c5c57da8591b5._comment
new file mode 100644
index 000000000..a99aa7bb9
--- /dev/null
+++ b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_2_7675ff78cf1b5f58618c5c57da8591b5._comment
@@ -0,0 +1,23 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-14T17:16:54Z"
+ content="""
+There are two possible things that could have happened.
+
+1. Maybe something somehow wrong got committed to the git repository
+   for the missing files. 
+   
+   If so, you can *always* use git to check out
+   the older version before that commit, and will then be able to see
+   the files; git-annex will know where the content of them is, etc.
+   You can also use `git revert` to revert such a bad commit.
+
+2. Maybe you accidentially dropped the content of the files with --force,
+   and if so, you may have lost the content.
+
+   If you run `git annex log` on one of the files, and look at the lines
+   starting with "-", you can see when the file content was removed from
+   repositories. But there's no way to get it back unless you have another
+   copy somewhere.
+"""]]

update; git bug fixed in pu
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_6_19c9dbdbcc2875cd6429f0ba06336fd2._comment b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_6_19c9dbdbcc2875cd6429f0ba06336fd2._comment
new file mode 100644
index 000000000..11d32e5f6
--- /dev/null
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_6_19c9dbdbcc2875cd6429f0ba06336fd2._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2017-11-13T17:09:31Z"
+ content="""
+The git bug is fixed in git's `pu` branch, or as Peff says,
+at least "papered over", and I've tested the fix with git-annex and
+it seems to avoid the problem. So as long as
+5eb0e541132208a9027cc090943276fa52f29c71 gets into a git release,
+this bug can be closed without any changes to git-annex.
+"""]]

Added a comment: possible cause (my error)
diff --git a/doc/forum/help__44___a_bunch_of_files_disappeared/comment_1_56cffd43ad7056a34c23b139f95ea58e._comment b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_1_56cffd43ad7056a34c23b139f95ea58e._comment
new file mode 100644
index 000000000..53fea0012
--- /dev/null
+++ b/doc/forum/help__44___a_bunch_of_files_disappeared/comment_1_56cffd43ad7056a34c23b139f95ea58e._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="astrophoenix"
+ avatar="http://cdn.libravatar.org/avatar/cb0b7982b6877c58c3860d8ad5fb3148"
+ subject="possible cause (my error)"
+ date="2017-11-12T21:01:28Z"
+ content="""
+oh, it might have been something dumb i did:
+
+at some point I wanted to delete a few files, so trying to be cute I did this:
+
+m=OneMovie.m4v git annex drop --force $m; git rm $m; git ci -m \"rm $m\"
+
+I noted that didn't work, but didn't realize it might've taken out most/all the files in the folder
+
+"""]]

diff --git a/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn b/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn
index 675ed094d..b3479e903 100644
--- a/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn
+++ b/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn
@@ -8,7 +8,7 @@ I hadn't done a git annex sync in a long time. so I updated via brew on my high
 
 when I ran git annex sync on my server, i saw it modify the 6 or so movies which I had copied over to the old lion laptop long ago. those six movies are now pointed to hashes which don't exist, and when I look at the history, the previous hashes don't exist either.
 
-then I ran a git annex fsck on the server, and it listed tons of movies which is says have so known copies. at some point, all of them were on the server. I suspect, but don't know how to confirm, that all the movies are still sitting in the annex on the server, but many many of the links are wrong.
+then I ran a git annex fsck on the server, and it listed tons of movies which is says have no known copies. at some point, all of them were on the server. I suspect, but don't know how to confirm, that all the movies are still sitting in the annex on the server, but many many of the links are wrong.
 
 how can I fix the links?
 

post question about missing files
diff --git a/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn b/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn
new file mode 100644
index 000000000..675ed094d
--- /dev/null
+++ b/doc/forum/help__44___a_bunch_of_files_disappeared.mdwn
@@ -0,0 +1,15 @@
+I have a large repo full of movies (dvd and blue-ray rips). I host it on my ubuntu linux 16.04 xenial system which sits next to my tv.
+
+I use git annex sync to copy stuff to my mac laptop running high sierra and I just updated to the brew version of git-annex there, 6.20171109
+
+I also have a copy on my really old laptop running mac os x lion, that one has a really really old version running.
+
+I hadn't done a git annex sync in a long time. so I updated via brew on my high sierra laptop to 6.20171109 and ran git annex sync on the laptop, and then on my server.
+
+when I ran git annex sync on my server, i saw it modify the 6 or so movies which I had copied over to the old lion laptop long ago. those six movies are now pointed to hashes which don't exist, and when I look at the history, the previous hashes don't exist either.
+
+then I ran a git annex fsck on the server, and it listed tons of movies which is says have so known copies. at some point, all of them were on the server. I suspect, but don't know how to confirm, that all the movies are still sitting in the annex on the server, but many many of the links are wrong.
+
+how can I fix the links?
+
+thanks.

Added a comment: remote "origin" missing some gcrypt commands?
diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_17_098998132fb0bcf130b5bf8a16ac8b8a._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_17_098998132fb0bcf130b5bf8a16ac8b8a._comment
new file mode 100644
index 000000000..8026567a6
--- /dev/null
+++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_17_098998132fb0bcf130b5bf8a16ac8b8a._comment
@@ -0,0 +1,54 @@
+[[!comment format=mdwn
+ username="spam@9590d16798fd27f4e38472862e296fc9828e3d39"
+ nickname="spam"
+ avatar="http://cdn.libravatar.org/avatar/838ad52bf6bddd532ff179811be47f52"
+ subject="remote &quot;origin&quot; missing some gcrypt commands?"
+ date="2017-11-11T19:51:56Z"
+ content="""
+
+I just discovered that cloning over ssh an gcrypt encrypted repository and enabling the remote afterwards is somehow messing up the git config:
+
+git clone grypt::ssh://user@ip.com:/mnt/encrypted_backup
+cd encrypted_backup
+git annex enableremote encrypted_backup gitrepo=/.../encrypted_backup
+
+leads to following in the .git/config of the just cloned repository:
+
+...
+[remote \"origin\"]
+url = grypt::ssh://user@ip.com:/mnt/encrypted_backup
+gcrypt-id = :id:12312312
+fetch = +refs/heads/*:refs/remotes/origin/*
+
+[remote \"encrypted_backup\"]
+url = grypt::ssh://user@ip.com:/mnt/encrypted_backup
+fetch = +refs/heads/*:refs/remotes/server/*
+gcrypt-participants = keyid
+gcrypt-signingkey = keyid
+gcrypt-publish-participants = true
+gcrypt-id = :id:adsasd
+annex-gcrypt = shell
+annex-uuid = 312312312
+...
+
+Note, that for the remote \"origin\" some config like the signingkey is missing compared to the remote \"encrypted_backup\"
+
+Then, running
+git annex sync --content
+
+leads to a error saying
+
+\"gcrypt: Failed to decrypt manifest!\"
+
+during the push process.
+After that I am not able to sync the repository anymore, even with the original repostitory, which initiated the remote.
+The encrypted_backup is then somehow messed up.
+
+Removing the \"origin\" remote via
+git remote remove origin
+
+solves the problem for me. But that command has to be launched right before the first sync, pull or push command! Otherwise the sync process cannot be done anymore.
+
+
+
+"""]]

still need stack-windows.yaml to specify newer versions of Win32 and unix-compat
diff --git a/doc/install/Windows.mdwn b/doc/install/Windows.mdwn
index 46a92fd6c..9ae82e4a9 100644
--- a/doc/install/Windows.mdwn
+++ b/doc/install/Windows.mdwn
@@ -39,8 +39,9 @@ for Windows. One way is to download it from
 Put it somewhere in PATH.
 
 Then open Git Bash, [[clone git-annex|download]], and in git-annex's source
-tree, run "stack build" to download and
-build all dependencies and git-annex. "stack install" will install git-annex.
+tree, run "stack build --stack-yaml stack-windows.yaml" to download and
+build all dependencies and git-annex. "stack install --stack-yaml
+stack-windows.yaml" will install git-annex.
 
 (To build the git-annex installer, you also need to install the NullSoft
 installer system. The script `standalone/windows/build.sh` is
diff --git a/git-annex.cabal b/git-annex.cabal
index f6cb10c05..3363d12a8 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -381,7 +381,7 @@ Executable git-annex
 
   if (os(windows))
     Build-Depends:
-      Win32 (>= 2.6.1.0),
+      Win32 (== 2.3.1.1),
       Win32-extras,
       unix-compat (>= 0.5),
       setenv,
diff --git a/stack-windows.yaml b/stack-windows.yaml
new file mode 100644
index 000000000..7253a5143
--- /dev/null
+++ b/stack-windows.yaml
@@ -0,0 +1,28 @@
+flags:
+  git-annex:
+    concurrentoutput: true
+    production: true
+    assistant: true
+    pairing: true
+    network-uri: true
+    s3: true
+    testsuite: true
+    webdav: true
+    torrentparser: true
+    webapp: true
+    magicmime: false
+    dbus: false
+    android: false
+    androidsplice: false
+packages:
+- '.'
+extra-deps:
+- aws-0.17.1
+- bloomfilter-2.0.1.0
+- torrent-10000.1.1
+- yesod-default-1.2.0
+- Win32-2.6.1.0
+- unix-compat-0.5
+explicit-setup-deps:
+  git-annex: true
+resolver: lts-9.10
diff --git a/stack.yaml b/stack.yaml
index dc181272b..d84c4682e 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -23,4 +23,4 @@ extra-deps:
 - yesod-default-1.2.0
 explicit-setup-deps:
   git-annex: true
-resolver: lts-9.10
+resolver: lts-9.9
diff --git a/standalone/windows/build.sh b/standalone/windows/build.sh
index e3be18af0..b0e6bdd4f 100755
--- a/standalone/windows/build.sh
+++ b/standalone/windows/build.sh
@@ -64,16 +64,17 @@ getextra rsync.exe 85cb7a4d16d274fcf8069b39042965ad26abd6aa
 stack --version
 
 # Build git-annex
-stack setup
-stack install -j 1 --no-haddock --local-bin-path .
+stack setup --stack-yaml stack-windows.yaml
+stack install -j 1 --stack-yaml stack-windows.yaml --no-haddock \
+	--local-bin-path .
 
 # Build the installer
-withcygpreferred stack ghc --no-haddock \
+withcygpreferred stack ghc --stack-yaml stack-windows.yaml --no-haddock \
 	--package nsis Build/NullSoftInstaller.hs
 ./Build/NullSoftInstaller
 
 mkdir -p dist
-stack ghc --no-haddock Build/BuildVersion.hs
+stack ghc --stack-yaml stack-windows.yaml --no-haddock Build/BuildVersion.hs
 ./Build/BuildVersion > dist/build-version
 
 # Test git-annex

use win32 2.6.1.0
That has my patches merged into it, so stack-windows.yaml is not needed
any longer.
This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
diff --git a/doc/install/Windows.mdwn b/doc/install/Windows.mdwn
index 9ae82e4a9..46a92fd6c 100644
--- a/doc/install/Windows.mdwn
+++ b/doc/install/Windows.mdwn
@@ -39,9 +39,8 @@ for Windows. One way is to download it from
 Put it somewhere in PATH.
 
 Then open Git Bash, [[clone git-annex|download]], and in git-annex's source
-tree, run "stack build --stack-yaml stack-windows.yaml" to download and
-build all dependencies and git-annex. "stack install --stack-yaml
-stack-windows.yaml" will install git-annex.
+tree, run "stack build" to download and
+build all dependencies and git-annex. "stack install" will install git-annex.
 
 (To build the git-annex installer, you also need to install the NullSoft
 installer system. The script `standalone/windows/build.sh` is
diff --git a/git-annex.cabal b/git-annex.cabal
index 3363d12a8..f6cb10c05 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -381,7 +381,7 @@ Executable git-annex
 
   if (os(windows))
     Build-Depends:
-      Win32 (== 2.3.1.1),
+      Win32 (>= 2.6.1.0),
       Win32-extras,
       unix-compat (>= 0.5),
       setenv,
diff --git a/stack-windows.yaml b/stack-windows.yaml
deleted file mode 100644
index bdacbec31..000000000
--- a/stack-windows.yaml
+++ /dev/null
@@ -1,30 +0,0 @@
-flags:
-  git-annex:
-    concurrentoutput: true
-    production: true
-    assistant: true
-    pairing: true
-    network-uri: true
-    s3: true
-    testsuite: true
-    webdav: true
-    torrentparser: true
-    webapp: true
-    magicmime: false
-    dbus: false
-    android: false
-    androidsplice: false
-packages:
-- '.'
-- location:
-    git: https://github.com/joeyh/win32
-    commit: 9250d2b8cadf5a98ffb3f68d933a6257c1b7e23d
-  extra-dep: true
-extra-deps:
-- aws-0.17.1
-- bloomfilter-2.0.1.0
-- torrent-10000.1.1
-- yesod-default-1.2.0
-explicit-setup-deps:
-  git-annex: true
-resolver: lts-9.10
diff --git a/stack.yaml b/stack.yaml
index d84c4682e..dc181272b 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -23,4 +23,4 @@ extra-deps:
 - yesod-default-1.2.0
 explicit-setup-deps:
   git-annex: true
-resolver: lts-9.9
+resolver: lts-9.10
diff --git a/standalone/windows/build.sh b/standalone/windows/build.sh
index b0e6bdd4f..e3be18af0 100755
--- a/standalone/windows/build.sh
+++ b/standalone/windows/build.sh
@@ -64,17 +64,16 @@ getextra rsync.exe 85cb7a4d16d274fcf8069b39042965ad26abd6aa
 stack --version
 
 # Build git-annex
-stack setup --stack-yaml stack-windows.yaml
-stack install -j 1 --stack-yaml stack-windows.yaml --no-haddock \
-	--local-bin-path .
+stack setup
+stack install -j 1 --no-haddock --local-bin-path .
 
 # Build the installer
-withcygpreferred stack ghc --stack-yaml stack-windows.yaml --no-haddock \
+withcygpreferred stack ghc --no-haddock \
 	--package nsis Build/NullSoftInstaller.hs
 ./Build/NullSoftInstaller
 
 mkdir -p dist
-stack ghc --stack-yaml stack-windows.yaml --no-haddock Build/BuildVersion.hs
+stack ghc --no-haddock Build/BuildVersion.hs
 ./Build/BuildVersion > dist/build-version
 
 # Test git-annex

Added a comment
diff --git a/doc/tips/ZSH_completion/comment_4_0e6f7035350ec0c9c61eeafb02ab7af7._comment b/doc/tips/ZSH_completion/comment_4_0e6f7035350ec0c9c61eeafb02ab7af7._comment
new file mode 100644
index 000000000..bc35bf98a
--- /dev/null
+++ b/doc/tips/ZSH_completion/comment_4_0e6f7035350ec0c9c61eeafb02ab7af7._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lykos153"
+ avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
+ subject="comment 4"
+ date="2017-11-09T17:35:14Z"
+ content="""
+Well I guess that depends on habit/workflow. I often let tab completion show me the directory content before beginning to type. Thanks for the links, I'll look into it (though I have no experience with Haskell yet).
+"""]]

add news item for git-annex 6.20171109
diff --git a/doc/news/version_6.20170818.mdwn b/doc/news/version_6.20170818.mdwn
deleted file mode 100644
index 388f36562..000000000
--- a/doc/news/version_6.20170818.mdwn
+++ /dev/null
@@ -1,49 +0,0 @@
-**Note** this is a security fix release. A prompt upgrade is strongly
-recommended. Attacks using this security hole will involve the attacker
-either providing a ssh repository url to the user, or the user pulling from
-a git-annex repository provided by an attacker and then running `git annex
-enableremote`. For details about the security hole, see
-[[bugs/dashed_ssh_hostname_security_hole]]. CVE-2017-12976
-
-git-annex 6.20170818 released with [[!toggle text="these changes"]]
-[[!toggleable text="""
-   * Security fix: Disallow hostname starting with a dash, which
-     would get passed to ssh and be treated an option. This could
-     be used by an attacker who provides a crafted repository url
-     to cause the victim to execute arbitrary code via -oProxyCommand.
-     (The same class of security hole recently affected git itself.)
-   * git-annex.cabal: Deal with breaking changes in Cabal 2.0.
-   * Fix build with QuickCheck 2.10.
-   * fsck: Support --json.
-   * move, copy: Support --batch.
-   * Added GIT\_ANNEX\_VECTOR\_CLOCK environment variable, which can be used to
-     override the default timestamps used in log files in the git-annex
-     branch. This is a dangerous environment variable; use with caution.
-   * Fix a git-annex test failure when run on NFS due to NFS lock files
-     preventing directory removal.
-   * test: Avoid most situations involving failure to delete test
-     directories, by forking a worker process and only deleting the test
-     directory once it exits.
-   * Disable http-client's default 30 second response timeout when HEADing
-     an url to check if it exists. Some web servers take quite a long time
-     to answer a HEAD request.
-   * Added remote configuration settings annex-ignore-command and
-     annex-sync-command, which are dynamic equivilants of the annex-ignore
-     and annex-sync configurations.
-   * Prevent spaces from being embedded in the name of new WORM keys,
-     as that handing spaces in keys would complicate things like the
-     external special remote protocol.
-   * migrate: WORM keys containing spaces will be migrated to not contain
-     spaces anymore.
-   * External special remotes will refuse to operate on keys with spaces in
-     their names. That has never worked correctly due to the design of the
-     external special remote protocol. Display an error message suggesting
-     migration.
-   * Fix incorrect external special remote documentation, which said that
-     the filename parameter to the TRANSFER command could not contain
-     spaces. It can in fact contain spaces. Special remotes implementors
-     that relied on that may need to fix bugs in their special remotes.
-   * Fix the external special remotes git-annex-remote-ipfs,
-     git-annex-remote-torrent and the example.sh template to correctly
-     support filenames with spaces.
-   * Windows: Win32 package has subsumed Win32-extras; update dependency."""]]
diff --git a/doc/news/version_6.20171109.mdwn b/doc/news/version_6.20171109.mdwn
new file mode 100644
index 000000000..ff4893c86
--- /dev/null
+++ b/doc/news/version_6.20171109.mdwn
@@ -0,0 +1,13 @@
+git-annex 6.20171109 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * Fix export of subdir of a branch.
+   * Fix exporting of non-annexed files to external special remotes.
+   * unlock, lock: Support --json.
+   * When there are multiple urls for a file, still treat it as being present
+     in the web when some urls don't work, as long as at least one url does
+     work.
+   * Makefile improvement for sudo make install.
+     Thanks, Eric Siegerman
+   * Makefile improvement for BUILDER=stack, use stack to run ghc.
+   * testremote: Test exporttree.
+   * Fix directory special remote's cleanup of empty export directories."""]]
\ No newline at end of file

reorder
diff --git a/doc/tips/ZSH_completion/comment_2_182c13292016baf46d8f53a306cb4b05._comment b/doc/tips/ZSH_completion/comment_5_182c13292016baf46d8f53a306cb4b05._comment
similarity index 100%
rename from doc/tips/ZSH_completion/comment_2_182c13292016baf46d8f53a306cb4b05._comment
rename to doc/tips/ZSH_completion/comment_5_182c13292016baf46d8f53a306cb4b05._comment

response
diff --git a/doc/tips/ZSH_completion/comment_2_182c13292016baf46d8f53a306cb4b05._comment b/doc/tips/ZSH_completion/comment_2_182c13292016baf46d8f53a306cb4b05._comment
new file mode 100644
index 000000000..9fd7b628f
--- /dev/null
+++ b/doc/tips/ZSH_completion/comment_2_182c13292016baf46d8f53a306cb4b05._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-09T16:11:10Z"
+ content="""
+Well, the code that generates the zsh completions could
+always be improved.
+<https://github.com/pcapriotti/optparse-applicative/issues>
+
+I don't really understand why completing options before filenames makes
+it unusable. It means that "git-annex tabtabtab" cycles through options
+first, but if you type part of the filename tab will complete the rest.
+It does seem that someone else agrees with you though, as they filed
+this issue: <https://github.com/pcapriotti/optparse-applicative/issues/173>
+"""]]

retitle; known bug
diff --git a/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn
index 442e17242..3d1b38556 100644
--- a/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn
+++ b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn
@@ -19,3 +19,5 @@ The only way to rectify this I can find is `git checkout` - but that means I hav
 ### What version of git-annex are you using? On what operating system?
 
 git-annex version: 6.20171026-gd451d333d (standalone binary) on Debian stretch.
+
+[[!meta title="v6 get/drop make git think files are changed"]]
diff --git a/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch/comment_1_d5e90cf5edf189bb023deb46b5ffe96c._comment b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch/comment_1_d5e90cf5edf189bb023deb46b5ffe96c._comment
new file mode 100644
index 000000000..d635819d7
--- /dev/null
+++ b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch/comment_1_d5e90cf5edf189bb023deb46b5ffe96c._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-09T15:56:41Z"
+ content="""
+This is one of the primary known problems with v6 mode. On the page
+[[todo/smudge]] that collects all such issues, it's the second item.
+Basically, when git-annex gets/drops the content of the file, it
+updates the work tree file, and so git thinks the file has changed.
+Currently the best way to deal with it is to run `git add` on the file.
+"""]]
diff --git a/stack.yaml b/stack.yaml
index d84c4682e..8f1f1dabd 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -21,6 +21,7 @@ extra-deps:
 - bloomfilter-2.0.1.0
 - torrent-10000.1.1
 - yesod-default-1.2.0
+- optparse-applicative-0.14.0.0
 explicit-setup-deps:
   git-annex: true
 resolver: lts-9.9

add links
diff --git a/doc/git-annex-test.mdwn b/doc/git-annex-test.mdwn
index 773f7404e..8d18f82e7 100644
--- a/doc/git-annex-test.mdwn
+++ b/doc/git-annex-test.mdwn
@@ -29,6 +29,8 @@ framework. Pass --help for details about those.
 
 [[git-annex]](1)
 
+[[git-annex-testremote]](1)
+
 # AUTHOR
 
 Joey Hess <id@joeyh.name>
diff --git a/doc/git-annex-testremote.mdwn b/doc/git-annex-testremote.mdwn
index a6793066c..df3c8ac50 100644
--- a/doc/git-annex-testremote.mdwn
+++ b/doc/git-annex-testremote.mdwn
@@ -36,6 +36,8 @@ and exporttree disabled and enabled.
 
 [[git-annex]](1)
 
+[[git-annex-test]](1)
+
 # AUTHOR
 
 Joey Hess <id@joeyh.name>

removed
diff --git a/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment b/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment
deleted file mode 100644
index 4e956f691..000000000
--- a/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="Lykos153"
- avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
- subject="comment 3"
- date="2017-11-09T02:15:25Z"
- content="""
-Doesn't work so well for me. The hand-build one performs way better in terms of clearness and efficiency. It groups the suggestions instead of sorting them by name and it prints a description for each command. But most importently, it does only suggest switches when `-` was typed and completes file names otherwise. The auto generated one (at least the one that is shipped with the Arch Linux package) puts the file names last, which makes it unusable.
-"""]]

removed
diff --git a/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment b/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment
deleted file mode 100644
index 4320617ce..000000000
--- a/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="Lykos153"
- avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
- subject="comment 2"
- date="2017-11-09T02:15:11Z"
- content="""
-Doesn't work so well for me. The hand-build one performs way better in terms clearness and efficiency. It groups the suggestions instead of sorting them by name and it prints a description for each command. But most importently, it does only suggest switches when `-` was typed and completes file names otherwise. The auto generated one (at least the one that is shipped with the Arch Linux package) puts the file names last, which makes it unusable.
-"""]]

Added a comment
diff --git a/doc/tips/ZSH_completion/comment_4_2bdaf6a00723b975808a52863e417321._comment b/doc/tips/ZSH_completion/comment_4_2bdaf6a00723b975808a52863e417321._comment
new file mode 100644
index 000000000..3c037688a
--- /dev/null
+++ b/doc/tips/ZSH_completion/comment_4_2bdaf6a00723b975808a52863e417321._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lykos153"
+ avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
+ subject="comment 4"
+ date="2017-11-09T02:15:37Z"
+ content="""
+Doesn't work so well for me. The hand-build one performs way better in terms of clearness and efficiency. It groups the suggestions instead of sorting them by name and it prints a description for each command. But most importently, it does only suggest switches when `-` was typed and completes file names otherwise. The auto generated one (at least the one that is shipped with the Arch Linux package) puts the file names last, which makes it unusable.
+"""]]

Added a comment
diff --git a/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment b/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment
new file mode 100644
index 000000000..4e956f691
--- /dev/null
+++ b/doc/tips/ZSH_completion/comment_3_28ebe919a65196d8abf4a5e192779912._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lykos153"
+ avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
+ subject="comment 3"
+ date="2017-11-09T02:15:25Z"
+ content="""
+Doesn't work so well for me. The hand-build one performs way better in terms of clearness and efficiency. It groups the suggestions instead of sorting them by name and it prints a description for each command. But most importently, it does only suggest switches when `-` was typed and completes file names otherwise. The auto generated one (at least the one that is shipped with the Arch Linux package) puts the file names last, which makes it unusable.
+"""]]

Added a comment
diff --git a/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment b/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment
new file mode 100644
index 000000000..4320617ce
--- /dev/null
+++ b/doc/tips/ZSH_completion/comment_2_a09e3adc117d1f0428378dda55c467f5._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="Lykos153"
+ avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
+ subject="comment 2"
+ date="2017-11-09T02:15:11Z"
+ content="""
+Doesn't work so well for me. The hand-build one performs way better in terms clearness and efficiency. It groups the suggestions instead of sorting them by name and it prints a description for each command. But most importently, it does only suggest switches when `-` was typed and completes file names otherwise. The auto generated one (at least the one that is shipped with the Arch Linux package) puts the file names last, which makes it unusable.
+"""]]

remove nix section to unbreak i386-ancient build
diff --git a/doc/bugs/Add_day_to_metadata./comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment b/doc/bugs/Add_day_to_metadata./comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment
new file mode 100644
index 000000000..73a5b5606
--- /dev/null
+++ b/doc/bugs/Add_day_to_metadata./comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-08T18:51:20Z"
+ content="""
+Unfortunately the added nix section broke the i386-ancient build,
+which uses stack 1.0.4. That version of stack complains:
+
+Executable named nix-shell not found on path: ["/usr/local/sbin","/usr/local/bin","/sbin","/bin","/usr/sbin","/usr/bin"]
+
+So, I've reverted the addition of that section.
+"""]]
diff --git a/stack.yaml b/stack.yaml
index ac601200e..d84c4682e 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -24,11 +24,3 @@ extra-deps:
 explicit-setup-deps:
   git-annex: true
 resolver: lts-9.9
-nix:
-  packages:
-    - ncurses
-    - icu
-    - libcxx
-    - gcc
-    - zlib
-    - rsync
\ No newline at end of file

testremote: Test exporttree.
As long as the class of remotes supports exporting, it's tested whether
or not the remote is configured with exporttree=yes.
Also, made testremote of a remote configured with exporttree=yes
disable that configuration for testing non-export storage.
This commit was supported by the NSF-funded DataLad project.
diff --git a/CHANGELOG b/CHANGELOG
index 358abb4f5..faa05e3d2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -9,6 +9,7 @@ git-annex (6.20171027) UNRELEASED; urgency=medium
   * Makefile improvement for sudo make install.
     Thanks, Eric Siegerman 
   * Makefile improvement for BUILDER=stack, use stack to run ghc.
+  * testremote: Test exporttree.
 
  -- Joey Hess <id@joeyh.name>  Mon, 30 Oct 2017 12:01:45 -0400
 
diff --git a/Command/TestRemote.hs b/Command/TestRemote.hs
index 8a21fdf35..75e438d79 100644
--- a/Command/TestRemote.hs
+++ b/Command/TestRemote.hs
@@ -21,6 +21,8 @@ import Utility.Metered
 import Utility.DataUnits
 import Utility.CopyFile
 import Types.Messages
+import Types.Export
+import Remote.Helper.Export
 import Remote.Helper.Chunked
 import Git.Types
 
@@ -57,21 +59,24 @@ seek o = commandAction $ start (fromInteger $ sizeOption o) (testRemote o)
 start :: Int -> RemoteName -> CommandStart
 start basesz name = do
 	showStart "testremote" name
-	r <- either giveup id <$> Remote.byName' name
-	showAction "generating test keys"
 	fast <- Annex.getState Annex.fast
-	ks <- mapM randKey (keySizes basesz fast)
+	r <- either giveup disableExportTree =<< Remote.byName' name
 	rs <- catMaybes <$> mapM (adjustChunkSize r) (chunkSizes basesz fast)
 	rs' <- concat <$> mapM encryptionVariants rs
 	unavailrs  <- catMaybes <$> mapM Remote.mkUnavailable [r]
-	next $ perform rs' unavailrs ks
+	exportr <- exportTreeVariant r
+	showAction "generating test keys"
+	ks <- mapM randKey (keySizes basesz fast)
+	next $ perform rs' unavailrs exportr ks
 
-perform :: [Remote] -> [Remote] -> [Key] -> CommandPerform
-perform rs unavailrs ks = do
+perform :: [Remote] -> [Remote] -> Maybe Remote -> [Key] -> CommandPerform
+perform rs unavailrs exportr ks = do
+	ea <- maybe exportUnsupported Remote.exportActions exportr
 	st <- Annex.getState id
 	let tests = testGroup "Remote Tests" $ concat
 		[ [ testGroup "unavailable remote" (testUnavailable st r (Prelude.head ks)) | r <- unavailrs ]
 		, [ testGroup (desc r k) (test st r k) | k <- ks, r <- rs ]
+		, [ testGroup (descexport k1 k2) (testExportTree st exportr ea k1 k2) | k1 <- take 2 ks, k2 <- take 2 (reverse ks) ]
 		]
 	ok <- case tryIngredients [consoleTestReporter] mempty tests of
 		Nothing -> error "No tests found!?"
@@ -83,6 +88,11 @@ perform rs unavailrs ks = do
 		, [ show (getChunkConfig (Remote.config r')) ]
 		, ["encryption", fromMaybe "none" (M.lookup "encryption" (Remote.config r'))]
 		]
+	descexport k1 k2 = intercalate "; " $ map unwords
+		[ [ "exporttree=yes" ]
+		, [ "key1 size", show (keySize k1) ]
+		, [ "key2 size", show (keySize k2) ]
+		]
 
 adjustChunkSize :: Remote -> Int -> Annex (Maybe Remote)
 adjustChunkSize r chunksize = adjustRemoteConfig r
@@ -98,6 +108,19 @@ encryptionVariants r = do
 		M.insert "highRandomQuality" "false"
 	return $ catMaybes [noenc, sharedenc]
 
+-- Variant of a remote with exporttree disabled.
+disableExportTree :: Remote -> Annex Remote
+disableExportTree r = maybe (error "failed disabling exportreee") return 
+		=<< adjustRemoteConfig r (M.delete "exporttree")
+
+-- Variant of a remote with exporttree enabled.
+exportTreeVariant :: Remote -> Annex (Maybe Remote)
+exportTreeVariant r = ifM (Remote.isExportSupported r)
+	( adjustRemoteConfig r $
+		M.insert "encryption" "none" . M.insert "exporttree" "yes"
+	, return Nothing
+	)
+
 -- Regenerate a remote with a modified config.
 adjustRemoteConfig :: Remote -> (Remote.RemoteConfig -> Remote.RemoteConfig) -> Annex (Maybe Remote)
 adjustRemoteConfig r adjustconfig = Remote.generate (Remote.remotetype r)
@@ -160,6 +183,50 @@ test st r k =
 	store = Remote.storeKey r k (AssociatedFile Nothing) nullMeterUpdate
 	remove = Remote.removeKey r k
 
+testExportTree :: Annex.AnnexState -> Maybe Remote -> Remote.ExportActions Annex -> Key -> Key -> [TestTree]
+testExportTree _ Nothing _ _ _ = []
+testExportTree st (Just _) ea k1 k2 =
+	[ check "check present export when not present" $
+		not <$> checkpresentexport k1
+	, check "remove export when not present" (removeexport k1)
+	, check "store export" (storeexport k1)
+	, check "check present export after store" $
+		checkpresentexport k1
+	, check "store export when already present" (storeexport k1)
+	, check "retrieve export" (retrieveexport k1)
+	, check "store new content to export" (storeexport k2)
+	, check "check present export after store of new content" $
+		checkpresentexport k2
+	, check "retrieve export new content" (retrieveexport k2)
+	, check "remove export" (removeexport k2)
+	, check "check present export after remove" $
+		not <$> checkpresentexport k2
+	, check "retrieve export fails after removal" $
+		not <$> retrieveexport k2
+	, check "remove export directory" removeexportdirectory
+	, check "remove export directory that is already removed" removeexportdirectory
+	-- renames are not tested because remotes do not need to support them
+	]
+  where
+	testexportdirectory = "testremote-export"
+	testexportlocation = mkExportLocation (testexportdirectory </> "location")
+	check desc a = testCase desc $
+		Annex.eval st (Annex.setOutput QuietOutput >> a) @? "failed"
+	storeexport k = do
+		loc <- Annex.calcRepo (gitAnnexLocation k)
+		Remote.storeExport ea loc k testexportlocation nullMeterUpdate
+	retrieveexport k = withTmpFile "exported" $ \tmp h -> do
+		liftIO $ hClose h
+		ifM (Remote.retrieveExport ea k testexportlocation tmp nullMeterUpdate)
+			( verifyKeyContent AlwaysVerify UnVerified k tmp
+			, return False
+			)
+	checkpresentexport k = Remote.checkPresentExport ea k testexportlocation
+	removeexport k = Remote.removeExport ea k testexportlocation
+	removeexportdirectory = case Remote.removeExportDirectory ea of
+		Nothing -> return True
+		Just a -> a (mkExportDirectory testexportdirectory)
+
 testUnavailable :: Annex.AnnexState -> Remote -> Key -> [TestTree]
 testUnavailable st r k =
 	[ check (== Right False) "removeKey" $
diff --git a/doc/git-annex-testremote.mdwn b/doc/git-annex-testremote.mdwn
index 5c3a5ca5b..a6793066c 100644
--- a/doc/git-annex-testremote.mdwn
+++ b/doc/git-annex-testremote.mdwn
@@ -19,7 +19,8 @@ tries to clean up after itself, if the remote being tested had a bug,
 the cleanup might fail, leaving test data in the remote.
 
 Testing will use the remote's configuration, automatically varying
-the chunk sizes, and with simple shared encryption enabled and disabled.
+the chunk sizes, and with simple shared encryption disabled and enabled,
+and exporttree disabled and enabled.
 
 # OPTIONS
 
diff --git a/doc/todo/Test_cases_for_exporttree_special_remotes.mdwn b/doc/todo/Test_cases_for_exporttree_special_remotes.mdwn
index 443369740..3d32b2c71 100644
--- a/doc/todo/Test_cases_for_exporttree_special_remotes.mdwn
+++ b/doc/todo/Test_cases_for_exporttree_special_remotes.mdwn
@@ -1 +1,3 @@
 As far as I can tell, `git annex testremote` doesn't test exporting yet
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/todo/Test_cases_for_exporttree_special_remotes/comment_2_0e280ec5691dbb0eef68f6e6c1424d08._comment b/doc/todo/Test_cases_for_exporttree_special_remotes/comment_2_0e280ec5691dbb0eef68f6e6c1424d08._comment
new file mode 100644
index 000000000..5cbb24e59
--- /dev/null
+++ b/doc/todo/Test_cases_for_exporttree_special_remotes/comment_2_0e280ec5691dbb0eef68f6e6c1424d08._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-08T17:38:02Z"
+ content="""
+Added some fairly comprehensive tests.
+"""]]

improve testremote docs
diff --git a/doc/git-annex-testremote.mdwn b/doc/git-annex-testremote.mdwn
index fff08fdcc..5c3a5ca5b 100644
--- a/doc/git-annex-testremote.mdwn
+++ b/doc/git-annex-testremote.mdwn
@@ -14,9 +14,12 @@ the remote, then redownloading them, removing them from the remote, etc.
 It's safe to run in an existing repository (the repository contents are
 not altered), although it may perform expensive data transfers.
 
-Testing a single remote will use the remote's configuration,
-automatically varying the chunk sizes, and with simple shared encryption
-enabled and disabled.
+It's best to make a new remote for testing purposes. While the test
+tries to clean up after itself, if the remote being tested had a bug,
+the cleanup might fail, leaving test data in the remote.
+
+Testing will use the remote's configuration, automatically varying
+the chunk sizes, and with simple shared encryption enabled and disabled.
 
 # OPTIONS
 

close already fixed bug (dup)
diff --git a/doc/bugs/Exporting_subdirs_fails.mdwn b/doc/bugs/Exporting_subdirs_fails.mdwn
index 12d465f62..9d21e0a94 100644
--- a/doc/bugs/Exporting_subdirs_fails.mdwn
+++ b/doc/bugs/Exporting_subdirs_fails.mdwn
@@ -20,3 +20,4 @@ ok
 ### What version of git-annex are you using? On what operating system?
 git-annex version: 6.20171026-g43d011a52 on Arch Linux
 
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/Exporting_subdirs_fails/comment_1_2113ddd317cd5d11cf1d7a56b30bb452._comment b/doc/bugs/Exporting_subdirs_fails/comment_1_2113ddd317cd5d11cf1d7a56b30bb452._comment
new file mode 100644
index 000000000..c3bddf2ff
--- /dev/null
+++ b/doc/bugs/Exporting_subdirs_fails/comment_1_2113ddd317cd5d11cf1d7a56b30bb452._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T21:08:48Z"
+ content="""
+This got fixed in [[!commit 24883e01cd487feb1be95146811d65e4f240465d]],
+the next release of git-annex will have the fix.
+"""]]

confirm
diff --git a/doc/todo/Test_cases_for_exporttree_special_remotes/comment_1_4db9a4b9cb94ba7ee460ee81fe52bf86._comment b/doc/todo/Test_cases_for_exporttree_special_remotes/comment_1_4db9a4b9cb94ba7ee460ee81fe52bf86._comment
new file mode 100644
index 000000000..18caa43f8
--- /dev/null
+++ b/doc/todo/Test_cases_for_exporttree_special_remotes/comment_1_4db9a4b9cb94ba7ee460ee81fe52bf86._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T21:07:00Z"
+ content="""
+That's right, exporttree is not tested yet.
+Thanks for the reminder.
+"""]]

response
diff --git a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_3_e0a2822d931a3129dde3e280a15cf500._comment b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_3_e0a2822d931a3129dde3e280a15cf500._comment
new file mode 100644
index 000000000..f124fe7f5
--- /dev/null
+++ b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_3_e0a2822d931a3129dde3e280a15cf500._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-11-07T21:02:59Z"
+ content="""
+There may be one url that works in one situation, and another that always
+works. Eg, a url on the work LAN, and a public, slower url. In such a
+situation, the user would not want fsck to remove an url that is
+temporarily not available.
+"""]]

followup
diff --git a/doc/install/fromsource/comment_68_059f1cd929228e131bf88d80aca0b573._comment b/doc/install/fromsource/comment_68_059f1cd929228e131bf88d80aca0b573._comment
new file mode 100644
index 000000000..e1f96f95e
--- /dev/null
+++ b/doc/install/fromsource/comment_68_059f1cd929228e131bf88d80aca0b573._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 68"""
+ date="2017-11-07T20:42:14Z"
+ content="""
+@erics, thanks for the useful feedback about stack. I've applied your
+patch to avoid the Build/InstallDesktopFile problem. And, I made it
+automatically set GHC when BUILDER=stack.
+
+Using the Makefile is documented, up in the "building from source on
+Debian" section. But there's a set of users who want to use stack,
+and don't want to mess around with Makefiles (including users on Windows
+without a "make"), and that's who the stack instructions are kind of
+targeted at. It's an unfortunate problem with stack that it doesn't provide
+any way to make the git-annex-shell symlink.
+
+I am doubtful that --allow-different-user is a good idea. I sometimes
+use stack to build git-annex for testing purposes, but I have never
+built it with stack and installed that with `sudo make install`.
+And it may well be that there's not a reasonable way to make that work;
+and the install target is mostly intended for use by distributions that
+are creating a package of git-annex, who probably set PREFIX and don't run
+it as root.
+"""]]

comment
diff --git a/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message/comment_1_2ec5e2dfe502b3357f7cb224bb28219a._comment b/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message/comment_1_2ec5e2dfe502b3357f7cb224bb28219a._comment
new file mode 100644
index 000000000..fba3a63f7
--- /dev/null
+++ b/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message/comment_1_2ec5e2dfe502b3357f7cb224bb28219a._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T20:39:22Z"
+ content="""
+Perhaps I'm blind as the one who wrote it, but I don't see anything
+unclear about the message.
+"""]]

simplify ikiwiki docs build testing and output
diff --git a/Makefile b/Makefile
index 6ac241f67..3d978df16 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,4 @@
-all=git-annex git-annex-shell mans docs Build/InstallDesktopFile
+all=git-annex git-annex-shell mans docs
 
 # set to "./Setup" if you lack a cabal program. Or can be set to "stack"
 BUILDER?=cabal
@@ -85,28 +85,26 @@ retest: git-annex
 tags:
 	(for f in $$(find . | grep -v /.git/ | grep -v /tmp/ | grep -v /dist/ | grep -v /doc/ | egrep '\.hs$$'); do hothasktags -c --cpp -c -traditional -c --include=dist/build/autogen/cabal_macros.h $$f; done) 2>/dev/null | sort > tags
 
-# If ikiwiki is available, build static html docs suitable for being
-# shipped in the software package.
-ifeq ($(shell which ikiwiki),)
-IKIWIKI=echo "** ikiwiki not found, skipping building docs" >&2; true
-else
-IKIWIKI=ikiwiki
-endif
-
 mans: Build/MakeMans
 	./Build/MakeMans
 
+# If ikiwiki is available, build static html docs suitable for being
+# shipped in the software package.
 docs: mans
-	LC_ALL=C TZ=UTC $(IKIWIKI) doc html -v --wikiname git-annex \
-		--plugin=goodstuff \
-		--no-usedirs --disable-plugin=openid --plugin=sidebar \
-		--plugin theme --set theme=actiontabs --set deterministic=1 \
-		--disable-plugin=shortcut --disable-plugin=smiley \
-		--plugin=comments --set comments_pagespec="*" \
-		--exclude='ikiwiki/*' \
-		--exclude='news/.*' --exclude='design/assistant/blog/*' \
-		--exclude='bugs/*' --exclude='todo/*' --exclude='forum/*' \
-		--exclude='users/*' --exclude='devblog/*' --exclude='thanks'
+	@if [ -n "`which ikiwiki`" ]; then \
+		LC_ALL=C TZ=UTC ikiwiki doc html -v --wikiname git-annex \
+			--plugin=goodstuff \
+			--no-usedirs --disable-plugin=openid --plugin=sidebar \
+			--plugin theme --set theme=actiontabs --set deterministic=1 \
+			--disable-plugin=shortcut --disable-plugin=smiley \
+			--plugin=comments --set comments_pagespec="*" \
+			--exclude='ikiwiki/*' \
+			--exclude='news/.*' --exclude='design/assistant/blog/*' \
+			--exclude='bugs/*' --exclude='todo/*' --exclude='forum/*' \
+			--exclude='users/*' --exclude='devblog/*' --exclude='thanks'; \
+	else \
+		echo "** ikiwiki not found, skipping building docs" >&2; \
+	fi
 
 clean:
 	if [ "$(BUILDER)" != ./Setup ] && [ "$(BUILDER)" != cabal ]; then $(BUILDER) clean; fi
diff --git a/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn
index 02fac5a6a..f2cea95f5 100644
--- a/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn
+++ b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn
@@ -40,3 +40,5 @@ No intended functional change; only what *make* prints should be different.
 			--plugin=goodstuff \
 			--no-usedirs --disable-plugin=openid --plugin=sidebar \
 			--plugin theme --set theme=actiontabs --set deterministic=1 \
+
+> [[done]] --[[Joey]]
diff --git a/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run/comment_1_b8304ff302805f7bec977002d733436a._comment b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run/comment_1_b8304ff302805f7bec977002d733436a._comment
new file mode 100644
index 000000000..31a670086
--- /dev/null
+++ b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run/comment_1_b8304ff302805f7bec977002d733436a._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T20:31:00Z"
+ content="""
+I don't like this amount of complication for a build system cosmetic
+improvement. Instead, I have added a "@" to the objectionably long
+command line in the Makefile.
+"""]]

Makefile improvement for sudo make install. Thanks, Eric Siegerman
diff --git a/CHANGELOG b/CHANGELOG
index 868db42af..ed5ab0c7f 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,6 +6,8 @@ git-annex (6.20171027) UNRELEASED; urgency=medium
   * When there are multiple urls for a file, still treat it as being present
     in the web when some urls don't work, as long as at least one url does
     work.
+  * Makefile improvement for sudo make install.
+    Thanks, Eric Siegerman 
 
  -- Joey Hess <id@joeyh.name>  Mon, 30 Oct 2017 12:01:45 -0400
 
diff --git a/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn b/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn
index 0ee250c6c..ef4e63ffa 100644
--- a/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn
+++ b/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn
@@ -22,3 +22,8 @@
 	 
 	 # set to "./Setup" if you lack a cabal program. Or can be set to "stack"
 	 BUILDER?=cabal
+
+> Applied [[done]]. Note that I had to do a considerable amount of editing to
+> get that in to a form that `git am` would accept. In the future,
+> providing a patch in a form that `git am` can use would be better.
+> --[[Joey]]

answer
diff --git a/doc/git-annex-adjust/comment_2_383af8e1e26e4119671437aace0a58f5._comment b/doc/git-annex-adjust/comment_2_383af8e1e26e4119671437aace0a58f5._comment
new file mode 100644
index 000000000..a0a270095
--- /dev/null
+++ b/doc/git-annex-adjust/comment_2_383af8e1e26e4119671437aace0a58f5._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-11-07T20:18:19Z"
+ content="""
+`git annex fix` fixes up the symlinks before they're committed.
+It's run by the pre-commit hook, so even when annexed files are
+manually moved around, the symlinks that get committed are always
+right.
+
+So then, if the symlinks committed are always right,
+how would `git annex adjust --fix` be useful? Well,
+there are ways to check out git repositories that make
+the .git directory not be in the usual place. For example,
+when using submodules, git puts that directory in a different place.
+And then the committed symlinks won't point to .git. So,
+`git annex adjust --fix` is useful as a way to adjust the symlinks
+locally, without committing any changes to them, in that kind of
+situation.
+"""]]

Added a comment
diff --git a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_2_d941d47c79ae9b72a936703aaba50ddd._comment b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_2_d941d47c79ae9b72a936703aaba50ddd._comment
new file mode 100644
index 000000000..ae6ac88e4
--- /dev/null
+++ b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_2_d941d47c79ae9b72a936703aaba50ddd._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="robert.schuetz@7942237bf71a2ae4f5d3cb047d167612b8c9e311"
+ nickname="robert.schuetz"
+ avatar="http://cdn.libravatar.org/avatar/89879460a9e84b9c736d982d9489d3d9"
+ subject="comment 2"
+ date="2017-11-07T20:22:04Z"
+ content="""
+If you want it that way, that's fine with me.
+But I think it would make more sense to let `fsck` call `rmurl` on all URLs that don't hold the content anymore.
+"""]]

Web.checkKey: Fix handling of multiple urls
When there are multiple urls for a file, still treat it as being present
in the web when some urls don't work, as long as at least one url does
work.
This is consistent with the other web methods handling of multiple urls.
This commit was sponsored by Ole-Morten Duesund on Patreon.
diff --git a/CHANGELOG b/CHANGELOG
index 06480e1ec..868db42af 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,6 +3,9 @@ git-annex (6.20171027) UNRELEASED; urgency=medium
   * Fix export of subdir of a branch.
   * Fix exporting of non-annexed files to external special remotes.
   * unlock, lock: Support --json.
+  * When there are multiple urls for a file, still treat it as being present
+    in the web when some urls don't work, as long as at least one url does
+    work.
 
  -- Joey Hess <id@joeyh.name>  Mon, 30 Oct 2017 12:01:45 -0400
 
diff --git a/Remote/Web.hs b/Remote/Web.hs
index f3580ca99..233c17eb3 100644
--- a/Remote/Web.hs
+++ b/Remote/Web.hs
@@ -119,8 +119,8 @@ checkKey' key us = firsthit us (Right False) $ \u -> do
 	firsthit (u:rest) _ a = do
 		r <- a u
 		case r of
-			Right _ -> return r
-			Left _ -> firsthit rest r a
+			Right True -> return r
+			_ -> firsthit rest r a
 
 getWebUrls :: Key -> Annex [URLString]
 getWebUrls key = filter supported <$> getUrls key
diff --git a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn
index 70bb8d06d..0e6f5d256 100644
--- a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn
+++ b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn
@@ -60,3 +60,4 @@ I'm running NixOS btw.
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 I'm lovin' it!
 
+> [[fixed|done]] --[[Joey]] 
diff --git a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_1_e1503374a87da976c831cd7fa0749e58._comment b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_1_e1503374a87da976c831cd7fa0749e58._comment
new file mode 100644
index 000000000..6c6957f60
--- /dev/null
+++ b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file/comment_1_e1503374a87da976c831cd7fa0749e58._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T19:58:14Z"
+ content="""
+I think that the real bug here is that, as long as one url still has the
+content, it's still present in the web, so should not be marked as not
+present. The rest of the operations in the web special remote work that
+way, trying urls until one succeeds, and so should `checkKey`.
+"""]]

link to git ML post
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment
index f6c6e5f4f..5cc9dc678 100644
--- a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment
@@ -22,6 +22,7 @@ It seems that git is looking at `$PWD` to get "/hd/repo" and normalizing
 the relative path via that. So there's no possible relative path
 that git will accept in this situation. This kind of seems like buggy
 behavior on git's part. I've posted about it to the git mailing list.
+<https://marc.info/?l=git&m=151008314122623&w=2>
 
 git-annex could avoid the problem by not using relative paths of course,
 but there are reasons including shorter path length (generally) 

followup
diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_3_24eadf45e4d8690470ed34686569a323._comment b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_3_24eadf45e4d8690470ed34686569a323._comment
new file mode 100644
index 000000000..ee761a7ce
--- /dev/null
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_3_24eadf45e4d8690470ed34686569a323._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-11-07T19:45:51Z"
+ content="""
+The sqlite failure does not seem to be due to the database getting too
+large or anything like that, because your loop does eventually manage to
+add all the files.
+
+Perhaps sqlite is failing intermittently for whatever reason..
+"""]]

followup
diff --git a/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_3_607d5b01eb95906391337f7351e0193c._comment b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_3_607d5b01eb95906391337f7351e0193c._comment
new file mode 100644
index 000000000..f2ba0e607
--- /dev/null
+++ b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_3_607d5b01eb95906391337f7351e0193c._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2017-11-07T19:35:17Z"
+ content="""
+Ah right, it looks in `uuid.log` to make sure the UUID provided is
+valid. Bit of a catch 22 there.
+
+Here's a way that will work:
+
+	git config remote.unknown.url foo
+	git config remote.unknown.annex-uuid 8bb266ed-453d-4489-9d8a-de38b2bc77c2
+	git annex dead unknown
+	git remote rm unknown
+"""]]

diff --git a/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn
new file mode 100644
index 000000000..70bb8d06d
--- /dev/null
+++ b/doc/bugs/git_annex_fsck_--from_web_removes_all_urls_of_a_file.mdwn
@@ -0,0 +1,62 @@
+### Please describe the problem.
+If `git annex fsck --from web` encounters a URL that is no longer available,
+it will remove all URLs from the appropriate file's location log.
+Surprisingly, adding one of the URLs the file was associated to before brings back all of them.
+
+### What steps will reproduce the problem?
+
+    $ git annex whereis file.txt
+    whereis file.txt (3 copies) 
+  	    ...
+
+      The following untrusted locations may also have copies:
+  	    00000000-0000-0000-0000-000000000001 -- web
+
+      web: http://example.org/dead/file.txt
+      web: http://example.org/alive/file.txt
+    ok
+    $ git annex fsck --from web
+    fsck file.txt (checking http://example.org/dead/file.txt...) (fixing location log)
+      ** Based on the location log, file.txt
+      ** was expected to be present, but its content is missing.
+    failed
+    (recording state in git...)
+    git-annex: fsck: 1 failed
+    $ git annex whereis file.txt
+    whereis file.txt (3 copies) 
+  	    ...
+    ok
+    $ git annex addurl http://example.org/alive/file.txt
+    addurl file.txt ok
+    (recording state in git...)
+    $ git annex whereis file.txt
+    whereis file.txt (3 copies) 
+  	    ...
+
+      The following untrusted locations may also have copies:
+  	    00000000-0000-0000-0000-000000000001 -- web
+
+      web: http://example.org/dead/file.txt
+      web: http://example.org/alive/file.txt
+    ok
+    
+
+The only fix that seems to fix this is using `rmurl` on the dead one.
+
+### What version of git-annex are you using? On what operating system?
+    $ git annex version
+    git-annex version: 6.20171026
+    build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+    dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+    local repository version: 5
+    supported repository versions: 3 5 6
+    upgrade supported from repository versions: 0 1 2 3 4 5
+    operating system: linux x86_64
+
+I'm running NixOS btw.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+I'm lovin' it!
+

analysis
Should add a link to my git ML post when it gets through.
This commit was sponsored by Bruno BEAUFILS on Patreon.
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment
new file mode 100644
index 000000000..f6c6e5f4f
--- /dev/null
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_5_cb5332119c54764b4ef1e000b3767b0e._comment
@@ -0,0 +1,35 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2017-11-07T18:21:20Z"
+ content="""
+With somerepo in ~/tmp/repo, and pwd in /hd/repo, which is a symlink from /tmp/hd/repo,
+this happens:
+
+	chat: git ["--git-dir=../../../home/joey/tmp/repo/.git","--work-tree=../../../home/joey/tmp/repo","--literal-pathspecs","--literal-pathspecs","cat-file","--batch"]
+	fatal: unable to normalize object directory: /hd/repo/../../../home/joey/tmp/repo/.git/objects
+
+Here it's making git operate on the remote git repo. The relative paths it uses
+to it seem legitimate, in that "ls ../.." shows the content of "/tmp" and
+"ls ../../.." shows the root directory.
+
+This is a very confusing situation, and different ways of getting the current
+directory give different results in this situation. In particular `$PWD`
+contains "/hd/repo" while getcwd(3) will return "/tmp/hd/repo". Paths
+need to be relative to "/tmp/hd/repo" as that's the *actual* cwd.
+
+It seems that git is looking at `$PWD` to get "/hd/repo" and normalizing
+the relative path via that. So there's no possible relative path
+that git will accept in this situation. This kind of seems like buggy
+behavior on git's part. I've posted about it to the git mailing list.
+
+git-annex could avoid the problem by not using relative paths of course,
+but there are reasons including shorter path length (generally) 
+for its use of relative paths. 
+
+git-annex could unset PWD before running git, which forces git to fall
+back to using getcwd(3). I've verified that does avoid the problem.
+Going to see if this just gets fixed in git before I put in such ugly
+and kind of expensive workarounds. (Expensive because, to unset PWD
+from the environment, git-annex would have to copy it.)
+"""]]

Added a comment: reply to comment 1
diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_2_74f3520d5eaecea35e52f3ffc2bbc559._comment b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_2_74f3520d5eaecea35e52f3ffc2bbc559._comment
new file mode 100644
index 000000000..8b00153de
--- /dev/null
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_2_74f3520d5eaecea35e52f3ffc2bbc559._comment
@@ -0,0 +1,31 @@
+[[!comment format=mdwn
+ username="webanck"
+ avatar="http://cdn.libravatar.org/avatar/cd273f76ef8c4218510b4f50ef7e1f3d"
+ subject="reply to comment 1"
+ date="2017-11-07T19:16:08Z"
+ content="""
+Hello joey, and thanks for the quick answer, I am honored !
+It's refreshing to enter an active community :)
+
+About the usage of \"bash on Windows\", well, git-annex had too many problems, especially with the symlinks, until the recent \"Windows 10 Fall Creators Update\".
+Now it goes pretty well.
+
+My main concern for the moment is the symlinks created in the \"bash\" can't be used by windows whereas symlinks created in windows can be used in \"bash\".
+Thus working in an annexed directory with Windows programs needs to unlock files. One note though, I didn't try with hardlinks.
+
+About the v6. I just figured that some minutes ago and, following <http://git-annex.branchable.com/tips/largefiles/>, created a .gitattributes file.
+
+	*.txt annex.largefiles=nothing
+
+It works better now.
+
+About the smudge/sqlite errors. I encountered those only after a big number of adds.
+I even managed to add all my files eventually with a little loop ignoring the errors (for 1000 files):
+
+	for i in {1..10}
+	do 
+	git add $(git ls-files --others | grep txt | head -100) 2> /dev/null
+	done
+
+
+"""]]

Added a comment
diff --git a/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_2_ddede3236f1d91ac4f0117b76267a021._comment b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_2_ddede3236f1d91ac4f0117b76267a021._comment
new file mode 100644
index 000000000..736f98ef6
--- /dev/null
+++ b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_2_ddede3236f1d91ac4f0117b76267a021._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="robert.schuetz@7942237bf71a2ae4f5d3cb047d167612b8c9e311"
+ nickname="robert.schuetz"
+ avatar="http://cdn.libravatar.org/avatar/89879460a9e84b9c736d982d9489d3d9"
+ subject="comment 2"
+ date="2017-11-07T19:14:18Z"
+ content="""
+Surprisingly, I can't use the UUID:
+
+    $ git annex dead 8bb266ed-453d-4489-9d8a-de38b2bc77c2
+    dead 8bb266ed-453d-4489-9d8a-de38b2bc77c2 git-annex: there is no available git remote named \"8bb266ed-453d-4489-9d8a-de38b2bc77c2\"
+"""]]

reroduced; reopen bug report
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory.mdwn b/doc/bugs/fatal__58___unable_to_normalize_object_directory.mdwn
index 9abfbd50c..b013ebbc9 100644
--- a/doc/bugs/fatal__58___unable_to_normalize_object_directory.mdwn
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory.mdwn
@@ -78,5 +78,3 @@ Linux XXX 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Li
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 Yes! I use git-annex without a problem since years. This is the first time I have really no idea what's going on.
-
-[[done]]
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_4_c4cf3df34625b17ed44077bec53b0ac4._comment b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_4_c4cf3df34625b17ed44077bec53b0ac4._comment
new file mode 100644
index 000000000..85e51a963
--- /dev/null
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_4_c4cf3df34625b17ed44077bec53b0ac4._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2017-11-07T18:19:39Z"
+ content="""
+Thanks for the reproducion recipe, @rfourquet. That worked for me.
+
+The bug submitter closed this bug, but I'm reopening it since this seems a
+legitimate bug report and we know how to reproduce it too.
+"""]]

response
diff --git a/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_1_85cc355213f894e8c1aeece432de67c3._comment b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_1_85cc355213f894e8c1aeece432de67c3._comment
new file mode 100644
index 000000000..1db8b0c19
--- /dev/null
+++ b/doc/forum/Remote_listed_in_whereis_but_not_in_info/comment_1_85cc355213f894e8c1aeece432de67c3._comment
@@ -0,0 +1,20 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T18:05:49Z"
+ content="""
+For this to happen, the repository must somehow not be listed in uuid.log,
+but the location log for the file includes its uuid. Normally, any time a
+git-annex repo is initialized, it gets recorded in uuid.log.
+
+One way that could have happened is if `git annex reinit` was run
+with some uuid that was not known in uuid.log. Or, if `git config
+annex.uuid` were manually set.
+
+I don't think it's a problem, other than you not knowing where this
+repository is -- unless some file is only present in that repository. 
+
+You can run `git annex dead` with the uuid as a parameter to mark it dead,
+or you could `git annex describe` with the uuid as a parameter to give
+it a description.
+"""]]

respond; retitle
diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
index 8a18fe441..3e3d749a1 100644
--- a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
@@ -59,3 +59,4 @@ Oh yeah, I am still discovering this powerfull git annex tool.
 
 In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other.
 
+[[!meta title="sqlite crash on Windows running linux binary via WSL"]]
diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_1_6f4179fe22d85b4a51e4aad60c4ff00d._comment b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_1_6f4179fe22d85b4a51e4aad60c4ff00d._comment
new file mode 100644
index 000000000..e22ea1d93
--- /dev/null
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes/comment_1_6f4179fe22d85b4a51e4aad60c4ff00d._comment
@@ -0,0 +1,27 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-11-07T17:48:28Z"
+ content="""
+There have been several bug reports with that same error message
+and different causes (permissions problems; filesystems on which sqlite
+doesn't work etc). So all that I know from the error is something went
+wrong with the sqlite database, which causes queries of it to fail.
+
+Using the linux version of git-annex on Winows via the (misleadingly named)
+"bash for windows" is pretty unusual. I have never seen it work; Microsoft
+were still adding missing linux system calls to their emulation layer the
+last time I tried it. It's quite possible that there remains a bug in that
+emulation layer, that prevents sqlite from working, or makes it unreliable,
+etc. I cannot reproduce the problem running git-annex on Linux.
+
+Note that since it's a v6 repository, running `git add` actually adds the
+files to git-annex. `git add` unfortunately runs `git-annex smudge` once
+per file. As you note, this is slow; this is one of the several reasons
+documented in [[todo/smudge]] why v6 mode is still considered experimental;
+and it will need changes in git to improve the speed).
+
+It would be useful to know if it's failing on the first file, 
+or if several files get processed ok before it begins to fail.
+You could set `GIT_TRACE=1` in the environment to find out.
+"""]]

diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
index dc6bda328..8a18fe441 100644
--- a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
@@ -52,6 +52,8 @@ Deleting git annex databases and running git annex fsck didnt do the trick:
 	git add $(git ls-files --others | grep txt)
 	# Again, plenty of sqlite errors :()
 
+It seems like a big overhead to add files tracked only by git in git annex repo. I know there are hooks/filters that catch and recover annexed files after modification but is it possible to disable these git annex hooks/filters when adding files that shouldn't be annexed ?
+
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 Oh yeah, I am still discovering this powerfull git annex tool. 
 

diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
index db2516267..dc6bda328 100644
--- a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
@@ -43,7 +43,7 @@ Create a git annex repo, add thousands of annexed binary files, and add thousand
 	# End of transcript or log.
 
 
-Triying to solve this problem, I found a part of answer in the form of a similar problem encounter here : <https://git-annex.branchable.com/forum/Problem_with_corrupt_SQLite_DB/>
+Triying to solve this problem, I found a part of answer in the form of a similar problem encountered here : <https://git-annex.branchable.com/forum/Problem_with_corrupt_SQLite_DB/>
 
 Deleting git annex databases and running git annex fsck didnt do the trick:
 	

diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
index dbf46a03b..db2516267 100644
--- a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
@@ -38,6 +38,8 @@ Create a git annex repo, add thousands of annexed binary files, and add thousand
 	error: external filter git-annex smudge --clean %f failed 1
 	error: external filter git-annex smudge --clean %f failed
 	# [...] plenty of errors follow
+	>git ls-files --others | grep txt | wc -l
+	1953
 	# End of transcript or log.
 
 

diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
new file mode 100644
index 000000000..dbf46a03b
--- /dev/null
+++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn
@@ -0,0 +1,57 @@
+### Please describe the problem.
+
+When adding plenty of files to my git annex repository, I encounter recurring sqlite errors.
+
+### What steps will reproduce the problem?
+Create a git annex repo, add thousands of annexed binary files, and add thousands of small files tracked only with git.
+
+
+### What version of git-annex are you using? On what operating system?
+
+	>git annex version
+	git-annex version: 6.20171003
+	build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+	dependency versions: aws-0.13.0 bloomfilter-2.0.1.0 cryptonite-0.10 DAV-1.2 feed-0.3.10.4 ghc-7.10.3 http-client-0.4.26.2 persistent-sqlite-2.2 torrent-10000.0.0 uuid-1.3.11 yesod-1.4.2
+	key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+	remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+	local repository version: 6
+	supported repository versions: 3 5 6
+	upgrade supported from repository versions: 0 1 2 3 4 5
+	operating system: linux x86_64
+
+	>lsb_release -d
+	Description:    Ubuntu 16.04.3 LTS
+
+	>uname -r
+	4.4.0-43-Microsoft
+	#(I am using Bash on Windows.)
+
+### Please provide any additional information below.
+
+
+	# If you can, paste a complete transcript of the problem occurring here.
+	>git ls-files --others | grep txt | wc -l
+	1953
+	>git add $(git ls-files --others | grep txt)
+	sqlite worker thread crashed: SQLite3 returned ErrorIO while attempting to perform prepare "SELECT null from content limit 1": disk I/O error
+	git-annex: sqlite query crashed
+	error: external filter git-annex smudge --clean %f failed 1
+	error: external filter git-annex smudge --clean %f failed
+	# [...] plenty of errors follow
+	# End of transcript or log.
+
+
+Triying to solve this problem, I found a part of answer in the form of a similar problem encounter here : <https://git-annex.branchable.com/forum/Problem_with_corrupt_SQLite_DB/>
+
+Deleting git annex databases and running git annex fsck didnt do the trick:
+	
+	rm -rf .git/annex/keys/db .git/annex/keys/db-wal
+	git annex fsck --incremental -J4
+	git add $(git ls-files --others | grep txt)
+	# Again, plenty of sqlite errors :()
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+Oh yeah, I am still discovering this powerfull git annex tool. 
+
+In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other.
+

update
diff --git a/doc/thanks/list b/doc/thanks/list
index 6a6938dfe..f9fc9ba3d 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -76,3 +76,6 @@ Silvio Ankermann,
 Paul Tötterman, 
 Erik Bjäreholt, 
 André Pereira, 
+Kanak Kshetri, 
+Josh, 
+tdittr, 

diff --git a/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn b/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn
index ef7b2caac..d7f7f18e0 100644
--- a/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn
+++ b/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn
@@ -2,7 +2,7 @@ I came across the following strange behaviour:
 
     $ git annex whereis "2013-WS/ecl/Algorithms for Scoring Coreference Chains.pdf"
     whereis 2013-WS/ecl/Algorithms for Scoring Coreference Chains.pdf (4 copies) 
-      	    04140d86-2ad5-4807-a789-f478dbf477c7 -- [mojzesz]
+   	    04140d86-2ad5-4807-a789-f478dbf477c7 -- [mojzesz]
    	    622fce61-6702-448f-8eee-9a31d8a67e14 -- here
    	    8bb266ed-453d-4489-9d8a-de38b2bc77c2
    	    d8149441-8b4d-4d37-bed4-c0f709165f32 -- [alonzo]

diff --git a/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn b/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn
new file mode 100644
index 000000000..ef7b2caac
--- /dev/null
+++ b/doc/forum/Remote_listed_in_whereis_but_not_in_info.mdwn
@@ -0,0 +1,29 @@
+I came across the following strange behaviour:
+
+    $ git annex whereis "2013-WS/ecl/Algorithms for Scoring Coreference Chains.pdf"
+    whereis 2013-WS/ecl/Algorithms for Scoring Coreference Chains.pdf (4 copies) 
+      	    04140d86-2ad5-4807-a789-f478dbf477c7 -- [mojzesz]
+   	    622fce61-6702-448f-8eee-9a31d8a67e14 -- here
+   	    8bb266ed-453d-4489-9d8a-de38b2bc77c2
+   	    d8149441-8b4d-4d37-bed4-c0f709165f32 -- [alonzo]
+    ok
+
+I have no idea what that remote without a name is. Is there a way to find that out?
+
+Plus, it is not shown by
+
+    $ git annex info
+    repository mode: indirect
+    trusted repositories: 0
+    semitrusted repositories: 3
+	    04140d86-2ad5-4807-a789-f478dbf477c7 -- [mojzesz]
+ 	    622fce61-6702-448f-8eee-9a31d8a67e14 -- here
+ 	    d8149441-8b4d-4d37-bed4-c0f709165f32 -- [alonzo]
+    untrusted repositories: 4
+	    00000000-0000-0000-0000-000000000001 -- web
+ 	    00000000-0000-0000-0000-000000000002 -- bittorrent
+ 	    11d4b299-0170-49b3-8b71-7ea2c47f212b -- nexus5
+ 	    dd22c018-65f8-4fa7-b880-48616016e272 -- miracle
+    ...
+
+Also, is there a way to mark that remote as dead?

Unchanged files are detected as modified
diff --git a/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn
new file mode 100644
index 000000000..442e17242
--- /dev/null
+++ b/doc/bugs/Unchanged_files_are_detected_as_modified_in_unlocked_branch.mdwn
@@ -0,0 +1,21 @@
+### Please describe the problem.
+
+I have a git-annex repo with lots of images and keep a unlocked v6 branch around to access them. When new files are added (i.e. after running `git annex sync --content`), they are often detected as "modified":
+
+    $ git annex status .
+    M image.jpg
+
+They are however identical to the checked-in files:
+
+    $ cp image.jpg image-changed.jpg
+    $ git checkout -- image.jpg
+    $ cmp image.jpg image-changed.jpg && echo "No change"
+    No change
+
+This seems to happen to older files from time to time as well, but I cannot reproduce that.
+
+The only way to rectify this I can find is `git checkout` - but that means I have no way to know whether I am actually throwing away changes. It also has the unfortunate side effect of changing the mtime, leading to previews having to be regenerated.
+
+### What version of git-annex are you using? On what operating system?
+
+git-annex version: 6.20171026-gd451d333d (standalone binary) on Debian stretch.

Added a comment
diff --git a/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_3_c49df6a33bbf8c3238f5eec1ef3e8152._comment b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_3_c49df6a33bbf8c3238f5eec1ef3e8152._comment
new file mode 100644
index 000000000..3910a24ed
--- /dev/null
+++ b/doc/bugs/fatal__58___unable_to_normalize_object_directory/comment_3_c49df6a33bbf8c3238f5eec1ef3e8152._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="rfourquet"
+ avatar="http://cdn.libravatar.org/avatar/2c78d7b5b3c6a417e5d666666ec40d51"
+ subject="comment 3"
+ date="2017-11-05T12:04:50Z"
+ content="""
+I ran into the same problem: assuming I have a repo at \"/mnt/hd/repo\", then `$ cd /; sudo ln -s /mnt/hd/ .; cd /hd/repo; git annex copy --to=someremote ./some-file` exhibits the problem. Thanks!
+"""]]

Added a comment: tutorial up to date?
diff --git a/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_4_6610f58b031d587f07f05b31754e4d97._comment b/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_4_6610f58b031d587f07f05b31754e4d97._comment
new file mode 100644
index 000000000..6fa189137
--- /dev/null
+++ b/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_4_6610f58b031d587f07f05b31754e4d97._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="ynikitenko"
+ avatar="http://cdn.libravatar.org/avatar/168d629704097ddc596f75ca32a687a3"
+ subject="tutorial up to date?"
+ date="2017-11-03T20:23:12Z"
+ content="""
+When I follow the tutorial and run
+
+    git push origin master git-annex
+
+it replies
+
+    error: src refspec master does not match any.
+    error: failed to push some refs to 'vds:~/test.git'
+
+However one can use `git-annex sync` instead and that works fine.
+"""]]

Added a comment: Is it essential to clone from the server?
diff --git a/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_3_371f22ec58cf953c2b8abe3af71d3f91._comment b/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_3_371f22ec58cf953c2b8abe3af71d3f91._comment
new file mode 100644
index 000000000..c39263fa5
--- /dev/null
+++ b/doc/tips/centralized_git_repository_tutorial/on_your_own_server/comment_3_371f22ec58cf953c2b8abe3af71d3f91._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="ynikitenko"
+ avatar="http://cdn.libravatar.org/avatar/168d629704097ddc596f75ca32a687a3"
+ subject="Is it essential to clone from the server?"
+ date="2017-11-03T11:28:01Z"
+ content="""
+    Now on your laptop, clone the git repository from the server:
+
+Is it possible to init git-annex repo on your local machine, add git-remote there and push local data to the server? Won't there be any problems with this 'non-official' approach?
+"""]]

Fixed typo: changed "an" to "a" in "an new feature"
diff --git a/doc/tips/unlocked_files.mdwn b/doc/tips/unlocked_files.mdwn
index 3efae6ba0..68c7f20aa 100644
--- a/doc/tips/unlocked_files.mdwn
+++ b/doc/tips/unlocked_files.mdwn
@@ -31,7 +31,7 @@ had many problems of its own.
 
 ## enter v6 mode
 
-/!\ This is an new feature; see its [[todo_list|todo/smudge]]
+/!\ This is a new feature; see its [[todo_list|todo/smudge]]
 for known issues.
 
 This led to the v6 repository mode, which makes unlocked files remain

Fixed typo: changed "copy" to "copies" in "two copy"
diff --git a/doc/tips/antipatterns.mdwn b/doc/tips/antipatterns.mdwn
index 39c7e24b7..cd8ffaf1a 100644
--- a/doc/tips/antipatterns.mdwn
+++ b/doc/tips/antipatterns.mdwn
@@ -22,7 +22,7 @@ disk space, is a horrible idea. The general antipattern is:
     mv repoB/.git/annex repoB/.git/annex.bak
     ln -s repoA/.git/annex repoB/.git/annex
 
-This is bad because git-annex will believe it has two copy of the
+This is bad because git-annex will believe it has two copies of the
 files and then would let you drop the single copy, therefore leading
 to data loss.
 

Fixed typo: added "that" to "to require [that] more than one copy of a file exists"
diff --git a/doc/walkthrough/backups.mdwn b/doc/walkthrough/backups.mdwn
index b51a88794..223e0b2ee 100644
--- a/doc/walkthrough/backups.mdwn
+++ b/doc/walkthrough/backups.mdwn
@@ -1,4 +1,4 @@
-git-annex can be configured to require more than one copy of a file exists,
+git-annex can be configured to require that more than one copy of a file exists,
 as a simple backup for your data. This is controlled by the
 numcopies setting, which defaults to 1 copy. Let's
 change that to require 2 copies, and send a copy of every file

Fixed typo: added "to" to "the `--size` parameter can adjust it [to] test using smaller files"
diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn
index bfb2f08d8..087cf081b 100644
--- a/doc/special_remotes.mdwn
+++ b/doc/special_remotes.mdwn
@@ -102,7 +102,7 @@ as usual, and it then runs a lot of tests, using random data. It's
 particularly useful to test new implementations of special remotes.
 
 By default it will upload and download files of around 1MiB to the remote
-it tests; the `--size` parameter can adjust it test using smaller files.
+it tests; the `--size` parameter can adjust it to test using smaller files.
 
 It's safe to use this command even when you're already storing data in a
 remote; it won't touch your existing files stored on the remote.

FreeBSD port is in devel/*hs-*git-annex (the URL was correct; the text wasn't)
diff --git a/doc/install/FreeBSD.mdwn b/doc/install/FreeBSD.mdwn
index 72b402c38..358c02d5a 100644
--- a/doc/install/FreeBSD.mdwn
+++ b/doc/install/FreeBSD.mdwn
@@ -1,2 +1,2 @@
 git-annex is in FreeBSD ports in
-[devel/git-annex](http://www.freshports.org/devel/hs-git-annex/)
+[devel/hs-git-annex](http://www.freshports.org/devel/hs-git-annex/)

Fixed typo: "repisitory"
diff --git a/doc/future_proofing.mdwn b/doc/future_proofing.mdwn
index 71a1b4a09..a4d9ac24f 100644
--- a/doc/future_proofing.mdwn
+++ b/doc/future_proofing.mdwn
@@ -37,7 +37,7 @@ problem:
   another clone of the repository. Even if the filenames are lost,
   it's possible to [[tips/recover_data_from_lost+found]].
 
-* What about upgrades to the git-annex repisitory format?
+* What about upgrades to the git-annex repository format?
   git-annex supports [[upgrades]] from all previous repository versions,
   and will always support upgrading from all of them to any new versions.
   Note that the upgrade process needs to modify the content of the

Added a comment: difference with git-annex-fix?
diff --git a/doc/git-annex-adjust/comment_1_364d1d5546c0f448a4b27cfe69a59a07._comment b/doc/git-annex-adjust/comment_1_364d1d5546c0f448a4b27cfe69a59a07._comment
new file mode 100644
index 000000000..1e2d7a602
--- /dev/null
+++ b/doc/git-annex-adjust/comment_1_364d1d5546c0f448a4b27cfe69a59a07._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="ynikitenko"
+ avatar="http://cdn.libravatar.org/avatar/168d629704097ddc596f75ca32a687a3"
+ subject="difference with git-annex-fix?"
+ date="2017-11-02T19:57:48Z"
+ content="""
+From git-annex-fix manual:
+
+       Fixes up symlinks that have become broken to again point to annexed content.
+       This is useful to run manually when you have been moving the symlinks around, but is done automatically when committing a change with git too.
+       Also, adjusts unlocked files to be copies or hard links as configured by annex.thin.
+
+So what is the difference between that and git-annex adjust --fix?
+"""]]

Added a comment: Works
diff --git a/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__/comment_1_cb691410d2a9ef4393af5362000a768c._comment b/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__/comment_1_cb691410d2a9ef4393af5362000a768c._comment
new file mode 100644
index 000000000..dfe735e61
--- /dev/null
+++ b/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__/comment_1_cb691410d2a9ef4393af5362000a768c._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="ynikitenko"
+ avatar="http://cdn.libravatar.org/avatar/168d629704097ddc596f75ca32a687a3"
+ subject="Works"
+ date="2017-11-02T19:21:43Z"
+ content="""
+Surprisingly, these very same repositories work today. I don't know why this happened. Seems that the commands I wrote above work.
+
+I've also found a useful post with various git-annex usage cases (basically the one descibed in this post).
+
+[http://ewen.mcneill.gen.nz/blog/entry/2014-11-15-git-annex-to-manage-media-backups/](http://ewen.mcneill.gen.nz/blog/entry/2014-11-15-git-annex-to-manage-media-backups/)
+"""]]

diff --git a/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message.mdwn b/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message.mdwn
new file mode 100644
index 000000000..79d6e3fa7
--- /dev/null
+++ b/doc/bugs/__91__PATCH__93___Cosmetic__58___clarify_a_warning_message.mdwn
@@ -0,0 +1,22 @@
+	commit 3ee8dc86cd831e975c80844924ef062b79e763b6
+	Author: Eric Siegerman <pub08-git@davor.org>
+	Date:   Tue Oct 31 21:12:38 2017 -0400
+
+	    Make a Makefile warning ... more obviously only a warning
+
+	diff --git a/Makefile b/Makefile
+	index aceb65cae..0381e7383 100644
+	--- a/Makefile
+	+++ b/Makefile
+	@@ -34,7 +34,10 @@ git-annex: tmp/configure-stamp
+	 # Work around https://github.com/haskell/cabal/issues/3524
+	 # when not linked dynamically to haskell libs
+		@if ! ldd git-annex | grep -q libHS; then \
+	-		chrpath -d git-annex || echo "** unable to chrpath git-annex; it will be a little bit slower than necessary"; \
+	+		chrpath -d git-annex || { \
+	+			echo "** warning: unable to chrpath git-annex; it will run OK..."; \
+	+			echo "** ... but maybe a little bit slower than necessary"; \
+	+		} \
+		fi
+	 
+	 git-annex-shell: git-annex

diff --git a/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn
new file mode 100644
index 000000000..02fac5a6a
--- /dev/null
+++ b/doc/bugs/__91__PATCH__93___Cosmetic__58___only_print_ikiwiki_command_if_it__39__s_run.mdwn
@@ -0,0 +1,42 @@
+No intended functional change; only what *make* prints should be different.
+
+	commit bb43afb0d70311dc9fd7633133c3c4fec32511e6
+	Author: Eric Siegerman <pub08-git@davor.org>
+	Date:   Tue Oct 31 02:33:13 2017 -0400
+
+	    Refactor "make docs" to eliminate confusing output
+	    
+	    The the many lines of arguments to the ikiwiki command would
+	    always be printed -- even if ikiwiki was unavailable.  Now
+	    you'll only see them if they're accomplishing something.
+
+	diff --git a/Makefile b/Makefile
+	index aceb65cae..121b19a99 100644
+	--- a/Makefile
+	+++ b/Makefile
+	@@ -88,16 +88,21 @@ tags:
+	 # If ikiwiki is available, build static html docs suitable for being
+	 # shipped in the software package.
+	 ifeq ($(shell which ikiwiki),)
+	-IKIWIKI=echo "** ikiwiki not found, skipping building docs" >&2; true
+	+BUILD_DOCS=_skip_building_docs
+	 else
+	-IKIWIKI=ikiwiki
+	+BUILD_DOCS = _build_docs
+	 endif
+	 
+	 mans: Build/MakeMans
+		./Build/MakeMans
+	 
+	-docs: mans
+	-	LC_ALL=C TZ=UTC $(IKIWIKI) doc html -v --wikiname git-annex \
+	+docs: mans $(BUILD_DOCS)
+	+
+	+_skip_building_docs:
+	+	@echo "** ikiwiki not found, skipping building docs" >&2
+	+
+	+_build_docs:
+	+	LC_ALL=C TZ=UTC ikiwiki doc html -v --wikiname git-annex \
+			--plugin=goodstuff \
+			--no-usedirs --disable-plugin=openid --plugin=sidebar \
+			--plugin theme --set theme=actiontabs --set deterministic=1 \

diff --git a/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn b/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn
new file mode 100644
index 000000000..0ee250c6c
--- /dev/null
+++ b/doc/bugs/__91__PATCH__93___Build_Build__47__InstallDesktopFile_at___34__make_all__34___time.mdwn
@@ -0,0 +1,24 @@
+	commit 69138285fd4671855184a2de68e1b99aa0a4f3a8
+	Author: Eric Siegerman <pub08-git@davor.org>
+	Date:   Tue Oct 31 02:17:27 2017 -0400
+
+	    Build Build/InstallDesktopFile at "make all" time
+	    
+	    If you run stack as root (e.g. for "make install"), any files it
+	    creates under ./ will, of course, be owned by root.  That's a
+	    problem for subsequent runs as non-root.
+	    
+	    Reduce the likelihood of that happening by building
+	    Build/InstallDesktopFile during "make all", so that it needn't be
+	    built by "make install".
+
+	diff --git a/Makefile b/Makefile
+	index aceb65cae..6ac241f67 100644
+	--- a/Makefile
+	+++ b/Makefile
+	@@ -1,4 +1,4 @@
+	-all=git-annex git-annex-shell mans docs
+	+all=git-annex git-annex-shell mans docs Build/InstallDesktopFile
+	 
+	 # set to "./Setup" if you lack a cabal program. Or can be set to "stack"
+	 BUILDER?=cabal

Added a comment: Woops, just found the "contributing" page...
diff --git a/doc/install/fromsource/comment_67_5655a80d206bd1f7cf1f326c645df7ed._comment b/doc/install/fromsource/comment_67_5655a80d206bd1f7cf1f326c645df7ed._comment
new file mode 100644
index 000000000..beecf6922
--- /dev/null
+++ b/doc/install/fromsource/comment_67_5655a80d206bd1f7cf1f326c645df7ed._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="erics"
+ avatar="http://cdn.libravatar.org/avatar/6e5f74c742128e5d98fd672ed6ea4865"
+ subject="Woops, just found the &quot;contributing&quot; page..."
+ date="2017-11-01T19:33:08Z"
+ content="""
+... so I have the answer to my question about how to submit patches.
+"""]]

Added a comment: Adventures with stack
diff --git a/doc/install/fromsource/comment_66_849c4c2cfc34991c13d55495e391cce3._comment b/doc/install/fromsource/comment_66_849c4c2cfc34991c13d55495e391cce3._comment
new file mode 100644
index 000000000..3f7dcacad
--- /dev/null
+++ b/doc/install/fromsource/comment_66_849c4c2cfc34991c13d55495e391cce3._comment
@@ -0,0 +1,78 @@
+[[!comment format=mdwn
+ username="erics"
+ avatar="http://cdn.libravatar.org/avatar/6e5f74c742128e5d98fd672ed6ea4865"
+ subject="Adventures with stack"
+ date="2017-11-01T19:19:51Z"
+ content="""
+I've played with git-annex before, but never used it seriously.  Building it from source for the first time, to get the latest and greatest, I ran into a few stumbling blocks.  I've got them sorted out now; this is to (a) document them, and (b) ask which if any of these should be considered bugs -- in which case I'll file bug reports about them.
+
+I'm a complete Haskell newbie, and don't want to dive into that right now, so I'm using *stack*, since I like the sound of
+> Using stack automates nearly everything, will work on many systems, and avoids build failures due to fast-changing haskell libraries.
+
+I'm working with Ubuntu 16.04 LTS, stack 1.5.1, ghc 8.0.2 (courtesy of stack), and git-annex 6.20171026.
+
+So, here's how things went.
+
+*Almost* following the *stack* section of [[install/fromsource]], after `stack setup` I did `stack build` instead of`stack install`, so that I could (a) run the tests before installing, and (b) install into /usr/local as root.
+
+Issue #1: **`stack test` doesn't run the tests**.  It took me a while to find `git annex test`.  This isn't a bug, just something for people to be aware of.
+
+Issue #2: **`stack build` doesn't create the *git-annex* and *git-annex-shell* symlinks**.  Which means that the *ssh remote* tests fail.
+
+Issue #3: **Unless they don't**.  If *./git-annex-shell* doesn't exist, the tests use whatever copy they find on $PATH.  Which lets them run -- but it's not testing the code you think it's testing.  This could lead to either false negatives or false positives under different circumstances.
+
+I'd noticed the Makefile, but dismissed it after a failed attempt to set `BUILDER=stack`, especially since it's undocumented.  Turns out it does important stuff -- like creating those symlinks.  So I dug more deeply.  The magic incantation is:
+
+	make BUILDER=stack GHC=\"stack ghc --\" [target]
+
+The \"--\" is the unobvious bit.
+
+There's even a `make test`.
+
+Issue #4: The Makefile does good and useful things, but **I can find no mention of it in [[install/fromsource]]**.
+
+OK, so at this point I've got the thing built, and the tests passing (legitimately, using the correct version of *git-annex-shell*).  Time to install it.
+
+Issue #5: **`sudo make install` doesn't work**.  *stack* refuses to run, complaining:
+
+> You are not the owner of '/home/erics/.stack/'. Aborting to protect file permissions.
+> Retry with '--allow-different-user' to disable this precaution.
+
+See <https://github.com/commercialhaskell/stack/issues/471> for what this is about.
+
+Three ways to add that option:
+
+1. Hack the Makefile -- add another `if [ $BUILDER = stack ]` section
+
+2. Add this line to git-annex's *stack.yaml* (or to *~/.stack/config.yaml* if you want it as a global setting):
+
+       allow-different-user: true
+
+   (It's a standalone entry, so should have *no* leading whitespace)
+
+3. For installing, use a variant of the above magic incantation:
+
+	  sudo make BUILDER=\"stack --allow-different-user\" GHC=\"stack ghc --\" install
+
+I did #2, but #3 is probably safest, since it means you're only allow(ing)-different-user when you actually need to.
+
+However you do it, be aware that if *stack* decides, at make-install time, to create any files under the source tree, of course it will create them as root, so subsequent builds as non-root will find them unwritable.  I presume that's what stack's error message, \"... to protect file permissions\", is talking about.
+
+Issue #6: **`make install` does indeed cause stack to create more files**.  Specifically, it builds *Build/InstallDesktopFile*.  It would make sense to build that at `make all` time, so that it gets done as the non-root user. This wouldn't guarantee not to pollute the source tree with root-owned files -- but as it is now, that's pretty much guaranteed *to* happen.
+
+After all those gripes, one very very good thing: I didn't have to apt-get any Haskell things at all!  Stack downloaded everything it needed, including the compiler.  Nice!
+
+Questions for @Joey:
+
+A. Is the Makefile undocumented because you don't really intend anyone else to use it, or is that just an oversight?
+
+B. Are any of the above issues legitimate bugs?  (Prime candidates, to my mind: #3 and #6)
+
+C. Given all the dependency grief in the comments from Cabal users, and that for all my troubles I experienced *none* of that ... maybe *stack* should be the recommended approach, even if you don't use it yourself?
+
+D. What's the preferred way to submit patches?  I have a couple of cosmetic ones for Makefile, besides the (trivial) fix for #6
+
+I'll edit the *stack* section of [[install/fromsource]] (the only part I'm competent to talk about), once I know which direction to go (i.e. the answer to (A)).
+
+
+"""]]

grammar...
diff --git a/doc/bugs/Exporting_subdirs_fails.mdwn b/doc/bugs/Exporting_subdirs_fails.mdwn
index dd16ede85..12d465f62 100644
--- a/doc/bugs/Exporting_subdirs_fails.mdwn
+++ b/doc/bugs/Exporting_subdirs_fails.mdwn
@@ -1,5 +1,5 @@
 ### Please describe the problem.
-git annex won't export subdirectories claiming they don't exist. `git rev-parse` however, returns a hash for said directories which then are accepted by `git annex export`.
+git annex won't export subdirectories claiming they don't exist. `git rev-parse` however, returns hashes for said directories which then are accepted by `git annex export`.
 
 
 ### What steps will reproduce the problem?

diff --git a/doc/bugs/Exporting_subdirs_fails.mdwn b/doc/bugs/Exporting_subdirs_fails.mdwn
new file mode 100644
index 000000000..dd16ede85
--- /dev/null
+++ b/doc/bugs/Exporting_subdirs_fails.mdwn
@@ -0,0 +1,22 @@
+### Please describe the problem.
+git annex won't export subdirectories claiming they don't exist. `git rev-parse` however, returns a hash for said directories which then are accepted by `git annex export`.
+
+
+### What steps will reproduce the problem?
+
+[[!format sh """
+$ git annex export master:folder --to exp
+fatal: Path 'folder:' does not exist in 'master'
+git-annex: unknown tree
+$ git rev-parse master:folder
+943218c141ab2a315b533fa9f92fe610916b5d1b
+$ git annex export 943218c141ab2a315b533fa9f92fe610916b5d1b --to exp
+export exp testfile2 
+ok                          
+(recording state in git...)
+"""]]
+
+
+### What version of git-annex are you using? On what operating system?
+git-annex version: 6.20171026-g43d011a52 on Arch Linux
+

rename todo/Test_cases_for_exportree_special_remotes.mdwn to todo/Test_cases_for_exporttree_special_remotes.mdwn
diff --git a/doc/todo/Test_cases_for_exportree_special_remotes.mdwn b/doc/todo/Test_cases_for_exporttree_special_remotes.mdwn
similarity index 100%
rename from doc/todo/Test_cases_for_exportree_special_remotes.mdwn
rename to doc/todo/Test_cases_for_exporttree_special_remotes.mdwn

Fix type exportree
diff --git a/doc/git-annex-export.mdwn b/doc/git-annex-export.mdwn
index 4807a6434..0a5a5516a 100644
--- a/doc/git-annex-export.mdwn
+++ b/doc/git-annex-export.mdwn
@@ -59,7 +59,7 @@ And, git-annex will never trust an export to retain the content of a key.
 # EXAMPLE
 
 	git annex initremote myexport type=directory directory=/mnt/myexport \
-		exportree=yes encryption=none
+		exporttree=yes encryption=none
 	git annex export master --to myexport
 
 After that, /mnt/myexport will contain the same tree of files as the master

diff --git a/doc/todo/Test_cases_for_exportree_special_remotes.mdwn b/doc/todo/Test_cases_for_exportree_special_remotes.mdwn
new file mode 100644
index 000000000..443369740
--- /dev/null
+++ b/doc/todo/Test_cases_for_exportree_special_remotes.mdwn
@@ -0,0 +1 @@
+As far as I can tell, `git annex testremote` doesn't test exporting yet

diff --git a/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__.mdwn b/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__.mdwn
new file mode 100644
index 000000000..9168bc5ef
--- /dev/null
+++ b/doc/forum/git-annex_for_multiple_repositories___40__ssh_server__41__.mdwn
@@ -0,0 +1,73 @@
+For git-annex version 6 with 'thin'. [I want 1) not store date twice 2) to be able to have self-sufficient repository at different machines (i.e. not special remotes, which don't contain the git tree)].
+
+Say I've two machines A and B and the server S. 
+
+I setup a local repository on A:
+
+    git init
+    git-annex init $HOSTNAME --version=6
+    git config annex.thin true
+    git add .
+
+on S:
+
+    mkdir test
+    cd test
+    git init
+    git-annex init $HOSTNAME --version=6
+    git config annex.thin true
+
+on A:
+
+    git remote add S:~/test.git
+    git-annex sync --content
+
+on B:
+
+    git clone S:~/test.git
+    cd test
+    git-annex init $HOSTNAME --version=6
+    git config annex.thin true
+    git-annex sync
+
+(the reply)
+
+    ...
+    merge: refs/remotes/origin/master - not something we can merge
+
+    failed
+    push origin 
+    Counting objects: 6, done.
+    Delta compression using up to 4 threads.
+    Compressing objects: 100% (5/5), done.
+    Writing objects: 100% (6/6), 714 bytes | 0 bytes/s, done.
+    Total 6 (delta 1), reused 1 (delta 0)
+    remote: git-annex: unknown command post-receive
+    ...
+
+and the file contains only '/annex/objects/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'.
+
+Another time with another repo which I tried to sync from S at B, I got
+
+    $ git-annex sync --content
+    Initial commit
+
+    Untracked files: 
+(all my files untracked)
+
+    merge: refs/remotes/origin/master - not something we can merge
+
+    error: Untracked working tree file 'applications/algorithms/file.pdf' would be overwritten by merge.
+    fatal: read-tree failed
+    failed
+    git-annex: sync: 1 failed
+
+On walkthrough it's written how to sync with USB. How should one initialize remote repositories and use them e.g. as central repositories (though they should work probably not depending on 'central' or not they are)? 
+
+I've read a lot of manuals on git-annex and the forum; the questions here remain unanswered: 
+
+<http://git-annex.branchable.com/forum/git-annex_SSH_server_+_cloud_remote/>
+
+<http://git-annex.branchable.com/forum/Synchronize_two_latops_with_a_ssh_remote/>
+
+I hope this is a basic and popular usage of git-annex and that one can write a howto for that or at least answer this question.

response
diff --git a/doc/forum/How_to_sync_without_enumerating_files__63__/comment_1_ad78c67ae1bb4dc8ed800779c0298158._comment b/doc/forum/How_to_sync_without_enumerating_files__63__/comment_1_ad78c67ae1bb4dc8ed800779c0298158._comment
new file mode 100644
index 000000000..f79b3592f
--- /dev/null
+++ b/doc/forum/How_to_sync_without_enumerating_files__63__/comment_1_ad78c67ae1bb4dc8ed800779c0298158._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-10-31T17:15:33Z"
+ content="""
+Well, `git annex sync` does not itself run `git status`. 
+
+Try running git-annex with the --debug option, it will say what git
+commands it runs, and that should help find out which git command is
+complaining.
+
+My guess is it's `git commit`, which  is probably getting the status
+unncessarily for a commit message template, despite a commit message
+being specified. If so, `git annex sync --no-commit` would avoid
+that.
+"""]]

close bug; copy benchmarking info to new page
diff --git a/doc/benchmarking.mdwn b/doc/benchmarking.mdwn
new file mode 100644
index 000000000..68fc328d0
--- /dev/null
+++ b/doc/benchmarking.mdwn
@@ -0,0 +1,3 @@
+This page exsists to collect benchmarking data about git-annex, so it can
+be referred to later. If you have a specific instance where git-annex seems
+unncessarily slow, please file a bug report about it.
diff --git a/doc/benchmarking/comment_10_1af4ac0d37c876912678522895c1656b._comment b/doc/benchmarking/comment_10_1af4ac0d37c876912678522895c1656b._comment
new file mode 100644
index 000000000..868b10364
--- /dev/null
+++ b/doc/benchmarking/comment_10_1af4ac0d37c876912678522895c1656b._comment
@@ -0,0 +1,61 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2016-09-29T18:33:33Z"
+ content="""
+* Optimised key2file and file2key. 18% scanning time speedup.
+* Optimised adjustGitEnv. 50% git-annex branch query speedup
+* Optimised parsePOSIXTime. 10% git-annex branch query speedup
+* Tried making catObjectDetails.receive use ByteString for parsing, 
+  but that did not seem to speed it up significantly.
+  So it parsing is already fairly optimal, it's just that a
+  lot of data passes through it when querying the git-annex
+  branch.
+
+After all that, profiling `git-annex find`:
+
+	        Thu Sep 29 16:51 2016 Time and Allocation Profiling Report  (Final)
+	
+	           git-annex.1 +RTS -p -RTS find
+	
+	        total time  =        1.73 secs   (1730 ticks @ 1000 us, 1 processor)
+	        total alloc = 1,812,406,632 bytes  (excludes profiling overheads)
+	
+	COST CENTRE            MODULE                  %time %alloc
+	
+	md5                    Data.Hash.MD5            28.0   37.9
+	catchIO                Utility.Exception        10.2   12.5
+	inAnnex'.checkindirect Annex.Content             9.9    3.7
+	catches                Control.Monad.Catch       8.7    5.7
+	readish                Utility.PartialPrelude    5.7    3.0
+	isAnnexLink            Annex.Link                5.0    8.4
+	keyFile                Annex.Locations           4.2    5.8
+	spanList               Data.List.Utils           4.0    6.3
+	startswith             Data.List.Utils           2.0    1.3
+
+And `git-annex find --not --in web`:
+
+	        Thu Sep 29 16:35 2016 Time and Allocation Profiling Report  (Final)
+	
+	           git-annex +RTS -p -RTS find --not --in web
+	
+	        total time  =        5.24 secs   (5238 ticks @ 1000 us, 1 processor)
+	        total alloc = 3,293,314,472 bytes  (excludes profiling overheads)
+	
+	COST CENTRE               MODULE                      %time %alloc
+	
+	catObjectDetails.receive  Git.CatFile                  12.9    5.5
+	md5                       Data.Hash.MD5                10.6   20.8
+	readish                   Utility.PartialPrelude        7.3    8.2
+	catchIO                   Utility.Exception             6.7    7.3
+	spanList                  Data.List.Utils               4.1    7.4
+	readFileStrictAnyEncoding Utility.Misc                  3.5    1.3
+	catches                   Control.Monad.Catch           3.3    3.2
+
+So, quite a large speedup overall!
+
+This leaves md5 still unoptimised at 10-28% of CPU use. I looked at switching
+it to cryptohash's implementation, but it would require quite a lot of
+bit-banging math to pull the used values out of the ByteString containing
+the md5sum.
+"""]]
diff --git a/doc/benchmarking/comment_11_1ca8d9765e6e3a18ae09df74bc390a00._comment b/doc/benchmarking/comment_11_1ca8d9765e6e3a18ae09df74bc390a00._comment
new file mode 100644
index 000000000..619351d4c
--- /dev/null
+++ b/doc/benchmarking/comment_11_1ca8d9765e6e3a18ae09df74bc390a00._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2017-05-15T21:56:52Z"
+ content="""
+Switched from MissingH to cryptonite for md5. It did move md5 out of the top CPU spot but
+the overall runtime didn't change much. Memory allocations did go down by a
+good amount.
+
+Updated profiles:
+
+	           git-annex +RTS -p -RTS find
+	
+	        total time  =        1.63 secs   (1629 ticks @ 1000 us, 1 processor)
+	        total alloc = 1,496,336,496 bytes  (excludes profiling overheads)
+	
+	COST CENTRE              MODULE                     SRC                                             %time %alloc
+	
+	catchIO                  Utility.Exception          Utility/Exception.hs:79:1-17                     14.1   15.1
+	inAnnex'.checkindirect   Annex.Content              Annex/Content.hs:(108,9)-(119,39)                10.6    4.8
+	catches                  Control.Monad.Catch        src/Control/Monad/Catch.hs:(432,1)-(436,76)       8.6    6.9
+	spanList                 Data.List.Utils            src/Data/List/Utils.hs:(150,1)-(155,36)           6.7   11.1
+	isAnnexLink              Annex.Link                 Annex/Link.hs:35:1-85                             5.0   10.2
+	keyFile                  Annex.Locations            Annex/Locations.hs:(456,1)-(462,19)               5.0    7.0
+	readish                  Utility.PartialPrelude     Utility/PartialPrelude.hs:(48,1)-(50,20)          3.8    2.0
+	startswith               Data.List.Utils            src/Data/List/Utils.hs:103:1-23                   3.6    2.3
+	splitc                   Utility.Misc               Utility/Misc.hs:(52,1)-(54,25)                    3.4    6.5
+	s2w8                     Data.Bits.Utils            src/Data/Bits/Utils.hs:65:1-15                    2.6    6.4
+	keyPath                  Annex.Locations            Annex/Locations.hs:(492,1)-(494,23)               2.5    4.4
+	fileKey.unesc            Annex.Locations            Annex/Locations.hs:(469,9)-(474,39)               2.0    3.5
+	copyAndFreeze            Data.ByteArray.Methods     Data/ByteArray/Methods.hs:(224,1)-(227,21)        1.8    0.5
+
+	           git-annex +RTS -p -RTS find --not --in web
+	
+	        total time  =        5.33 secs   (5327 ticks @ 1000 us, 1 processor)
+	        total alloc = 2,908,489,000 bytes  (excludes profiling overheads)
+	
+	COST CENTRE          MODULE                     SRC                                             %time %alloc
+	
+	catObjectDetails.\   Git.CatFile                Git/CatFile.hs:(80,72)-(88,97)                    7.8    2.8
+	catchIO              Utility.Exception          Utility/Exception.hs:79:1-17                      7.6    8.3
+	spanList             Data.List.Utils            src/Data/List/Utils.hs:(150,1)-(155,36)           5.8    9.1
+	readish              Utility.PartialPrelude     Utility/PartialPrelude.hs:(48,1)-(50,20)          4.5    4.0
+	parseResp            Git.CatFile                Git/CatFile.hs:(113,1)-(124,28)                   4.4    2.9
+	readFileStrict       Utility.Misc               Utility/Misc.hs:33:1-59                           3.7    1.6
+	catches              Control.Monad.Catch        src/Control/Monad/Catch.hs:(432,1)-(436,76)       3.1    3.6
+	encodeW8             Utility.FileSystemEncoding Utility/FileSystemEncoding.hs:(131,1)-(133,70)    3.1    2.3
+	
+"""]]
diff --git a/doc/benchmarking/comment_8_c1f99493f5e5c362d5c39f048280b11b._comment b/doc/benchmarking/comment_8_c1f99493f5e5c362d5c39f048280b11b._comment
new file mode 100644
index 000000000..e0f4987a0
--- /dev/null
+++ b/doc/benchmarking/comment_8_c1f99493f5e5c362d5c39f048280b11b._comment
@@ -0,0 +1,45 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""profiling"""
+ date="2016-09-26T19:20:36Z"
+ content="""
+Built git-annex with profiling, using `stack build --profile`
+
+(For reproduciblity, running git-annex in a clone of the git-annex repo
+https://github.com/RichiH/conference_proceedings with rev
+2797a49023fc24aff6fcaec55421572e1eddcfa2 checked out. It has 9496 annexed
+objects.)
+
+Profiling `git-annex find +RTS -p`:
+
+	        total time  =        3.53 secs   (3530 ticks @ 1000 us, 1 processor)
+	        total alloc = 3,772,700,720 bytes  (excludes profiling overheads)
+	
+	COST CENTRE            MODULE                  %time %alloc
+	
+	spanList               Data.List.Utils          32.6   37.7
+	startswith             Data.List.Utils          14.3    8.1
+	md5                    Data.Hash.MD5            12.4   18.2
+	join                   Data.List.Utils           6.9   13.7
+	catchIO                Utility.Exception         5.9    6.0
+	catches                Control.Monad.Catch       5.0    2.8
+	inAnnex'.checkindirect Annex.Content             4.6    1.8
+	readish                Utility.PartialPrelude    3.0    1.4
+	isAnnexLink            Annex.Link                2.6    4.0
+	split                  Data.List.Utils           1.5    0.8
+	keyPath                Annex.Locations           1.2    1.7
+
+
+This is interesting!
+
+Fully 40% of CPU time and allocations are in list (really String) processing,
+and the details of the profiling report show that `spanList` and `startsWith`
+and `join` are all coming from calls to `replace` in `keyFile` and `fileKey`.
+Both functions nest several calls to replace, so perhaps that could be unwound
+into a single pass and/or a ByteString used to do it more efficiently.
+
+12% of run time is spent calculating the md5 hashes for the hash
+directories for .git/annex/objects. Data.Hash.MD5 is from missingh, and
+it is probably a quite unoptimised version. Switching to the version
+if cryptonite would probably speed it up a lot.
+"""]]
diff --git a/doc/benchmarking/comment_9_f4d802a28b79905da0cb24af6cb65b0a._comment b/doc/benchmarking/comment_9_f4d802a28b79905da0cb24af6cb65b0a._comment
new file mode 100644
index 000000000..9692ad2d7
--- /dev/null
+++ b/doc/benchmarking/comment_9_f4d802a28b79905da0cb24af6cb65b0a._comment
@@ -0,0 +1,42 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""more profiling"""
+ date="2016-09-26T19:59:43Z"
+ content="""
+Instead of profiling `git annex copy --to remote`, I profiled `git annex
+find --not --in web`, which needs to do the same kind of location log lookup.
+
+	        total time  =       12.41 secs   (12413 ticks @ 1000 us, 1 processor)
+	        total alloc = 8,645,057,104 bytes  (excludes profiling overheads)
+	
+	COST CENTRE               MODULE                      %time %alloc

(Diff truncated)
Added a comment: thanks
diff --git a/doc/todo/make_copy_--fast__faster/comment_13_0f8e2127cea96c4f9609fa7599b1eec9._comment b/doc/todo/make_copy_--fast__faster/comment_13_0f8e2127cea96c4f9609fa7599b1eec9._comment
new file mode 100644
index 000000000..e52d1bc2f
--- /dev/null
+++ b/doc/todo/make_copy_--fast__faster/comment_13_0f8e2127cea96c4f9609fa7599b1eec9._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="thanks"
+ date="2017-10-30T20:04:48Z"
+ content="""
+I do think that things became smoother/faster since then.  I guess we could consider this one closed for now, and I will keep in mind that --from mode is faster.
+
+Cheers,
+"""]]

close; export
diff --git a/doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn b/doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn
index 25c27efbf..3c53d114c 100644
--- a/doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn
+++ b/doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn
@@ -7,3 +7,7 @@ TRANSFER-SUCCESS STORE Key URL
 response when upon STORE success special remote provides a url under which content should be registered available from.
 
 [[!meta author=yoh]]
+
+> I think this was trying to implement something like what `git annex
+> export`, and since that's implemented now, we shouldn't need to worry
+> about this. [[done]] --[[Joey]]

close
diff --git a/doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn b/doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn
index bd7fa8050..50444ff5f 100644
--- a/doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn
+++ b/doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn
@@ -57,3 +57,5 @@ $> git lg --stat git-annex | head -n 30
 """]]
 
 [[!meta author=yoh]]
+
+> Seems we're ok, so [[done]] --[[Joey]]

followup
diff --git a/doc/todo/make_annex_info_more_efficient/comment_8_90a22c51b19707ee0cecbe652c6ffa98._comment b/doc/todo/make_annex_info_more_efficient/comment_8_90a22c51b19707ee0cecbe652c6ffa98._comment
new file mode 100644
index 000000000..25e3cb9ad
--- /dev/null
+++ b/doc/todo/make_annex_info_more_efficient/comment_8_90a22c51b19707ee0cecbe652c6ffa98._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 8"""
+ date="2017-10-30T18:56:14Z"
+ content="""
+It's not clear to me what needs to be done here.
+
+I tried cloning the freesurfer repository, and in it with a cold
+disk cache, `git annex info` took 0.6 seconds (warm cache, 0.1 seconds).
+
+It seems like you wanted to get the "local annex size" value, perhaps
+without the overhead of the "size of annexed files in working tree"
+value? In an indirect mode repository, the former value is obtained
+the same as `du .git/annex/objects`, and should be similarly fast.
+""]]