Recent changes to this wiki:
Added a comment
diff --git a/doc/bugs/concurrent_get_from_separate_clones_fails/comment_5_bec4fd724e73338171a5b7bef9612a45._comment b/doc/bugs/concurrent_get_from_separate_clones_fails/comment_5_bec4fd724e73338171a5b7bef9612a45._comment new file mode 100644 index 0000000000..207aa2c59f --- /dev/null +++ b/doc/bugs/concurrent_get_from_separate_clones_fails/comment_5_bec4fd724e73338171a5b7bef9612a45._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2026-04-28T18:11:12Z" + content=""" +> fixed in [10.20260115-105-gfc28e5d81e AKA 10.20260213~17](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=fc28e5d81e0fc300865c44ad7688e3a60f25e858) +"""]]
Added a comment
diff --git a/doc/bugs/git_annex_get_is_silently_stuck_on__P2P___62___GET_0/comment_5_52862d85555e3810fd4717badd27c4d8._comment b/doc/bugs/git_annex_get_is_silently_stuck_on__P2P___62___GET_0/comment_5_52862d85555e3810fd4717badd27c4d8._comment new file mode 100644 index 0000000000..8513f9d2b8 --- /dev/null +++ b/doc/bugs/git_annex_get_is_silently_stuck_on__P2P___62___GET_0/comment_5_52862d85555e3810fd4717badd27c4d8._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2026-04-28T12:06:32Z" + content=""" +FWIW, I guess could be closed -- can no longer reproduce, git-annex 10.20241031-1~ndall+1 : `git annex copy --debug --to typhon --fast --not --in typhon` took super long to transfer everything but completed without errors. I do not think I have any system accessible to me with `/tmp` on NFS. ATM it is + +``` +[bids@rolando 1076_spacetop.git] > df /tmp +Filesystem 1K-blocks Used Available Use% Mounted on +/dev/sdb1 961102624 78224 912179920 1% /tmp +[bids@rolando 1076_spacetop.git] > mount | grep /tmp +/dev/sdb1 on /tmp type ext4 (rw,relatime,stripe=192,data=ordered) +``` + +"""]]
pull does not drop from remote, and push does not drop from local
Behavior change to git-annex pull and push's handling of unwanted files.
While previously both commands dropped unwanted files from both the remote
and the local repository, now pull only drops unwanted files from the local
repository, and push only drops unwanted files from the remote.
Sponsored-by: Jack Hill
Behavior change to git-annex pull and push's handling of unwanted files.
While previously both commands dropped unwanted files from both the remote
and the local repository, now pull only drops unwanted files from the local
repository, and push only drops unwanted files from the remote.
Sponsored-by: Jack Hill
diff --git a/CHANGELOG b/CHANGELOG
index a904dbbe15..445b34ba47 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,11 @@ git-annex (10.20260422) UNRELEASED; urgency=medium
* info: Report the total size of unused keys found by the last run
of git-annex unused.
+ * Behavior change to git-annex pull and push's handling of unwanted
+ files. While previously both commands dropped unwanted files from
+ both the remote and the local repository, now pull only drops unwanted
+ files from the local repository, and push only drops unwanted files
+ from the remote.
-- Joey Hess <id@joeyh.name> Mon, 27 Apr 2026 11:41:49 -0400
diff --git a/Command/Sync.hs b/Command/Sync.hs
index 5fed2027bf..54fcf48e0c 100644
--- a/Command/Sync.hs
+++ b/Command/Sync.hs
@@ -910,10 +910,11 @@ seekSyncContent o rs currbranch = do
- Send it to each remote that doesn't have it, and for which it's
- preferred content.
-
- - Drop it locally if it's not preferred content (honoring numcopies).
- -
- - Drop it from each remote that has it, where it's not preferred content
+ - When pulling, drop it locally if it's not preferred content
- (honoring numcopies).
+ -
+ - When pushing, drop it from each remote that has it, where it's
+ - not preferred content (honoring numcopies).
-
- Returns True if any file transfers were made.
-}
@@ -949,13 +950,14 @@ syncFile o ebloom rs af k = do
-- includeCommandAction for drops,
-- because a failure to drop does not mean
-- the sync failed.
- handleDropsFrom locs' rs "unwanted" True k af si []
+ handleDropsFrom locs' dropfromrs "unwanted" dropfromhere
+ k af si []
callCommandAction
return (got || not (null putrs))
where
wantget lu have inhere = allM id
- [ pure (pullOption o || operationMode o == SatisfyMode)
+ [ pure (pullOption o || satisfymode)
, pure (not $ null have)
, pure (not inhere)
, wantGet lu True (Just k) af
@@ -971,7 +973,7 @@ syncFile o ebloom rs af k = do
next $ return True
wantput lu r
- | pushOption o == False && operationMode o /= SatisfyMode = return False
+ | pushOption o == False && not satisfymode = return False
| Remote.readonly r || remoteAnnexReadOnly (Remote.gitconfig r) = return False
| isImport r && not (isExport r) = return False
| isExport r && not (exportHasAnnexObjects r) = return False
@@ -989,6 +991,14 @@ syncFile o ebloom rs af k = do
| otherwise = return []
put lu dest = includeCommandAction $
Command.Move.toStart' lu dest Command.Move.RemoveNever af k ai si
+
+ dropfromhere = pullOption o || satisfymode
+
+ dropfromrs
+ | pushOption o || satisfymode = rs
+ | otherwise = []
+
+ satisfymode = operationMode o == SatisfyMode
ai = mkActionItem (k, af)
si = SeekInput []
diff --git a/doc/git-annex-pull.mdwn b/doc/git-annex-pull.mdwn
index 29833e46c6..2ea45dd081 100644
--- a/doc/git-annex-pull.mdwn
+++ b/doc/git-annex-pull.mdwn
@@ -36,8 +36,8 @@ this updates the content in the local repository for those migrations as well.
Normally this tries to download the content of each annexed file,
from any remote that it's pulling from that has a copy.
To control which files it downloads, configure the preferred
-content of the local repository. It will also drop files from a
-remote that are not preferred content of the remote.
+content of the local repository. It will also drop files
+from the local repository that are not preferred content.
See [[git-annex-preferred-content]](1).
# OPTIONS
diff --git a/doc/git-annex-push.mdwn b/doc/git-annex-push.mdwn
index e8e140ba5f..6a1509a86e 100644
--- a/doc/git-annex-push.mdwn
+++ b/doc/git-annex-push.mdwn
@@ -44,7 +44,7 @@ Normally this tries to upload the content of each annexed file that is
in the working tree, to any remote that it's pushing to that does not have
a copy. To control which files are uploaded to a remote, configure the preferred
content of the remote. When a file is not the preferred content of a remote,
-or of the local repository, this command will try to drop the file's content.
+this command will try to drop the file's content from it.
See [[git-annex-preferred-content]](1).
# OPTIONS
diff --git a/doc/todo/should_pull_drop_from_remote.mdwn b/doc/todo/should_pull_drop_from_remote.mdwn
index 8fc051ca53..23c2432103 100644
--- a/doc/todo/should_pull_drop_from_remote.mdwn
+++ b/doc/todo/should_pull_drop_from_remote.mdwn
@@ -40,3 +40,8 @@ possible to only document this behavior change, and if a user has set up
such a preferred content, they can of course change it to something that
picks the repository they want to keep the copy.
--[[Joey]]
+
+> While this is a behavior change, it's also clearly a bug fix.
+> So it's ok to change behavior. I did call it out as a behavior change in
+> the changelog so affected users will hopefully notice. It did not seem
+> a big enough behavior change for a news entry though. [[done]] --[[Joey]]
some thoughts on this, no clear path forward
diff --git a/doc/todo/support_--not_--unused/comment_2_6924c4c2c02d81d9f05937dc01b82da6._comment b/doc/todo/support_--not_--unused/comment_2_6924c4c2c02d81d9f05937dc01b82da6._comment new file mode 100644 index 0000000000..d9df25d045 --- /dev/null +++ b/doc/todo/support_--not_--unused/comment_2_6924c4c2c02d81d9f05937dc01b82da6._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-04-27T17:18:02Z" + content=""" +`git-annex find` does not accept the `--unused` option. Only copy, drop, +get, whereis, and a few other commands like that do. + +But, the bug still holds, for example `git-annex get --not --unused` will +confusingly get all unused files and not any other files. + +There is a similar problem with `--not --key=foo`, which might be expected +to get all other keys. Or even `--not --branch=foo`. + +It seems difficult to detect all such mistakes though. Consider +`git-annex get --not --unused --smallerthan=1mb`. +The user might expect that to get small files that are not unused, +but actually it gets large files that are unused. To detect +that mistake, the option parser for `--unused` (and `--key` and `--branch`) +would need to detect that a file matching expression was started before +it, and is not complete. + +I think at some point it just doesn't make sense to prevent user foot +shooting. `--not` is well documented in what it can be used with, and if +the user comes up with some combination of options that includes it and +tries to read it as english, we'd even have to worry about things like +rejecting `--not --quiet --smallerthan=1mb` since the user might expect +it to behave like `--verbose`. +"""]] diff --git a/doc/todo/support_--not_--unused/comment_3_cddadb4cb92195f77a32a0a6016a5c95._comment b/doc/todo/support_--not_--unused/comment_3_cddadb4cb92195f77a32a0a6016a5c95._comment new file mode 100644 index 0000000000..5cffda1edb --- /dev/null +++ b/doc/todo/support_--not_--unused/comment_3_cddadb4cb92195f77a32a0a6016a5c95._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-04-27T17:28:19Z" + content=""" +A `--used` type file matching option would only make sense combined +with `--all` or perhaps `--key`. Otherwise, it's the default to not operate +on unused keys, because commands operate on keys that are referenced by +files in the working tree. + +Having that as a file matching option would open a new user confusion. +Consider "git-annex drop --not --used". That would never drop any files +because `drop` by default operates only on used files. + +(Except for in the case where `git-annex unused` was run in the past, +then an unused file were added back to the tree.. then it would surprise +the user in a different way!) + +Given that `--unused` is already potentially confusing since it's not a +file matching option and can behave in surprising ways when treated as one, +having a `--used` or similar that *is* a file matching option but doesn't +usually do anything seems like it only adds to the confusion. +"""]]
info: Report the total size of unused keys
Sponsored-by: unqueued
Sponsored-by: unqueued
diff --git a/CHANGELOG b/CHANGELOG
index 09807bbe27..c1831b7ca6 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,9 @@
+git-annex (10.20260422) UNRELEASED; urgency=medium
+
+ * info: Report the total size of unused keys.
+
+ -- Joey Hess <id@joeyh.name> Mon, 27 Apr 2026 11:41:49 -0400
+
git-annex (10.20260421) upstream; urgency=medium
* disableremote: New command.
diff --git a/Command/Info.hs b/Command/Info.hs
index 689fb2b130..dc6dc9d592 100644
--- a/Command/Info.hs
+++ b/Command/Info.hs
@@ -1,6 +1,6 @@
{- git-annex command
-
- - Copyright 2011-2025 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -33,6 +33,7 @@ import Annex.WorkTree
import Logs.UUID
import Logs.Trust
import Logs.Location
+import Logs.Unused
import Annex.Branch (UnmergedBranches(..), getUnmergedRefs)
import Annex.NumCopies
import Git.Config (boolConfig)
@@ -293,6 +294,7 @@ global_slow_stats :: [Stat]
global_slow_stats =
[ tmp_size
, bad_data_size
+ , unused_data_size
, local_annex_keys
, local_annex_size
, known_annex_files True
@@ -469,10 +471,15 @@ treeDesc True = "working tree"
treeDesc False = "tree"
tmp_size :: Stat
-tmp_size = staleSize "temporary object directory size" gitAnnexTmpObjectDir
+tmp_size = staleDirSize "temporary object directory size" gitAnnexTmpObjectDir
bad_data_size :: Stat
-bad_data_size = staleSize "bad keys size" gitAnnexBadDir
+bad_data_size = staleDirSize "bad keys size" gitAnnexBadDir
+
+unused_data_size :: Stat
+unused_data_size = staleSize "unused keys size"
+ (M.keys <$> readUnusedLog (literalOsPath ""))
+ (pure (calcRepo . gitAnnexLocation))
key_size :: Key -> Stat
key_size k = simpleStat "size" $ showSizeKeys $ addKey k emptyKeyInfo
@@ -829,10 +836,17 @@ showSizeKeys d = do
"+ " ++ show (unknownSizeKeys d) ++
" unknown size"
-staleSize :: String -> (Git.Repo -> OsPath) -> Stat
-staleSize label dirspec = Stat label $ do
- keys <- lift $ dirKeys dirspec
- onsize =<< sum <$> keysizes keys
+staleDirSize :: String -> (Git.Repo -> OsPath) -> Stat
+staleDirSize label dirspec = staleSize label (dirKeys dirspec) getpath
+ where
+ getpath = do
+ dir <- fromRepo dirspec
+ return (\k -> pure (dir </> keyFile k))
+
+staleSize :: String -> Annex [Key] -> (Annex (Key -> Annex OsPath)) -> Stat
+staleSize label listkeys getpath = Stat label $ do
+ keys <- lift listkeys
+ onsize =<< sum <$> lift (keysizes keys)
where
onsize 0 = return Nothing
onsize size = return $ Just $
@@ -841,9 +855,10 @@ staleSize label dirspec = Stat label $ do
return $ sizer storageUnits False size
in json (++ aside "clean up with git-annex unused") val label
keysizes keys = do
- dir <- lift $ fromRepo dirspec
- liftIO $ forM keys $ \k ->
- catchDefaultIO 0 $ getFileSize (dir </> keyFile k)
+ getpath' <- getpath
+ forM keys $ \k -> do
+ p <- getpath' k
+ liftIO $ catchDefaultIO 0 $ getFileSize p
aside :: String -> String
aside s = " (" ++ s ++ ")"
diff --git a/doc/todo/Size_of_unused_files.mdwn b/doc/todo/Size_of_unused_files.mdwn
index d16b4e09fc..7af07cfe66 100644
--- a/doc/todo/Size_of_unused_files.mdwn
+++ b/doc/todo/Size_of_unused_files.mdwn
@@ -12,3 +12,5 @@ Maybe I'm missing something, but it feels like having `git annex unused` and `gi
Cheers,
Yann
+
+> [[done]] --[[Joey]]
diff --git a/doc/todo/Size_of_unused_files/comment_2_e94da18cff0d57c1e0216d91388b19f0._comment b/doc/todo/Size_of_unused_files/comment_2_e94da18cff0d57c1e0216d91388b19f0._comment
new file mode 100644
index 0000000000..b4ccc67a8a
--- /dev/null
+++ b/doc/todo/Size_of_unused_files/comment_2_e94da18cff0d57c1e0216d91388b19f0._comment
@@ -0,0 +1,26 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2026-04-27T15:48:40Z"
+ content="""
+Note that `git-annex info` already shows the size of the keys in the
+temporary directory, and the bad directory. `git-annex unused` lists
+the keys in those 2 directories, in separate tables from the unused keys.
+
+This is making me rethink whether it might make sense for `git-annex info`
+to display the total size of the unused keys as well. While that would make
+the output of info depend on when `git-annex unused` was last run, and
+also on *how* it was run, it would be consistent.
+
+Also, someone may run `git-annex unused` regularly, or have run it and not
+paid much attention to the output, and not want to run it again in order to
+see how much space is used by the unused keys.
+
+And `info` has repository size information, so it would make sense to look
+at it when considering a `git-annex move --unused`.
+
+As for making `git-annex unused` also report the size, it would necessarily
+use a different output format than `git-annex info`. This feels like
+unnecessary duplication of functionality, even if it shared a common
+implementation.
+"""]]
comment
diff --git a/doc/todo/using_file_metadata_for_preferred___40__wanted__41___content/comment_2_665b5497c23201fa2ba4d4e50cd99088._comment b/doc/todo/using_file_metadata_for_preferred___40__wanted__41___content/comment_2_665b5497c23201fa2ba4d4e50cd99088._comment new file mode 100644 index 0000000000..1854243ead --- /dev/null +++ b/doc/todo/using_file_metadata_for_preferred___40__wanted__41___content/comment_2_665b5497c23201fa2ba4d4e50cd99088._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-04-27T15:32:13Z" + content=""" +I could imagine adding something to preferred content expressions like +`metadatawanted` that looks to see if the metadata contains a `wanted` +field that is the uuid of the repository. + +I doubt that this belongs in the standard preferred content expression +though. But `annex.initwanted` can be used to configure a preferred content +expression that you want to use in all new repositories that `git-annex` +inits. +"""]]
remove OSX and Windows from sidebar
As there are no OSX or Windows builds currently, it's confusing to have
the old builds listed there.
As there are no OSX or Windows builds currently, it's confusing to have
the old builds listed there.
diff --git a/doc/builds.mdwn b/doc/builds.mdwn index 5e4cc1fbeb..5a0f03134d 100644 --- a/doc/builds.mdwn +++ b/doc/builds.mdwn @@ -18,12 +18,6 @@ <h2>Linux arm64-ancient</h2> <iframe width=1024 height=40em scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuild/arm64-ancient/build-version"> </iframe> -<h2>OSX</h2> -<iframe width=1024 height=40em scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuild/x86_64-apple-catalina/build-version"> -</iframe> -<h2>Windows</h2> -<iframe width=1024 height=40em scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuild/windows/build-version"> -</iframe> """]] # build logs
update
diff --git a/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn index 8b8fcf76bb..b83d65c375 100644 --- a/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn +++ b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn @@ -13,7 +13,7 @@ A possible alternative for OSX would be to stop providing builds, and rely on homebrew's builds. And/or the [pipi/uv](https://pypi.org/project/git-annex/) builds could -perhaps be blessed as the official builds. Although they may currently be -built from on datalad/git-annex? +perhaps be blessed as the official builds. mih assures me those builds +will contiue to be from git-annex master. --[[Joey]]
update
diff --git a/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn index d9dd02df9a..8b8fcf76bb 100644 --- a/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn +++ b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn @@ -10,8 +10,10 @@ Note that CI for Windows and OSX is still available through appveyor. But I don't think build artifacts can be extracted from there? A possible alternative for OSX would be to stop providing builds, -and rely on homebrew's builds. And/or the -[pipi/uv](https://pypi.org/project/git-annex/) -builds could perhaps be blessed as the official builds. +and rely on homebrew's builds. + +And/or the [pipi/uv](https://pypi.org/project/git-annex/) builds could +perhaps be blessed as the official builds. Although they may currently be +built from on datalad/git-annex? --[[Joey]]
unfortunate todo
diff --git a/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn new file mode 100644 index 0000000000..d9dd02df9a --- /dev/null +++ b/doc/todo/new_windows_and_OSX_build_hosts_needed.mdwn @@ -0,0 +1,17 @@ +The datalad/git-annex repository is no longer used as an autobuilder of +of git-annex. <https://github.com/datalad/git-annex/issues/260> + +Specifically, it was used before for the Windows and OSX autobuilds. + +A new build host is needed in order for git-annex releases to be +accompanied by Windows and OSX builds. + +Note that CI for Windows and OSX is still available through +appveyor. But I don't think build artifacts can be extracted from there? + +A possible alternative for OSX would be to stop providing builds, +and rely on homebrew's builds. And/or the +[pipi/uv](https://pypi.org/project/git-annex/) +builds could perhaps be blessed as the official builds. + +--[[Joey]]
remove some old and no longer present branches from list
diff --git a/doc/download.mdwn b/doc/download.mdwn index 9fbd9878d7..2d2f45249d 100644 --- a/doc/download.mdwn +++ b/doc/download.mdwn @@ -19,9 +19,6 @@ others need some manual work. See [[install]] for details. The git repository has some branches, including: -* `ghc7.0` is a by now very out of date branch that can be built with - ghc 7.0. -* `debian-*-backport` contains the latest backport of git-annex. * `setup` contains configuration for this website ----
despam
diff --git a/doc/forum/OSX__39__s_default_sshd_behaviour_has_limited_paths_set/comment_4_fc4bc5c0f4e3f75b862adc517739c334._comment b/doc/forum/OSX__39__s_default_sshd_behaviour_has_limited_paths_set/comment_4_fc4bc5c0f4e3f75b862adc517739c334._comment deleted file mode 100644 index beeca24e25..0000000000 --- a/doc/forum/OSX__39__s_default_sshd_behaviour_has_limited_paths_set/comment_4_fc4bc5c0f4e3f75b862adc517739c334._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="cxararea" - avatar="http://cdn.libravatar.org/avatar/2718f71ca02c851974140f2a0c457b1b" - subject="Poor Bunny" - date="2026-01-21T07:29:04Z" - content=""" -Another standout feature is replayability. Each run feels different due to <a href=\"https://poorbunnygame.com\">Poor Bunny</a> random trap patterns, and the desire to beat your previous high score creates a strong “one more try” loop. -"""]]
remove links to datalad fork of git-annex
https://github.com/datalad/git-annex/ is now being used as a place to
file AI generated bug reports, which I have deleted from here. As such,
I condider it a fork. Since the OSX and Windows autobuilds are built
from there, they're built from a fork and cannot be linked to from here.
https://github.com/datalad/git-annex/ is now being used as a place to
file AI generated bug reports, which I have deleted from here. As such,
I condider it a fork. Since the OSX and Windows autobuilds are built
from there, they're built from a fork and cannot be linked to from here.
diff --git a/doc/builds.mdwn b/doc/builds.mdwn index 48744dda62..5e4cc1fbeb 100644 --- a/doc/builds.mdwn +++ b/doc/builds.mdwn @@ -43,17 +43,5 @@ <h2>Linux arm64-ancient</h2> <iframe width=1024 scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuild/arm64-ancient/"> </iframe> -<h2>OSX</h2> -<a href="https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+macOS%22"> -<img src="https://github.com/datalad/git-annex/actions/workflows/build-macos.yaml/badge.svg"> -</a> -<h2>Windows</h2> -<a href="https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+Windows%22"> -<img src="https://github.com/datalad/git-annex/actions/workflows/build-windows.yaml/badge.svg"> -</a> -<h2>Debian standalone packages</h2> -<a href="https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+Ubuntu%22"> -<img src="https://github.com/datalad/git-annex/actions/workflows/build-ubuntu.yaml/badge.svg"> -</a> <h2>Appveyor</h2> <a href="https://ci.appveyor.com/project/mih/git-annex">here</a> diff --git a/doc/install/OSX.mdwn b/doc/install/OSX.mdwn index eba4050544..32ee0e856e 100644 --- a/doc/install/OSX.mdwn +++ b/doc/install/OSX.mdwn @@ -18,10 +18,8 @@ several more. Handy if you don't otherwise have git installed. ## autobuilds -An autobuild is also available, thanks to the Datalad project. - -* [git-annex.dmg](https://downloads.kitenet.net/git-annex/autobuild/x86_64-apple-catalina/git-annex.dmg) - ([build logs](https://github.com/datalad/git-annex/actions/workflows/build-macos.yaml?query=Build+git-annex+on+macOS)) +No autobuilds are currently available, however there is a Appveyor CI +[here](https://ci.appveyor.com/project/mih/git-annex). ## download security diff --git a/doc/install/Windows.mdwn b/doc/install/Windows.mdwn index 6ea4b5f756..b795d2919f 100644 --- a/doc/install/Windows.mdwn +++ b/doc/install/Windows.mdwn @@ -13,10 +13,8 @@ current status. ## autobuilds -An autobuild is also available, thanks to the Datalad project. - -* Windows 10 [download](https://downloads.kitenet.net/git-annex/autobuild/windows/git-annex-installer.exe) - ([build logs](https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+Windows%22)) +No autobuilds are currently available, however there is a Appveyor CI +[here](https://ci.appveyor.com/project/mih/git-annex). ## download security
remove unhelpful arguments
These are not bug reports in any valuable sense. They do not belong in
this repository or website.
These are not bug reports in any valuable sense. They do not belong in
this repository or website.
diff --git a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn
deleted file mode 100644
index 2fbda7139b..0000000000
--- a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn
+++ /dev/null
@@ -1 +0,0 @@
-Deleted AI generated bug report. [[done]] --[[Joey]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
deleted file mode 100644
index 9d80e78e5f..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
+++ /dev/null
@@ -1,3 +0,0 @@
-Deleted AI generated bug report. --[[Joey]]
-
-[[done]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment
deleted file mode 100644
index f597ded910..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment
+++ /dev/null
@@ -1,10 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 10"""
- date="2026-04-22T14:15:37Z"
- content="""
-I have no interest in drafting an AI policy. I do have interest in
-communicating with human beings who respect my time and work. By causing an
-AI to blather at me in corpspeak ("smoothed") you once again demonstrate
-you do not.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment
deleted file mode 100644
index 2165b2e652..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment
+++ /dev/null
@@ -1,12 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 1"""
- date="2026-04-21T18:33:33Z"
- content="""
-I refuse to read, respond, or track AI generated bug reports.
-
-As such, I am closing this bug report.
-
-I am asking for a second time, that you cease submitting such bug reports
-to this site. I will not ask a third time.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment
deleted file mode 100644
index 7b8a193fac..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment
+++ /dev/null
@@ -1,10 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 2"
- date="2026-04-21T20:59:45Z"
- content="""
-The **report** was generated by me and there was nothing in the report itself produced by AI: I cited a few invocations of git commands from the script produced by AI (not alone, but with me) -- output there was from git and git-annex. I am not sure how more non-AI the report itself could have been.
-
-The issue seems to be real and I did use claude code, as disclosed, to identify it to get myself unstuck since otherwise there were no feedback or information provided.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment
deleted file mode 100644
index 5e75df6bf5..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment
+++ /dev/null
@@ -1,14 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 3"""
- date="2026-04-22T00:59:58Z"
- content="""
-The report did not include instructions for reproducing it save for 315
-lines of slop. It would take a long time to read and comprehend what is
-being done in there. Dumping that on me shows that you do not respect my
-time.
-
-Putting that on a different page than directly in the bug report as a way
-to attempt to work around my request that it not be posted here just shows
-that you do not respect my boundaries.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment
deleted file mode 100644
index 755f25e4dc..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment
+++ /dev/null
@@ -1,10 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 4"
- date="2026-04-22T03:24:29Z"
- content="""
-**This comment consisted of AI slop and as such has been deleted.
-(It was also condescending corporate speak bullshit and techinically wrong
-to boot.)** --[[Joey]]
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment
deleted file mode 100644
index 5c165cc691..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 5"
- date="2026-04-22T04:13:37Z"
- content="""
-FWIW, the issue is likely not in git-annex itself but rather in forgejo-aneksjo not populating annex.uuid in the config although populating annex.url upon \"push to create\". Filed [forgejo-aneksajo/issues/113](https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/issues/113).
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment
deleted file mode 100644
index 12dd223414..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment
+++ /dev/null
@@ -1,7 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 6"""
- date="2026-04-22T06:23:40Z"
- content="""
-Was your comment #4 above written with the assistance of AI?
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment
deleted file mode 100644
index afa17cecec..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment
+++ /dev/null
@@ -1,11 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 7"""
- date="2026-04-22T06:33:59Z"
- content="""
-Re "This forces git-annex clients to probe the p2p-http endpoint to
-discover the remote's UUID" in your forgejo-aneksajo bug report.
-
-There is no remote UUID discovery operation in the P2P protocol.
-That is an AI hallucination.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment
deleted file mode 100644
index c7eb89f639..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 8"
- date="2026-04-22T13:25:47Z"
- content="""
-thanks, roger that.
-"""]]
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment
deleted file mode 100644
index 62456ebf18..0000000000
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment
+++ /dev/null
@@ -1,17 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 9"
- date="2026-04-22T13:43:59Z"
- content="""
-now spotted:
-
-> [!comment format=mdwn username=\"joey\" subject=\"\"\"comment 6\"\"\" date=\"2026-04-22T06:23:40Z\" content=\"\"\" Was your comment #4 above written with the assistance of AI? \"\"\"]]
-
-and again, out of respect — I am neither ignoring your question nor hiding provenance:
-
-\"with assistance\" -- yes, as with assistance of a keyboard, monitor, spell checker, etc. But it was not written by AI: I wrote it first in full first, then shortened & \"smoothed\" with AI (as I am not a native English speaker etc), reread and re-extended by myself in few iterations. It did come out long but representing my thoughts adequately.
-
-The request for clear a policy remains outstanding.
-I guess it could be a variant (or better exact) version of the [forgejo's AI agreement](https://codeberg.org/forgejo/governance/src/branch/main/AIAgreement.md) agreement.
-"""]]
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn
deleted file mode 100644
index 11b4895ce4..0000000000
--- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn
+++ /dev/null
@@ -1,110 +0,0 @@
-### Please describe the problem.
-
-Finally got to try the neat p2p (inspired by trr379 use-case raised in matrix), ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally.
-Unfortunately
-
-<details>
-<summary>whenever p2phttp worked fine (for a full file) request on 10.20251029</summary>
-
-```shell
-❯ curl http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz
- % Total % Received % Xferd Average Speed Time Time Time Current
(Diff truncated)
Added a comment
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_2_a37f1a012f0988be433177bf39a0447b._comment b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_2_a37f1a012f0988be433177bf39a0447b._comment
new file mode 100644
index 0000000000..53a9986752
--- /dev/null
+++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_2_a37f1a012f0988be433177bf39a0447b._comment
@@ -0,0 +1,97 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 2"
+ date="2026-04-22T17:02:04Z"
+ content="""
+FWIW: Nothing in the original report was produced by AI. Mention of claude and absent range requests support was only to provide provenance on what brought me here to discover this issue.
+
+Here is a full trivial reproducer collating those manually ran commands into 1 script:
+
+```
+#!/bin/bash
+
+export PS4='> '
+set -xeu
+
+cd \"$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)\"
+git clone https://datasets.datalad.org/dbic/QA/.git
+cd QA
+git annex version --raw; echo;
+
+sha256=02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f
+key=SHA256E-s101318091--$sha256.tgz
+
+git annex get --key \"$key\"
+uuid=$(git config annex.uuid)
+
+git annex --debug p2phttp --port 8081 --wideopen &
+
+sleep 2
+
+curl http://localhost:8081/git-annex/$uuid/key/$key >| out.gz
+
+ls -l out.gz; sha256sum out.gz
+
+kill %1
+
+echo \"done ok\"
+```
+
+which with recent git-annex produces
+
+```
+...
+> git annex version --raw
+10.20260316+git92-g28d90e468f-1~ndall+1> echo
+...
+curl: (52) Empty reply from server
+...
+```
+
+while passing all the way to 'done ok' with `10.20251029`.
+
+
+<details>
+<summary>with bleeding edge snapshot 10.20260421+git5-g38a24cc9df-1~ndall+1 - I still get \"Empty reply from server\" but now there are also logs from the git-annex server for `P2P .* DATA 101318091` but no `P2P .* SUCCESS`</summary>
+
+```shell
+> git annex version --raw
+10.20260421+git5-g38a24cc9df-1~ndall+1> echo
+
+> sha256=02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f
+> key=SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz
+> git annex get --key SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz
+get SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz (from origin...) (scanning for annexed files...)
+ok
+(recording state in git...)
+>> git config annex.uuid
+> uuid=6a1c5631-aa13-45be-babb-48aed74bcff9
+> sleep 2
+> git annex --debug p2phttp --port 8081 --wideopen
+[2026-04-22 12:59:21.489519787] (Utility.Process) process [3021850] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"show-ref\",\"git-annex\"]
+[2026-04-22 12:59:21.493864396] (Utility.Process) process [3021850] done ExitSuccess
+[2026-04-22 12:59:21.494532963] (Utility.Process) process [3021851] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+[2026-04-22 12:59:21.498327575] (Utility.Process) process [3021851] done ExitSuccess
+[2026-04-22 12:59:21.498911996] (Utility.Process) process [3021854] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log\",\"refs/heads/git-annex..7487ff3e269d9c37208e4f2a93e31a1ff6103f4c\",\"--pretty=%H\",\"-n1\"]
+[2026-04-22 12:59:21.50282289] (Utility.Process) process [3021854] done ExitSuccess
+[2026-04-22 12:59:21.504379992] (Utility.Process) process [3021855] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat-file\",\"--batch\"]
+[2026-04-22 12:59:21.509479669] (Annex.Branch) read proxy.log
+> curl http://localhost:8081/git-annex/6a1c5631-aa13-45be-babb-48aed74bcff9/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz
+ % Total % Received % Xferd Average Speed Time Time Time Current
+ Dload Upload Total Spent Left Speed
+ 0 0 0 0 0 0 0 0 0[2026-04-22 12:59:23.475306344] (P2P.IO) [http client] [ThreadId 23] P2P > GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz
+[2026-04-22 12:59:23.475728505] (P2P.IO) [http server] [ThreadId 22] P2P < GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz
+[2026-04-22 12:59:23.477029319] (Utility.Process) process [3022009] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"-c\",\"filter.annex.smudge=\",\"-c\",\"filter.annex.clean=\",\"-c\",\"filter.annex.process=\",\"write-tree\"]
+[2026-04-22 12:59:23.487752805] (Utility.Process) process [3022009] done ExitSuccess
+[2026-04-22 12:59:23.488176365] (Utility.Process) process [3022015] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"show-ref\",\"--hash\",\"refs/annex/last-index\"]
+[2026-04-22 12:59:23.492356509] (Utility.Process) process [3022015] done ExitSuccess
+[2026-04-22 12:59:23.493004854] (P2P.IO) [http server] [ThreadId 22] P2P > DATA 101318091
+[2026-04-22 12:59:23.493137804] (P2P.IO) [http client] [ThreadId 23] P2P < DATA 101318091
+ 0 0 0 0 0 0 0 0 0
+curl: (52) Empty reply from server
+bash p2p-notworking.sh 10.64s user 3.16s system 75% cpu 18.358 total
+
+```
+</details>
+"""]]
comment
diff --git a/doc/todo/test_script_against_headless_forgejo-aneksajo_/comment_1_bd6dbbbdec6fb0fba34dcda824a0568f._comment b/doc/todo/test_script_against_headless_forgejo-aneksajo_/comment_1_bd6dbbbdec6fb0fba34dcda824a0568f._comment new file mode 100644 index 0000000000..595f720b0c --- /dev/null +++ b/doc/todo/test_script_against_headless_forgejo-aneksajo_/comment_1_bd6dbbbdec6fb0fba34dcda824a0568f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-04-22T15:06:30Z" + content=""" +I have no forgejo-aneksajo experience. In any case, I don't see how this is +a git-annex todo. +"""]]
uninterested
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index 6f3492dc88..11b4895ce4 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -107,3 +107,4 @@ FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile bu [[!meta author=yoh]] +> [[done]] with this BS --[[Joey]] diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_1_3672a7e460436c3c100e2a1287f4bdb6._comment b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_1_3672a7e460436c3c100e2a1287f4bdb6._comment new file mode 100644 index 0000000000..2ad861f3b9 --- /dev/null +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses/comment_1_3672a7e460436c3c100e2a1287f4bdb6._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-04-22T14:59:36Z" + content=""" +> since was reported to be lacking by claude on a forgejo+aneksjo instance + +Your slop machine is hallucinating again. forgejo+aneksjo has range +support. + +The git-annex p2phttp endpoint, being an API endpoint for a protocol that +does not use range support, does not have range support. + +I suspect that the rest of this bug report is similar AI-addled thinking. +It's not worth my time to try to understand what you or an AI were doing +here. Thus, closing. +"""]]
close non-bug report
diff --git a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn index dfed9e9a7e..2fbda7139b 100644 --- a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn +++ b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn @@ -1 +1 @@ -Deleted AI generated bug report. +Deleted AI generated bug report. [[done]] --[[Joey]]
deslop
diff --git a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn index 9f301ff7f3..dfed9e9a7e 100644 --- a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn +++ b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn @@ -1,49 +1 @@ -### Please describe the problem. - -I was investigating the safety of parallel workers pushing + annex copying to the same original local origin location. -It gets stuck (I have not even tried yet that "ssh" option...) and it seems potentially the counter play of 'annex copy' like - -``` -yoh 3858496 0.0 0.0 7628 3844 pts/29 S 15:25 0:00 | \_ /usr/bin/bash -c worker 3 -yoh 3861583 0.0 0.0 7984 3936 pts/29 S 15:25 0:00 | | \_ git annex copy --to origin file-3-1.txt file-3-2.txt file-3-3.txt file-3-4.txt file-3-5.txt file-common.txt -yoh 3861585 3.7 0.1 1075475816 72956 pts/29 Sl 15:25 0:05 | | \_ /usr/bin/git-annex copy --to origin file-3-1.txt file-3-2.txt file-3-3.txt file-3-4.txt file-3-5.txt file-common.txt -yoh 3861886 0.0 0.0 8116 4564 pts/29 S 15:25 0:00 | | \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch - -``` -(of which I have ATM multiple going on) - -and 'annex post-receive' hook (of which I have only one) running upon `git push` - -``` -yoh 3858498 0.0 0.0 7628 3764 pts/29 S 15:25 0:00 | \_ /usr/bin/bash -c worker 4 -yoh 3863122 0.0 0.0 17340 4948 pts/29 Sl 15:25 0:00 | | \_ git push origin master:br-4 -yoh 3863130 0.0 0.0 2692 1912 pts/29 S 15:25 0:00 | | \_ /bin/sh -c git-receive-pack '/home/yoh/.tmp/parallel-push-3858314/origin' git-receive-pack '/home/yoh/.tmp/parallel-push-3858314/origin' -yoh 3863134 0.0 0.0 16980 5312 pts/29 Sl 15:25 0:00 | | \_ git-receive-pack /home/yoh/.tmp/parallel-push-3858314/origin -yoh 3863172 0.0 0.0 2692 1872 pts/29 S 15:25 0:00 | | \_ /bin/sh hooks/post-receive -yoh 3863192 0.0 0.0 7984 4060 pts/29 S 15:25 0:00 | | \_ git annex post-receive -yoh 3863195 0.3 0.0 1074074572 17816 pts/29 Sl 15:25 0:00 | | \_ /usr/bin/git-annex post-receive -``` - - -### What steps will reproduce the problem? - -[here](https://www.oneukrainian.com/tmp/parallel-push.sh) is a claude-code (with use of LLMs and HI; I did not even review in detail/try yet 'ssh' part coded there) generated script (look or use at your own discretion), on execution of which as `./parallel-push.sh --max-file-size 409600 --max-files 10 40 20` it gets stuck (this is the 4th run I think, consistent stuck at different places) with - -``` -copy file-common.txt ok -copy file-12-1.txt (to origin...) (checksum...) ok -(recording state in git...) -clone-12: done (1 files) -Cloning into '/home/yoh/.tmp/parallel-push-3858314/clone-12'... -done. -remote: (recording state in git...) -remote: (recovering from race...) -To /home/yoh/.tmp/parallel-push-3858314/origin - * [new branch] master -> br-12 - - -``` - -### What version of git-annex are you using? On what operating system? - -this run is with `10.20251029` but I tried bleeding edge standalone build `10.20260213+git57-gffa771e735-1~ndall+1` to the same result but process traces are more garbled so for the benefit of our both HI I pasted from the non-standalone built version +Deleted AI generated bug report.
fixed
diff --git a/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished.mdwn b/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished.mdwn index 1600d0ed3e..b55912deea 100644 --- a/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished.mdwn +++ b/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished.mdwn @@ -45,3 +45,5 @@ upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) This is my first post here. First, I use git-annex for both my personal files and professional datasets since around 5 years. So I have to say this wonderful piece of software helped me a lot, therefore a big thank you Joey and other contributors. ;) Second, as a reminder to myself, I really should consider to add tips to the website, since I accumulated a lot of self-wrote documentation in my notes! + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished/comment_5_0bd5fed989b1b97ad9d19d5e40d455fe._comment b/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished/comment_5_0bd5fed989b1b97ad9d19d5e40d455fe._comment new file mode 100644 index 0000000000..04114b7a2d --- /dev/null +++ b/doc/bugs/git-annex__58_____60__stdout__62____58___hPutBuf__58___resource_vanished/comment_5_0bd5fed989b1b97ad9d19d5e40d455fe._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-04-22T14:43:30Z" + content=""" +Fixed in [[!commit de5dee49da1dc9ca7733f723299235da1da52b10]]. +"""]]
close
diff --git a/doc/bugs/does_not_clean_sub_processes_until_freeze.mdwn b/doc/bugs/does_not_clean_sub_processes_until_freeze.mdwn index 3291e35910..b66702eaa0 100644 --- a/doc/bugs/does_not_clean_sub_processes_until_freeze.mdwn +++ b/doc/bugs/does_not_clean_sub_processes_until_freeze.mdwn @@ -63,3 +63,6 @@ upgrade supported from repository versions: 0 1 2 3 4 5 6 7 I use it for all kind of data I have both in private and at work. Amazing piece of software. I'm sure my colleagues/IT are annoyed of me plugging it to every possible discussion. + +> [[done]] as this seems like a bug that was already fixed in a newer +> version. --[[Joey]]
markup
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment index 82a74472bf..755f25e4dc 100644 --- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment @@ -4,7 +4,7 @@ subject="comment 4" date="2026-04-22T03:24:29Z" content=""" -This comment consisted of AI slop and as such has been deleted. +**This comment consisted of AI slop and as such has been deleted. (It was also condescending corporate speak bullshit and techinically wrong -to boot.) --[[Joey]] +to boot.)** --[[Joey]] """]]
fix formatting
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment index 256698d02d..12dd223414 100644 --- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment @@ -1,4 +1,4 @@ -[!comment format=mdwn +[[!comment format=mdwn username="joey" subject="""comment 6""" date="2026-04-22T06:23:40Z"
deslop
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment index 2d6a74bdb4..82a74472bf 100644 --- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment @@ -4,11 +4,7 @@ subject="comment 4" date="2026-04-22T03:24:29Z" content=""" -Joey, You asked that AI-generated bug reports not be posted -- and I didn't post one. The report itself was written by me. What it contained inline was a short, focused excerpt of `git-annex`'s own debug output showing exactly where `git credential fill` exits 128 and where git-annex then sets `annex-ignore` on the remote. That is the evidence for the bug, extracted and summarized by me so you would not have to wade through anything longer. It is not 315 lines, and it is not AI output -- it is your program's log output. If to understand more fully you need more information -- just ask, and I will try to distill. - -The fuller reproducer script lived behind a link, deliberately, so that you would not have to read it, and so that it is not posted on this site, as per your request. I included that link, together with the explicit AI disclaimer, because I wanted to be transparent about provenance rather than hide how I arrived at the diagnosis. Reading that as an attempt to \"work around\" your request is the opposite of what I intended, and I'd ask you to reconsider it. - -In the past I've done my best to produce minimal reproducers, and I'll keep doing that as feasible and time permits -- here I did not have one hand-crafted. But I don't recall a stated policy that reports without a complete hand-crafted reproducer would be deleted as AI slop — and that's the part that rubs me wrong, because the concise summary I did write is the part that seems went unread. - -I would genuinely appreciate a clearer statement of what you will and will not accept for bug and todo contributions, so I can either meet that bar or stop contributing here. I will respect whatever policy you set for this site. I would hope that policy would concern what gets posted here, and not what tools I use in my own work elsewhere, where I will continue to use what I need, including AI assistance, to do my job. I won't ask you to look at external reproducers going forward, but I would appreciate if the policy would not forbid me to link such materials when they exist, for provenance. +This comment consisted of AI slop and as such has been deleted. +(It was also condescending corporate speak bullshit and techinically wrong +to boot.) --[[Joey]] """]]
no
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment
new file mode 100644
index 0000000000..f597ded910
--- /dev/null
+++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_10_1db64b7075ec519dd68d68ea24342a83._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 10"""
+ date="2026-04-22T14:15:37Z"
+ content="""
+I have no interest in drafting an AI policy. I do have interest in
+communicating with human beings who respect my time and work. By causing an
+AI to blather at me in corpspeak ("smoothed") you once again demonstrate
+you do not.
+"""]]
Added a comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment new file mode 100644 index 0000000000..62456ebf18 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_9_6437b029e09543eed8c342a95fa75185._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 9" + date="2026-04-22T13:43:59Z" + content=""" +now spotted: + +> [!comment format=mdwn username=\"joey\" subject=\"\"\"comment 6\"\"\" date=\"2026-04-22T06:23:40Z\" content=\"\"\" Was your comment #4 above written with the assistance of AI? \"\"\"]] + +and again, out of respect — I am neither ignoring your question nor hiding provenance: + +\"with assistance\" -- yes, as with assistance of a keyboard, monitor, spell checker, etc. But it was not written by AI: I wrote it first in full first, then shortened & \"smoothed\" with AI (as I am not a native English speaker etc), reread and re-extended by myself in few iterations. It did come out long but representing my thoughts adequately. + +The request for clear a policy remains outstanding. +I guess it could be a variant (or better exact) version of the [forgejo's AI agreement](https://codeberg.org/forgejo/governance/src/branch/main/AIAgreement.md) agreement. +"""]]
Added a comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment new file mode 100644 index 0000000000..c7eb89f639 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_8_eb2b702793821159d41dee698e215366._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 8" + date="2026-04-22T13:25:47Z" + content=""" +thanks, roger that. +"""]]
bs
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment new file mode 100644 index 0000000000..afa17cecec --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_7_4a4445b46d177942e73bd8741ece7011._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2026-04-22T06:33:59Z" + content=""" +Re "This forces git-annex clients to probe the p2p-http endpoint to +discover the remote's UUID" in your forgejo-aneksajo bug report. + +There is no remote UUID discovery operation in the P2P protocol. +That is an AI hallucination. +"""]]
comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment new file mode 100644 index 0000000000..256698d02d --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_6_0254810faab3f3392cdb255bf0e7b186._comment @@ -0,0 +1,7 @@ +[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2026-04-22T06:23:40Z" + content=""" +Was your comment #4 above written with the assistance of AI? +"""]]
Added a comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment new file mode 100644 index 0000000000..5c165cc691 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_5_dacbb561579fd22f86b435d582cb3375._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2026-04-22T04:13:37Z" + content=""" +FWIW, the issue is likely not in git-annex itself but rather in forgejo-aneksjo not populating annex.uuid in the config although populating annex.url upon \"push to create\". Filed [forgejo-aneksajo/issues/113](https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/issues/113). +"""]]
Added a comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment new file mode 100644 index 0000000000..2d6a74bdb4 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_4_f5b8938e1c4ffec8c3d9dfeef1b2cd57._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 4" + date="2026-04-22T03:24:29Z" + content=""" +Joey, You asked that AI-generated bug reports not be posted -- and I didn't post one. The report itself was written by me. What it contained inline was a short, focused excerpt of `git-annex`'s own debug output showing exactly where `git credential fill` exits 128 and where git-annex then sets `annex-ignore` on the remote. That is the evidence for the bug, extracted and summarized by me so you would not have to wade through anything longer. It is not 315 lines, and it is not AI output -- it is your program's log output. If to understand more fully you need more information -- just ask, and I will try to distill. + +The fuller reproducer script lived behind a link, deliberately, so that you would not have to read it, and so that it is not posted on this site, as per your request. I included that link, together with the explicit AI disclaimer, because I wanted to be transparent about provenance rather than hide how I arrived at the diagnosis. Reading that as an attempt to \"work around\" your request is the opposite of what I intended, and I'd ask you to reconsider it. + +In the past I've done my best to produce minimal reproducers, and I'll keep doing that as feasible and time permits -- here I did not have one hand-crafted. But I don't recall a stated policy that reports without a complete hand-crafted reproducer would be deleted as AI slop — and that's the part that rubs me wrong, because the concise summary I did write is the part that seems went unread. + +I would genuinely appreciate a clearer statement of what you will and will not accept for bug and todo contributions, so I can either meet that bar or stop contributing here. I will respect whatever policy you set for this site. I would hope that policy would concern what gets posted here, and not what tools I use in my own work elsewhere, where I will continue to use what I need, including AI assistance, to do my job. I won't ask you to look at external reproducers going forward, but I would appreciate if the policy would not forbid me to link such materials when they exist, for provenance. +"""]]
no
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment new file mode 100644 index 0000000000..5e75df6bf5 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_3_5c9478eac4078f697d04096b2d5f32c8._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-04-22T00:59:58Z" + content=""" +The report did not include instructions for reproducing it save for 315 +lines of slop. It would take a long time to read and comprehend what is +being done in there. Dumping that on me shows that you do not respect my +time. + +Putting that on a different page than directly in the bug report as a way +to attempt to work around my request that it not be posted here just shows +that you do not respect my boundaries. +"""]]
Added a comment
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment new file mode 100644 index 0000000000..7b8a193fac --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_2_1862351058301fbe147b6ddbcfb0f44a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2026-04-21T20:59:45Z" + content=""" +The **report** was generated by me and there was nothing in the report itself produced by AI: I cited a few invocations of git commands from the script produced by AI (not alone, but with me) -- output there was from git and git-annex. I am not sure how more non-AI the report itself could have been. + +The issue seems to be real and I did use claude code, as disclosed, to identify it to get myself unstuck since otherwise there were no feedback or information provided. +"""]]
add news item for git-annex 10.20260420
diff --git a/doc/news/version_10.20251114.mdwn b/doc/news/version_10.20251114.mdwn deleted file mode 100644 index 63255f2897..0000000000 --- a/doc/news/version_10.20251114.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -git-annex 10.20251114 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * p2p --pair: Fix to work with external P2P networks. - * p2phttp: Significant robustness fixes for bugs that caused the - server to stall. - * p2phttp: Fix a file descriptor leak. - * p2phttp: Added the --lockedfiles option. - * dropunused: Run the annex.secure-erase-command - (or .git/hooks/secure-erase-annex) when deleting - temp and bad object files. - * remotedaemon: Avoid crashing when run with --debug."""]] \ No newline at end of file diff --git a/doc/news/version_10.20260420.mdwn b/doc/news/version_10.20260420.mdwn new file mode 100644 index 0000000000..abd090e745 --- /dev/null +++ b/doc/news/version_10.20260420.mdwn @@ -0,0 +1,10 @@ +git-annex 10.20260420 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * disableremote: New command. + * Fix annexUrl to inherit any password that is set in the remote url. + * Add DELEGATE extension to the external special remote protocol. + * Avoid dying of an exception when when stdout gets closed by eg head(1), + and avoid a crash loop when stderr is closed and git-annex dies of an + exception. + Fixes reversion introduced in version 10.20230407. + * Improve UUID sanitization. + * Deal with breaking changes to test concurrency in tasty-1.5.4."""]] \ No newline at end of file
close
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn index f823a12899..9d80e78e5f 100644 --- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn @@ -1 +1,3 @@ Deleted AI generated bug report. --[[Joey]] + +[[done]]
delete slop
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
index 2db7b5f377..f823a12899 100644
--- a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
+++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
@@ -1,50 +1 @@
-### Please describe the problem.
-
-Disclaimer: this is heavily AI-driven investigation since neither I figured it out myself, nor a complete reproducer to setup such an env existed/was provided. Origin: [PR in datalad-fuse](https://github.com/datalad/datalad-fuse/pull/127) where I want to establish a complete turnkey testing fixture against forgejo+aneksjo instance.
-
-In case of working with forgejo+anexksjo, it seems that if read `url` is publicly (no auth needed) available, but `pushurl` requires authentication and token is specified:
-
-```
-[remote "forgejo"]
- url = http://127.0.0.1:34201/testadmin/repro-1776783743.git
- fetch = +refs/heads/*:refs/remotes/forgejo/*
- pushurl = http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git
-
-```
-
-`git annex init` (10.20260316+git92-g28d90e468f-1~ndall+1) calls `git credential fill` which might fail e.g. in non interactive sessions and where no credential helper is available/configured, and that leads git-annex to announcement that remote is "not usable by git-annex; setting annex-ignore"
-
-
-<details>
-<summary>relevant excerpt from a run by an AI-produced script using 10.20260316+git92-g28d90e468f-1~ndall+1</summary>
-
-```shell
-❯ NO_CRED_HELPER=1 ANNEX=1 tools/forgejo-repro.sh
-[repro] git-annex version: 10.20260316+git92-g28d90e468f-1~ndall+1
-...
-+ git fetch -q forgejo
-+ git annex init --debug -q repro-local
-+ grep -i -E '(credential|annex-ignore|usable|error|fail|uuid)'
-+ sed 's/^/ [annex init] /'
- [annex init] [2026-04-21 11:02:29.572106245] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
- [annex init] [2026-04-21 11:02:29.581477964] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
- [annex init] [2026-04-21 11:02:29.588763529] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
- [annex init] [2026-04-21 11:02:29.605283022] (Annex.Branch) read uuid.log
- [annex init] [2026-04-21 11:02:29.605605509] (Annex.Branch) set uuid.log
- [annex init] [2026-04-21 11:02:29.620280198] (Utility.Process) process [1561063] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
- [annex init] [2026-04-21 11:02:29.623557836] (Utility.Process) process [1561063] done ExitFailure 128
- [annex init] Remote forgejo not usable by git-annex; setting annex-ignore
-```
-
-</details>
-
-### What steps will reproduce the problem?
-
-Full version of the script is [available in the aforementioned PR](https://github.com/datalad/datalad-fuse/pull/127/changes#diff-604675e9a53a0ae87fcf02ab81e613cf17b48b3f354ecb8c491a83dcd8d345e1) and could be used to reproduce above via `NO_CRED_HELPER=1 ANNEX=1 tools/forgejo-repro.sh`.
-
-
-
-### What version of git-annex are you using? On what operating system?
-
-10.20260316+git92-g28d90e468f-1~ndall+1
-
+Deleted AI generated bug report. --[[Joey]]
no
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment new file mode 100644 index 0000000000..2165b2e652 --- /dev/null +++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err/comment_1_fb77f4b017401ba3a9fb3c114919bedb._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-04-21T18:33:33Z" + content=""" +I refuse to read, respond, or track AI generated bug reports. + +As such, I am closing this bug report. + +I am asking for a second time, that you cease submitting such bug reports +to this site. I will not ask a third time. +"""]]
initial report on credential fill
diff --git a/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
new file mode 100644
index 0000000000..2db7b5f377
--- /dev/null
+++ b/doc/bugs/init_unnecessarily_errors_if_credentials_fill_err.mdwn
@@ -0,0 +1,50 @@
+### Please describe the problem.
+
+Disclaimer: this is heavily AI-driven investigation since neither I figured it out myself, nor a complete reproducer to setup such an env existed/was provided. Origin: [PR in datalad-fuse](https://github.com/datalad/datalad-fuse/pull/127) where I want to establish a complete turnkey testing fixture against forgejo+aneksjo instance.
+
+In case of working with forgejo+anexksjo, it seems that if read `url` is publicly (no auth needed) available, but `pushurl` requires authentication and token is specified:
+
+```
+[remote "forgejo"]
+ url = http://127.0.0.1:34201/testadmin/repro-1776783743.git
+ fetch = +refs/heads/*:refs/remotes/forgejo/*
+ pushurl = http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git
+
+```
+
+`git annex init` (10.20260316+git92-g28d90e468f-1~ndall+1) calls `git credential fill` which might fail e.g. in non interactive sessions and where no credential helper is available/configured, and that leads git-annex to announcement that remote is "not usable by git-annex; setting annex-ignore"
+
+
+<details>
+<summary>relevant excerpt from a run by an AI-produced script using 10.20260316+git92-g28d90e468f-1~ndall+1</summary>
+
+```shell
+❯ NO_CRED_HELPER=1 ANNEX=1 tools/forgejo-repro.sh
+[repro] git-annex version: 10.20260316+git92-g28d90e468f-1~ndall+1
+...
++ git fetch -q forgejo
++ git annex init --debug -q repro-local
++ grep -i -E '(credential|annex-ignore|usable|error|fail|uuid)'
++ sed 's/^/ [annex init] /'
+ [annex init] [2026-04-21 11:02:29.572106245] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
+ [annex init] [2026-04-21 11:02:29.581477964] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
+ [annex init] [2026-04-21 11:02:29.588763529] (Git.Config) git config read: [("",[""]),("annex.security.allowed-http-addresses",["all"]),("annex.security.allowed-ip-addresses",["127.0.0.1"]),("annex.security.allowed-url-schemes",["http https file"]),("annex.uuid",["d6b16f32-7ddd-4107-8b0e-8db60a996119"]),("annex.version",["10"]),("core.bare",["false"]),("core.filemode",["true"]),("core.logallrefupdates",["true"]),("core.repositoryformatversion",["0"]),("filter.annex.clean",["git-annex smudge --clean -- %f"]),("filter.annex.process",["git-annex filter-process"]),("filter.annex.smudge",["git-annex smudge -- %f"]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("remote.forgejo.fetch",["+refs/heads/*:refs/remotes/forgejo/*"]),("remote.forgejo.pushurl",["http://32c116cc671332bf9518d31371a8b515266aa80e:@127.0.0.1:34201/testadmin/repro-1776783743.git"]),("remote.forgejo.url",["http://127.0.0.1:34201/testadmin/repro-1776783743.git"]),("user.email",["test@test.nil"]),("user.name",["Test User"])]
+ [annex init] [2026-04-21 11:02:29.605283022] (Annex.Branch) read uuid.log
+ [annex init] [2026-04-21 11:02:29.605605509] (Annex.Branch) set uuid.log
+ [annex init] [2026-04-21 11:02:29.620280198] (Utility.Process) process [1561063] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
+ [annex init] [2026-04-21 11:02:29.623557836] (Utility.Process) process [1561063] done ExitFailure 128
+ [annex init] Remote forgejo not usable by git-annex; setting annex-ignore
+```
+
+</details>
+
+### What steps will reproduce the problem?
+
+Full version of the script is [available in the aforementioned PR](https://github.com/datalad/datalad-fuse/pull/127/changes#diff-604675e9a53a0ae87fcf02ab81e613cf17b48b3f354ecb8c491a83dcd8d345e1) and could be used to reproduce above via `NO_CRED_HELPER=1 ANNEX=1 tools/forgejo-repro.sh`.
+
+
+
+### What version of git-annex are you using? On what operating system?
+
+10.20260316+git92-g28d90e468f-1~ndall+1
+
forgot to add a comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_4_dda5420ba6169e8e9c449aea3532cbf3._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_4_dda5420ba6169e8e9c449aea3532cbf3._comment new file mode 100644 index 0000000000..c9f52d4730 --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_4_dda5420ba6169e8e9c449aea3532cbf3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-04-13T16:58:54Z" + content=""" +Was able to get git-annex back to typical unix-like behavior on sigpipe, +and also avoided the crash loop when stderr is closed. +"""]]
Added a comment
diff --git a/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment b/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment new file mode 100644 index 0000000000..911e23259c --- /dev/null +++ b/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 11" + date="2026-04-15T19:55:19Z" + content=""" +FTR: fixed in [10.20251215-51-g69e6c4d024 AKA 10.20260115~59](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=69e6c4d024dcff7c2f8ea1a2ed3b483a86b2cc7d) +"""]]
bug report
diff --git a/doc/bugs/annex.adjustedbranchrefresh_of_remote_not_honored.mdwn b/doc/bugs/annex.adjustedbranchrefresh_of_remote_not_honored.mdwn new file mode 100644 index 0000000000..c527c40ae9 --- /dev/null +++ b/doc/bugs/annex.adjustedbranchrefresh_of_remote_not_honored.mdwn @@ -0,0 +1,8 @@ +When a git remote has annex.adjustedbranchrefresh set to 1, `git-annex sync --content` +run with that remote should only refresh the adjusted branch there once. +But, I've verified it does so after every file is sent. Which is quite +slow. + +Apparently this is because in this case it registers a cleanup action, and +the way the Annex monad works for a local git remote, it runs those each +time, due to calling quiesce. --[[Joey]]
Added a comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment new file mode 100644 index 0000000000..072cc430af --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2026-04-13T17:30:16Z" + content=""" +FWIW -- never returned, doing on another box -- here is a complete command : `git clone https://github.com/dandisets/000026.git && cd 000026 && git annex init && git annex find --not --in here --fast | head -n 1` -- stalled as well. I wonder if replicates for you. Also there with `2>/dev/null` returned right away, but on rerun without - stalled again. +"""]]
fix sanitizeTopLevelExceptionMessages
Avoid dying of an exception when when stdout gets closed by eg head(1), and
avoid a crash loop when stderr is closed and git-annex dies of an
exception. Fixes reversion introduced in version 10.20230407.
Note that giveup sanitizes escape characters, so
sanitizeTopLevelExceptionMessages can just use it to rethrow
the exception. But the exception type information is lost, so
an exception caused by sigPIPE would be displayed.
I don't entirely understand the crash loop, but it seemed to be caused
by the exitWith $ ExitFailure 1. Which is not necessary when using
giveup.
It's a bit scary to mess with the sigPIPE handler, but restoring the
Default does get the behavior we want. If that turned out to cause a
problem though, that part of this could be reverted, and git-annex would
only display the kind of ugly error message when piped to head.
Sponsored-by: Dartmouth College's OpenNeuro project
Avoid dying of an exception when when stdout gets closed by eg head(1), and
avoid a crash loop when stderr is closed and git-annex dies of an
exception. Fixes reversion introduced in version 10.20230407.
Note that giveup sanitizes escape characters, so
sanitizeTopLevelExceptionMessages can just use it to rethrow
the exception. But the exception type information is lost, so
an exception caused by sigPIPE would be displayed.
I don't entirely understand the crash loop, but it seemed to be caused
by the exitWith $ ExitFailure 1. Which is not necessary when using
giveup.
It's a bit scary to mess with the sigPIPE handler, but restoring the
Default does get the behavior we want. If that turned out to cause a
problem though, that part of this could be reverted, and git-annex would
only display the kind of ugly error message when piped to head.
Sponsored-by: Dartmouth College's OpenNeuro project
diff --git a/CHANGELOG b/CHANGELOG
index a791dfdaa8..aa29f974f7 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,10 @@ git-annex (10.20260317) UNRELEASED; urgency=medium
* disableremote: New command.
* Add DELEGATE extension to the external special remote protocol.
* Improve UUID sanitization.
+ * Avoid dying of an exception when when stdout gets closed by eg head(1),
+ and avoid a crash loop when stderr is closed and git-annex dies of an
+ exception.
+ Fixes reversion introduced in version 10.20230407.
-- Joey Hess <id@joeyh.name> Mon, 23 Mar 2026 11:37:09 -0400
diff --git a/Messages.hs b/Messages.hs
index 704d5cfeac..25b129e93d 100644
--- a/Messages.hs
+++ b/Messages.hs
@@ -1,6 +1,6 @@
{- git-annex output messages
-
- - Copyright 2010-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -65,6 +65,9 @@ import qualified Data.ByteString.Char8 as S8
import System.Exit
import qualified Control.Monad.Catch as M
import Data.String
+#ifndef mingw32_HOST_OS
+import System.Posix.Signals
+#endif
import Common
import Types
@@ -355,15 +358,23 @@ mkPrompter = getConcurrency >>= \case
{- Catch all (non-async and not ExitCode) exceptions and display,
- sanitizing any control characters in the exceptions.
-
- - Exits nonzero on exception, so should only be used at topmost level.
+ - Should only be used at topmost level.
-}
sanitizeTopLevelExceptionMessages :: IO a -> IO a
-sanitizeTopLevelExceptionMessages a = a `catches`
- ((M.Handler (\ (e :: ExitCode) -> throwM e)) : nonAsyncHandler go)
+sanitizeTopLevelExceptionMessages a = do
+#ifndef mingw32_HOST_OS
+ -- By default ghc Ignores sigPIPE, and then does not display
+ -- exceptions like <stdout>: hFlush: resource vanished (Broken pipe)
+ --
+ -- Since this would display such exceptions, instead restore the
+ -- Default sigPIPE behavior, which is for the program to
+ -- immediately exit.
+ void $ installHandler sigPIPE Default Nothing
+#endif
+ a `catches`
+ ((M.Handler (\ (e :: ExitCode) -> throwM e)) : nonAsyncHandler go)
where
- go e = do
- hPutStrLn stderr $ safeOutput $ toplevelMsg (show e)
- exitWith $ ExitFailure 1
+ go e = giveup $ show e
{- Used to only run an action that displays a message after the specified
- number of steps. This is useful when performing an action that can
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
index 58fea8f47a..0558fc1204 100644
--- a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
@@ -41,3 +41,5 @@ I didn't check other commands , but they all should behave sane as not to stall
[[!meta author=yoh]]
[[!tag projects/openneuro]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
index 529216f696..789fdcdc0f 100644
--- a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
@@ -12,12 +12,9 @@ pipe exception, and so the crash loop never happens when stderr is closed.
That is relatively new; it was added in git-annex 10.20230407.
And apparently what ghc's default exception display mechanism does is
-actually to ignore the broken pipe exception for stdout. Which leads
-to the more typical unix behavior of silently stopping on SIGPIPE.
-See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html>
+actually to avoid displaying anything for the broken pipe exception for
+stdout. Which leads to the more typical unix behavior of silently
+stopping on SIGPIPE.
-Note that the loop when displaying an exception fails with an exception due
-to stderr being closed still seems to be the fault of ghc/base, not
-git-annex. All that sanitizeTopLevelExceptionMessages does is cause the
-exception to be displayed.
+See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html>
"""]]
comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment new file mode 100644 index 0000000000..529216f696 --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-04-13T15:22:17Z" + content=""" +Turns out that a git-annex that does not +use sanitizeTopLevelExceptionMessages will not have this behavior. + +Without that, `git-annex find | head -n1` does not display the broken +pipe exception, and so the crash loop never happens when stderr is closed. + +That is relatively new; it was added in git-annex 10.20230407. + +And apparently what ghc's default exception display mechanism does is +actually to ignore the broken pipe exception for stdout. Which leads +to the more typical unix behavior of silently stopping on SIGPIPE. +See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html> + +Note that the loop when displaying an exception fails with an exception due +to stderr being closed still seems to be the fault of ghc/base, not +git-annex. All that sanitizeTopLevelExceptionMessages does is cause the +exception to be displayed. +"""]]
Added a comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment new file mode 100644 index 0000000000..037cabf9dc --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment @@ -0,0 +1,37 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 3" + date="2026-04-13T15:56:23Z" + content=""" +> I'm curious if you also ran git-annex with FD 2 closed? + +how do I discover that ? it worked fine if I redirect stderr: + +``` +$> git annex find --in here 2>/dev/null | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat + +$> echo $? +0 + +``` + +and damn it -- after that it started to work in that shell: + +``` +$> git annex find --in here | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +git-annex: <stdout>: hFlush: resource vanished (Broken pipe) +``` + +:-/ I think it relates to dataset size as if I go to other small datasets in that shell now -- it returns fine as expected, but in huge `/mnt/btrfs/datasets/datalad/crawl/dandi/dandisets/000026` -- still running (likely more stuff spit out to stdout), even with + +``` +(git)smaug:/mnt/btrfs/datasets/datalad/crawl/dandi/dandisets/000026[draft]git +$> git annex find --not --in here --fast 2>/dev/null | head -n 1 +derivatives/Depth_normalized_OCT_volume/I38/sub-I38_ses-OCT_sample-BrocaAreaS01_depth-normed_OCT.ome.tiff +``` + +I will keep it going to see if ever returns, will check in an hour. FWIW -- that dataset is [here](https://github.com/dandisets/000026) +"""]]
comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment
new file mode 100644
index 0000000000..13ff17f759
--- /dev/null
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment
@@ -0,0 +1,36 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2026-04-13T14:29:51Z"
+ content="""
+I was able to reproduce this, but only when I ran git-annex with FD 2
+closed.
+
+ perl -e 'close(STDERR); system("git-annex find --not --in here --fast | head -n 1")'
+
+I'm curious if you also ran git-annex with FD 2 closed?
+
+I suspect what must be happening is that the exception handler crashes when
+outputting to stderr, which causes an exception to be thrown,
+leading to a crash loop.
+
+If so, it would seem likely to be a bug in ghc/base. And I'd not be surprised to find
+such a bug somewhere in its bug tracker. In fact, I almost remember finding this same
+behavior before, which may have helped me guess this due was FD 2 being closed.
+
+With that said, I've not been able to reproduce the behavior yet with a simpler
+haskell program like this one:
+
+ import System.IO
+
+ main = do
+ print "foo"
+ hFlush stdout
+ print "bar"
+ hFlush stdout
+ main
+
+But, that simple program also doesn't throw
+"hFlush: resource vanished (Broken pipe)" when piped to `head -n1`,
+so it's not quite replicating what git-annex does.
+"""]]
comment
diff --git a/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment b/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment new file mode 100644 index 0000000000..bdee883797 --- /dev/null +++ b/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""Re: git-annex: potential data loss with initremote and push with S3 special remotes.""" + date="2026-04-13T13:59:28Z" + content=""" +That needs to be a bug report. I have copied it to one here: +[[bugs/S3_pointed_to_minio_console_potential_data_loss]] +"""]]
comment
diff --git a/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment new file mode 100644 index 0000000000..3e13fe424f --- /dev/null +++ b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment @@ -0,0 +1,28 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-04-13T14:01:37Z" + content=""" +It's arguably not git-annex's fault if it was pointed to an API endpoint +that behaves enough like S3 to make it seems like it's stored data, +but does not store the data. + +With that said, for there to be data loss here, the file would need to be +dropped from the local repository, relying on the copy "stored" in S3. +So the S3 special remote's checkPresent could be improved to prevent +such a bad endpoint from being treated as containing the content of an +object. + +For S3, checkPresent does pass in the VersionID when git-annex knows one. +(Which that doesn't help if the API endpoint ignores that header.) +What it does not do is check the response from S3 for a VersionID or ETag. +Improving that seems like a possible way to avoid this kind of problem. + +It does check the ETag when the S3 remote is configured with +exporttree=true. + +As for the idea of checking the annex-uuid write by reading the file back, +the difficulty with that is S3 `DEEP_ARCHIVE` and similar can have an +hours-long delay to get back out a file that is already stored in the +bucket. Also, the annex-uuid file is not used for exporttree=yes remotes. +"""]]
promote comment to bug report
diff --git a/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn
new file mode 100644
index 0000000000..51b043b042
--- /dev/null
+++ b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn
@@ -0,0 +1,24 @@
+A colleague used a wrong config, which was pointing to minio console rather than the S3 endpoint. When they ran initremote, the console wrongfully replied 200-OK when PUTting the annex-uuid file, same when they then pushed the data. The minio console always redirect to a login page, and doesn't fail on PUT ( which is non-compliant ). So the dataset recorded all the data being present in that remote, while there was no trace of any buckets or objects in the S3.
+
+## steps to reproduce:
+
+```
+git init test_s3
+cd test_s3/
+git-annex init
+export AWS_ACCESS_KEY_ID=john AWS_SECRET_ACCESS_KEY=doe
+git annex initremote -d test_remote host=\"play.min.io\" bucket=\"test_bucket\" type=S3 encryption=none autoenable=true port=9443 protocol=https chunk=1GiB requeststyle=pathecho test > test_annexed_file
+git-annex add test_annexed_file
+git commit -m 'add annexed file'
+git-annex copy --fast --to test_remote
+```
+
+I am showing it with `--fast` flag here, as this is what datalad uses by default. Without `--fast`, it fails with (HeaderException {headerErrorMessage = \"ETag missing\"}) failed which is better.
+
+So to sum it up, the unfortunate circumstances are:
+
+1. the initremote PUT of annex-uuid is not performing check that the annex-uuid file was effectively pushed in a bucket.
+2. minio console replies with 200-OK for all http requests
+3. datalad uses `push --fast` by default, which recorded files as being pushed without performing a HEAD after push. I guess that's for performance reason, but that is dangerous if a server or reverse-proxy ends-up responding 200-OK to all requests after init.
+
+Thanks for your help!
Added a comment
diff --git a/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment b/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment new file mode 100644 index 0000000000..075b3b1b14 --- /dev/null +++ b/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2026-04-10T13:03:09Z" + content=""" +FWIW, I think this might be useful for openneuro as I think we run into cases (e.g. ds000113) where we have records on some elderly keys (such as SHA1--771e0eea7ceb32216a5a06c89c50d1f02bc79d6d) for which I think we no longer have any commit in the history pointing to them or even if we do -- we have no such key exported in the bucket (I think). +"""]]
done with DELEGATE extension
diff --git a/doc/todo/Ephemeral_special_remotes.mdwn b/doc/todo/Ephemeral_special_remotes.mdwn index 7082e52512..79fce6c3d6 100644 --- a/doc/todo/Ephemeral_special_remotes.mdwn +++ b/doc/todo/Ephemeral_special_remotes.mdwn @@ -12,3 +12,5 @@ A use case would be to have an "orchestration" special remotes that maybe repres in some way also an alternative to the `sameas` approach, where the alternatives are hidden in the implementation of a special remote, rather than in *each* repository. [[!tag projects/INM7]] + +> [[done]]! --[[Joey]] diff --git a/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment b/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment new file mode 100644 index 0000000000..8610d3b9b6 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 13""" + date="2026-04-08T19:22:38Z" + content=""" +Finished implementing this! +"""]]
implement DELEGATE with ephemeral=yes
Factored Annex.DisableRemote out of Command.DisableRemote
Factored Annex.DisableRemote out of Command.DisableRemote
diff --git a/Annex/DisableRemote.hs b/Annex/DisableRemote.hs
new file mode 100644
index 0000000000..bcaed58d03
--- /dev/null
+++ b/Annex/DisableRemote.hs
@@ -0,0 +1,189 @@
+{- disable a remote
+ -
+ - Copyright 2026 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+{-# LANGUAGE OverloadedStrings #-}
+
+module Annex.DisableRemote (disableRemote) where
+
+import Git
+import Annex.Common
+import qualified Annex
+import qualified Git.Remote.Remove
+import qualified Git.Ref
+import Types.Remote
+import Annex.Journal
+import qualified Annex.Branch
+import Annex.Branch.Transitions
+import Types.Transitions
+import Logs
+import Logs.Remote.Pure
+import Logs.MapLog
+import Logs.Transfer
+import Types.Transfer
+import qualified Database.Export
+import qualified Database.Fsck
+import qualified Database.RepoSize
+import qualified Database.ContentIdentifier
+import qualified Utility.OsString as OS
+
+import Data.ByteString.Builder
+import qualified Data.Map as M
+import qualified Data.Set as S
+import qualified Data.ByteString.Lazy as L
+
+disableRemote :: Remote -> RemoteName -> [Remote] -> Annex ()
+disableRemote r remotename remotelist = do
+ let uniqueuuid = not $
+ any (\r' -> uuid r' == uuid r && name r' /= name r) remotelist
+
+ when uniqueuuid $
+ -- If there are transfers to/from the remote still
+ -- running, this fails. That's why it's run early.
+ removeTransferLogs (uuid r)
+
+ cleanPrivateJournal r uniqueuuid
+
+ when uniqueuuid $ do
+ -- Each uuid has its own export and fsck database,
+ -- so always remove them, so long as this is the
+ -- only remote using this uuid.
+ Database.Export.removeDb (uuid r)
+ Database.Fsck.removeDb (uuid r)
+ -- These databases are updated from information
+ -- in the git-annex branch, so there is no point in
+ -- removing the uuid from them unless it's private.
+ whenM (isPrivateUUID (uuid r)) $ do
+ Database.RepoSize.removeUUID (uuid r)
+ Database.ContentIdentifier.removeUUID (uuid r) True
+
+ removeFsckState (uuid r)
+ removeImportLog (uuid r)
+ removeCredFiles (uuid r)
+
+ inRepo $ Git.Remote.Remove.remove remotename
+ removeRemoteTrackingBranches remotename
+
+-- Remove any remote branches.
+-- This is done because git remote remove only removes the configured
+-- remote tracking branch, not other remote branches.
+removeRemoteTrackingBranches :: String -> Annex ()
+removeRemoteTrackingBranches remotename = do
+ branches <- filter (\b -> branchprefix `isPrefixOf` fromRef b)
+ . map snd
+ <$> inRepo Git.Ref.list
+ forM_ branches $ \b ->
+ inRepo $ Git.Ref.delete' b
+ where
+ branchprefix = "refs/remotes/" ++ remotename ++ "/"
+
+isPrivateUUID :: UUID -> Annex Bool
+isPrivateUUID u =
+ (\c -> u `S.member` annexPrivateRepos c)
+ <$> Annex.getGitConfig
+
+{- Remove a private remote's uuid from the private journal
+ - entirely when it is the only remote using that uuid.
+ -
+ - And, when the remote is a sameas remote, its config is stored
+ - under its config-uuid. Remove that from the private remote log.
+ -}
+cleanPrivateJournal :: Remote -> Bool -> Annex ()
+cleanPrivateJournal r uniqueuuid
+ | uniqueuuid == True = do
+ whenM (isPrivateUUID (uuid r)) $ do
+ gc <- Annex.getGitConfig
+ let tc = filterBranch (\u -> u /= uuid r) gc
+ let handlestale = \_ b -> return (b, Nothing)
+ lockJournal $ \jl ->
+ overPrivateJournalFileContents handlestale Just
+ (go jl tc)
+ removeconfiguuid
+ | otherwise = removeconfiguuid
+ where
+ go jl tc getfilecontents = getfilecontents >>= \case
+ Just (_, p, Just (b, _)) -> do
+ cleaner jl tc p b
+ go jl tc getfilecontents
+ Just (_, _, Nothing) ->
+ go jl tc getfilecontents
+ Nothing -> return ()
+
+ cleaner jl tc p b = case tc p b of
+ PreserveFile -> return ()
+ ChangeFile builder ->
+ setlocaljournal jl (RegardingUUID [uuid r]) p builder
+
+ removeconfiguuid = case remoteAnnexConfigUUID (gitconfig r) of
+ Nothing -> return ()
+ Just cu -> whenM (isPrivateUUID cu) $
+ lockJournal $ \jl ->
+ getJournalFile jl (GetPrivate True) remoteLog >>= \case
+ JournalledContent b ->
+ scrub jl cu b
+ PossiblyStaleJournalledContent b ->
+ scrub jl cu b
+ NoJournalledContent ->
+ return ()
+ where
+ scrub jl cu b =
+ setlocaljournal jl (RegardingUUID [cu]) remoteLog $
+ buildRemoteConfigLog $ MapLog $ M.delete cu $
+ fromMapLog $ parseRemoteConfigLog b
+
+ setlocaljournal jl ru p builder =
+ let b' = toLazyByteString builder
+ in if L.null b'
+ then deleteJournalFile jl ru p
+ else do
+ b'' <- Annex.Branch.getLocal' (GetPrivate False) p
+ if b'' == b'
+ then deleteJournalFile jl ru p
+ else setJournalFile jl ru p builder
+
+removeFsckState :: UUID -> Annex ()
+removeFsckState u = do
+ d <- fromRepo (gitAnnexFsckStateDir u)
+ liftIO $ whenM (doesDirectoryExist d) $
+ removeDirectoryRecursive d
+ f <- fromRepo (gitAnnexFsckResultsLog u)
+ liftIO $ removeWhenExistsWith removeFile f
+
+removeImportLog :: UUID -> Annex ()
+removeImportLog u = do
+ f <- calcRepo' (gitAnnexImportLog u)
+ liftIO $ removeWhenExistsWith removeFile f
+
+removeTransferLogs :: UUID -> Annex ()
+removeTransferLogs u = do
+ -- getTransfers calls checkTransfer, which cleans up
+ -- transfer log files for transfers that are no longer running.
+ whenM (any foru <$> getTransfers) $
+ error "Active trasfers, cannot disable the remote."
+ forM_ [Upload, Download] $ \direction -> do
+ d <- fromRepo $ gitAnnexTransferUUIDDirectionDir u direction
+ liftIO $ void $ tryNonAsync $ removeDirectory d
+ d' <- fromRepo $ gitAnnexFailedTransferDir u direction
+ liftIO $ void $ tryNonAsync $ removeDirectory d'
+ clearFailedTransfers u
+ where
+ foru (t, _) = transferUUID t == u
+
+removeCredFiles :: UUID -> Annex ()
+removeCredFiles u = do
+ d <- fromRepo gitAnnexCredsDir
+ liftIO $ whenM (doesDirectoryExist d) $ do
+ mapM_ (removeWhenExistsWith removeFile)
+ =<< filter foru <$> dirContents d
+ where
+ us = fromUUID u
+
+ -- Remotes that use creds always include their UUID as part of the
+ -- filename. However, some remotes (Remote.External) need more than
+ -- one creds file, and add a "-foo" suffix to the UUID.
+ foru p =
+ let f = takeFileName p
+ in f == us || (us <> literalOsPath "-") `OS.isPrefixOf` f
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 46130bc273..d3bfe80d05 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -5,36 +5,11 @@
(Diff truncated)
initial request re forgejo-aneksjo e2e loop script
diff --git a/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn b/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn new file mode 100644 index 0000000000..d7810f7c9c --- /dev/null +++ b/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn @@ -0,0 +1,19 @@ +The need came up in the scope of trying to extend datalad-fuse to support forgejo-aneksajo ([AI assisted PR, look at your discretion](https://github.com/datalad/datalad-fuse/pull/127/changes)). There, to test, I would like to establish test fixture which would be used in tests to test interactions against containerized `codeberg.org/forgejo-aneksajo/forgejo-aneksajo:forgejo-rootless` instance . But I failed to programmatically establish the flow to make git-annex authenticate there even with some drums and bells. + +So I thought if you have some basic shell script (not copy/paste of a manual execution since I think there might be some races I am confronting with) which could reliably demonstrate sequence of commands to + +- boot up fresh forgejo-rootless (e.g. using podman) +- create basic account there +- create or push (to create) repo +- clone it locally (so it gets recognized as git-annex repo) +- add file under annex +- git push and git-annex copy annexed file (so authentication should work out) to forgejo-rootless + +would be greatly appreciated. + +It would allow us to establish local forgejo+aneksjo for testing of datalad-fuse (and other components) which would then use to access remote on forgejo instances openneuro preprocessed datasets. + +[[!meta author=yoh]] +[[!tag projects/openneuro]] + +
make ephemeral=no the default
This is a good default because a remote may be expensive to set up, and
ephemeral=yes would then not be desirable.
This is a good default because a remote may be expensive to set up, and
ephemeral=yes would then not be desirable.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index dc476852fe..e85333ae9f 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 82c54630ca..771d16f926 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value ephemeral=yes|no [params]
+ DELEGATE type=value [ephemeral=yes|no] [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -15,7 +15,8 @@ is not stored to the git-annex branch.
With ephemeral=yes, the delegate special remote is created and exists only
as long as the external special remote program is used by git-annex. Then
it is removed. With ephemeral=no, the delegate special remote is cached for
-use next time, avoiding the overhead of initializing it again.
+use next time, avoiding the overhead of initializing it again. When it is
+not specified, the default is ephemeral=no.
For example:
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index d1834385d3..ce1ec411bf 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
Revert "temporarily remove ephemeral setting"
This reverts commit 4ca54f030fdbbc0452b568b7ae6528f89f78b310.
This reverts commit 4ca54f030fdbbc0452b568b7ae6528f89f78b310.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index e85333ae9f..dc476852fe 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 5318a0683c..82c54630ca 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value [params]
+ DELEGATE type=value ephemeral=yes|no [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -12,18 +12,25 @@ and `--private` options. The delegate special remote inherits
the encryption settings, and uses the same annex-uuid. Its configuration
is not stored to the git-annex branch.
+With ephemeral=yes, the delegate special remote is created and exists only
+as long as the external special remote program is used by git-annex. Then
+it is removed. With ephemeral=no, the delegate special remote is cached for
+use next time, avoiding the overhead of initializing it again.
+
For example:
TRANSFER STORE somekey tmpfile
- DELEGATE type=directory directory=/mnt/disk/
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
This makes the annex object be stored in a directory special remote.
+Since initializing a directory special remote is inexpensive, it's
+made ephemeral.
When the key is later requested to be retrieved,
the same delegate can be used:
TRANSFER RETRIEVE somekey tmpfile
- DELEGATE type=directory directory=/mnt/disk/
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
----
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index ce1ec411bf..d1834385d3 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment b/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment new file mode 100644 index 0000000000..300931dd27 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 12""" + date="2026-04-08T18:23:11Z" + content=""" +`git-annex disableremote` is completed and I hope to wrap this up soon. +"""]]
TODO for fsck against exported S3
diff --git a/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn b/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn new file mode 100644 index 0000000000..2049f40479 --- /dev/null +++ b/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn @@ -0,0 +1,55 @@ +More info could be found at fresh [bug report against openneuro.org](https://github.com/OpenNeuroOrg/openneuro/issues/3875) and reproduction helpers in quick&dirty [20260408-noversionid](https://github.com/yarikoptic/20260408-noversionid) + +But boils down to the fact that if for some reason (older buggy git-annex used back then?) `git-annex` branch lacks log.rmet entries for files **present** exported on S3, it should populate those .log.rmet entries, but ATM it just fails to `fsck` but then manages to `get`. + +```shell +(git)smaug:/mnt/btrfs/datasets/datalad/tmp/20260408-noversionid/ds000113[master]git +$> git annex whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (3 copies) + 66a7004d-a15e-4764-90cd-54bbd179f74a -- [s3-PRIVATE] + b8b60a40-f339-4ddc-b08a-2a6f645bd3ef -- root@8dc3dbd70baf:/datalad/ds001473 + e28d70a7-9314-4542-a4ce-7d95b862070f -- [s3-PUBLIC] +ok + +$> git annex fsck --from s3-PUBLIC sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +fsck sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz + Remote is configured to use versioning, but no S3 version ID is recorded for this key +(cannot check content) failed +(recording state in git...) +fsck: 1 failed + +$> git annex whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (3 copies) + 66a7004d-a15e-4764-90cd-54bbd179f74a -- [s3-PRIVATE] + b8b60a40-f339-4ddc-b08a-2a6f645bd3ef -- root@8dc3dbd70baf:/datalad/ds001473 + e28d70a7-9314-4542-a4ce-7d95b862070f -- [s3-PUBLIC] +ok + +$> git annex get sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +get sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (from s3-PRIVATE...) + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(Delaying 1s before retrying....) + + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(Delaying 2s before retrying....) + + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(from s3-PUBLIC...) + Remote is configured to use versioning, but no S3 version ID is recorded for this key +ok + +$> git annex version | head -n 1 +git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05 + +``` + +Note that there could be multiple versions for the remote file and ideally git-annex does "its best" to figure out which one is the one to use, potentially even downloading them until hitting the matching one unless could associate using ETag etc. (FWIW ETag is deterministic even for multipart uploads, given knowing how upload was chunked. [Here](https://github.com/dandi/dandi-schema/blob/master/dandischema/digests/dandietag.py) our python code could be found. + +[[!meta author=yoh]] +[[!tag projects/openneuro]]
disableremote cred file removal
This includes removing the cred files used to accress P2P peers.
Note that the P2P auth token, which is used to auth incoming
connections, is stored in a cred file, but not one named with the uuid
of a remote, so it will not be deleted by this. Which is as intended,
see commit ddad858a817e58e26c21417196c3aac28591c8db
This includes removing the cred files used to accress P2P peers.
Note that the P2P auth token, which is used to auth incoming
connections, is stored in a cred file, but not one named with the uuid
of a remote, so it will not be deleted by this. Which is as intended,
see commit ddad858a817e58e26c21417196c3aac28591c8db
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 81016ebd19..dfc1990bfb 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -5,6 +5,8 @@
- Licensed under the GNU AGPL version 3 or higher.
-}
+{-# LANGUAGE OverloadedStrings #-}
+
module Command.DisableRemote where
import Command
@@ -27,6 +29,7 @@ import qualified Database.Export
import qualified Database.Fsck
import qualified Database.RepoSize
import qualified Database.ContentIdentifier
+import qualified Utility.OsString as OS
import Data.ByteString.Builder
import qualified Data.Map as M
@@ -73,10 +76,7 @@ start (remotename:[]) = byName' remotename >>= \case
removeFsckState (uuid r)
removeImportLog (uuid r)
-
- -- It would be good to remove cred files, but there
- -- is currently no way to list cred files belonging
- -- to a remote, or even to a UUID.
+ removeCredFiles (uuid r)
inRepo $ Git.Remote.Remove.remove remotename
removeRemoteTrackingBranches remotename
@@ -187,3 +187,18 @@ removeTransferLogs u = do
where
foru (t, _) = transferUUID t == u
+removeCredFiles :: UUID -> Annex ()
+removeCredFiles u = do
+ d <- fromRepo gitAnnexCredsDir
+ liftIO $ whenM (doesDirectoryExist d) $ do
+ mapM_ (removeWhenExistsWith removeFile)
+ =<< filter foru <$> dirContents d
+ where
+ us = fromUUID u
+
+ -- Remotes that use creds always include their UUID as part of the
+ -- filename. However, some remotes (Remote.External) need more than
+ -- one creds file, and add a "-foo" suffix to the UUID.
+ foru p =
+ let f = takeFileName p
+ in f == us || (us <> literalOsPath "-") `OS.isPrefixOf` f
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
index 75cc0dd97d..0c0e4f6775 100644
--- a/doc/git-annex-disableremote.mdwn
+++ b/doc/git-annex-disableremote.mdwn
@@ -20,9 +20,6 @@ use with the `git-annex enableremote` command.
When `git-annex initremote --private` was used to initialize a private
remote, disabling it also removes its information from the private journal.
-Note that this does not currently delete any cached credentials for a
-special remote or P2P peer. That may be improved in the future.
-
# OPTIONS
* `--json`
Added a comment: others stall too -- workaround <()
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment new file mode 100644 index 0000000000..2ea98a49b0 --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="others stall too -- workaround <()" + date="2026-04-08T14:42:59Z" + content=""" +well -- head also stalls + +``` +smaug:/tmp/ds000113 +$> git annex list | head +here +|origin +||s3-PRIVATE +|||s3-PUBLIC +||||web +|||||bittorrent +|||||| +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-02_bold.mat +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-03_bold.mat + +``` + +workaround + + +``` +*$> head -n1 <(git-annex find --not --in here --fast) +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +``` + +"""]]
stalling find if piped
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn new file mode 100644 index 0000000000..58fea8f47a --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn @@ -0,0 +1,43 @@ +### Please describe the problem. + +``` +$> git-annex version | head -n 1 +git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05 + + +(git)smaug:/tmp/ds000113[master]git +$> time git-annex find --not --in here --fast | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat + +``` + +is hanging although IMHO should have exited right after spitting out first entry. Overall it takes under second to list them + +``` +$> time git-annex find --not --in here --fast > /dev/null +git-annex find --not --in here --fast > /dev/null 0.45s user 0.09s system 132% cpu 0.408 total + +``` + +but if piped -- stall! + +it did manage to quit once + +``` +$> git annex find --not --in here --fast | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +git-annex: <stdout>: hFlush: resource vanished (Broken pipe) + +``` + +with system wide installed 10.20250416 but I failed to reproduce + +### What steps will reproduce the problem? + +FWIW -- above on https://github.com/OpenNeuroDatasets/ds000113.git + + +I didn't check other commands , but they all should behave sane as not to stall if piped ! Would be great if there was a test to ensure that long term. + +[[!meta author=yoh]] +[[!tag projects/openneuro]]
disableremote private journal cleanup
Was able to reuse the code that drops dead remotes from the git-annex
branch to filter the remote uuid out of the private journal log files.
This makes me very happy about the design of that code.
Also there is a special case for --sameas to remove the config uuid from
remote.log.
Was able to reuse the code that drops dead remotes from the git-annex
branch to filter the remote uuid out of the private journal log files.
This makes me very happy about the design of that code.
Also there is a special case for --sameas to remove the config uuid from
remote.log.
diff --git a/Annex/Branch.hs b/Annex/Branch.hs
index d0196d8875..f1e1103948 100644
--- a/Annex/Branch.hs
+++ b/Annex/Branch.hs
@@ -21,6 +21,8 @@ module Annex.Branch (
forceUpdate,
updateTo,
get,
+ getLocal,
+ getLocal',
getHistorical,
getRef,
getUnmergedRefs,
@@ -1052,7 +1054,7 @@ overBranchFileContents' select go st = do
Nothing
| journalIgnorable st -> return Nothing
| otherwise ->
- overJournalFileContents' buf (handlestale branchsha) select
+ overJournalFileContents' buf False (handlestale branchsha) select
res <- catObjectStreamLsTree l (select' . getTopFilePath . Git.LsTree.file) g go'
`finally` liftIO (void cleanup)
return (res, branchsha)
diff --git a/Annex/Journal.hs b/Annex/Journal.hs
index 55eae36b28..74d85b0bc7 100644
--- a/Annex/Journal.hs
+++ b/Annex/Journal.hs
@@ -102,6 +102,15 @@ setJournalFile _jl ru file content = withOtherTmp $ \tmp -> do
-- exists
mv `catchIO` (const (createAnnexDirectory jd >> mv))
+deleteJournalFile :: JournalLocked -> RegardingUUID -> OsPath -> Annex ()
+deleteJournalFile _jl ru file = do
+ st <- getState
+ jd <- fromRepo =<< ifM (regardingPrivateUUID ru)
+ ( return (gitAnnexPrivateJournalDir st)
+ , return (gitAnnexJournalDir st)
+ )
+ liftIO $ removeWhenExistsWith removeFile (jd </> journalFile file)
+
newtype AppendableJournalFile = AppendableJournalFile (OsPath, OsPath)
{- If the journal file does not exist, it cannot be appended to, because
@@ -321,17 +330,32 @@ overJournalFileContents
-> Annex a
overJournalFileContents handlestale select go = do
buf <- liftIO newEmptyMVar
- go $ overJournalFileContents' buf handlestale select
+ go $ overJournalFileContents' buf False handlestale select
+
+{- Like overJournalFileContents, but only over files that are in the
+ - private journal. However, the file content still includes the public
+ - content, concacenated with the private content. -}
+overPrivateJournalFileContents
+ :: (OsPath -> L.ByteString -> Annex (L.ByteString, Maybe b))
+ -> (OsPath -> Maybe v)
+ -> (Annex (FileContents v b) -> Annex a)
+ -> Annex a
+overPrivateJournalFileContents handlestale select go = do
+ buf <- liftIO newEmptyMVar
+ go $ overJournalFileContents' buf True handlestale select
overJournalFileContents'
:: MVar ([OsPath], [OsPath])
+ -> Bool
-> (OsPath -> L.ByteString -> Annex (L.ByteString, Maybe b))
-> (OsPath -> Maybe a)
-> Annex (FileContents a b)
-overJournalFileContents' buf handlestale select =
+overJournalFileContents' buf onlyprivate handlestale select =
liftIO (tryTakeMVar buf) >>= \case
Nothing -> do
- jfs <- journalledFiles
+ jfs <- if onlyprivate
+ then return []
+ else journalledFiles
pjfs <- journalledFilesPrivate
drain jfs pjfs
Just (jfs, pjfs) -> drain jfs pjfs
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 4b000fd01f..6d2f4e20a3 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -10,8 +10,22 @@ module Command.DisableRemote where
import Command
import Remote
import Git
+import qualified Annex
import qualified Git.Remote.Remove
import qualified Git.Ref
+import Types.Remote
+import Annex.Journal
+import qualified Annex.Branch
+import Annex.Branch.Transitions
+import Types.Transitions
+import Logs
+import Logs.Remote.Pure
+import Logs.MapLog
+
+import Data.ByteString.Builder
+import qualified Data.Map as M
+import qualified Data.Set as S
+import qualified Data.ByteString.Lazy as L
cmd :: Command
cmd = withAnnexOptions [jsonOptions] $
@@ -27,6 +41,10 @@ start :: [String] -> CommandStart
start (remotename:[]) = byName' remotename >>= \case
Left err -> giveup err
Right r -> starting "disableremote" ai si $ do
+ uniqueuuid <- not
+ . any (\r' -> uuid r' == uuid r && name r' /= name r)
+ <$> remoteList
+
-- It would be good to remove export databases, fsck
-- databases, and transfer logs, but all of those are
-- uuid based, so would need to avoid deleting any if the
@@ -39,12 +57,7 @@ start (remotename:[]) = byName' remotename >>= \case
-- It would be good to remove the AuthToken used for a P2P
-- remote.
- -- If the remote is private, it would be good to remove
- -- the remote from remote.log and uuid.log in the private
- -- journal, and also to remove any private logs for the
- -- uuid. (Unless there are other remotes using the same
- -- uuid.)
-
+ cleanPrivateJournal r uniqueuuid
inRepo $ Git.Remote.Remove.remove remotename
removeRemoteTrackingBranches remotename
@@ -66,3 +79,67 @@ removeRemoteTrackingBranches remotename = do
inRepo $ Git.Ref.delete' b
where
branchprefix = "refs/remotes/" ++ remotename ++ "/"
+
+{- Remove a private remote's uuid from the private journal
+ - entirely when it is the only remote using that uuid.
+ -
+ - And, when the remote is a sameas remote, its config is stored
+ - under its config-uuid. Remove that from the private remote log.
+ -}
+cleanPrivateJournal :: Remote -> Bool -> Annex ()
+cleanPrivateJournal r uniqueuuid
+ | uniqueuuid == True = do
+ whenM (isprivateuuid (uuid r)) $ do
+ gc <- Annex.getGitConfig
+ let tc = filterBranch (\u -> u /= uuid r) gc
+ let handlestale = \_ b -> return (b, Nothing)
+ lockJournal $ \jl ->
+ overPrivateJournalFileContents handlestale Just
+ (go jl tc)
+ removeconfiguuid
+ | otherwise = removeconfiguuid
+ where
+ isprivateuuid u =
+ (\c -> u `S.member` annexPrivateRepos c)
+ <$> Annex.getGitConfig
+
+ go jl tc getfilecontents = getfilecontents >>= \case
+ Just (_, p, Just (b, _)) -> do
+ cleaner jl tc p b
+ go jl tc getfilecontents
+ Just (_, _, Nothing) ->
+ go jl tc getfilecontents
+ Nothing -> return ()
+
+ cleaner jl tc p b = case tc p b of
+ PreserveFile -> return ()
+ ChangeFile builder ->
+ setlocaljournal jl (RegardingUUID [uuid r]) p builder
+
+ removeconfiguuid = case remoteAnnexConfigUUID (gitconfig r) of
+ Nothing -> return ()
+ Just cu -> whenM (isprivateuuid cu) $
+ lockJournal $ \jl ->
+ getJournalFile jl (GetPrivate True) remoteLog >>= \case
+ JournalledContent b ->
+ scrub jl cu b
+ PossiblyStaleJournalledContent b ->
+ scrub jl cu b
+ NoJournalledContent ->
+ return ()
+ where
+ scrub jl cu b =
+ setlocaljournal jl (RegardingUUID [cu]) remoteLog $
+ buildRemoteConfigLog $ MapLog $ M.delete cu $
+ fromMapLog $ parseRemoteConfigLog b
+
+ setlocaljournal jl ru p builder =
+ let b' = toLazyByteString builder
+ in if L.null b'
+ then deleteJournalFile jl ru p
+ else do
+ b'' <- Annex.Branch.getLocal' (GetPrivate False) p
+ if b'' == b'
+ then deleteJournalFile jl ru p
+ else setJournalFile jl ru p builder
+
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
index faf7cec8c9..75cc0dd97d 100644
(Diff truncated)
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment b/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment new file mode 100644 index 0000000000..452dd0330e --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 11""" + date="2026-04-01T15:57:20Z" + content=""" +I have started a `disableremote` branch that adds a `git-annex +disableremote` command. That same command will be able to be used +for `ephemeral=yes`. + +As it stands, the command works, but there is some information about the +remote that it does not remove. I think for this purpose, +it needs to at least remove the config uuid of the sameas remote +from the private remote.log journal file. Otherwise that file will grow +by one line each time an ephemeral delegate is used. + +Removing cached annex creds, etc would also be good, but is not essential; they +would get overwritten on next use. And, since cached creds use a filename +based on the uuid, there's actually no way to know if a cached creds is for +an delegate or are creds used by the external special remote with the +same uuid. This actually seems like a bit of a bug with --sameas in +general, that if two sameas remotes both use cached creds, they will clash. +(Perhaps creds should use the config-uuid.) +"""]]
basic disableremote
There are several things it ought to delete, but doesn't yet. Still,
it's usable as far as it goes.
There are several things it ought to delete, but doesn't yet. Still,
it's usable as far as it goes.
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs index 6efd0b9e43..4b000fd01f 100644 --- a/Command/DisableRemote.hs +++ b/Command/DisableRemote.hs @@ -8,19 +8,15 @@ module Command.DisableRemote where import Command -import qualified Annex.SpecialRemote -import Annex.SpecialRemote.Config (nameField, sameasNameField) -import qualified Logs.Remote -import qualified Types.Remote as R -import qualified Remote -import Types.ProposedAccepted - -import qualified Data.Map as M +import Remote +import Git +import qualified Git.Remote.Remove +import qualified Git.Ref cmd :: Command cmd = withAnnexOptions [jsonOptions] $ command "disableremote" SectionSetup - "stops git-annex from using a remote" + "stop using a remote" paramName (withParams seek) @@ -28,5 +24,45 @@ seek :: CmdParams -> CommandSeek seek = withWords (commandAction . start) start :: [String] -> CommandStart -start ps@(name:[]) = error "TODO" +start (remotename:[]) = byName' remotename >>= \case + Left err -> giveup err + Right r -> starting "disableremote" ai si $ do + -- It would be good to remove export databases, fsck + -- databases, and transfer logs, but all of those are + -- uuid based, so would need to avoid deleting any if the + -- same uuid is still in use by another remote. + + -- It would be good to remove cred files, but there + -- is currently no way to list cred files belonging to a + -- remote. + + -- It would be good to remove the AuthToken used for a P2P + -- remote. + + -- If the remote is private, it would be good to remove + -- the remote from remote.log and uuid.log in the private + -- journal, and also to remove any private logs for the + -- uuid. (Unless there are other remotes using the same + -- uuid.) + + inRepo $ Git.Remote.Remove.remove remotename + removeRemoteTrackingBranches remotename + + next $ return True + where + ai = ActionItemOther (Just (UnquotedString remotename)) + si = SeekInput [remotename] start _ = giveup "Specify the remote's name." + +-- Remove any remote branches. +-- This is done because git remote remove only removes the configured +-- remote tracking branch, not other remote branches. +removeRemoteTrackingBranches :: String -> Annex () +removeRemoteTrackingBranches remotename = do + branches <- filter (\b -> branchprefix `isPrefixOf` fromRef b) + . map snd + <$> inRepo Git.Ref.list + forM_ branches $ \b -> + inRepo $ Git.Ref.delete' b + where + branchprefix = "refs/remotes/" ++ remotename ++ "/" diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn index 2b063fbff2..faf7cec8c9 100644 --- a/doc/git-annex-disableremote.mdwn +++ b/doc/git-annex-disableremote.mdwn @@ -1,6 +1,6 @@ # NAME -git-annex disableremote - stops git-annex from using a remote +git-annex disableremote - stop using a remote # SYNOPSIS @@ -8,16 +8,17 @@ git annex disableremote `name` # DESCRIPTION -Stops git-annex in the current repository from using a remote that was -earlier set up by `git-annex initremote` or `git-annex enableremote`. +This removes the remote from the local git configuration, the same as +`git remote remove` would. It removes remote tracking branches. +It also cleans up various other files in `.git/annex/` that belong to +or mention the remote. -This does not prevent the same remote from continuing to be used in other -clones of the repository. +Note that this does not currently delete any cached credentials for a +special remote or P2P peer. That may be improved in the future. -This removes the remote from the local git configuration, the same as -`git remote remove` would. It also cleans up various other files in -`.git/annex/` that mention the remote, including deleting any cached -credentials for the remote. +This does not prevent the same special remote from continuing to be used +in other clones of the repository, and it can be re-enabled for local +use with the `git-annex enableremote` command. # OPTIONS
stub in disableremote command
diff --git a/CHANGELOG b/CHANGELOG
index ecc6037a6e..e1eb0a8d81 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,6 +1,7 @@
git-annex (10.20260317) UNRELEASED; urgency=medium
* Fix annexUrl to inherit any password that is set in the remote url.
+ * disableremote: New command.
-- Joey Hess <id@joeyh.name> Mon, 23 Mar 2026 11:37:09 -0400
diff --git a/CmdLine/GitAnnex.hs b/CmdLine/GitAnnex.hs
index 7290b660e9..65b5fac680 100644
--- a/CmdLine/GitAnnex.hs
+++ b/CmdLine/GitAnnex.hs
@@ -58,6 +58,7 @@ import qualified Command.InitRemote
import qualified Command.EnableRemote
import qualified Command.ConfigRemote
import qualified Command.RenameRemote
+import qualified Command.DisableRemote
import qualified Command.EnableTor
import qualified Command.Multicast
import qualified Command.Expire
@@ -173,6 +174,7 @@ cmds testoptparser testrunner mkbenchmarkgenerator = map addGitAnnexCommonOption
, Command.EnableRemote.cmd
, Command.ConfigRemote.cmd
, Command.RenameRemote.cmd
+ , Command.DisableRemote.cmd
, Command.EnableTor.cmd
, Command.Multicast.cmd
, Command.Reinject.cmd
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
new file mode 100644
index 0000000000..6efd0b9e43
--- /dev/null
+++ b/Command/DisableRemote.hs
@@ -0,0 +1,32 @@
+{- git-annex command
+ -
+ - Copyright 2026 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+module Command.DisableRemote where
+
+import Command
+import qualified Annex.SpecialRemote
+import Annex.SpecialRemote.Config (nameField, sameasNameField)
+import qualified Logs.Remote
+import qualified Types.Remote as R
+import qualified Remote
+import Types.ProposedAccepted
+
+import qualified Data.Map as M
+
+cmd :: Command
+cmd = withAnnexOptions [jsonOptions] $
+ command "disableremote" SectionSetup
+ "stops git-annex from using a remote"
+ paramName
+ (withParams seek)
+
+seek :: CmdParams -> CommandSeek
+seek = withWords (commandAction . start)
+
+start :: [String] -> CommandStart
+start ps@(name:[]) = error "TODO"
+start _ = giveup "Specify the remote's name."
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
new file mode 100644
index 0000000000..2b063fbff2
--- /dev/null
+++ b/doc/git-annex-disableremote.mdwn
@@ -0,0 +1,50 @@
+# NAME
+
+git-annex disableremote - stops git-annex from using a remote
+
+# SYNOPSIS
+
+git annex disableremote `name`
+
+# DESCRIPTION
+
+Stops git-annex in the current repository from using a remote that was
+earlier set up by `git-annex initremote` or `git-annex enableremote`.
+
+This does not prevent the same remote from continuing to be used in other
+clones of the repository.
+
+This removes the remote from the local git configuration, the same as
+`git remote remove` would. It also cleans up various other files in
+`.git/annex/` that mention the remote, including deleting any cached
+credentials for the remote.
+
+# OPTIONS
+
+* `--json`
+
+ Enable JSON output. This is intended to be parsed by programs that use
+ git-annex.
+
+* `--json-error-messages`
+
+ Messages that would normally be output to standard error are included in
+ the JSON instead.
+
+* Also, the [[git-annex-common-options]](1) can be used.
+
+# SEE ALSO
+
+[[git-annex]](1)
+
+[[git-annex-initremote]](1)
+
+[[git-annex-enableremote]](1)
+
+[[git-annex-renameremote]](1)
+
+# AUTHOR
+
+Joey Hess <id@joeyh.name>
+
+Warning: Automatically converted into a man page by mdwn2man. Edit with care.
diff --git a/doc/git-annex-enableremote.mdwn b/doc/git-annex-enableremote.mdwn
index 7c0a70d558..a74f81f75b 100644
--- a/doc/git-annex-enableremote.mdwn
+++ b/doc/git-annex-enableremote.mdwn
@@ -86,6 +86,8 @@ has found didn't work before and gave up on using, setting
[[git-annex-renameremote]](1)
+[[git-annex-disableremote]](1)
+
# AUTHOR
Joey Hess <id@joeyh.name>
diff --git a/doc/git-annex-initremote.mdwn b/doc/git-annex-initremote.mdwn
index f4d5705308..beb2f3c96e 100644
--- a/doc/git-annex-initremote.mdwn
+++ b/doc/git-annex-initremote.mdwn
@@ -151,6 +151,8 @@ want to use `git annex renameremote`.
[[git-annex-renameremote]](1)
+[[git-annex-disableremote]](1)
+
# AUTHOR
Joey Hess <id@joeyh.name>
diff --git a/git-annex.cabal b/git-annex.cabal
index d02c567360..0d422341af 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -640,6 +640,7 @@ Executable git-annex
Command.Dead
Command.Describe
Command.DiffDriver
+ Command.DisableRemote
Command.Drop
Command.DropKey
Command.DropUnused
Added a comment: a note on potential use case(s)
diff --git a/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment b/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment new file mode 100644 index 0000000000..fa2daca752 --- /dev/null +++ b/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="a note on potential use case(s)" + date="2026-03-28T12:23:26Z" + content=""" +> because noone has requested it + +I came to this while considering using git-annex to importtree from rclone. + +The use case could be automation of workflow to work with docs on google drive or alike ([docflow](https://github.com/con/docflow)) or even more so for the [con/serve](https://con.github.io/serve/) \"concept\" since people do keep and collaborate via google drives and dropboxes. I do not have (yet) a clear idea on for which funded project this could apply, but it might become relevant/affordable one way or another if we guestimate the amount of work needed. Also the [protocol enhancement](http://git-annex.branchable.com/design/external_special_remote_protocol/export_and_import_appendix/) for import/export-tree already looks quite thought-through. +"""]]
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment index 8a901671bf..ff1e5ea0d4 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment @@ -13,4 +13,8 @@ Their config does not get written to the git-annex branch and their configuration prevents the user from accidentially using them. The traces that hit the disk are just an implementation detail which can easily be ignored. + +So I'm inclined to merge the branch as it stands. But that would mean that, +if ephemeral support is later added it would need to be a new extension eg +"DELEGATE-EPHEMERAL". """]]
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment new file mode 100644 index 0000000000..8a901671bf --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2026-03-26T16:08:11Z" + content=""" +The `delegate` branch now has the exension working. As it stands, I left +out the `ephemeral=yes|no`, so the delegate remote remains +in the git config and elsewhere after git-annex stops running, and will be +reused next time it's needed. + +I suspect that it might not matter that delegate remotes are not ephemeral. +Their config does not get written to the git-annex branch and their +configuration prevents the user from accidentially using them. The traces +that hit the disk are just an implementation detail which can easily be +ignored. +"""]]
temporarily remove ephemeral setting
The delegate extension seems to be fully implemented, save for this
setting. Which will be a lot of extra work to implement.
The delegate extension seems to be fully implemented, save for this
setting. Which will be a lot of extra work to implement.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index dc476852fe..e85333ae9f 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 82c54630ca..5318a0683c 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value ephemeral=yes|no [params]
+ DELEGATE type=value [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -12,25 +12,18 @@ and `--private` options. The delegate special remote inherits
the encryption settings, and uses the same annex-uuid. Its configuration
is not stored to the git-annex branch.
-With ephemeral=yes, the delegate special remote is created and exists only
-as long as the external special remote program is used by git-annex. Then
-it is removed. With ephemeral=no, the delegate special remote is cached for
-use next time, avoiding the overhead of initializing it again.
-
For example:
TRANSFER STORE somekey tmpfile
- DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+ DELEGATE type=directory directory=/mnt/disk/
This makes the annex object be stored in a directory special remote.
-Since initializing a directory special remote is inexpensive, it's
-made ephemeral.
When the key is later requested to be retrieved,
the same delegate can be used:
TRANSFER RETRIEVE somekey tmpfile
- DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+ DELEGATE type=directory directory=/mnt/disk/
----
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index d1834385d3..ce1ec411bf 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
fix link
diff --git a/doc/todo/import_tree_from_rsync_special_remote.mdwn b/doc/todo/import_tree_from_rsync_special_remote.mdwn index c31dd0b199..d65d6116d4 100644 --- a/doc/todo/import_tree_from_rsync_special_remote.mdwn +++ b/doc/todo/import_tree_from_rsync_special_remote.mdwn @@ -37,7 +37,7 @@ importtree, but there are several roadblocks: So, it seems that, importtree would need to be able to run commands other than rsync on the server. --[[Joey]] -Or, implement [[todo/importtree_only_remote]] and make rsync special +Or, implement [[todo/importtree_only_remotes]] and make rsync special remotes support either importtree or extporttree, but not both, which avoids that problem. --[[Joey]]
initial report on the dances with annexurl
diff --git a/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn b/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn new file mode 100644 index 0000000000..0decc22896 --- /dev/null +++ b/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn @@ -0,0 +1,71 @@ +### Please describe the problem. + +I am trying to orchestrate testing against a forgejo+aneksjo fixture instance in [datalad-fuse](https://github.com/datalad/datalad-fuse/pull/127) where I run forgejo+aneksjo in podman container mapping to a different (fixed) external ports. Overall verdict - "have difficulties". + + +One particular, potentially a core issue (besides that odd "push master first to get things rolling", likely to initiate smth on server side I guess), is that `annexurl`, even if I predefine it correct: + +```shell +❯ git annex version | head -n 1 +git-annex version: 10.20260316+git1-g768707adf4-1~ndall+1 +❯ git annex copy --to=forgejo testfile.bin < /dev/null +Username for 'http://127.0.0.1:41713': ^C +❯ tail -n 5 .git/config +[remote "forgejo"] + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://127.0.0.1:41713/git-annex-p2phttp +❯ git push forgejo master +Enumerating objects: 9, done. +Counting objects: 100% (9/9), done. +Delta compression using up to 20 threads +Compressing objects: 100% (8/8), done. +Writing objects: 100% (9/9), 1003 bytes | 1003.00 KiB/s, done. +Total 9 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0) +To http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + * [new branch] master -> master +❯ tail -n 5 .git/config +[remote "forgejo"] + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://127.0.0.1:41713/git-annex-p2phttp +``` + +it would get overridden by git-annex: + +```shell +❯ git annex copy --to=forgejo testfile.bin < /dev/null +copy testfile.bin (unable to connect to HTTP server: Network.Socket.connect: <socket: 30>: does not exist (Connection refused)) failed +copy: 1 failed +❯ tail -n 5 .git/config + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://localhost:3000/git-annex-p2phttp + annex-uuid = ab484350-97c4-48a7-9d3d-8321dc966cb4 +``` + +and fixing port is not enough: + +```shell +❯ sed -i -e 's,:3000,:41713,g' .git/config +❯ tail -n 5 .git/config + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://localhost:41713/git-annex-p2phttp + annex-uuid = ab484350-97c4-48a7-9d3d-8321dc966cb4 +❯ git annex copy --to=forgejo testfile.bin < /dev/null +Username for 'annex+http://localhost:41713/git-annex-p2phttp': ^C +``` + +seems that pointing to IP "fixes" it + +``` +❯ sed -i -e 's,localhost:41713,127.0.0.1:41713,g' .git/config +❯ git annex copy --to=forgejo testfile.bin < /dev/null +copy testfile.bin (to forgejo...) ok +(recording state in git...) +```
update
diff --git a/doc/todo/RawFilePath_conversion.mdwn b/doc/todo/RawFilePath_conversion.mdwn index cd0dec9b1a..444d6f322f 100644 --- a/doc/todo/RawFilePath_conversion.mdwn +++ b/doc/todo/RawFilePath_conversion.mdwn @@ -49,9 +49,8 @@ of the status. any noticiable performance impact. The `require_OsPath` branch removes the OsPath build flag, -and merging it would resolve this. That will need packagers to do some work -to package the libraries though. Or to upgrade ghc, since file-io and -os-string are boot libraries since ghc 9.12.2 and 9.10.1 respectively. +and merging it would resolve this. But it needs a newer version of file-io +than is in Debian yet. --[[Joey]] [[!tag confirmed]]
Added a comment: Re: registering multi-file torrent urls for existing files
diff --git a/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment b/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment new file mode 100644 index 0000000000..dc1b2cea7e --- /dev/null +++ b/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="miris" + avatar="http://cdn.libravatar.org/avatar/bd975774ecba53f7454a61d50fc7d8cc" + subject="Re: registering multi-file torrent urls for existing files" + date="2026-03-22T12:36:40Z" + content=""" +Coming back to this one, turns out this is feasible using `git annex registerurl`: + +``` +aria2c --show-files multi-file.torrent +# get the number of the file +git annex registerurl \"$(git annex lookupkey file.mp4)\" \"magnet:...#<number>\" +``` + +Works as intended; however, this will skip verification, so it's imperative that this is only used when you are 100% certain the file in the torrent is exactly the same your local one. :) +"""]]
diff --git a/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn b/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn new file mode 100644 index 0000000000..11900c6a50 --- /dev/null +++ b/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn @@ -0,0 +1,7 @@ +What is the best way to use adb special remote with device connected and paired over wifi? It uses random ports for connection every time, so I haven't even bothered yet to try IP:PORT as a serialnumber. + + +``` +$ adb devices -l +XXXXX:12345 device product:XXX model:XXX device:XXX transport_id:1 +```
mih seems stated that it works for him just fine.
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index a657aa92b1..6f3492dc88 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -106,4 +106,4 @@ FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile bu [[!meta author=yoh]] -[[!tag projects/FZJ]] +
update
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment index e943321592..05b7cf0035 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment @@ -3,20 +3,21 @@ subject="""comment 1""" date="2026-03-17T17:47:39Z" content=""" -Note that forgejo does support raw file access, and I expect that supports -range requests for annex objected. +Note that forgejo does support raw file access, and I expect that it +supports range requests for annex objects. -The endpoint where it might make sense to support range requests -is `https://git-annex.branchable.com/design/p2p_protocol_over_http/#index1h3` +The p2phttp endpoint where it might make sense to support range requests +is `/git-annex/$uuid/key/$key` When p2phttp is proxying to a special remote, it would need to download the whole file from the special remote even if the range request was for a small part. So I don't think it should be supported for proxying. One way to implement this might be to use Servant.Server.StaticFiles -to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint -redirect requests to that. That uses warp's built-in static file serving, -which supports range requests. +with a StaticSettings ssLookupFiles that returns the file location under +`.git/annex/objects/` (or even a location in another git-annex repository +when proxying to one, eg a cluster node.) That uses warp's built-in static +file serving, which supports range requests. But how to handle authentication? It seems like the only way would be to reimplement p2phttp's authentication checking as
comment
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment new file mode 100644 index 0000000000..e943321592 --- /dev/null +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-03-17T17:47:39Z" + content=""" +Note that forgejo does support raw file access, and I expect that supports +range requests for annex objected. + +The endpoint where it might make sense to support range requests +is `https://git-annex.branchable.com/design/p2p_protocol_over_http/#index1h3` + +When p2phttp is proxying to a special remote, it would need to download +the whole file from the special remote even if the range request was for a +small part. So I don't think it should be supported for proxying. + +One way to implement this might be to use Servant.Server.StaticFiles +to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint +redirect requests to that. That uses warp's built-in static file serving, +which supports range requests. + +But how to handle authentication? It seems like +the only way would be to reimplement p2phttp's authentication checking as +WAI middleware. +"""]] diff --git a/doc/todo/p2phttp_ranged_requests.mdwn b/doc/todo/p2phttp_ranged_requests.mdwn deleted file mode 100644 index 674125a12c..0000000000 --- a/doc/todo/p2phttp_ranged_requests.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -The p2phttp endpoint for raw download of a key does not currently support -range requests or other such things. While it's not always possible for -p2phttp to support that, eg when proxying to a special remote it cannot, it -would be useful if it supported it in configurations where it is possible -to do so. - -One way to implement this might be to use Servant.Server.StaticFiles -to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint -redirect requests to that. And reject ranged requests when proxying. - -But how to handle authentication? It seems like -the only way would be to reimplement p2phttp's authentication checking as -WAI middleware. --[[Joey]]
removed association with FZJ project as out of current scope/need
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn index bbbbb8775b..368d470e88 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -9,4 +9,3 @@ and It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. [[!meta author=yoh]] -[[!tag projects/FZJ]]
todo
diff --git a/doc/todo/p2phttp_ranged_requests.mdwn b/doc/todo/p2phttp_ranged_requests.mdwn new file mode 100644 index 0000000000..674125a12c --- /dev/null +++ b/doc/todo/p2phttp_ranged_requests.mdwn @@ -0,0 +1,13 @@ +The p2phttp endpoint for raw download of a key does not currently support +range requests or other such things. While it's not always possible for +p2phttp to support that, eg when proxying to a special remote it cannot, it +would be useful if it supported it in configurations where it is possible +to do so. + +One way to implement this might be to use Servant.Server.StaticFiles +to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint +redirect requests to that. And reject ranged requests when proxying. + +But how to handle authentication? It seems like +the only way would be to reimplement p2phttp's authentication checking as +WAI middleware. --[[Joey]]
reassign to existing FZJ
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index 5b19dcb9f3..a657aa92b1 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -106,4 +106,4 @@ FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile bu [[!meta author=yoh]] -[[!tag projects/trr379.de]] +[[!tag projects/FZJ]]
reassign to already existing FZJ
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn index 985fd7d0a2..bbbbb8775b 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -9,4 +9,4 @@ and It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. [[!meta author=yoh]] -[[!tag projects/trr379.de]] +[[!tag projects/FZJ]]
TODO on range requests
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn new file mode 100644 index 0000000000..985fd7d0a2 --- /dev/null +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -0,0 +1,12 @@ +ATM according to [.../design/p2p_protocol_over_http/](https://git-annex.branchable.com/design/p2p_protocol_over_http/) + +> Request headers are currently ignored, so eg Range requests are not supported. (This would be possible to implement, up to a point.) + +and + +> Note that there is no Content-Length header. + +It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. + +[[!meta author=yoh]] +[[!tag projects/trr379.de]]
added server side log and relation to trr379 project
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index ddf14504b5..5b19dcb9f3 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -1,6 +1,6 @@ ### Please describe the problem. -Finally got to try the neat p2p, ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. +Finally got to try the neat p2p (inspired by trr379 use-case raised in matrix), ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. Unfortunately <details> @@ -77,6 +77,20 @@ curl: (52) Empty reply from server curl: (56) Recv failure: Connection reset by peer ``` +edit: added the server side: + +``` +❯ git annex version --raw; echo; git annex --debug p2phttp --port 8081 --wideopen + +10.20260316+git1-g768707adf4-1~ndall+1 +[2026-03-17 10:16:22.435272107] (Utility.Process) process [2104460] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2026-03-17 10:16:22.441213037] (Utility.Process) process [2104460] done ExitSuccess +[2026-03-17 10:16:22.441610323] (Utility.Process) process [2104461] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2026-03-17 10:16:22.446123885] (Utility.Process) process [2104461] done ExitSuccess +[2026-03-17 10:16:22.447208284] (Utility.Process) process [2104462] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2026-03-17 10:16:22.453605123] (Annex.Branch) read proxy.log + +``` </details> @@ -89,3 +103,7 @@ this was on a local clone of https://datasets.datalad.org/dbic/QA/.git with `sou 10.20260316+git1-g768707adf4-1~ndall+1 - not ok; 10.20251029 - not ok. FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile but I think it was due to some change in behavior affecting RIA archives (I did not have yet chance to troubleshoot manually to report) , thus unrelated here. + + +[[!meta author=yoh]] +[[!tag projects/trr379.de]]
reporting about inability for p2p with recent version
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn new file mode 100644 index 0000000000..ddf14504b5 --- /dev/null +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -0,0 +1,91 @@ +### Please describe the problem. + +Finally got to try the neat p2p, ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. +Unfortunately + +<details> +<summary>whenever p2phttp worked fine (for a full file) request on 10.20251029</summary> + +```shell +❯ curl http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 96.62M 0 96.62M 0 0 563.2M 0 0 +❯ sha256sum out.tgz +02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f out.tgz + +``` + +and on server side: + +``` +❯ git annex version --raw; echo; git annex --debug p2phttp --port 8081 --wideopen +10.20251029 +[2026-03-17 10:12:16.496986375] (Utility.Process) process [2098931] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2026-03-17 10:12:16.499045819] (Utility.Process) process [2098931] done ExitSuccess +[2026-03-17 10:12:16.499500423] (Utility.Process) process [2098934] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2026-03-17 10:12:16.501265293] (Utility.Process) process [2098934] done ExitSuccess +[2026-03-17 10:12:16.502428567] (Utility.Process) process [2098935] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2026-03-17 10:12:16.505874247] (Annex.Branch) read proxy.log +[2026-03-17 10:12:26.349175199] (P2P.IO) [http client] [ThreadId 22] P2P > GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz +[2026-03-17 10:12:26.349539971] (P2P.IO) [http server] [ThreadId 19] P2P < GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz +[2026-03-17 10:12:26.350670225] (Utility.Process) process [2099177] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"] +[2026-03-17 10:12:26.366674858] (Utility.Process) process [2099177] done ExitSuccess +[2026-03-17 10:12:26.367081645] (Utility.Process) process [2099178] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"] +[2026-03-17 10:12:26.369153188] (Utility.Process) process [2099178] done ExitSuccess +[2026-03-17 10:12:26.370404636] (P2P.IO) [http server] [ThreadId 19] P2P > DATA 101318091 +[2026-03-17 10:12:26.370496387] (P2P.IO) [http client] [ThreadId 22] P2P < DATA 101318091 +[2026-03-17 10:12:26.518654447] (P2P.IO) [http client] [ThreadId 22] P2P > SUCCESS +[2026-03-17 10:12:26.518800582] (P2P.IO) [http server] [ThreadId 19] P2P < SUCCESS + +``` +</details> + + +<details> +<summary> +with the most recent version 10.20260316+git1-g768707adf4-1~ndall+1 we get silent treatment -- no content provided and nothing in the --debug log +</summary> + + +``` +❯ curl http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 0 +curl: (52) Empty reply from server +❯ curl -v http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz +* Host localhost:8081 was resolved. +* IPv6: ::1 +* IPv4: 127.0.0.1 + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 0* Trying [::1]:8081... +* connect to ::1 port 8081 from ::1 port 56576 failed: Connection refused +* Trying 127.0.0.1:8081... +* Established connection to localhost (127.0.0.1 port 8081) from 127.0.0.1 port 58866 +* using HTTP/1.x +> GET /git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz HTTP/1.1 +> Host: localhost:8081 +> User-Agent: curl/8.18.0 +> Accept: */* +> +* Request completely sent off +* Recv failure: Connection reset by peer + +* closing connection #0 +curl: (56) Recv failure: Connection reset by peer +``` + +</details> + + +### What steps will reproduce the problem? + +this was on a local clone of https://datasets.datalad.org/dbic/QA/.git with `sourcedata/sub-qa64/ses-20240715/func/sub-qa64_ses-20240715_acq-faX77_bold.dicom.tgz` content + +### What version of git-annex are you using? On what operating system? + +10.20260316+git1-g768707adf4-1~ndall+1 - not ok; 10.20251029 - not ok. + +FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile but I think it was due to some change in behavior affecting RIA archives (I did not have yet chance to troubleshoot manually to report) , thus unrelated here.
improve docs to close bug
diff --git a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn index ae34d49200..a9ad943cf8 100644 --- a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn +++ b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn @@ -45,3 +45,5 @@ local repository version: 10 [[!tag projects/ICE4]] + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment new file mode 100644 index 0000000000..6c45a883ab --- /dev/null +++ b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-03-17T13:39:50Z" + content=""" +Updated documentation. +"""]] diff --git a/doc/git-annex-export.mdwn b/doc/git-annex-export.mdwn index 0ec85668d8..c87051c07c 100644 --- a/doc/git-annex-export.mdwn +++ b/doc/git-annex-export.mdwn @@ -59,12 +59,22 @@ tell it what branch to track. For example: git config remote.myremote.annex-tracking-branch master git annex push myremote -You can combine using `git annex export` to send changes to a special +When the special remote is not also configured with `importtree=yes`, +git-annex does not try to preserve other files that may be written to the +special remote by other means. Exporting a tree of files to a special remote +will overwrite any files already stored on it with the same filenames. +In some cases, an update to what is exported that deletes a subdirectory +will delete not only the exported files that were in that subdirectory, +but any other files that might have been written to the same subdirectory +by other means. + +When the special remote is also configured with `importtree=yes`, +you can combine using `git annex export` to send changes to a special remote with `git annex import` to fetch changes from a special remote. -When a file on a special remote has been modified by software other than -git-annex, exporting to it will not overwrite the modified file, and the -export will not succeed. You can resolve this conflict by using -`git annex import`. +In this case, when a file on a special remote has been modified by +software other than git-annex, exporting to it will not overwrite the +modified file, and the export will not succeed. You can resolve this +conflict by using `git annex import`. (Some types of special remotes such as S3 with versioning may instead let an export overwrite the modified file; then `git annex import`
correct branch name
diff --git a/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment index f4a292dd86..fa0d3f7a4d 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment @@ -3,7 +3,7 @@ subject="""comment 9""" date="2026-03-05T19:00:16Z" content=""" -Started developing this in the `ephemeral` branch. +Started developing this in the `delegate` branch. It seems to also make sense to allow DELEGATE as a response to WHEREIS.
add news item for git-annex 10.20260316
diff --git a/doc/news/version_10.20251029.mdwn b/doc/news/version_10.20251029.mdwn deleted file mode 100644 index b98b28583c..0000000000 --- a/doc/news/version_10.20251029.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -git-annex 10.20251029 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * Support ssh remotes with '#' and '?' in the path to the repository, - the same way git does. - * assistant: Fix reversion that caused files to be added locked by - default."""]] \ No newline at end of file diff --git a/doc/news/version_10.20260316.mdwn b/doc/news/version_10.20260316.mdwn new file mode 100644 index 0000000000..7ca6f4e414 --- /dev/null +++ b/doc/news/version_10.20260316.mdwn @@ -0,0 +1,17 @@ +git-annex 10.20260316 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * Added CHECKPRESENT-URL extension to the external special remote protocol. + * Fix reversion in previous version that caused auto-initializing of + local git remotes that have annex-ignore set. + * Fix bug that caused git credential to be rejected when a http request + failed for some reason other than 401. + * Importing from the directory special remote will no longer add sizes + to keys, which overrode backends that generate unsized keys. + * Fix retrival from http git remotes of keys with '%' in their names. + * Fix behavior when initremote is used with --sameas= + combined with --private. + * web, S3, git: Fix bugs in checking if content is present on a remote + when configuration does not allow accessing it. + * httpalso: Fix bugs in handling content not being present on the remote. + * adb: Avoid deleting contents of a non-empty directory when + removing the last exported file from the directory. + * Improve display of http exceptions."""]] \ No newline at end of file
response
diff --git a/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment b/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment new file mode 100644 index 0000000000..456d576eff --- /dev/null +++ b/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-03-16T10:48:29Z" + content=""" +I have not looked at borg 2 in any detail, but it seems they are trying to +keep the CLI to some extent the same. So it seems it would depend on whether +CLI changes break something git-annex relies on. + +I'd treat any incompatability as a [[bug|bugs]] or [[todo]] on the git-annex +side, so if you try it and find problems, please report them. +"""]]
Added a comment: borg 2.0
diff --git a/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment b/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment new file mode 100644 index 0000000000..40c5e76b46 --- /dev/null +++ b/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="nadir" + avatar="http://cdn.libravatar.org/avatar/2af9174cf6c06de802104d632dc40071" + subject="borg 2.0" + date="2026-03-15T13:27:42Z" + content=""" +With borg 2.0 on the horizon, I was wondering how support for that would look should that be planned. + +With how much has changed and the necessity to create new repos, it might make the most sense to create a separate borg2 remote, but I have now idea. Mostly just curious if there are any plans for that at all. +"""]]
update
diff --git a/doc/thanks/list b/doc/thanks/list index 0c9056062d..0e8eb59c97 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -129,3 +129,4 @@ Andrew Poelstra, joshingly, Melody Tolly, username, +Steffen Vogel,
update
diff --git a/doc/thanks/list b/doc/thanks/list index 0a65388abb..0c9056062d 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -128,3 +128,4 @@ mpol, Andrew Poelstra, joshingly, Melody Tolly, +username,