Recent changes to this wiki:
Added a comment
diff --git a/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment b/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment new file mode 100644 index 0000000000..911e23259c --- /dev/null +++ b/doc/bugs/s3_imported_branch_is___34__git_buggy__34____58____bad_blobs/comment_11_2db0adbd5d15eba0442afec572a84413._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 11" + date="2026-04-15T19:55:19Z" + content=""" +FTR: fixed in [10.20251215-51-g69e6c4d024 AKA 10.20260115~59](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=69e6c4d024dcff7c2f8ea1a2ed3b483a86b2cc7d) +"""]]
Added a comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment new file mode 100644 index 0000000000..072cc430af --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_5_640df954a31d87bcda0dbfe4463e2a4b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2026-04-13T17:30:16Z" + content=""" +FWIW -- never returned, doing on another box -- here is a complete command : `git clone https://github.com/dandisets/000026.git && cd 000026 && git annex init && git annex find --not --in here --fast | head -n 1` -- stalled as well. I wonder if replicates for you. Also there with `2>/dev/null` returned right away, but on rerun without - stalled again. +"""]]
fix sanitizeTopLevelExceptionMessages
Avoid dying of an exception when when stdout gets closed by eg head(1), and
avoid a crash loop when stderr is closed and git-annex dies of an
exception. Fixes reversion introduced in version 10.20230407.
Note that giveup sanitizes escape characters, so
sanitizeTopLevelExceptionMessages can just use it to rethrow
the exception. But the exception type information is lost, so
an exception caused by sigPIPE would be displayed.
I don't entirely understand the crash loop, but it seemed to be caused
by the exitWith $ ExitFailure 1. Which is not necessary when using
giveup.
It's a bit scary to mess with the sigPIPE handler, but restoring the
Default does get the behavior we want. If that turned out to cause a
problem though, that part of this could be reverted, and git-annex would
only display the kind of ugly error message when piped to head.
Sponsored-by: Dartmouth College's OpenNeuro project
Avoid dying of an exception when when stdout gets closed by eg head(1), and
avoid a crash loop when stderr is closed and git-annex dies of an
exception. Fixes reversion introduced in version 10.20230407.
Note that giveup sanitizes escape characters, so
sanitizeTopLevelExceptionMessages can just use it to rethrow
the exception. But the exception type information is lost, so
an exception caused by sigPIPE would be displayed.
I don't entirely understand the crash loop, but it seemed to be caused
by the exitWith $ ExitFailure 1. Which is not necessary when using
giveup.
It's a bit scary to mess with the sigPIPE handler, but restoring the
Default does get the behavior we want. If that turned out to cause a
problem though, that part of this could be reverted, and git-annex would
only display the kind of ugly error message when piped to head.
Sponsored-by: Dartmouth College's OpenNeuro project
diff --git a/CHANGELOG b/CHANGELOG
index a791dfdaa8..aa29f974f7 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,10 @@ git-annex (10.20260317) UNRELEASED; urgency=medium
* disableremote: New command.
* Add DELEGATE extension to the external special remote protocol.
* Improve UUID sanitization.
+ * Avoid dying of an exception when when stdout gets closed by eg head(1),
+ and avoid a crash loop when stderr is closed and git-annex dies of an
+ exception.
+ Fixes reversion introduced in version 10.20230407.
-- Joey Hess <id@joeyh.name> Mon, 23 Mar 2026 11:37:09 -0400
diff --git a/Messages.hs b/Messages.hs
index 704d5cfeac..25b129e93d 100644
--- a/Messages.hs
+++ b/Messages.hs
@@ -1,6 +1,6 @@
{- git-annex output messages
-
- - Copyright 2010-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -65,6 +65,9 @@ import qualified Data.ByteString.Char8 as S8
import System.Exit
import qualified Control.Monad.Catch as M
import Data.String
+#ifndef mingw32_HOST_OS
+import System.Posix.Signals
+#endif
import Common
import Types
@@ -355,15 +358,23 @@ mkPrompter = getConcurrency >>= \case
{- Catch all (non-async and not ExitCode) exceptions and display,
- sanitizing any control characters in the exceptions.
-
- - Exits nonzero on exception, so should only be used at topmost level.
+ - Should only be used at topmost level.
-}
sanitizeTopLevelExceptionMessages :: IO a -> IO a
-sanitizeTopLevelExceptionMessages a = a `catches`
- ((M.Handler (\ (e :: ExitCode) -> throwM e)) : nonAsyncHandler go)
+sanitizeTopLevelExceptionMessages a = do
+#ifndef mingw32_HOST_OS
+ -- By default ghc Ignores sigPIPE, and then does not display
+ -- exceptions like <stdout>: hFlush: resource vanished (Broken pipe)
+ --
+ -- Since this would display such exceptions, instead restore the
+ -- Default sigPIPE behavior, which is for the program to
+ -- immediately exit.
+ void $ installHandler sigPIPE Default Nothing
+#endif
+ a `catches`
+ ((M.Handler (\ (e :: ExitCode) -> throwM e)) : nonAsyncHandler go)
where
- go e = do
- hPutStrLn stderr $ safeOutput $ toplevelMsg (show e)
- exitWith $ ExitFailure 1
+ go e = giveup $ show e
{- Used to only run an action that displays a message after the specified
- number of steps. This is useful when performing an action that can
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
index 58fea8f47a..0558fc1204 100644
--- a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn
@@ -41,3 +41,5 @@ I didn't check other commands , but they all should behave sane as not to stall
[[!meta author=yoh]]
[[!tag projects/openneuro]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
index 529216f696..789fdcdc0f 100644
--- a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment
@@ -12,12 +12,9 @@ pipe exception, and so the crash loop never happens when stderr is closed.
That is relatively new; it was added in git-annex 10.20230407.
And apparently what ghc's default exception display mechanism does is
-actually to ignore the broken pipe exception for stdout. Which leads
-to the more typical unix behavior of silently stopping on SIGPIPE.
-See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html>
+actually to avoid displaying anything for the broken pipe exception for
+stdout. Which leads to the more typical unix behavior of silently
+stopping on SIGPIPE.
-Note that the loop when displaying an exception fails with an exception due
-to stderr being closed still seems to be the fault of ghc/base, not
-git-annex. All that sanitizeTopLevelExceptionMessages does is cause the
-exception to be displayed.
+See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html>
"""]]
comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment new file mode 100644 index 0000000000..529216f696 --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_2e95dcff2f5776f97eaad22e8b284f24._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-04-13T15:22:17Z" + content=""" +Turns out that a git-annex that does not +use sanitizeTopLevelExceptionMessages will not have this behavior. + +Without that, `git-annex find | head -n1` does not display the broken +pipe exception, and so the crash loop never happens when stderr is closed. + +That is relatively new; it was added in git-annex 10.20230407. + +And apparently what ghc's default exception display mechanism does is +actually to ignore the broken pipe exception for stdout. Which leads +to the more typical unix behavior of silently stopping on SIGPIPE. +See also <https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html> + +Note that the loop when displaying an exception fails with an exception due +to stderr being closed still seems to be the fault of ghc/base, not +git-annex. All that sanitizeTopLevelExceptionMessages does is cause the +exception to be displayed. +"""]]
Added a comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment new file mode 100644 index 0000000000..037cabf9dc --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_3_9c1bd4016e326cf9e100a697798382df._comment @@ -0,0 +1,37 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 3" + date="2026-04-13T15:56:23Z" + content=""" +> I'm curious if you also ran git-annex with FD 2 closed? + +how do I discover that ? it worked fine if I redirect stderr: + +``` +$> git annex find --in here 2>/dev/null | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat + +$> echo $? +0 + +``` + +and damn it -- after that it started to work in that shell: + +``` +$> git annex find --in here | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +git-annex: <stdout>: hFlush: resource vanished (Broken pipe) +``` + +:-/ I think it relates to dataset size as if I go to other small datasets in that shell now -- it returns fine as expected, but in huge `/mnt/btrfs/datasets/datalad/crawl/dandi/dandisets/000026` -- still running (likely more stuff spit out to stdout), even with + +``` +(git)smaug:/mnt/btrfs/datasets/datalad/crawl/dandi/dandisets/000026[draft]git +$> git annex find --not --in here --fast 2>/dev/null | head -n 1 +derivatives/Depth_normalized_OCT_volume/I38/sub-I38_ses-OCT_sample-BrocaAreaS01_depth-normed_OCT.ome.tiff +``` + +I will keep it going to see if ever returns, will check in an hour. FWIW -- that dataset is [here](https://github.com/dandisets/000026) +"""]]
comment
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment
new file mode 100644
index 0000000000..13ff17f759
--- /dev/null
+++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_2_a906714f7e4ab7d9e50745f60e6da5d0._comment
@@ -0,0 +1,36 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2026-04-13T14:29:51Z"
+ content="""
+I was able to reproduce this, but only when I ran git-annex with FD 2
+closed.
+
+ perl -e 'close(STDERR); system("git-annex find --not --in here --fast | head -n 1")'
+
+I'm curious if you also ran git-annex with FD 2 closed?
+
+I suspect what must be happening is that the exception handler crashes when
+outputting to stderr, which causes an exception to be thrown,
+leading to a crash loop.
+
+If so, it would seem likely to be a bug in ghc/base. And I'd not be surprised to find
+such a bug somewhere in its bug tracker. In fact, I almost remember finding this same
+behavior before, which may have helped me guess this due was FD 2 being closed.
+
+With that said, I've not been able to reproduce the behavior yet with a simpler
+haskell program like this one:
+
+ import System.IO
+
+ main = do
+ print "foo"
+ hFlush stdout
+ print "bar"
+ hFlush stdout
+ main
+
+But, that simple program also doesn't throw
+"hFlush: resource vanished (Broken pipe)" when piped to `head -n1`,
+so it's not quite replicating what git-annex does.
+"""]]
comment
diff --git a/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment b/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment new file mode 100644 index 0000000000..bdee883797 --- /dev/null +++ b/doc/special_remotes/S3/comment_41_138b9c48237626506b42a9f133612a40._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""Re: git-annex: potential data loss with initremote and push with S3 special remotes.""" + date="2026-04-13T13:59:28Z" + content=""" +That needs to be a bug report. I have copied it to one here: +[[bugs/S3_pointed_to_minio_console_potential_data_loss]] +"""]]
comment
diff --git a/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment new file mode 100644 index 0000000000..3e13fe424f --- /dev/null +++ b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss/comment_1_de78ed6d5da146b31516c0c830a5742d._comment @@ -0,0 +1,28 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-04-13T14:01:37Z" + content=""" +It's arguably not git-annex's fault if it was pointed to an API endpoint +that behaves enough like S3 to make it seems like it's stored data, +but does not store the data. + +With that said, for there to be data loss here, the file would need to be +dropped from the local repository, relying on the copy "stored" in S3. +So the S3 special remote's checkPresent could be improved to prevent +such a bad endpoint from being treated as containing the content of an +object. + +For S3, checkPresent does pass in the VersionID when git-annex knows one. +(Which that doesn't help if the API endpoint ignores that header.) +What it does not do is check the response from S3 for a VersionID or ETag. +Improving that seems like a possible way to avoid this kind of problem. + +It does check the ETag when the S3 remote is configured with +exporttree=true. + +As for the idea of checking the annex-uuid write by reading the file back, +the difficulty with that is S3 `DEEP_ARCHIVE` and similar can have an +hours-long delay to get back out a file that is already stored in the +bucket. Also, the annex-uuid file is not used for exporttree=yes remotes. +"""]]
promote comment to bug report
diff --git a/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn
new file mode 100644
index 0000000000..51b043b042
--- /dev/null
+++ b/doc/bugs/S3_pointed_to_minio_console_potential_data_loss.mdwn
@@ -0,0 +1,24 @@
+A colleague used a wrong config, which was pointing to minio console rather than the S3 endpoint. When they ran initremote, the console wrongfully replied 200-OK when PUTting the annex-uuid file, same when they then pushed the data. The minio console always redirect to a login page, and doesn't fail on PUT ( which is non-compliant ). So the dataset recorded all the data being present in that remote, while there was no trace of any buckets or objects in the S3.
+
+## steps to reproduce:
+
+```
+git init test_s3
+cd test_s3/
+git-annex init
+export AWS_ACCESS_KEY_ID=john AWS_SECRET_ACCESS_KEY=doe
+git annex initremote -d test_remote host=\"play.min.io\" bucket=\"test_bucket\" type=S3 encryption=none autoenable=true port=9443 protocol=https chunk=1GiB requeststyle=pathecho test > test_annexed_file
+git-annex add test_annexed_file
+git commit -m 'add annexed file'
+git-annex copy --fast --to test_remote
+```
+
+I am showing it with `--fast` flag here, as this is what datalad uses by default. Without `--fast`, it fails with (HeaderException {headerErrorMessage = \"ETag missing\"}) failed which is better.
+
+So to sum it up, the unfortunate circumstances are:
+
+1. the initremote PUT of annex-uuid is not performing check that the annex-uuid file was effectively pushed in a bucket.
+2. minio console replies with 200-OK for all http requests
+3. datalad uses `push --fast` by default, which recorded files as being pushed without performing a HEAD after push. I guess that's for performance reason, but that is dangerous if a server or reverse-proxy ends-up responding 200-OK to all requests after init.
+
+Thanks for your help!
Added a comment
diff --git a/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment b/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment new file mode 100644 index 0000000000..075b3b1b14 --- /dev/null +++ b/doc/todo/forget_dead_keys/comment_2_574c34d5fdc7b141970459af5985ff67._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2026-04-10T13:03:09Z" + content=""" +FWIW, I think this might be useful for openneuro as I think we run into cases (e.g. ds000113) where we have records on some elderly keys (such as SHA1--771e0eea7ceb32216a5a06c89c50d1f02bc79d6d) for which I think we no longer have any commit in the history pointing to them or even if we do -- we have no such key exported in the bucket (I think). +"""]]
done with DELEGATE extension
diff --git a/doc/todo/Ephemeral_special_remotes.mdwn b/doc/todo/Ephemeral_special_remotes.mdwn index 7082e52512..79fce6c3d6 100644 --- a/doc/todo/Ephemeral_special_remotes.mdwn +++ b/doc/todo/Ephemeral_special_remotes.mdwn @@ -12,3 +12,5 @@ A use case would be to have an "orchestration" special remotes that maybe repres in some way also an alternative to the `sameas` approach, where the alternatives are hidden in the implementation of a special remote, rather than in *each* repository. [[!tag projects/INM7]] + +> [[done]]! --[[Joey]] diff --git a/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment b/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment new file mode 100644 index 0000000000..8610d3b9b6 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_13_7527e441ff29366ab5a289090767f930._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 13""" + date="2026-04-08T19:22:38Z" + content=""" +Finished implementing this! +"""]]
implement DELEGATE with ephemeral=yes
Factored Annex.DisableRemote out of Command.DisableRemote
Factored Annex.DisableRemote out of Command.DisableRemote
diff --git a/Annex/DisableRemote.hs b/Annex/DisableRemote.hs
new file mode 100644
index 0000000000..bcaed58d03
--- /dev/null
+++ b/Annex/DisableRemote.hs
@@ -0,0 +1,189 @@
+{- disable a remote
+ -
+ - Copyright 2026 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+{-# LANGUAGE OverloadedStrings #-}
+
+module Annex.DisableRemote (disableRemote) where
+
+import Git
+import Annex.Common
+import qualified Annex
+import qualified Git.Remote.Remove
+import qualified Git.Ref
+import Types.Remote
+import Annex.Journal
+import qualified Annex.Branch
+import Annex.Branch.Transitions
+import Types.Transitions
+import Logs
+import Logs.Remote.Pure
+import Logs.MapLog
+import Logs.Transfer
+import Types.Transfer
+import qualified Database.Export
+import qualified Database.Fsck
+import qualified Database.RepoSize
+import qualified Database.ContentIdentifier
+import qualified Utility.OsString as OS
+
+import Data.ByteString.Builder
+import qualified Data.Map as M
+import qualified Data.Set as S
+import qualified Data.ByteString.Lazy as L
+
+disableRemote :: Remote -> RemoteName -> [Remote] -> Annex ()
+disableRemote r remotename remotelist = do
+ let uniqueuuid = not $
+ any (\r' -> uuid r' == uuid r && name r' /= name r) remotelist
+
+ when uniqueuuid $
+ -- If there are transfers to/from the remote still
+ -- running, this fails. That's why it's run early.
+ removeTransferLogs (uuid r)
+
+ cleanPrivateJournal r uniqueuuid
+
+ when uniqueuuid $ do
+ -- Each uuid has its own export and fsck database,
+ -- so always remove them, so long as this is the
+ -- only remote using this uuid.
+ Database.Export.removeDb (uuid r)
+ Database.Fsck.removeDb (uuid r)
+ -- These databases are updated from information
+ -- in the git-annex branch, so there is no point in
+ -- removing the uuid from them unless it's private.
+ whenM (isPrivateUUID (uuid r)) $ do
+ Database.RepoSize.removeUUID (uuid r)
+ Database.ContentIdentifier.removeUUID (uuid r) True
+
+ removeFsckState (uuid r)
+ removeImportLog (uuid r)
+ removeCredFiles (uuid r)
+
+ inRepo $ Git.Remote.Remove.remove remotename
+ removeRemoteTrackingBranches remotename
+
+-- Remove any remote branches.
+-- This is done because git remote remove only removes the configured
+-- remote tracking branch, not other remote branches.
+removeRemoteTrackingBranches :: String -> Annex ()
+removeRemoteTrackingBranches remotename = do
+ branches <- filter (\b -> branchprefix `isPrefixOf` fromRef b)
+ . map snd
+ <$> inRepo Git.Ref.list
+ forM_ branches $ \b ->
+ inRepo $ Git.Ref.delete' b
+ where
+ branchprefix = "refs/remotes/" ++ remotename ++ "/"
+
+isPrivateUUID :: UUID -> Annex Bool
+isPrivateUUID u =
+ (\c -> u `S.member` annexPrivateRepos c)
+ <$> Annex.getGitConfig
+
+{- Remove a private remote's uuid from the private journal
+ - entirely when it is the only remote using that uuid.
+ -
+ - And, when the remote is a sameas remote, its config is stored
+ - under its config-uuid. Remove that from the private remote log.
+ -}
+cleanPrivateJournal :: Remote -> Bool -> Annex ()
+cleanPrivateJournal r uniqueuuid
+ | uniqueuuid == True = do
+ whenM (isPrivateUUID (uuid r)) $ do
+ gc <- Annex.getGitConfig
+ let tc = filterBranch (\u -> u /= uuid r) gc
+ let handlestale = \_ b -> return (b, Nothing)
+ lockJournal $ \jl ->
+ overPrivateJournalFileContents handlestale Just
+ (go jl tc)
+ removeconfiguuid
+ | otherwise = removeconfiguuid
+ where
+ go jl tc getfilecontents = getfilecontents >>= \case
+ Just (_, p, Just (b, _)) -> do
+ cleaner jl tc p b
+ go jl tc getfilecontents
+ Just (_, _, Nothing) ->
+ go jl tc getfilecontents
+ Nothing -> return ()
+
+ cleaner jl tc p b = case tc p b of
+ PreserveFile -> return ()
+ ChangeFile builder ->
+ setlocaljournal jl (RegardingUUID [uuid r]) p builder
+
+ removeconfiguuid = case remoteAnnexConfigUUID (gitconfig r) of
+ Nothing -> return ()
+ Just cu -> whenM (isPrivateUUID cu) $
+ lockJournal $ \jl ->
+ getJournalFile jl (GetPrivate True) remoteLog >>= \case
+ JournalledContent b ->
+ scrub jl cu b
+ PossiblyStaleJournalledContent b ->
+ scrub jl cu b
+ NoJournalledContent ->
+ return ()
+ where
+ scrub jl cu b =
+ setlocaljournal jl (RegardingUUID [cu]) remoteLog $
+ buildRemoteConfigLog $ MapLog $ M.delete cu $
+ fromMapLog $ parseRemoteConfigLog b
+
+ setlocaljournal jl ru p builder =
+ let b' = toLazyByteString builder
+ in if L.null b'
+ then deleteJournalFile jl ru p
+ else do
+ b'' <- Annex.Branch.getLocal' (GetPrivate False) p
+ if b'' == b'
+ then deleteJournalFile jl ru p
+ else setJournalFile jl ru p builder
+
+removeFsckState :: UUID -> Annex ()
+removeFsckState u = do
+ d <- fromRepo (gitAnnexFsckStateDir u)
+ liftIO $ whenM (doesDirectoryExist d) $
+ removeDirectoryRecursive d
+ f <- fromRepo (gitAnnexFsckResultsLog u)
+ liftIO $ removeWhenExistsWith removeFile f
+
+removeImportLog :: UUID -> Annex ()
+removeImportLog u = do
+ f <- calcRepo' (gitAnnexImportLog u)
+ liftIO $ removeWhenExistsWith removeFile f
+
+removeTransferLogs :: UUID -> Annex ()
+removeTransferLogs u = do
+ -- getTransfers calls checkTransfer, which cleans up
+ -- transfer log files for transfers that are no longer running.
+ whenM (any foru <$> getTransfers) $
+ error "Active trasfers, cannot disable the remote."
+ forM_ [Upload, Download] $ \direction -> do
+ d <- fromRepo $ gitAnnexTransferUUIDDirectionDir u direction
+ liftIO $ void $ tryNonAsync $ removeDirectory d
+ d' <- fromRepo $ gitAnnexFailedTransferDir u direction
+ liftIO $ void $ tryNonAsync $ removeDirectory d'
+ clearFailedTransfers u
+ where
+ foru (t, _) = transferUUID t == u
+
+removeCredFiles :: UUID -> Annex ()
+removeCredFiles u = do
+ d <- fromRepo gitAnnexCredsDir
+ liftIO $ whenM (doesDirectoryExist d) $ do
+ mapM_ (removeWhenExistsWith removeFile)
+ =<< filter foru <$> dirContents d
+ where
+ us = fromUUID u
+
+ -- Remotes that use creds always include their UUID as part of the
+ -- filename. However, some remotes (Remote.External) need more than
+ -- one creds file, and add a "-foo" suffix to the UUID.
+ foru p =
+ let f = takeFileName p
+ in f == us || (us <> literalOsPath "-") `OS.isPrefixOf` f
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 46130bc273..d3bfe80d05 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -5,36 +5,11 @@
(Diff truncated)
initial request re forgejo-aneksjo e2e loop script
diff --git a/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn b/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn new file mode 100644 index 0000000000..d7810f7c9c --- /dev/null +++ b/doc/todo/test_script_against_headless_forgejo-aneksajo_.mdwn @@ -0,0 +1,19 @@ +The need came up in the scope of trying to extend datalad-fuse to support forgejo-aneksajo ([AI assisted PR, look at your discretion](https://github.com/datalad/datalad-fuse/pull/127/changes)). There, to test, I would like to establish test fixture which would be used in tests to test interactions against containerized `codeberg.org/forgejo-aneksajo/forgejo-aneksajo:forgejo-rootless` instance . But I failed to programmatically establish the flow to make git-annex authenticate there even with some drums and bells. + +So I thought if you have some basic shell script (not copy/paste of a manual execution since I think there might be some races I am confronting with) which could reliably demonstrate sequence of commands to + +- boot up fresh forgejo-rootless (e.g. using podman) +- create basic account there +- create or push (to create) repo +- clone it locally (so it gets recognized as git-annex repo) +- add file under annex +- git push and git-annex copy annexed file (so authentication should work out) to forgejo-rootless + +would be greatly appreciated. + +It would allow us to establish local forgejo+aneksjo for testing of datalad-fuse (and other components) which would then use to access remote on forgejo instances openneuro preprocessed datasets. + +[[!meta author=yoh]] +[[!tag projects/openneuro]] + +
make ephemeral=no the default
This is a good default because a remote may be expensive to set up, and
ephemeral=yes would then not be desirable.
This is a good default because a remote may be expensive to set up, and
ephemeral=yes would then not be desirable.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index dc476852fe..e85333ae9f 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 82c54630ca..771d16f926 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value ephemeral=yes|no [params]
+ DELEGATE type=value [ephemeral=yes|no] [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -15,7 +15,8 @@ is not stored to the git-annex branch.
With ephemeral=yes, the delegate special remote is created and exists only
as long as the external special remote program is used by git-annex. Then
it is removed. With ephemeral=no, the delegate special remote is cached for
-use next time, avoiding the overhead of initializing it again.
+use next time, avoiding the overhead of initializing it again. When it is
+not specified, the default is ephemeral=no.
For example:
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index d1834385d3..ce1ec411bf 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
Revert "temporarily remove ephemeral setting"
This reverts commit 4ca54f030fdbbc0452b568b7ae6528f89f78b310.
This reverts commit 4ca54f030fdbbc0452b568b7ae6528f89f78b310.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index e85333ae9f..dc476852fe 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value ephemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 5318a0683c..82c54630ca 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value [params]
+ DELEGATE type=value ephemeral=yes|no [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -12,18 +12,25 @@ and `--private` options. The delegate special remote inherits
the encryption settings, and uses the same annex-uuid. Its configuration
is not stored to the git-annex branch.
+With ephemeral=yes, the delegate special remote is created and exists only
+as long as the external special remote program is used by git-annex. Then
+it is removed. With ephemeral=no, the delegate special remote is cached for
+use next time, avoiding the overhead of initializing it again.
+
For example:
TRANSFER STORE somekey tmpfile
- DELEGATE type=directory directory=/mnt/disk/
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
This makes the annex object be stored in a directory special remote.
+Since initializing a directory special remote is inexpensive, it's
+made ephemeral.
When the key is later requested to be retrieved,
the same delegate can be used:
TRANSFER RETRIEVE somekey tmpfile
- DELEGATE type=directory directory=/mnt/disk/
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
----
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index ce1ec411bf..d1834385d3 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value [params]`
+ * `DELEGATE type=value emphemeral=yes|no [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment b/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment new file mode 100644 index 0000000000..300931dd27 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_12_46e3a4fb89f9946527efe0c8e8648c62._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 12""" + date="2026-04-08T18:23:11Z" + content=""" +`git-annex disableremote` is completed and I hope to wrap this up soon. +"""]]
TODO for fsck against exported S3
diff --git a/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn b/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn new file mode 100644 index 0000000000..2049f40479 --- /dev/null +++ b/doc/todo/fsck_against_versioned_S3_should_populate_log.rmet.mdwn @@ -0,0 +1,55 @@ +More info could be found at fresh [bug report against openneuro.org](https://github.com/OpenNeuroOrg/openneuro/issues/3875) and reproduction helpers in quick&dirty [20260408-noversionid](https://github.com/yarikoptic/20260408-noversionid) + +But boils down to the fact that if for some reason (older buggy git-annex used back then?) `git-annex` branch lacks log.rmet entries for files **present** exported on S3, it should populate those .log.rmet entries, but ATM it just fails to `fsck` but then manages to `get`. + +```shell +(git)smaug:/mnt/btrfs/datasets/datalad/tmp/20260408-noversionid/ds000113[master]git +$> git annex whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (3 copies) + 66a7004d-a15e-4764-90cd-54bbd179f74a -- [s3-PRIVATE] + b8b60a40-f339-4ddc-b08a-2a6f645bd3ef -- root@8dc3dbd70baf:/datalad/ds001473 + e28d70a7-9314-4542-a4ce-7d95b862070f -- [s3-PUBLIC] +ok + +$> git annex fsck --from s3-PUBLIC sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +fsck sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz + Remote is configured to use versioning, but no S3 version ID is recorded for this key +(cannot check content) failed +(recording state in git...) +fsck: 1 failed + +$> git annex whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +whereis sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (3 copies) + 66a7004d-a15e-4764-90cd-54bbd179f74a -- [s3-PRIVATE] + b8b60a40-f339-4ddc-b08a-2a6f645bd3ef -- root@8dc3dbd70baf:/datalad/ds001473 + e28d70a7-9314-4542-a4ce-7d95b862070f -- [s3-PUBLIC] +ok + +$> git annex get sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz +get sub-01/ses-auditoryperception/func/sub-01_ses-auditoryperception_task-auditoryperception_run-01_physio.tsv.gz (from s3-PRIVATE...) + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(Delaying 1s before retrying....) + + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(Delaying 2s before retrying....) + + Remote is configured to use versioning, but no S3 version ID is recorded for this key + + download failed: invalid url +(from s3-PUBLIC...) + Remote is configured to use versioning, but no S3 version ID is recorded for this key +ok + +$> git annex version | head -n 1 +git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05 + +``` + +Note that there could be multiple versions for the remote file and ideally git-annex does "its best" to figure out which one is the one to use, potentially even downloading them until hitting the matching one unless could associate using ETag etc. (FWIW ETag is deterministic even for multipart uploads, given knowing how upload was chunked. [Here](https://github.com/dandi/dandi-schema/blob/master/dandischema/digests/dandietag.py) our python code could be found. + +[[!meta author=yoh]] +[[!tag projects/openneuro]]
disableremote cred file removal
This includes removing the cred files used to accress P2P peers.
Note that the P2P auth token, which is used to auth incoming
connections, is stored in a cred file, but not one named with the uuid
of a remote, so it will not be deleted by this. Which is as intended,
see commit ddad858a817e58e26c21417196c3aac28591c8db
This includes removing the cred files used to accress P2P peers.
Note that the P2P auth token, which is used to auth incoming
connections, is stored in a cred file, but not one named with the uuid
of a remote, so it will not be deleted by this. Which is as intended,
see commit ddad858a817e58e26c21417196c3aac28591c8db
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 81016ebd19..dfc1990bfb 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -5,6 +5,8 @@
- Licensed under the GNU AGPL version 3 or higher.
-}
+{-# LANGUAGE OverloadedStrings #-}
+
module Command.DisableRemote where
import Command
@@ -27,6 +29,7 @@ import qualified Database.Export
import qualified Database.Fsck
import qualified Database.RepoSize
import qualified Database.ContentIdentifier
+import qualified Utility.OsString as OS
import Data.ByteString.Builder
import qualified Data.Map as M
@@ -73,10 +76,7 @@ start (remotename:[]) = byName' remotename >>= \case
removeFsckState (uuid r)
removeImportLog (uuid r)
-
- -- It would be good to remove cred files, but there
- -- is currently no way to list cred files belonging
- -- to a remote, or even to a UUID.
+ removeCredFiles (uuid r)
inRepo $ Git.Remote.Remove.remove remotename
removeRemoteTrackingBranches remotename
@@ -187,3 +187,18 @@ removeTransferLogs u = do
where
foru (t, _) = transferUUID t == u
+removeCredFiles :: UUID -> Annex ()
+removeCredFiles u = do
+ d <- fromRepo gitAnnexCredsDir
+ liftIO $ whenM (doesDirectoryExist d) $ do
+ mapM_ (removeWhenExistsWith removeFile)
+ =<< filter foru <$> dirContents d
+ where
+ us = fromUUID u
+
+ -- Remotes that use creds always include their UUID as part of the
+ -- filename. However, some remotes (Remote.External) need more than
+ -- one creds file, and add a "-foo" suffix to the UUID.
+ foru p =
+ let f = takeFileName p
+ in f == us || (us <> literalOsPath "-") `OS.isPrefixOf` f
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
index 75cc0dd97d..0c0e4f6775 100644
--- a/doc/git-annex-disableremote.mdwn
+++ b/doc/git-annex-disableremote.mdwn
@@ -20,9 +20,6 @@ use with the `git-annex enableremote` command.
When `git-annex initremote --private` was used to initialize a private
remote, disabling it also removes its information from the private journal.
-Note that this does not currently delete any cached credentials for a
-special remote or P2P peer. That may be improved in the future.
-
# OPTIONS
* `--json`
Added a comment: others stall too -- workaround <()
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment new file mode 100644 index 0000000000..2ea98a49b0 --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes/comment_1_2fad011efa895ac18185c322be4945fa._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="others stall too -- workaround <()" + date="2026-04-08T14:42:59Z" + content=""" +well -- head also stalls + +``` +smaug:/tmp/ds000113 +$> git annex list | head +here +|origin +||s3-PRIVATE +|||s3-PUBLIC +||||web +|||||bittorrent +|||||| +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-02_bold.mat +___X__ derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-03_bold.mat + +``` + +workaround + + +``` +*$> head -n1 <(git-annex find --not --in here --fast) +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +``` + +"""]]
stalling find if piped
diff --git a/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn new file mode 100644 index 0000000000..58fea8f47a --- /dev/null +++ b/doc/bugs/find_should_quite_if_output_of_the_pipe_closes.mdwn @@ -0,0 +1,43 @@ +### Please describe the problem. + +``` +$> git-annex version | head -n 1 +git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05 + + +(git)smaug:/tmp/ds000113[master]git +$> time git-annex find --not --in here --fast | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat + +``` + +is hanging although IMHO should have exited right after spitting out first entry. Overall it takes under second to list them + +``` +$> time git-annex find --not --in here --fast > /dev/null +git-annex find --not --in here --fast > /dev/null 0.45s user 0.09s system 132% cpu 0.408 total + +``` + +but if piped -- stall! + +it did manage to quit once + +``` +$> git annex find --not --in here --fast | head -n 1 +derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat +git-annex: <stdout>: hFlush: resource vanished (Broken pipe) + +``` + +with system wide installed 10.20250416 but I failed to reproduce + +### What steps will reproduce the problem? + +FWIW -- above on https://github.com/OpenNeuroDatasets/ds000113.git + + +I didn't check other commands , but they all should behave sane as not to stall if piped ! Would be great if there was a test to ensure that long term. + +[[!meta author=yoh]] +[[!tag projects/openneuro]]
disableremote private journal cleanup
Was able to reuse the code that drops dead remotes from the git-annex
branch to filter the remote uuid out of the private journal log files.
This makes me very happy about the design of that code.
Also there is a special case for --sameas to remove the config uuid from
remote.log.
Was able to reuse the code that drops dead remotes from the git-annex
branch to filter the remote uuid out of the private journal log files.
This makes me very happy about the design of that code.
Also there is a special case for --sameas to remove the config uuid from
remote.log.
diff --git a/Annex/Branch.hs b/Annex/Branch.hs
index d0196d8875..f1e1103948 100644
--- a/Annex/Branch.hs
+++ b/Annex/Branch.hs
@@ -21,6 +21,8 @@ module Annex.Branch (
forceUpdate,
updateTo,
get,
+ getLocal,
+ getLocal',
getHistorical,
getRef,
getUnmergedRefs,
@@ -1052,7 +1054,7 @@ overBranchFileContents' select go st = do
Nothing
| journalIgnorable st -> return Nothing
| otherwise ->
- overJournalFileContents' buf (handlestale branchsha) select
+ overJournalFileContents' buf False (handlestale branchsha) select
res <- catObjectStreamLsTree l (select' . getTopFilePath . Git.LsTree.file) g go'
`finally` liftIO (void cleanup)
return (res, branchsha)
diff --git a/Annex/Journal.hs b/Annex/Journal.hs
index 55eae36b28..74d85b0bc7 100644
--- a/Annex/Journal.hs
+++ b/Annex/Journal.hs
@@ -102,6 +102,15 @@ setJournalFile _jl ru file content = withOtherTmp $ \tmp -> do
-- exists
mv `catchIO` (const (createAnnexDirectory jd >> mv))
+deleteJournalFile :: JournalLocked -> RegardingUUID -> OsPath -> Annex ()
+deleteJournalFile _jl ru file = do
+ st <- getState
+ jd <- fromRepo =<< ifM (regardingPrivateUUID ru)
+ ( return (gitAnnexPrivateJournalDir st)
+ , return (gitAnnexJournalDir st)
+ )
+ liftIO $ removeWhenExistsWith removeFile (jd </> journalFile file)
+
newtype AppendableJournalFile = AppendableJournalFile (OsPath, OsPath)
{- If the journal file does not exist, it cannot be appended to, because
@@ -321,17 +330,32 @@ overJournalFileContents
-> Annex a
overJournalFileContents handlestale select go = do
buf <- liftIO newEmptyMVar
- go $ overJournalFileContents' buf handlestale select
+ go $ overJournalFileContents' buf False handlestale select
+
+{- Like overJournalFileContents, but only over files that are in the
+ - private journal. However, the file content still includes the public
+ - content, concacenated with the private content. -}
+overPrivateJournalFileContents
+ :: (OsPath -> L.ByteString -> Annex (L.ByteString, Maybe b))
+ -> (OsPath -> Maybe v)
+ -> (Annex (FileContents v b) -> Annex a)
+ -> Annex a
+overPrivateJournalFileContents handlestale select go = do
+ buf <- liftIO newEmptyMVar
+ go $ overJournalFileContents' buf True handlestale select
overJournalFileContents'
:: MVar ([OsPath], [OsPath])
+ -> Bool
-> (OsPath -> L.ByteString -> Annex (L.ByteString, Maybe b))
-> (OsPath -> Maybe a)
-> Annex (FileContents a b)
-overJournalFileContents' buf handlestale select =
+overJournalFileContents' buf onlyprivate handlestale select =
liftIO (tryTakeMVar buf) >>= \case
Nothing -> do
- jfs <- journalledFiles
+ jfs <- if onlyprivate
+ then return []
+ else journalledFiles
pjfs <- journalledFilesPrivate
drain jfs pjfs
Just (jfs, pjfs) -> drain jfs pjfs
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
index 4b000fd01f..6d2f4e20a3 100644
--- a/Command/DisableRemote.hs
+++ b/Command/DisableRemote.hs
@@ -10,8 +10,22 @@ module Command.DisableRemote where
import Command
import Remote
import Git
+import qualified Annex
import qualified Git.Remote.Remove
import qualified Git.Ref
+import Types.Remote
+import Annex.Journal
+import qualified Annex.Branch
+import Annex.Branch.Transitions
+import Types.Transitions
+import Logs
+import Logs.Remote.Pure
+import Logs.MapLog
+
+import Data.ByteString.Builder
+import qualified Data.Map as M
+import qualified Data.Set as S
+import qualified Data.ByteString.Lazy as L
cmd :: Command
cmd = withAnnexOptions [jsonOptions] $
@@ -27,6 +41,10 @@ start :: [String] -> CommandStart
start (remotename:[]) = byName' remotename >>= \case
Left err -> giveup err
Right r -> starting "disableremote" ai si $ do
+ uniqueuuid <- not
+ . any (\r' -> uuid r' == uuid r && name r' /= name r)
+ <$> remoteList
+
-- It would be good to remove export databases, fsck
-- databases, and transfer logs, but all of those are
-- uuid based, so would need to avoid deleting any if the
@@ -39,12 +57,7 @@ start (remotename:[]) = byName' remotename >>= \case
-- It would be good to remove the AuthToken used for a P2P
-- remote.
- -- If the remote is private, it would be good to remove
- -- the remote from remote.log and uuid.log in the private
- -- journal, and also to remove any private logs for the
- -- uuid. (Unless there are other remotes using the same
- -- uuid.)
-
+ cleanPrivateJournal r uniqueuuid
inRepo $ Git.Remote.Remove.remove remotename
removeRemoteTrackingBranches remotename
@@ -66,3 +79,67 @@ removeRemoteTrackingBranches remotename = do
inRepo $ Git.Ref.delete' b
where
branchprefix = "refs/remotes/" ++ remotename ++ "/"
+
+{- Remove a private remote's uuid from the private journal
+ - entirely when it is the only remote using that uuid.
+ -
+ - And, when the remote is a sameas remote, its config is stored
+ - under its config-uuid. Remove that from the private remote log.
+ -}
+cleanPrivateJournal :: Remote -> Bool -> Annex ()
+cleanPrivateJournal r uniqueuuid
+ | uniqueuuid == True = do
+ whenM (isprivateuuid (uuid r)) $ do
+ gc <- Annex.getGitConfig
+ let tc = filterBranch (\u -> u /= uuid r) gc
+ let handlestale = \_ b -> return (b, Nothing)
+ lockJournal $ \jl ->
+ overPrivateJournalFileContents handlestale Just
+ (go jl tc)
+ removeconfiguuid
+ | otherwise = removeconfiguuid
+ where
+ isprivateuuid u =
+ (\c -> u `S.member` annexPrivateRepos c)
+ <$> Annex.getGitConfig
+
+ go jl tc getfilecontents = getfilecontents >>= \case
+ Just (_, p, Just (b, _)) -> do
+ cleaner jl tc p b
+ go jl tc getfilecontents
+ Just (_, _, Nothing) ->
+ go jl tc getfilecontents
+ Nothing -> return ()
+
+ cleaner jl tc p b = case tc p b of
+ PreserveFile -> return ()
+ ChangeFile builder ->
+ setlocaljournal jl (RegardingUUID [uuid r]) p builder
+
+ removeconfiguuid = case remoteAnnexConfigUUID (gitconfig r) of
+ Nothing -> return ()
+ Just cu -> whenM (isprivateuuid cu) $
+ lockJournal $ \jl ->
+ getJournalFile jl (GetPrivate True) remoteLog >>= \case
+ JournalledContent b ->
+ scrub jl cu b
+ PossiblyStaleJournalledContent b ->
+ scrub jl cu b
+ NoJournalledContent ->
+ return ()
+ where
+ scrub jl cu b =
+ setlocaljournal jl (RegardingUUID [cu]) remoteLog $
+ buildRemoteConfigLog $ MapLog $ M.delete cu $
+ fromMapLog $ parseRemoteConfigLog b
+
+ setlocaljournal jl ru p builder =
+ let b' = toLazyByteString builder
+ in if L.null b'
+ then deleteJournalFile jl ru p
+ else do
+ b'' <- Annex.Branch.getLocal' (GetPrivate False) p
+ if b'' == b'
+ then deleteJournalFile jl ru p
+ else setJournalFile jl ru p builder
+
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
index faf7cec8c9..75cc0dd97d 100644
(Diff truncated)
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment b/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment new file mode 100644 index 0000000000..452dd0330e --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_11_722bc7d47295de9dd52f083d8114f9e1._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 11""" + date="2026-04-01T15:57:20Z" + content=""" +I have started a `disableremote` branch that adds a `git-annex +disableremote` command. That same command will be able to be used +for `ephemeral=yes`. + +As it stands, the command works, but there is some information about the +remote that it does not remove. I think for this purpose, +it needs to at least remove the config uuid of the sameas remote +from the private remote.log journal file. Otherwise that file will grow +by one line each time an ephemeral delegate is used. + +Removing cached annex creds, etc would also be good, but is not essential; they +would get overwritten on next use. And, since cached creds use a filename +based on the uuid, there's actually no way to know if a cached creds is for +an delegate or are creds used by the external special remote with the +same uuid. This actually seems like a bit of a bug with --sameas in +general, that if two sameas remotes both use cached creds, they will clash. +(Perhaps creds should use the config-uuid.) +"""]]
basic disableremote
There are several things it ought to delete, but doesn't yet. Still,
it's usable as far as it goes.
There are several things it ought to delete, but doesn't yet. Still,
it's usable as far as it goes.
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs index 6efd0b9e43..4b000fd01f 100644 --- a/Command/DisableRemote.hs +++ b/Command/DisableRemote.hs @@ -8,19 +8,15 @@ module Command.DisableRemote where import Command -import qualified Annex.SpecialRemote -import Annex.SpecialRemote.Config (nameField, sameasNameField) -import qualified Logs.Remote -import qualified Types.Remote as R -import qualified Remote -import Types.ProposedAccepted - -import qualified Data.Map as M +import Remote +import Git +import qualified Git.Remote.Remove +import qualified Git.Ref cmd :: Command cmd = withAnnexOptions [jsonOptions] $ command "disableremote" SectionSetup - "stops git-annex from using a remote" + "stop using a remote" paramName (withParams seek) @@ -28,5 +24,45 @@ seek :: CmdParams -> CommandSeek seek = withWords (commandAction . start) start :: [String] -> CommandStart -start ps@(name:[]) = error "TODO" +start (remotename:[]) = byName' remotename >>= \case + Left err -> giveup err + Right r -> starting "disableremote" ai si $ do + -- It would be good to remove export databases, fsck + -- databases, and transfer logs, but all of those are + -- uuid based, so would need to avoid deleting any if the + -- same uuid is still in use by another remote. + + -- It would be good to remove cred files, but there + -- is currently no way to list cred files belonging to a + -- remote. + + -- It would be good to remove the AuthToken used for a P2P + -- remote. + + -- If the remote is private, it would be good to remove + -- the remote from remote.log and uuid.log in the private + -- journal, and also to remove any private logs for the + -- uuid. (Unless there are other remotes using the same + -- uuid.) + + inRepo $ Git.Remote.Remove.remove remotename + removeRemoteTrackingBranches remotename + + next $ return True + where + ai = ActionItemOther (Just (UnquotedString remotename)) + si = SeekInput [remotename] start _ = giveup "Specify the remote's name." + +-- Remove any remote branches. +-- This is done because git remote remove only removes the configured +-- remote tracking branch, not other remote branches. +removeRemoteTrackingBranches :: String -> Annex () +removeRemoteTrackingBranches remotename = do + branches <- filter (\b -> branchprefix `isPrefixOf` fromRef b) + . map snd + <$> inRepo Git.Ref.list + forM_ branches $ \b -> + inRepo $ Git.Ref.delete' b + where + branchprefix = "refs/remotes/" ++ remotename ++ "/" diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn index 2b063fbff2..faf7cec8c9 100644 --- a/doc/git-annex-disableremote.mdwn +++ b/doc/git-annex-disableremote.mdwn @@ -1,6 +1,6 @@ # NAME -git-annex disableremote - stops git-annex from using a remote +git-annex disableremote - stop using a remote # SYNOPSIS @@ -8,16 +8,17 @@ git annex disableremote `name` # DESCRIPTION -Stops git-annex in the current repository from using a remote that was -earlier set up by `git-annex initremote` or `git-annex enableremote`. +This removes the remote from the local git configuration, the same as +`git remote remove` would. It removes remote tracking branches. +It also cleans up various other files in `.git/annex/` that belong to +or mention the remote. -This does not prevent the same remote from continuing to be used in other -clones of the repository. +Note that this does not currently delete any cached credentials for a +special remote or P2P peer. That may be improved in the future. -This removes the remote from the local git configuration, the same as -`git remote remove` would. It also cleans up various other files in -`.git/annex/` that mention the remote, including deleting any cached -credentials for the remote. +This does not prevent the same special remote from continuing to be used +in other clones of the repository, and it can be re-enabled for local +use with the `git-annex enableremote` command. # OPTIONS
stub in disableremote command
diff --git a/CHANGELOG b/CHANGELOG
index ecc6037a6e..e1eb0a8d81 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,6 +1,7 @@
git-annex (10.20260317) UNRELEASED; urgency=medium
* Fix annexUrl to inherit any password that is set in the remote url.
+ * disableremote: New command.
-- Joey Hess <id@joeyh.name> Mon, 23 Mar 2026 11:37:09 -0400
diff --git a/CmdLine/GitAnnex.hs b/CmdLine/GitAnnex.hs
index 7290b660e9..65b5fac680 100644
--- a/CmdLine/GitAnnex.hs
+++ b/CmdLine/GitAnnex.hs
@@ -58,6 +58,7 @@ import qualified Command.InitRemote
import qualified Command.EnableRemote
import qualified Command.ConfigRemote
import qualified Command.RenameRemote
+import qualified Command.DisableRemote
import qualified Command.EnableTor
import qualified Command.Multicast
import qualified Command.Expire
@@ -173,6 +174,7 @@ cmds testoptparser testrunner mkbenchmarkgenerator = map addGitAnnexCommonOption
, Command.EnableRemote.cmd
, Command.ConfigRemote.cmd
, Command.RenameRemote.cmd
+ , Command.DisableRemote.cmd
, Command.EnableTor.cmd
, Command.Multicast.cmd
, Command.Reinject.cmd
diff --git a/Command/DisableRemote.hs b/Command/DisableRemote.hs
new file mode 100644
index 0000000000..6efd0b9e43
--- /dev/null
+++ b/Command/DisableRemote.hs
@@ -0,0 +1,32 @@
+{- git-annex command
+ -
+ - Copyright 2026 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+module Command.DisableRemote where
+
+import Command
+import qualified Annex.SpecialRemote
+import Annex.SpecialRemote.Config (nameField, sameasNameField)
+import qualified Logs.Remote
+import qualified Types.Remote as R
+import qualified Remote
+import Types.ProposedAccepted
+
+import qualified Data.Map as M
+
+cmd :: Command
+cmd = withAnnexOptions [jsonOptions] $
+ command "disableremote" SectionSetup
+ "stops git-annex from using a remote"
+ paramName
+ (withParams seek)
+
+seek :: CmdParams -> CommandSeek
+seek = withWords (commandAction . start)
+
+start :: [String] -> CommandStart
+start ps@(name:[]) = error "TODO"
+start _ = giveup "Specify the remote's name."
diff --git a/doc/git-annex-disableremote.mdwn b/doc/git-annex-disableremote.mdwn
new file mode 100644
index 0000000000..2b063fbff2
--- /dev/null
+++ b/doc/git-annex-disableremote.mdwn
@@ -0,0 +1,50 @@
+# NAME
+
+git-annex disableremote - stops git-annex from using a remote
+
+# SYNOPSIS
+
+git annex disableremote `name`
+
+# DESCRIPTION
+
+Stops git-annex in the current repository from using a remote that was
+earlier set up by `git-annex initremote` or `git-annex enableremote`.
+
+This does not prevent the same remote from continuing to be used in other
+clones of the repository.
+
+This removes the remote from the local git configuration, the same as
+`git remote remove` would. It also cleans up various other files in
+`.git/annex/` that mention the remote, including deleting any cached
+credentials for the remote.
+
+# OPTIONS
+
+* `--json`
+
+ Enable JSON output. This is intended to be parsed by programs that use
+ git-annex.
+
+* `--json-error-messages`
+
+ Messages that would normally be output to standard error are included in
+ the JSON instead.
+
+* Also, the [[git-annex-common-options]](1) can be used.
+
+# SEE ALSO
+
+[[git-annex]](1)
+
+[[git-annex-initremote]](1)
+
+[[git-annex-enableremote]](1)
+
+[[git-annex-renameremote]](1)
+
+# AUTHOR
+
+Joey Hess <id@joeyh.name>
+
+Warning: Automatically converted into a man page by mdwn2man. Edit with care.
diff --git a/doc/git-annex-enableremote.mdwn b/doc/git-annex-enableremote.mdwn
index 7c0a70d558..a74f81f75b 100644
--- a/doc/git-annex-enableremote.mdwn
+++ b/doc/git-annex-enableremote.mdwn
@@ -86,6 +86,8 @@ has found didn't work before and gave up on using, setting
[[git-annex-renameremote]](1)
+[[git-annex-disableremote]](1)
+
# AUTHOR
Joey Hess <id@joeyh.name>
diff --git a/doc/git-annex-initremote.mdwn b/doc/git-annex-initremote.mdwn
index f4d5705308..beb2f3c96e 100644
--- a/doc/git-annex-initremote.mdwn
+++ b/doc/git-annex-initremote.mdwn
@@ -151,6 +151,8 @@ want to use `git annex renameremote`.
[[git-annex-renameremote]](1)
+[[git-annex-disableremote]](1)
+
# AUTHOR
Joey Hess <id@joeyh.name>
diff --git a/git-annex.cabal b/git-annex.cabal
index d02c567360..0d422341af 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -640,6 +640,7 @@ Executable git-annex
Command.Dead
Command.Describe
Command.DiffDriver
+ Command.DisableRemote
Command.Drop
Command.DropKey
Command.DropUnused
Added a comment: a note on potential use case(s)
diff --git a/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment b/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment new file mode 100644 index 0000000000..fa2daca752 --- /dev/null +++ b/doc/todo/importtree_only_remotes/comment_1_32cb04d928afb93aea50684a093b3aef._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="a note on potential use case(s)" + date="2026-03-28T12:23:26Z" + content=""" +> because noone has requested it + +I came to this while considering using git-annex to importtree from rclone. + +The use case could be automation of workflow to work with docs on google drive or alike ([docflow](https://github.com/con/docflow)) or even more so for the [con/serve](https://con.github.io/serve/) \"concept\" since people do keep and collaborate via google drives and dropboxes. I do not have (yet) a clear idea on for which funded project this could apply, but it might become relevant/affordable one way or another if we guestimate the amount of work needed. Also the [protocol enhancement](http://git-annex.branchable.com/design/external_special_remote_protocol/export_and_import_appendix/) for import/export-tree already looks quite thought-through. +"""]]
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment index 8a901671bf..ff1e5ea0d4 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment @@ -13,4 +13,8 @@ Their config does not get written to the git-annex branch and their configuration prevents the user from accidentially using them. The traces that hit the disk are just an implementation detail which can easily be ignored. + +So I'm inclined to merge the branch as it stands. But that would mean that, +if ephemeral support is later added it would need to be a new extension eg +"DELEGATE-EPHEMERAL". """]]
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment new file mode 100644 index 0000000000..8a901671bf --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_10_cd5ae0b22d4c9c6389be2887764db2b8._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2026-03-26T16:08:11Z" + content=""" +The `delegate` branch now has the exension working. As it stands, I left +out the `ephemeral=yes|no`, so the delegate remote remains +in the git config and elsewhere after git-annex stops running, and will be +reused next time it's needed. + +I suspect that it might not matter that delegate remotes are not ephemeral. +Their config does not get written to the git-annex branch and their +configuration prevents the user from accidentially using them. The traces +that hit the disk are just an implementation detail which can easily be +ignored. +"""]]
temporarily remove ephemeral setting
The delegate extension seems to be fully implemented, save for this
setting. Which will be a lot of extra work to implement.
The delegate extension seems to be fully implemented, save for this
setting. Which will be a lot of extra work to implement.
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index dc476852fe..e85333ae9f 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -148,7 +148,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENT Key`
@@ -170,7 +170,7 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVE Key`
@@ -180,7 +180,7 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -288,7 +288,7 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
- * `DELEGATE type=value ephemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `GETINFO`
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
index 82c54630ca..5318a0683c 100644
--- a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -2,7 +2,7 @@ This is an appendix to the [[external_special_remote_protocol]].
The `DELEGATE` extension allows many requests to be responded to with:
- DELEGATE type=value ephemeral=yes|no [params]
+ DELEGATE type=value [params]
This delegates the request to a special remote of the specified
type, which is initialized with the provided parameters.
@@ -12,25 +12,18 @@ and `--private` options. The delegate special remote inherits
the encryption settings, and uses the same annex-uuid. Its configuration
is not stored to the git-annex branch.
-With ephemeral=yes, the delegate special remote is created and exists only
-as long as the external special remote program is used by git-annex. Then
-it is removed. With ephemeral=no, the delegate special remote is cached for
-use next time, avoiding the overhead of initializing it again.
-
For example:
TRANSFER STORE somekey tmpfile
- DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+ DELEGATE type=directory directory=/mnt/disk/
This makes the annex object be stored in a directory special remote.
-Since initializing a directory special remote is inexpensive, it's
-made ephemeral.
When the key is later requested to be retrieved,
the same delegate can be used:
TRANSFER RETRIEVE somekey tmpfile
- DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+ DELEGATE type=directory directory=/mnt/disk/
----
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index d1834385d3..ce1ec411bf 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,7 +59,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
@@ -79,7 +79,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORT Key`
@@ -90,7 +90,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
@@ -107,7 +107,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
@@ -118,7 +118,7 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
- * `DELEGATE type=value emphemeral=yes|no [params]`
+ * `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
fix link
diff --git a/doc/todo/import_tree_from_rsync_special_remote.mdwn b/doc/todo/import_tree_from_rsync_special_remote.mdwn index c31dd0b199..d65d6116d4 100644 --- a/doc/todo/import_tree_from_rsync_special_remote.mdwn +++ b/doc/todo/import_tree_from_rsync_special_remote.mdwn @@ -37,7 +37,7 @@ importtree, but there are several roadblocks: So, it seems that, importtree would need to be able to run commands other than rsync on the server. --[[Joey]] -Or, implement [[todo/importtree_only_remote]] and make rsync special +Or, implement [[todo/importtree_only_remotes]] and make rsync special remotes support either importtree or extporttree, but not both, which avoids that problem. --[[Joey]]
initial report on the dances with annexurl
diff --git a/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn b/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn new file mode 100644 index 0000000000..0decc22896 --- /dev/null +++ b/doc/bugs/annex_overwrites_existing_p2p_annexurl.mdwn @@ -0,0 +1,71 @@ +### Please describe the problem. + +I am trying to orchestrate testing against a forgejo+aneksjo fixture instance in [datalad-fuse](https://github.com/datalad/datalad-fuse/pull/127) where I run forgejo+aneksjo in podman container mapping to a different (fixed) external ports. Overall verdict - "have difficulties". + + +One particular, potentially a core issue (besides that odd "push master first to get things rolling", likely to initiate smth on server side I guess), is that `annexurl`, even if I predefine it correct: + +```shell +❯ git annex version | head -n 1 +git-annex version: 10.20260316+git1-g768707adf4-1~ndall+1 +❯ git annex copy --to=forgejo testfile.bin < /dev/null +Username for 'http://127.0.0.1:41713': ^C +❯ tail -n 5 .git/config +[remote "forgejo"] + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://127.0.0.1:41713/git-annex-p2phttp +❯ git push forgejo master +Enumerating objects: 9, done. +Counting objects: 100% (9/9), done. +Delta compression using up to 20 threads +Compressing objects: 100% (8/8), done. +Writing objects: 100% (9/9), 1003 bytes | 1003.00 KiB/s, done. +Total 9 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0) +To http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + * [new branch] master -> master +❯ tail -n 5 .git/config +[remote "forgejo"] + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://127.0.0.1:41713/git-annex-p2phttp +``` + +it would get overridden by git-annex: + +```shell +❯ git annex copy --to=forgejo testfile.bin < /dev/null +copy testfile.bin (unable to connect to HTTP server: Network.Socket.connect: <socket: 30>: does not exist (Connection refused)) failed +copy: 1 failed +❯ tail -n 5 .git/config + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://localhost:3000/git-annex-p2phttp + annex-uuid = ab484350-97c4-48a7-9d3d-8321dc966cb4 +``` + +and fixing port is not enough: + +```shell +❯ sed -i -e 's,:3000,:41713,g' .git/config +❯ tail -n 5 .git/config + url = http://127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + fetch = +refs/heads/*:refs/remotes/forgejo/* + pushurl = http://testadmin:c8bacefb551205d5b1f75bba7af38aabc2dd7287@127.0.0.1:41713/testadmin/test-annex-91e0e3b7.git + annexurl = annex+http://localhost:41713/git-annex-p2phttp + annex-uuid = ab484350-97c4-48a7-9d3d-8321dc966cb4 +❯ git annex copy --to=forgejo testfile.bin < /dev/null +Username for 'annex+http://localhost:41713/git-annex-p2phttp': ^C +``` + +seems that pointing to IP "fixes" it + +``` +❯ sed -i -e 's,localhost:41713,127.0.0.1:41713,g' .git/config +❯ git annex copy --to=forgejo testfile.bin < /dev/null +copy testfile.bin (to forgejo...) ok +(recording state in git...) +```
update
diff --git a/doc/todo/RawFilePath_conversion.mdwn b/doc/todo/RawFilePath_conversion.mdwn index cd0dec9b1a..444d6f322f 100644 --- a/doc/todo/RawFilePath_conversion.mdwn +++ b/doc/todo/RawFilePath_conversion.mdwn @@ -49,9 +49,8 @@ of the status. any noticiable performance impact. The `require_OsPath` branch removes the OsPath build flag, -and merging it would resolve this. That will need packagers to do some work -to package the libraries though. Or to upgrade ghc, since file-io and -os-string are boot libraries since ghc 9.12.2 and 9.10.1 respectively. +and merging it would resolve this. But it needs a newer version of file-io +than is in Debian yet. --[[Joey]] [[!tag confirmed]]
Added a comment: Re: registering multi-file torrent urls for existing files
diff --git a/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment b/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment new file mode 100644 index 0000000000..dc1b2cea7e --- /dev/null +++ b/doc/special_remotes/bittorrent/comment_6_588119c1ac99558a94631689dce9363a._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="miris" + avatar="http://cdn.libravatar.org/avatar/bd975774ecba53f7454a61d50fc7d8cc" + subject="Re: registering multi-file torrent urls for existing files" + date="2026-03-22T12:36:40Z" + content=""" +Coming back to this one, turns out this is feasible using `git annex registerurl`: + +``` +aria2c --show-files multi-file.torrent +# get the number of the file +git annex registerurl \"$(git annex lookupkey file.mp4)\" \"magnet:...#<number>\" +``` + +Works as intended; however, this will skip verification, so it's imperative that this is only used when you are 100% certain the file in the torrent is exactly the same your local one. :) +"""]]
diff --git a/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn b/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn new file mode 100644 index 0000000000..11900c6a50 --- /dev/null +++ b/doc/forum/__91__android__93_____91__adb__93___Support_for_adb_over_tcp.mdwn @@ -0,0 +1,7 @@ +What is the best way to use adb special remote with device connected and paired over wifi? It uses random ports for connection every time, so I haven't even bothered yet to try IP:PORT as a serialnumber. + + +``` +$ adb devices -l +XXXXX:12345 device product:XXX model:XXX device:XXX transport_id:1 +```
mih seems stated that it works for him just fine.
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index a657aa92b1..6f3492dc88 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -106,4 +106,4 @@ FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile bu [[!meta author=yoh]] -[[!tag projects/FZJ]] +
update
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment index e943321592..05b7cf0035 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment @@ -3,20 +3,21 @@ subject="""comment 1""" date="2026-03-17T17:47:39Z" content=""" -Note that forgejo does support raw file access, and I expect that supports -range requests for annex objected. +Note that forgejo does support raw file access, and I expect that it +supports range requests for annex objects. -The endpoint where it might make sense to support range requests -is `https://git-annex.branchable.com/design/p2p_protocol_over_http/#index1h3` +The p2phttp endpoint where it might make sense to support range requests +is `/git-annex/$uuid/key/$key` When p2phttp is proxying to a special remote, it would need to download the whole file from the special remote even if the range request was for a small part. So I don't think it should be supported for proxying. One way to implement this might be to use Servant.Server.StaticFiles -to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint -redirect requests to that. That uses warp's built-in static file serving, -which supports range requests. +with a StaticSettings ssLookupFiles that returns the file location under +`.git/annex/objects/` (or even a location in another git-annex repository +when proxying to one, eg a cluster node.) That uses warp's built-in static +file serving, which supports range requests. But how to handle authentication? It seems like the only way would be to reimplement p2phttp's authentication checking as
comment
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment new file mode 100644 index 0000000000..e943321592 --- /dev/null +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs/comment_1_fa5d62289f76fefaf808edfea41622cd._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-03-17T17:47:39Z" + content=""" +Note that forgejo does support raw file access, and I expect that supports +range requests for annex objected. + +The endpoint where it might make sense to support range requests +is `https://git-annex.branchable.com/design/p2p_protocol_over_http/#index1h3` + +When p2phttp is proxying to a special remote, it would need to download +the whole file from the special remote even if the range request was for a +small part. So I don't think it should be supported for proxying. + +One way to implement this might be to use Servant.Server.StaticFiles +to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint +redirect requests to that. That uses warp's built-in static file serving, +which supports range requests. + +But how to handle authentication? It seems like +the only way would be to reimplement p2phttp's authentication checking as +WAI middleware. +"""]] diff --git a/doc/todo/p2phttp_ranged_requests.mdwn b/doc/todo/p2phttp_ranged_requests.mdwn deleted file mode 100644 index 674125a12c..0000000000 --- a/doc/todo/p2phttp_ranged_requests.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -The p2phttp endpoint for raw download of a key does not currently support -range requests or other such things. While it's not always possible for -p2phttp to support that, eg when proxying to a special remote it cannot, it -would be useful if it supported it in configurations where it is possible -to do so. - -One way to implement this might be to use Servant.Server.StaticFiles -to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint -redirect requests to that. And reject ranged requests when proxying. - -But how to handle authentication? It seems like -the only way would be to reimplement p2phttp's authentication checking as -WAI middleware. --[[Joey]]
removed association with FZJ project as out of current scope/need
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn index bbbbb8775b..368d470e88 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -9,4 +9,3 @@ and It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. [[!meta author=yoh]] -[[!tag projects/FZJ]]
todo
diff --git a/doc/todo/p2phttp_ranged_requests.mdwn b/doc/todo/p2phttp_ranged_requests.mdwn new file mode 100644 index 0000000000..674125a12c --- /dev/null +++ b/doc/todo/p2phttp_ranged_requests.mdwn @@ -0,0 +1,13 @@ +The p2phttp endpoint for raw download of a key does not currently support +range requests or other such things. While it's not always possible for +p2phttp to support that, eg when proxying to a special remote it cannot, it +would be useful if it supported it in configurations where it is possible +to do so. + +One way to implement this might be to use Servant.Server.StaticFiles +to serve `.git/annex/objects/`, and make the `serveGetGeneric` API endpoint +redirect requests to that. And reject ranged requests when proxying. + +But how to handle authentication? It seems like +the only way would be to reimplement p2phttp's authentication checking as +WAI middleware. --[[Joey]]
reassign to existing FZJ
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index 5b19dcb9f3..a657aa92b1 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -106,4 +106,4 @@ FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile bu [[!meta author=yoh]] -[[!tag projects/trr379.de]] +[[!tag projects/FZJ]]
reassign to already existing FZJ
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn index 985fd7d0a2..bbbbb8775b 100644 --- a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -9,4 +9,4 @@ and It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. [[!meta author=yoh]] -[[!tag projects/trr379.de]] +[[!tag projects/FZJ]]
TODO on range requests
diff --git a/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn new file mode 100644 index 0000000000..985fd7d0a2 --- /dev/null +++ b/doc/todo/p2p__58___add_Range___40__in__41___and_Content-Length___40__out__41___hdrs.mdwn @@ -0,0 +1,12 @@ +ATM according to [.../design/p2p_protocol_over_http/](https://git-annex.branchable.com/design/p2p_protocol_over_http/) + +> Request headers are currently ignored, so eg Range requests are not supported. (This would be possible to implement, up to a point.) + +and + +> Note that there is no Content-Length header. + +It would be great to have those supported. The former should facilitate datalad-fuse via fsspec sparse access to data via p2p protocol, the latter -- any additional verification/treatment of the fetched content by any standard http* library/downloader. + +[[!meta author=yoh]] +[[!tag projects/trr379.de]]
added server side log and relation to trr379 project
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn index ddf14504b5..5b19dcb9f3 100644 --- a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -1,6 +1,6 @@ ### Please describe the problem. -Finally got to try the neat p2p, ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. +Finally got to try the neat p2p (inspired by trr379 use-case raised in matrix), ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. Unfortunately <details> @@ -77,6 +77,20 @@ curl: (52) Empty reply from server curl: (56) Recv failure: Connection reset by peer ``` +edit: added the server side: + +``` +❯ git annex version --raw; echo; git annex --debug p2phttp --port 8081 --wideopen + +10.20260316+git1-g768707adf4-1~ndall+1 +[2026-03-17 10:16:22.435272107] (Utility.Process) process [2104460] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2026-03-17 10:16:22.441213037] (Utility.Process) process [2104460] done ExitSuccess +[2026-03-17 10:16:22.441610323] (Utility.Process) process [2104461] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2026-03-17 10:16:22.446123885] (Utility.Process) process [2104461] done ExitSuccess +[2026-03-17 10:16:22.447208284] (Utility.Process) process [2104462] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2026-03-17 10:16:22.453605123] (Annex.Branch) read proxy.log + +``` </details> @@ -89,3 +103,7 @@ this was on a local clone of https://datasets.datalad.org/dbic/QA/.git with `sou 10.20260316+git1-g768707adf4-1~ndall+1 - not ok; 10.20251029 - not ok. FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile but I think it was due to some change in behavior affecting RIA archives (I did not have yet chance to troubleshoot manually to report) , thus unrelated here. + + +[[!meta author=yoh]] +[[!tag projects/trr379.de]]
reporting about inability for p2p with recent version
diff --git a/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn new file mode 100644 index 0000000000..ddf14504b5 --- /dev/null +++ b/doc/bugs/recent_annex_p2phttp_silently___40__in_--debug__41___refuses.mdwn @@ -0,0 +1,91 @@ +### Please describe the problem. + +Finally got to try the neat p2p, ultimately with the hope to connect to datalad-fuse. Wanted to test range request support (since was reported to be lacking by claude on a forgejo+aneksjo instance) and thus thought to try on the most recent version locally. +Unfortunately + +<details> +<summary>whenever p2phttp worked fine (for a full file) request on 10.20251029</summary> + +```shell +❯ curl http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 96.62M 0 96.62M 0 0 563.2M 0 0 +❯ sha256sum out.tgz +02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f out.tgz + +``` + +and on server side: + +``` +❯ git annex version --raw; echo; git annex --debug p2phttp --port 8081 --wideopen +10.20251029 +[2026-03-17 10:12:16.496986375] (Utility.Process) process [2098931] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2026-03-17 10:12:16.499045819] (Utility.Process) process [2098931] done ExitSuccess +[2026-03-17 10:12:16.499500423] (Utility.Process) process [2098934] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2026-03-17 10:12:16.501265293] (Utility.Process) process [2098934] done ExitSuccess +[2026-03-17 10:12:16.502428567] (Utility.Process) process [2098935] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2026-03-17 10:12:16.505874247] (Annex.Branch) read proxy.log +[2026-03-17 10:12:26.349175199] (P2P.IO) [http client] [ThreadId 22] P2P > GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz +[2026-03-17 10:12:26.349539971] (P2P.IO) [http server] [ThreadId 19] P2P < GET 0 SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz +[2026-03-17 10:12:26.350670225] (Utility.Process) process [2099177] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"] +[2026-03-17 10:12:26.366674858] (Utility.Process) process [2099177] done ExitSuccess +[2026-03-17 10:12:26.367081645] (Utility.Process) process [2099178] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"] +[2026-03-17 10:12:26.369153188] (Utility.Process) process [2099178] done ExitSuccess +[2026-03-17 10:12:26.370404636] (P2P.IO) [http server] [ThreadId 19] P2P > DATA 101318091 +[2026-03-17 10:12:26.370496387] (P2P.IO) [http client] [ThreadId 22] P2P < DATA 101318091 +[2026-03-17 10:12:26.518654447] (P2P.IO) [http client] [ThreadId 22] P2P > SUCCESS +[2026-03-17 10:12:26.518800582] (P2P.IO) [http server] [ThreadId 19] P2P < SUCCESS + +``` +</details> + + +<details> +<summary> +with the most recent version 10.20260316+git1-g768707adf4-1~ndall+1 we get silent treatment -- no content provided and nothing in the --debug log +</summary> + + +``` +❯ curl http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 0 +curl: (52) Empty reply from server +❯ curl -v http://localhost:8081/git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz >| out.tgz +* Host localhost:8081 was resolved. +* IPv6: ::1 +* IPv4: 127.0.0.1 + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 0* Trying [::1]:8081... +* connect to ::1 port 8081 from ::1 port 56576 failed: Connection refused +* Trying 127.0.0.1:8081... +* Established connection to localhost (127.0.0.1 port 8081) from 127.0.0.1 port 58866 +* using HTTP/1.x +> GET /git-annex/90d896aa-00d0-4f85-bcae-2fd1e992fcab/key/SHA256E-s101318091--02b4a96d66121ddbb5d51fa3f22c2b929bc16d955f438421c1f1b04a1264a50f.tgz HTTP/1.1 +> Host: localhost:8081 +> User-Agent: curl/8.18.0 +> Accept: */* +> +* Request completely sent off +* Recv failure: Connection reset by peer + +* closing connection #0 +curl: (56) Recv failure: Connection reset by peer +``` + +</details> + + +### What steps will reproduce the problem? + +this was on a local clone of https://datasets.datalad.org/dbic/QA/.git with `sourcedata/sub-qa64/ses-20240715/func/sub-qa64_ses-20240715_acq-faX77_bold.dicom.tgz` content + +### What version of git-annex are you using? On what operating system? + +10.20260316+git1-g768707adf4-1~ndall+1 - not ok; 10.20251029 - not ok. + +FWIW -- https://github.com/datalad/git-annex testing was not happy for awhile but I think it was due to some change in behavior affecting RIA archives (I did not have yet chance to troubleshoot manually to report) , thus unrelated here.
improve docs to close bug
diff --git a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn index ae34d49200..a9ad943cf8 100644 --- a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn +++ b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote.mdwn @@ -45,3 +45,5 @@ local repository version: 10 [[!tag projects/ICE4]] + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment new file mode 100644 index 0000000000..6c45a883ab --- /dev/null +++ b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_9_31a1d65c06fff526c08eea4d35e5c4d5._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-03-17T13:39:50Z" + content=""" +Updated documentation. +"""]] diff --git a/doc/git-annex-export.mdwn b/doc/git-annex-export.mdwn index 0ec85668d8..c87051c07c 100644 --- a/doc/git-annex-export.mdwn +++ b/doc/git-annex-export.mdwn @@ -59,12 +59,22 @@ tell it what branch to track. For example: git config remote.myremote.annex-tracking-branch master git annex push myremote -You can combine using `git annex export` to send changes to a special +When the special remote is not also configured with `importtree=yes`, +git-annex does not try to preserve other files that may be written to the +special remote by other means. Exporting a tree of files to a special remote +will overwrite any files already stored on it with the same filenames. +In some cases, an update to what is exported that deletes a subdirectory +will delete not only the exported files that were in that subdirectory, +but any other files that might have been written to the same subdirectory +by other means. + +When the special remote is also configured with `importtree=yes`, +you can combine using `git annex export` to send changes to a special remote with `git annex import` to fetch changes from a special remote. -When a file on a special remote has been modified by software other than -git-annex, exporting to it will not overwrite the modified file, and the -export will not succeed. You can resolve this conflict by using -`git annex import`. +In this case, when a file on a special remote has been modified by +software other than git-annex, exporting to it will not overwrite the +modified file, and the export will not succeed. You can resolve this +conflict by using `git annex import`. (Some types of special remotes such as S3 with versioning may instead let an export overwrite the modified file; then `git annex import`
correct branch name
diff --git a/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment index f4a292dd86..fa0d3f7a4d 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment @@ -3,7 +3,7 @@ subject="""comment 9""" date="2026-03-05T19:00:16Z" content=""" -Started developing this in the `ephemeral` branch. +Started developing this in the `delegate` branch. It seems to also make sense to allow DELEGATE as a response to WHEREIS.
add news item for git-annex 10.20260316
diff --git a/doc/news/version_10.20251029.mdwn b/doc/news/version_10.20251029.mdwn deleted file mode 100644 index b98b28583c..0000000000 --- a/doc/news/version_10.20251029.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -git-annex 10.20251029 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * Support ssh remotes with '#' and '?' in the path to the repository, - the same way git does. - * assistant: Fix reversion that caused files to be added locked by - default."""]] \ No newline at end of file diff --git a/doc/news/version_10.20260316.mdwn b/doc/news/version_10.20260316.mdwn new file mode 100644 index 0000000000..7ca6f4e414 --- /dev/null +++ b/doc/news/version_10.20260316.mdwn @@ -0,0 +1,17 @@ +git-annex 10.20260316 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * Added CHECKPRESENT-URL extension to the external special remote protocol. + * Fix reversion in previous version that caused auto-initializing of + local git remotes that have annex-ignore set. + * Fix bug that caused git credential to be rejected when a http request + failed for some reason other than 401. + * Importing from the directory special remote will no longer add sizes + to keys, which overrode backends that generate unsized keys. + * Fix retrival from http git remotes of keys with '%' in their names. + * Fix behavior when initremote is used with --sameas= + combined with --private. + * web, S3, git: Fix bugs in checking if content is present on a remote + when configuration does not allow accessing it. + * httpalso: Fix bugs in handling content not being present on the remote. + * adb: Avoid deleting contents of a non-empty directory when + removing the last exported file from the directory. + * Improve display of http exceptions."""]] \ No newline at end of file
response
diff --git a/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment b/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment new file mode 100644 index 0000000000..456d576eff --- /dev/null +++ b/doc/special_remotes/borg/comment_5_d6c9d91578ddcb12a5a33e313094179e._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-03-16T10:48:29Z" + content=""" +I have not looked at borg 2 in any detail, but it seems they are trying to +keep the CLI to some extent the same. So it seems it would depend on whether +CLI changes break something git-annex relies on. + +I'd treat any incompatability as a [[bug|bugs]] or [[todo]] on the git-annex +side, so if you try it and find problems, please report them. +"""]]
Added a comment: borg 2.0
diff --git a/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment b/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment new file mode 100644 index 0000000000..40c5e76b46 --- /dev/null +++ b/doc/special_remotes/borg/comment_4_2327577bd65e66d1c88b5d05de38cb5c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="nadir" + avatar="http://cdn.libravatar.org/avatar/2af9174cf6c06de802104d632dc40071" + subject="borg 2.0" + date="2026-03-15T13:27:42Z" + content=""" +With borg 2.0 on the horizon, I was wondering how support for that would look should that be planned. + +With how much has changed and the necessity to create new repos, it might make the most sense to create a separate borg2 remote, but I have now idea. Mostly just curious if there are any plans for that at all. +"""]]
update
diff --git a/doc/thanks/list b/doc/thanks/list index 0c9056062d..0e8eb59c97 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -129,3 +129,4 @@ Andrew Poelstra, joshingly, Melody Tolly, username, +Steffen Vogel,
update
diff --git a/doc/thanks/list b/doc/thanks/list index 0a65388abb..0c9056062d 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -128,3 +128,4 @@ mpol, Andrew Poelstra, joshingly, Melody Tolly, +username,
comment
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys/comment_4_18d10c9f97fec86ce16776021777bd17._comment b/doc/bugs/import_adds_size_to_external_backend_keys/comment_4_18d10c9f97fec86ce16776021777bd17._comment new file mode 100644 index 0000000000..b0924ad2fe --- /dev/null +++ b/doc/bugs/import_adds_size_to_external_backend_keys/comment_4_18d10c9f97fec86ce16776021777bd17._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-03-11T15:14:44Z" + content=""" +Turns out Remote.Directory did not need to add the size at all. I've fixed +it. +"""]]
close
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys.mdwn b/doc/bugs/import_adds_size_to_external_backend_keys.mdwn index df9ed4c6f5..7574182477 100644 --- a/doc/bugs/import_adds_size_to_external_backend_keys.mdwn +++ b/doc/bugs/import_adds_size_to_external_backend_keys.mdwn @@ -85,3 +85,5 @@ index 0000000..8620ffe [[!tag projects/ICE4]] + +> [[fixed|done]] --[[Joey]]
removed
diff --git a/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment b/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment deleted file mode 100644 index 0643ca5fee..0000000000 --- a/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="miris" - avatar="http://cdn.libravatar.org/avatar/bd975774ecba53f7454a61d50fc7d8cc" - subject="registering multi-file torrent urls for existing files" - date="2026-03-11T15:09:29Z" - content=""" -Heyo, - -Is it possible to assign multi-file torrent links to files, without needing to download the entire torrent again using Annex? - -Consider the following scenario: - -A multi-file torrent has been downloaded in the past, say, using qBittorrent... -... I then add the files to Git Annex, and would like to efficiently register torrent/magnet URLs for them... - -Annex maps files to a multi-file torrent using `#<n>` suffixes, so I've tried the following: - -1. `aria2c --show-files <torrent file>` -- to get the numbers for each file -2. `git annex addurl --file \"file.mp4\" \"magnet:...#<N>\" -- to map the file to its original torrent/magnet - -The following error occurs: - -> (downloading torrent file...) git-annex: That url contains multiple files according to the bittorrent remote; cannot add it to a single file. - -If there's any information or clarification needed, please don't hesitate to let me know :) -"""]]
Added a comment: registering multi-file torrent urls for existing files
diff --git a/doc/special_remotes/bittorrent/comment_6_ee0e1e0ed6e97dffc5840db0baf7c25c._comment b/doc/special_remotes/bittorrent/comment_6_ee0e1e0ed6e97dffc5840db0baf7c25c._comment new file mode 100644 index 0000000000..61faf94ccd --- /dev/null +++ b/doc/special_remotes/bittorrent/comment_6_ee0e1e0ed6e97dffc5840db0baf7c25c._comment @@ -0,0 +1,27 @@ +[[!comment format=mdwn + username="miris" + avatar="http://cdn.libravatar.org/avatar/bd975774ecba53f7454a61d50fc7d8cc" + subject="registering multi-file torrent urls for existing files" + date="2026-03-11T15:10:37Z" + content=""" +Heyo, + +Is it possible to assign multi-file torrent links to files, without needing to download the entire torrent again using Annex? + +Consider the following scenario: + +A multi-file torrent has been downloaded in the past, say, using qBittorrent... + +... I then add the files to Git Annex, and would like to efficiently register torrent/magnet URLs for them... + +Annex maps files to a multi-file torrent using `#<n>` suffixes, so I've tried the following: + +1. `aria2c --show-files <torrent file>` -- to get the numbers for each file +2. `git annex addurl --file \"file.mp4\" \"magnet:...#<N>\"` -- to map the file to its original torrent/magnet + +The following error occurs: + +> (downloading torrent file...) git-annex: That url contains multiple files according to the bittorrent remote; cannot add it to a single file. + +If there's any information or clarification needed, please don't hesitate to let me know :) +"""]]
Added a comment: registering multi-file torrent urls for existing files
diff --git a/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment b/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment new file mode 100644 index 0000000000..0643ca5fee --- /dev/null +++ b/doc/special_remotes/bittorrent/comment_5_45b73c0d9f8b2fd2238412fb9feb3e5b._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="miris" + avatar="http://cdn.libravatar.org/avatar/bd975774ecba53f7454a61d50fc7d8cc" + subject="registering multi-file torrent urls for existing files" + date="2026-03-11T15:09:29Z" + content=""" +Heyo, + +Is it possible to assign multi-file torrent links to files, without needing to download the entire torrent again using Annex? + +Consider the following scenario: + +A multi-file torrent has been downloaded in the past, say, using qBittorrent... +... I then add the files to Git Annex, and would like to efficiently register torrent/magnet URLs for them... + +Annex maps files to a multi-file torrent using `#<n>` suffixes, so I've tried the following: + +1. `aria2c --show-files <torrent file>` -- to get the numbers for each file +2. `git annex addurl --file \"file.mp4\" \"magnet:...#<N>\" -- to map the file to its original torrent/magnet + +The following error occurs: + +> (downloading torrent file...) git-annex: That url contains multiple files according to the bittorrent remote; cannot add it to a single file. + +If there's any information or clarification needed, please don't hesitate to let me know :) +"""]]
comment
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys/comment_3_f6c47c2feb1d1fc1351bc21f9bc1e2d7._comment b/doc/bugs/import_adds_size_to_external_backend_keys/comment_3_f6c47c2feb1d1fc1351bc21f9bc1e2d7._comment new file mode 100644 index 0000000000..0bc811b44b --- /dev/null +++ b/doc/bugs/import_adds_size_to_external_backend_keys/comment_3_f6c47c2feb1d1fc1351bc21f9bc1e2d7._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-03-11T14:48:31Z" + content=""" + $ git annex migrate --remove-size + migrate data.bin + git-annex: failed creating link from old to new key + +That happens when the file is not present in the local repository. +If you get the file first it will work. + +Migrate needs the content present so it can populate the new key. +Otherwise, there can be situation where the new key never ends up being +populated with the content. +"""]]
comment
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys/comment_2_b5f15e7850a6c162349010157e2b55be._comment b/doc/bugs/import_adds_size_to_external_backend_keys/comment_2_b5f15e7850a6c162349010157e2b55be._comment new file mode 100644 index 0000000000..8ec73b1954 --- /dev/null +++ b/doc/bugs/import_adds_size_to_external_backend_keys/comment_2_b5f15e7850a6c162349010157e2b55be._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-03-11T14:21:42Z" + content=""" +This bug only happens with the directory special remote and not others. + +The problem is that the directory special remote implements `importKey`, +and explicitly adds the size back! +[[!commit 15000dee07a06d04285351616915794bd6ec7f14]] explains why +it needs to do that. +"""]]
rename
diff --git a/doc/todo/extrahader_config.mdwn b/doc/todo/extraheader_config.mdwn similarity index 100% rename from doc/todo/extrahader_config.mdwn rename to doc/todo/extraheader_config.mdwn
todo
diff --git a/doc/todo/extrahader_config.mdwn b/doc/todo/extrahader_config.mdwn new file mode 100644 index 0000000000..f753319af7 --- /dev/null +++ b/doc/todo/extrahader_config.mdwn @@ -0,0 +1,23 @@ +Support git config `http.extraHeader` and `http.<url>.extraheader`. + +This would particularly be useful for P2P over HTTP, where an `annex+https` +url could be configured to send headers for http basic auth. + +And related to that case, it might also make sense to make the +`remote.<name>.annexUrl` default to inheriting the extraheader +of the `remote.<name>.url`. So that when using eg, forgejo-aneksajo, +the user only needs to configure the auth header in one place. + +One concern with `http.extraHeader` is that, since git only uses that for +git repo access, it could easily contain auth headers that the user would +be surprised to find `git-annex addurl` using, for example. So it might +make sense to only support the `http.<url>.extraheader` form for uses in +git-annex. + +The P2P over HTTP inheriting idea above could still use +`http.extraheader` when the annexUrl and url are on the same host. +(As is already done when querying `git credential` in that case, +via `isP2PHttpSameHost`.) +--[[Joey]] + +[[!tag projects/INM7]]
Added a comment
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys/comment_1_6bc5340138bfcbe312a0fec446370094._comment b/doc/bugs/import_adds_size_to_external_backend_keys/comment_1_6bc5340138bfcbe312a0fec446370094._comment new file mode 100644 index 0000000000..1df2668084 --- /dev/null +++ b/doc/bugs/import_adds_size_to_external_backend_keys/comment_1_6bc5340138bfcbe312a0fec446370094._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="matrss" + avatar="http://cdn.libravatar.org/avatar/cd1c0b3be1af288012e49197918395f0" + subject="comment 1" + date="2026-03-06T13:53:07Z" + content=""" +Unfortunately it seems like I can't even remove the size after the import: + +``` +$ git switch --detach importdir/import +HEAD is now at 6d3aa43 import from importdir +$ git annex migrate --remove-size +migrate data.bin +git-annex: failed creating link from old to new key +failed +migrate: 1 failed +[ble: exit 1] +``` +"""]]
diff --git a/doc/bugs/import_adds_size_to_external_backend_keys.mdwn b/doc/bugs/import_adds_size_to_external_backend_keys.mdwn new file mode 100644 index 0000000000..df9ed4c6f5 --- /dev/null +++ b/doc/bugs/import_adds_size_to_external_backend_keys.mdwn @@ -0,0 +1,87 @@ +### Please describe the problem. + +I have an external backend for grib files that deliberately does not include a size in its generated keys. It intentionally produces the same key for data that is different on a binary level, but equivalent in practice. This means I don't want to include a size, because two files with equivalent data can have a different size but generate the same key after normalization. + +Now, the problem is that `git annex import` seems to unconditionally add a size to imported keys anyway, as seen in the log below. + + +### What steps will reproduce the problem? + +1. Take the XFOO example backend from here: <https://git-annex.branchable.com/design/external_backend_protocol/git-annex-backend-XFOO> +2. Make this change to remove the size from generated keys: + + ``` + 38c38 + < echo "GENKEY-SUCCESS" "XFOO-s$sz--$hash" + --- + > echo "GENKEY-SUCCESS" "XFOO--$hash" + ``` +3. Create a repository and `git annex add --backend XFOO` a file +4. Validate that this file's key does not have a size +5. Set up a directory special remote with importtree=yes and import the same file from it +6. Observe that the imported file's key has a size set + +### What version of git-annex are you using? On what operating system? + +``` +git-annex version: 10.20260213-g1b947233f21755c0c4d1f00e5a24f39d62fa3f1e +build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath +dependency versions: aws-0.25.2 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +``` + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +$ datalad create test-import-key-size +create(ok): /home/icg149/Playground/test-import-key-size (dataset) +$ head -c 10K /dev/urandom > test-import-key-size/data.bin +$ mkdir test-import-key-size-import-dir +$ cp test-import-key-size/data.bin test-import-key-size-import-dir/ +$ cd test-import-key-size +$ git annex add --backend XFOO data.bin +add data.bin +ok +(recording state in git...) +$ ls -l +total 4 +lrwxrwxrwx 1 icg149 icg149 102 Mär 6 14:17 data.bin -> .git/annex/objects/jM/ZJ/XFOO--cc9401a4c19c25864b650740c215b3bd/XFOO--cc9401a4c19c25864b650740c215b3bd +$ git annex initremote importdir type=directory directory=../test-import-key-size-import-dir importtree=yes encryption=none +initremote importdir ok +(recording state in git...) +$ git annex import --no-content --backend XFOO import --from importdir +list importdir ok +import importdir data.bin +ok +update refs/remotes/importdir/import ok +(recording state in git...) +$ git show importdir/import +commit 6d3aa436a9c643cef7d6f02647b952107bc09951 (importdir/import) +Author: Matthias Riße <m.risse@fz-juelich.de> +Date: Fri Mar 6 14:18:14 2026 +0100 + + import from importdir + +diff --git a/data.bin b/data.bin +new file mode 120000 +index 0000000..8620ffe +--- /dev/null ++++ b/data.bin +@@ -0,0 +1 @@ ++.git/annex/objects/GF/8Z/XFOO-s10240--cc9401a4c19c25864b650740c215b3bd/XFOO-s10240--cc9401a4c19c25864b650740c215b3bd +\ No newline at end of file + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + + +[[!tag projects/ICE4]]
update
diff --git a/doc/thanks/list b/doc/thanks/list index 0a65388abb..0c9056062d 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -128,3 +128,4 @@ mpol, Andrew Poelstra, joshingly, Melody Tolly, +username,
Added a comment: FTR commands to check and "fix up"
diff --git a/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_3_f911957e4dc3c2c61cbdb04e54218705._comment b/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_3_f911957e4dc3c2c61cbdb04e54218705._comment
new file mode 100644
index 0000000000..77c7838a67
--- /dev/null
+++ b/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_3_f911957e4dc3c2c61cbdb04e54218705._comment
@@ -0,0 +1,91 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="FTR commands to check and "fix up""
+ date="2026-03-05T22:46:18Z"
+ content="""
+in fears against modification of files in git-annex branch directly, here is the commands to 'check'
+
+
+```
+$> f=Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+$> key=$(readlink \"$f\" | xargs basename); alog=$(git ls-tree -r git-annex | grep \"$key\" | awk '/.web$/{print $4;}'); git show \"git-annex:$alog\"
+1772708470s 1 https://www.youtube.com/watch?v=0fcKYGsBZxU
+```
+
+First I tried to fix via re-addurl, and we do get some difference:
+
+$> git rm \"$f\"; git annex addurl --no-raw --file \"$f\" \"$url\"
+rm 'Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv'
+addurl https://www.youtube.com/watch?v=0fcKYGsBZxU (using yt-dlp) (to Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv) ok
+(recording state in git...)
+$> git status
+On branch master
+Your branch is up to date with 'origin/master'.
+
+Changes to be committed:
+ (use \"git restore --staged <file>...\" to unstage)
+ modified: Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+
+$> git diff --cached
+diff --git a/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv b/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+index e59a58c35..e12bb1280 120000
+--- a/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
++++ b/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+@@ -1 +1 @@
+-../.git/annex/objects/KQ/x1/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU
+\ No newline at end of file
++../.git/annex/objects/wq/jM/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU
+\ No newline at end of file
+
+```
+
+for which I did not really care as long as I got that file if metadata transferred, but it didn't:
+
+```
+$> git commit -m 'redownloaded \"unlucky\" video for which no yt: was added' $f
+[master 379d379ea] redownloaded \"unlucky\" video for which no yt: was added
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+$> git annex metadata \"$f\"
+metadata Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+
+ok
+```
+
+Also when I used recent after 202601 version which would auto-upgrade to use VURL key difference was to switch from URL to VURL. Could you please point me on where to read up on VURLs and their benefit for relaxed URLs?
+
+
+then I tried to do the dance with unregisterurl, rmurl, addurl, which ended up having
+
+```
+$> key=$(readlink \"$f\" | xargs basename); alog=$(git ls-tree -r git-annex | grep \"$key\" | awk '/.web$/{print $4;}'); git show \"git-annex:$alog\"
+1772750261s 0 https://www.youtube.com/watch?v=0fcKYGsBZxU
+1772750309s 1 yt:https://www.youtube.com/watch?v=0fcKYGsBZxU
+
+```
+
+and for which I still was not able to get it:
+
+```
+$> git annex get \"$f\"
+get Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv (from web...)
+ Verification of content failed
+
+ Unable to access these remotes: web
+
+ No other repository is known to contain the file.
+
+ (Note that these git remotes have annex-ignore set: origin)
+failed
+get: 1 failed
+git annex get \"$f\" 8.22s user 3.63s system 112% cpu 10.505 total
+```
+
+although I think it did fetch it. But i guess it is because of the `-s0` in the original key! So original way with `git rm` + `addurl` was kinda legit as it also fixed up the URL BUT it lost the metadata for the key.
+
+Is there a quick way to copy metadata from another key? (like internally it does for the same path?)
+
+Or is there a better way to 'fix up URL/key' which would you recommend Joey so I could retain metadata?
+
+
+"""]]
Added a comment: FTR commands to check and "fix up"
diff --git a/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_2_fe63fdf9a0d957c7944ad5fe241243fc._comment b/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_2_fe63fdf9a0d957c7944ad5fe241243fc._comment
new file mode 100644
index 0000000000..8eee05f75d
--- /dev/null
+++ b/doc/bugs/tries_to_download_a_.mkv_video_without_yt-dlp/comment_2_fe63fdf9a0d957c7944ad5fe241243fc._comment
@@ -0,0 +1,91 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="FTR commands to check and "fix up""
+ date="2026-03-05T22:45:31Z"
+ content="""
+in fears against modification of files in git-annex branch directly, here is the commands to 'check'
+
+
+```
+$> f=Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+$> key=$(readlink \"$f\" | xargs basename); alog=$(git ls-tree -r git-annex | grep \"$key\" | awk '/.web$/{print $4;}'); git show \"git-annex:$alog\"
+1772708470s 1 https://www.youtube.com/watch?v=0fcKYGsBZxU
+```
+
+First I tried to fix via re-addurl, and we do get some difference:
+
+$> git rm \"$f\"; git annex addurl --no-raw --file \"$f\" \"$url\"
+rm 'Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv'
+addurl https://www.youtube.com/watch?v=0fcKYGsBZxU (using yt-dlp) (to Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv) ok
+(recording state in git...)
+$> git status
+On branch master
+Your branch is up to date with 'origin/master'.
+
+Changes to be committed:
+ (use \"git restore --staged <file>...\" to unstage)
+ modified: Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+
+$> git diff --cached
+diff --git a/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv b/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+index e59a58c35..e12bb1280 120000
+--- a/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
++++ b/Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+@@ -1 +1 @@
+-../.git/annex/objects/KQ/x1/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU
+\ No newline at end of file
++../.git/annex/objects/wq/jM/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU
+\ No newline at end of file
+
+```
+
+for which I did not really care as long as I got that file if metadata transferred, but it didn't:
+
+```
+$> git commit -m 'redownloaded \"unlucky\" video for which no yt: was added' $f
+[master 379d379ea] redownloaded \"unlucky\" video for which no yt: was added
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+$> git annex metadata \"$f\"
+metadata Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
+
+ok
+```
+
+Also when I used recent after 202601 version which would auto-upgrade to use VURL key difference was to switch from URL to VURL. Could you please point me on where to read up on VURLs and their benefit for relaxed URLs?
+
+
+then I tried to do the dance with unregisterurl, rmurl, addurl, which ended up having
+
+```
+$> key=$(readlink \"$f\" | xargs basename); alog=$(git ls-tree -r git-annex | grep \"$key\" | awk '/.web$/{print $4;}'); git show \"git-annex:$alog\"
+1772750261s 0 https://www.youtube.com/watch?v=0fcKYGsBZxU
+1772750309s 1 yt:https://www.youtube.com/watch?v=0fcKYGsBZxU
+
+```
+
+and for which I still was not able to get it:
+
+```
+$> git annex get \"$f\"
+get Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv (from web...)
+ Verification of content failed
+
+ Unable to access these remotes: web
+
+ No other repository is known to contain the file.
+
+ (Note that these git remotes have annex-ignore set: origin)
+failed
+get: 1 failed
+git annex get \"$f\" 8.22s user 3.63s system 112% cpu 10.505 total
+```
+
+although I think it did fetch it. But i guess it is because of the `-s0` in the original key! So original way with `git rm` + `addurl` was kinda legit as it also fixed up the URL BUT it lost the metadata for the key.
+
+Is there a quick way to copy metadata from another key? (like internally it does for the same path?)
+
+Or is there a better way to 'fix up URL/key' which would you recommend Joey so I could retain metadata?
+
+
+"""]]
initial report on stuck'iness
diff --git a/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn new file mode 100644 index 0000000000..9f301ff7f3 --- /dev/null +++ b/doc/bugs/git_annex_copy_+_git_push_get_stuck__in_parallel.mdwn @@ -0,0 +1,49 @@ +### Please describe the problem. + +I was investigating the safety of parallel workers pushing + annex copying to the same original local origin location. +It gets stuck (I have not even tried yet that "ssh" option...) and it seems potentially the counter play of 'annex copy' like + +``` +yoh 3858496 0.0 0.0 7628 3844 pts/29 S 15:25 0:00 | \_ /usr/bin/bash -c worker 3 +yoh 3861583 0.0 0.0 7984 3936 pts/29 S 15:25 0:00 | | \_ git annex copy --to origin file-3-1.txt file-3-2.txt file-3-3.txt file-3-4.txt file-3-5.txt file-common.txt +yoh 3861585 3.7 0.1 1075475816 72956 pts/29 Sl 15:25 0:05 | | \_ /usr/bin/git-annex copy --to origin file-3-1.txt file-3-2.txt file-3-3.txt file-3-4.txt file-3-5.txt file-common.txt +yoh 3861886 0.0 0.0 8116 4564 pts/29 S 15:25 0:00 | | \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch + +``` +(of which I have ATM multiple going on) + +and 'annex post-receive' hook (of which I have only one) running upon `git push` + +``` +yoh 3858498 0.0 0.0 7628 3764 pts/29 S 15:25 0:00 | \_ /usr/bin/bash -c worker 4 +yoh 3863122 0.0 0.0 17340 4948 pts/29 Sl 15:25 0:00 | | \_ git push origin master:br-4 +yoh 3863130 0.0 0.0 2692 1912 pts/29 S 15:25 0:00 | | \_ /bin/sh -c git-receive-pack '/home/yoh/.tmp/parallel-push-3858314/origin' git-receive-pack '/home/yoh/.tmp/parallel-push-3858314/origin' +yoh 3863134 0.0 0.0 16980 5312 pts/29 Sl 15:25 0:00 | | \_ git-receive-pack /home/yoh/.tmp/parallel-push-3858314/origin +yoh 3863172 0.0 0.0 2692 1872 pts/29 S 15:25 0:00 | | \_ /bin/sh hooks/post-receive +yoh 3863192 0.0 0.0 7984 4060 pts/29 S 15:25 0:00 | | \_ git annex post-receive +yoh 3863195 0.3 0.0 1074074572 17816 pts/29 Sl 15:25 0:00 | | \_ /usr/bin/git-annex post-receive +``` + + +### What steps will reproduce the problem? + +[here](https://www.oneukrainian.com/tmp/parallel-push.sh) is a claude-code (with use of LLMs and HI; I did not even review in detail/try yet 'ssh' part coded there) generated script (look or use at your own discretion), on execution of which as `./parallel-push.sh --max-file-size 409600 --max-files 10 40 20` it gets stuck (this is the 4th run I think, consistent stuck at different places) with + +``` +copy file-common.txt ok +copy file-12-1.txt (to origin...) (checksum...) ok +(recording state in git...) +clone-12: done (1 files) +Cloning into '/home/yoh/.tmp/parallel-push-3858314/clone-12'... +done. +remote: (recording state in git...) +remote: (recovering from race...) +To /home/yoh/.tmp/parallel-push-3858314/origin + * [new branch] master -> br-12 + + +``` + +### What version of git-annex are you using? On what operating system? + +this run is with `10.20251029` but I tried bleeding edge standalone build `10.20260213+git57-gffa771e735-1~ndall+1` to the same result but process traces are more garbled so for the benefit of our both HI I pasted from the non-standalone built version
don't allow delegating GETORDERED
maybe later if there is a use case?
maybe later if there is a use case?
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index bb4fd6f3fe..dc476852fe 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -231,9 +231,6 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that files are written in order.
* `UNORDERED`
Indicates that files are not written in order.
- * `DELEGATE type=value ephemeral=yes|no [params]`
- Delegate this request to a different type of special remote.
- See [[delegate_appendix]].
* `GETAVAILABILITY`
Asks the remote if it is locally or globally available.
(Ie stored in the cloud vs on a local disk.)
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment new file mode 100644 index 0000000000..f4a292dd86 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_9_c0fb4d7034229dd12d89c54c046f15e4._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-03-05T19:00:16Z" + content=""" +Started developing this in the `ephemeral` branch. + +It seems to also make sense to allow DELEGATE as a response to WHEREIS. + +I'm on the fence about delegating GETORDERED. Probably most remotes won't +bother to respond to GETORDERED at all, and the only time it makes sense to +delegate it is when always delegating to the same type of special remote. +If delegating to different special remotes at different times, it +doesn't make sense to delegate it to a single on of them. + +Similarly I don't think it makes sense to delegate GETINFO unless only +delegating to a single special remote. Will probably wait to see if someone +has a use case before supporting GETINFO, GETAVAILABILITY, CLAIMURL, etc. +"""]]
document DELEGATE extension to external special remote protocol
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 7c97d721b2..bb4fd6f3fe 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -45,7 +45,7 @@ Recent versions of git-annex respond with a message indicating
protocol extensions that it supports. Older versions of
git-annex do not send this message.
- EXTENSIONS INFO ASYNC GETGITREMOTENAME UNAVAILABLERESPONSE TRANSFER-RETRIEVE-URL CHECKPRESENT-URL
+ EXTENSIONS INFO ASYNC GETGITREMOTENAME UNAVAILABLERESPONSE TRANSFER-RETRIEVE-URL CHECKPRESENT-URL DELEGATE
The special remote can respond to that with its own EXTENSIONS message, listing
any extensions it wants to use.
@@ -148,6 +148,9 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
+ * `DELEGATE type=value ephemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `CHECKPRESENT Key`
Requests the remote to check if a key is present in it.
It's important that, while a key is being transferred to a remote,
@@ -167,6 +170,9 @@ The following requests *must* all be supported by the special remote.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
+ * `DELEGATE type=value ephemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `REMOVE Key`
Requests the remote to remove a key's contents.
* `REMOVE-SUCCESS Key`
@@ -174,6 +180,9 @@ The following requests *must* all be supported by the special remote.
the remote didn't have the key at the point removal was requested.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the key was unable to be removed from the remote.
+ * `DELEGATE type=value ephemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
Special remotes can optionally support tree exports and imports,
which makes the [[git-annex-export]] and [[git-annex-import]] commands
@@ -222,6 +231,9 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that files are written in order.
* `UNORDERED`
Indicates that files are not written in order.
+ * `DELEGATE type=value ephemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `GETAVAILABILITY`
Asks the remote if it is locally or globally available.
(Ie stored in the cloud vs on a local disk.)
@@ -240,8 +252,6 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
trying to use the remote.
Older versions of git-annex do not support this response, so avoid
sending it unless the `UNAVAILABLERESPONSE` extension is enabled.
-* `ORDERED`
-
* `CLAIMURL Url`
Asks the remote if it wishes to claim responsibility for downloading
an url.
@@ -281,6 +291,9 @@ the special remote can reply with `UNSUPPORTED-REQUEST`.
Indicates that no location is known for a key.
This is not needed when `SETURIPRESENT` is used, since such uris are
automatically displayed by `git annex whereis`.
+ * `DELEGATE type=value ephemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `GETINFO`
Requests the remote to send some information describing its
configuration, for display by `git annex info`. A block of responses
@@ -493,9 +506,14 @@ These protocol extensions are currently supported.
* `TRANSFER-RETRIEVE-URL`
This allows the `TRANSFER-RETRIEVE-URL` response to be used
in reply to `TRANSFER` and `TRANSFEREXPORT`.
-* `CHECKPRESENT-URL`
+* `CHECKPRESENT-URL`
This allows the `CHECKPRESENT-URL` response to be used
in reply to `CHECKPRESENT` and `CHECKPRESENTEXPORT`.
+* `DELEGATE`
+ This allows the `DELEGATE` response to be used in reply to many requests.
+ That causes the request to be handled by a delegate special remote,
+ which is initialized as needed. See [[delegate_appendix]]
+ for details.
## signals
diff --git a/doc/design/external_special_remote_protocol/delegate_appendix.mdwn b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
new file mode 100644
index 0000000000..82c54630ca
--- /dev/null
+++ b/doc/design/external_special_remote_protocol/delegate_appendix.mdwn
@@ -0,0 +1,48 @@
+This is an appendix to the [[external_special_remote_protocol]].
+
+The `DELEGATE` extension allows many requests to be responded to with:
+
+ DELEGATE type=value ephemeral=yes|no [params]
+
+This delegates the request to a special remote of the specified
+type, which is initialized with the provided parameters.
+
+It is similar to using `git-annex initremote` with the `--sameas`
+and `--private` options. The delegate special remote inherits
+the encryption settings, and uses the same annex-uuid. Its configuration
+is not stored to the git-annex branch.
+
+With ephemeral=yes, the delegate special remote is created and exists only
+as long as the external special remote program is used by git-annex. Then
+it is removed. With ephemeral=no, the delegate special remote is cached for
+use next time, avoiding the overhead of initializing it again.
+
+For example:
+
+ TRANSFER STORE somekey tmpfile
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+
+This makes the annex object be stored in a directory special remote.
+Since initializing a directory special remote is inexpensive, it's
+made ephemeral.
+
+When the key is later requested to be retrieved,
+the same delegate can be used:
+
+ TRANSFER RETRIEVE somekey tmpfile
+ DELEGATE type=directory ephemeral=yes directory=/mnt/disk/
+
+----
+
+Internally, git-annex has to make a git remote for the delegate special
+remote. It comes up with a name by hashing the configuration of the
+delegate, and puts it in a namespace under the name of the external special
+remote that used it. For example, if the special remote doing the above
+delegation is named "foo", the delegate special remote might be named
+"foo-delegate-69fc17c8f97d751014b36de5c328464b76c41af7".
+
+Currently, delegate special remotes are user visible in eg `git remote`.
+While they are configured with `annex-ignore` set to avoid git-annex
+commands from ususally using them, the user can choose to directly use them
+if they desire. This is just a consequence of the current implementation,
+and may change in future versions of git-annex.
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index f89256798c..d1834385d3 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -59,6 +59,9 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a
protocol extension; it's only safe to send it to git-annex after
it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
+ * `DELEGATE type=value emphemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `CHECKPRESENTEXPORT Key`
Requests the remote to check if the previously provided `EXPORT` Name is present
in it.
@@ -76,6 +79,9 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex. This response is a protocol
extension; it's only safe to send it to git-annex after it sent an
`EXTENSIONS` that included `CHECKPRESENT-URL`.
+ * `DELEGATE type=value emphemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `REMOVEEXPORT Key`
Requests the remote to remove content stored by `TRANSFEREXPORT`
with the previously provided `EXPORT` Name.
@@ -84,6 +90,9 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
the content was already not present.
* `REMOVE-FAILURE Key ErrorMsg`
Indicates that the content was unable to be removed from the remote.
+ * `DELEGATE type=value emphemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `REMOVEEXPORTDIRECTORY Directory`
Requests the remote remove an exported directory.
If the remote does not use directories, or `REMOVEEXPORT` cleans up
@@ -98,6 +107,9 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
* `REMOVEEXPORTDIRECTORY-FAILURE`
Indicates that a `REMOVEEXPORTDIRECTORY` failed for whatever reason.
Should not be returned if the directory did not exist.
+ * `DELEGATE type=value emphemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `RENAMEEXPORT Key NewName`
Requests the remote rename a file stored on it from the previously
provided `EXPORT` Name to the NewName. Remotes that support exports but not
@@ -106,6 +118,9 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that a `RENAMEEXPORT` was done successfully.
* `RENAMEEXPORT-FAILURE Key`
Indicates that a `RENAMEEXPORT` failed for whatever reason.
+ * `DELEGATE type=value emphemeral=yes|no [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
## import/export interface
format
diff --git a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment index 5c8fedfa69..0443160cc2 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment @@ -21,4 +21,4 @@ of reinitialization. With ephemeral=yes, the delegate is automatically removed when the external special remote program shuts down (unless another one is using it.) With ephemeral=no, the delegate remains initialized for use next time. -""""]] +"""]]
remove name
diff --git a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment index 0ed6c2b9ae..5c8fedfa69 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment @@ -6,7 +6,7 @@ Add to external special remote protocol, enabled by the `DELEGATE` extension: - DELEGATE name type=whatever ephemeral=yes|no [params] + DELEGATE type=whatever ephemeral=yes|no [params] Which can be used as a response to TRANSFER, REMOVE, CHECKPRESENT, TRANSFEREXPORT, CHECKPRESENTEXPORT, REMOVEEXPORT, REMOVEEXPORTDIRECTORY,
format
diff --git a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment index 76679a1fa4..0ed6c2b9ae 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment @@ -21,4 +21,4 @@ of reinitialization. With ephemeral=yes, the delegate is automatically removed when the external special remote program shuts down (unless another one is using it.) With ephemeral=no, the delegate remains initialized for use next time. -""]] +""""]]
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment new file mode 100644 index 0000000000..76679a1fa4 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_8_e89046bfe2cd0cfff9d317367e3675f1._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""simplified design with better name""" + date="2026-03-05T15:30:59Z" + content=""" +Add to external special remote protocol, enabled by the `DELEGATE` +extension: + + DELEGATE name type=whatever ephemeral=yes|no [params] + +Which can be used as a response to TRANSFER, REMOVE, CHECKPRESENT, +TRANSFEREXPORT, CHECKPRESENTEXPORT, REMOVEEXPORT, REMOVEEXPORTDIRECTORY, +RENAMEEXPORT + +This initializes a delegate special remote in a private namespace, and +uses it to perform the operation. + +Subsequent uses of DELEGATE with the same configuration avoid the overhead +of reinitialization. + +With ephemeral=yes, the delegate is automatically removed when the external +special remote program shuts down (unless another one is using it.) +With ephemeral=no, the delegate remains initialized for use next time. +""]]
rename
diff --git a/doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment b/doc/todo/Ephemeral_special_remotes/comment_7_213e927f9d25acc01c9c647712c349ee._comment similarity index 98% rename from doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment rename to doc/todo/Ephemeral_special_remotes/comment_7_213e927f9d25acc01c9c647712c349ee._comment index 2eab7302d7..2699c1fc66 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_7_213e927f9d25acc01c9c647712c349ee._comment @@ -1,6 +1,6 @@ [[!comment format=mdwn username="joey" - subject="""comment 6""" + subject="""Re: comment 6""" date="2026-03-05T14:44:39Z" content=""" > Is a non-ephemeral aspect visible/accessible outside the context of the special remote that set it up? Would it appear as a regular special remote for a CLI user, as if they ran initremote?
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment b/doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment new file mode 100644 index 0000000000..2eab7302d7 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_6_213e927f9d25acc01c9c647712c349ee._comment @@ -0,0 +1,59 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2026-03-05T14:44:39Z" + content=""" +> Is a non-ephemeral aspect visible/accessible outside the context of the special remote that set it up? Would it appear as a regular special remote for a CLI user, as if they ran initremote? + +I think it would be best for it not to be visible to the user. Since these +remotes can still set their own git configs though, they will necessarily +show up in `git remote list`. (Any `git config remote.foo.bar` setting is +enough for that.) It would be possible for git-annex to not treat them as +valid remotes when used outside of the aspect context though. +Easiest would be to set annex-ignore on them. + +It would be possible to point `GIT_CONFIG` at a different config file +when setting up and using the ephemeral special remote. That would have +the problem though that if the special remote looks at some user-set +git configs, it wouldn't see them. An example that comes to mind +that a special remote would be expected to see is the +"credential.helper" configuration. Maybe git-annex could merge .git/config +into the ephemeral remote's version when using it? Seems complex and +potentially slow though. + +(BTW, Even ephemeral aspects will be user-visible while git-annex is running.) + +> At which stage would INITASPECT (have to) be used? The PREPARE stage, I guess. + +I think it could be used at any point. + +> How expensive could INITASPECT be? Would it (immediately) trigger init/prepare of the aspect-remote? + +As expensive as `git-annex initremote` initially, but subsequenty close to +a noop when the remote configuration includes emphemeral=no + +Also, calling it repeatedly in the same session with the same configuration +should be a noop after the first time. So you could call it immediately +before USEASPECT. + +That does suggest a simplification: Rather than having a separate +INITASPECT command: + +USEASPECT type=whatever ephemeral=yes|no [params] + +Neat, this avoids needing to name the aspect! And avoids any problem with +the aspect name having been used before with a different config. + +It also means that any +failure to initialize will necessarily make the USEASPECT response be +an error message, so error handling takes care of itself. + +git-annex would still need a remote name internally; it could eg hash the +configuration to get a name. + +I'm inclined to go with this simplification. + +> Do I understand correctly that it would be possible to set the active aspect on a per-key and per-operation basis? + +It's per-operation. If you want different aspects for different types of keys it would be up to you to pick between them. +"""]]
Added a comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_6_1d9cd49500c4b896dd26095c487db142._comment b/doc/todo/Ephemeral_special_remotes/comment_6_1d9cd49500c4b896dd26095c487db142._comment new file mode 100644 index 0000000000..10c4eac6ff --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_6_1d9cd49500c4b896dd26095c487db142._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="mih" + avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd" + subject="comment 6" + date="2026-03-05T08:08:45Z" + content=""" +I concur with you reasoning, also in particular with the observation that making this about URLs would be a mistake. I was already trying to have the \"redirect\" approach do things, it did not want to be used for. + +Here is my understanding of the proposed design: + +I could use this to implement an \"orchestration\" special remote that, rather then implementing store and retrieve procedures, is focused on what other implementations shall be used. For this, it can rely on the full set of special remotes available on a system. It would be possible to have a single remote (using this new feature) abstract a data holding site that can be talked to via various protocols, and the specific access approach can be selected dynamically. This would, therefore, include the ability to use a redirect special remote for URL-based downloads. + +Few questions which I could not answer with confidence: + +- Is a non-ephemeral aspect visible/accessible outside the context of the special remote that set it up? Would it appear as a regular special remote for a CLI user, as if they ran `initremote`? +- At which stage would `INITASPECT` (have to) be used? The `PREPARE` stage, I guess. +- How expensive could `INITASPECT` be? Would it (immediately) trigger init/prepare of the aspect-remote? +- `INITASPECT-OK|INITASPECT-FAILURE` are responses sent by the main git-annex process to the special remote, right? Any implementation would need to implement some kind of error handline (try another aspect, or error also). +- Do I understand correctly that it would be possible to set the active aspect on a per-key and per-operation basis? + + +"""]]
update
diff --git a/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment index 3a186efadb..f7272e1c8f 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment @@ -25,7 +25,12 @@ With ephemeral=yes, the aspect is automatically removed when the external special remote program shuts down (unless another one is using it.) With ephemeral=no, the aspect remains initialized for use next time. -Note that INITASPECT will successfully do nothing if the remote already -exists with the same config. If a remote exists with that name but a -different config, it will remove the old one and init the new one. +Note that INITASPECT will successfully do nothing if the aspect already +exists with the same config. If an aspect exists with that name but a +different config, it will fail. I earlier thought it could remove the old +one and make a new one, but that risks removing an aspect that is still +in use by another process, which could result in unexpected behavior when +that aspect reads its git config or cached creds or etc. It should +be easy enough in most cases to avoid reusing the same aspect name for +two different configs. """]]
layout
diff --git a/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment index 423e78f281..3a186efadb 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment @@ -2,7 +2,7 @@ username="joey" subject="""proposed design""" date="2026-03-04T16:31:28Z" - content="""" + content=""" I'm here going with the name "aspect" to refer to a sameas remote that is in a private namespace belonging to the external special remote that uses it. This name is a bit of a placeholder, but I think some name is needed,
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment new file mode 100644 index 0000000000..423e78f281 --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_5_4ec8597802b2f76839a7e5050417409d._comment @@ -0,0 +1,31 @@ +[[!comment format=mdwn + username="joey" + subject="""proposed design""" + date="2026-03-04T16:31:28Z" + content="""" +I'm here going with the name "aspect" to refer to a sameas remote that is +in a private namespace belonging to the external special remote that uses +it. This name is a bit of a placeholder, but I think some name is needed, +because it would be surprising if "INITREMOTE" did a different thing +than `git-annex initremote`. + +Add to external special remote protocol, enabled by the `REDIRECTREMOTE` +extension: + + INITASPECT name type=whatever ephemeral=yes|no [params] + INITASPECT-OK + INITASPECT-FAILURE reason + +Add response to TRANSFER, REMOVE, CHECKPRESENT, TRANSFEREXPORT, +CHECKPRESENTEXPORT, REMOVEEXPORT, REMOVEEXPORTDIRECTORY, RENAMEEXPORT: + + USEASPECT name + +With ephemeral=yes, the aspect is automatically removed when the external +special remote program shuts down (unless another one is using it.) +With ephemeral=no, the aspect remains initialized for use next time. + +Note that INITASPECT will successfully do nothing if the remote already +exists with the same config. If a remote exists with that name but a +different config, it will remove the old one and init the new one. +"""]]
formatting
diff --git a/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment b/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment index 2533a7a46d..dca6b4ccc2 100644 --- a/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment +++ b/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment @@ -24,12 +24,12 @@ automatically use that namespace. For example: - INITREMOTE blah type=blah url=whatever - INITREMOTE-OK - [...] - REDIRECT_REMOTE blah - [...] - REMOVEREMOTE blah + INITREMOTE blah type=blah url=whatever + INITREMOTE-OK + [...] + REDIRECT_REMOTE blah + [...] + REMOVEREMOTE blah That might make a remote named eg "foo-$foouuid-blah" where $foouuid is the uuid of the special remote foo that owns it. So there is no possibility
comment
diff --git a/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment b/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment new file mode 100644 index 0000000000..2533a7a46d --- /dev/null +++ b/doc/todo/Ephemeral_special_remotes/comment_4_69f32af134b36304ca01f93c62c0c9cf._comment @@ -0,0 +1,51 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-03-04T15:47:39Z" + content=""" +Continuing my line of thought, `REDIRECT_REMOTE` would I guess be +provided with a remote name, not a uuid, since with --sameas the +remote would have the same uuid. + +While special remote "foo" could use "foo-bar", "foo-baz" etc as +the name of its not-really-ephemeral helper remotes, that is not +entirely satisfactory, since the user might have their own "foo-bar" +remote. Or the user might notice "foo-bar" exists, and start using it, +and then it would be painful if "foo" later removes it. + +And, new protocol command like `initremote` does seem to be needed, +because if a special remote runs `git-annex initremote` itself, the +git-annex process that is using the special remote won't know about +the new remote. + +If there's an `initremote`-like protocol command, the special remotes +it inits could be in a separate namespace, and `REDIRECT_REMOTE` could +automatically use that namespace. + +For example: + + INITREMOTE blah type=blah url=whatever + INITREMOTE-OK + [...] + REDIRECT_REMOTE blah + [...] + REMOVEREMOTE blah + +That might make a remote named eg "foo-$foouuid-blah" where $foouuid is +the uuid of the special remote foo that owns it. So there is no possibility +of collision. That would be in `.git/config` for the reasons I discussed +earlier. + +Depending on the type of remote, it might be cheap enough to INITREMOTE +and REMOVEREMOTE in the same session. Making it emphmeral, athough with +some disk writes happening behind the scenes to update the git config etc. +Or, the REMOVEREMOTE could be skipped to leave it set up for the next +session. Then an `INITREMOTE` with the same settings would be optimised +to a no-op. + +That would have `git remote remote foo` leave behind the configs for +the not-so-ephemeral remotes that it set up. Not a big problem, the user +can go in and delete them or a `git-annex removeremote` could handle it, +as well as deleting `.git/annex/journal-private/remote.log`, cached creds, +etc. +"""]]
removed
diff --git a/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment b/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment deleted file mode 100644 index 215ab35502..0000000000 --- a/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="Reigniting interested in this topic and linking to related efforts (BABS etc)" - date="2026-03-02T16:18:28Z" - content=""" -I just saw support for [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout) merged in [BABS](https://github.com/PennLINC/babs/pull/337) and frankly I never knew/used it before! Inspired by an enthusiastic [Meng](https://github.com/just-meng) who ~~made a strategic mistake for her PhD progress~~ [pioneered use of use of git worktrees](https://blog.datalad.org/posts/git-worktree-workflow/) in DataLad having attended Distribits 2025, I thought to check if git-annex has support for the `sparse-checkout`. - -In conjunction with `sparse-checkout` (existing already) support for worktrees in `git-annex` can make a perfect \"couple\" for an efficient [ephemeral](https://myyoda.github.io/principles-examples/stamped_principles/e/) compute where we checkout only what is really needed, e.g. following the `datalad run` input/output specifications. - -This is just a summary of the potential research/implementation since may be it even somehow magically all works already given that BABS merged the sparse-checkout support and they extensively use `git annex` already...? -"""]]
comment
diff --git a/doc/forum/support_for_git_sparse_checkout/comment_3_cba791314c59f61dd49e10b92e41637a._comment b/doc/forum/support_for_git_sparse_checkout/comment_3_cba791314c59f61dd49e10b92e41637a._comment new file mode 100644 index 0000000000..e51735c17c --- /dev/null +++ b/doc/forum/support_for_git_sparse_checkout/comment_3_cba791314c59f61dd49e10b92e41637a._comment @@ -0,0 +1,40 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-03-03T22:50:38Z" + content=""" +In a sparse checkout, most git commands behave as is they were run in a +worktree that contains only the files in the sparse checkout, and not other +files. Since git-annex uses git commands extensively when identifying files +to work on, its commands skip over files not in the sparse checkout. + +There are exceptions, the main one seems to be `git ls-files`, which +does list files not in the sparse checkout. All commands that operate on +annexed files and that use `git ls-files` to enumerate files though feed +the files into `git cat-file --batch`, and that will say a file is not +found when it's not part of the sparse checkout. So git-annex skips those. + +The only exception I can find, and possibily the only one, is that +`git-annex add` will add files that are in a subdirectory that is not +included in the sparse checkout. (It uses `git ls-files` without `git +cat-file`.) That is a different behavior than `git +add`, which refuses to add such files (though the --sparse option +overrides and causes them to be added). + +I don't know if this `git-annex add` behavior would be a problem. +The documentation for `--sparse` says that the reason git add doesn't +default to it is because, after it adds such a file, it could get removed +from the worktree without warning. Which would make it hard to get the +file's content back if it didn't get committed first. + +If that were a problem, it could be fixed by making git-annex run `git +ls-files` with the `--sparse` option, which is supposed to filter out files +not in the sparse checkout... Except that doesn't seem to work right when +I try it. Maybe a bug in git (2.51.0)? + +Anyway, my impression is that this would all need +playing with to determine if it happens to meet your needs. +Bearing in mind that sparse checkout is itself an experimental feature +(for 6+ years?) that is documented to be subject to future behavior +changes. +"""]]
Added a comment: Reigniting interested in this topic and linking to related efforts (BABS etc)
diff --git a/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment b/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment new file mode 100644 index 0000000000..215ab35502 --- /dev/null +++ b/doc/forum/support_for_git_sparse_checkout/comment_2_3eadd702526c31b73b187d2e89f2e1ab._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="Reigniting interested in this topic and linking to related efforts (BABS etc)" + date="2026-03-02T16:18:28Z" + content=""" +I just saw support for [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout) merged in [BABS](https://github.com/PennLINC/babs/pull/337) and frankly I never knew/used it before! Inspired by an enthusiastic [Meng](https://github.com/just-meng) who ~~made a strategic mistake for her PhD progress~~ [pioneered use of use of git worktrees](https://blog.datalad.org/posts/git-worktree-workflow/) in DataLad having attended Distribits 2025, I thought to check if git-annex has support for the `sparse-checkout`. + +In conjunction with `sparse-checkout` (existing already) support for worktrees in `git-annex` can make a perfect \"couple\" for an efficient [ephemeral](https://myyoda.github.io/principles-examples/stamped_principles/e/) compute where we checkout only what is really needed, e.g. following the `datalad run` input/output specifications. + +This is just a summary of the potential research/implementation since may be it even somehow magically all works already given that BABS merged the sparse-checkout support and they extensively use `git annex` already...? +"""]]
comment
diff --git a/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_8_ed8bec2b8b941fd816cfb17704f5290f._comment b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_8_ed8bec2b8b941fd816cfb17704f5290f._comment new file mode 100644 index 0000000000..2df3b55051 --- /dev/null +++ b/doc/bugs/git_annex_export_--fast_deletes_files_on_remote/comment_8_ed8bec2b8b941fd816cfb17704f5290f._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 8""" + date="2026-03-02T16:02:51Z" + content=""" +Ok, I've tagged the todos about import support from rsync, and hopefully +that will be able to get implemented. + +As for this bug, it seems that at least documentation improvements are +needed in order to close it. I have also fixed the adb special remote to +avoid the behavior, which leaves webdav and any external special remotes +that might have the behavior. +"""]]
Added a comment: Reigniting interested in this topic and linking to related efforts (BABS etc)
diff --git a/doc/forum/support_for_git_sparse_checkout/comment_1_d65f3931efc3bdc101d12594694db4f4._comment b/doc/forum/support_for_git_sparse_checkout/comment_1_d65f3931efc3bdc101d12594694db4f4._comment new file mode 100644 index 0000000000..2388998b37 --- /dev/null +++ b/doc/forum/support_for_git_sparse_checkout/comment_1_d65f3931efc3bdc101d12594694db4f4._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="Reigniting interested in this topic and linking to related efforts (BABS etc)" + date="2026-03-02T16:17:46Z" + content=""" +I just saw support for [git sparse-checkout](https://git-scm.com/docs/git-sparse-checkout) merged in [BABS](https://github.com/PennLINC/babs/pull/337) and frankly I never knew/used it before! Inspired by an enthusiastic [Meng](https://github.com/just-meng) who ~~made a strategic mistake for her PhD progress~~ [pioneered use of use of git worktrees](https://blog.datalad.org/posts/git-worktree-workflow/) in DataLad having attended Distribits 2025, I thought to check if git-annex has support for the `sparse-checkout`. + +In conjunction with `sparse-checkout` (existing already) support for worktrees in `git-annex` can make a perfect \"couple\" for an efficient [ephemeral](https://myyoda.github.io/principles-examples/stamped_principles/e/) compute where we checkout only what is really needed, e.g. following the `datalad run` input/output specifications. + +This is just a summary of the potential research/implementation since may be it even somehow magically all works already given that BABS merged the sparse-checkout support and they extensively use `git annex` already...? +"""]]
adb: Avoid deleting contents of a non-empty directory when removing the last exported file from the directory
Same as was done for rsync in commit e1ce4a530bbcd7ded1bd67613c15f9f75005336d
toybox rmdir supports --ignore-fail-on-non-empty, as does busybox rmdir,
so I assume this will work broadly across current android.
Same as was done for rsync in commit e1ce4a530bbcd7ded1bd67613c15f9f75005336d
toybox rmdir supports --ignore-fail-on-non-empty, as does busybox rmdir,
so I assume this will work broadly across current android.
diff --git a/CHANGELOG b/CHANGELOG
index 2fb70abd8b..951afee0c9 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,6 +10,8 @@ git-annex (10.20260214) UNRELEASED; urgency=medium
* Improve display of http exceptions.
* Fix reversion in previous version that caused auto-initializing of
local git remotes that have annex-ignore set.
+ * adb: Avoid deleting contents of a non-empty directory when
+ removing the last exported file from the directory.
-- Joey Hess <id@joeyh.name> Mon, 16 Feb 2026 13:38:21 -0400
diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index 745e4dc207..1352296057 100644
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -274,7 +274,7 @@ removeExportDirectoryM serial abase dir =
unlessM go $
giveup "adb failed"
where
- go = adbShellBool serial [Param "rm", Param "-rf", File (fromAndroidPath adir)]
+ go = adbShellBool serial [Param "rmdir", Param "--ignore-fail-on-non-empty", File (fromAndroidPath adir)]
adir = androidExportLocation abase (mkExportLocation (fromExportDirectory dir))
checkPresentExportM :: AndroidSerial -> AndroidPath -> Key -> ExportLocation -> Annex Bool
diff --git a/Types/Remote.hs b/Types/Remote.hs
index 87a4f1009e..e83395a3c2 100644
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -288,7 +288,7 @@ data ExportActions a = ExportActions
, removeExport :: Key -> ExportLocation -> a ()
-- Removes an exported directory. Typically the directory will be
-- empty, but it could possibly contain files or other directories,
- -- and it's ok to delete those (but not required to).
+ -- and it's ok to delete those (but better to avoid doing so).
-- If the remote does not use directories, or automatically cleans
-- up empty directories, this can be Nothing.
--
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 0bb70b7e60..f89256798c 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -91,8 +91,8 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
The directory will be in the form of a relative path, and may contain path
separators, whitespace, and other special characters.
Typically the directory will be empty, but it could possibly contain
- files or other directories, and it's ok to remove those, but not required
- to do so.
+ files or other directories, and it's ok to remove those, but better to
+ avoid doing so.
* `REMOVEEXPORTDIRECTORY-SUCCESS`
Indicates that a `REMOVEEXPORTDIRECTORY` was done successfully.
* `REMOVEEXPORTDIRECTORY-FAILURE`