Recent changes to this wiki:
Added a comment
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_5_e13e18adc7af4ceb0f38b30b16469341._comment b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_5_e13e18adc7af4ceb0f38b30b16469341._comment
new file mode 100644
index 0000000000..f643f92a42
--- /dev/null
+++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_5_e13e18adc7af4ceb0f38b30b16469341._comment
@@ -0,0 +1,52 @@
+[[!comment format=mdwn
+ username="vekhir@e0f2b370dc5ee632235b2b67ea19272db6560883"
+ nickname="vekhir"
+ avatar="http://cdn.libravatar.org/avatar/4c48a5ac216c1a5dc55c1653ed26a3a2"
+ subject="comment 5"
+ date="2026-10-03T01:09:05Z"
+ content="""
+I had success compiling with crypton 1.1.0 with the patch below. We need to specify `ram` to use it in Utility/Hash/Crypton.hs
+
+```
+diff --git a/Utility/Hash/Crypton.hs b/Utility/Hash/Crypton.hs
+index 520dd2ff79..101ad3e00f 100644
+--- a/Utility/Hash/Crypton.hs
++++ b/Utility/Hash/Crypton.hs
+@@ -7,6 +7,7 @@
+
+ {-# LANGUAGE BangPatterns, PackageImports #-}
+ {-# LANGUAGE RankNTypes #-}
++{-# LANGUAGE CPP #-}
+
+ module Utility.Hash.Crypton (
+ sha1,
+@@ -70,7 +71,11 @@ module Utility.Hash.Crypton (
+ import qualified Data.ByteString as S
+ import qualified Data.ByteString.Lazy as L
+ import Data.IORef
++#if MIN_VERSION_crypton(1,1,0)
++import qualified \"ram\" Data.ByteArray as BA
++#else
+ import qualified \"memory\" Data.ByteArray as BA
++#endif
+ import \"crypton\" Crypto.Hash
+
+ import Utility.Hash.Types
+diff --git a/git-annex.cabal b/git-annex.cabal
+index 806d23f60f..0fe8bfa522 100644
+--- a/git-annex.cabal
++++ b/git-annex.cabal
+@@ -318,8 +318,9 @@ Executable git-annex
+ Build-Depends:
+ base (>= 4.18.2.1 && < 5),
+ persistent (>= 2.13.3),
+- crypton,
+- tls
++ crypton (>= 1.1.0),
++ ram (>= 0.20.1 && < 0.23),
++ tls (>= 2.3.0 && < 2.5)
+
+ -- Fully optimize for production.
+ if flag(Production)
+```
+"""]]
Added a comment
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_4_e1b1985c59d130321222cdd7df07df44._comment b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_4_e1b1985c59d130321222cdd7df07df44._comment new file mode 100644 index 0000000000..f39ba1a055 --- /dev/null +++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_4_e1b1985c59d130321222cdd7df07df44._comment @@ -0,0 +1,47 @@ +[[!comment format=mdwn + username="vekhir@e0f2b370dc5ee632235b2b67ea19272db6560883" + nickname="vekhir" + avatar="http://cdn.libravatar.org/avatar/4c48a5ac216c1a5dc55c1653ed26a3a2" + subject="comment 4" + date="2026-10-03T00:01:09Z" + content=""" +For the 10.20260901 release, the two errors are + +``` +[ 41 of 757] Compiling Utility.Hash.Crypton ( Utility/Hash/Crypton.hs, dist/build/git-annex/git-annex-tmp/Utility/Hash/Crypton.dyn_o ) + +Utility/Hash/Crypton.hs:227:27: error: [GHC-39999] + • No instance for ‘BA.ByteArrayAccess (Digest a)’ + arising from a use of ‘BA.convert’ + • In the second argument of ‘(.)’, namely ‘BA.convert’ + In the expression: HashDigest . BA.convert + In an equation for ‘hashDigest’: + hashDigest = HashDigest . BA.convert + | +227 | hashDigest = HashDigest . BA.convert + | ^^^^^^^^^^ +``` +and +``` +[187 of 757] Compiling Annex.Balanced ( Annex/Balanced.hs, dist/build/git-annex/git-annex-tmp/Annex/Balanced.dyn_o ) + +Annex/Balanced.hs:39:19: error: [GHC-39999] + • No instance for ‘BA.ByteArrayAccess (Digest a)’ + arising from a use of ‘BA.unpack’ + • In the second argument of ‘(.)’, namely ‘BA.unpack’ + In the expression: + L.foldl' (\ i b -> (i `shiftL` 8) + fromIntegral b) 0 . BA.unpack + In an equation for ‘tointeger’: + tointeger + = L.foldl' (\ i b -> (i `shiftL` 8) + fromIntegral b) 0 . BA.unpack + | +39 | . BA.unpack + | ^^^^^^^^^ +``` + +Going off `master`, the latter issue in Annex/Balanced.hs has been resolved while Utility/Hash/Crypton.hs is still affected. + +On a different note, I've tried building with botan-low, which failed to build due to the same issue above, i.e. Utility/Hash/Crypton.hs gets compiled even when using botan. I would expect that using botan would remove any use of crypton. + +I like your work regarding the use of LLMs even in dependencies. I wasn't previously aware that there were already so many (and so widely used!) projects that included LLM generated code. +"""]]
testremote: Add --ensure-readonly mode
I've tested this against a ssh git remote with
GIT_ANNEX_SHELL_READONLY=true set, and it passes.
I've tested this against a ssh git remote with
GIT_ANNEX_SHELL_READONLY=true set, and it passes.
diff --git a/CHANGELOG b/CHANGELOG
index 38c0ec142c..964a0d725a 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -30,6 +30,7 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* When built with Botan, also use it for HMAC.
* test, testremote: Support --tap output when built with TastyTap build
flag.
+ * testremote: Add --ensure-readonly mode.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Command/TestRemote.hs b/Command/TestRemote.hs
index d6ed27a29c..dda678c57c 100644
--- a/Command/TestRemote.hs
+++ b/Command/TestRemote.hs
@@ -60,6 +60,7 @@ data TestRemoteOptions = TestRemoteOptions
, sizeOption :: ByteSize
, tapOutput :: Bool
, testReadonlyFile :: [FilePath]
+ , ensureReadonlyFile :: Maybe FilePath
}
optParser :: CmdParamsDesc -> Parser TestRemoteOptions
@@ -75,10 +76,14 @@ optParser desc = TestRemoteOptions
<> help "use TAP output"
)
<*> many testreadonly
+ <*> optional (option str
+ ( long "ensure-readonly" <> metavar paramFile
+ <> help "readonly test object (may be deleted from remote)"
+ ))
where
testreadonly = option str
( long "test-readonly" <> metavar paramFile
- <> help "readonly test object"
+ <> help "readonly test object (will not be modified)"
)
seek :: TestRemoteOptions -> CommandSeek
@@ -90,15 +95,17 @@ start o = starting "testremote" (ActionItemOther (Just (UnquotedString (testRemo
cache <- liftIO newRemoteVariantCache
r <- either giveup (disableExportTree cache)
=<< Remote.byName' (testRemote o)
- ks <- case testReadonlyFile o of
- [] -> if Remote.readonly r
- then giveup "This remote is readonly, so you need to use the --test-readonly option."
- else do
- showAction "generating test keys"
- NE.fromList
- <$> mapM randKey (keySizes basesz fast)
- fs -> NE.fromList <$> mapM (getReadonlyKey r . toOsPath) fs
- let r' = if null (testReadonlyFile o)
+ ks <- if ensureReadonlyFile o == Nothing
+ then case testReadonlyFile o of
+ [] -> if Remote.readonly r
+ then giveup "This remote is readonly, so you need to use the --test-readonly option."
+ else gentestkeys fast
+ fs -> NE.fromList <$> mapM (getReadonlyKey r . toOsPath) fs
+ else gentestkeys fast
+ ensurereadonlyk <- case ensureReadonlyFile o of
+ Nothing -> pure Nothing
+ Just f -> Just <$> getReadonlyKey r (toOsPath f)
+ let r' = if null (testReadonlyFile o) && ensureReadonlyFile o == Nothing
then r
else r { Remote.readonly = True }
let drs = if Remote.readonly r'
@@ -108,13 +115,16 @@ start o = starting "testremote" (ActionItemOther (Just (UnquotedString (testRemo
let exportr = if Remote.readonly r'
then return Nothing
else exportTreeVariant cache r'
- perform o drs unavailr exportr ks
+ perform o drs unavailr exportr ks ensurereadonlyk
where
basesz = fromInteger $ sizeOption o
si = SeekInput [testRemote o]
+ gentestkeys fast = do
+ showAction "generating test keys"
+ NE.fromList <$> mapM randKey (keySizes basesz fast)
-perform :: TestRemoteOptions -> [Described (Annex (Maybe Remote))] -> Maybe Remote -> Annex (Maybe Remote) -> NE.NonEmpty Key -> CommandPerform
-perform o drs unavailr exportr ks = do
+perform :: TestRemoteOptions -> [Described (Annex (Maybe Remote))] -> Maybe Remote -> Annex (Maybe Remote) -> NE.NonEmpty Key -> Maybe Key -> CommandPerform
+perform o drs unavailr exportr ks ensurereadonlyk = do
st <- liftIO . newTVarIO =<< (,)
<$> Annex.getState id
<*> Annex.getRead id
@@ -123,7 +133,8 @@ perform o drs unavailr exportr ks = do
drs
(pure unavailr)
exportr
- (NE.map (\k -> Described (desck k) (pure k)) ks)
+ (NE.map mkdesck ks)
+ (mkdesck <$> ensurereadonlyk)
ok <- case tryIngredients testingredients mempty tests of
Nothing -> error "No tests found!?"
Just act -> liftIO act
@@ -131,6 +142,7 @@ perform o drs unavailr exportr ks = do
next $ cleanup rs (NE.toList ks) ok
where
desck k = unwords [ "key size", show (fromKey keySize k) ]
+ mkdesck k = Described (desck k) (pure k)
testingredients =
if tapOutput o
#ifdef WITH_TASTYTAP
@@ -239,22 +251,31 @@ mkTestTrees
-> Annex (Maybe Remote)
-> Annex (Maybe Remote)
-> (NE.NonEmpty (Described (Annex Key)))
+ -> (Maybe (Described (Annex Key)))
-> [TestTree]
-mkTestTrees runannex mkrs mkunavailr mkexportr mkks = concat $
- [ [ inOrderTestGroup "unavailable remote" (testUnavailable runannex mkunavailr (getVal (NE.head mkks))) ]
- , [ inOrderTestGroup (desc mkr mkk) (test runannex (getVal mkr) (getVal mkk)) | mkk <- NE.toList mkks, mkr <- mkrs ]
- , [ inOrderTestGroup (descexport mkk1 mkk2) (testExportTree runannex mkexportr (getVal mkk1) (getVal mkk2)) | mkk1 <- take 2 (NE.toList mkks), mkk2 <- take 2 (reverse (NE.toList mkks)) ]
- ]
+mkTestTrees runannex mkrs mkunavailr mkexportr mkks mkensurereadonlyk = concat $
+ case mkensurereadonlyk of
+ Nothing ->
+ [ [ inOrderTestGroup "unavailable remote" (testUnavailable runannex mkunavailr (getVal (NE.head mkks))) ]
+ , [ inOrderTestGroup (desc mkr mkk) (test runannex (getVal mkr) (getVal mkk)) | mkk <- NE.toList mkks, mkr <- mkrs ]
+ , [ inOrderTestGroup (descexport mkk1 mkk2) (testExportTree runannex mkexportr (getVal mkk1) (getVal mkk2)) | mkk1 <- take 2 (NE.toList mkks), mkk2 <- take 2 (reverse (NE.toList mkks)) ]
+ ]
+ Just ensurereadonlyk ->
+ [ [ inOrderTestGroup (desc mkr mkk) (testEnsureReadOnly runannex (getVal mkr) (getVal mkk) (getVal ensurereadonlyk)) | mkk <- NE.toList mkks, mkr <- mkrs ]
+ ]
where
- desc r k = intercalate "; " $ map unwords
+ desc r k = combinedescs
[ [ getDesc k ]
, [ getDesc r ]
+ , map getDesc $ maybeToList mkensurereadonlyk
]
- descexport k1 k2 = intercalate "; " $ map unwords
+ descexport k1 k2 = combinedescs
[ [ "exporttree=yes" ]
, [ getDesc k1 ]
, [ getDesc k2 ]
+ , map getDesc $ maybeToList mkensurereadonlyk
]
+ combinedescs = intercalate "; " . map unwords . filter (not . null)
test :: RunAnnex -> Annex (Maybe Remote) -> Annex Key -> [TestTree]
test runannex mkr mkk =
@@ -327,6 +348,33 @@ test runannex mkr mkk =
store r k = Remote.storeKey r k (AssociatedFile Nothing) Nothing nullMeterUpdate
remove r k = Remote.removeKey r Nothing k
+testEnsureReadOnly :: RunAnnex -> Annex (Maybe Remote) -> Annex Key -> Annex Key -> [TestTree]
+testEnsureReadOnly runannex mkr mkk mkensurereadonlyk =
+ [ check "removeKey when present fails" $ \r _ k ->
+ shouldfail $ runBool (remove r k)
+ , check "removeKey did not remove" $ \r _ k ->
+ present r k True
+ , check "removeKey when not present fails" $ \r k _ ->
+ shouldfail $ runBool (remove r k)
+ , check "storeKey when not present fails" $ \r k _ ->
+ shouldfail $ runBool (store r k)
+ ]
+ where
+ check desc a = testCase desc $ do
+ let a' = mkr >>= \case
+ Just r -> do
+ k <- mkk
+ ensurereadonlyk <- mkensurereadonlyk
+ a r k ensurereadonlyk
+ Nothing -> return True
+ runannex a' @? "failed"
+ shouldfail a = tryNonAsync a >>= \case
+ Left _ -> return True
+ Right _ -> return False
+ present r k b = (== Right b) <$> Remote.hasKey r k
+ store r k = Remote.storeKey r k (AssociatedFile Nothing) Nothing nullMeterUpdate
+ remove r k = Remote.removeKey r Nothing k
+
testExportTree :: RunAnnex -> Annex (Maybe Remote) -> Annex Key -> Annex Key -> [TestTree]
testExportTree runannex mkr mkk1 mkk2 =
[ check "check present export when not present" $ \ea k1 _k2 ->
diff --git a/Test.hs b/Test.hs
index 27857569be..d2992dc25c 100644
--- a/Test.hs
+++ b/Test.hs
@@ -266,7 +266,7 @@ testRemote testvariants remotetype setupremote =
cv <- annexeval cache
liftIO $ atomically $ putTMVar v
(r, (unavailr, (exportr, (ks, cv))))
- go getv = Command.TestRemote.mkTestTrees runannex mkrs mkunavailr mkexportr (NE.fromList mkks)
+ go getv = Command.TestRemote.mkTestTrees runannex mkrs mkunavailr mkexportr (NE.fromList mkks) Nothing
where
runannex = inmainrepo . annexeval
mkrs = if testvariants
diff --git a/doc/git-annex-testremote.mdwn b/doc/git-annex-testremote.mdwn
index 42bf01e76f..6d16bb7599 100644
--- a/doc/git-annex-testremote.mdwn
+++ b/doc/git-annex-testremote.mdwn
@@ -41,6 +41,19 @@ is limited to checking various properties of downloading from it.
This option can be repeated.
+ When this option is used, testremote will avoid changing the remote in
(Diff truncated)
comment
diff --git a/doc/todo/More_fine-grained_testremote_command/comment_5_20d4a64cf52386a99b9624a6a5e3c2a8._comment b/doc/todo/More_fine-grained_testremote_command/comment_5_20d4a64cf52386a99b9624a6a5e3c2a8._comment new file mode 100644 index 0000000000..ae61c2533e --- /dev/null +++ b/doc/todo/More_fine-grained_testremote_command/comment_5_20d4a64cf52386a99b9624a6a5e3c2a8._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-10-02T17:30:10Z" + content=""" +> Now that you are saying this, is a new option even necessary? +> --test-readonly already takes a filename that is expected to be present +> on the remote, so instead of adding a new option --test-readonly could +> ensure that this key can't be removed, and that a different key can't be +> stored (and that removeKey succeeds on this not-present key). + +I think it needs a new option. Someone might be using --test-readonly with +a file that they don't want to lose. Currently it never tries to remove a +file from a readonly remote, so that is safe enough, though not really +documented as such. +"""]]
testremote --tap
Note that the use of noMessages in Command.TestRemote is to prevent
"recording state in git" at the end, which is not tap output. There may
be others.
It seemed fine to also suppress those messages when not running --tap
mode.
Note that the use of noMessages in Command.TestRemote is to prevent
"recording state in git" at the end, which is not tap output. There may
be others.
It seemed fine to also suppress those messages when not running --tap
mode.
diff --git a/CHANGELOG b/CHANGELOG
index 015f6497c4..38c0ec142c 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -28,7 +28,8 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* NoLLMDependencies: Update for tls.
* Support bup 0.34, while also still working with previous versions.
* When built with Botan, also use it for HMAC.
- * test: Support --tap output when built with TastyTap build flag.
+ * test, testremote: Support --tap output when built with TastyTap build
+ flag.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Command/TestRemote.hs b/Command/TestRemote.hs
index f8cfde2e2a..d6ed27a29c 100644
--- a/Command/TestRemote.hs
+++ b/Command/TestRemote.hs
@@ -1,11 +1,12 @@
{- git-annex command
-
- - Copyright 2014-2020 Joey Hess <id@joeyh.name>
+ - Copyright 2014-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
{-# LANGUAGE RankNTypes, DeriveFunctor, PackageImports, OverloadedStrings #-}
+{-# LANGUAGE CPP #-}
module Command.TestRemote where
@@ -38,6 +39,9 @@ import Test.Framework
import Test.Tasty
import Test.Tasty.Runners
import Test.Tasty.HUnit
+#ifdef WITH_TASTYTAP
+import Test.Tasty.Runners.TAP
+#endif
import "crypto-api" Crypto.Random
import qualified Data.ByteString as B
import qualified Data.ByteString.Lazy as L
@@ -47,13 +51,14 @@ import Control.Concurrent.STM hiding (check)
import qualified Data.List.NonEmpty as NE
cmd :: Command
-cmd = command "testremote" SectionTesting
+cmd = noMessages $ command "testremote" SectionTesting
"test transfers to/from a remote"
paramRemote (seek <$$> optParser)
data TestRemoteOptions = TestRemoteOptions
{ testRemote :: RemoteName
, sizeOption :: ByteSize
+ , tapOutput :: Bool
, testReadonlyFile :: [FilePath]
}
@@ -65,6 +70,10 @@ optParser desc = TestRemoteOptions
<> value (1024 * 1024)
<> help "base key size (default 1MiB)"
)
+ <*> switch
+ ( long "tap"
+ <> help "use TAP output"
+ )
<*> many testreadonly
where
testreadonly = option str
@@ -99,13 +108,13 @@ start o = starting "testremote" (ActionItemOther (Just (UnquotedString (testRemo
let exportr = if Remote.readonly r'
then return Nothing
else exportTreeVariant cache r'
- perform drs unavailr exportr ks
+ perform o drs unavailr exportr ks
where
basesz = fromInteger $ sizeOption o
si = SeekInput [testRemote o]
-perform :: [Described (Annex (Maybe Remote))] -> Maybe Remote -> Annex (Maybe Remote) -> NE.NonEmpty Key -> CommandPerform
-perform drs unavailr exportr ks = do
+perform :: TestRemoteOptions -> [Described (Annex (Maybe Remote))] -> Maybe Remote -> Annex (Maybe Remote) -> NE.NonEmpty Key -> CommandPerform
+perform o drs unavailr exportr ks = do
st <- liftIO . newTVarIO =<< (,)
<$> Annex.getState id
<*> Annex.getRead id
@@ -115,13 +124,22 @@ perform drs unavailr exportr ks = do
(pure unavailr)
exportr
(NE.map (\k -> Described (desck k) (pure k)) ks)
- ok <- case tryIngredients [consoleTestReporter] mempty tests of
+ ok <- case tryIngredients testingredients mempty tests of
Nothing -> error "No tests found!?"
Just act -> liftIO act
rs <- catMaybes <$> mapM getVal drs
next $ cleanup rs (NE.toList ks) ok
where
desck k = unwords [ "key size", show (fromKey keySize k) ]
+ testingredients =
+ if tapOutput o
+#ifdef WITH_TASTYTAP
+ then [ tapRunner ]
+#else
+ then error "git-annex was built without --tap support"
+#endif
+ else []
+ ++ [ consoleTestReporter ]
remoteVariants :: RemoteVariantCache -> Described (Annex Remote) -> Int -> Bool -> [Described (Annex (Maybe Remote))]
remoteVariants cache dr basesz fast =
diff --git a/doc/git-annex-testremote.mdwn b/doc/git-annex-testremote.mdwn
index 77f7aebaed..42bf01e76f 100644
--- a/doc/git-annex-testremote.mdwn
+++ b/doc/git-annex-testremote.mdwn
@@ -45,6 +45,10 @@ is limited to checking various properties of downloading from it.
Tune the base size of generated objects. The default is 1MiB.
+* `--tap`
+
+ Use TAP output (see https://testanything.org/).
+
* Also the [[git-annex-common-options]](1) can be used.
# SEE ALSO
diff --git a/doc/todo/More_fine-grained_testremote_command/comment_4_224024f3e8344aacb81ba5848e22948f._comment b/doc/todo/More_fine-grained_testremote_command/comment_4_224024f3e8344aacb81ba5848e22948f._comment
new file mode 100644
index 0000000000..86f3088849
--- /dev/null
+++ b/doc/todo/More_fine-grained_testremote_command/comment_4_224024f3e8344aacb81ba5848e22948f._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2026-10-02T17:17:38Z"
+ content="""
+TAP output is now supported: `git-annex testremote --tap`
+
+It needs git-annex to be built with the TastyTap build flag.
+"""]]
update
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
index 7b1ace7c7e..7a4f255232 100644
--- a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
+++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
@@ -3,7 +3,7 @@
subject="""comment 2"""
date="2026-10-02T14:33:10Z"
content="""
-I have thought of three different ways to support concurrent tests with TAP
+I have thought of several different ways to support concurrent tests with TAP
output:
1. Include a TAP parser (or at least a parser for the subset of TAP that
@@ -20,6 +20,10 @@ output:
processes can be concacenated easily. This would probably entail forking
tasty-tap, but it is a small amount of code, so total code would
probably be less than option #1.
+4. Clean up git-annex's hacks around parallelism and tasty by making
+ tasty itself support running the sub-test processes. That would leave
+ tasty-tap reporting on the collection of all the tasty test results at
+ the top level. This would be the best solution.
However, I think this should be deferred to a later todo if it turns out to
be needed.
done
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn
index 9c8fc05146..de0e728d88 100644
--- a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn
+++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn
@@ -3,3 +3,5 @@ There is also a somewhat related [SARIF](https://sarifweb.azurewebsites.net/) (S
[[!meta author=yoh]]
[[!tag projects/repronim]]
+
+> implemented TastyTap build flag and --tap, [[done]] --[[Joey]]
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
index 1091add508..7b1ace7c7e 100644
--- a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
+++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment
@@ -20,4 +20,7 @@ output:
processes can be concacenated easily. This would probably entail forking
tasty-tap, but it is a small amount of code, so total code would
probably be less than option #1.
+
+However, I think this should be deferred to a later todo if it turns out to
+be needed.
"""]]
test: Support --tap output when built with TastyTap build flag
Parallel testing is not supported with --tap since it would result in multiple
TAP output blocks.
parallelTestRunner still used, even though it needed several special cases to
work in this mode. The reason is the comment on it:
- An added benefit of using child processes is that any files they may
- leave open are closed before finalCleanup is run at the end. This
- prevents some failures to clean up after the test suite.
Also though, I anticipate maybe supporting parallel testing with --tap
in the future, so it didn't seem to make sense to refactor parallelTestRunner
in the ways that would be needed to avoid using it.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Parallel testing is not supported with --tap since it would result in multiple
TAP output blocks.
parallelTestRunner still used, even though it needed several special cases to
work in this mode. The reason is the comment on it:
- An added benefit of using child processes is that any files they may
- leave open are closed before finalCleanup is run at the end. This
- prevents some failures to clean up after the test suite.
Also though, I anticipate maybe supporting parallel testing with --tap
in the future, so it didn't seem to make sense to refactor parallelTestRunner
in the ways that would be needed to avoid using it.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
diff --git a/BuildFlags.hs b/BuildFlags.hs
index 97d92d838d..7e6bcc8a69 100644
--- a/BuildFlags.hs
+++ b/BuildFlags.hs
@@ -82,6 +82,11 @@ buildFlags = filter (not . null)
#endif
#ifdef WITH_NOLLMDEPENDENCIES
, "NoLLMDependencies"
+#endif
+#ifdef WITH_TASTYTAP
+ , "TastyTap"
+#else
+#warning Building without tasty-tap support.
#endif
]
diff --git a/CHANGELOG b/CHANGELOG
index cbf7a43f15..015f6497c4 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -28,6 +28,7 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* NoLLMDependencies: Update for tls.
* Support bup 0.34, while also still working with previous versions.
* When built with Botan, also use it for HMAC.
+ * test: Support --tap output when built with TastyTap build flag.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Test.hs b/Test.hs
index 0234bb5088..27857569be 100644
--- a/Test.hs
+++ b/Test.hs
@@ -98,7 +98,7 @@ import qualified Utility.Gpg
optParser :: Parser TestOptions
optParser = TestOptions
- <$> snd (tastyParser (tests 1 False defaulttos))
+ <$> snd (tastyParser defaulttos (tests 1 False defaulttos))
<*> switch
( long "keep-failures"
<> help "preserve repositories on test failure"
@@ -121,6 +121,10 @@ optParser = TestOptions
( long "test-debug"
<> help "show debug messages for commands run by test suite"
)
+ <*> switch
+ ( long "tap"
+ <> help "use TAP output"
+ )
<*> cmdParams "non-options are for internal use only"
where
parseconfigvalue s = case break (== '=') s of
@@ -136,11 +140,12 @@ optParser = TestOptions
, concurrentJobs = Nothing
, testGitConfig = mempty
, testDebug = False
+ , tapOutput = False
, internalData = mempty
}
runner :: TestOptions -> IO ()
-runner opts = parallelTestRunner opts tests
+runner opts = testRunner opts tests
tests :: Int -> Bool -> TestOptions -> [TestTree]
tests numparts crippledfilesystem opts =
diff --git a/Test/Framework.hs b/Test/Framework.hs
index f47de2c91d..7dd8c3cd1d 100644
--- a/Test/Framework.hs
+++ b/Test/Framework.hs
@@ -21,6 +21,9 @@ import Test.Tasty.Options
import Test.Tasty.Ingredients.Rerun
import Test.Tasty.Ingredients.ConsoleReporter
import qualified Test.Tasty.Patterns.Types as TP
+#ifdef WITH_TASTYTAP
+import Test.Tasty.Runners.TAP
+#endif
import Options.Applicative.Types
import Control.Concurrent
import Control.Concurrent.Async
@@ -772,6 +775,23 @@ runFakeSsh (_host:cmd:[]) =
\_ _ _ pid -> exitWith =<< waitForProcess pid
runFakeSsh ps = error $ "fake ssh option parse error: " ++ show ps
+testRunner :: TestOptions -> (Int -> Bool -> TestOptions -> [TestTree]) -> IO ()
+testRunner opts mkts
+ | fakeSsh opts = runFakeSsh (internalData opts)
+ | tapOutput opts =
+#ifdef WITH_TASTYTAP
+ parallelTestRunner 1 opts mkts
+#else
+ error "git-annex was built without --tap support"
+#endif
+ | otherwise = do
+ numjobs <- case concurrentJobs opts of
+ Just NonConcurrent -> pure 1
+ Just (Concurrent n) -> pure n
+ Just ConcurrentPerCpu -> getNumProcessors
+ Nothing -> getNumProcessors
+ parallelTestRunner numjobs opts mkts
+
{- Tests each TestTree in parallel, and exits with success/failure.
-
- Tasty supports parallel tests, but this does not use it, because
@@ -782,17 +802,8 @@ runFakeSsh ps = error $ "fake ssh option parse error: " ++ show ps
- leave open are closed before finalCleanup is run at the end. This
- prevents some failures to clean up after the test suite.
-}
-parallelTestRunner :: TestOptions -> (Int -> Bool -> TestOptions -> [TestTree]) -> IO ()
-parallelTestRunner opts mkts = do
- numjobs <- case concurrentJobs opts of
- Just NonConcurrent -> pure 1
- Just (Concurrent n) -> pure n
- Just ConcurrentPerCpu -> getNumProcessors
- Nothing -> getNumProcessors
- parallelTestRunner' numjobs opts mkts
-
-parallelTestRunner' :: Int -> TestOptions -> (Int -> Bool -> TestOptions -> [TestTree]) -> IO ()
-parallelTestRunner' numjobs opts mkts
+parallelTestRunner :: Int -> TestOptions -> (Int -> Bool -> TestOptions -> [TestTree]) -> IO ()
+parallelTestRunner numjobs opts mkts
| fakeSsh opts = runFakeSsh (internalData opts)
| otherwise = go =<< Utility.Env.getEnv subenv
where
@@ -808,6 +819,11 @@ parallelTestRunner' numjobs opts mkts
then 1
else numjobs * 2
+ mkts' crippledfilesystem
+ | tapOutput opts =
+ [topLevelTestGroup $ mkts 1 crippledfilesystem opts]
+ | otherwise = mkts numparts crippledfilesystem opts
+
worker rs nvar a = do
(n, m) <- atomically $ do
(n, m) <- readTVar nvar
@@ -819,7 +835,12 @@ parallelTestRunner' numjobs opts mkts
r <- a n
worker (r:rs) nvar a
- summarizeresults a = do
+ summarizeresults a
+ | tapOutput opts = do
+ _ <- a
+ return ()
+ | otherwise = summarizeresults' a
+ summarizeresults' a = do
starttime <- getCurrentTime
(numts, exitcodes) <- a
duration <- Utility.HumanTime.durationSince starttime
@@ -846,8 +867,8 @@ parallelTestRunner' numjobs opts mkts
<$> Annex.Init.probeCrippledFileSystem'
(toOsPath tmpdir)
Nothing Nothing False
- let ts = mkts numparts crippledfilesystem opts
- let warnings = fst (tastyParser ts)
+ let ts = mkts' crippledfilesystem
+ let warnings = fst (tastyParser opts ts)
unless (null warnings) $ do
hPutStrLn stderr "warnings from tasty:"
mapM_ (hPutStrLn stderr) warnings
@@ -855,9 +876,11 @@ parallelTestRunner' numjobs opts mkts
args <- getArgs
pp <- fromOsPath <$> Annex.Path.programPath
termcolor <- hSupportsANSIColor stdout
- let ps = if useColor (lookupOption tastyopts) termcolor
- then "--color=always":args
- else "--color=never":args
+ let ps = if tapOutput opts
+ then args
+ else if useColor (lookupOption tastyopts) termcolor
+ then "--color=always":args
+ else "--color=never":args
let runone n = do
let subdir = fromOsPath $ toOsPath tmpdir </> toOsPath (show n)
ensuredir subdir
@@ -875,9 +898,11 @@ parallelTestRunner' numjobs opts mkts
go (Just subenvval) = case readish subenvval of
Nothing -> error ("Bad " ++ subenv)
Just (n, crippledfilesystem) -> setTestEnv $ do
- let ts = mkts numparts crippledfilesystem opts
- let t = topLevelTestGroup [ ts !! (n - 1) ]
- case tryIngredients ingredients tastyopts t of
+ let ts = mkts' crippledfilesystem
+ let t = if tapOutput opts
+ then ts !! 0
+ else topLevelTestGroup [ ts !! (n - 1) ]
+ case tryIngredients (ingredients opts) tastyopts t of
Nothing -> error "No tests found!?"
Just act -> ifM act
( exitSuccess
@@ -901,14 +926,20 @@ topLevelTestGroup = inOrderTestGroup "Tests"
initTestsName :: String
initTestsName = "Init Tests"
-tastyParser :: [TestTree] -> ([String], Parser Test.Tasty.Options.OptionSet)
-tastyParser ts = suiteOptionParser ingredients (topLevelTestGroup ts)
-
-ingredients :: [Ingredient]
-ingredients =
(Diff truncated)
comment
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment new file mode 100644 index 0000000000..1091add508 --- /dev/null +++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_2_38754cd75fb11a237372a7e20bfdb024._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-10-02T14:33:10Z" + content=""" +I have thought of three different ways to support concurrent tests with TAP +output: + +1. Include a TAP parser (or at least a parser for the subset of TAP that + tasty-tap uses, which does not include YAML blocks), so subtest + process output is gathered, parsed, and re-emitted. (Also needs a TAP + generator.) +2. Add an option like `--test-portion=1/3` so the caller can run multiple + processes in parallel and do whatever combining they like. This is close + to the way subtest processes are implemented currently, so probably the + easiest option. Also more flexible since it can be used without using + TAP. +3. Since test numbers are actually optional in TAP output. modify tasty-tap + to not include test numbers, and then the outputs of concurrent + processes can be concacenated easily. This would probably entail forking + tasty-tap, but it is a small amount of code, so total code would + probably be less than option #1. +"""]]
crypton
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index b4b1b6f7bd..504c0e38ae 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -120,14 +120,21 @@ seems likely to have attribution or licencing issues. For example, see [this issue](https://github.com/kazu-yamamoto/crypton/issues/232). +The developer +[states](https://kazu-yamamoto.hatenablog.jp/entry/2026/09/25/110313) +that "I am not a cryptography expert, so I cannot write code that is +both correct and performant." This begs the question: Does anyone +understand and has anyone reviewed the LLM generated code? + The point of cryptonite was to package up industry standard C implementations of crypto functions in haskell. This is no longer that, and as such, does not seem like a trustworthly package to use. -Unfortunately, it is a dependency of several other packages, -including `tls`. git-annex supports being built with older versions. The botan build flag -enables using botan for most things rather than crypton. +makes git-annex use botan rather than crypton. + +Unfortunately, crypton is a dependency of several other packages, +including `tls`. ### tls
comment
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_3_6913dbd4d3ebcfc44a6dda6894871aab._comment b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_3_6913dbd4d3ebcfc44a6dda6894871aab._comment new file mode 100644 index 0000000000..d4bda26c91 --- /dev/null +++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_3_6913dbd4d3ebcfc44a6dda6894871aab._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-10-01T15:29:58Z" + content=""" +I suspect but have not verified that the problem is the use of memory in +Utility/Hash/Crypton.hs, which is the only place it should touch crypton +types. And it will probably need to be ifdefed in order to support old and +new versions of crypton. +"""]]
comment
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_2_050fd6c285c32a44f6a94b533a9e809d._comment b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_2_050fd6c285c32a44f6a94b533a9e809d._comment new file mode 100644 index 0000000000..35def2a162 --- /dev/null +++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_2_050fd6c285c32a44f6a94b533a9e809d._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-10-01T14:34:45Z" + content=""" +Please show the build error message. +"""]]
Added a comment: Alternative: botan instead of crypton
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_1_a833d747aa209641b86972e6ac7e9e5a._comment b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_1_a833d747aa209641b86972e6ac7e9e5a._comment new file mode 100644 index 0000000000..9c533a90d6 --- /dev/null +++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram/comment_1_a833d747aa209641b86972e6ac7e9e5a._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="vekhir@e0f2b370dc5ee632235b2b67ea19272db6560883" + nickname="vekhir" + avatar="http://cdn.libravatar.org/avatar/4c48a5ac216c1a5dc55c1653ed26a3a2" + subject="Alternative: botan instead of crypton" + date="2026-09-30T02:19:53Z" + content=""" +The botan flag allows avoiding the dependency on crypton, instead depending on botan which is free from LLM generated code and better supported. More info in the blog entry [no llm code](https://git-annex.branchable.com/no_llm_code/#index7h3). +"""]]
diff --git a/doc/bugs/crypton_1.1.0_support___47___switch_to_ram.mdwn b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram.mdwn new file mode 100644 index 0000000000..fae7eece3f --- /dev/null +++ b/doc/bugs/crypton_1.1.0_support___47___switch_to_ram.mdwn @@ -0,0 +1,23 @@ +### Please describe the problem in your own words. + +crypton 1.1.0 switched from memory to ram, the latter providing the same interface with its own definitions. Thus, using memory is incompatible with crypton 1.1.0. + +### What steps will reproduce the problem? + +Build the latest release 10.20260901 against crypton 1.1.0. +The build fails with an error message regarding mismatching types. + +### What version of git-annex are you using? On what operating system? + +* OS: Arch Linux +* Kernel: Linux 7.2.6-arch2-1 +* GHC: 9.6.6 +* git-annex: 10.20260901 +* crypton: 1.1.0 +* memory: 0.18.0 + +ram: 0.22.1 + +### Please provide any additional information below. + +The relevant imports explicitly mention the memory module: `import "memory" ...` and `import qualified "memory" ...`. Without the explicit module mention, it would work as-is, as ram provides the same interface.
todo
diff --git a/doc/todo/server_side_mirroring.mdwn b/doc/todo/server_side_mirroring.mdwn new file mode 100644 index 0000000000..d5e8eab7c1 --- /dev/null +++ b/doc/todo/server_side_mirroring.mdwn @@ -0,0 +1,52 @@ +Use case is wanting to set up a server with a git-annex repository which is +a mirror of an original repository. +All git refs are mirrored, using eg `git push --mirror`. + +So, the git-annex branch cannot diverge. Or if it does, any divergence will get +overwritten the next time the refs mirror is updated. + +The mirror can contain some, but not necessarily all annex objects as the +original repository. If a client requents an annex object that is not +present in the mirror, it transparently proxies the request to the original +repository. It may also, optionally, cache the object in the mirror (if +the mirror's preferred content configuration wants it). + +A client dropping an object from the mirror would have it proxy the drop to +the original repository, and only delete its cache if that drop succeeded. + +It seems at first that the mirror repository would need to have +the same uuid as the original repository. Since the client sees the +original git-annex branch, it only knows about the uuid in that branch +for location tracking. + +But using the same uuid would bring in a ton of special cases and problems. +Just a few of them are: +* `git-annex get` in the mirror repository would fail to get files from the + original repository, because the local repo uuid is filtered out of the + remote list in `keyPossibilities'`. +* The mirror couldn't have its own preferred content configuration. + +The mirror repository could have the same uuid as the original set in +annex.uuid, so the client *thinks* it has the same uuid, but actually have +a different uuid that is configured elsewhere. Make git-annex use that +other uuid, and with annex.private set, it will never reach the git-annex +branch. + +In the P2P protocol, `AUTH-SUCCESS` includes the server's uuid. +That would need to be the original repository's uuid, to avoid a +client thinking it's talking to the wrong repository. + +--- + +Concretely, a mirror could be configured like this: + + git remote add original http://example.com/foo + git pull --prune mirror + git config annex.mirroring original + git config annex.private true + git-annex init + # that sets annex.uuid to the discovered uuid of original + # and it sets annex.mirror-uuid to the repo's true uuid + git-annex mirror --from=original # optional + +[[!tag projects/INM7]]
support bup 0.34, while also still working with previous versions
bup no longer allows -r with a path, only supporting host:path and some uri
forms. The fix is to use bup -d when the buprepo is a path. That also works
with older versions of bup.
It's worth noting that there is a small behavior difference with bup init.
bup init -r /tmp/foo creates /tmp/foo but also ~/.bup
bup -d /tmp/foo init creates only /tmp/foo
I think this is ok, the ~/.bup that created did not actually get used in
any significant way. The only thing it wrote to is ~/.bup/index-cache/
and it seems to work w/o that.
One exception to that is if initremote is run with this method, but then
git-annex is downgraded, and the bup special remote is accessed. The way
git-annex used to run bup, it will then complain that ~/.bup/ doesn't
exist.
Sponsored-by: Leon Schuermann
bup no longer allows -r with a path, only supporting host:path and some uri
forms. The fix is to use bup -d when the buprepo is a path. That also works
with older versions of bup.
It's worth noting that there is a small behavior difference with bup init.
bup init -r /tmp/foo creates /tmp/foo but also ~/.bup
bup -d /tmp/foo init creates only /tmp/foo
I think this is ok, the ~/.bup that created did not actually get used in
any significant way. The only thing it wrote to is ~/.bup/index-cache/
and it seems to work w/o that.
One exception to that is if initremote is run with this method, but then
git-annex is downgraded, and the bup special remote is accessed. The way
git-annex used to run bup, it will then complain that ~/.bup/ doesn't
exist.
Sponsored-by: Leon Schuermann
diff --git a/CHANGELOG b/CHANGELOG
index 218e65fb19..5cb8fe0ee0 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -26,6 +26,7 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
is no recorded content identifier.
* NoLLMDependencies: Update for crypton.
* NoLLMDependencies: Update for tls.
+ * Support bup 0.34, while also still working with previous versions.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Remote/Bup.hs b/Remote/Bup.hs
index 5ff440ff45..3bc0e5041c 100644
--- a/Remote/Bup.hs
+++ b/Remote/Bup.hs
@@ -1,6 +1,6 @@
{- Using bup as a remote.
-
- - Copyright 2011-2022 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -41,7 +41,22 @@ import Annex.Perms
import Utility.Metered
import Types.ProposedAccepted
-type BupRepo = String
+data BupRepo
+ = BupRepoPath FilePath
+ | BupRepoRemote String
+
+parseBupRepo :: String -> BupRepo
+parseBupRepo s
+ | ':' `elem` s = BupRepoRemote s
+ | otherwise = BupRepoPath s
+
+serializeBupRepo :: BupRepo -> String
+serializeBupRepo (BupRepoPath p) = p
+serializeBupRepo (BupRepoRemote r) = r
+
+bupRepoLocal :: BupRepo -> Bool
+bupRepoLocal (BupRepoPath _) = True
+bupRepoLocal (BupRepoRemote _ ) = False
remote :: RemoteType
remote = specialRemoteType $ RemoteType
@@ -67,7 +82,7 @@ gen r u rc gc rs = do
c <- parsedRemoteConfig remote rc
bupr <- liftIO $ bup2GitRemote buprepo
cst <- remoteCost gc c $
- if bupLocal buprepo
+ if bupRepoLocal buprepo
then nearlyCheapRemoteCost
else expensiveRemoteCost
(u', bupr') <- getBupUUID bupr u
@@ -86,7 +101,7 @@ gen r u rc gc rs = do
, removeKey = removeKeyDummy
, lockContent = Nothing
, checkPresent = checkPresentDummy
- , checkPresentCheap = bupLocal buprepo
+ , checkPresentCheap = bupRepoLocal buprepo
, exportActions = exportUnsupported
, importActions = importUnsupported
, exportImportActions = exportImportUnsupported
@@ -97,18 +112,21 @@ gen r u rc gc rs = do
, config = c
, getRepo = return r
, gitconfig = gc
- , localpath = if bupLocal buprepo && not (null buprepo)
- then Just (toOsPath buprepo)
- else Nothing
+ , localpath = case buprepo of
+ BupRepoPath p | not (null p) -> Just (toOsPath p)
+ _ -> Nothing
, remotetype = remote
- , availability = if null buprepo
- then pure LocallyAvailable
- else checkPathAvailability (bupLocal buprepo) (toOsPath buprepo)
+ , availability = case buprepo of
+ BupRepoPath p
+ | null p -> pure LocallyAvailable
+ | otherwise ->
+ checkPathAvailability True (toOsPath p)
+ BupRepoRemote _ -> pure GloballyAvailable
, readonly = False
, appendonly = False
, untrustworthy = False
, mkUnavailable = return Nothing
- , getInfo = return [("repo", buprepo)]
+ , getInfo = return [("repo", serializeBupRepo buprepo)]
, claimUrl = Nothing
, checkUrl = Nothing
, remoteStateHandle = rs
@@ -124,15 +142,17 @@ gen r u rc gc rs = do
(checkKey bupr')
this
where
- buprepo = fromMaybe (giveup "missing buprepo") $ remoteAnnexBupRepo gc
+ buprepo = maybe (giveup "missing buprepo") parseBupRepo $
+ remoteAnnexBupRepo gc
bupSetup :: SetupStage -> Maybe UUID -> RemoteName -> Maybe CredPair -> RemoteConfig -> RemoteGitConfig -> Annex (RemoteConfig, UUID)
bupSetup ss mu _ _ c gc = do
u <- maybe (liftIO genUUID) return mu
-- verify configuration is sane
- let buprepo = maybe (giveup "Specify buprepo=") fromProposedAccepted $
- M.lookup buprepoField c
+ let buprepo = maybe (giveup "Specify buprepo=")
+ (parseBupRepo . fromProposedAccepted)
+ (M.lookup buprepoField c)
(c', _encsetup) <- encryptionSetup ss c gc
-- bup init will create the repository.
@@ -144,13 +164,15 @@ bupSetup ss mu _ _ c gc = do
-- The buprepo is stored in git config, as well as this repo's
-- persistent state, so it can vary between hosts.
- gitConfigSpecialRemote u c' [("buprepo", buprepo)]
+ gitConfigSpecialRemote u c' [("buprepo", serializeBupRepo buprepo)]
return (c', u)
bupParams :: String -> BupRepo -> [CommandParam] -> [CommandParam]
-bupParams command buprepo params =
- Param command : [Param "-r", Param buprepo] ++ params
+bupParams command (BupRepoPath buprepo) params =
+ Param "-d" : Param buprepo : Param command : params
+bupParams command (BupRepoRemote buprepo) params =
+ Param command : Param "-r" : Param buprepo : params
bup :: String -> BupRepo -> [CommandParam] -> Annex Bool
bup command buprepo params = do
@@ -295,19 +317,18 @@ getBupUUID r u
Right r' -> return (toUUID $ Git.Config.get configkeyUUID mempty r', r')
Left _ -> return (NoUUID, r)
-{- Converts a bup remote path spec into a Git.Repo. There are some
- - differences in path representation between git and bup. -}
+{- Converts a BupRepo into a Git.Repo. There are some
+ - differences in representation between git and bup. -}
bup2GitRemote :: BupRepo -> IO Git.Repo
-bup2GitRemote "" = do
- -- bup -r "" operates on ~/.bup
+bup2GitRemote (BupRepoPath "") = do
+ -- bup -d "" operates on ~/.bup
h <- myHomeDir
Git.Construct.fromPath $ toOsPath h </> literalOsPath ".bup"
-bup2GitRemote r
- | bupLocal r =
- if "/" `isPrefixOf` r
- then Git.Construct.fromPath (toOsPath r)
- else giveup "please specify an absolute path"
- | otherwise = Git.Construct.fromUrl False $ "ssh://" ++ host ++ slash dir
+bup2GitRemote (BupRepoPath r)
+ | "/" `isPrefixOf` r = Git.Construct.fromPath (toOsPath r)
+ | otherwise = giveup "please specify an absolute path"
+bup2GitRemote (BupRepoRemote r) =
+ Git.Construct.fromUrl False $ "ssh://" ++ host ++ slash dir
where
bits = splitc ':' r
host = fromMaybe "" $ headMaybe bits
@@ -329,9 +350,6 @@ bupRef k
where
shown = serializeKey k
-bupLocal :: BupRepo -> Bool
-bupLocal = notElem ':'
-
{- Bup is not concurrency safe, so use a lock file. Only one writer process
- should run at a time; multiple readers may run if no writer is running. -}
lockBup :: Bool -> Remote -> Annex a -> Annex a
diff --git a/doc/bugs/support_new_bup_version.mdwn b/doc/bugs/support_new_bup_version.mdwn
index 4f4993d559..cfabf4086e 100644
--- a/doc/bugs/support_new_bup_version.mdwn
+++ b/doc/bugs/support_new_bup_version.mdwn
@@ -10,3 +10,5 @@ even though the new bup does also support `bup init <path>`
The `bup -d` approach seems to work with older versions of bup as
well. --[[Joey]]
+
+> [[done]] --[[Joey]]
bug
diff --git a/doc/bugs/support_new_bup_version.mdwn b/doc/bugs/support_new_bup_version.mdwn new file mode 100644 index 0000000000..4f4993d559 --- /dev/null +++ b/doc/bugs/support_new_bup_version.mdwn @@ -0,0 +1,12 @@ +bup 0.34 or so changed in ways that breaks the bup special remote +when not operating on a ssh remote. This affects `git-annex test`. + +--remote no longer accepts a local path, but only a host:path or +some url forms (not file:// though even though it is documented to be +supported). + +Instead, use eg `bup -d <path> split`. Will also work for `init`, +even though the new bup does also support `bup init <path>` + +The `bup -d` approach seems to work with older versions of bup as +well. --[[Joey]]
on crypton
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 13276f3b9a..b4b1b6f7bd 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -111,10 +111,21 @@ changelog. ### crypton -[cryton](https://hackage.haskell.org/package/crypton) since 1.1.0 +[crypton](https://hackage.haskell.org/package/crypton) since 1.1.0 [First LLM generated code](https://github.com/kazu-yamamoto/crypton/commit/cfe36d439a075aa4cce72baafa8bbcc58dabf85b) +Later LLM generated code includes large amounts of C code that +seems likely to have attribution or licencing issues. +For example, see +[this issue](https://github.com/kazu-yamamoto/crypton/issues/232). + +The point of cryptonite was to package up industry standard C +implementations of crypto functions in haskell. This is no longer +that, and as such, does not seem like a trustworthly package to use. +Unfortunately, it is a dependency of several other packages, +including `tls`. + git-annex supports being built with older versions. The botan build flag enables using botan for most things rather than crypton.
NoLLMDependencies: Update for tls
stack-NoLLMDependencies.yaml has tls-2.1.8 rather than tls-2.4.3
which is the last version w/o LLM use. 2.2.2 is the last one that
depends on a version of crypton w/o LLM use. Various other dependency
issues prevented using 2.2.2; 2.1.8 is what lts-24.52 uses.
stack-NoLLMDependencies.yaml has tls-2.1.8 rather than tls-2.4.3
which is the last version w/o LLM use. 2.2.2 is the last one that
depends on a version of crypton w/o LLM use. Various other dependency
issues prevented using 2.2.2; 2.1.8 is what lts-24.52 uses.
diff --git a/CHANGELOG b/CHANGELOG
index 86546639d3..218e65fb19 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -25,6 +25,7 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* Allow retrieval from some importtree=yes special remotes when there
is no recorded content identifier.
* NoLLMDependencies: Update for crypton.
+ * NoLLMDependencies: Update for tls.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn
index 7948ded69d..13276f3b9a 100644
--- a/doc/no_llm_code.mdwn
+++ b/doc/no_llm_code.mdwn
@@ -118,6 +118,14 @@ changelog.
git-annex supports being built with older versions. The botan build flag
enables using botan for most things rather than crypton.
+### tls
+
+[tls](https://hackage.haskell.org/package/tls) since 2.4.4
+
+[First LLM generated code](https://github.com/haskell-tls/hs-tls/commit/218abdd67a4cde30854b5ca9b146b269da53bc07)
+
+git-annex supports being built with older versions.
+
### Cabal
[First LLM generated code](https://github.com/haskell/cabal/commit/da8b314563feb15a3df7bc1baeef4b7aa08f7578)
diff --git a/git-annex.cabal b/git-annex.cabal
index d4970cfea5..90bcda4cdd 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -291,7 +291,6 @@ Executable git-annex
warp-tls (>= 3.2.2),
crypton-connection (>= 0.4.3),
crypton-x509-store,
- tls,
aws (>= 0.24.1)
CC-Options: -Wall
GHC-Options: -Wall -fno-warn-tabs -Wincomplete-uni-patterns
@@ -309,12 +308,14 @@ Executable git-annex
persistent (>= 2.13.3) && (< 2.15.0.0),
warp (< 3.4.11),
magic (<= 1.1),
- crypton (< 1.1.0)
+ crypton (< 1.1.0),
+ tls (< 2.4.4)
else
Build-Depends:
base (>= 4.18.2.1 && < 5),
persistent (>= 2.13.3),
- crypton
+ crypton,
+ tls
-- Fully optimize for production.
if flag(Production)
diff --git a/stack-NoLLMDependencies.yaml b/stack-NoLLMDependencies.yaml
index 0b54175eca..8d2f3ee7b9 100644
--- a/stack-NoLLMDependencies.yaml
+++ b/stack-NoLLMDependencies.yaml
@@ -30,3 +30,4 @@ extra-deps:
- yesod-static-1.6.1.2
- magic-1.1
- crypton-1.0.6
+- tls-2.1.8
poc
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_1_99a2c2e59eef83fb9d24f503101a3606._comment b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_1_99a2c2e59eef83fb9d24f503101a3606._comment new file mode 100644 index 0000000000..6f02803a16 --- /dev/null +++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__/comment_1_99a2c2e59eef83fb9d24f503101a3606._comment @@ -0,0 +1,62 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-26T16:45:53Z" + content=""" +<https://hackage.haskell.org/package/tasty-tap> can be used to do this. + +A proof of concept patch is below. + +It always uses TAP format, which I would not want because personally I +find it hard to read and scan. Using TAP needs to be an option. + +Note that every concurrent test job produces its own TAP output. +To avoid that problem, in TAP mode the test suite would need to not run +concurrent tests. + +The added dependency will need a build flag as tasty-tap is not packaged in eg, Debian. + + diff --git a/Test/Framework.hs b/Test/Framework.hs + index f47de2c91d..8729bbc531 100644 + --- a/Test/Framework.hs + +++ b/Test/Framework.hs + @@ -20,6 +20,7 @@ import Test.Tasty.HUnit + import Test.Tasty.Options + import Test.Tasty.Ingredients.Rerun + import Test.Tasty.Ingredients.ConsoleReporter + +import Test.Tasty.Runners.TAP + import qualified Test.Tasty.Patterns.Types as TP + import Options.Applicative.Types + import Control.Concurrent + @@ -855,9 +856,7 @@ parallelTestRunner' numjobs opts mkts + args <- getArgs + pp <- fromOsPath <$> Annex.Path.programPath + termcolor <- hSupportsANSIColor stdout + - let ps = if useColor (lookupOption tastyopts) termcolor + - then "--color=always":args + - else "--color=never":args + + let ps = args + let runone n = do + let subdir = fromOsPath $ toOsPath tmpdir </> toOsPath (show n) + ensuredir subdir + @@ -907,6 +906,7 @@ tastyParser ts = suiteOptionParser ingredients (topLevelTestGroup ts) + ingredients :: [Ingredient] + ingredients = + [ listingTests + + , tapRunner + , rerunningTests [consoleTestReporter] + ] + + diff --git a/git-annex.cabal b/git-annex.cabal + index d4970cfea5..b48f2323de 100644 + --- a/git-annex.cabal + +++ b/git-annex.cabal + @@ -277,6 +277,7 @@ Executable git-annex + tasty-hunit, + tasty-quickcheck, + tasty-rerun, + + tasty-tap, + ansi-terminal >= 0.9, + DAV (>= 1.0), + network (>= 3.0.0.0), +"""]]
crypton 1.1.0 did not depend on ram 0.21.0
Not that it matters since that version of crypton has LLM generated
code.
Not that it matters since that version of crypton has LLM generated
code.
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 67230622e2..7948ded69d 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -66,7 +66,6 @@ Rather than use ram, git-annex continues to use the unmaintained But ram is an dependency of other dependencies, and these in particular depend on 0.21.0 or newer: -* crypton since 1.1.0 (itself containing LLM code) * tls since 2.3.1 [Reverse dependencies of ram](https://packdeps.haskellers.com/reverse/ram)
crypton 

diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index ee66360bc0..67230622e2 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -66,7 +66,7 @@ Rather than use ram, git-annex continues to use the unmaintained But ram is an dependency of other dependencies, and these in particular depend on 0.21.0 or newer: -* crypton since 1.1.0 +* crypton since 1.1.0 (itself containing LLM code) * tls since 2.3.1 [Reverse dependencies of ram](https://packdeps.haskellers.com/reverse/ram) @@ -110,6 +110,15 @@ disclosed their earlier LLM use, but it is apparent, including in their communications to [[Joey]] about incorrect LLM-generated statements in the changelog. +### crypton + +[cryton](https://hackage.haskell.org/package/crypton) since 1.1.0 + +[First LLM generated code](https://github.com/kazu-yamamoto/crypton/commit/cfe36d439a075aa4cce72baafa8bbcc58dabf85b) + +git-annex supports being built with older versions. The botan build flag +enables using botan for most things rather than crypton. + ### Cabal [First LLM generated code](https://github.com/haskell/cabal/commit/da8b314563feb15a3df7bc1baeef4b7aa08f7578)
TODO: on TAP output
diff --git a/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn new file mode 100644 index 0000000000..9c8fc05146 --- /dev/null +++ b/doc/todo/provide_TAP_protocol_logging_for___39__annex_test__39__.mdwn @@ -0,0 +1,5 @@ +ATM outputs from `git annex test` are custom. I personally find it difficult to find results of failure etc. But what is more important is that it is difficult to script deterministic analytics on top of such results (e.g. as xfailing some specific tests in some specific contexts). I know of [TAP](https://testanything.org) protocol which I believe a number of generic testing harnesses support. May be could be some form of `--output-format text,tap` options for `git annex test`. +There is also a somewhat related [SARIF](https://sarifweb.azurewebsites.net/) (Static Analysis Results Interchange Format), which was suggested in another scope, but not familiar with it at all in practice. + +[[!meta author=yoh]] +[[!tag projects/repronim]]
layout
diff --git a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn index 3d85be2808..066e468bd8 100644 --- a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn +++ b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn @@ -67,6 +67,6 @@ If you find combinations that work, please edit this page to list them. `git-annex export` to the directory remote. Once that is done, `git-annex get` will work from the rsync remote. * directory configured with importtree=yes and/or exporttree=yes and rsync configured with - importtree=yes + importtree=yes Works same as the previous combination, since git-annex version 10.20260924.
done
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn index 26df585a3f..17a624955d 100644 --- a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn @@ -99,3 +99,5 @@ get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f [[!tag projects/ICE4]] + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_5_9702e602bfd32e0ab6816fd679dd4d00._comment b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_5_9702e602bfd32e0ab6816fd679dd4d00._comment new file mode 100644 index 0000000000..01a6540188 --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_5_9702e602bfd32e0ab6816fd679dd4d00._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-09-24T16:43:32Z" + content=""" +Patch applied. +"""]]
allow retrieval from importtree=yes special remote when there is no recorded content identifier
This allows using a combination of a directory special remote
configured with importtree=yes and/or exporttree=yes and rsync
special remote configured with importtree=yes. After import or export
with the directory special remote, get from the rsync special remote will
work, despite no content identifier being recorded for the rsync special
remote.
This kind of combination only works when the second special remote type
does not itself support importtree=yes. Since currently retrieveImport
is implemented for such using retrieveExportWithContentIdentifier,
which will fail if it does not get any content identifiers.
Note that the fall back to retrieveFromExport inside retrieveFromImport
was dead code, because retrieveFromImport is not used for import+export
remotes.
This allows using a combination of a directory special remote
configured with importtree=yes and/or exporttree=yes and rsync
special remote configured with importtree=yes. After import or export
with the directory special remote, get from the rsync special remote will
work, despite no content identifier being recorded for the rsync special
remote.
This kind of combination only works when the second special remote type
does not itself support importtree=yes. Since currently retrieveImport
is implemented for such using retrieveExportWithContentIdentifier,
which will fail if it does not get any content identifiers.
Note that the fall back to retrieveFromExport inside retrieveFromImport
was dead code, because retrieveFromImport is not used for import+export
remotes.
diff --git a/CHANGELOG b/CHANGELOG
index 41cdaa6786..32473c39c2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -22,6 +22,8 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* Automate updating remote.name.annexUrl for annex+https servers,
by re-probing the http remote's annex.url config when unable to connect
to an annex+https server.
+ * Allow retrieval from importtree=yes special remote when there
+ is no recorded content identifier.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Remote/Helper/ExportImport.hs b/Remote/Helper/ExportImport.hs
index 23bd86c1ec..768b05f365 100644
--- a/Remote/Helper/ExportImport.hs
+++ b/Remote/Helper/ExportImport.hs
@@ -452,20 +452,15 @@ adjustExportImport' isexport isimport isexportimport annexobjects r rs gc = do
else retrieveWithoutContentIdentifier $
retrieveFromExport getlocs k af dest p
- retrieveFromImport getlocs ciddbv k af dest p = do
+ retrieveFromImport getlocs ciddbv k _af dest p = do
cids <- getkeycids ciddbv k
- if not (null cids)
- then getlocs $ \loc ->
- -- retrieveImport does not guarantee that
- -- the file it retrieves has the content
- -- identifier, so it must be strongly
- -- verified.
- stronglyverify $
- snd <$> retrieveImport (importActions r) loc cids dest (Left k) p
- -- In case a content identifier is somehow missing,
- -- try this instead.
- else retrieveWithoutContentIdentifier $
- retrieveFromExport getlocs k af dest p
+ getlocs $ \loc ->
+ -- retrieveImport does not guarantee that
+ -- the file it retrieves corresponds to any
+ -- content identifier, so it must be strongly
+ -- verified.
+ stronglyverify $
+ snd <$> retrieveImport (importActions r) loc cids dest (Left k) p
retrieveWithoutContentIdentifier a
| isexport = a
diff --git a/Types/Remote.hs b/Types/Remote.hs
index 16b3e43e64..560af0c746 100644
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -481,8 +481,8 @@ data ImportActions a = ImportActions
-- key.
, importKey :: a (Maybe (ImportLocation -> ContentIdentifier -> ByteSize -> MeterUpdate -> a (Maybe Key)))
-- Like retrieveExportWithContentIdentifier, but does not
- -- need to guarantee that the file it retrieves has one
- -- of the requested ContentIdentifiers.
+ -- need to guarantee that the file it retrieves corresponds
+ -- to any of the listed ContentIdentifiers.
, retrieveImport
:: ImportLocation
-> [ContentIdentifier]
diff --git a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn
index 030eb6ad6b..3d85be2808 100644
--- a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn
+++ b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn
@@ -66,3 +66,7 @@ If you find combinations that work, please edit this page to list them.
`git-annex import --fast` from the directory remote, or
`git-annex export` to the directory remote.
Once that is done, `git-annex get` will work from the rsync remote.
+* directory configured with importtree=yes and/or exporttree=yes and rsync configured with
+ importtree=yes
+ Works same as the previous combination, since git-annex
+ version 10.20260924.
document combination
diff --git a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn index 2900655835..030eb6ad6b 100644 --- a/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn +++ b/doc/tips/multiple_remotes_accessing_the_same_data_store.mdwn @@ -60,3 +60,9 @@ If you find combinations that work, please edit this page to list them. * httpalso and rclone (using git-remote-rclone) (any layout except for frankencase) * httpalso and any special remote that uses exporttree=yes +* directory configured with importtree=yes and/or exporttree=yes and rsync configured with + exporttree=yes + This combination allows populating both remotes using + `git-annex import --fast` from the directory remote, or + `git-annex export` to the directory remote. + Once that is done, `git-annex get` will work from the rsync remote.
comment
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_4_65f6004f36b429abc01f9991d2ea698c._comment b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_4_65f6004f36b429abc01f9991d2ea698c._comment new file mode 100644 index 0000000000..b0515d3e35 --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_4_65f6004f36b429abc01f9991d2ea698c._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-09-24T15:35:02Z" + content=""" +Maybe this patch is ok to apply without worrying about those other cases. +--sameas is not guaranteed to work for all combinations of special remotes +and that's why [[tips/multiple_remotes_accessing_the_same_data_store]] +documents combinations that work. + +So it could just be documented that this one specific combination also works. +And it happens to be a useful combination to support. If we later want some +other combination like import from rsync followed by get from sameas +directory to also work, the extra code can be written then to support it. +(By eg making Remote.Directory not implement importActions using +exportImportActions, or making its retrieveExportWithContentIdentifier +not fail when the `[ContentIdentifier]` list is empty) +"""]]
patch
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_3_916dd3a154384b8b48890d11b95c67dd._comment b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_3_916dd3a154384b8b48890d11b95c67dd._comment new file mode 100644 index 0000000000..3293e115b5 --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_3_916dd3a154384b8b48890d11b95c67dd._comment @@ -0,0 +1,55 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2026-09-24T15:11:26Z" + content=""" +Following is a small patch that makes import from directory, followed by get +from the sameas rsync work. + +Note that, import from rsync followed by get from sameas directory *won't* +work with this patch. The reason is that directory special remotes support +importree+exporttree. And even if they are only configured with exporttree, +they currently verify ContentIdentifiers when retrieiving. And when no +ContentIdentifier is known, it will fail. + +So, applying this patch seems like it would need remotes like directory to +have a separate implementation of importActions, rather than the current way +that exportImportActions is used to implement importActions. There are not +too many remotes that support importtree+exporttree, so maybe that is worth +doing. + +The other limitation of the patch is that import from rsync followed by get +from sameas directory that is configured with importree+exporttree still +won't work. And this approach cannot deal with that case. + + diff --git a/Remote/Helper/ExportImport.hs b/Remote/Helper/ExportImport.hs + index 23bd86c1ec..7db9ddebfb 100644 + --- a/Remote/Helper/ExportImport.hs + +++ b/Remote/Helper/ExportImport.hs + @@ -454,18 +454,13 @@ adjustExportImport' isexport isimport isexportimport annexobjects r rs gc = do + + retrieveFromImport getlocs ciddbv k af dest p = do + cids <- getkeycids ciddbv k + - if not (null cids) + - then getlocs $ \loc -> + - -- retrieveImport does not guarantee that + - -- the file it retrieves has the content + - -- identifier, so it must be strongly + - -- verified. + - stronglyverify $ + - snd <$> retrieveImport (importActions r) loc cids dest (Left k) p + - -- In case a content identifier is somehow missing, + - -- try this instead. + - else retrieveWithoutContentIdentifier $ + - retrieveFromExport getlocs k af dest p + + getlocs $ \loc -> + + -- retrieveImport does not guarantee that + + -- the file it retrieves has any content + + -- identifier, so it must be strongly + + -- verified. + + stronglyverify $ + + snd <$> retrieveImport (importActions r) loc cids dest (Left k) p + + retrieveWithoutContentIdentifier a + | isexport = a +"""]]
comment
diff --git a/doc/todo/support_local_git_remote_acting_as_a_proxy/comment_1_991ec3e4cc29b0293c475ab99baf1ee6._comment b/doc/todo/support_local_git_remote_acting_as_a_proxy/comment_1_991ec3e4cc29b0293c475ab99baf1ee6._comment new file mode 100644 index 0000000000..0c11aadc57 --- /dev/null +++ b/doc/todo/support_local_git_remote_acting_as_a_proxy/comment_1_991ec3e4cc29b0293c475ab99baf1ee6._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-24T14:37:05Z" + content=""" +See also [[todo/optimise_use_of_keys_db_on_local_remote]] +"""]]
fix tag
diff --git a/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn b/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn index 3462021af6..310b642fe2 100644 --- a/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn +++ b/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn @@ -15,4 +15,4 @@ that a local git remote that acts as a proxy will keep a removable drive in use as long as git-annex is running or b) avoid tearing down the git-annex-shell for a few seconds so it's still available for the next use. -[[!tag projects/INM7]] +[[!tag projects/ICE4]]
comment
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_2_7a2e23906b6160f375f1193c947fa636._comment b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_2_7a2e23906b6160f375f1193c947fa636._comment new file mode 100644 index 0000000000..4a8df353f9 --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_2_7a2e23906b6160f375f1193c947fa636._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-09-24T14:23:45Z" + content=""" +A use case for this is having a directory special remote that is --sameas a +rsync special remote. `git-annex import --fast` from the directory special +remote imports the tree and that avoids needing to import from the rsync +special remote, which would be much slower for a large data set. + +A second import step would be acceptable in this situation. +The user would need to make sure that the files on the remotes +are unchanged from the directory import when running the rsync import. +This seems like it could be handled as a special flag to `git-annex import`. +"""]]
comment
diff --git a/doc/todo/optimise_use_of_keys_db_on_local_remote/comment_1_52c39340a7ceb20f3519836bb6a152f8._comment b/doc/todo/optimise_use_of_keys_db_on_local_remote/comment_1_52c39340a7ceb20f3519836bb6a152f8._comment new file mode 100644 index 0000000000..e1e4f73cce --- /dev/null +++ b/doc/todo/optimise_use_of_keys_db_on_local_remote/comment_1_52c39340a7ceb20f3519836bb6a152f8._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-24T14:15:47Z" + content=""" +When this was filed in 2020 I'm not sure why it would do that, +but later on [[!commit ba7ecbc6a9c3763e8152e4f46522d14a4ee2b59d]] +made `quiesce` call `Database.Keys.closeDb` and so this is entirely the +expected behavior. + +There is certainly the potential for less frequent use of `quiesce` +avoiding this overhead though. It could eg wait 1 second after last +use before quiescing. +"""]]
add todo
diff --git a/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn b/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn new file mode 100644 index 0000000000..3462021af6 --- /dev/null +++ b/doc/todo/support_local_git_remote_acting_as_a_proxy.mdwn @@ -0,0 +1,18 @@ +It would be useful in some cases to be able to have a local git remote +that acts as a proxy to other remotes. That is not currently supported. +See [[!commit f98605bce7ecfa7fd155d501accb2734d3e9a422]]. + +Supporting this could be as easy as using git-annex-shell on the path of +the local git remote and speaking P2P protocol to it. + +A possible complication with that: In between uses of a local git remote, +git-annex is careful to flush files and close handles. That is to support +removable drives, so nothing is lost if one is ejected while git-annex is +running, and to make it possible to unmount them. If git-annex-shell +has to be spun up once per file, and then shut down afterwards, that might +be quite slow. Possible approaches to deal with that would be a) document +that a local git remote that acts as a proxy will keep a removable drive in +use as long as git-annex is running or b) avoid tearing down the +git-annex-shell for a few seconds so it's still available for the next use. + +[[!tag projects/INM7]]
comment
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_1_7094cab70cc38bb31fc0ea558c036522._comment b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_1_7094cab70cc38bb31fc0ea558c036522._comment new file mode 100644 index 0000000000..b217bd04d4 --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails/comment_1_7094cab70cc38bb31fc0ea558c036522._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-24T13:45:26Z" + content=""" +This happens because the meaning of a content identifier is dependent on +the implementation of the special remote. Using a content identifier +generated for a directory special remote with a rsync special remote will +not work. + +So, for the --sameas remote, the content identifier is accessed using +its annex-config-uuid rather than the annex-uuid. And that's why +it says no content identifier is recorded, since none has been stored +using the annex-config-uuid. + +For this to work there would need to be another step taken to record +the content identifiers for the --sameas remote. Which could be something +like `git-annex import` from it -- except when I tried doing that, it +doesn't record them, because it skips over already imported files. + +I doubt that needing a second import step would meet your needs though? + +It may be that the best fix for this is to prohibit initializing a --sameas +remote with importtree=yes. +"""]]
deslop
diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn b/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn deleted file mode 100644 index 9b072e0a92..0000000000 --- a/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn +++ /dev/null @@ -1,76 +0,0 @@ -### Please describe the problem. -`git annex export` to a directory special remote can delete pre-existing files -that it never wrote. -When export encounters a path that already exists on the export remote, it does -not overwrite it — the existing file's contents are left untouched. But it still -prints `export <remote> <file> ok` and records the file as exported. That record -later authorises a deletion: once the exported tree stops listing that path, -export prints `unexport <remote> <file> ok` and removes the pre-existing file. -So the behaviour is asymmetric in an unfortunate direction: too conservative to -overwrite a file it does not own, but willing to delete that same file later. -### What steps will reproduce the problem? -Set up a directory holding data that git-annex did not put there: - - mkdir -p /tmp/target - echo "PRECIOUS-PREEXISTING-DATA" > /tmp/target/a.txt - echo "ALSO-PRECIOUS" > /tmp/target/keep.txt - -Make an annex whose tree happens to contain a file of the same name, plus one -new file: - - mkdir /tmp/work && cd /tmp/work - git init -q . - git annex init -q work - echo "DIFFERENT-CONTENT-FROM-TREE" > a.txt - echo "new" > b.txt - git annex add a.txt b.txt - git commit -qm tree - git annex initremote t type=directory encryption=none \ - directory=/tmp/target exporttree=yes - -Export: - - $ git annex export main --to t - export t a.txt ok - export t b.txt ok - $ cat /tmp/target/a.txt - PRECIOUS-PREEXISTING-DATA - -Note `a.txt` was reported as exported, but its contents were (correctly) not -overwritten. -Now export a tree that no longer contains those files. The empty tree is used -here for brevity; in practice this is just an ordinary change that drops a path. - - $ git annex export $(git hash-object -t tree /dev/null) --to t - unexport t b.txt ok - unexport t a.txt ok - $ cat /tmp/target/a.txt - cat: /tmp/target/a.txt: No such file or directory - $ cat /tmp/target/keep.txt - ALSO-PRECIOUS - -`a.txt` is gone. Its contents were never exported by git-annex, and never -existed anywhere in the annex — they are simply lost. -`keep.txt` survives, which isolates the cause: it was never named in an exported -tree, so no export record was created for it. Deletion follows the export -record, and the export record was created for a file that was never written. -### What version of git-annex are you using? On what operating system? -10.20251215 on Linux (Manjaro, x86_64). Also reproduced with a build of -10.20260718 from git. -### Please provide any additional information below. -The impact depends on what else lives in the export remote's directory. If it -holds a checked-out git repository, the deletion can remove that repository's -`.git/config`, `HEAD`, `refs/*`, `logs/*` and hooks in one pass — every one of -which export had previously declined to overwrite. In a test here that took a -working repository from 31 files to 4, after which `git log` in it reported -`fatal: not a git repository`. -A couple of related observations from the same testing, in case they are useful: -* Exporting to an *empty* directory writes every file in the tree as expected, - so the non-overwriting behaviour above is specific to paths that already - exist. -* Once export has written a file, later modifying that file on the remote and - re-running export does not restore it, with or without `--force`. -I have deliberately not proposed a fix, since the right behaviour is a design -question — whether export should refuse such a path, warn, overwrite it, or -simply not record a file it did not write, all have different consequences for -existing users. diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment deleted file mode 100644 index f628487e20..0000000000 --- a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2026-09-05T14:41:07Z" - content=""" -This bug report appears to have been generated by an LLM. - -I do not appreciate and will not engage with LLM generated content, so if -you would like this bug to be looked at and fixed in a timely manner, -please use your own words. - -I will delete this bug as probably LLM generated if you do not do so within -the next few days. -"""]] diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment deleted file mode 100644 index 82e3a4cba4..0000000000 --- a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2026-09-07T15:05:20Z" - content=""" -See [[bugs/export_does_not_overwrite_existing_third_party_files]]. -"""]]
Added a comment: Can fileprefix be changed on S3 remotes?
diff --git a/doc/special_remotes/S3/comment_42_5ef8cdbe292d0e41fcd3f6e53f5cb5ab._comment b/doc/special_remotes/S3/comment_42_5ef8cdbe292d0e41fcd3f6e53f5cb5ab._comment new file mode 100644 index 0000000000..45c8bcc33d --- /dev/null +++ b/doc/special_remotes/S3/comment_42_5ef8cdbe292d0e41fcd3f6e53f5cb5ab._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="gioele@678b7c03f524f2669b179b603f65352fcc16774e" + nickname="gioele" + avatar="http://cdn.libravatar.org/avatar/366dbda84e78aff8a8a070622aeb63ce" + subject="Can fileprefix be changed on S3 remotes?" + date="2026-09-19T22:28:36Z" + content=""" +git-annex stored a bunch of files on a S3 bucket under `foo/` and now I'd like to \"move\" them to `bar/`. + +Assuming I manually \"move\" the files on S3 to `bar/`, would `git annex enableremote s3 fileprefix=bar/` be enough to access the existing encrypted files? + +Or does `fileprefix` influence the S3 location of future files only? +"""]]
retroactively tag closed bug report
diff --git a/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn index 91935ea5ca..9df04ec875 100644 --- a/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn +++ b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn @@ -68,4 +68,6 @@ fsck f3.dat # stalls here Plenty - and I think I must have used mask special remote with more than one file before. +[[!tag projects/INM7]] + > [[fixed|done]]
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn index 22ac88131d..26df585a3f 100644 --- a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn @@ -98,3 +98,4 @@ get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +[[!tag projects/ICE4]]
diff --git a/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn new file mode 100644 index 0000000000..22ac88131d --- /dev/null +++ b/doc/bugs/get_from_sameas_remote_with_importtree__61__yes_fails.mdwn @@ -0,0 +1,100 @@ +### Please describe the problem in your own words. + +I have a directory special remote with importtree=yes for which I have ran a `git annex import` to make the files stored there known to git-annex. The same directory is also available via ssh, so I have added another rsync special remote with importtree=yes and --sameas= the directory remote. The content tracking properly shows that they both have the same keys available. The issue is that get'ing a key from the rsync remote fails with the error message "no content identifier is recorded, unable to retrieve". + + +### What steps will reproduce the problem? + +``` +mkdir importdir +echo test1 > importdir/test1.txt +echo test2 > importdir/test2.txt +mkdir test-sameas-importtree +cd test-sameas-importtree/ +git init +git annex init +git annex initremote importdir type=directory directory=../importdir importtree=yes encryption=none +git annex import importdir --from importdir +git annex drop --all --force +git annex get --key SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt +git annex initremote --sameas=importdir sameas-importdir type=directory directory=../importdir importtree=yes encryption=none +git annex findkeys --in sameas-importdir +git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from sameas-importdir +git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from importdir +``` + +(full transcript of this sequence with outputs is below) + + +### What version of git-annex are you using? On what operating system? + +``` +$ git annex version +git-annex version: 10.20260717-g0c917920c80ab1e8cc3d8f5886537708949e1659 +build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath Botan Blake3 XXH3 +dependency versions: aws-0.24.4 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 BLAKE3_256E BLAKE3_256 XXH3E XXH3 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 10 +``` + + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +$ mkdir importdir +$ echo test1 > importdir/test1.txt +$ echo test2 > importdir/test2.txt +$ mkdir test-sameas-importtree +$ cd test-sameas-importtree/ +$ git init +Leeres Git-Repository in /home/icg149/Playground/test-sameas-importtree/.git/ initialisiert +$ git annex init +init ok +(recording state in git...) +$ git annex initremote importdir type=directory directory=../importdir importtree=yes encryption=none +initremote importdir ok +(recording state in git...) +$ git annex import importdir --from importdir +list importdir ok +import importdir test2.txt +ok +import importdir test1.txt +ok +update refs/remotes/importdir/importdir ok +(recording state in git...) +$ git annex drop --all --force +drop SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt ok +drop SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt ok +(recording state in git...) +$ git annex get --key SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt +get SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt (from importdir...) ok +(recording state in git...) +$ git annex initremote --sameas=importdir sameas-importdir type=directory directory=../importdir importtree=yes encryption=none +initremote sameas-importdir ok +(recording state in git...) +$ git annex findkeys --in sameas-importdir +SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt +SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt +$ git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from sameas-importdir +get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt (from sameas-importdir...) + no content identifier is recorded, unable to retrieve +failed +get: 1 failed +[ble: exit 1] +$ git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from importdir +get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt (from importdir...) ok +(recording state in git...) + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +
Added a comment: docker build
diff --git a/doc/forum/git-annex_confused_by_a_case-preserving_filesystem__63__/comment_1_0124cb3e63ac5828848b1c3ed04e4ab8._comment b/doc/forum/git-annex_confused_by_a_case-preserving_filesystem__63__/comment_1_0124cb3e63ac5828848b1c3ed04e4ab8._comment new file mode 100644 index 0000000000..a403d671ec --- /dev/null +++ b/doc/forum/git-annex_confused_by_a_case-preserving_filesystem__63__/comment_1_0124cb3e63ac5828848b1c3ed04e4ab8._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="matteo@4122eaeed50d58332e6ae9e6ba7237fe81294399" + nickname="matteo" + avatar="http://cdn.libravatar.org/avatar/b113a0f7867c398c5e9c24646f8495b8" + subject="docker build" + date="2026-09-15T07:52:41Z" + content=""" +This issue still exists today with version 10.20260213 + +I came across it while trying to run a docker build from macOS host to a linux image. All annex symlinks became broken inside of the image. Thank you for reporting this! +"""]]
diff --git a/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn b/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn index bd7533b176..288a68af92 100644 --- a/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn +++ b/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn @@ -1,7 +1,7 @@ I have a group called external for my different external hard drives and numcopies set to 2. I was trying to make sure that the external group itself had two copies of everything versus having one copy in the external group and one copy on my laptop. I tried and got: ```bash -$ git annex --lackingcopies=external:1 +$ git annex find --lackingcopies=external:1 git-annex: bad value for number of lacking copies ```
diff --git a/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn b/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn new file mode 100644 index 0000000000..bd7533b176 --- /dev/null +++ b/doc/forum/Bad_value_for_lackingcopies_inside_a_group.mdwn @@ -0,0 +1,10 @@ +I have a group called external for my different external hard drives and numcopies set to 2. I was trying to make sure that the external group itself had two copies of everything versus having one copy in the external group and one copy on my laptop. I tried and got: + +```bash +$ git annex --lackingcopies=external:1 +git-annex: bad value for number of lacking copies +``` + +Am I missing something about the syntax? + +git annex version: 10.20251215-132
close
diff --git a/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config.mdwn b/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config.mdwn index e483b00c76..00fa30d55e 100644 --- a/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config.mdwn +++ b/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config.mdwn @@ -6,3 +6,5 @@ From my experimentation it seems to be that git-annex does not discover the `ann This automatic discovery would be nice for p2phttp support in forgejo-aneksajo, as existing clones could automatically start making use of it as soon as the instance is updated to support it on the server-side and the git-annex version is updated to be recent enough on the client-side. [[!tag projects/ICE4]] + +> [[done]] --[[Joey]]
re-probing of annex.url for annex+https on connection failure
Automate updating remote.name.annexUrl for annex+https servers, by
re-probing the http remote's annex.url config when unable to connect to an
annex+https server.
p2pHttpReprobe was able to reuse tryGitConfigRead. However, if the remote
git repo somehow lost its annex.uuid config at this point, it does not make
sense to set annex-ignore on it. So when tryGitConfigRead hasuuid is set,
made it avoid doing that.
Automate updating remote.name.annexUrl for annex+https servers, by
re-probing the http remote's annex.url config when unable to connect to an
annex+https server.
p2pHttpReprobe was able to reuse tryGitConfigRead. However, if the remote
git repo somehow lost its annex.uuid config at this point, it does not make
sense to set annex-ignore on it. So when tryGitConfigRead hasuuid is set,
made it avoid doing that.
diff --git a/CHANGELOG b/CHANGELOG
index 0f5fbde07d..41cdaa6786 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -19,6 +19,9 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
remote that is configured with exporttree=yes but without
importtree=yes.
* Fix mask special remote to not hang after the first file.
+ * Automate updating remote.name.annexUrl for annex+https servers,
+ by re-probing the http remote's annex.url config when unable to connect
+ to an annex+https server.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/P2P/Http/Client.hs b/P2P/Http/Client.hs
index 6e9dc7c797..0182c81d0e 100644
--- a/P2P/Http/Client.hs
+++ b/P2P/Http/Client.hs
@@ -65,48 +65,64 @@ type ClientAction a
p2pHttpClient
:: Remote
+ -> Annex Git.Repo
-> (String -> Annex a)
-> ClientAction a
-> Annex a
-p2pHttpClient rmt fallback clientaction =
- p2pHttpClientVersions (const True) rmt fallback clientaction >>= \case
+p2pHttpClient rmt reprobeurl fallback clientaction =
+ p2pHttpClientVersions (const True) rmt reprobeurl fallback clientaction >>= \case
Just res -> return res
Nothing -> fallback "git-annex HTTP API server is missing an endpoint"
p2pHttpClientVersions
:: (ProtocolVersion -> Bool)
-> Remote
+ -> Annex Git.Repo
-> (String -> Annex a)
-> ClientAction a
-> Annex (Maybe a)
-p2pHttpClientVersions allowedversion rmt fallback clientaction = do
+p2pHttpClientVersions allowedversion rmt reprobeurl fallback clientaction = do
rmtrepo <- getRepo rmt
- p2pHttpClientVersions' allowedversion rmt rmtrepo fallback clientaction
+ case p2purl of
+ Just p2purl' -> p2pHttpClientVersions' allowedversion p2purl' rmt rmtrepo fallbackreprobe clientaction
+ Nothing -> error "internal"
+ where
+ p2purl = remoteAnnexP2PHttpUrl (gitconfig rmt)
+ -- When unable to speak to the server, re-probe for its url, and
+ -- try again if it changed. This avoids the user needing to manually
+ -- update the remote.name.annexUrl config.
+ fallbackreprobe s = do
+ r <- reprobeurl
+ gc <- Annex.getRemoteGitConfig r
+ case remoteAnnexP2PHttpUrl gc of
+ Just p2purl' | Just p2purl' /= p2purl ->
+ p2pHttpClientVersions' allowedversion p2purl' rmt r fallback clientaction >>= \case
+ Just res -> return res
+ Nothing -> fallback s
+ _ -> fallback s
p2pHttpClientVersions'
:: (ProtocolVersion -> Bool)
+ -> P2PHttpUrl
-> Remote
-> Git.Repo
-> (String -> Annex a)
-> ClientAction a
-> Annex (Maybe a)
-p2pHttpClientVersions' allowedversion rmt rmtrepo fallback clientaction =
- case p2pHttpBaseUrl <$> remoteAnnexP2PHttpUrl (gitconfig rmt) of
- Nothing -> error "internal"
- Just baseurl -> do
- uo <- getUrlOptions (Just (gitconfig rmt))
- let clientenv = mkClientEnv (httpManager uo) baseurl
- let clientenv' = clientenv
- { makeClientRequest = \u r ->
- applyRequest uo
- <$> makeClientRequest clientenv u r
- }
- ccv <- Annex.getRead Annex.gitcredentialcache
- Git.CredentialCache cc <- liftIO $ atomically $
- readTMVar ccv
- case M.lookup (Git.CredentialBaseURL credentialbaseurl) cc of
- Nothing -> go clientenv' Nothing False Nothing versions
- Just cred -> go clientenv' (Just cred) True (credauth cred) versions
+p2pHttpClientVersions' allowedversion p2phttpurl rmt rmtrepo fallback clientaction = do
+ uo <- getUrlOptions (Just (gitconfig rmt))
+ let clientenv = mkClientEnv (httpManager uo) (p2pHttpBaseUrl p2phttpurl)
+ let clientenv' = clientenv
+ { makeClientRequest = \u r ->
+ applyRequest uo
+ <$> makeClientRequest clientenv u r
+ }
+ ccv <- Annex.getRead Annex.gitcredentialcache
+ Git.CredentialCache cc <- liftIO $ atomically $
+ readTMVar ccv
+ case M.lookup (Git.CredentialBaseURL credentialbaseurl) cc of
+ Nothing -> go clientenv' Nothing False Nothing versions
+ Just cred -> go clientenv' (Just cred) True (credauth cred) versions
where
versions = filter allowedversion allProtocolVersions
go clientenv mcred credcached mauth (v:vs) = do
@@ -151,13 +167,11 @@ p2pHttpClientVersions' allowedversion rmt rmtrepo fallback clientaction =
++ " " ++
decodeBS (statusMessage (responseStatusCode resp))
- credentialbaseurl = case remoteAnnexP2PHttpUrl (gitconfig rmt) of
- Just p2phttpurl
- | isP2PHttpSameHost p2phttpurl rmtrepo ->
- Git.repoLocation rmtrepo
- | otherwise ->
- p2pHttpUrlString p2phttpurl
- Nothing -> error "internal"
+ credentialbaseurl
+ | isP2PHttpSameHost p2phttpurl rmtrepo =
+ Git.repoLocation rmtrepo
+ | otherwise =
+ p2pHttpUrlString p2phttpurl
credauth cred = do
ba <- Git.credentialBasicAuth cred
@@ -239,16 +253,17 @@ clientRemoveWithProof
-> Key
-> Annex RemoveResultPlus
-> Remote
+ -> Annex Git.Repo
-> Annex RemoveResultPlus
-clientRemoveWithProof proof k unabletoremove remote =
+clientRemoveWithProof proof k unabletoremove remote reprobeurl =
case safeDropProofEndTime =<< proof of
Nothing -> removeanytime
Just endtime -> removebefore endtime
where
- removeanytime = p2pHttpClient remote giveup (clientRemove k)
+ removeanytime = p2pHttpClient remote reprobeurl giveup (clientRemove k)
removebefore endtime =
- p2pHttpClientVersions useversion remote giveup clientGetTimestamp >>= \case
+ p2pHttpClientVersions useversion remote reprobeurl giveup clientGetTimestamp >>= \case
Just (GetTimestampResult (Timestamp remotetime)) ->
removebefore' endtime remotetime
-- Peer is too old to support REMOVE-BEFORE.
@@ -256,7 +271,7 @@ clientRemoveWithProof proof k unabletoremove remote =
removebefore' endtime remotetime =
canRemoveBefore endtime remotetime (liftIO getPOSIXTime) >>= \case
- Just remoteendtime -> p2pHttpClient remote giveup $
+ Just remoteendtime -> p2pHttpClient remote reprobeurl giveup $
clientRemoveBefore k (Timestamp remoteendtime)
Nothing -> unabletoremove
diff --git a/P2P/Http/Url.hs b/P2P/Http/Url.hs
index b2a1e4c293..d035628763 100644
--- a/P2P/Http/Url.hs
+++ b/P2P/Http/Url.hs
@@ -30,7 +30,7 @@ data P2PHttpUrl = P2PHttpUrl
{ p2pHttpUrlString :: String
, p2pHttpBaseUrl :: BaseUrl
}
- deriving (Show)
+ deriving (Show, Eq)
parseP2PHttpUrl :: String -> Maybe P2PHttpUrl
parseP2PHttpUrl us
diff --git a/Remote/Git.hs b/Remote/Git.hs
index c6fdde638f..ba09987ccc 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -377,10 +377,12 @@ tryGitConfigRead gc autoinit r hasuuid
setremote (setConfig . annexUrlConfigKey) u
_ -> noop
return r'
- Left err -> do
- set_ignore "not usable by git-annex" False
- warning $ UnquotedString $ configurl (Git.repoLocationUserVisible r) ++ " " ++ err
- return r
+ Left err
+ | hasuuid -> return r
+ | otherwise -> do
+ set_ignore "not usable by git-annex" False
+ warning $ UnquotedString $ configurl (Git.repoLocationUserVisible r) ++ " " ++ err
+ return r
configlist_failed = set_ignore "does not have git-annex installed" True
@@ -474,7 +476,7 @@ inAnnex' repo rmt st@(State connpool duc _ _ _ _) key
| Git.repoIsUrl repo = checkremote
| otherwise = checklocal
where
- checkp2phttp = p2pHttpClient rmt giveup (clientCheckPresent key)
+ checkp2phttp = p2pHttpClient rmt (p2pHttpReprobe rmt) giveup (clientCheckPresent key)
checkhttp = do
gc <- Annex.getGitConfig
Url.withUrlOptionsPromptingCreds (Just (gitconfig rmt)) $ \uo ->
@@ -514,7 +516,7 @@ dropKey r st proof key = do
dropKey' :: Git.Repo -> Remote -> State -> Maybe SafeDropProof -> Key -> Annex ()
dropKey' repo r st@(State connpool duc _ _ _ _) proof key
| isP2PHttp r =
(Diff truncated)
comment
diff --git a/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config/comment_5_9b0e6f0cb6523252f249c247e267a4ae._comment b/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config/comment_5_9b0e6f0cb6523252f249c247e267a4ae._comment new file mode 100644 index 0000000000..e6ffd8c720 --- /dev/null +++ b/doc/todo/p2phttp__58___regularly_re-check_for_annex.url_config/comment_5_9b0e6f0cb6523252f249c247e267a4ae._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-09-10T16:41:00Z" + content=""" +> Would this include re-checking when remote.name.annexUrl is unset? + +If that would mean checking on every use of the remote, it seems too often. + +And even Debian stable now includes a new enough git-annex to support +p2phttp, so the benefits of that check are more limited now +to detecting server-side changes. + +> Given that the clone happened in the knowledge that "dumb http" was the +> only supported http protocol and read only, I am now questioning if such +> a automatic upgrade to p2phttp would really be needed, or even desirable + +I think that's a good point. + +So I'm inclined to make this todo only about re-checking when +remote.name.annexUrl is configured to a annex+http url +and it fails to connect. +"""]]
fix closing of this
diff --git a/doc/bugs/auth.mdwn b/doc/bugs/auth.mdwn index f49438eb73..45876d03fe 100644 --- a/doc/bugs/auth.mdwn +++ b/doc/bugs/auth.mdwn @@ -70,4 +70,4 @@ My expectation would be that a git-annex push would be able to trigger the oauth [[!tag projects/INM7]] -> [[fixed]] in forgejo-aneksajo --[[Joey]] +> [[fixed|done]] in forgejo-aneksajo --[[Joey]]
response
diff --git a/doc/forum/How_to_export_subfolders_and_their_contents/comment_1_bfbb622ab698ddca20a6cb339e0113a1._comment b/doc/forum/How_to_export_subfolders_and_their_contents/comment_1_bfbb622ab698ddca20a6cb339e0113a1._comment new file mode 100644 index 0000000000..d365ff1f09 --- /dev/null +++ b/doc/forum/How_to_export_subfolders_and_their_contents/comment_1_bfbb622ab698ddca20a6cb339e0113a1._comment @@ -0,0 +1,27 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-07T15:52:37Z" + content=""" +git-annex will filter the tree it exports to only the files that are +configured to be preferred content of the remote. So you could do this: + + git annex wanted android 'include=album/*' + git-annex export master --to android + +See [[git-annex-preferred-content]] for details about the syntax, but +basically you can extend that to as many directories as you want, separated by +"or"; it will include any files in any of those directories. + +---- + +> How do I create a tree that contains the subfolder itself too? + +Worth noting that is a general git question really, with several valid +answers that don't involve git-annex at all. (One easy one is to check out +a new branch, `git rm` everything except the subfolder, and `git commit`.) + +And once you have such a tree stored on a git branch, you can pass that +branch to `git-annex export`. So that's an option if you don't want to need +to deal with git-annex's own filtering features. +"""]]
Fix mask special remote to not hang after the first file
Refill TMVar.
Also, when there is a configuation problem, it would error out, leaving the
TMVar empty, so refill with Nothing on exception to avoid a hang in that
case.
Refill TMVar.
Also, when there is a configuation problem, it would error out, leaving the
TMVar empty, so refill with Nothing on exception to avoid a hang in that
case.
diff --git a/CHANGELOG b/CHANGELOG
index d912eb4d90..0f5fbde07d 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -18,6 +18,7 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
of an export conflict, or has been unsafely writing directly to a
remote that is configured with exporttree=yes but without
importtree=yes.
+ * Fix mask special remote to not hang after the first file.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Remote/Mask.hs b/Remote/Mask.hs
index b5fbbb4f1b..4f3cc85ccf 100644
--- a/Remote/Mask.hs
+++ b/Remote/Mask.hs
@@ -166,12 +166,17 @@ mkMaskedRemote :: RemoteConfig -> RemoteGitConfig -> UUID -> Annex MaskedRemote
mkMaskedRemote c gc u = do
v <- liftIO $ newTMVarIO Nothing
return $ MaskedRemote $
- liftIO (atomically (takeTMVar v)) >>= \case
- Just maskedremote -> return maskedremote
- Nothing -> do
- maskedremote <- findMaskedRemote c gc u
- liftIO $ atomically $ putTMVar v (Just maskedremote)
- return maskedremote
+ liftIO (atomically (takeTMVar v)) >>= \d ->
+ go v d `onException` restore v
+ where
+ go v (Just maskedremote) = do
+ liftIO $ atomically $ putTMVar v (Just maskedremote)
+ return maskedremote
+ go v Nothing = do
+ maskedremote <- findMaskedRemote c gc u
+ liftIO $ atomically $ putTMVar v (Just maskedremote)
+ return maskedremote
+ restore v = liftIO (atomically (void (tryPutTMVar v Nothing)))
findMaskedRemote :: RemoteConfig -> RemoteGitConfig -> UUID -> Annex Remote
findMaskedRemote c gc myuuid = case remoteAnnexMask gc of
diff --git a/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn
index 5ae37d82f1..91935ea5ca 100644
--- a/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn
+++ b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn
@@ -67,3 +67,5 @@ fsck f3.dat # stalls here
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Plenty - and I think I must have used mask special remote with more than one file before.
+
+> [[fixed|done]]
diff --git a/doc/bugs/copying_to_mask_stalls_after_first_file/comment_2_dadf7839444231569d44c83ee44c923e._comment b/doc/bugs/copying_to_mask_stalls_after_first_file/comment_2_dadf7839444231569d44c83ee44c923e._comment
new file mode 100644
index 0000000000..a8a0b9f903
--- /dev/null
+++ b/doc/bugs/copying_to_mask_stalls_after_first_file/comment_2_dadf7839444231569d44c83ee44c923e._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2026-09-07T15:17:19Z"
+ content="""
+This is a simple failure to refill a TMVar on use, so easy to fix.
+
+T the bug has always been there since the first release of Remote.Mask.
+I am as surprised as you that apparently none of us tried it with multiple
+files before.
+"""]]
comment
diff --git a/doc/bugs/copying_to_mask_stalls_after_first_file/comment_1_a038d136783aae130474f63b7931a6cb._comment b/doc/bugs/copying_to_mask_stalls_after_first_file/comment_1_a038d136783aae130474f63b7931a6cb._comment
new file mode 100644
index 0000000000..35e940016d
--- /dev/null
+++ b/doc/bugs/copying_to_mask_stalls_after_first_file/comment_1_a038d136783aae130474f63b7931a6cb._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2026-09-07T15:13:31Z"
+ content="""
+Reproduced. Note that `{1..4}` is a bashism or something and didn't work
+when I used it in a shell script, but `seq 1 4` worked.
+"""]]
cleanup
diff --git a/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn index 7f965678d4..d654ff5a28 100644 --- a/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn +++ b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn @@ -16,10 +16,7 @@ location and it's skipped being written if so, under the assumption that the export already was done in another clone of the repository. See [[!commit c3fa1f2b0869a87a9788ebbf881993c8093f3196]]. -Would it be possible to deal with that better? The export log should -indicate if the file is known to have been exported to the special remote, -and so it should be able to skip trying to re-export it in that case, -without needing the `checkPresentExport`. +Would it be possible to deal with that better? (FWIW: There is another, probably LLM generated bug report, about this which, as is usual for LLM generated bug reports, contains specious
comment
diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment new file mode 100644 index 0000000000..82e3a4cba4 --- /dev/null +++ b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_2_e6bca381097fc943704432e19109d871._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-09-07T15:05:20Z" + content=""" +See [[bugs/export_does_not_overwrite_existing_third_party_files]]. +"""]]
add export warning on checkPresentExport = True
export: Display a warning when the user is either in the middle of an
export conflict, or has been unsafely writing directly to a remote that is
configured with exporttree=yes but without importtree=yes.
export: Display a warning when the user is either in the middle of an
export conflict, or has been unsafely writing directly to a remote that is
configured with exporttree=yes but without importtree=yes.
diff --git a/CHANGELOG b/CHANGELOG
index c0f8535350..d912eb4d90 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -14,6 +14,10 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
* importfeed: Avoid displaying empty feed titles.
* webdav: Fix a hang when credentials are not available.
(Reversion introduced in 10.20260901)
+ * export: Display a warning when the user is either in the middle
+ of an export conflict, or has been unsafely writing directly to a
+ remote that is configured with exporttree=yes but without
+ importtree=yes.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/Command/Export.hs b/Command/Export.hs
index f2e06810aa..1ce24083e3 100644
--- a/Command/Export.hs
+++ b/Command/Export.hs
@@ -283,7 +283,10 @@ startExport r srcrs db cvar allfilledvar ti = do
stopUnless (notrecordedpresent ek) $
starting ("export " ++ name r) ai si $
ifM (either (const False) id <$> tryNonAsync (checkPresentExport (exportActions r) ek loc))
- ( next $ cleanupExport r db ek loc False
+ ( do
+ unlessM (isImportSupported r) $
+ warning presentwarning
+ next $ cleanupExport r db ek loc False
, do
liftIO $ modifyMVar_ cvar (pure . const (FileUploaded True))
performExport r srcrs db ek af (Git.LsTree.sha ti) loc allfilledvar
@@ -308,6 +311,16 @@ startExport r srcrs db cvar allfilledvar ti = do
then return False
else notElem (uuid r) <$> loggedLocations ek
)
+
+ presentwarning = UnquotedString $ unwords
+ [ "A file by this name is already present in the remote."
+ , "This is typically due to an export conflict, which will"
+ , "be resolved by this command once the git-annex branch"
+ , "is in sync across all writers. (Or it may be due to"
+ , "something other than git-annex writing to the remote."
+ , "Since the remote is not configured with importtree=yes,"
+ , "such modifications will not be preserved.)"
+ ]
performExport :: Remote -> [Remote] -> ExportHandle -> Key -> AssociatedFile -> Sha -> ExportLocation -> MVar AllFilled -> CommandPerform
performExport r srcrs db ek af contentsha loc allfilledvar = do
diff --git a/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn
index 878edbb448..7f965678d4 100644
--- a/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn
+++ b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn
@@ -26,3 +26,5 @@ which, as is usual for LLM generated bug reports, contains specious
pseudo-reasoning which it's not worth engaging with.
I am filing this bug report in anticipation of deleting that one.)
--[[Joey]]
+
+> [[done]] --[[Joey]]
diff --git a/doc/bugs/export_does_not_overwrite_existing_third_party_files/comment_1_f25701f967b519619e2273caf6fbeaec._comment b/doc/bugs/export_does_not_overwrite_existing_third_party_files/comment_1_f25701f967b519619e2273caf6fbeaec._comment
new file mode 100644
index 0000000000..2861ba49f8
--- /dev/null
+++ b/doc/bugs/export_does_not_overwrite_existing_third_party_files/comment_1_f25701f967b519619e2273caf6fbeaec._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2026-09-07T13:26:05Z"
+ content="""
+FWIW, I did add a note to the git-annex-export man page that git-annex
+expects to be the only thing writing to an exporttree=yes remote unless
+importtree=yes. So hopefully there will be less confused users. However,
+I'm doubtful that will be enough to avoid all confusion.
+
+So, I've also made it display a warning in this situation.
+"""]]
add "in your own words"
it's ridiculous I need to ask for this
it's ridiculous I need to ask for this
diff --git a/doc/templates/bugtemplate.mdwn b/doc/templates/bugtemplate.mdwn index f255df90e6..33b4a1a5a2 100644 --- a/doc/templates/bugtemplate.mdwn +++ b/doc/templates/bugtemplate.mdwn @@ -1,4 +1,4 @@ -### Please describe the problem. +### Please describe the problem in your own words. ### What steps will reproduce the problem?
add warning about other writers to exporttree=yes only special remotes
This may not be the best place for such a warning, but putting it in
every special remote's documentation of exportree=yes also doesn't feel
great.
This may not be the best place for such a warning, but putting it in
every special remote's documentation of exportree=yes also doesn't feel
great.
diff --git a/doc/git-annex-export.mdwn b/doc/git-annex-export.mdwn index c87051c07c..113fa5001a 100644 --- a/doc/git-annex-export.mdwn +++ b/doc/git-annex-export.mdwn @@ -31,6 +31,11 @@ being exported to it. (Note that things in the expression like Any files in the treeish that are stored on git will also be exported to the special remote. +Note that, unless the special remote is also configured with +`importtree=yes`, git-annex should be the only thing writing to it. +Files written to it in other ways can be overwritten by git-annex, +or can confuse git-annex. + Repeated exports are done efficiently, by diffing the old and new tree, and transferring only the changed files, and renaming files as necessary.
bug report
diff --git a/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn new file mode 100644 index 0000000000..878edbb448 --- /dev/null +++ b/doc/bugs/export_does_not_overwrite_existing_third_party_files.mdwn @@ -0,0 +1,28 @@ +If a directory special remote, which is configured with exporttree=yes +but without importtree=yes, has a file written to it not by git-annex, +a later git-annex export of a file with the same name but a different +content fails to overwrite it, but the output of the command indicates +the export succeeded. + +Of course, it's fine for the file to be overwritten. The user is not +supposed to be writing files to such a location themselves; if they +want such files preserved they need to use importtree=yes. +It would also be fine for the command to error out. +But it's surprising for it to silently fail to write to it. + +This is not limited to the directory special remote. It happens because +`checkPresentExport` is used to check if there is a file in the export +location and it's skipped being written if so, under the assumption +that the export already was done in another clone of the repository. +See [[!commit c3fa1f2b0869a87a9788ebbf881993c8093f3196]]. + +Would it be possible to deal with that better? The export log should +indicate if the file is known to have been exported to the special remote, +and so it should be able to skip trying to re-export it in that case, +without needing the `checkPresentExport`. + +(FWIW: There is another, probably LLM generated bug report, about this +which, as is usual for LLM generated bug reports, contains specious +pseudo-reasoning which it's not worth engaging with. +I am filing this bug report in anticipation of deleting that one.) +--[[Joey]]
no llm please
diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment new file mode 100644 index 0000000000..f628487e20 --- /dev/null +++ b/doc/bugs/export_deletes_preexisting_files_it_never_wrote/comment_1_2dfeb980ffe87dc6edf0b24a8558b765._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-05T14:41:07Z" + content=""" +This bug report appears to have been generated by an LLM. + +I do not appreciate and will not engage with LLM generated content, so if +you would like this bug to be looked at and fixed in a timely manner, +please use your own words. + +I will delete this bug as probably LLM generated if you do not do so within +the next few days. +"""]]
wording
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 8cc5ae7c8e..ee66360bc0 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -129,7 +129,7 @@ discussed at the top of this web page. See [this thread](https://lore.kernel.org/git/aooRdiVdjovWSFiG@fruit.crustytoothpaste.net/T/#md349b22d89eb933d22ec7a77dbc45ab562de5348). [First Assisted-By LLM code](https://github.com/git/git/commit/d7971544fe17378f44f49983010dbfc1834f7bef), -but the git developers think this and other similar patches were +but the git developers think this and other similar patches [mimicked other code in git](https://lore.kernel.org/git/aooRdiVdjovWSFiG@fruit.crustytoothpaste.net/T/#mf9ad560764da1645d1fa9a13cf68e4e2cd579d9a) is a way that makes them not be a potential license problem.
git situation is unclear/nuanced
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 5653e4b074..8cc5ae7c8e 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -120,8 +120,20 @@ git-annex.cabal that prevent an old version building it. ### git -Since 2.53 +Status is unclear. Possibly since 2.53 or 2.55. -[First LLM generated code](https://github.com/git/git/commit/d7971544fe17378f44f49983010dbfc1834f7bef) +git's developer documentation +[states](https://git-scm.com/docs/SubmittingPatches#ai) that they will "reject +anything that looks AI generated", with similar legal concerns as those +discussed at the top of this web page. See [this +thread](https://lore.kernel.org/git/aooRdiVdjovWSFiG@fruit.crustytoothpaste.net/T/#md349b22d89eb933d22ec7a77dbc45ab562de5348). + +[First Assisted-By LLM code](https://github.com/git/git/commit/d7971544fe17378f44f49983010dbfc1834f7bef), +but the git developers think this and other similar patches were +[mimicked other code in git](https://lore.kernel.org/git/aooRdiVdjovWSFiG@fruit.crustytoothpaste.net/T/#mf9ad560764da1645d1fa9a13cf68e4e2cd579d9a) +is a way that makes them not be a potential license problem. + +[First Co-Authored-By LLM code](https://github.com/git/git/commit/d9ee2ab501089e0ee22305ae99e13c7b730bc798) +which was merged into gitk upstream of git and so avoided git's policy. git-annex supports git back to 2.22.
fixed formatting of code snippets
diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn b/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn index dd821b8c98..9b072e0a92 100644 --- a/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn +++ b/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn @@ -10,11 +10,14 @@ So the behaviour is asymmetric in an unfortunate direction: too conservative to overwrite a file it does not own, but willing to delete that same file later. ### What steps will reproduce the problem? Set up a directory holding data that git-annex did not put there: + mkdir -p /tmp/target echo "PRECIOUS-PREEXISTING-DATA" > /tmp/target/a.txt echo "ALSO-PRECIOUS" > /tmp/target/keep.txt + Make an annex whose tree happens to contain a file of the same name, plus one new file: + mkdir /tmp/work && cd /tmp/work git init -q . git annex init -q work @@ -24,16 +27,20 @@ new file: git commit -qm tree git annex initremote t type=directory encryption=none \ directory=/tmp/target exporttree=yes + Export: + $ git annex export main --to t export t a.txt ok export t b.txt ok $ cat /tmp/target/a.txt PRECIOUS-PREEXISTING-DATA + Note `a.txt` was reported as exported, but its contents were (correctly) not overwritten. Now export a tree that no longer contains those files. The empty tree is used here for brevity; in practice this is just an ordinary change that drops a path. + $ git annex export $(git hash-object -t tree /dev/null) --to t unexport t b.txt ok unexport t a.txt ok @@ -41,6 +48,7 @@ here for brevity; in practice this is just an ordinary change that drops a path. cat: /tmp/target/a.txt: No such file or directory $ cat /tmp/target/keep.txt ALSO-PRECIOUS + `a.txt` is gone. Its contents were never exported by git-annex, and never existed anywhere in the annex — they are simply lost. `keep.txt` survives, which isolates the cause: it was never named in an exported
fix import concurrency waiting
Fix concurrent import of identical files to not fail
with "transfer already in progress".
This bug seems to have been present all the way back to the beginning in
commit e412129523e9b64975c936cfd15b4da0df1276fd.
The download action was run in a next block, which prevents the
waitstart/signaldone bracket from waiting on it.
Note that I've opted to keep d4633d61c881c68c61bab8eeb2eb1a23603c5582
as a second layer of guard against concurrency problems. If 2 imports
of a small file are run at the same time, that commit's use of locking
is still useful.
Sponsored-by: Dartmouth College's DANDI project
Fix concurrent import of identical files to not fail
with "transfer already in progress".
This bug seems to have been present all the way back to the beginning in
commit e412129523e9b64975c936cfd15b4da0df1276fd.
The download action was run in a next block, which prevents the
waitstart/signaldone bracket from waiting on it.
Note that I've opted to keep d4633d61c881c68c61bab8eeb2eb1a23603c5582
as a second layer of guard against concurrency problems. If 2 imports
of a small file are run at the same time, that commit's use of locking
is still useful.
Sponsored-by: Dartmouth College's DANDI project
diff --git a/Annex/Import.hs b/Annex/Import.hs
index 033b5a4a20..7267c9794a 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -787,10 +787,10 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
job <- liftIO $ newEmptyTMVarIO
let ai = ActionItemOther (Just (QuotedPath (fromImportLocation loc)))
let si = SeekInput []
- let importaction = starting ("import " ++ Remote.name remote) ai si $ do
+ let importaction = do
when oldversion $
showNote "old version"
- tryNonAsync (importordownload cidmap i largematcher) >>= \case
+ res <- tryNonAsync (importordownload cidmap i largematcher) >>= \case
Left e -> next $ do
warning (UnquotedString (show e))
liftIO $ atomically $
@@ -800,10 +800,12 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
liftIO $ atomically $
putTMVar job r
return (isJust r)
- commandAction $ bracket_
- (waitstart importing cid)
- (signaldone importing cid)
- importaction
+ return res
+ commandAction $ starting ("import " ++ Remote.name remote) ai si $
+ bracket_
+ (waitstart importing cid)
+ (signaldone importing cid)
+ importaction
return (Right job)
thirdpartypopulatedimport db (loc, (cid, sz)) =
diff --git a/CHANGELOG b/CHANGELOG
index 6d6104f850..2a463344fc 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -7,8 +7,9 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
was already documented to do.
* external: Support git-credential in TRANSFER-RETRIEVE-URL,
CHECKPRESENT-URL, and RETRIEVEIMPORT-URL
- * Fix import bug that could cause data corruption when importing with -J
- multiple small files that all have identical content.
+ * Fix concurrent import of identical files to not fail with
+ "transfer already in progress".
+ * Also fixes possible data corruption when importing identical small files.
* Support building with QuickCheck 2.17.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn
index df57f36b4b..a77bd347ac 100644
--- a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn
+++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn
@@ -63,3 +63,5 @@ git -C ephys-compression annex initremote s3-bucket type=S3 bucket=aind-benchmar
[[!meta author=yoh]]
[[!tag projects/dandi]]
+
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_3_6e56e113833b3aebd8a06bfb58aedef3._comment b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_3_6e56e113833b3aebd8a06bfb58aedef3._comment
new file mode 100644
index 0000000000..6abdd0dbd7
--- /dev/null
+++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_3_6e56e113833b3aebd8a06bfb58aedef3._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2026-09-04T16:37:51Z"
+ content="""
+This bug has been present as long as import has supported -J,
+although it had code that was supposed to prevent this problem.
+
+I've fixed it now.
+"""]]
import small files as a download transfer
Fix import bug that could cause data corruption when importing with -J
multiple small files that all have identical content.
I don't know if there actually was data corruption, but the
retrieveImport action could run concurrently on the same tmp file.
This does add a notifyTransfer for imports of small files, which is a
small behavior change but does not seem like a problem.
Sponsored-by: Dartmouth College's DANDI project
Fix import bug that could cause data corruption when importing with -J
multiple small files that all have identical content.
I don't know if there actually was data corruption, but the
retrieveImport action could run concurrently on the same tmp file.
This does add a notifyTransfer for imports of small files, which is a
small behavior change but does not seem like a problem.
Sponsored-by: Dartmouth College's DANDI project
diff --git a/Annex/Import.hs b/Annex/Import.hs
index 8b73859d1f..2a5da78214 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -853,7 +853,7 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
islargefile <- checkMatcher' matcher mi NoLiveUpdate mempty
metered Nothing sz bwlimit $ const $ if islargefile
then doimportlarge importkey cidmap loc cid sz f
- else doimportsmall cidmap loc cid sz
+ else doimportsmall cidmap loc cid sz f
doimportlarge importkey cidmap loc cid sz f p =
tryNonAsync importer >>= \case
@@ -901,25 +901,28 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
-- The file is small, so is added to git, so while importing
-- without content does not retrieve annexed files, it does
-- need to retrieve this file.
- doimportsmall cidmap loc cid sz p = do
- let downloader tmpfile = do
+ doimportsmall cidmap loc cid sz f p = do
+ let downloader p' tmpfile = do
(k, _) <- Remote.retrieveImport
(Remote.importActions remote)
loc [cid] tmpfile
(Right (mkkey tmpfile))
- p
+ (combineMeterUpdate p' p)
case keyGitSha k of
Just sha -> do
recordcidkey cidmap cid k
return sha
Nothing -> error "internal"
+ let af = AssociatedFile (Just f)
checkDiskSpaceToGet tmpkey Nothing Nothing $
- withTmp tmpkey $ \tmpfile ->
- tryNonAsync (downloader tmpfile) >>= \case
- Right sha -> return $ Just (loc, Left sha)
- Left e -> do
- warning (UnquotedString (show e))
- return Nothing
+ notifyTransfer Download af $
+ download' (Remote.uuid remote) tmpkey af Nothing stdRetry $ \p' ->
+ withTmp tmpkey $ \tmpfile ->
+ tryNonAsync (downloader p' tmpfile) >>= \case
+ Right sha -> return $ Just (loc, Left sha)
+ Left e -> do
+ warning (UnquotedString (show e))
+ return Nothing
where
tmpkey = tmpImportKey cid sz
mkkey tmpfile = gitShaKey <$> hashFile tmpfile
diff --git a/CHANGELOG b/CHANGELOG
index 12e2f2aaf7..6d6104f850 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -7,6 +7,8 @@ git-annex (10.20260902) UNRELEASED; urgency=medium
was already documented to do.
* external: Support git-credential in TRANSFER-RETRIEVE-URL,
CHECKPRESENT-URL, and RETRIEVEIMPORT-URL
+ * Fix import bug that could cause data corruption when importing with -J
+ multiple small files that all have identical content.
* Support building with QuickCheck 2.17.
-- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_2_cf4b4946be0c4f2544fafeda97d59be7._comment b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_2_cf4b4946be0c4f2544fafeda97d59be7._comment
new file mode 100644
index 0000000000..52e26b10bb
--- /dev/null
+++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_2_cf4b4946be0c4f2544fafeda97d59be7._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2026-09-04T15:17:15Z"
+ content="""
+Looks like there is a similar problem when importing small files,
+except in the `doimportsmall` code path it does not use `download`,
+so doesn't notice that it's downloading the same key twice. Which can leave
+2 threads both writing to the same tmpfile, with possibly bad results.
+
+I've fixed that, but only by making that case fail with
+the same "transfer already in progress".
+"""]]
reproduced
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_1_7b8a7438b7602fb048db8448c8d77f0e._comment b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_1_7b8a7438b7602fb048db8448c8d77f0e._comment new file mode 100644 index 0000000000..186ffeaf71 --- /dev/null +++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__/comment_1_7b8a7438b7602fb048db8448c8d77f0e._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-09-04T14:38:35Z" + content=""" +Reporoduced but only once I used -J3. + +This is due to the several `contact_vector.npy` files, which are all +identical. So they have the same etag and size, and importKey generates +the same temporary key for them before downloading. + +A similar problem could happen when the remote supports +`importKeyWithContentIdentifier` and that produces the same key for 2 +files. I expect a -J import can fail in that situation as well. +"""]]
fix
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment index cf7fe2b9b3..a8b485bdac 100644 --- a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment @@ -34,4 +34,4 @@ git-credential, and I see no reason for them not to, once the web special remote does. Implemented all of the above. -""]] +"""]]
comment
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment new file mode 100644 index 0000000000..cf7fe2b9b3 --- /dev/null +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_9_0f6c35e6a3bfc02dbb1bd8c45a35405a._comment @@ -0,0 +1,37 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-09-03T15:51:01Z" + content=""" +If the web special remote supports it, does it also make sense for any +other special remotes to support it? Not *all* of them, since that is +impractical. + +Since implementing this is as easy as swapping `withUrlOptions` to +`withUrlOptionsPromptingCreds`, I grepped for that. The special remotes +it may make sense for are bittorrent, httpalso, and external. + +bittorrent I think might as well, it's very much like the web special +remote. + +httpalso seems like a reasonable use case too + +external could for `DOWNLOAD-URL` -- which is already documented to +support git-credential but did not. Oops. Also `TRANSFER-RETRIEVE-URL`, +`CHECKPRESENT-URL`, and `RETRIEVEIMPORT_URL`, which I think should support it. + +But external special remotes with `remote.name.annex-externaltype=readonly` +should not; that is documented to be for cases where no authentication is needed +to download. + +One remote where it would not make sense is S3, which uses withUrlOptions +in only when "publicurl=" is configured. Which is not supposed to involve +password prompting, and if it did support git-credential, would be very +confusing since that is different from the usual S3 authentication. + +Also, `git-annex addurl` and `git-annex importfeed` could support +git-credential, and I see no reason for them not to, once the web special +remote does. + +Implemented all of the above. +""]]
Use git-credential in more situations
* Use git-credential in more situations when accessing urls that need a
password or other authentication, including the web, bittorrent and
httpalso special remotes, and git-annex addurl and importfeed.
* external: Fix DOWNLOAD-URL to support git-credential, which it
was already documented to do.
* external: Support git-credential in TRANSFER-RETRIEVE-URL,
CHECKPRESENT-URL, and RETRIEVEIMPORT-URL
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
* Use git-credential in more situations when accessing urls that need a
password or other authentication, including the web, bittorrent and
httpalso special remotes, and git-annex addurl and importfeed.
* external: Fix DOWNLOAD-URL to support git-credential, which it
was already documented to do.
* external: Support git-credential in TRANSFER-RETRIEVE-URL,
CHECKPRESENT-URL, and RETRIEVEIMPORT-URL
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
diff --git a/CHANGELOG b/CHANGELOG index 865eaef433..12e2f2aaf7 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,5 +1,12 @@ git-annex (10.20260902) UNRELEASED; urgency=medium + * Use git-credential in more situations when accessing urls that need a + password or other authentication, including the web, bittorrent and + httpalso special remotes, and git-annex addurl and importfeed. + * external: Fix DOWNLOAD-URL to support git-credential, which it + was already documented to do. + * external: Support git-credential in TRANSFER-RETRIEVE-URL, + CHECKPRESENT-URL, and RETRIEVEIMPORT-URL * Support building with QuickCheck 2.17. -- Joey Hess <id@joeyh.name> Wed, 02 Sep 2026 11:04:59 -0400 diff --git a/Command/AddUrl.hs b/Command/AddUrl.hs index 5c725df7a5..f56acc5fe9 100644 --- a/Command/AddUrl.hs +++ b/Command/AddUrl.hs @@ -251,7 +251,7 @@ startWeb addunlockedmatcher o si urlstring = go $ fromMaybe bad $ parseURIPortab go url = startingAddUrl si urlstring o $ if relaxedOption (downloadOptions o) then go' url Url.assumeUrlExists - else Url.withUrlOptions Nothing (Url.getUrlInfo urlstring) >>= \case + else Url.withUrlOptionsPromptingCreds Nothing (Url.getUrlInfo urlstring) >>= \case Right urlinfo -> go' url urlinfo Left err -> do warning (UnquotedString err) @@ -352,7 +352,7 @@ downloadWeb addunlockedmatcher o url urlinfo file = go =<< downloadWith' downloader urlkey webUUID url file where urlkey = addSizeUrlKey urlinfo $ Backend.URL.fromUrl url Nothing True - downloader f p = Url.withUrlOptions Nothing $ + downloader f p = Url.withUrlOptionsPromptingCreds Nothing $ downloadUrl False urlkey p Nothing [url] f go Nothing = return Nothing go (Just (tmp, backend)) = ifM (useYoutubeDl o <&&> liftIO (isHtmlFile tmp)) diff --git a/Command/ImportFeed.hs b/Command/ImportFeed.hs index b22ccc8685..e5a3ebf2b2 100644 --- a/Command/ImportFeed.hs +++ b/Command/ImportFeed.hs @@ -274,7 +274,7 @@ findDownloads u f feeddesc = catMaybes $ map mk (feedItems f) downloadFeed :: URLString -> OsPath -> Annex Bool downloadFeed url f | Url.parseURIRelaxed url == Nothing = giveup "invalid feed url" - | otherwise = Url.withUrlOptions Nothing $ + | otherwise = Url.withUrlOptionsPromptingCreds Nothing $ Url.download nullMeterUpdate Nothing url f startDownload :: AddUnlockedMatcher -> ImportFeedOptions -> Cache -> TMVar Bool -> ToDownload -> CommandStart @@ -373,7 +373,7 @@ downloadEnclosure addunlockedmatcher opts cache cv todownload url = let go urlinfo = Just . maybeToList <$> addUrlFile addunlockedmatcher dlopts url urlinfo f if relaxedOption (downloadOptions opts) then go Url.assumeUrlExists - else Url.withUrlOptions Nothing (Url.getUrlInfo url) >>= \case + else Url.withUrlOptionsPromptingCreds Nothing (Url.getUrlInfo url) >>= \case Right urlinfo -> go urlinfo Left err -> do warning (UnquotedString err) diff --git a/Remote/BitTorrent.hs b/Remote/BitTorrent.hs index 0923fc9205..6c1335f0eb 100644 --- a/Remote/BitTorrent.hs +++ b/Remote/BitTorrent.hs @@ -216,7 +216,7 @@ downloadTorrentFile gc u = do withTmpFileIn othertmp (literalOsPath "torrent") $ \f h -> do liftIO $ hClose h resetAnnexFilePerm f - ok <- Url.withUrlOptions (Just gc) $ + ok <- Url.withUrlOptionsPromptingCreds (Just gc) $ Url.download nullMeterUpdate Nothing u f when ok $ liftIO $ moveFile f torrent diff --git a/Remote/External.hs b/Remote/External.hs index 949fe6eab3..2197338591 100644 --- a/Remote/External.hs +++ b/Remote/External.hs @@ -100,9 +100,9 @@ gen rt externalprogram r u rc gc rs importUnsupported return $ Just $ specialRemote c readonlyStorer - (retrieveUrl gc) + (retrieveUrlReadOnly gc) readonlyRemoveKey - (checkKeyUrl gc) + (checkKeyUrlReadOnly gc) rmt | otherwise = do c <- parsedRemoteConfig remote rc @@ -335,7 +335,7 @@ retrieveKeyFileM external gc = fileRetriever $ \dest k p -> | k == k' -> result $ Left $ respErrorMessage "TRANSFER" errmsg TRANSFER_RETRIEVE_URL k' url - | k == k' -> getResult $ retrieveUrl' gc url dest k p + | k == k' -> getResult $ retrieveUrl gc url dest k p DELEGATE ps -> getResult $ do delegate <- getDelegateRemote external ps _ <- retrieveKeyFile delegate k @@ -373,7 +373,7 @@ checkPresentM external gc k = either giveup id <$> go | k' == k -> result $ Left $ respErrorMessage "CHECKPRESENT" errmsg CHECKPRESENT_URL k' url - | k == k' -> checkKeyUrl' gc k url + | k == k' -> checkKeyUrl gc k url DELEGATE ps -> Just $ do delegate <- getDelegateRemote external ps Result . Right <$> checkPresent delegate k @@ -432,7 +432,7 @@ retrieveExportM external gc k loc dest p = do TRANSFER_FAILURE Download k' errmsg | k == k' -> result $ Left $ respErrorMessage "TRANSFER" errmsg TRANSFER_RETRIEVE_URL k' url - | k == k' -> Just $ Result <$> retrieveUrl' gc url dest k p + | k == k' -> Just $ Result <$> retrieveUrl gc url dest k p DELEGATE ps -> getResult $ do delegate <- getDelegateRemote external ps _ <- retrieveExport (exportActions delegate) k loc dest p @@ -466,7 +466,7 @@ retrieveImportM external gc loc cids dest gk p = RETRIEVEIMPORT_FAILURE errmsg -> result $ Left $ respErrorMessage "RETRIEVEIMPORT" errmsg RETRIEVEIMPORT_URL url -> getResult $ do - retrieveUrl' gc url dest UnknownSize p >>= \case + retrieveUrl gc url dest UnknownSize p >>= \case Right () -> Right <$> either pure id gk Left msg -> pure (Left msg) DELEGATE ps -> getResult $ do @@ -522,7 +522,7 @@ checkPresentExportImport request srequest delegateaction handlereq external gc k | k' == k -> result $ Left $ respErrorMessage srequest errmsg CHECKPRESENT_URL k' url - | k == k' -> checkKeyUrl' gc k url + | k == k' -> checkKeyUrl gc k url DELEGATE ps -> Just $ do delegate <- getDelegateRemote external ps Result . Right <$> delegateaction delegate k loc @@ -864,7 +864,7 @@ handleRequest' st external req mp responsehandler liftIO $ atomically $ do l <- takeTMVar cleanupv putTMVar cleanupv (removeTmpFile tmpf:l) - res <- withUrlOptions (Just gc) $ + res <- withUrlOptionsPromptingCreds (Just gc) $ downloadUrl' False UnknownSize nullMeterUpdate Nothing [url] tmpf @@ -1206,15 +1206,15 @@ checkUrlM external url = where mkmulti (u, s, f) = (u, s, toOsPath f) -retrieveUrl :: RemoteGitConfig -> Retriever -retrieveUrl gc = fileRetriever' $ \f k p iv -> do +retrieveUrlReadOnly :: RemoteGitConfig -> Retriever +retrieveUrlReadOnly gc = fileRetriever' $ \f k p iv -> do us <- getWebUrls k unlessM (withUrlOptions (Just gc) $ downloadUrl True k p iv us f) $ giveup downloadFailed -retrieveUrl' :: MeterSize sizer => RemoteGitConfig -> URLString -> OsPath -> sizer -> MeterUpdate -> Annex (Either String ()) -retrieveUrl' gc url dest sizer p = - withUrlOptions (Just gc) $ \uo -> +retrieveUrl :: MeterSize sizer => RemoteGitConfig -> URLString -> OsPath -> sizer -> MeterUpdate -> Annex (Either String ()) +retrieveUrl gc url dest sizer p = + withUrlOptionsPromptingCreds (Just gc) $ \uo -> downloadUrl' False sizer p Nothing [url] dest uo >>= return . \case Left msg -> Left msg Right True -> Right () @@ -1223,14 +1223,14 @@ retrieveUrl' gc url dest sizer p = downloadFailed :: String downloadFailed = "failed to download content" -checkKeyUrl :: RemoteGitConfig -> CheckPresent -checkKeyUrl gc k = do +checkKeyUrlReadOnly :: RemoteGitConfig -> CheckPresent +checkKeyUrlReadOnly gc k = do us <- getWebUrls k anyM (\u -> withUrlOptions (Just gc) $ checkBoth u (fromKey keySize k)) us -checkKeyUrl' :: RemoteGitConfig -> Key -> URLString -> Maybe (Annex (ResponseHandlerResult (Either String Bool))) -checkKeyUrl' gc k url = - Just $ withUrlOptions (Just gc) $ \uo -> +checkKeyUrl :: RemoteGitConfig -> Key -> URLString -> Maybe (Annex (ResponseHandlerResult (Either String Bool))) +checkKeyUrl gc k url = + Just $ withUrlOptionsPromptingCreds (Just gc) $ \uo -> Result <$> checkBoth' url (fromKey keySize k) uo getWebUrls :: Key -> Annex [URLString] diff --git a/Remote/HttpAlso.hs b/Remote/HttpAlso.hs index 8725015e6d..fb68d2f6ec 100644 --- a/Remote/HttpAlso.hs +++ b/Remote/HttpAlso.hs @@ -134,7 +134,7 @@ retriveExportHttpAlso gc baseurl key loc dest p = do downloadAction :: RemoteGitConfig -> OsPath -> MeterUpdate -> Maybe IncrementalVerifier -> ((URLString -> Annex (Either String ())) -> Annex (Either String ())) -> Annex () downloadAction gc dest p iv run = - Url.withUrlOptions (Just gc) $ \uo -> + Url.withUrlOptionsPromptingCreds (Just gc) $ \uo -> run (\url -> Url.download' p iv url dest uo) >>= either giveup (const (return ())) @@ -144,7 +144,7 @@ checkKey gc baseurl ll key = (Diff truncated)
comment
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_8_ae629ba94860bb9bfb35d2d45e787f3b._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_8_ae629ba94860bb9bfb35d2d45e787f3b._comment new file mode 100644 index 0000000000..51975b6297 --- /dev/null +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_8_ae629ba94860bb9bfb35d2d45e787f3b._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 8""" + date="2026-09-03T15:23:08Z" + content=""" +I started writing docs for `remote.web.annex-gitcredentials`, +and quickly realized that it's confusing that, for a git remote, +`remote.<name>.annex-gitcredentials` is not needed in order for git-credential +to be used, while for a web special remote, it is needed. + +That makes me wonder if the config is needed at all, or if it should +default to true. Either choice would mean that the security considerations +need to be considered again. + +But: A git remote can also be set up to be autoenabled, with an attacker +controlled url. In that case, git-annex will already use the git credential +config when accessing files in that git remote, and so will git when +pulling from that remote. + +So, there does not seem to be any additional security exposure in making the +web special remote use git-credential by default. +"""]]
comment
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_7_01b136c085294412f7b38da1177b824a._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_7_01b136c085294412f7b38da1177b824a._comment new file mode 100644 index 0000000000..e74bdbc0eb --- /dev/null +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_7_01b136c085294412f7b38da1177b824a._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2026-09-03T14:47:57Z" + content=""" +I think if there is a config that is only used by the web special remote, +it needs to mention the web special remote in its name (or value) somehow. + +So rather than `credential.<url>.annex-ignore = false`, something like +`credential.<url>.annex-web-special-remote = true`. But that is a mouthful. + +Or `remote.web.annex-gitcredentials = true`. Which avoids git-annex needing +to handle `credential.` config matching at all, git-credential would use +whatever configs are set. + +That would make the web special remote use git credential for any urls that +need auth. Which may be too broad for some use case I suppose, but if the +user wants it to only be used for some specific urls, they can make a +separate web special remote with `urlinclude=` and set the git config for +that remote. +"""]]
remove old comments
These are at this point clutter on an important top-level page.
These are at this point clutter on an important top-level page.
diff --git a/doc/install/comment_10_484a35a4739e8168019668aaf474bae9._comment b/doc/install/comment_10_484a35a4739e8168019668aaf474bae9._comment deleted file mode 100644 index 0457f11cbb..0000000000 --- a/doc/install/comment_10_484a35a4739e8168019668aaf474bae9._comment +++ /dev/null @@ -1,28 +0,0 @@ -[[!comment format=mdwn - username="nobodyinperson" - avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5" - subject="Use an older version e.g. from archive.org" - date="2025-07-30T16:17:44Z" - content=""" -This happens sometimes and will eventually get fixed when joey notices it. Currently the binary is only on his laptop apparently. - -You can use an older version which is eventually available on archive.org: - -[[!format bash \"\"\" -yann in yann-desktop-nixos in …/OSX/current/10.15_Catalina on master took 2s123ms -🐟 ❯ git co d8a7d5d54d24d17810f07c0756e7334e998650fe -HEAD ist jetzt bei d8a7d5d54d publishing git-annex 10.20250630 10.20250606 -yann in yann-desktop-nixos in …/OSX/current/10.15_Catalina on HEAD (d8a7d5d) -🐟 ❯ git annex whereis -whereis git-annex.dmg (1 copy) - 5dc2ccd1-e534-4dae-8e8c-f31c8015e26e -- archive.org via S3 - - The following untrusted locations may also have copies: - 00000000-0000-0000-0000-000000000001 -- web - - web: http://archive.org/download/git-annex-builds/SHA256E-s28610967--7fc0dbf3f0a1f275a95730899327694b90dcd60c4ba8d8070a3efde44983a719.dmg -ok -\"\"\"]] - -So for example [this link](http://archive.org/download/git-annex-builds/SHA256E-s28610967--7fc0dbf3f0a1f275a95730899327694b90dcd60c4ba8d8070a3efde44983a719.dmg): -"""]] diff --git a/doc/install/comment_1_0aa16754fb08d8f2a54c8c3f78b6c187._comment b/doc/install/comment_1_0aa16754fb08d8f2a54c8c3f78b6c187._comment deleted file mode 100644 index 1bf53f02a9..0000000000 --- a/doc/install/comment_1_0aa16754fb08d8f2a54c8c3f78b6c187._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawm7eqCMh_B7mxE0tnchbr0JoYu11FUAFRY" - nickname="Stéphane" - subject="Old versions from distributions (e.g. Debian stable) fail with online instructions." - date="2014-06-28T15:36:12Z" - content=""" -Hello everyone. - -Be aware that your distribution's package may be very old. -For example, at the time I write this, latest Debian stable is Debian 7.5 which is 2 months old. -But git-annex package there is two *years* old (tomorrow, it will be exactly two yezrs old). - -So, beware. If following [online walkthrough](https://git-annex.branchable.com/walkthrough/), either install a more recent git-annex (e.g. from [Debain backports](http://backports.debian.org/Instructions/)) or follow instructions from your local `/usr/share/doc/git-annex/html/walkthrough.html` instead. -"""]] diff --git a/doc/install/comment_2_ba3985a5cbd9f5682807d2bdbb9874e2._comment b/doc/install/comment_2_ba3985a5cbd9f5682807d2bdbb9874e2._comment deleted file mode 100644 index 4db93ac341..0000000000 --- a/doc/install/comment_2_ba3985a5cbd9f5682807d2bdbb9874e2._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="For Debian/Ubuntu users -- get git-annex-standalone from NeuroDebian" - date="2017-07-12T17:57:21Z" - content=""" -We provide quite an up-to-date standalone backport build of git-annex (package name [git-annex-standalone](http://neuro.debian.net/pkgs/git-annex-standalone.html)) through NeuroDebian for all Debian/Ubuntus, so you might want to enable NeuroDebian repository (`apt-get install neurodebian` on a recent debian/ubuntu or follow [NeuroDebian website](http://neuro.debian.net) for instructions). -"""]] diff --git a/doc/install/comment_3_9a2118d6f967585cb21f9d9b372f4017._comment b/doc/install/comment_3_9a2118d6f967585cb21f9d9b372f4017._comment deleted file mode 100644 index 499b2c3999..0000000000 --- a/doc/install/comment_3_9a2118d6f967585cb21f9d9b372f4017._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Alan" - avatar="http://cdn.libravatar.org/avatar/9cbc26346f1c693d7df198e662a5fdae" - subject="No package for debian stretch, armh?" - date="2017-08-25T11:58:38Z" - content=""" -I cannot find a package for Debian Stretch for the armh architecture. I don't see git-annex in the official packages, and armh is not available in neurodebian. Is building from source the only option? -"""]] diff --git a/doc/install/comment_5_a2532a0fea59d15a2efa11748ff0d70a._comment b/doc/install/comment_5_a2532a0fea59d15a2efa11748ff0d70a._comment deleted file mode 100644 index 809c683172..0000000000 --- a/doc/install/comment_5_a2532a0fea59d15a2efa11748ff0d70a._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="dittigas" - avatar="http://cdn.libravatar.org/avatar/58cabd2b30471004288fac2b535faaa8" - subject="git-annex-webapp missing from Fedora 27" - date="2018-03-27T07:43:53Z" - content=""" -My installation does not seem to include the webapp feature. I.e git-annex webapp, git annex webapp or git-annex-webapp are not abailble. -"""]] diff --git a/doc/install/comment_6_4ae37152bc9765cea921b436fb2785d8._comment b/doc/install/comment_6_4ae37152bc9765cea921b436fb2785d8._comment deleted file mode 100644 index c06c13ee23..0000000000 --- a/doc/install/comment_6_4ae37152bc9765cea921b436fb2785d8._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""re: git-annex-webapp missing from Fedora 27""" - date="2018-04-04T15:59:51Z" - content=""" -@dittigas git-annex can be built without the webapp, -if the libraries it uses are not made available at build time. -I suggest you get in touch with the Fedora maintainers and ask them to -enable the webapp in their builds. -"""]] diff --git a/doc/install/comment_7_bc273d60cb74241231183186aefbc147._comment b/doc/install/comment_7_bc273d60cb74241231183186aefbc147._comment deleted file mode 100644 index 94e3ace489..0000000000 --- a/doc/install/comment_7_bc273d60cb74241231183186aefbc147._comment +++ /dev/null @@ -1,36 +0,0 @@ -[[!comment format=mdwn - username="seregynp@3214c4138198e0fe5615d11af832f69f8f5b6873" - nickname="seregynp" - avatar="http://cdn.libravatar.org/avatar/9070bf4684f1f7ed88564c6d75f29d59" - subject="Cannot upgrade to v6" - date="2018-09-15T10:20:58Z" - content=""" -I'm running Ubuntu: - - Distributor ID: Ubuntu │ - Description: Ubuntu 16.04.5 LTS │ - Release: 16.04 │ - Codename: xenial - -Installed git annex with: \"apt-get install git-annex\". - -And here is the my \"git-annex version\": - - git-annex version: 5.20151208-1build1 │ - build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus Des│ - ktopNotify XMPP DNS Feeds Quvi TDFA TorrentParser Database │ - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256│ - E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 S│ - KEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL │ - remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar│ - hook external - -When i'm trying to \"git annex init --version=6\" it says: \"Usage: git-annex init [DESC]\". -Meaning v5 git annex does not support \"--version\" option for \"init\" command. - -If i'm running \"git annex upgrade\", it says: \"upgrade . ok\". -But then \"git annex version\" shows v5 again. - -Is it possible to upgrade from v5 to v6 currently ? -Am i doing anything wrong ? -"""]] diff --git a/doc/install/comment_8_1330a5413b720eca0103a0880d24eb2d._comment b/doc/install/comment_8_1330a5413b720eca0103a0880d24eb2d._comment deleted file mode 100644 index 29b8c07a31..0000000000 --- a/doc/install/comment_8_1330a5413b720eca0103a0880d24eb2d._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2018-10-04T18:24:07Z" - content=""" -@seregynp, you need git-annex version 7 to use that. -"""]] diff --git a/doc/install/comment_8_768498659909c37044f2b4dd08bacda3._comment b/doc/install/comment_8_768498659909c37044f2b4dd08bacda3._comment deleted file mode 100644 index 73e75dee54..0000000000 --- a/doc/install/comment_8_768498659909c37044f2b4dd08bacda3._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="Git for Windows installation -- support for symlinks" - date="2019-08-08T15:04:07Z" - content=""" -The Git for Windows installer has a setting to turn on support for symlinks, which is not the default. It says something about symlinks requiring special permissions. What setting should be used with git-annex? -"""]] diff --git a/doc/install/comment_9_a17ccf32e2e450b2b744f11a1c5edc8c._comment b/doc/install/comment_9_a17ccf32e2e450b2b744f11a1c5edc8c._comment deleted file mode 100644 index e5bd2b3b47..0000000000 --- a/doc/install/comment_9_a17ccf32e2e450b2b744f11a1c5edc8c._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="h0b0" - avatar="http://cdn.libravatar.org/avatar/bf8483b4623b379c3443d63ecdff22a2" - subject="Cataline build missing" - date="2025-07-30T15:19:06Z" - content=""" -Somehow [the Catalina binary got lost](https://downloads.kitenet.net/git-annex/OSX/current/10.15_Catalina/). Considering that homebrew also fails due to the old OS this is a problem. I'd be happy if this was made available again. -"""]]
reorg
diff --git a/doc/install.mdwn b/doc/install.mdwn index e6e0d1101a..d6259e415e 100644 --- a/doc/install.mdwn +++ b/doc/install.mdwn @@ -28,6 +28,12 @@ detailed instructions | quick install [[PyPI]] | `uv tool install git-annex` """]] +## Building it yourself + +git-annex is [[Free Software|license]], written in [Haskell](http://www.haskell.org/). +Experienced users should not find it too hard to build and install +it [[from source|fromsource]]. + ## Historical builds Many historical builds are available from the @@ -36,12 +42,6 @@ git-annex repository.Visit [downloads.kitenet.net](https://downloads.kitenet.net/) for more information. -## Building it yourself - -git-annex is [[Free Software|license]], written in [Haskell](http://www.haskell.org/). -Experienced users should not find it too hard to build and install -it [[from source|fromsource]]. - ## See also * [[autobuild overview|builds]]
wording
diff --git a/doc/install.mdwn b/doc/install.mdwn index cbbe67c5d2..e6e0d1101a 100644 --- a/doc/install.mdwn +++ b/doc/install.mdwn @@ -30,7 +30,11 @@ detailed instructions | quick install ## Historical builds -Many of historical builds available from the [https://downloads.kitenet.net/.git/](https://downloads.kitenet.net/.git/) git-annex repository. Visit [downloads.kitenet.net](https://downloads.kitenet.net/) for more information. +Many historical builds are available from the +[https://downloads.kitenet.net/.git/](https://downloads.kitenet.net/.git/) +git-annex repository.Visit +[downloads.kitenet.net](https://downloads.kitenet.net/) for more +information. ## Building it yourself
added back manual install
diff --git a/doc/install/Android.mdwn b/doc/install/Android.mdwn index 55c77953b4..f8f09ce482 100644 --- a/doc/install/Android.mdwn +++ b/doc/install/Android.mdwn @@ -1,9 +1,12 @@ -The easiest way to install git-annex on Android is using the [[termux]] -package. +The easiest way to install git-annex on Android is using [[Termux]] +or [[Nix-On-Droid]]. It's also possible to install git-annex manually, using -the [[Linux_standalone]] build, run inside the Termux or Nix-On-Droid -app. [[Manual installation instructions here|/Android]]. +the [[Linux_standalone]] build, run inside Termux or a similar app: + + pkg install wget + wget https://git-annex.branchable.com/install/Android/git-annex-install + source git-annex-install The old git-annex Android app, is no longer bEing updated, details about it are at [[oldapp]].
fix name
diff --git a/doc/install/Nix-On_Droid.mdwn b/doc/install/Nix-On-Droid.mdwn similarity index 100% rename from doc/install/Nix-On_Droid.mdwn rename to doc/install/Nix-On-Droid.mdwn
git-annex in termux and Android instructions reorg and split
diff --git a/doc/Android.mdwn b/doc/Android.mdwn index 7117526c68..5f9d79f42a 100644 --- a/doc/Android.mdwn +++ b/doc/Android.mdwn @@ -12,66 +12,8 @@ on using git-annex that way.) ## Installation (Termux) -First, install [Termux](https://termux.com/). This is an Android app that can -run some Linux software in a terminal, including git-annex. - -git-annex is not currently part of the Termux distribution, but it's easy -to install it. Paste these commands into Termux: - - pkg install wget - wget https://git-annex.branchable.com/install/Android/git-annex-install - source git-annex-install +See [[/install/termux]] ## Installation (Nix-On-Droid) -Installing git-annex using -[Nix-On-Droid](https://github.com/t184256/nix-on-droid) is recommended -for more advanced users who want to use git-annex at the command line. -The git-annex webapp does not currently work in Nix-On-Droid (as a workaround, run `git annex webapp --listen 127.0.0.1` and copy-paste the URL in a browser). - -To enter a shell with git-annex available to use, -run inside Nix-On-Droid: `nix-shell -p git git-annex` - -To avoid needing to do that every time you start Nix-On-Droid, -you can add git and git-annex to your `environment.packages` in -`~/.config/nixpkgs/nix-on-droid.nix` and then run -`nix-on-droid switch` - -## Starting git-annex assistant - -Just run "git-annex webapp" inside Termux. -A browser window will open with the git-annex interface. - -[[!img webapp.png alt="git-annex webapp"]] - -## Closing and reopening the webapp - -The webapp does not need to be left open after you've set up your -repository. As long as Termux (or Nix-On-Droid) is left open, git-annex -will remain running and sync your files. - -## Starting at power on - -If you install the [Termux:Boot app](https://wiki.termux.com/wiki/Termux:Boot), -git-annex will be automatically started when your Android device -powers on. It will run in the background in whatever repositories you have -set up in the webapp. - -## Using the command line - -If you prefer to use `git-annex` at the command line, you can do so inside -Termux or Nix-On-Droid. Here we'll make a repository for photos: - - cd ~/storage/dcim - git init - git-annex init - -You can go on to set up a ssh remote pointing to a server, and sync -your files to and from it. - -And so on. Most ways you would use git-annex on a Linux system work fairly -well in the Termux environment. - -## Upgrading (Termux) - -To upgrade to a new git-annex release, just run `git-annex-install` again. +See [[/install/Nix-On-Droid]] diff --git a/doc/install.mdwn b/doc/install.mdwn index 23f440689d..cbbe67c5d2 100644 --- a/doc/install.mdwn +++ b/doc/install.mdwn @@ -21,8 +21,10 @@ detailed instructions | quick install [[OSX/Homebrew]] | `brew install git-annex` [[FreeBSD]] | `pkg install hs-git-annex` [[OpenBSD]] | `pkg_add git-annex` -[[Android]] | **beta** -[[Windows]] | **beta** +[[Android]] | + [[termux]] | `pkg install git-annex` + [[Nix-On-Droid]] | +[[Windows]] | [[PyPI]] | `uv tool install git-annex` """]] diff --git a/doc/install/Android.mdwn b/doc/install/Android.mdwn index af9f70fc9a..55c77953b4 100644 --- a/doc/install/Android.mdwn +++ b/doc/install/Android.mdwn @@ -1,7 +1,9 @@ -Now git-annex can be used on Android! +The easiest way to install git-annex on Android is using the [[termux]] +package. -[[Installation instructions here|/Android]]. +It's also possible to install git-annex manually, using +the [[Linux_standalone]] build, run inside the Termux or Nix-On-Droid +app. [[Manual installation instructions here|/Android]]. -The way it works now is the [[Linux_standalone]] builds of git-annex are run -inside the Termux app. The old git-annex Android app, -is no longer being updated, details about it are at [[oldapp]]. +The old git-annex Android app, +is no longer bEing updated, details about it are at [[oldapp]]. diff --git a/doc/install/Nix-On_Droid.mdwn b/doc/install/Nix-On_Droid.mdwn new file mode 100644 index 0000000000..c8ae45daca --- /dev/null +++ b/doc/install/Nix-On_Droid.mdwn @@ -0,0 +1,15 @@ +Installing git-annex using +[Nix-On-Droid](https://github.com/t184256/nix-on-droid) is recommended +for more advanced users who want to use git-annex at the command line. + +To enter a shell with git-annex available to use, +run inside Nix-On-Droid: `nix-shell -p git git-annex` + +To avoid needing to do that every time you start Nix-On-Droid, +you can add git and git-annex to your `environment.packages` in +`~/.config/nixpkgs/nix-on-droid.nix` and then run +`nix-on-droid switch` + +The git-annex webapp does not currently work in Nix-On-Droid (as a +workaround, run `git annex webapp --listen 127.0.0.1` and copy-paste the +URL in a browser). diff --git a/doc/install/termux.mdwn b/doc/install/termux.mdwn new file mode 100644 index 0000000000..8e30f0c48c --- /dev/null +++ b/doc/install/termux.mdwn @@ -0,0 +1,43 @@ +The [termux Android app](https://termux.dev/) includes git-annex in its +package repository, so installation is easy: + + pkg install git-annex + +## Starting git-annex assistant + +Just run "git-annex webapp" inside Termux. +A browser window will open with the git-annex interface. + +[[!img webapp.png alt="git-annex webapp"]] + +## Closing and reopening the webapp + +The webapp does not need to be left open after you've set up your +repository. As long as Termux is left open, git-annex +will remain running and sync your files. + +## Starting at power on + +If you install the [Termux:Boot app](https://wiki.termux.com/wiki/Termux:Boot), +git-annex will be automatically started when your Android device +powers on. It will run in the background in whatever repositories you have +set up in the webapp. + +## Using the command line + +If you prefer to use `git-annex` at the command line, you can do so inside +Termux. Here we'll make a repository for photos: + + cd ~/storage/dcim + git init + git-annex init + +You can go on to set up a ssh remote pointing to a server, and sync +your files to and from it. + +And so on. Most ways you would use git-annex on a Linux system work fairly +well in the Termux environment. + +## Upgrading + +To upgrade to a new git-annex release, just run `git-annex-install` again.
add news item for git-annex 10.20260901
diff --git a/doc/news/version_10.20260520.mdwn b/doc/news/version_10.20260520.mdwn deleted file mode 100644 index 3744b131a7..0000000000 --- a/doc/news/version_10.20260520.mdwn +++ /dev/null @@ -1,24 +0,0 @@ -git-annex 10.20260520 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * Behavior change: git-annex sync now defaults to syncing content, - for consistency with push and pull. However, to avoid surprising - behavior, this only affects repositories that have preferred content - configured. - (Use --no-content or configure annex.synccontent to avoid this.) - * Behavior change to git-annex pull and push's handling of unwanted - files. While previously both commands dropped unwanted files from - both the remote and the local repository, now pull only drops unwanted - files from the local repository, and push only drops unwanted files - from the remote. - * info: Report the total size of unused keys found by the last run - of git-annex unused. - * push: When pushing to a bare git repository, display git push - progress before the display of pushed branches. - * push, pull, assist: Fix behavior of --content to override the - annex.synccontent configuration. - * Send git-annex (or other configured) User-Agent when connecting to - annex+http remotes. - * Support GIT\_SSL\_CAINFO, GIT\_SSL\_CAPATH, http.sslCAPath, and http.sslCAPath - when connecting to https servers. - * Linux standalone builds now bundle CA certs. They are used only when - the system does not have its own CA cert store. - * Linux standalone build supports using Fedora's CA cert store location."""]] \ No newline at end of file diff --git a/doc/news/version_10.20260901.mdwn b/doc/news/version_10.20260901.mdwn new file mode 100644 index 0000000000..9ae064b54a --- /dev/null +++ b/doc/news/version_10.20260901.mdwn @@ -0,0 +1,35 @@ +git-annex 10.20260901 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * Behavior change: drop --auto --from a remote does not any longer + try to drop content that is not known to be present on the remote. + This avoids unncessary work and makes it consistent with the behavior + of git-annex sync and push. + * External special remote protocol extended to support IMPORTKEY. + * git-annex-remote-internetarchive supports --no-content imports. + * diffdriver: Avoid crashing when git passes an (undocumented) 8th + parameter. + * The preferred content groupwanted expression will no longer + consider a groupwanted expression of "" to be set, which allows + another group's groupwanted expression to be used instead. + * Fixed buggy handling of preferred content + "balanced=groupname:lackingcopies" + * Expand preferred content "lackingcopies" and "approxlackingcopies" + expression syntax to support "groupname:number" + * Expand preferred content "copies", "lackingcopies", and + "approxlackingcopies" expression syntax to support group limits which + can include/exclude multiple groups. Eg + "copies=archive+backup-offsite=3" + * Also expanded --lackingcopies, --approxlackingcopies, and --copies + with the same syntax. + * Expand preferred content "balanced", "fullybalanced", + "sizebalanced" and "fullysizebalanced" expression syntax to support + group limits as well. Eg + "balanced=backup:lackingcopies=archive-offsite" + * importfeed: Fix reporting and logging of problems with feeds. + * importfeed: When adding an url, indicate which feed it is from. + * Fix reversion in 8.20200226 that broke git-annex benchmark --databases + * Remove the ParallelBuild cabal flag and add cabal.project that + enables parallel build by default with ghc 9.8+ and cabal-install 3.12. + * NoLLMDependencies: Update for warp and magic. + * git-annex.cabal: Pin magic to 1.1 avoiding build failure on Windows + with newer version. + * stack.yaml: Update to lts-24.52"""]] \ No newline at end of file
diffdriver: Avoid crashing when git passes an (undocumented) 8th parameter
diff --git a/CHANGELOG b/CHANGELOG
index 95ef828865..cca6502e7e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,8 @@ git-annex (10.20260718) UNRELEASED; urgency=medium
* git-annex-remote-internetarchive supports --no-content imports.
* importfeed: Fix reporting and logging of problems with feeds.
* importfeed: When adding an url, indicate which feed it is from.
+ * diffdriver: Avoid crashing when git passes an (undocumented) 8th
+ parameter.
* Fixed buggy handling of preferred content
"balanced=groupname:lackingcopies"
* Expand preferred content "lackingcopies" and "approxlackingcopies"
diff --git a/Command/DiffDriver.hs b/Command/DiffDriver.hs
index bfcc917ec7..6baedfbfba 100644
--- a/Command/DiffDriver.hs
+++ b/Command/DiffDriver.hs
@@ -1,6 +1,6 @@
{- git-annex command
-
- - Copyright 2014-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2014-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -88,6 +88,10 @@ parseReq opts
, rNewHex = new_hex
, rNewMode = new_mode
}
+ -- git documents 7 parameters, but there can be an additional parameter
+ -- containing a similarity index description.
+ mk (path:old_file:old_hex:old_mode:new_file:new_hex:new_mode:_:[]) =
+ mk (path:old_file:old_hex:old_mode:new_file:new_hex:new_mode:[])
mk (unmergedpath:[]) = UnmergedReq { rPath = unmergedpath }
mk _ = badopts
diff --git a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn
index 2699904bd3..90bb8e8bb6 100644
--- a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn
+++ b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn
@@ -101,3 +101,4 @@ rename to moved-test.txt
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+> [[fixed|done]] --[[Joey]]
diff --git a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files/comment_1_c4963f1e8b025f55562917d32b0d6ab0._comment b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files/comment_1_c4963f1e8b025f55562917d32b0d6ab0._comment
new file mode 100644
index 0000000000..79b3b705ca
--- /dev/null
+++ b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files/comment_1_c4963f1e8b025f55562917d32b0d6ab0._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2026-08-31T17:22:09Z"
+ content="""
+git's not behaving as it's documented to; the diffdriver is
+documented to take 7 parameters the same as `GIT_EXTERNAL_DIFF` does,
+but here it's passing an 8th parameter with some textual description of the
+move.
+
+This does not seem to be new behavior, it goes back to 2010 or so.
+
+Anyway, it will be easy to make git-annex support this, since it can simply
+ignore the 8th parameter.
+"""]]
todo
diff --git a/doc/todo/add_excesscopies_to_preferred_content.mdwn b/doc/todo/add_excesscopies_to_preferred_content.mdwn new file mode 100644 index 0000000000..705092b908 --- /dev/null +++ b/doc/todo/add_excesscopies_to_preferred_content.mdwn @@ -0,0 +1,15 @@ +"not excesscopies=1" would match when the number of copies is not larger +than the configured numcopies. It complements "lackingcopies". + +I had an implementation in +[[!commit 26a7f4d89b9e73bf0a0989934e938968cbbbf522]], +but reverted it in [[!commit e962788bd46eddf8c5a4b2ae96d5fab7298f1cca]] +because it was buggy. + +I think that what's needed to implement this is for it to ignore the +AssumeNotPresent. But when I tried that, `git-annex drop --auto` +would drop, but then `git-annex get --auto` would get, so it wasn't stable. + +So, I think there needs to also be a AssumePresent, which gets populated +with the uuid of the repository that is getting a file. "excesscopies" +would look at AssumePresent. --[[Joey]]
gave up on this todo
diff --git a/doc/todo/should_balanced_lackingcopies_drop.mdwn b/doc/todo/should_balanced_lackingcopies_drop.mdwn index 74fe6098a0..b689944388 100644 --- a/doc/todo/should_balanced_lackingcopies_drop.mdwn +++ b/doc/todo/should_balanced_lackingcopies_drop.mdwn @@ -28,3 +28,5 @@ work the same as "fullybalanced=groupname:lackingcopies or (present and not lackingcopies=groupname:0)" --[[Joey]] + +> Gave up on this one, as not able to be implemented. [[done]] --[[Joey]] diff --git a/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment index 2f05c809cb..adc9866c5d 100644 --- a/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment +++ b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment @@ -39,4 +39,7 @@ Maybe better to leave this up to the user. They could use eg "balanced=pool:lackingcopies=backup and not excesscopies=pool+backup=1" if excesscopies were implemented. Sadly, I failed to implement it, see [[!commit e962788bd46eddf8c5a4b2ae96d5fab7298f1cca]] + +Anyway, with no way to implement excesscopies, I don't think balanced +lackingcopies can drop either. So this todo seems unable to be implemented. """]]
update
diff --git a/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment index d935e424c9..2f05c809cb 100644 --- a/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment +++ b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment @@ -37,4 +37,6 @@ But then, how to make it want to drop in the one case, and not in the other case Maybe better to leave this up to the user. They could use eg "balanced=pool:lackingcopies=backup and not excesscopies=pool+backup=1" +if excesscopies were implemented. Sadly, I failed to implement it, see +[[!commit e962788bd46eddf8c5a4b2ae96d5fab7298f1cca]] """]]
Revert "excesscopies and approxexcesscopies"
This reverts commit 26a7f4d89b9e73bf0a0989934e938968cbbbf522.
These were buggy unfortunately when dropping.
Eg, with numcopies=2, and "not excesscopies=1", and 3 copies, `git-annex
drop --auto` would not do anything, because it looks at whether the
preferred content expression would match *after* the drop, at which
point there will be 2 copies, so excesscopies=1 does not match, making
the whole preferred content expression match.
lackingcopies does not have the same problem because with eg
"lackingcopies=1", after the drop there is no lacking copy so the drop
can proceed.
Maybe lackingcopies needs to not take AssumeNotPresent into account?
But, I implemented that, and it made it not be stable; `git-annex get
--auto` would get a file, and then `git-annex drop --auto` would drop it.
I suppose this must be why this otherwise obvious thing to have in
preferred content wasn't in it. Because it can't be implemented.
This reverts commit 26a7f4d89b9e73bf0a0989934e938968cbbbf522.
These were buggy unfortunately when dropping.
Eg, with numcopies=2, and "not excesscopies=1", and 3 copies, `git-annex
drop --auto` would not do anything, because it looks at whether the
preferred content expression would match *after* the drop, at which
point there will be 2 copies, so excesscopies=1 does not match, making
the whole preferred content expression match.
lackingcopies does not have the same problem because with eg
"lackingcopies=1", after the drop there is no lacking copy so the drop
can proceed.
Maybe lackingcopies needs to not take AssumeNotPresent into account?
But, I implemented that, and it made it not be stable; `git-annex get
--auto` would get a file, and then `git-annex drop --auto` would drop it.
I suppose this must be why this otherwise obvious thing to have in
preferred content wasn't in it. Because it can't be implemented.

diff --git a/Annex/FileMatcher.hs b/Annex/FileMatcher.hs
index 2397e7b5c6..5e31404ed0 100644
--- a/Annex/FileMatcher.hs
+++ b/Annex/FileMatcher.hs
@@ -192,8 +192,6 @@ preferredContentTokens pcd =
, ValueToken "copies" (usev limitCopies)
, ValueToken "lackingcopies" (usev $ limitLackingCopies "lackingcopies" False)
, ValueToken "approxlackingcopies" (usev $ limitLackingCopies "approxlackingcopies" True)
- , ValueToken "excesscopies" (usev $ limitExcessCopies "excesscopies" False)
- , ValueToken "approxexcesscopies" (usev $ limitExcessCopies "approxexcesscopies" True)
, ValueToken "inbackend" (usev limitInBackend)
, ValueToken "metadata" (usev limitMetaData)
, ValueToken "url" (usev limitUrl)
diff --git a/CHANGELOG b/CHANGELOG
index 87a665bca3..95ef828865 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,4 +1,4 @@
-git-annex (10.20260731) UNRELEASED; urgency=medium
+git-annex (10.20260718) UNRELEASED; urgency=medium
* Behavior change: drop --auto --from a remote does not any longer
try to drop content that is not known to be present on the remote.
@@ -22,8 +22,6 @@ git-annex (10.20260731) UNRELEASED; urgency=medium
"sizebalanced" and "fullysizebalanced" expression syntax to support
group limits as well. Eg
"balanced=backup:lackingcopies=archive-offsite"
- * Add excesscopies and approxexcesscopies to preferred content expressions.
- * Added --excesscopies and --approxexcesscopies.
* The preferred content groupwanted expression will no longer
consider a groupwanted expression of "" to be set, which allows
another group's groupwanted expression to be used instead.
diff --git a/CmdLine/GitAnnex/Options.hs b/CmdLine/GitAnnex/Options.hs
index bc4334983a..5bc6f0074e 100644
--- a/CmdLine/GitAnnex/Options.hs
+++ b/CmdLine/GitAnnex/Options.hs
@@ -328,16 +328,6 @@ keyMatchingOptions' =
<> help "match files that need more copies (faster)"
<> hidden
)
- , annexOption (setAnnexState . Limit.addExcessCopies "excesscopies" False) $ strOption
- ( long "excesscopies" <> metavar paramNumber
- <> help "match files with more than numcopies"
- <> hidden
- )
- , annexOption (setAnnexState . Limit.addExcessCopies "approxexcesscopies" True) $ strOption
- ( long "approxexcesscopies" <> metavar paramNumber
- <> help "match files with more than numcopies (faster)"
- <> hidden
- )
, annexOption (setAnnexState . Limit.addInBackend) $ strOption
( long "inbackend" <> short 'B' <> metavar paramName
<> help "match files using a key-value backend"
diff --git a/Limit.hs b/Limit.hs
index 595362dc62..0061f74603 100644
--- a/Limit.hs
+++ b/Limit.hs
@@ -450,22 +450,8 @@ limitCopies want = case splitc ':' want of
addLackingCopies :: String -> Bool -> String -> Annex ()
addLackingCopies desc approx = addLimit . limitLackingCopies desc approx
-{- Adds a limit to match files that have more copies than needed. -}
-addExcessCopies :: String -> Bool -> String -> Annex ()
-addExcessCopies desc approx = addLimit . limitExcessCopies desc approx
-
limitLackingCopies :: String -> Bool -> MkLimit Annex
-limitLackingCopies = limitCopiesBy "lacking" vs
- where
- vs needed nhave numcopies = numcopies - nhave >= needed
-
-limitExcessCopies :: String -> Bool -> MkLimit Annex
-limitExcessCopies = limitCopiesBy "excess" vs
- where
- vs needed nhave numcopies = nhave >= numcopies + needed
-
-limitCopiesBy :: String -> (Int -> Int -> Int -> Bool) -> String -> Bool -> MkLimit Annex
-limitCopiesBy by vs desc approx want = case readish numwant of
+limitLackingCopies desc approx want = case readish numwant of
Just needed -> Right $ MatchFiles
{ matchAction = const $ \notpresent mi -> flip checkKey mi $
go mi needed notpresent
@@ -477,7 +463,7 @@ limitCopiesBy by vs desc approx want = case readish numwant of
, matchNegationUnstable = False
, matchDesc = matchDescSimple desc
}
- Nothing -> Left $ "bad value for number of " ++ by ++ " copies"
+ Nothing -> Left "bad value for number of lacking copies"
where
go mi needed notpresent key = case (groupwant, grouplimit) of
(Nothing, []) -> check (const True)
@@ -488,9 +474,8 @@ limitCopiesBy by vs desc approx want = case readish numwant of
m <- uuidsByGroup <$> groupMap
check (checkGroupLimit gl m)
where
- check uuidp = limitCheckNumCopies approx mi
- notpresent uuidp key
- (vs needed)
+ check uuidp = limitCheckNumCopies approx mi notpresent uuidp key vs
+ vs nhave numcopies' = numcopies' - nhave >= needed
(groupwant, grouplimit, numwant) = case splitc ':' want of
(g:n:[]) -> (Just (toGroup g), [], n)
_ -> case splitc '=' want of
diff --git a/doc/git-annex-matching-options.mdwn b/doc/git-annex-matching-options.mdwn
index abf50df09a..4e490a9f2e 100644
--- a/doc/git-annex-matching-options.mdwn
+++ b/doc/git-annex-matching-options.mdwn
@@ -169,39 +169,6 @@ in either of two repositories.
Like `--lackingcopies`, but does not look at .gitattributes annex.numcopies
settings. This makes it significantly faster.
-* `--excesscopies=number`
-
- Matches only when git-annex believes that there are the specified number
- or more of additional copies beyond the numcopies settings.
-
-* `--excesscopies=groupname:number`
-
- Providing the name of a group limits `--excesscopies` to only
- considering repositories in that group toward the numcopies count.
-
-* `--excesscopies=grouplimit=number`
-
- Providing a group limit makes `--excesscopies` only consider repositories
- in groups that match the group limit toward the numcopies count.
-
- In a group limit, use "+" before a group to include that group, and "-"
- before a group to exclude it. (The first group to be included does
- not need to be prefixed with a "+" though it may be.)
-
- A group limit matches repositories that are in any of the included
- groups and are in none of the excluded groups.
-
- For example, "--excesscopies=backup+archive-offsite=1" matches
- files that are stored in more backup or archive repositories than needed
- to satisfy numcopies, excluding offsite repositories. And
- "--excesscopies=-offsite=1" matches files that are stored in more
- repositories than needed, excluding offsite repositories.
-
-* `--approxexcesscopies=value`
-
- Like `--excesscopies`, but does not look at .gitattributes annex.numcopies
- settings. This makes it significantly faster.
-
* `--inbackend=name`
Matches only when content is stored using the specified key-value
diff --git a/doc/git-annex-preferred-content.mdwn b/doc/git-annex-preferred-content.mdwn
index 48aaab4cd3..89aed4f2c2 100644
--- a/doc/git-annex-preferred-content.mdwn
+++ b/doc/git-annex-preferred-content.mdwn
@@ -153,42 +153,6 @@ content not being configured.
Like lackingcopies, but does not look at .gitattributes annex.numcopies
settings. This makes it significantly faster.
-* `excesscopies=number`
-
- Matches only files that git-annex believes have the specified number or
- more of additional copies beyond their numcopies settings.
-
-* `excesscopies=groupname:number`
-
- Providing the name of a group limits `excesscopies` to only
- considering repositories in that group toward the numcopies count.
-
- For example, `excesscopies=backup:1` matches files that are stored
- in more backup repositories than needed to satisfy numcopies.
-
-* `excesscopies=grouplimit=number`
-
- Providing a group limit makes `excesscopies` only consider repositories
- in groups that match the group limit toward the numcopies count.
-
- In a group limit, use "+" before a group to include that group, and "-"
- before a group to exclude it. (The first group to be included does
- not need to be prefixed with a "+" though it may be.)
-
- A group limit matches repositories that are in any of the included
- groups and are in none of the excluded groups.
-
- For example, "excesscopies=backup+archive-offsite=1" matches
- files that are stored in more backup or archive repositories than needed
- to satisfy numcopies, excluding offsite repositories. And
- "excesscopies=-offsite=1" matches files that are stored in more
- repositories than needed, excluding offsite repositories.
-
-* `approxexcesscopies=value`
-
- Like excesscopies, but does not look at .gitattributes annex.numcopies
- settings. This makes it significantly faster.
-
* `inbackend=backendname`
Matches only files whose content is stored using the specified key-value
diff --git a/doc/preferred_content.mdwn b/doc/preferred_content.mdwn
index e8c6cdb227..293c2e9648 100644
--- a/doc/preferred_content.mdwn
+++ b/doc/preferred_content.mdwn
@@ -58,7 +58,6 @@ it assumes all files that are currently present are preferred content.
Here are changes to preferred content expressions, and the version
they were added in.
(Diff truncated)
comment
diff --git a/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment new file mode 100644 index 0000000000..d935e424c9 --- /dev/null +++ b/doc/todo/should_balanced_lackingcopies_drop/comment_4_2f15645b89b2dbbc47457e13f8e2d6b9._comment @@ -0,0 +1,40 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-08-31T15:44:32Z" + content=""" +Hmm, this behavior is not limited to balanced lackingcopies. +Eg "balanced=pool:2" will keep 3 copies if there are already 3, until +--rebalance is used. + +But in that case, the only way there can be 3 copies is if one is manually +made. And one reason to manually make an extra copy might be when moving a file +between two nodes of the pool, where it's copied to the new node first, +and then dropped from the second. It seems that the user would not want +a `git-annex sync` run in the middle of that process to undo their copy. + +And the documentation does mention that: + + Some of the ways that it can get out of balance include [...] + a file getting copied into more repositories in the + group than the specified number + +In the case of eg, "balanced=pool:lackingcopies=backup", a new backup +repository could be set up, and get a copy of a file (backup repositories +want all files). Then the file would have more copies in the pool than the +expression wants it to have. + +Is that different enough to justify a different behavior of dropping a copy +from the pool? At this point, before balanced lackingcopies is in a release, +it can be done without being a behavior change still. It would be harder +to justify it as a later behavior change. + +What about the similar situation with a manual move between nodes +with balanced lackingcopies? It seems the same reasoning applies to that as +to balanced copies. + +But then, how to make it want to drop in the one case, and not in the other case? + +Maybe better to leave this up to the user. They could use eg +"balanced=pool:lackingcopies=backup and not excesscopies=pool+backup=1" +"""]]
excesscopies and approxexcesscopies
* Add excesscopies and approxexcesscopies to preferred content expressions.
* Added --excesscopies and --approxexcesscopies.
* Add excesscopies and approxexcesscopies to preferred content expressions.
* Added --excesscopies and --approxexcesscopies.
diff --git a/Annex/FileMatcher.hs b/Annex/FileMatcher.hs
index 5e31404ed0..2397e7b5c6 100644
--- a/Annex/FileMatcher.hs
+++ b/Annex/FileMatcher.hs
@@ -192,6 +192,8 @@ preferredContentTokens pcd =
, ValueToken "copies" (usev limitCopies)
, ValueToken "lackingcopies" (usev $ limitLackingCopies "lackingcopies" False)
, ValueToken "approxlackingcopies" (usev $ limitLackingCopies "approxlackingcopies" True)
+ , ValueToken "excesscopies" (usev $ limitExcessCopies "excesscopies" False)
+ , ValueToken "approxexcesscopies" (usev $ limitExcessCopies "approxexcesscopies" True)
, ValueToken "inbackend" (usev limitInBackend)
, ValueToken "metadata" (usev limitMetaData)
, ValueToken "url" (usev limitUrl)
diff --git a/CHANGELOG b/CHANGELOG
index 95ef828865..87a665bca3 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,4 +1,4 @@
-git-annex (10.20260718) UNRELEASED; urgency=medium
+git-annex (10.20260731) UNRELEASED; urgency=medium
* Behavior change: drop --auto --from a remote does not any longer
try to drop content that is not known to be present on the remote.
@@ -22,6 +22,8 @@ git-annex (10.20260718) UNRELEASED; urgency=medium
"sizebalanced" and "fullysizebalanced" expression syntax to support
group limits as well. Eg
"balanced=backup:lackingcopies=archive-offsite"
+ * Add excesscopies and approxexcesscopies to preferred content expressions.
+ * Added --excesscopies and --approxexcesscopies.
* The preferred content groupwanted expression will no longer
consider a groupwanted expression of "" to be set, which allows
another group's groupwanted expression to be used instead.
diff --git a/CmdLine/GitAnnex/Options.hs b/CmdLine/GitAnnex/Options.hs
index 5bc6f0074e..bc4334983a 100644
--- a/CmdLine/GitAnnex/Options.hs
+++ b/CmdLine/GitAnnex/Options.hs
@@ -328,6 +328,16 @@ keyMatchingOptions' =
<> help "match files that need more copies (faster)"
<> hidden
)
+ , annexOption (setAnnexState . Limit.addExcessCopies "excesscopies" False) $ strOption
+ ( long "excesscopies" <> metavar paramNumber
+ <> help "match files with more than numcopies"
+ <> hidden
+ )
+ , annexOption (setAnnexState . Limit.addExcessCopies "approxexcesscopies" True) $ strOption
+ ( long "approxexcesscopies" <> metavar paramNumber
+ <> help "match files with more than numcopies (faster)"
+ <> hidden
+ )
, annexOption (setAnnexState . Limit.addInBackend) $ strOption
( long "inbackend" <> short 'B' <> metavar paramName
<> help "match files using a key-value backend"
diff --git a/Limit.hs b/Limit.hs
index 11a94a5ced..7c836343a6 100644
--- a/Limit.hs
+++ b/Limit.hs
@@ -450,8 +450,22 @@ limitCopies want = case splitc ':' want of
addLackingCopies :: String -> Bool -> String -> Annex ()
addLackingCopies desc approx = addLimit . limitLackingCopies desc approx
+{- Adds a limit to match files that have more copies than needed. -}
+addExcessCopies :: String -> Bool -> String -> Annex ()
+addExcessCopies desc approx = addLimit . limitExcessCopies desc approx
+
limitLackingCopies :: String -> Bool -> MkLimit Annex
-limitLackingCopies desc approx want = case readish numwant of
+limitLackingCopies = limitCopiesBy "lacking" vs
+ where
+ vs needed nhave numcopies = numcopies - nhave >= needed
+
+limitExcessCopies :: String -> Bool -> MkLimit Annex
+limitExcessCopies = limitCopiesBy "excess" vs
+ where
+ vs needed nhave numcopies = nhave >= numcopies + needed
+
+limitCopiesBy :: String -> (Int -> Int -> Int -> Bool) -> String -> Bool -> MkLimit Annex
+limitCopiesBy by vs desc approx want = case readish numwant of
Just needed -> Right $ MatchFiles
{ matchAction = const $ \notpresent mi -> flip checkKey mi $
go mi needed notpresent
@@ -463,7 +477,7 @@ limitLackingCopies desc approx want = case readish numwant of
, matchNegationUnstable = False
, matchDesc = matchDescSimple desc
}
- Nothing -> Left "bad value for number of lacking copies"
+ Nothing -> Left $ "bad value for number of " ++ by ++ " copies"
where
go mi needed notpresent key = case (groupwant, grouplimit) of
(Nothing, []) -> check (const True)
@@ -474,8 +488,9 @@ limitLackingCopies desc approx want = case readish numwant of
m <- uuidsByGroup <$> groupMap
check (checkGroupLimit gl m)
where
- check uuidp = limitCheckNumCopies approx mi notpresent uuidp key vs
- vs nhave numcopies' = numcopies' - nhave >= needed
+ check uuidp = limitCheckNumCopies approx mi
+ notpresent uuidp key
+ (vs needed)
(groupwant, grouplimit, numwant) = case splitc ':' want of
(g:n:[]) -> (Just (toGroup g), [], n)
_ -> case splitc '=' want of
diff --git a/doc/git-annex-matching-options.mdwn b/doc/git-annex-matching-options.mdwn
index 4e490a9f2e..abf50df09a 100644
--- a/doc/git-annex-matching-options.mdwn
+++ b/doc/git-annex-matching-options.mdwn
@@ -169,6 +169,39 @@ in either of two repositories.
Like `--lackingcopies`, but does not look at .gitattributes annex.numcopies
settings. This makes it significantly faster.
+* `--excesscopies=number`
+
+ Matches only when git-annex believes that there are the specified number
+ or more of additional copies beyond the numcopies settings.
+
+* `--excesscopies=groupname:number`
+
+ Providing the name of a group limits `--excesscopies` to only
+ considering repositories in that group toward the numcopies count.
+
+* `--excesscopies=grouplimit=number`
+
+ Providing a group limit makes `--excesscopies` only consider repositories
+ in groups that match the group limit toward the numcopies count.
+
+ In a group limit, use "+" before a group to include that group, and "-"
+ before a group to exclude it. (The first group to be included does
+ not need to be prefixed with a "+" though it may be.)
+
+ A group limit matches repositories that are in any of the included
+ groups and are in none of the excluded groups.
+
+ For example, "--excesscopies=backup+archive-offsite=1" matches
+ files that are stored in more backup or archive repositories than needed
+ to satisfy numcopies, excluding offsite repositories. And
+ "--excesscopies=-offsite=1" matches files that are stored in more
+ repositories than needed, excluding offsite repositories.
+
+* `--approxexcesscopies=value`
+
+ Like `--excesscopies`, but does not look at .gitattributes annex.numcopies
+ settings. This makes it significantly faster.
+
* `--inbackend=name`
Matches only when content is stored using the specified key-value
diff --git a/doc/git-annex-preferred-content.mdwn b/doc/git-annex-preferred-content.mdwn
index 89aed4f2c2..48aaab4cd3 100644
--- a/doc/git-annex-preferred-content.mdwn
+++ b/doc/git-annex-preferred-content.mdwn
@@ -153,6 +153,42 @@ content not being configured.
Like lackingcopies, but does not look at .gitattributes annex.numcopies
settings. This makes it significantly faster.
+* `excesscopies=number`
+
+ Matches only files that git-annex believes have the specified number or
+ more of additional copies beyond their numcopies settings.
+
+* `excesscopies=groupname:number`
+
+ Providing the name of a group limits `excesscopies` to only
+ considering repositories in that group toward the numcopies count.
+
+ For example, `excesscopies=backup:1` matches files that are stored
+ in more backup repositories than needed to satisfy numcopies.
+
+* `excesscopies=grouplimit=number`
+
+ Providing a group limit makes `excesscopies` only consider repositories
+ in groups that match the group limit toward the numcopies count.
+
+ In a group limit, use "+" before a group to include that group, and "-"
+ before a group to exclude it. (The first group to be included does
+ not need to be prefixed with a "+" though it may be.)
+
+ A group limit matches repositories that are in any of the included
+ groups and are in none of the excluded groups.
+
+ For example, "excesscopies=backup+archive-offsite=1" matches
+ files that are stored in more backup or archive repositories than needed
+ to satisfy numcopies, excluding offsite repositories. And
+ "excesscopies=-offsite=1" matches files that are stored in more
+ repositories than needed, excluding offsite repositories.
+
+* `approxexcesscopies=value`
+
+ Like excesscopies, but does not look at .gitattributes annex.numcopies
+ settings. This makes it significantly faster.
+
* `inbackend=backendname`
Matches only files whose content is stored using the specified key-value
diff --git a/doc/preferred_content.mdwn b/doc/preferred_content.mdwn
index 293c2e9648..e8c6cdb227 100644
--- a/doc/preferred_content.mdwn
+++ b/doc/preferred_content.mdwn
@@ -58,6 +58,7 @@ it assumes all files that are currently present are preferred content.
Here are changes to preferred content expressions, and the version
they were added in.
(Diff truncated)
diff --git a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn index e25f1779ce..2699904bd3 100644 --- a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn +++ b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn @@ -36,7 +36,7 @@ upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 local repository version: 10 ``` -from conda-forge. I'll retry with the latest version once the update hits the repositories... +from conda-forge. ~~I'll retry with the latest version once the update hits the repositories...~~ Same issue with `10.20260717-g0c917920c80ab1e8cc3d8f5886537708949e1659`. ### Please provide any additional information below.
diff --git a/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn new file mode 100644 index 0000000000..e25f1779ce --- /dev/null +++ b/doc/bugs/git-annex_diffdriver_fails_with_renamed_files.mdwn @@ -0,0 +1,103 @@ +### Please describe the problem. + +The git-annex diffdriver fails to create a diff for renamed annexed files. + + +### What steps will reproduce the problem? + +``` +mkdir repo +cd repo/ +git init +git annex init +echo '*.txt diff=annexedtext' >> .gitattributes +git annex add . +git commit -m 'commit' +git config diff.annexedtext.command 'git annex diffdriver --text' +echo test > test.txt +git annex add . +git commit -m 'commit' +git mv test.txt moved-test.txt +git commit -m 'commit' +git diff HEAD~1..HEAD +``` + +### What version of git-annex are you using? On what operating system? + +``` +git-annex version: 10.20260601-gd153453dde35bb4c90443996bf4af054ef04a6a9 +build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath +dependency versions: aws-0.24.4 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 10 +``` + +from conda-forge. I'll retry with the latest version once the update hits the repositories... + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +$ mkdir repo +$ cd repo/ +$ git init +Leeres Git-Repository in /home/icg149/Playground/repo/.git/ initialisiert +$ git annex init +init ok +(recording state in git...) +$ echo '*.txt diff=annexedtext' >> .gitattributes +$ git annex add . +add .gitattributes (dotfile; adding content to git repository) ok +(recording state in git...) +$ git commit -m 'commit' +[main (Root-Commit) ad61e2d] commit + 1 file changed, 1 insertion(+) + create mode 100644 .gitattributes +$ git config diff.annexedtext.command 'git annex diffdriver --text' +$ echo test > test.txt +$ git annex add . +add test.txt +ok +(recording state in git...) +$ git commit -m 'commit' +[main 5ae489e] commit + 1 file changed, 1 insertion(+) + create mode 120000 test.txt +$ git mv test.txt moved-test.txt +$ git commit -m 'commit' +[main 34045af] commit + 1 file changed, 0 insertions(+), 0 deletions(-) + rename test.txt => moved-test.txt (100%) +$ git diff HEAD~1..HEAD +git-annex: Unexpected input: test.txt /tmp/git-blob-bAAkbd/test.txt e7eabeaf9f3ed585eea5b4b687fb09b951899559 120000 /tmp/git-blob-zNUKeQ/moved-test.txt e7eabeaf9f3ed585eea5b4b687fb09b951899559 120000 moved-test.txt similarity index 100% +rename from test.txt +rename to moved-test.txt + + +Schwerwiegend: externes Diff-Programm unerwartet beendet, angehalten bei test.txt +[ble: exit 128] +$ git show +commit 34045afbd0015edd52d19bb7973be6c0cb56309b (HEAD -> main) +Author: Matthias Riße <m.risse@fz-juelich.de> +Date: Thu Aug 27 17:32:30 2026 +0200 + + commit + +diff --git a/test.txt b/moved-test.txt +similarity index 100% +rename from test.txt +rename to moved-test.txt + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +
magic LLM use now disclosed
though not the earlier use
though not the earlier use
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index ac4bf6f6f2..5653e4b074 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -103,9 +103,12 @@ and 10,000+ lines of changes. [magic](https://hackage.haskell.org/package/magic) since 1.1.1 -New maintainer has not disclosed their LLM use, but it is apparent, -including in their communications to [[Joey]] about incorrect -LLM-generated statements in the changelog. +[First disclosed LLM generated code](https://github.com/philippedev101/magic-haskell/commit/f52a3628626644d68e1cd1d2c36e98f78693bda5) + +The person who took over magic from its long-time maintainer has not +disclosed their earlier LLM use, but it is apparent, including in their +communications to [[Joey]] about incorrect LLM-generated statements in the +changelog. ### Cabal
add magic to NoLLMDependencies
I emailed its new maintainer asking them to disclose apparent LLM
generated text in the changelog. I also pointed out an incorrect
statement in that text.
Their reply was very clearly LLM generated, did not disclose LLM use,
contained additional incorrect and misleading statements, and had an
attached 667 LOC tarball of LLM slop.
While it's currently redundant to have magic pinned twice at the same
version, the pins are for 2 different reasons. If for some reason it
makes sense to support git-annex building with the new version, eg if
the bugs in it get fixed and someone wants to litter the code with
ifdefs to deal with the new API, the pinned version still needs to be
present in NoLLMDependencies.
I emailed its new maintainer asking them to disclose apparent LLM
generated text in the changelog. I also pointed out an incorrect
statement in that text.
Their reply was very clearly LLM generated, did not disclose LLM use,
contained additional incorrect and misleading statements, and had an
attached 667 LOC tarball of LLM slop.
While it's currently redundant to have magic pinned twice at the same
version, the pins are for 2 different reasons. If for some reason it
makes sense to support git-annex building with the new version, eg if
the bugs in it get fixed and someone wants to litter the code with
ifdefs to deal with the new API, the pinned version still needs to be
present in NoLLMDependencies.
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn
index 2ba3ab76ad..ac4bf6f6f2 100644
--- a/doc/no_llm_code.mdwn
+++ b/doc/no_llm_code.mdwn
@@ -99,6 +99,14 @@ and 10,000+ lines of changes.
(See [[todo/ditch_yesod]])
+### magic
+
+[magic](https://hackage.haskell.org/package/magic) since 1.1.1
+
+New maintainer has not disclosed their LLM use, but it is apparent,
+including in their communications to [[Joey]] about incorrect
+LLM-generated statements in the changelog.
+
### Cabal
[First LLM generated code](https://github.com/haskell/cabal/commit/da8b314563feb15a3df7bc1baeef4b7aa08f7578)
diff --git a/git-annex.cabal b/git-annex.cabal
index 13625b4ee0..4aaa74d39a 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -308,7 +308,8 @@ Executable git-annex
base (>= 4.18.2.1 && < 4.23),
ram (< 0.21.0),
persistent (>= 2.13.3) && (< 2.15.0.0),
- warp (< 3.4.11)
+ warp (< 3.4.11),
+ magic (<= 1.1)
else
Build-Depends:
base (>= 4.18.2.1 && < 5),
diff --git a/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn b/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn new file mode 100644 index 0000000000..dd821b8c98 --- /dev/null +++ b/doc/bugs/export_deletes_preexisting_files_it_never_wrote.mdwn @@ -0,0 +1,68 @@ +### Please describe the problem. +`git annex export` to a directory special remote can delete pre-existing files +that it never wrote. +When export encounters a path that already exists on the export remote, it does +not overwrite it — the existing file's contents are left untouched. But it still +prints `export <remote> <file> ok` and records the file as exported. That record +later authorises a deletion: once the exported tree stops listing that path, +export prints `unexport <remote> <file> ok` and removes the pre-existing file. +So the behaviour is asymmetric in an unfortunate direction: too conservative to +overwrite a file it does not own, but willing to delete that same file later. +### What steps will reproduce the problem? +Set up a directory holding data that git-annex did not put there: + mkdir -p /tmp/target + echo "PRECIOUS-PREEXISTING-DATA" > /tmp/target/a.txt + echo "ALSO-PRECIOUS" > /tmp/target/keep.txt +Make an annex whose tree happens to contain a file of the same name, plus one +new file: + mkdir /tmp/work && cd /tmp/work + git init -q . + git annex init -q work + echo "DIFFERENT-CONTENT-FROM-TREE" > a.txt + echo "new" > b.txt + git annex add a.txt b.txt + git commit -qm tree + git annex initremote t type=directory encryption=none \ + directory=/tmp/target exporttree=yes +Export: + $ git annex export main --to t + export t a.txt ok + export t b.txt ok + $ cat /tmp/target/a.txt + PRECIOUS-PREEXISTING-DATA +Note `a.txt` was reported as exported, but its contents were (correctly) not +overwritten. +Now export a tree that no longer contains those files. The empty tree is used +here for brevity; in practice this is just an ordinary change that drops a path. + $ git annex export $(git hash-object -t tree /dev/null) --to t + unexport t b.txt ok + unexport t a.txt ok + $ cat /tmp/target/a.txt + cat: /tmp/target/a.txt: No such file or directory + $ cat /tmp/target/keep.txt + ALSO-PRECIOUS +`a.txt` is gone. Its contents were never exported by git-annex, and never +existed anywhere in the annex — they are simply lost. +`keep.txt` survives, which isolates the cause: it was never named in an exported +tree, so no export record was created for it. Deletion follows the export +record, and the export record was created for a file that was never written. +### What version of git-annex are you using? On what operating system? +10.20251215 on Linux (Manjaro, x86_64). Also reproduced with a build of +10.20260718 from git. +### Please provide any additional information below. +The impact depends on what else lives in the export remote's directory. If it +holds a checked-out git repository, the deletion can remove that repository's +`.git/config`, `HEAD`, `refs/*`, `logs/*` and hooks in one pass — every one of +which export had previously declined to overwrite. In a test here that took a +working repository from 31 files to 4, after which `git log` in it reported +`fatal: not a git repository`. +A couple of related observations from the same testing, in case they are useful: +* Exporting to an *empty* directory writes every file in the tree as expected, + so the non-overwriting behaviour above is specific to paths that already + exist. +* Once export has written a file, later modifying that file on the remote and + re-running export does not restore it, with or without `--force`. +I have deliberately not proposed a fix, since the right behaviour is a design +question — whether export should refuse such a path, warn, overwrite it, or +simply not record a file it did not write, all have different consequences for +existing users.
diff --git a/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn
new file mode 100644
index 0000000000..5ae37d82f1
--- /dev/null
+++ b/doc/bugs/copying_to_mask_stalls_after_first_file.mdwn
@@ -0,0 +1,69 @@
+### Please describe the problem.
+
+There appears to be an issue with the mask remote with encryption: annex-copying several files stalls after first file.
+
+I first noticed this issue with forgejo-aneksajo and reported as <https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/issues/125> - only then did I realize that this also affects bare repositories. I added a comment to the forgejo-aneksajo issue and will amend it further, linking to this issue here.
+
+### What steps will reproduce the problem?
+
+This would create a repo with four small files and try to push it to an encrypted mask remote:
+
+```
+git init /tmp/foo
+git init --bare /tmp/local-bare-repo
+cd /tmp/foo
+for n in {1..4}; do head -c 1M < /dev/urandom > f$n.dat; done
+git annex init
+git annex add *dat
+git commit -m "Add 4x1M"
+git remote add local /tmp/local-bare-repo
+git annex sync --no-content local
+git annex sync --no-content local
+git annex info local | grep uuid
+git annex initremote local-mask type=mask remote=local encryption=hybrid keyid=<GPG KEY ID HERE>
+git annex copy --to local-mask
+```
+
+This stalls for me after the first file (ie. after printing copy f2.dat) and I interrupt with Ctrl+c.
+
+On one attempt I got "gpg: signal Interrupt caught ... exiting" after Ctrl+c but usually I got nothing.
+
+### What version of git-annex are you using? On what operating system?
+
+I tried this on two systems:
+
+- git-annex 10.20260717 (from PyPI), git 2.47.3, gpg (GnuPG) 2.4.7 on Debian GNU/Linux 13 (trixie)
+- git-annex 10.20250416, git 2.39.5, gpg (GnuPG) 2.2.40 on Debian GNU/Linux 12 (bookworm)
+
+
+### Please provide any additional information below.
+
+This seems to be specific for copying multiple files, because copying the files one-by-one works fine:
+
+```
+git annex copy f1.dat --to local-mask
+git annex copy f2.dat --to local-mask
+git annex copy f3.dat --to local-mask
+git annex copy f4.dat --to local-mask
+```
+
+This seems to be specific to the mask special remote (and not just a problem with gpg), because e.g. copy to directory remote works:
+
+```
+mkdir /tmp/safe
+git annex initremote safe type=directory directory=/tmp/safe encryption=hybrid keyid=<GPG KEY ID HERE>
+git annex copy --to safe
+```
+
+And it does not seem to be specific to copy, since (after pushing files one by one) fsck shows similar behavior (all at once stalls, one-by-one works).
+
+```
+❱ git annex fsck --fast --from local-mask
+fsck f1.dat ok
+fsck f2.dat ok
+fsck f3.dat # stalls here
+```
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Plenty - and I think I must have used mask special remote with more than one file before.
added project tag
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn index 88418d3872..df57f36b4b 100644 --- a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn +++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn @@ -59,3 +59,7 @@ git -C ephys-compression annex initremote s3-bucket type=S3 bucket=aind-benchmar 10.20260717-g698698a3c787a39d6ebe444d85b3eed81a60fb2d Debian GNU/Linux + + +[[!meta author=yoh]] +[[!tag projects/dandi]]
initial report on "transfer already in progress" and non-0 exit
diff --git a/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn new file mode 100644 index 0000000000..88418d3872 --- /dev/null +++ b/doc/bugs/import_runs_into___34__transfer_already_in_progress__34__.mdwn @@ -0,0 +1,61 @@ +### Please describe the problem. + +already the 2nd attempt since on first, with annex version from march this year, it did run + +``` +import s3-bucket aind-np1/625749_2022-08-03_15-15-06_ProbeA/provenance.json ok +import s3-bucket aind-np1/634568_2022-08-05_15-59-46_ProbeA/properties/contact_vector.npy + transfer already in progress, or unable to take transfer lock +failed +import s3-bucket aind-np1/634568_2022-08-05_15-59-46_ProbeA/properties/offset_to_uV.npy + transfer already in progress, or unable to take transfer lock +failed +import s3-bucket aind-np1/625749_2022-08-03_15-15-06_ProbeA/properties/channel_name.npy ok +``` + +and then after fetching all the rest 400GB without a hiccup, it did exit with non-0... i decided to make a fresh run with "bleeding edge" release 10.20260717-g698698a3c787a39d6ebe444d85b3eed81a60fb2d to similar result + +``` +(git)smaug:/mnt/datasets/datalad/crawl/aind-benchmark-data[master]git +$> datalad run -m "Initial update (import; 2nd attempt; first had 2 files failed)" duct code/update ephys-compression +[INFO ] == Command start (output follows) ===== +2026-08-18T17:44:04-0400 [INFO ] con-duct: python-dotenv not installed, skipping .env file loading +2026-08-18T17:44:04-0400 [INFO ] con-duct: duct 0.18.0 is executing 'code/update ephys-compression'... +2026-08-18T17:44:04-0400 [INFO ] con-duct: Log files will be written to .duct/logs/2026.08.18T17.44.04-2688207_ +I: annex version 10.20260717-g698698a3c787a39d6ebe444d85b3eed81a60fb2d +list s3-bucket ok +import s3-bucket aind-np1/625749_2022-08-03_15-15-06_ProbeA/binary.json ok +import s3-bucket aind-np1/625749_2022-08-03_15-15-06_ProbeA/properties/gain_to_uV.npy ok +import s3-bucket aind-np1/634568_2022-08-05_15-59-46_ProbeA/properties/channel_name.npy + transfer already in progress, or unable to take transfer lock +failed +import s3-bucket aind-np1/625749_2022-08-03_15-15-06_ProbeA/properties/channel_name.npy ok +... +``` + +which I just interrupted then. + +I have + +``` +$> git config --list | grep '^annex' +annex.retry=3 +annex.jobs=5 +annex.autoupgraderepository=false +annex.autoupgraderepository=false +annex.diskreserve=0M +``` + + +### What steps will reproduce the problem? + +s3 importree remote was initiated via + +``` +git -C ephys-compression annex initremote s3-bucket type=S3 bucket=aind-benchmark-data datacenter=US encryption=none fileprefix=ephys-compression/ host=s3.amazonaws.com importtree=yes publicurl=https://aind-benchmark-data.s3.amazonaws.com/ region=us-west-2 signature=anonymous versioning=no +``` + +### What version of git-annex are you using? On what operating system? + +10.20260717-g698698a3c787a39d6ebe444d85b3eed81a60fb2d Debian GNU/Linux +
diff --git a/doc/forum/createSymbolicLink_fails_with_already_exists.mdwn b/doc/forum/createSymbolicLink_fails_with_already_exists.mdwn new file mode 100644 index 0000000000..826c5ef7ff --- /dev/null +++ b/doc/forum/createSymbolicLink_fails_with_already_exists.mdwn @@ -0,0 +1,21 @@ +What happened here? It's not a small repo (around 200k files). + +``` +# git annex add . +add snapshot-20250204-2159/backups/USERNAME/HOSTNAME/home_USERNAME_20190121/.local/share/evolution/mail/local/.Sent.ibex.index.data +git-annex: createSymbolicLink '../../../../../../../../../../.git/annex/objects/23/Qj/SHA256E-s8--aedcc86ad84d1a969ae06583f122eab63c402af85b5435f1c77c9a7c02fb401e.data/SHA256E-s8--aedcc86ad84d1a969ae06583f122eab63c402af85b5435f1c77c9a7c02fb401e.data' to '.git/annex/othertmp/.0': already exists (File exists) +failed +add snapshot-20250204-2159/backups/USERNAME/HOSTNAME/home_USERNAME_20190121/.local/share/evolution/mail/local/.Templates.ibex.index +git-annex: createSymbolicLink '../../../../../../../../../../.git/annex/objects/Z5/6j/SHA256E-s7168--225731ef7798fb6626c969d0ea114bf4ae53f4801027ca8cdb21b1a9878b3e79/SHA256E-s7168--225731ef7798fb6626c969d0ea114bf4ae53f4801027ca8cdb21b1a9878b3e79' to '.git/annex/othertmp/.1': already exists (File exists) +failed +add snapshot-20250204-2159/backups/USERNAME/HOSTNAME/home_USERNAME/.local/share/evolution/mail/local/.Sent.ibex.index.data +git-annex: createSymbolicLink '../../../../../../../../../../.git/annex/objects/23/Qj/SHA256E-s8--aedcc86ad84d1a969ae06583f122eab63c402af85b5435f1c77c9a7c02fb401e.data/SHA256E-s8--aedcc86ad84d1a969ae06583f122eab63c402af85b5435f1c77c9a7c02fb401e.data' to '.git/annex/othertmp/.0': already exists (File exists) +failed +add snapshot-20250204-2159/backups/USERNAME/HOSTNAME/home_USERNAME/.local/share/evolution/mail/local/.Templates.ibex.index +git-annex: createSymbolicLink '../../../../../../../../../../.git/annex/objects/Z5/6j/SHA256E-s7168--225731ef7798fb6626c969d0ea114bf4ae53f4801027ca8cdb21b1a9878b3e79/SHA256E-s7168--225731ef7798fb6626c969d0ea114bf4ae53f4801027ca8cdb21b1a9878b3e79' to '.git/annex/othertmp/.0': already exists (File exists) +failed +add: 4 failed + +``` + +Btw: In the FormattingHelp in this Wiki the triple backticks for a fenced code block should be mentioned.
reorg
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_6_cd7056373fcabd688cc150304ebc72ee._comment similarity index 95% rename from doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment rename to doc/todo/web_remote__58___reuse_git_credential_authentication/comment_6_cd7056373fcabd688cc150304ebc72ee._comment index b9c9046be3..79103f2bd3 100644 --- a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_6_cd7056373fcabd688cc150304ebc72ee._comment @@ -1,6 +1,6 @@ [[!comment format=mdwn username="joey" - subject="""comment 5""" + subject="""comment 6""" date="2026-08-17T19:34:28Z" content=""" `credential.<url>.annex-ignore` seems like the right track,
comment
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment new file mode 100644 index 0000000000..b9c9046be3 --- /dev/null +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_cd7056373fcabd688cc150304ebc72ee._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-08-17T19:34:28Z" + content=""" +`credential.<url>.annex-ignore` seems like the right track, +and I suppose there could also be something like +`http.<url>.sslCAInfo.annex-ignore` etc too. Such configs would need to +be clearly documented as only being supported by the web special remote +and not other special remotes though. + +FWIW I don't want DELEGATE to supplant every other use of http in special +remotes. Any applicable web API library should be able to be used by a +special remote, when that's the lowest friction way to implement one. +"""]]
Added a comment: thoughts on git credential. scope
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_35d995e9bfb2e8e66e433eaeddedbd4c._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_35d995e9bfb2e8e66e433eaeddedbd4c._comment new file mode 100644 index 0000000000..72202aac7b --- /dev/null +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_5_35d995e9bfb2e8e66e433eaeddedbd4c._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="thoughts on git credential. scope" + date="2026-08-17T19:24:21Z" + content=""" +> @yoh what do you think about that idea? + +I might be missing the point for extra parameter (`urlinclude`) for web remote itself as it might complicate handling multiple `<url>` settings, and overall feels more of a user/system wide configuration and not remote. + +As for the need of support by all special remotes -- now that there is a [`DELEGATE`](https://git-annex.branchable.com/design/external_special_remote_protocol/delegate_appendix/) functionality for special remotes, I think special remotes should generally be advised to just delegate HTTP downloads to git-annex for robust/centralized operation overall where feasible. Then it would fall under centralized `credential.<url>.` handling by git-annex. + +Related: just an idea, what about adding `credential.<url>.annex-ignore` defaulting to `True`, just to let folks where needed explicitly set to `False` so to tell credential provider to be used for that `<url>`? Since manual configuration of a `credential.` is anyways required, IMHO it is ok to request explicitly specify if it should apply to git-annex or not, with default being \"not\". +"""]]
update
diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_2_407a6e789f41f998c9cc8318ed511d19._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_2_407a6e789f41f998c9cc8318ed511d19._comment index 425b064e97..a386059113 100644 --- a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_2_407a6e789f41f998c9cc8318ed511d19._comment +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_2_407a6e789f41f998c9cc8318ed511d19._comment @@ -15,4 +15,6 @@ There could be room for a middle ground, and it might be something like a separate web special remote that is configured with `urlinclude=https://datalad-test.local.lan` inheriting your git configs for that url. + +@yoh what do you think about that idea? """]] diff --git a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_4_c45e87af4b3bd5213db8f5f9cea9f2df._comment b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_4_c45e87af4b3bd5213db8f5f9cea9f2df._comment index 567212c0b1..10a4ea2415 100644 --- a/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_4_c45e87af4b3bd5213db8f5f9cea9f2df._comment +++ b/doc/todo/web_remote__58___reuse_git_credential_authentication/comment_4_c45e87af4b3bd5213db8f5f9cea9f2df._comment @@ -17,4 +17,9 @@ the consequences. But, if I had considered this problem in 2015, I may have thought twice about implementing autoenable. (FWIW, git started supporting url wildcards in configs in 2013.) + +Anyway, if that is a security hole then autoenabling is at the root of it, +and it would need to be fixed by changing the default autoenable behavior. +Eg, git-annex could default to not allow autoenable, and the config to +allow it could discuss using config wildcards combined with autoenable. """]]