Recent changes to this wiki:
todo
diff --git a/doc/todo/optimize_drop_with_present_in_preferred_content_expression.mdwn b/doc/todo/optimize_drop_with_present_in_preferred_content_expression.mdwn new file mode 100644 index 0000000000..c9b7a872c3 --- /dev/null +++ b/doc/todo/optimize_drop_with_present_in_preferred_content_expression.mdwn @@ -0,0 +1,33 @@ +`git-annex drop --auto --from` a remote when the remote's preferred content +expression is "present" and when the key is not present will do +work to drop the key, unless --fast is used. + +In the vast majority of cases, that is unncessary work. Only if the +location tracking is out of date and the remote actually has the content +does the content get dropped. + +(As well as being unncessary work, it looks pretty weird. I thought +for a while that my preferred content expression was broken or that +git-annex was dropping things it shouldn't.) + +Generally drop does not default to --fast because the goal is that +dropping a specific file from a remote really guarantees that the remote +does not contain the file. Even if the local repository is not in sync. + +In this case though, the drop is not dropping a specific file, it's +working to satisfy the preferred content. If the local repository is out +of sync there are surely other ways that a preferred content expression +won't be fully satisfied. For example, if a repository only wants content +when no other repository in its group contains it, the local repository +being out of sync can make a drop not happen when it "should" +(or happen when it "shouldn't"). + +It might be that the solution to this is to make `drop --auto` default +to `--fast`. Or a more targeted solution that checks if the preferred +content depends on location tracking might be better. Unsure. I'd lean +toward the simpler one. + +Note that `git-annex sync` only drops from remotes that location +tracking thinks contain the content. So this would bring `git-annex drop +--auto` into alignment with it. +--[[Joey]]
todo
diff --git a/doc/todo/balanced_lackingcopies_excluding_a_group.mdwn b/doc/todo/balanced_lackingcopies_excluding_a_group.mdwn new file mode 100644 index 0000000000..261ae25d72 --- /dev/null +++ b/doc/todo/balanced_lackingcopies_excluding_a_group.mdwn @@ -0,0 +1,13 @@ +Similar to [[todo/preferred_content_lackingcopies_excluding_a_group]], +"balanced=lackingcopies" could have a way to exclude a group from being +treated as a copy. + +Eg, if there are client repositories and backup repositories, the goal +might be to make the backup repositories balance enough copies to satisfy +numcopies on their own, not counting client repositories. +"balanced=backup:lackingcopies" will not do that. + +Maybe "balanced=backup:lackingcopies-client" and +"balanced=backup:lackingcopies+backup". The former counts copies that +are not on clients, while the latter only counts copies that are on +backups. --[[Joey]]
fix groupwanted ""
The preferred content groupwanted expression will no longer consider a
groupwanted expression of "" to be set, which allows another group's
groupwanted expression to be used instead.
See commit 7d8558548b371c669c958b72d216b47893a3ba78. I was not able to use
isEmpty here though, so groupwanted " " will still be treated as a value.
The preferred content groupwanted expression will no longer consider a
groupwanted expression of "" to be set, which allows another group's
groupwanted expression to be used instead.
See commit 7d8558548b371c669c958b72d216b47893a3ba78. I was not able to use
isEmpty here though, so groupwanted " " will still be treated as a value.
diff --git a/Annex/FileMatcher.hs b/Annex/FileMatcher.hs
index 385e23a16e..5e31404ed0 100644
--- a/Annex/FileMatcher.hs
+++ b/Annex/FileMatcher.hs
@@ -342,7 +342,7 @@ makeMatcher groupmap configmap groupwantedmap u matcherf mktokens unknownmatcher
(groupwanted mygroups)
| otherwise = unknownmatcher
mygroups = fromMaybe S.empty (u `M.lookup` groupsByUUID groupmap)
- groupwanted s = case M.elems $ M.filterWithKey (\k _ -> S.member k s) groupwantedmap of
+ groupwanted s = case filter (not . null) $ M.elems $ M.filterWithKey (\k _ -> S.member k s) groupwantedmap of
[pc] -> Just pc
_ -> Nothing
diff --git a/CHANGELOG b/CHANGELOG
index 9b3f4615c8..64aaadeb31 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -16,6 +16,9 @@ git-annex (10.20260718) UNRELEASED; urgency=medium
"copies=archive+backup-offsite=3"
* Also expanded --lackingcopies, --approxlackingcopies, and --copies
with the same syntax.
+ * The preferred content groupwanted expression will no longer
+ consider a groupwanted expression of "" to be set, which allows
+ another group's groupwanted expression to be used instead.
-- Joey Hess <id@joeyh.name> Fri, 24 Jul 2026 10:07:07 -0400
diff --git a/doc/git-annex-groupwanted.mdwn b/doc/git-annex-groupwanted.mdwn
index 249f82fb11..c94ca63b26 100644
--- a/doc/git-annex-groupwanted.mdwn
+++ b/doc/git-annex-groupwanted.mdwn
@@ -14,7 +14,7 @@ content expression set to "groupwanted".
For example, to configure a group named redundantarchive, and
make repositories in the group want to contain 3 copies of every file:
-
+
git annex groupwanted redundantarchive "not (copies=redundantarchive:3)"
for repo in foo bar baz; do
git annex group $repo redundantarchive
@@ -25,6 +25,11 @@ Note that there must be exactly one groupwanted expression configured
amoung all the groups that a repository is in; if there's more than one,
none of them will be used.
+To clear a groupwanted expression, use an empty value for the expression,
+eg:
+
+ git-annex groupwanted foo ""
+
# OPTIONS
* The [[git-annex-common-options]](1) can be used.
grouplimit syntax
* Expand preferred content "copies", "lackingcopies", and
"approxlackingcopies" expression syntax to support group limits which
can include/exclude multiple groups.
* Also, expand preferred content "lackingcopies" and "approxlackingcopies"
expression syntax to support "groupname:number". Just for consistency
with "copies".
* Also expanded --lackingcopies, --approxlackingcopies, and --copies
with the same syntax.
Using "grouplimit=number" rather than "grouplimit:number" makes old
versions of git-annex treat this as an unparseable number, so when
encountering such a preferred content expression they fall back to
"present" which is nicer behavior than not matching.
I punted a bit on the grouplimit syntax, perhaps there are combinations
of groups someone would want to match that cannot be represented. Putting a
fully general Utility.Matcher style syntax inside a value that is itself
inside such a syntax would need some form of quoting or bracketing.
And this feels too niche for such a complication.
Could always be added later I suppose.
Sponsored-by: Walltime
* Expand preferred content "copies", "lackingcopies", and
"approxlackingcopies" expression syntax to support group limits which
can include/exclude multiple groups.
* Also, expand preferred content "lackingcopies" and "approxlackingcopies"
expression syntax to support "groupname:number". Just for consistency
with "copies".
* Also expanded --lackingcopies, --approxlackingcopies, and --copies
with the same syntax.
Using "grouplimit=number" rather than "grouplimit:number" makes old
versions of git-annex treat this as an unparseable number, so when
encountering such a preferred content expression they fall back to
"present" which is nicer behavior than not matching.
I punted a bit on the grouplimit syntax, perhaps there are combinations
of groups someone would want to match that cannot be represented. Putting a
fully general Utility.Matcher style syntax inside a value that is itself
inside such a syntax would need some form of quoting or bracketing.
And this feels too niche for such a complication.
Could always be added later I suppose.
Sponsored-by: Walltime
diff --git a/CHANGELOG b/CHANGELOG
index 18ef9dd263..9b3f4615c8 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -8,6 +8,14 @@ git-annex (10.20260718) UNRELEASED; urgency=medium
* Remove the ParallelBuild cabal flag and add cabal.project that
enables parallel build by default with ghc 9.8+ and cabal-install 3.12.
* stack.yaml: Update to lts-24.52
+ * Expand preferred content "lackingcopies" and "approxlackingcopies"
+ expression syntax to support "groupname:number"
+ * Expand preferred content "copies", "lackingcopies", and
+ "approxlackingcopies" expression syntax to support group limits which
+ can include/exclude multiple groups. Eg
+ "copies=archive+backup-offsite=3"
+ * Also expanded --lackingcopies, --approxlackingcopies, and --copies
+ with the same syntax.
-- Joey Hess <id@joeyh.name> Fri, 24 Jul 2026 10:07:07 -0400
diff --git a/Limit.hs b/Limit.hs
index 28ab1be653..82cdddf5ba 100644
--- a/Limit.hs
+++ b/Limit.hs
@@ -1,6 +1,6 @@
{- user-specified limits on files to act on
-
- - Copyright 2011-2025 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
@@ -415,8 +415,10 @@ limitCopies want = case splitc ':' want of
[v, n] -> case parsetrustspec v of
Just checker -> go n $ checktrust checker
Nothing -> go n $ checkgroup (toGroup v)
- [n] -> go n $ const $ return True
- _ -> Left "bad value for copies"
+ _ -> case splitc '=' want of
+ [gl, n] -> go n $ checkgrouplimit (parseGroupLimit gl)
+ [n] -> go n $ const $ return True
+ _ -> Left "bad value for copies"
where
go num good = case readish num of
Nothing -> Left "bad number for copies"
@@ -437,6 +439,9 @@ limitCopies want = case splitc ':' want of
return $ numCopiesCount us >= n
checktrust checker u = checker <$> lookupTrust u
checkgroup g u = S.member g <$> lookupGroups u
+ checkgrouplimit gl u = do
+ m <- uuidsByGroup <$> groupMap
+ return (checkGroupLimit gl m u)
parsetrustspec s
| "+" `isSuffixOf` s = (<=) <$> readTrustLevel (beginning s)
| otherwise = (==) <$> readTrustLevel s
@@ -446,7 +451,7 @@ addLackingCopies :: String -> Bool -> String -> Annex ()
addLackingCopies desc approx = addLimit . limitLackingCopies desc approx
limitLackingCopies :: String -> Bool -> MkLimit Annex
-limitLackingCopies desc approx want = case readish want of
+limitLackingCopies desc approx want = case readish numwant of
Just needed -> Right $ MatchFiles
{ matchAction = const $ \notpresent mi -> flip checkKey mi $
go mi needed notpresent
@@ -460,13 +465,28 @@ limitLackingCopies desc approx want = case readish want of
}
Nothing -> Left "bad value for number of lacking copies"
where
- go mi needed notpresent key =
- limitCheckNumCopies approx mi notpresent key vs
+ go mi needed notpresent key = case (groupwant, grouplimit) of
+ (Nothing, []) -> check (const True)
+ (Just g, _) -> do
+ s <- fromMaybe S.empty
+ . M.lookup g
+ . uuidsByGroup
+ <$> groupMap
+ check (`S.member` s)
+ (Nothing, gl) -> do
+ m <- uuidsByGroup <$> groupMap
+ check (checkGroupLimit gl m)
where
+ check uuidp = limitCheckNumCopies approx mi notpresent uuidp key vs
vs nhave numcopies' = numcopies' - nhave >= needed
-
-limitCheckNumCopies :: Bool -> MatchInfo -> AssumeNotPresent -> Key -> (Int -> Int -> v) -> Annex v
-limitCheckNumCopies approx mi notpresent key vs = do
+ (groupwant, grouplimit, numwant) = case splitc ':' want of
+ (g:n:[]) -> (Just (toGroup g), [], n)
+ _ -> case splitc '=' want of
+ (gl:n:[]) -> (Nothing, parseGroupLimit gl, n)
+ _ -> (Nothing, [], want)
+
+limitCheckNumCopies :: Bool -> MatchInfo -> AssumeNotPresent -> (UUID -> Bool) -> Key -> (Int -> Int -> v) -> Annex v
+limitCheckNumCopies approx mi notpresent uuidp key vs = do
numcopies <- if approx
then approxNumCopies
else case mi of
@@ -474,7 +494,7 @@ limitCheckNumCopies approx mi notpresent key vs = do
matchFile fi
MatchingInfo {} -> approxNumCopies
MatchingUserInfo {} -> approxNumCopies
- us <- filter (`S.notMember` notpresent)
+ us <- filter (\u -> uuidp u && u `S.notMember` notpresent)
<$> (trustExclude UnTrusted =<< Remote.keyLocations key)
return $ numCopiesCheck'' us vs numcopies
where
@@ -684,7 +704,7 @@ limitFullyBalanced'' filtercandidates termname mu getgroupmap want =
| n == "lackingcopies" -> go g $
Left $ \mi notpresent key ->
let vs nhave numcopies = numcopies - nhave
- in limitCheckNumCopies False mi notpresent key vs
+ in limitCheckNumCopies False mi notpresent (const True) key vs
| otherwise -> maybe
(Left $ "bad number for " ++ termname)
(go g . Right)
@@ -964,3 +984,30 @@ matchDescSimple s Nothing = Utility.Matcher.MatchDesc s
(=?) :: String -> String -> (Maybe Bool -> Utility.Matcher.MatchDesc)
k =? v = matchDescSimple (k ++ "=" ++ v)
+
+data GroupLimit
+ = GroupInclude Group
+ | GroupExclude Group
+ deriving (Show)
+
+parseGroupLimit :: String -> [GroupLimit]
+parseGroupLimit = go GroupInclude
+ where
+ go b s = case break (\c -> c == '+' || c == '-') s of
+ ("", "") -> []
+ ("", ('+':s')) -> go GroupInclude s'
+ ("", ('-':s')) -> go GroupExclude s'
+ (groupname, s') -> b (toGroup groupname) : go GroupInclude s'
+
+checkGroupLimit :: Ord t => [GroupLimit] -> M.Map Group (S.Set t) -> t -> Bool
+checkGroupLimit gl m u =
+ let (includes, excludes) = partition fst (map decompose gl)
+ in (any member includes || null includes) && not (any member excludes)
+ where
+ decompose (GroupInclude g) = (True, g)
+ decompose (GroupExclude g) = (False, g)
+
+ member (_, g) =
+ case M.lookup g m of
+ Nothing -> False
+ Just s -> u `S.member` s
diff --git a/doc/git-annex-matching-options.mdwn b/doc/git-annex-matching-options.mdwn
index 5bfeb73f05..4e490a9f2e 100644
--- a/doc/git-annex-matching-options.mdwn
+++ b/doc/git-annex-matching-options.mdwn
@@ -106,7 +106,7 @@ in either of two repositories.
* `--copies=trustlevel:number`
Matches only when git-annex believes there are the specified number of
- copies, on remotes with the specified trust level. For example,
+ copies, or more, on remotes with the specified trust level. For example,
`--copies=trusted:2`
To match any trust level at or higher than a given level,
@@ -115,18 +115,58 @@ in either of two repositories.
* `--copies=groupname:number`
Matches only when git-annex believes there are the specified number of
- copies, on remotes in the specified group. For example,
+ copies, or more, on remotes in the specified group. For example,
`--copies=archive:2`
+* `copies=grouplimit=number`
+
+ Matches only files that git-annex believes have the specified number of
+ copies, or more, on repositories in groups allowed by the group limit.
+
+ In a group limit, use "+" before a group to include that group, and "-"
+ before a group to exclude it. (The first group to be included does
+ not need to be prefixed with a "+" though it may be.)
+
+ A group limit matches repositories that are in any of the included
+ groups and are in none of the excluded groups.
+
+ For example, "--copies=backup+archive-offsite=3" matches
+ files that are stored in 3 backup or archive repositories,
+ excluding offsite repositories. And "--copies=-offsite=3" matches
+ files that are stored in 3 repositories excluding offsite repositories.
+
* `--lackingcopies=number`
Matches only when git-annex believes that the specified number or
more additional copies need to be made in order to satisfy numcopies
settings.
-* `--approxlackingcopies=number`
+* `--lackingcopies=groupname:number`
+
+ Providing the name of a group limits `--lackingcopies` to only
+ considering repositories in that group toward the numcopies count.
+
+* `--lackingcopies=grouplimit=number`
+
+ Providing a group limit makes `--lackingcopies` only consider repositories
+ in groups that match the group limit toward the numcopies count.
(Diff truncated)
refine
diff --git a/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn b/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn index b46de3bbb0..aa31220b66 100644 --- a/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn +++ b/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn @@ -1,12 +1,23 @@ -I want a way to make a preferred content expression like -`lackingcopies` but that excludes copies in a group. - My example use case is I have a number of archive remotes. The archive groupwanted includes `approxlackingcopies=1` so numcopies-1 copies get sent to the archives, along with the 1 local copy. This means that `git-annex drop` isn't able to drop files; they need to be manually sent to one more archive remote first. -Syntax could be: `lackingcopies=number-group` eg `lackingcopies=1-client` +Idea: Add a way to make a preferred content expression like +`lackingcopies` but that excludes copies in a group. + +Syntax could be: `lackingcopies=number-groupname` eg `lackingcopies=1-client` + +Alternatively to excluding a group, it could only look at copies in a +particular group: `lackingcopies=groupname:number`. This would +correspond well with `copies=groupname:number`, and so seems better +on that basis. + +Both of these approaches would be more general if they supported multiple +groups. Eg `lackingcopies=number-client-othergroup` or +`lackingcopies=archive,othergroup:1`. +The nice thing about `lackingcopies=archive,othergroup:1` is that it +could be back-generalized to `copies=archive,othergroup:1` --[[Joey]]
idea
diff --git a/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn b/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn new file mode 100644 index 0000000000..b46de3bbb0 --- /dev/null +++ b/doc/todo/preferred_content_lackingcopies_excluding_a_group.mdwn @@ -0,0 +1,12 @@ +I want a way to make a preferred content expression like +`lackingcopies` but that excludes copies in a group. + +My example use case is I have a number of archive remotes. The archive +groupwanted includes `approxlackingcopies=1` so numcopies-1 copies get +sent to the archives, along with the 1 local copy. This means that +`git-annex drop` isn't able to drop files; they need to be manually sent to +one more archive remote first. + +Syntax could be: `lackingcopies=number-group` eg `lackingcopies=1-client` + +--[[Joey]]
close one todo and open another
diff --git a/doc/todo/make_more_special_remotes_support_importtree.mdwn b/doc/todo/make_more_special_remotes_support_importtree.mdwn index e182541d46..f85fc69bbc 100644 --- a/doc/todo/make_more_special_remotes_support_importtree.mdwn +++ b/doc/todo/make_more_special_remotes_support_importtree.mdwn @@ -6,7 +6,7 @@ more special remotes support tree importing. These built-in ones could support it: -* webdav +* webdav (partially done; see [[webdav_importtree_support]] ## external special remotes @@ -64,3 +64,5 @@ worthwhile. --[[Joey]] * https://github.com/Lykos153/AnnexRemote <https://github.com/Lykos153/AnnexRemote/issues/132> + +[[done]] --[[Joey]] diff --git a/doc/todo/webdav_importtree_support.mdwn b/doc/todo/webdav_importtree_support.mdwn new file mode 100644 index 0000000000..2379171591 --- /dev/null +++ b/doc/todo/webdav_importtree_support.mdwn @@ -0,0 +1,5 @@ +Make the webdav special remote support importtree=yes + +The `webdav-importtree` branch has a partial implementation, but finishing +it will need either improvements to the haskell DAV library, or parsing a +webdav response document in git-annex. --[[Joey]]
remove S3 from todo
I misunderstood some documentation; S3 supports importtree=yes for
unversioned buckets when not combined with exporttree=yes.
I misunderstood some documentation; S3 supports importtree=yes for
unversioned buckets when not combined with exporttree=yes.
diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn index 70d88e53cf..999841fbb2 100644 --- a/doc/special_remotes/S3.mdwn +++ b/doc/special_remotes/S3.mdwn @@ -140,7 +140,8 @@ the S3 remote. the content of files overwritten in this way can still be recovered, but you may have to look through the git history to find them. When versioning is not enabled, this risks data loss, and so git-annex - will not let you enable a remote with that configuration unless forced. + will not let you enable a remote with this set in combination + with exporttree unless forced. * `annexobjects` - When set to "yes" along with "exporttree=yes", this allows storing other objects in the remote along with the diff --git a/doc/todo/make_more_special_remotes_support_importtree.mdwn b/doc/todo/make_more_special_remotes_support_importtree.mdwn index d670ab1ff9..e182541d46 100644 --- a/doc/todo/make_more_special_remotes_support_importtree.mdwn +++ b/doc/todo/make_more_special_remotes_support_importtree.mdwn @@ -7,13 +7,6 @@ more special remotes support tree importing. These built-in ones could support it: * webdav -* S3 - - importtree=yes is only supported when versioning is enabled - (or when forced when version is not enabled). - - It would be possible to support importtree=yes without exporttree=yes - for any S3 bucket with no need of forcing. ## external special remotes
comment
diff --git a/doc/todo/support_more_backup_software_like_borg/comment_1_d6b85e37db0cdcadd224885758e4cbc4._comment b/doc/todo/support_more_backup_software_like_borg/comment_1_d6b85e37db0cdcadd224885758e4cbc4._comment new file mode 100644 index 0000000000..51e9bd581c --- /dev/null +++ b/doc/todo/support_more_backup_software_like_borg/comment_1_d6b85e37db0cdcadd224885758e4cbc4._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-07-28T15:46:54Z" + content=""" +Now that external special remotes can support importtree, it does seem like +all that is missing to have this in an external special remote is a way to +set `thirdPartyPopulated`. +"""]]
initial report on lack of support of http.SITE.sslCAInfo
diff --git a/doc/todo/Support_per_SITE_http.SITE.sslCAInfo_etc.mdwn b/doc/todo/Support_per_SITE_http.SITE.sslCAInfo_etc.mdwn new file mode 100644 index 0000000000..180b839cf6 --- /dev/null +++ b/doc/todo/Support_per_SITE_http.SITE.sslCAInfo_etc.mdwn @@ -0,0 +1,57 @@ +https://git-annex.branchable.com/git-annex/ mentions + +> (The http.sslCAInfo and http.sslCAPath git configs have the same effect.) + +but unfortunately it seems that there is no symmetric to git support for per-SITE settings, e.g. + +``` +(tmp) yoh@typhon:/tmp$ git clone https://datalad-test.local.lan/DLTC/sample_datalad_provider /tmp/my_clone +Cloning into '/tmp/my_clone'... +fatal: unable to access 'https://datalad-test.local.lan/DLTC/sample_datalad_provider/': server verification failed: certificate signer not trusted. (CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none) +``` + +can be overcome for git with + +``` +(tmp) yoh@typhon:/tmp$ openssl s_client -connect datalad-test.local.lan:443 </dev/null 2>/dev/null | openssl x509 > ~/.config/git-credential-diab/datalad-test.local.lan-ca.pem +(tmp) yoh@typhon:/tmp$ git config --global http.https://datalad-test.local.lan.sslCAInfo ~/.config/git-credential-diab/datalad-test.local.lan-ca.pem +(tmp) yoh@typhon:/tmp$ git clone https://datalad-test.local.lan/DLTC/sample_datalad_provider /tmp/my_clone +Cloning into '/tmp/my_clone'... +remote: Enumerating objects: 47, done. +remote: Counting objects: 100% (47/47), done. +remote: Compressing objects: 100% (38/38), done. +remote: Total 47 (delta 12), reused 0 (delta 0), pack-reused 0 (from 0) +Receiving objects: 100% (47/47), 4.33 KiB | 4.33 MiB/s, done. +Resolving deltas: 100% (12/12), done. +``` + +but for recent annex it remains "not good" (until we override for all sites): + +``` +(tmp) yoh@typhon:/tmp/my_clone$ git annex version --raw; echo +10.20260624-g1f50df25f767231afaf9de9119f4a339d0619256 + +(tmp) yoh@typhon:/tmp$ cd my_clone/ + +(tmp) yoh@typhon:/tmp/my_clone$ git annex whereis annexed_demo.bin + + Remote origin not usable by git-annex; setting annex-ignore + + https://datalad-test.local.lan/DLTC/sample_datalad_provider/config download failed: HandshakeFailed (Error_Protocol "certificate has unknown CA" UnknownCa) +whereis annexed_demo.bin (2 copies) + 01486323-6af3-4c59-a372-dea7d9340f6d -- git@02915365b276:/data/git/repositories/dltc/sample_datalad_provider.git + a69e3dc7-4554-4aae-819f-c65b7dd414e9 -- datalad@datalad-kbdm-datalad-provider:/tmp/push_test +ok + +(tmp) yoh@typhon:/tmp/my_clone$ git config --unset remote.origin.annex-ignore + +(tmp) yoh@typhon:/tmp/my_clone$ git annex whereis annexed_demo.bin +whereis annexed_demo.bin (2 copies) + 01486323-6af3-4c59-a372-dea7d9340f6d -- git@02915365b276:/data/git/repositories/dltc/sample_datalad_provider.git [origin] + a69e3dc7-4554-4aae-819f-c65b7dd414e9 -- datalad@datalad-kbdm-datalad-provider:/tmp/push_test +ok + +# so no complains about ssl certificate and subsequent annex copy actually works out nicely +``` + +
close
diff --git a/doc/todo/fast_import_for_externals_with_importKey.mdwn b/doc/todo/fast_import_for_externals_with_importKey.mdwn index 39120b3265..774a41a30f 100644 --- a/doc/todo/fast_import_for_externals_with_importKey.mdwn +++ b/doc/todo/fast_import_for_externals_with_importKey.mdwn @@ -16,3 +16,7 @@ although it does document the key format and the external could be expected to generate it by itself. Or there could be a reply like "IMPORTKEY-SUCCESS SHA1 $hash". --[[Joey]]: + +> [[done]], but I punted on the key generation and left that up to +> the external program. It may make sense for git-annex to later +> grow a command to handle that. --[[Joey]]
Merge branch 'importKey'
External special remote protocol extended to support IMPORTKEY.
* External special remote protocol extended to support IMPORTKEY.
* git-annex-remote-internetarchive supports --no-content imports.
Made IMPORTKEY an protocol extension because it would be a lot of overhead
for IMPORTKEY requests to be made for each key when the external special
remote doesn't support it.
This needed importKeyWithContentIdentifier and importKey to be changed to
take an Annex action, which returns the Maybe. Annoying, but in the case of
the external special remote, that is the only way to access the content of
the extensions list. This Annex action should only ever do such a thing, so
it could be an IO action, or even a STM action, but I left it Annex for
now.
* External special remote protocol extended to support IMPORTKEY.
* git-annex-remote-internetarchive supports --no-content imports.
Made IMPORTKEY an protocol extension because it would be a lot of overhead
for IMPORTKEY requests to be made for each key when the external special
remote doesn't support it.
This needed importKeyWithContentIdentifier and importKey to be changed to
take an Annex action, which returns the Maybe. Annoying, but in the case of
the external special remote, that is the only way to access the content of
the extensions list. This Annex action should only ever do such a thing, so
it could be an IO action, or even a STM action, but I left it Annex for
now.
diff --git a/Annex/Import.hs b/Annex/Import.hs
index d8dd1057df..fdc6af7cab 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -499,9 +499,9 @@ buildImportTreesHistory
buildImportTreesHistory converttree basetree msubdir history hdl = S.fromList
<$> mapM (\ic -> buildImportTreesGeneric' converttree basetree msubdir ic hdl) history
-canImportKeys :: Remote -> Bool -> Bool
+canImportKeys :: Remote -> Bool -> Annex Bool
canImportKeys remote importcontent =
- importcontent || isJust (Remote.importKey ia)
+ pure (importcontent) <||> (isJust <$> Remote.importKey ia)
where
ia = Remote.importActions remote
@@ -671,7 +671,7 @@ importKeys
-> ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)
-> Annex (ImportResult (ImportableContentsChunkable Annex (Either Sha Key)))
importKeys remote importtreeconfig importcontent thirdpartypopulated importablecontents = do
- unless (canImportKeys remote importcontent) $
+ unlessM (canImportKeys remote importcontent) $
giveup "This remote does not support importing without downloading content."
-- This map is used to remember content identifiers that
-- were just imported, before they have necessarily been
@@ -807,7 +807,7 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
return (Right job)
thirdpartypopulatedimport db (loc, (cid, sz)) =
- case Remote.importKey (Remote.importActions remote) of
+ Remote.importKey (Remote.importActions remote) >>= \case
Nothing -> return Nothing
Just importkey ->
tryNonAsync (importkey loc cid sz nullMeterUpdate) >>= \case
@@ -823,20 +823,21 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
importordownload cidmap (loc, (cid, sz)) largematcher = do
f <- locworktreefile loc
matcher <- largematcher f
+ let usedownload = dodownload cidmap (loc, (cid, sz)) f matcher
+ let useimport = doimport cidmap (loc, (cid, sz)) f matcher
-- When importing a key is supported, always use it rather
-- than downloading and retrieving a key, to avoid
-- generating trees with different keys for the same content.
- let act = if importcontent
- then case Remote.importKey (Remote.importActions remote) of
- Nothing -> dodownload
+ if importcontent
+ then Remote.importKey (Remote.importActions remote) >>= \case
+ Nothing -> usedownload
Just _ -> if Utility.Matcher.introspect matchNeedsFileContent (fst matcher)
- then dodownload
- else doimport
- else doimport
- act cidmap (loc, (cid, sz)) f matcher
+ then usedownload
+ else useimport
+ else useimport
doimport cidmap (loc, (cid, sz)) f matcher =
- case Remote.importKey (Remote.importActions remote) of
+ Remote.importKey (Remote.importActions remote) >>= \case
Nothing -> error "internal" -- checked earlier
Just importkey -> do
when (Utility.Matcher.introspect matchNeedsFileContent (fst matcher)) $
diff --git a/CHANGELOG b/CHANGELOG
index b2abdec86b..78eb239927 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,5 +1,7 @@
git-annex (10.20260718) UNRELEASED; urgency=medium
+ * External special remote protocol extended to support IMPORTKEY.
+ * git-annex-remote-internetarchive supports --no-content imports.
* Fix reversion in 8.20200226 that broke git-annex benchmark --databases
* importfeed: Fix reporting and logging of problems with feeds.
* importfeed: When adding an url, indicate which feed it is from.
diff --git a/Command/Sync.hs b/Command/Sync.hs
index dabd6bf3ce..91d655227e 100644
--- a/Command/Sync.hs
+++ b/Command/Sync.hs
@@ -627,8 +627,8 @@ importRemote importcontent o remote currbranch
Nothing -> noop
Just b -> do
let (branch, subdir) = splitRemoteAnnexTrackingBranchSubdir b
- if canImportKeys remote importcontent
- then do
+ ifM (canImportKeys remote importcontent)
+ ( do
addunlockedmatcher <- addUnlockedMatcher
Command.Import.seekRemote remote branch subdir importcontent (CheckGitIgnore True) addunlockedmatcher []
-- Importing generates a branch
@@ -638,7 +638,8 @@ importRemote importcontent o remote currbranch
-- mergeing it.
mc <- mergeConfig True
void $ mergeRemote remote currbranch mc o
- else warning $ UnquotedString $ "Cannot import from " ++ Remote.name remote ++ " when not syncing content."
+ , warning $ UnquotedString $ "Cannot import from " ++ Remote.name remote ++ " when not syncing content."
+ )
where
wantpull = remoteAnnexPull (Remote.gitconfig remote)
@@ -651,10 +652,12 @@ importRemote importcontent o remote currbranch
pullThirdPartyPopulated :: SyncOptions -> Remote -> CommandSeek
pullThirdPartyPopulated o remote
| not (pullOption o) || not wantpull = noop
- | not (canImportKeys remote False) = noop
- | otherwise = void $ includeCommandAction $ starting "list" ai si $
- Command.Import.listContents' remote ImportTree (CheckGitIgnore False) go
+ | otherwise =
+ whenM (canImportKeys remote False)
+ start
where
+ start = void $ includeCommandAction $ starting "list" ai si $
+ Command.Import.listContents' remote ImportTree (CheckGitIgnore False) go
go (Just importable) = importChanges remote ImportTree False True importable >>= \case
ImportFinished postexportlogupdate imported -> do
(_t, updatestate) <- recordImportTree remote ImportTree Nothing imported postexportlogupdate
diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index 50d77b6d07..a5e7e85a5b 100644
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -101,7 +101,7 @@ gen r u rc gc rs = do
, importActions = importUnsupported
, exportImportActions = ExportImportActions
{ listImportableOrExportedContents = listImportableOrExportedContentsM serial adir c
- , importKeyWithContentIdentifier = Nothing
+ , importKeyWithContentIdentifier = pure Nothing
, retrieveExportWithContentIdentifier = retrieveExportWithContentIdentifierM serial adir
, storeExportWithContentIdentifier = storeExportWithContentIdentifierM serial adir
, removeExportWithContentIdentifier = removeExportWithContentIdentifierM serial adir
diff --git a/Remote/Borg.hs b/Remote/Borg.hs
index 28d283048c..59ec3d03ae 100644
--- a/Remote/Borg.hs
+++ b/Remote/Borg.hs
@@ -98,7 +98,8 @@ gen r u rc gc rs = do
, importActions = importUnsupported
, exportImportActions = ExportImportActions
{ listImportableOrExportedContents = listImportableOrExportedContentsM u borgrepo c
- , importKeyWithContentIdentifier = Just ThirdPartyPopulated.importKey
+ , importKeyWithContentIdentifier = pure $
+ Just ThirdPartyPopulated.importKey
, retrieveExportWithContentIdentifier = retrieveExportWithContentIdentifierM borgrepo
, checkPresentExportWithContentIdentifier = checkPresentExportWithContentIdentifierM borgrepo
-- This remote is thirdPartyPopulated, so these
diff --git a/Remote/Directory.hs b/Remote/Directory.hs
index a87519b0ba..9ab7e4a77b 100644
--- a/Remote/Directory.hs
+++ b/Remote/Directory.hs
@@ -119,7 +119,8 @@ gen r u rc gc rs = do
, importActions = importUnsupported
, exportImportActions = ExportImportActions
{ listImportableOrExportedContents = listImportableOrExportedContentsM ii dir
- , importKeyWithContentIdentifier = Just (importKeyWithContentIdentifierM ii dir)
+ , importKeyWithContentIdentifier = pure $
+ Just (importKeyWithContentIdentifierM ii dir)
, retrieveExportWithContentIdentifier = retrieveExportWithContentIdentifierM ii dir cow
, storeExportWithContentIdentifier = storeExportWithContentIdentifierM ii dir cow fastcopy
, removeExportWithContentIdentifier = removeExportWithContentIdentifierM ii dir
diff --git a/Remote/External.hs b/Remote/External.hs
index 62bc76fed3..949fe6eab3 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -129,7 +129,7 @@ gen rt externalprogram r u rc gc rs
let importactions = if importsupported
then ImportActions
{ listImportableContents = listImportableContentsM external
- , importKey = Nothing
+ , importKey = importKeyM external
, retrieveImport = retrieveImportM external gc
, checkPresentImport = checkPresentImportM external gc
}
@@ -613,6 +613,32 @@ listImportableContentsM external =
go _ _ UNSUPPORTED_REQUEST = result Nothing
go _ _ _ = Nothing
+importKeyM
+ :: External
+ -> Annex (Maybe (ImportLocation -> ContentIdentifier -> ByteSize -> MeterUpdate -> Annex (Maybe Key)))
+importKeyM external =
+ withExternalState external $ \st ->
+ return $ if importKeyExtensionEnabled (externalExtensions st)
+ then Just go
+ else Nothing
+ where
+ go loc cid sz p =
+ handleRequestImport' external loc (IMPORTKEY sz cid) Nothing $ \case
+ IMPORTKEY_SUCCESS k ->
+ result (Just k)
+ IMPORTKEY_FAILURE err ->
+ giveup err
+ IMPORTKEY_SKIP ->
+ result Nothing
+ DELEGATE ps -> Just $ do
+ delegate <- getDelegateRemote external ps
+ importKey (importActions delegate) >>= \case
+ Just a -> Result <$> a loc cid sz p
+ Nothing -> giveup "IMPORTKEY delegated to a special remote that does not support it"
+ UNSUPPORTED_REQUEST ->
+ giveup "IMPORTKEY not implemented by external special remote, but it claimed to support it"
+ _ -> Nothing
(Diff truncated)
IMPORTKEY extension to external special remote protocol
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 43a14ea90f..9fd61b5f6b 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -45,7 +45,7 @@ Recent versions of git-annex respond with a message indicating
protocol extensions that it supports. Older versions of
git-annex do not send this message.
- EXTENSIONS INFO ASYNC GETGITREMOTENAME UNAVAILABLERESPONSE TRANSFER-RETRIEVE-URL CHECKPRESENT-URL DELEGATE
+ EXTENSIONS INFO ASYNC GETGITREMOTENAME UNAVAILABLERESPONSE TRANSFER-RETRIEVE-URL CHECKPRESENT-URL DELEGATE IMPORTKEY
The special remote can respond to that with its own EXTENSIONS message, listing
any extensions it wants to use.
@@ -529,6 +529,9 @@ These protocol extensions are currently supported.
* `CHECKPRESENT-URL`
This allows the `CHECKPRESENT-URL` response to be used
in reply to `CHECKPRESENT` and `CHECKPRESENTEXPORT`.
+* `IMPORTKEY`
+ This indicates that the external special remote supports the `IMPORTKEY`
+ request documented in the [[export_and_import_appendix]].
* `DELEGATE`
This allows the `DELEGATE` response to be used in reply to many requests.
That causes the request to be handled by a delegate special remote,
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 66f2d30569..77ab32a31d 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -207,7 +207,10 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
No response is made to this message.
* `RETRIEVEIMPORT File`
Retrieves the content of a file with the previously provided
- `IMPORT` Name from the special remote to the File on local disk.
+ `IMPORT` Name from the special remote to the File on local disk.
+ Note that the special remote can send `DOWNLOAD-URL` messages
+ without negotiating support for it in the protocol, because versions
+ of git-annex that support this also support `DOWNLOAD-URL`.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
@@ -220,7 +223,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
this lets it offload that work to git-annex.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
- See [[delegate_appendix]].
+ See [[delegate_appendix]].
* `CHECKPRESENTIMPORT Key`
Requests the remote to check if the previously provided `IMPORT` Name
is present in it.
@@ -239,10 +242,35 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
-
-Note that the special remote can send `DOWNLOAD-URL` without negotiating
-support for it in the protocol, because versions of git-annex that support
-this also support `DOWNLOAD-URL`.
+* `IMPORTKEY Size ContentIdentifier`
+ Requests the remote generate a git-annex key for content stored
+ on it without downloading that content.
+ This request is a protocol extension and will only be sent if the
+ special remote sent an `EXTENSIONS` that included `IMPORTKEY`.
+ Implementing this allows using `git-annex import --no-content` from
+ the remote.
+ The remote can generate any [[type of git-annex key|backends]] that uses
+ a checksum. For example, if a cloud storage service allows querying for
+ a md5, key, it's ok to generate a MD5 key.
+ To generate the key, see [[internals/key_format]].
+ The previously provided `IMPORT` Name specifies which file on the remote
+ to generate the key for.
+ The Size is provided because the key should include the size of the file.
+ The ContentIdentifier is provided in case it can be used to avoid
+ extra work in determining the key. For example, if a remote uses a md5
+ as the ContentIdentifier in `LISTIMPORTABLECONTENTS`, it
+ can reuse the md5 to generate the key. It is not necessary to guarantee
+ that the file on the remote still has the same ContentIdentifier.
+ * `IMPORTKEY-SUCCESS Key`
+ Responding with the imported key.
+ * `IMPORTKEY-FAILURE ErrorMsg`
+ Indicates that it was not possible to import the key.
+ (Perhaps because the remote cannot be contacted.)
+ * `IMPORTKEY-SKIP`
+ This makes the file not be included in the imported tree.
+ * `DELEGATE type=value [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
### import example
no longer a draft
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn index 66f2d30569..6267b1024c 100644 --- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn +++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn @@ -132,12 +132,12 @@ a request, it can reply with `UNSUPPORTED-REQUEST`. ## import interface -(This part is a draft, not implemented yet.) - git-annex will use this interface when the special remote is initialized with importtree=yes and indicates that it supports imports. +git-annex version 10.20260717 was the first to support this. + ### content identifiers The special remote needs to have some way to identify a particular version of a
Revert "comment"
This reverts commit 71711a5670dde105e4ec183dd40898a974aaa4c9.
Turns out that's an owncloud extension.
This reverts commit 71711a5670dde105e4ec183dd40898a974aaa4c9.
Turns out that's an owncloud extension.
diff --git a/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment b/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment deleted file mode 100644 index 27ec74b2a7..0000000000 --- a/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2026-07-27T16:42:33Z" - content=""" -webdav supports reading hashes -<https://central.owncloud.org/t/reading-hashes-md5-or-sha1-via-webdav/14348/2> -so this could also be used for git-annex's built-in webdav special remote. -"""]]
comment
diff --git a/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment b/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment new file mode 100644 index 0000000000..27ec74b2a7 --- /dev/null +++ b/doc/todo/fast_import_for_externals_with_importKey/comment_2_51850e59c2ef8f58b9f656f3b23c0274._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-07-27T16:42:33Z" + content=""" +webdav supports reading hashes +<https://central.owncloud.org/t/reading-hashes-md5-or-sha1-via-webdav/14348/2> +so this could also be used for git-annex's built-in webdav special remote. +"""]]
format
diff --git a/doc/todo/make_more_special_remotes_support_importtree.mdwn b/doc/todo/make_more_special_remotes_support_importtree.mdwn index 8861eed693..d670ab1ff9 100644 --- a/doc/todo/make_more_special_remotes_support_importtree.mdwn +++ b/doc/todo/make_more_special_remotes_support_importtree.mdwn @@ -25,49 +25,49 @@ Update: Went down the list and thought about if it might make sense for each external to support it, and pinged those that seem likely to be worthwhile. --[[Joey]] -* https://github.com/DanielDent/git-annex-remote-rclone +* https://github.com/DanielDent/git-annex-remote-rclone <https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/issues/81> -* rclone gitannex +* rclone gitannex <https://github.com/rclone/rclone/issues/9666> -* https://github.com/encryptio/git-annex-remote-b2 +* https://github.com/encryptio/git-annex-remote-b2 maybe useful? -* https://github.com/mhameed/git-annex-remote-ceph +* https://github.com/mhameed/git-annex-remote-ceph seems unlikely to make sense to import from a ceph pool? -* https://pypi.org/project/git-annex-remote-internxt/ +* https://pypi.org/project/git-annex-remote-internxt/ <https://hub.datalad.org/git-annex/git-annex-remote-internxt/issues/21> -* https://git.sr.ht/~ehmry/git-annex-remote-nncp +* https://git.sr.ht/~ehmry/git-annex-remote-nncp probably not useful, since the purpose is P2P transfers with both sides using git-annex -* https://github.com/grawity/code/blob/master/net/git-annex-remote-gvfs +* https://github.com/grawity/code/blob/master/net/git-annex-remote-gvfs (repository is gone) -* https://github.com/3ofcoins/knife-annex/ +* https://github.com/3ofcoins/knife-annex/ old and uses hook backend, so not applicable -* https://github.com/xloem/gitlakepy - Seems to use a non-defunct blockchain storage -* https://github.com/Schnouki/git-annex-remote-hubic +* https://github.com/xloem/gitlakepy + Seems to use a now-defunct blockchain storage +* https://github.com/Schnouki/git-annex-remote-hubic repository is archived -* https://pypi.org/project/git-annex-remote-dbx/ +* https://pypi.org/project/git-annex-remote-dbx/ does not seem active and rclone also supports dropbox -* https://github.com/bgilbert/gcsannex +* https://github.com/bgilbert/gcsannex repository is archived -* https://github.com/Lykos153/git-annex-remote-googledrive +* https://github.com/Lykos153/git-annex-remote-googledrive commented on existing issue <https://github.com/Lykos153/git-annex-remote-googledrive/issues/21> -* https://github.com/dxtr/megaannex-go +* https://github.com/dxtr/megaannex-go seems inactive -* https://github.com/TobiasTheViking/megaannex +* https://github.com/TobiasTheViking/megaannex (no longer working) -* https://github.com/TobiasTheViking/owncloudannex +* https://github.com/TobiasTheViking/owncloudannex open question about whether this has any benefit over regular webdav: <https://github.com/TobiasTheViking/owncloudannex/issues/6> -* https://github.com/TobiasTheViking/skydriveannex +* https://github.com/TobiasTheViking/skydriveannex too old/inactive to seem worth filing an issue on -* https://github.com/TobiasTheViking/flickrannex +* https://github.com/TobiasTheViking/flickrannex not applicable -* https://github.com/TobiasTheViking/dropboxannex +* https://github.com/TobiasTheViking/dropboxannex does not seem active and rclone also supports dropbox ## libraries -* https://github.com/Lykos153/AnnexRemote +* https://github.com/Lykos153/AnnexRemote <https://github.com/Lykos153/AnnexRemote/issues/132>
pinged external maintainers
diff --git a/doc/todo/make_more_special_remotes_support_importtree.mdwn b/doc/todo/make_more_special_remotes_support_importtree.mdwn
index df83ef9b92..8861eed693 100644
--- a/doc/todo/make_more_special_remotes_support_importtree.mdwn
+++ b/doc/todo/make_more_special_remotes_support_importtree.mdwn
@@ -2,6 +2,8 @@ Now that importtree=yes is supported for special remotes without also
needing to implement exporttree=yes, it should be possible to make a lot
more special remotes support tree importing.
+## builtin
+
These built-in ones could support it:
* webdav
@@ -13,6 +15,59 @@ These built-in ones could support it:
It would be possible to support importtree=yes without exporttree=yes
for any S3 bucket with no need of forcing.
+## external special remotes
+
Lots of external special remotes could also probably support it.
That will be left up to their maintainers, but it would make sense to ping
the maintainers.
+
+Update: Went down the list and thought about if it might make sense for
+each external to support it, and pinged those that seem likely to be
+worthwhile. --[[Joey]]
+
+* https://github.com/DanielDent/git-annex-remote-rclone
+ <https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/issues/81>
+* rclone gitannex
+ <https://github.com/rclone/rclone/issues/9666>
+* https://github.com/encryptio/git-annex-remote-b2
+ maybe useful?
+* https://github.com/mhameed/git-annex-remote-ceph
+ seems unlikely to make sense to import from a ceph pool?
+* https://pypi.org/project/git-annex-remote-internxt/
+ <https://hub.datalad.org/git-annex/git-annex-remote-internxt/issues/21>
+* https://git.sr.ht/~ehmry/git-annex-remote-nncp
+ probably not useful, since the purpose is P2P
+ transfers with both sides using git-annex
+* https://github.com/grawity/code/blob/master/net/git-annex-remote-gvfs
+ (repository is gone)
+* https://github.com/3ofcoins/knife-annex/
+ old and uses hook backend, so not applicable
+* https://github.com/xloem/gitlakepy
+ Seems to use a non-defunct blockchain storage
+* https://github.com/Schnouki/git-annex-remote-hubic
+ repository is archived
+* https://pypi.org/project/git-annex-remote-dbx/
+ does not seem active and rclone also supports dropbox
+* https://github.com/bgilbert/gcsannex
+ repository is archived
+* https://github.com/Lykos153/git-annex-remote-googledrive
+ commented on existing issue
+ <https://github.com/Lykos153/git-annex-remote-googledrive/issues/21>
+* https://github.com/dxtr/megaannex-go
+ seems inactive
+* https://github.com/TobiasTheViking/megaannex
+ (no longer working)
+* https://github.com/TobiasTheViking/owncloudannex
+ open question about whether this has any benefit over regular
+ webdav: <https://github.com/TobiasTheViking/owncloudannex/issues/6>
+* https://github.com/TobiasTheViking/skydriveannex
+ too old/inactive to seem worth filing an issue on
+* https://github.com/TobiasTheViking/flickrannex
+ not applicable
+* https://github.com/TobiasTheViking/dropboxannex
+ does not seem active and rclone also supports dropbox
+
+## libraries
+
+* https://github.com/Lykos153/AnnexRemote
+ <https://github.com/Lykos153/AnnexRemote/issues/132>
comment
diff --git a/doc/todo/fast_import_for_externals_with_importKey/comment_1_620576b1a304cbefd1b8a7eaab13159c._comment b/doc/todo/fast_import_for_externals_with_importKey/comment_1_620576b1a304cbefd1b8a7eaab13159c._comment new file mode 100644 index 0000000000..ff06b1e742 --- /dev/null +++ b/doc/todo/fast_import_for_externals_with_importKey/comment_1_620576b1a304cbefd1b8a7eaab13159c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2026-07-27T15:30:28Z" + content=""" +rclone would be another use case, some of its remotes support +server-side generation of md5, sha1, or other hashes. +"""]]
remove redundant IA via S3 in list
The IA page covers that.
The IA page covers that.
diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn index cb7991f745..501709c740 100644 --- a/doc/special_remotes.mdwn +++ b/doc/special_remotes.mdwn @@ -50,7 +50,6 @@ Here are specific instructions for using git-annex with various services: * [[IMAP|forum/special_remote_for_IMAP]] * [Internxt Drive](https://pypi.org/project/git-annex-remote-internxt/) * [[Internet_Archive]] -* [[tips/Internet_Archive_via_S3]] * [[ipfs]] * [[Jottacloud|rclone]] * [[Mega|tips/megaannex]]
Added a comment
diff --git a/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_2_0210a1cc9b01cac62b1da555250ba060._comment b/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_2_0210a1cc9b01cac62b1da555250ba060._comment new file mode 100644 index 0000000000..8eab5e9528 --- /dev/null +++ b/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_2_0210a1cc9b01cac62b1da555250ba060._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="nobodyinperson" + avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5" + subject="comment 2" + date="2026-07-26T11:07:26Z" + content=""" +Well you can already do that with `git annex config --set annex.addunlocked true`, right? 🤔 +"""]]
Added a comment
diff --git a/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_1_6de4f4546dd797a223186ec536e8a24b._comment b/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_1_6de4f4546dd797a223186ec536e8a24b._comment new file mode 100644 index 0000000000..7e32b47dca --- /dev/null +++ b/doc/forum/Disable_autoswitching_to_adjusted__40__unlocked__41___branch/comment_1_6de4f4546dd797a223186ec536e8a24b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 1" + date="2026-07-21T17:08:46Z" + content=""" +I feel like I had asked similar question in the past but failed to find it or an answer ;-) + +Indeed it feels like it would be great to be able to just declare to always do only \"unlocked\" mode operations in a given git-annex repo, and thus potentially keeping it always in the \"unlocked\" mode (even when on system with symlinks support), but without requiring any `adjusted/` \"synchronization\" altogether. +"""]]
add news item for git-annex 10.20260717
diff --git a/doc/news/version_10.20260420.mdwn b/doc/news/version_10.20260420.mdwn deleted file mode 100644 index abd090e745..0000000000 --- a/doc/news/version_10.20260420.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -git-annex 10.20260420 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * disableremote: New command. - * Fix annexUrl to inherit any password that is set in the remote url. - * Add DELEGATE extension to the external special remote protocol. - * Avoid dying of an exception when when stdout gets closed by eg head(1), - and avoid a crash loop when stderr is closed and git-annex dies of an - exception. - Fixes reversion introduced in version 10.20230407. - * Improve UUID sanitization. - * Deal with breaking changes to test concurrency in tasty-1.5.4."""]] \ No newline at end of file diff --git a/doc/news/version_10.20260717.mdwn b/doc/news/version_10.20260717.mdwn new file mode 100644 index 0000000000..37abe8d757 --- /dev/null +++ b/doc/news/version_10.20260717.mdwn @@ -0,0 +1,18 @@ +git-annex 10.20260717 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * External special remote protocol extended to support import. + Developers of external special remotes should consider if import makes + sense for them and add support. + * Support importtree=yes with rsync special remotes. + * Added git-annex-remote-internetarchive special remote that imports an + Internet Archive item and serves as an example for how to easily write + importtree=yes external special remotes. + https://git-annex.branchable.com/tips/how\_to\_make\_a\_simple\_importtree\_special\_remote/ + * Added DOWNLOAD-URL extension to the external special remote protocol. + * git-annex is guaranteed to not contain LLM generated code, + and will attempt to remain buildable with versions of dependencies + that predate the addition of any LLM generated code. + See https://git-annex.branchable.com/no\_llm\_code/ + * git-annex.cabal: Added NoLLMDependencies build flag. + * Added stack-NoLLMDependencies.yaml + * Avoid redundant hash verification after a hash verification fails. + * Fix build with time-1.15."""]] \ No newline at end of file
fix link
diff --git a/doc/special_remotes/Internet_Archive.mdwn b/doc/special_remotes/Internet_Archive.mdwn
index 9ede028a0b..1a65585e9c 100644
--- a/doc/special_remotes/Internet_Archive.mdwn
+++ b/doc/special_remotes/Internet_Archive.mdwn
@@ -2,6 +2,6 @@
remote in two different ways:
1. To upload files to it, see
- [[tips/[The Internet Archive](http://www.archive.org/)]]
+ [[tips/Internet_Archive_via_S3]]
2. To [[import a tree of files|git-annex-import]] from an existing
Internet Archive item, use [[external/git-annex-remote-internetarchive]].
reorg
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index fe8de64306..4e1cc9d942 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -161,16 +161,16 @@ like this:
Similarly, the CHECKPRESENTIMPORT request is git-annex just checking
if the file still appears to be present in the remote.
-(It does not need to verify that the has the same content that it did when
-imported, checking the size or even if HTTP HEAD succeeds is enough.
-Although if you can verify the content inexpensively, by all means do so.)
CHECKPRESENTIMPORT)
key="$2"
echo CHECKPRESENT-URL "$key" "https://archive.org/download/$item//$importlocation"
;;
-And if you need to hit the url yourself, you could do something like this:
+If you need to hit the url yourself, you could do something like the
+following. It does not need to verify that the has the same content that it did when
+imported, checking the size or even if HTTP HEAD succeeds is enough.
+Although if you can verify the content inexpensively, by all means do so.
CHECKPRESENTIMPORT)
key="$2"
typo
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index e62811063f..fe8de64306 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -126,7 +126,7 @@ well as using its User-Agent etc. Here's how to do that:
tmpfile="$@"
# now process "$tmpfile" and output IMPORTABLECONTENT
# and IMPORTABLECONTENTIDENTIFIER as before
- # (Left as an exersise for the reader.)
+ # (Left as an exercise for the reader.)
echo LISTIMPORTABLECONTENTS-SUCCESS
;;
DOWNLOAD-URL-FAILURE)
DOWNLOAD-URL
Also LISTIMPORTABLECONTENTS-FAILURE and renamed IMPORTABLECONTENTEND
to LISTIMPORTABLECONTENTS-SUCCESS for consistency.
Note that the temp file that DOWNLOAD-URL writes to is scoped to the
lifetime of the protocol request that git-annex makes.
Also LISTIMPORTABLECONTENTS-FAILURE and renamed IMPORTABLECONTENTEND
to LISTIMPORTABLECONTENTS-SUCCESS for consistency.
Note that the temp file that DOWNLOAD-URL writes to is scoped to the
lifetime of the protocol request that git-annex makes.
diff --git a/CHANGELOG b/CHANGELOG
index 24658ba4be..131e564db2 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -10,6 +10,7 @@ git-annex (10.20260717) UNRELEASED; urgency=medium
Developers of external special remotes should consider if import makes
sense for them and add support.
* Support importtree=yes with rsync special remotes.
+ * Added DOWNLOAD-URL extension to the external special remote protocol.
* Avoid redundant hash verification after a hash verification fails.
* Fix build with time-1.15.
diff --git a/Remote/External.hs b/Remote/External.hs
index 4bacaa352b..28ccc382c1 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -34,6 +34,7 @@ import Remote.Helper.ExportImport
import Remote.Helper.ReadOnly
import Utility.Metered
import Utility.Hash
+import Utility.Tmp
import Types.Transfer
import Logs.PreferredContent.Raw
import Logs.RemoteState
@@ -597,12 +598,14 @@ listImportableContentsM external =
go c (Just (sz, loc)) (IMPORTABLECONTENTIDENTIFIER cid) =
Just $ return $ GetNextMessage $
go ((loc, (cid, sz)):c) Nothing
- go c _ IMPORTABLECONTENTEND =
+ go c _ LISTIMPORTABLECONTENTS_SUCCESS =
result $ Just $
ImportableContentsComplete $ ImportableContents
{ importableContents = c
, importableHistory = []
}
+ go _ _ (LISTIMPORTABLECONTENTS_FAILURE err) =
+ giveup err
go _ _ (DELEGATE ps) = Just $ do
delegate <- getDelegateRemote external ps
Result <$> listImportableContents (importActions delegate)
@@ -721,18 +724,24 @@ handleRequest' st external req mp responsehandler
where
go = do
sendMessage st req
- loop
- loop = receiveMessage st external responsehandler
- (\rreq -> Just $ handleRemoteRequest rreq >> loop)
- (\msg -> Just $ handleExceptionalMessage msg >> loop)
+ cleanupv <- liftIO $ atomically $ newTMVar []
+ loop cleanupv
+ `finally` cleanup cleanupv
+
+ loop cleanupv = receiveMessage st external responsehandler
+ (\rreq -> Just $ handleRemoteRequest cleanupv rreq >> loop cleanupv)
+ (\msg -> Just $ handleExceptionalMessage msg >> loop cleanupv)
+
+ cleanup cleanupv = liftIO $
+ sequence =<< atomically (takeTMVar cleanupv)
- handleRemoteRequest (PROGRESS bytesprocessed) =
+ handleRemoteRequest _ (PROGRESS bytesprocessed) =
maybe noop (\a -> liftIO $ a bytesprocessed) mp
- handleRemoteRequest (DIRHASH k) =
+ handleRemoteRequest _ (DIRHASH k) =
send $ VALUE $ fromOsPath $ hashDirMixed def k
- handleRemoteRequest (DIRHASH_LOWER k) =
+ handleRemoteRequest _ (DIRHASH_LOWER k) =
send $ VALUE $ fromOsPath $ hashDirLower def k
- handleRemoteRequest (SETCONFIG setting value) =
+ handleRemoteRequest _ (SETCONFIG setting value) =
liftIO $ atomically $ do
ParsedRemoteConfig m c <- takeTMVar (externalConfig st)
let !m' = M.insert
@@ -747,13 +756,13 @@ handleRequest' st external req mp responsehandler
f <- takeTMVar (externalConfigChanges st)
let !f' = M.insert (Accepted setting) (Accepted value) . f
putTMVar (externalConfigChanges st) f'
- handleRemoteRequest (GETCONFIG setting) = do
+ handleRemoteRequest _ (GETCONFIG setting) = do
value <- maybe "" fromProposedAccepted
. (M.lookup (Accepted setting))
. unparsedRemoteConfig
<$> liftIO (atomically $ readTMVar $ externalConfig st)
send $ VALUE value
- handleRemoteRequest (SETCREDS setting login password) = case (externalUUID external, externalGitConfig external) of
+ handleRemoteRequest _ (SETCREDS setting login password) = case (externalUUID external, externalGitConfig external) of
(Just u, Just gc) -> do
pc <- liftIO $ atomically $ takeTMVar (externalConfig st)
pc' <- setRemoteCredPair' pc encryptionAlreadySetup gc
@@ -769,56 +778,78 @@ handleRequest' st external req mp responsehandler
let !f' = M.union configchanges . f
putTMVar (externalConfigChanges st) f'
_ -> senderror "cannot send SETCREDS here"
- handleRemoteRequest (GETCREDS setting) = case (externalUUID external, externalGitConfig external) of
+ handleRemoteRequest _ (GETCREDS setting) = case (externalUUID external, externalGitConfig external) of
(Just u, Just gc) -> do
c <- liftIO $ atomically $ readTMVar $ externalConfig st
creds <- fromMaybe ("", "") <$>
getRemoteCredPair c gc (credstorage setting u)
send $ CREDS (fst creds) (snd creds)
_ -> senderror "cannot send GETCREDS here"
- handleRemoteRequest GETUUID = case externalUUID external of
+ handleRemoteRequest _ GETUUID = case externalUUID external of
Just u -> send $ VALUE $ fromUUID u
Nothing -> senderror "cannot send GETUUID here"
- handleRemoteRequest GETGITDIR =
+ handleRemoteRequest _ GETGITDIR =
send . VALUE . fromOsPath =<< fromRepo Git.localGitDir
- handleRemoteRequest GETGITREMOTENAME =
+ handleRemoteRequest _ GETGITREMOTENAME =
case externalRemoteName external of
Just n -> send $ VALUE n
Nothing -> senderror "git remote name not known"
- handleRemoteRequest (SETWANTED expr) = case externalUUID external of
+ handleRemoteRequest _ (SETWANTED expr) = case externalUUID external of
Just u -> preferredContentSet u expr
Nothing -> senderror "cannot send SETWANTED here"
- handleRemoteRequest GETWANTED = case externalUUID external of
+ handleRemoteRequest _ GETWANTED = case externalUUID external of
Just u -> do
expr <- fromMaybe "" . M.lookup u
<$> preferredContentMapRaw
send $ VALUE expr
Nothing -> senderror "cannot send GETWANTED here"
- handleRemoteRequest (SETSTATE key state) =
+ handleRemoteRequest _ (SETSTATE key state) =
case externalRemoteStateHandle external of
Just h -> setRemoteState h key state
Nothing -> senderror "cannot send SETSTATE here"
- handleRemoteRequest (GETSTATE key) =
+ handleRemoteRequest _ (GETSTATE key) =
case externalRemoteStateHandle external of
Just h -> do
state <- fromMaybe ""
<$> getRemoteState h key
send $ VALUE state
Nothing -> senderror "cannot send GETSTATE here"
- handleRemoteRequest (SETURLPRESENT key url) =
+ handleRemoteRequest _ (SETURLPRESENT key url) =
setUrlPresent key url
- handleRemoteRequest (SETURLMISSING key url) =
+ handleRemoteRequest _ (SETURLMISSING key url) =
setUrlMissing key url
- handleRemoteRequest (SETURIPRESENT key uri) =
- withurl (SETURLPRESENT key) uri
- handleRemoteRequest (SETURIMISSING key uri) =
- withurl (SETURLMISSING key) uri
- handleRemoteRequest (GETURLS key prefix) = do
+ handleRemoteRequest cleanupv (SETURIPRESENT key uri) =
+ withurl cleanupv (SETURLPRESENT key) uri
+ handleRemoteRequest cleanupv (SETURIMISSING key uri) =
+ withurl cleanupv (SETURLMISSING key) uri
+ handleRemoteRequest _ (GETURLS key prefix) = do
mapM_ (send . VALUE) =<< getUrlsWithPrefix key prefix
send (VALUE "") -- end of list
- handleRemoteRequest (DEBUG msg) = fastDebug "Remote.External" msg
- handleRemoteRequest (INFO msg) = showInfo (UnquotedString msg)
- handleRemoteRequest (VERSION _) = senderror "too late to send VERSION"
+ handleRemoteRequest _ (DEBUG msg) = fastDebug "Remote.External" msg
+ handleRemoteRequest _ (INFO msg) = showInfo (UnquotedString msg)
+ handleRemoteRequest cleanupv (DOWNLOAD_URL url) = do
+ case externalGitConfig external of
+ Just gc -> do
+ (tmpf, h) <- liftIO $ do
+ tmpdir <- systemTmpDirectory
+ openTmpFileIn tmpdir (literalOsPath "url")
+ liftIO $ hClose h
+ liftIO $ atomically $ do
+ l <- takeTMVar cleanupv
+ putTMVar cleanupv (removeTmpFile tmpf:l)
+ res <- withUrlOptions (Just gc) $
+ downloadUrl' False UnknownSize
+ nullMeterUpdate Nothing [url]
+ tmpf
+ case res of
+ Right True ->
+ send $ DOWNLOAD_URL_SUCCESS (fromOsPath tmpf)
+ Left err ->
+ send $ DOWNLOAD_URL_FAILURE err
+ Right False ->
+ send $ DOWNLOAD_URL_FAILURE "download failed"
+ _ -> senderror "cannot send DOWNLOAD-URL here"
+ handleRemoteRequest _ (VERSION _) = senderror "too late to send VERSION"
handleExceptionalMessage (ERROR err) = giveup $ "external special remote error: " ++ err
@@ -832,8 +863,8 @@ handleRequest' st external req mp responsehandler
}
where
base = replace "/" "_" $ fromUUID u ++ "-" ++ setting
-
- withurl mk uri = handleRemoteRequest $ mk $
+
+ withurl cleanupv mk uri = handleRemoteRequest cleanupv $ mk $
setDownloader (show uri) OtherDownloader
sendMessage
diff --git a/Remote/External/Types.hs b/Remote/External/Types.hs
index fe276438ec..8d408a91a6 100644
--- a/Remote/External/Types.hs
(Diff truncated)
todo
diff --git a/doc/todo/fast_import_for_externals_with_importKey.mdwn b/doc/todo/fast_import_for_externals_with_importKey.mdwn new file mode 100644 index 0000000000..39120b3265 --- /dev/null +++ b/doc/todo/fast_import_for_externals_with_importKey.mdwn @@ -0,0 +1,18 @@ +External special remotes using importree=yes are not able to provide +`importKey`. If they were, when it's possible to query a hash from the API +the external is using, it would be able to import without downloading the +file. + +`git-annex-remote-internetarchive` is an example of a program that could +use this. + +`importKey` is currently only in ExportImportActions, +not in ImportActions so would first need to be added to it. Easy. + +The external would need to convert whatever ContentIdentifier it chooses to +use into a key. Eg if it uses a sha1, it would make a SHA1 key. IIRC +git-annex does not provide an interface to generate a key from a hash +although it does document the key format and the external could be expected +to generate it by itself. Or there could be a reply like +"IMPORTKEY-SUCCESS SHA1 $hash". +--[[Joey]]:
version for external importtree
diff --git a/CHANGELOG b/CHANGELOG
index c1404ef6f8..24658ba4be 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,4 +1,4 @@
-git-annex (10.20260625) UNRELEASED; urgency=medium
+git-annex (10.20260717) UNRELEASED; urgency=medium
* git-annex is guaranteed to not contain LLM generated code,
and will attempt to remain buildable with versions of dependencies
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index 30faeb1e64..85176101d0 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -17,6 +17,8 @@ Using it looks like this:
...
> git merge mobydick/master --allow-unrelated-histories
+Note that this needs git-annex 10.20260717 or newer.
+
Here we'll walk through that example program and explain what it does,
which is an easier start for you building you own than reading the
[[design/external_special_remote_protocol]] and
diff --git a/git-annex.cabal b/git-annex.cabal
index 4d81acf611..6d5c99d105 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -1,5 +1,5 @@
Name: git-annex
-Version: 10.20260624
+Version: 10.20260717
Cabal-Version: 1.12
License: AGPL-3
Maintainer: Joey Hess <id@joeyh.name>
improve
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index 564bbe475d..30faeb1e64 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -95,8 +95,8 @@ about an item, and `jq` is used to extract the information it needs.
echo IMPORTABLECONTENTEND
;;
-Don't worry too much about the details of the above example though. This is
-where you'll code up something entirely different.
+Don't worry too much about the parsing details of the above example though.
+This is where you'll code up something entirely different.
The main thing you need to determine is what content identifier to use
for a file in your remote. In the example, the Internet Archive has a handy
fix
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index e30f3f549b..564bbe475d 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -153,7 +153,7 @@ And if you need to hit the url yourself, you could do something like this:
;;
And that's all! Well, all except for some necessary boilerplate code
-to handle the rest of the protocol that you should need to modify,
+to handle the rest of the protocol that you should never need to modify,
feel free to just copy the rest of this.
IMPORTSUPPORTED)
update
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
index a2a4c4a294..e30f3f549b 100644
--- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -113,14 +113,14 @@ request is git-annex asking that a file be downloaded from the remote.
(This uses an `$importlocation` while we've not dealt with setting
yet, see below.)
-If anyone can download the url, perhaps with some HTTP basic auth,
-you can just tell git-annex what url to download and let it do
-the work. That's what the example does.
-
RETRIEVEIMPORT)
echo RETRIEVEIMPORT-URL "https://archive.org/download/$item/$importlocation"
;;
+If anyone can download the url, perhaps with some HTTP basic auth,
+you can just tell git-annex what url to download and let it do
+the work. That's what the example above does.
+
If you need to download the url yourself, you'll instead need something
like this:
@@ -131,8 +131,10 @@ like this:
;;
Similarly, the CHECKPRESENTIMPORT request is git-annex just checking
-if the file still appears to be present in the remote. (It does not need to
-verify that the has the same content.)
+if the file still appears to be present in the remote.
+(It does not need to verify that the has the same content that it did when
+imported, checking the size or even if HTTP HEAD succeeds is enough.
+Although if you can verify the content inexpensively, by all means do so.)
CHECKPRESENTIMPORT)
key="$2"
links
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn index 4b31af10ab..a2a4c4a294 100644 --- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn +++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn @@ -19,8 +19,8 @@ Using it looks like this: Here we'll walk through that example program and explain what it does, which is an easier start for you building you own than reading the -[[doc/design/external_special_remote_protocol]] and -its [[import_appendix|doc/design/external_special_remote_protocol/export_and_import_appendix]]. +[[design/external_special_remote_protocol]] and +its [[import_appendix|design/external_special_remote_protocol/export_and_import_appendix]]. Starting off, it's a portable shell script. But of course you could use some other language, probably with better results.
link
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn index bcb422d607..4b31af10ab 100644 --- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn +++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn @@ -3,7 +3,7 @@ importing a tree of files from it with [[git-annex-import]]? This is really simple to build with a short shell script. As an example, see [[special_remotes/external/git-annex-remote-internetarchive]]. -This treats an [[special_remote/Internet_Archive]] item as an +This treats an [[special_remotes/Internet_Archive]] item as an [[external_special_remote|special_remotes/external]]. Using it looks like this:
link
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn index 24c4dd5a8a..bcb422d607 100644 --- a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn +++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn @@ -1,5 +1,5 @@ Would you like to treat some web resource as a git-annex special remote, -importing a tree of files from it with [[git-annex-importree]]? This is +importing a tree of files from it with [[git-annex-import]]? This is really simple to build with a short shell script. As an example, see [[special_remotes/external/git-annex-remote-internetarchive]].
close
diff --git a/doc/todo/importree_only_external_remotes.mdwn b/doc/todo/importree_only_external_remotes.mdwn index 39af61b95f..b86f1a1761 100644 --- a/doc/todo/importree_only_external_remotes.mdwn +++ b/doc/todo/importree_only_external_remotes.mdwn @@ -58,3 +58,6 @@ only external remote in about 1 page of shell script: ;; esac done + +> [[done]], I didn't implement `DOWNLOAD-URL` though, being unsure if it is +> really an ergonomic benefit worth the complexity. --[[Joey]]
fix
diff --git a/doc/special_remotes/external.mdwn b/doc/special_remotes/external.mdwn index ebac41afea..372fff0184 100644 --- a/doc/special_remotes/external.mdwn +++ b/doc/special_remotes/external.mdwn @@ -23,7 +23,7 @@ It's not hard! It's *especially* easy to implement special remotes that are read-only imports of some web resource. See -[[tips/how_to_make_a_simple_importree_special_remote]]. +[[tips/how_to_make_a_simple_importtree_special_remote]]. ## libraries
tip
diff --git a/doc/special_remotes/external.mdwn b/doc/special_remotes/external.mdwn
index 94b2bad263..ebac41afea 100644
--- a/doc/special_remotes/external.mdwn
+++ b/doc/special_remotes/external.mdwn
@@ -15,12 +15,16 @@ It's not hard!
* When the user runs `git annex initremote foo type=external externaltype=$bar`,
it will use your program.
* See [[design/external_special_remote_protocol]] for what the program
- needs to do. There's an example at the end of this page.
+ needs to do. There are some examples below.
* If things don't seem to work, pass `--debug` and you'll see, amoung other
things, a transcript of git-annex's communication with your program.
* If you build a new special remote, please add it to the list
of [[special_remotes]].
+It's *especially* easy to implement special remotes that are read-only
+imports of some web resource. See
+[[tips/how_to_make_a_simple_importree_special_remote]].
+
## libraries
For Python, there is a [library by Lykos153](https://github.com/Lykos153/AnnexRemote)
@@ -29,11 +33,13 @@ that take care of all the protocol details.
## examples
-Here's an example of using an external special remote to add torrent
-support to git-annex: [[external/git-annex-remote-torrent]]
-
Here's a simple shell script example, which can easily be adapted
to run whatever commands you need. Or better, re-written in some better
-language of your choice.
+language of your choice: [[external/example.sh]]
-[[!inline pages="special_remotes/external/example.sh" feeds=no]]
+Here's an example Internet Archive item special remote,
+which can be used with [[git-annex-importtree]]:
+[[external/git-annex-remote-internetarchive]]
+
+Here's an example of using an external special remote to add torrent
+support to git-annex: [[external/git-annex-remote-torrent]]
diff --git a/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
new file mode 100644
index 0000000000..24c4dd5a8a
--- /dev/null
+++ b/doc/tips/how_to_make_a_simple_importtree_special_remote.mdwn
@@ -0,0 +1,175 @@
+Would you like to treat some web resource as a git-annex special remote,
+importing a tree of files from it with [[git-annex-importree]]? This is
+really simple to build with a short shell script.
+
+As an example, see [[special_remotes/external/git-annex-remote-internetarchive]].
+This treats an [[special_remote/Internet_Archive]] item as an
+[[external_special_remote|special_remotes/external]].
+
+Using it looks like this:
+
+ > git-annex initremote mobydick type=external encryption=none \
+ externaltype=internetarchive importtree=yes item=moby_dick_librivox
+ > git-annex wanted mobydick include="*.mp3"
+ > git-annex import master --from mobydick
+ list mobydick ok
+ import mobydick mobydick_135_melville_64kb.mp3
+ ...
+ > git merge mobydick/master --allow-unrelated-histories
+
+Here we'll walk through that example program and explain what it does,
+which is an easier start for you building you own than reading the
+[[doc/design/external_special_remote_protocol]] and
+its [[import_appendix|doc/design/external_special_remote_protocol/export_and_import_appendix]].
+
+Starting off, it's a portable shell script. But of course you could use
+some other language, probably with better results.
+
+ #!/bin/sh
+ set -e
+
+This is using version 2 of the protocol, which we start off as follows.
+
+ echo VERSION 2
+
+Probably you'll want some configuration. In this case, it expects
+the user to provide an "item=" parameter, which is the Internet Archive
+item's name. The `GETCONFIG` message can be used to query for such parameters,
+and git-annex will respond with "VALUE whatever". This is a shell
+function since it's needed in a couple of different places below.
+
+ getconfig () {
+ echo GETCONFIG item
+ read resp
+ item=$(echo "$resp" | sed 's/^VALUE \?//')
+ }
+
+Now the meat of the program, which is just a loop that reads lines from
+stdin and dispatches on requests that git-annex sends to it.
+
+ while read line; do
+ set -- $line
+ case "$1" in
+
+The INITREMOTE request is sent when `git-annex initremote` is run,
+and it's where you validate any configuration, like the "item" parameter.
+
+ INITREMOTE)
+ getconfig
+ if [ -z "$item" ]; then
+ echo INITREMOTE-FAILURE "Specify item="
+ else
+ echo INITREMOTE-SUCCESS
+ fi
+ ;;
+
+The PREPARE request is sent when git-annex is starting to use the
+remote, and is just a place to do any one-time startup actions.
+Here it's used to call the `getconfig` function which sets
+"$item", which is used below.
+
+ PREPARE)
+ getconfig
+ echo PREPARE-SUCCESS
+ ;;
+
+Now the main event, the LISTIMPORTABLECONTENTS request is
+sent when git-annex wants to import from your remote.
+It outputs 2 lines for each file in the remote,
+followed by IMPORTABLECONTENTEND.
+
+In the example, the Internet Archive has a JSON API to get information
+about an item, and `jq` is used to extract the information it needs.
+
+ LISTIMPORTABLECONTENTS)
+ curl --silent "https://archive.org/metadata/$item/files" | \
+ jq -r '.result[] | (.name, .size, .sha1)' | \
+ while read name; do
+ read size
+ read sha1
+ if [ "$sha1" != null ]; then
+ echo IMPORTABLECONTENT "$size" "$name"
+ echo IMPORTABLECONTENTIDENTIFIER "$sha1"
+ fi
+ done
+ echo IMPORTABLECONTENTEND
+ ;;
+
+Don't worry too much about the details of the above example though. This is
+where you'll code up something entirely different.
+
+The main thing you need to determine is what content identifier to use
+for a file in your remote. In the example, the Internet Archive has a handy
+sha1 available, which is a perfect content identifier. If you're not so
+lucky, you can use things like the size+mtime of the file as the content
+identifier. It just needs to be unique enough to uniquely identify a
+particular version of a file in the web resource that you're treating as a
+remote. It's ok if it turns out to not be as unique as you think, because
+git-annex of course hashes the files itself, and will notice if a hash
+changed.
+
+Continuing on, and nearing the home stretch, the RETRIEVEIMPORT
+request is git-annex asking that a file be downloaded from the remote.
+(This uses an `$importlocation` while we've not dealt with setting
+yet, see below.)
+
+If anyone can download the url, perhaps with some HTTP basic auth,
+you can just tell git-annex what url to download and let it do
+the work. That's what the example does.
+
+ RETRIEVEIMPORT)
+ echo RETRIEVEIMPORT-URL "https://archive.org/download/$item/$importlocation"
+ ;;
+
+If you need to download the url yourself, you'll instead need something
+like this:
+
+ RETRIEVEIMPORT)
+ shift 1
+ file="$@"
+ curl -o "$file" "https://archive.org/download/$item/$importlocation"
+ ;;
+
+Similarly, the CHECKPRESENTIMPORT request is git-annex just checking
+if the file still appears to be present in the remote. (It does not need to
+verify that the has the same content.)
+
+ CHECKPRESENTIMPORT)
+ key="$2"
+ echo CHECKPRESENT-URL "$key" "https://archive.org/download/$item//$importlocation"
+ ;;
+
+And if you need to hit the url yourself, you could do something like this:
+
+ CHECKPRESENTIMPORT)
+ key="$2"
+ if curl --output /dev/null --silent --head --fail "https://archive.org/download/$item/$importlocation"; then
+ echo CHECKPRESENTIMPORT-SUCCESS
+ else
+ echo CHECKPRESENTIMPORT-FAILURE
+ fi
+ ;;
+
+And that's all! Well, all except for some necessary boilerplate code
(Diff truncated)
document how to limit
diff --git a/doc/special_remotes/external/git-annex-remote-internetarchive b/doc/special_remotes/external/git-annex-remote-internetarchive
index 241688aaf9..4269d75a3a 100755
--- a/doc/special_remotes/external/git-annex-remote-internetarchive
+++ b/doc/special_remotes/external/git-annex-remote-internetarchive
@@ -10,6 +10,9 @@
# git-annex import master --from iaitem
# git-annex merge iaitem/master
#
+# To limit the files that are imported to a particular extension use eg:
+# git-annex wanted iaitem include="*.mp3"
+#
# Copyright 2013-2026 Joey Hess; licenced under the GNU GPL version 3 or higher.
set -e
getconfig () {
add git-annex-remote-internetarchive
diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn
index 01b4567e0d..cb7991f745 100644
--- a/doc/special_remotes.mdwn
+++ b/doc/special_remotes.mdwn
@@ -49,6 +49,7 @@ Here are specific instructions for using git-annex with various services:
* [[hubiC|tips/using_Hubic]]
* [[IMAP|forum/special_remote_for_IMAP]]
* [Internxt Drive](https://pypi.org/project/git-annex-remote-internxt/)
+* [[Internet_Archive]]
* [[tips/Internet_Archive_via_S3]]
* [[ipfs]]
* [[Jottacloud|rclone]]
diff --git a/doc/special_remotes/Internet_Archive.mdwn b/doc/special_remotes/Internet_Archive.mdwn
new file mode 100644
index 0000000000..9ede028a0b
--- /dev/null
+++ b/doc/special_remotes/Internet_Archive.mdwn
@@ -0,0 +1,7 @@
+[The Internet Archive](http://www.archive.org/) can be used as a special
+remote in two different ways:
+
+1. To upload files to it, see
+ [[tips/[The Internet Archive](http://www.archive.org/)]]
+2. To [[import a tree of files|git-annex-import]] from an existing
+ Internet Archive item, use [[external/git-annex-remote-internetarchive]].
diff --git a/doc/special_remotes/external/git-annex-remote-internetarchive b/doc/special_remotes/external/git-annex-remote-internetarchive
new file mode 100755
index 0000000000..241688aaf9
--- /dev/null
+++ b/doc/special_remotes/external/git-annex-remote-internetarchive
@@ -0,0 +1,73 @@
+#!/bin/sh
+# This makes an Internet Archive item into a git-annex special remote.
+#
+# Dependencies: curl, jq
+#
+# Install in PATH, and then use as follows:
+# git-annex initremote iaitem type=external encryption=none \
+# externaltype=internetarchive importtree=yes \
+# item=<item name here>
+# git-annex import master --from iaitem
+# git-annex merge iaitem/master
+#
+# Copyright 2013-2026 Joey Hess; licenced under the GNU GPL version 3 or higher.
+set -e
+getconfig () {
+ echo GETCONFIG item
+ read resp
+ item=$(echo "$resp" | sed 's/^VALUE \?//')
+}
+
+echo VERSION 2
+while read line; do
+ set -- $line
+ case "$1" in
+ INITREMOTE)
+ getconfig
+ if [ -z "$item" ]; then
+ echo INITREMOTE-FAILURE "Specify item="
+ else
+ echo INITREMOTE-SUCCESS
+ fi
+ ;;
+ PREPARE)
+ getconfig
+ echo PREPARE-SUCCESS
+ ;;
+ LISTIMPORTABLECONTENTS)
+ curl --silent "https://archive.org/metadata/$item/files" | \
+ jq -r '.result[] | (.name, .size, .sha1)' | \
+ while read name; do
+ read size
+ read sha1
+ if [ "$sha1" != null ]; then
+ echo IMPORTABLECONTENT "$size" "$name"
+ echo IMPORTABLECONTENTIDENTIFIER "$sha1"
+ fi
+ done
+ echo IMPORTABLECONTENTEND
+ ;;
+ RETRIEVEIMPORT)
+ echo RETRIEVEIMPORT-URL "https://archive.org/download/$item/$importlocation"
+ ;;
+ CHECKPRESENTIMPORT)
+ key="$2"
+ echo CHECKPRESENT-URL "$key" "https://archive.org/download/$item//$importlocation"
+ ;;
+ # Below is all the boilerplate needed to make this work as an
+ # importtree-only remote.
+ IMPORTSUPPORTED)
+ echo IMPORTREQUIRED
+ ;;
+ IMPORT)
+ shift 1
+ importlocation="$@"
+ ;;
+ INITREMOTE)
+ echo INITREMOTE-SUCCESS
+ ;;
+ *)
+ echo UNSUPPORTED-REQUEST
+ ;;
+ esac
+done
diff --git a/doc/tips/Internet_Archive_via_S3.mdwn b/doc/tips/Internet_Archive_via_S3.mdwn
index b982588e4c..9a21939acd 100644
--- a/doc/tips/Internet_Archive_via_S3.mdwn
+++ b/doc/tips/Internet_Archive_via_S3.mdwn
@@ -1,8 +1,10 @@
[The Internet Archive](http://www.archive.org/) allows members to upload
-collections using an Amazon S3
+items using an Amazon S3
[compatible API](http://www.archive.org/help/abouts3.txt), and this can
be used with git-annex's [[special_remotes/S3]] support.
+(If you only want to import files from an Internet Archive item,
+
So, you can locally archive things with git-annex, define remotes that
correspond to "items" at the Internet Archive, and use git-annex to upload
your files to there. Of course, your use of the Internet Archive must
implement IMPORTREQUIRED
This allows for some very small external special remote programs that
import from somewhere on the web, and don't need any of the rest of the
external special remote protocol to be implemented.
This allows for some very small external special remote programs that
import from somewhere on the web, and don't need any of the rest of the
external special remote protocol to be implemented.
diff --git a/Annex/SpecialRemote/Config.hs b/Annex/SpecialRemote/Config.hs
index e21a9c3e39..9d1ab13c7e 100644
--- a/Annex/SpecialRemote/Config.hs
+++ b/Annex/SpecialRemote/Config.hs
@@ -288,6 +288,10 @@ yesNoParser f mdef fd = genParser yesno f mdef fd
yesno "no" = Just False
yesno _ = Nothing
+yesNoGenerator :: Bool -> String
+yesNoGenerator True = "yes"
+yesNoGenerator False = "no"
+
trueFalseParser :: RemoteConfigField -> Maybe Bool -> FieldDesc -> RemoteConfigFieldParser
trueFalseParser f mdef fd = genParser trueFalseParser' f mdef fd
(Just (ValueDesc "true or false"))
diff --git a/Remote/External.hs b/Remote/External.hs
index fe9f68cd06..4bacaa352b 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -110,10 +110,10 @@ gen rt externalprogram r u rc gc rs
Annex.addCleanupAction (RemoteCleanup u) $ stopExternal external
cst <- getCost external r gc c
exportsupported <- if exportTree c
- then checkExportSupported external
+ then isExportSupported' <$> checkExportSupported (Just external)
else return False
importsupported <- if importTree c
- then checkImportSupported external
+ then isImportSupported' <$> checkImportSupported (Just external)
else return False
let exportactions = if exportsupported
then ExportActions
@@ -267,37 +267,42 @@ externalSetup externalprogram setgitconfig ss mu remotename _ c gc = do
checkSupportedWith
:: Maybe ExternalProgram
- -> (External -> Annex Bool)
+ -> (Maybe External -> Annex a)
-> ParsedRemoteConfig
-> RemoteGitConfig
- -> Annex Bool
+ -> Annex a
checkSupportedWith Nothing checker c gc = do
let externaltype = fromMaybe (giveup "Specify externaltype=") $
remoteAnnexExternalType gc <|> getRemoteConfigValue externaltypeField c
if externaltype == "readonly"
- then return False
+ then checker Nothing
else checkSupportedWith (Just (ExternalType externaltype)) checker c gc
checkSupportedWith (Just externalprogram) checker c gc =
- checker
+ checker . Just
=<< newExternal externalprogram Nothing c (Just gc) Nothing Nothing
-checkExportSupported :: External -> Annex Bool
-checkExportSupported external = go `catchNonAsync` (const (return False))
+checkExportSupported :: Maybe External -> Annex ExportSupported
+checkExportSupported (Just external) = go
+ `catchNonAsync` (const (return (ExportSupported False)))
where
go = handleRequest external EXPORTSUPPORTED Nothing $ \resp -> case resp of
- EXPORTSUPPORTED_SUCCESS -> result True
- EXPORTSUPPORTED_FAILURE -> result False
- UNSUPPORTED_REQUEST -> result False
+ EXPORTSUPPORTED_SUCCESS -> result (ExportSupported True)
+ EXPORTSUPPORTED_FAILURE -> result (ExportSupported False)
+ UNSUPPORTED_REQUEST -> result (ExportSupported False)
_ -> Nothing
+checkExportSupported Nothing = return (ExportSupported False)
-checkImportSupported :: External -> Annex Bool
-checkImportSupported external = go `catchNonAsync` (const (return False))
+checkImportSupported :: Maybe External -> Annex ImportSupported
+checkImportSupported (Just external) = go
+ `catchNonAsync` (const (return (ImportSupported False)))
where
go = handleRequest external IMPORTSUPPORTED Nothing $ \resp -> case resp of
- IMPORTSUPPORTED_SUCCESS -> result True
- IMPORTSUPPORTED_FAILURE -> result False
- UNSUPPORTED_REQUEST -> result False
+ IMPORTSUPPORTED_SUCCESS -> result (ImportSupported True)
+ IMPORTSUPPORTED_FAILURE -> result (ImportSupported False)
+ IMPORTREQUIRED -> result ImportRequired
+ UNSUPPORTED_REQUEST -> result (ImportSupported False)
_ -> Nothing
+checkImportSupported Nothing = return (ImportSupported False)
storeKeyM :: External -> Storer
storeKeyM external = fileStorer $ \k f p ->
diff --git a/Remote/External/Types.hs b/Remote/External/Types.hs
index 4008769aa3..fe276438ec 100644
--- a/Remote/External/Types.hs
+++ b/Remote/External/Types.hs
@@ -310,6 +310,7 @@ data Response
| RENAMEEXPORT_FAILURE Key
| IMPORTSUPPORTED_SUCCESS
| IMPORTSUPPORTED_FAILURE
+ | IMPORTREQUIRED
| IMPORTABLECONTENT Size FilePath
| IMPORTABLECONTENTIDENTIFIER ContentIdentifier
| IMPORTABLECONTENTEND
@@ -359,6 +360,7 @@ instance Proto.Receivable Response where
parseCommand "RENAMEEXPORT-FAILURE" = Proto.parse1 RENAMEEXPORT_FAILURE
parseCommand "IMPORTSUPPORTED-SUCCESS" = Proto.parse0 IMPORTSUPPORTED_SUCCESS
parseCommand "IMPORTSUPPORTED-FAILURE" = Proto.parse0 IMPORTSUPPORTED_FAILURE
+ parseCommand "IMPORTREQUIRED" = Proto.parse0 IMPORTREQUIRED
parseCommand "IMPORTABLECONTENT" = Proto.parse2 IMPORTABLECONTENT
parseCommand "IMPORTABLECONTENTIDENTIFIER" = Proto.parse1 IMPORTABLECONTENTIDENTIFIER
parseCommand "IMPORTABLECONTENTEND" = Proto.parse0 IMPORTABLECONTENTEND
diff --git a/Remote/Helper/ExportImport.hs b/Remote/Helper/ExportImport.hs
index 0192911da2..23b2ff7a97 100644
--- a/Remote/Helper/ExportImport.hs
+++ b/Remote/Helper/ExportImport.hs
@@ -33,8 +33,8 @@ import Control.Concurrent.STM
class HasExportUnsupported a where
exportUnsupported :: a
-instance HasExportUnsupported (ParsedRemoteConfig -> RemoteGitConfig -> Annex Bool) where
- exportUnsupported = \_ _ -> return False
+instance HasExportUnsupported (ParsedRemoteConfig -> RemoteGitConfig -> Annex ExportSupported) where
+ exportUnsupported = \_ _ -> return (ExportSupported False)
instance HasExportUnsupported (ExportActions Annex) where
exportUnsupported = ExportActions
@@ -52,8 +52,8 @@ instance HasExportUnsupported (ExportActions Annex) where
class HasImportUnsupported a where
importUnsupported :: a
-instance HasImportUnsupported (ParsedRemoteConfig -> RemoteGitConfig -> Annex Bool) where
- importUnsupported = \_ _ -> return False
+instance HasImportUnsupported (ParsedRemoteConfig -> RemoteGitConfig -> Annex ImportSupported) where
+ importUnsupported = \_ _ -> return (ImportSupported False)
instance HasImportUnsupported (ImportActions Annex) where
importUnsupported = ImportActions
@@ -84,11 +84,11 @@ instance HasExportImportUnsupported (ExportImportActions Annex) where
where
nope = giveup "import combined with export not supported"
-exportIsSupported :: ParsedRemoteConfig -> RemoteGitConfig -> Annex Bool
-exportIsSupported = \_ _ -> return True
+exportIsSupported :: ParsedRemoteConfig -> RemoteGitConfig -> Annex ExportSupported
+exportIsSupported = \_ _ -> return (ExportSupported True)
-importIsSupported :: ParsedRemoteConfig -> RemoteGitConfig -> Annex Bool
-importIsSupported = \_ _ -> return True
+importIsSupported :: ParsedRemoteConfig -> RemoteGitConfig -> Annex ImportSupported
+importIsSupported = \_ _ -> return (ImportSupported True)
exportImportIsSupported :: ParsedRemoteConfig -> RemoteGitConfig -> Annex Bool
exportImportIsSupported = \_ _ -> return True
@@ -103,7 +103,7 @@ adjustExportImportRemoteType rt = rt { setup = setup' }
=<< configParser rt c
let checkconfig supported configured configfield cont =
let allowed =
- ( supported rt pc gc
+ ( pure supported
<||> exportImportSupported rt pc gc)
<&&> pure (not (thirdPartyPopulated rt))
in ifM allowed
@@ -128,8 +128,13 @@ adjustExportImportRemoteType rt = rt { setup = setup' }
++ fromProposedAccepted exportTreeField
)
| otherwise = cont
- checkconfig exportSupported exportTree exportTreeField $
- checkconfig importSupported importTree importTreeField $
+ exportsupported <- exportSupported rt pc gc
+ importsupported <- importSupported rt pc gc
+ when (isImportRequired importsupported && not (importTree pc)) $
+ giveup $ "This special remote must be configured with " ++
+ fromProposedAccepted importTreeField ++ "=" ++ yesNoGenerator True
+ checkconfig (isExportSupported' exportsupported) exportTree exportTreeField $
+ checkconfig (isImportSupported' importsupported) importTree importTreeField $
checkexportimport $
setup rt st mu remotename cp c gc
@@ -148,7 +153,7 @@ adjustExportImport r rs = do
-- Use ExportImportActions even when
-- not configured with exporttree=yes,
-- when it's supported, since it
- -- handled content identifiers more
+ -- handles content identifiers more
-- strongly than ImportActions does.
( importconfigured
-- thirdPartyPopulated is handled using
diff --git a/Types/Remote.hs b/Types/Remote.hs
index 0756ee827b..17e69b9d2d 100644
--- a/Types/Remote.hs
+++ b/Types/Remote.hs
@@ -20,8 +20,13 @@ module Types.Remote
, Verification(..)
, unVerified
, RetrievalSecurityPolicy(..)
+ , ExportSupported(..)
, isExportSupported
+ , isExportSupported'
+ , ImportSupported(..)
, isImportSupported
+ , isImportSupported'
+ , isImportRequired
(Diff truncated)
allow using -URL responses without negotiating a protocol extension
Versions of git-annex that support the import appendix will always
support those responses. This will allow for simpler and smaller
external programs for import.
Versions of git-annex that support the import appendix will always
support those responses. This will allow for simpler and smaller
external programs for import.
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 3c8ba15539..4946dd893c 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -207,9 +207,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates the retrieve failed.
* `RETRIEVEIMPORT-URL Url`
Rather than the special remote downloading content from an url itself,
- this lets it offload that work to git-annex. This response is a
- protocol extension; it's only safe to send it to git-annex after
- it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
+ this lets it offload that work to git-annex.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -227,9 +225,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
present in the remote. (Perhaps the remote cannot be contacted.)
* `CHECKPRESENT-URL Key Url`
Rather than the special remote checking an url itself,
- this lets it offload that work to git-annex. This response is a protocol
- extension; it's only safe to send it to git-annex after it sent an
- `EXTENSIONS` that included `CHECKPRESENT-URL`.
+ this lets it offload that work to git-annex.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
diff --git a/doc/todo/importree_only_external_remotes.mdwn b/doc/todo/importree_only_external_remotes.mdwn
index 233ab51b30..eb518fb382 100644
--- a/doc/todo/importree_only_external_remotes.mdwn
+++ b/doc/todo/importree_only_external_remotes.mdwn
@@ -33,12 +33,6 @@ only external remote in about 1 page of shell script:
PREPARE)
echo PREPARE-SUCCESS
;;
- EXTENSIONS)
- # not strictly necessary because git-annex versions that support
- # importtree should support these extensions.
- # Need to verify if it works w/o this.
- echo EXTENSIONS TRANSFER-RETRIEVE-URL CHECKPRESENT-URL
- ;;
IMPORTSUPPORTED)
echo IMPORTSUPPORTED-REQUIRED
;;
todo
diff --git a/doc/todo/importree_only_external_remotes.mdwn b/doc/todo/importree_only_external_remotes.mdwn new file mode 100644 index 0000000000..233ab51b30 --- /dev/null +++ b/doc/todo/importree_only_external_remotes.mdwn @@ -0,0 +1,66 @@ +It could be simpler to write an external special remote that +imports a tree from some web resource that is tree-like. + +Conceptually, all such a special +remote needs to do is access an url that lists the files in the data +store, parse it, and then use `RETRIEVEIMPORT-URL` and `CHECKPRESENT-URL` +to tell git-annex what urls to use. + +But currently, the whole external special remote protocol needs to be +implemented including storing keys. Even if there is no way to write to the +data store, and even if key-value storage is not appropriate. + +If the external special remote had a way to indicate that it only supported +importtree, it would not even be necessary to configure it with +importtree=yes. And it could skip implementing the rest of the protocol. +This could be eg an `IMPORTSUPPORTED-REQUIRED` response. + +Also it would be good to have a way for `LISTIMPORTABLECONTENTS` +to outsource an url download to git-annex. This might take the form +of a special remote message `DOWNLOAD-URL` that downloads a given url +to a file and responds with the filename. (The file would be deleted +once the current request is done.) + +Putting all this together, it should be possible to write an importtree +only external remote in about 1 page of shell script: + + #!/bin/sh + set -e + echo VERSION 2 + while read line; do + set -- $line + case "$1" in + PREPARE) + echo PREPARE-SUCCESS + ;; + EXTENSIONS) + # not strictly necessary because git-annex versions that support + # importtree should support these extensions. + # Need to verify if it works w/o this. + echo EXTENSIONS TRANSFER-RETRIEVE-URL CHECKPRESENT-URL + ;; + IMPORTSUPPORTED) + echo IMPORTSUPPORTED-REQUIRED + ;; + LISTIMPORTABLECONTENTS) + echo DOWNLOAD-URL http://example.com/ + read file + # parse $file here and output IMPORTABLECONTENT and IMPORTABLECONTENTIDENTIFIER lines + echo IMPORTABLECONTENTEND + ;; + IMPORT) + shift 1 + importlocation="$@" + ;; + RETRIEVEIMPORT) + echo RETRIEVEIMPORT-URL "http://example.com/$importlocation" + ;; + CHECKPRESENTIMPORT) + key="$2" + echo CHECKPRESENT-URL "$key" "http://example.com/$importlocation" + ;; + *) + echo UNSUPPORTED-REQUEST + ;; + esac + done
cleanu
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn index 4659f45eb2..3c8ba15539 100644 --- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn +++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn @@ -25,7 +25,7 @@ remote. ## export interface git-annex will use this when the special remote is initialized with -exporttree=yes and without importtree=yes and indicates that it supports +exporttree=yes and indicates that it supports exports. These are requests git-annex sends to the special remote program.
wording
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn index 3ebf7d2cbd..4659f45eb2 100644 --- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn +++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn @@ -4,7 +4,7 @@ Some special remotes interface to a key/value datastore using keys that are eg hashes, and it won't make sense for them to implement any of this. When a special remote interfaces with something that looks like a directory -of files, it can either or both of the interfaces described here. +of files, it can implement either or both of the interfaces described here. The export interface can be implemented to allow the [[git-annex-export]] command to be used to store file trees on the special
close one todo and open another
diff --git a/doc/todo/importtree_only_remotes.mdwn b/doc/todo/importtree_only_remotes.mdwn index 5d518feab1..8724320215 100644 --- a/doc/todo/importtree_only_remotes.mdwn +++ b/doc/todo/importtree_only_remotes.mdwn @@ -86,3 +86,5 @@ Or by complicating Remote.Helper.ExportImport further.. [[!tag confirmed]] [[!tag projects/ICE4]] + +> [[done]] --[[Joey]] diff --git a/doc/todo/importtree_only_remotes/comment_5_3ce0fa1729d716db94bbd2458c319fd2._comment b/doc/todo/importtree_only_remotes/comment_5_3ce0fa1729d716db94bbd2458c319fd2._comment new file mode 100644 index 0000000000..a65ce54b4a --- /dev/null +++ b/doc/todo/importtree_only_remotes/comment_5_3ce0fa1729d716db94bbd2458c319fd2._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-07-16T16:19:19Z" + content=""" +The external special remote interface is implemented, so I am closing this. + +See new [[todo/make_more_special_remotes_support_importtree]] +"""]] diff --git a/doc/todo/make_more_special_remotes_support_importtree.mdwn b/doc/todo/make_more_special_remotes_support_importtree.mdwn new file mode 100644 index 0000000000..df83ef9b92 --- /dev/null +++ b/doc/todo/make_more_special_remotes_support_importtree.mdwn @@ -0,0 +1,18 @@ +Now that importtree=yes is supported for special remotes without also +needing to implement exporttree=yes, it should be possible to make a lot +more special remotes support tree importing. + +These built-in ones could support it: + +* webdav +* S3 + + importtree=yes is only supported when versioning is enabled + (or when forced when version is not enabled). + + It would be possible to support importtree=yes without exporttree=yes + for any S3 bucket with no need of forcing. + +Lots of external special remotes could also probably support it. +That will be left up to their maintainers, but it would make sense to ping +the maintainers.
working importtree from external special remotes
diff --git a/Remote/External.hs b/Remote/External.hs
index 4f32121637..5266e74b02 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -525,11 +525,6 @@ checkPresentExportImport request srequest delegateaction handlereq external gc k
Left $ srequest ++ " not implemented by external special remote"
_ -> Nothing
-listImportableContentsM
- :: External
- -> Annex (Maybe (ImportableContentsChunkable a (ContentIdentifier, ByteSize)))
-listImportableContentsM = undefined
-
removeExportM :: External -> Key -> ExportLocation -> Annex ()
removeExportM external k loc = either giveup return =<< go
where
@@ -585,6 +580,32 @@ renameExportM external k src dest = either giveup return =<< go
_ -> Nothing
req sk = RENAMEEXPORT sk dest
+listImportableContentsM
+ :: External
+ -> Annex (Maybe (ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)))
+listImportableContentsM external =
+ handleRequest external LISTIMPORTABLECONTENTS Nothing
+ (go [] Nothing)
+ where
+ go c _ (IMPORTABLECONTENT sz loc) =
+ let loc' = mkImportLocation (toOsPath loc)
+ in Just $ return $ GetNextMessage $
+ go c (Just (sz, loc'))
+ go c (Just (sz, loc)) (IMPORTABLECONTENTIDENTIFIER cid) =
+ Just $ return $ GetNextMessage $
+ go ((loc, (cid, sz)):c) Nothing
+ go c _ IMPORTABLECONTENTEND =
+ result $ Just $
+ ImportableContentsComplete $ ImportableContents
+ { importableContents = c
+ , importableHistory = []
+ }
+ go _ _ (DELEGATE ps) = Just $ do
+ delegate <- getDelegateRemote external ps
+ Result <$> listImportableContents (importActions delegate)
+ go _ _ UNSUPPORTED_REQUEST = result Nothing
+ go _ _ _ = Nothing
+
{- Sends a Request to the external remote, and waits for it to generate
- a Response. That is fed into the responsehandler, which should return
- the action to run for it (or Nothing if there's a protocol error).
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 0cc8cbedfa..3ebf7d2cbd 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -255,24 +255,24 @@ git-annex asks for a list of files stored in the special remote:
Which responds:
- CONTENT 100 foo
- CONTENTIDENTIFIER 100 48511528411921470
- CONTENT 200 bar
- CONTENTIDENTIFIER 200 48511528411963410
- END
+ IMPORTABLECONTENT 100 foo
+ IMPORTABLECONTENTIDENTIFIER 100 48511528411921470
+ IMPORTABLECONTENT 200 bar
+ IMPORTABLECONTENTIDENTIFIER 200 48511528411963410
+ IMPORTABLECONTENTEND
Next git-annex asks for the content of a file to be retrieved.
IMPORT foo
- RETRIEVEIMPORTED tmpfile
+ RETRIEVEIMPORT tmpfile
If the file no longer exists, the response would be:
- RETRIEVE-FAILURE file not found
+ RETRIEVEIMPORT-FAILURE file not found
If the file exists, the response would be:
- RETRIEVE-SUCCESS
+ RETRIEVEIMPORT-SUCCESS
What if the file exists but has been modified in the time
since LISTIMPORTABLECONTENTS? The external special remote
diff --git a/doc/special_remotes/external/example.sh b/doc/special_remotes/external/example.sh
index e3ddee60cd..c0400e178b 100755
--- a/doc/special_remotes/external/example.sh
+++ b/doc/special_remotes/external/example.sh
@@ -5,7 +5,7 @@
#
# Install in PATH as git-annex-remote-directory
#
-# Copyright 2013 Joey Hess; licenced under the GNU GPL version 3 or higher.
+# Copyright 2013-2026 Joey Hess; licenced under the GNU GPL version 3 or higher.
set -e
@@ -90,7 +90,7 @@ dostore () {
&& runcmd mv -f "$tmp" "$loc"; then
echo TRANSFER-SUCCESS STORE "$key"
else
- echo TRANSFER-FAILURE STORE "$key"
+ echo TRANSFER-FAILURE STORE "$key" "failed to write to file"
fi
rmdir "$mydirectory/tmp"
}
@@ -105,10 +105,10 @@ doretrieve () {
if runcmd cp "$loc" "$file"; then
echo TRANSFER-SUCCESS RETRIEVE "$key"
else
- echo TRANSFER-FAILURE RETRIEVE "$key"
+ echo TRANSFER-FAILURE RETRIEVE "$key" "failed to read file"
fi
else
- echo TRANSFER-FAILURE RETRIEVE "$key"
+ echo TRANSFER-FAILURE RETRIEVE "$key" "file does not exist"
fi
}
@@ -142,7 +142,7 @@ doremove () {
if runcmd rm -f "$loc"; then
echo REMOVE-SUCCESS "$key"
else
- echo REMOVE-FAILURE "$key"
+ echo REMOVE-FAILURE "$key" "file removal failed"
fi
else
echo REMOVE-SUCCESS "$key"
@@ -242,6 +242,15 @@ while read line; do
# that are required to be supported, so it's fine
# to respond to any others with UNSUPPORTED-REQUEST.
+ # This is optional, only provided as an example.
+ GETINFO)
+ echo INFOFIELD "repository location"
+ echo INFOVALUE "$mydirectory"
+ echo INFOFIELD "login"
+ echo INFOVALUE "$MYLOGIN"
+ echo INFOEND
+ ;;
+
# Let's also support exporting...
EXPORTSUPPORTED)
echo EXPORTSUPPORTED-SUCCESS
@@ -296,14 +305,37 @@ while read line; do
echo RENAMEEXPORT-FAILURE "$key"
fi
;;
-
- # This is optional, only provided as an example.
- GETINFO)
- echo INFOFIELD "repository location"
- echo INFOVALUE "$mydirectory"
- echo INFOFIELD "login"
- echo INFOVALUE "$MYLOGIN"
- echo INFOEND
+
+ # Let's also support importing...
+ IMPORTSUPPORTED)
+ echo IMPORTSUPPORTED-SUCCESS
+ ;;
+ LISTIMPORTABLECONTENTS)
+ find "$mydirectory" -type f -printf 'IMPORTABLECONTENT %s %P\nIMPORTABLECONTENTIDENTIFIER %s %T@\n'
+ echo IMPORTABLECONTENTEND
+ ;;
+ IMPORT)
+ shift 1
+ importlocation="$mydirectory/$@"
+ # No response to this one; this value is used below.
+ ;;
+ RETRIEVEIMPORT)
+ shift 1
+ file="$@"
+ # XXX when easy to do, send PROGRESS while transferring the file
+ if [ -e "$importlocation" ]; then
+ if runcmd cp "$importlocation" "$file"; then
+ echo RETRIEVEIMPORT-SUCCESS
+ else
+ echo RETRIEVEIMPORT-FAILURE "failed to read file"
+ fi
+ else
+ echo RETRIEVEIMPORT-FAILURE "does not exist"
+ fi
+ ;;
+ CHECKPRESENTIMPORT)
+ key="$2"
+ docheckpresent "$key" "$importlocation"
;;
*)
follow renaming
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 1eeaad7190..0cc8cbedfa 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -181,7 +181,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
in bytes. The Name is the name of the file on the remote,
in the form of a relative path, and may contain path separators,
whitespace, and other special characters.
- Always followed by `CONTENTIDENTIFIER`.
+ Always followed by `IMPORTABLECONTENTIDENTIFIER`.
* `IMPORTABLECONTENTIDENTIFIER ContentIdentifier`
Provide the ContentIdentifier for the previous `CONTENT`.
* `IMPORTABLECONTENTEND`
more consistent protocol for import
Note that CHECKPRESENT-URL already worked since code was reused from CHECKPRESENTEXPORT
Note that CHECKPRESENT-URL already worked since code was reused from CHECKPRESENTEXPORT
diff --git a/Remote/External/Types.hs b/Remote/External/Types.hs
index 76059c3312..4008769aa3 100644
--- a/Remote/External/Types.hs
+++ b/Remote/External/Types.hs
@@ -313,6 +313,9 @@ data Response
| IMPORTABLECONTENT Size FilePath
| IMPORTABLECONTENTIDENTIFIER ContentIdentifier
| IMPORTABLECONTENTEND
+ | RETRIEVEIMPORT_SUCCESS
+ | RETRIEVEIMPORT_FAILURE ErrorMsg
+ | RETRIEVEIMPORT_URL URLString
| DELEGATE [String]
| UNSUPPORTED_REQUEST
deriving (Show)
@@ -359,6 +362,9 @@ instance Proto.Receivable Response where
parseCommand "IMPORTABLECONTENT" = Proto.parse2 IMPORTABLECONTENT
parseCommand "IMPORTABLECONTENTIDENTIFIER" = Proto.parse1 IMPORTABLECONTENTIDENTIFIER
parseCommand "IMPORTABLECONTENTEND" = Proto.parse0 IMPORTABLECONTENTEND
+ parseCommand "RETRIEVEIMPORT-SUCCESS" = Proto.parse0 RETRIEVEIMPORT_SUCCESS
+ parseCommand "RETRIEVEIMPORT-FAILURE" = Proto.parse1 RETRIEVEIMPORT_FAILURE
+ parseCommand "RETRIEVEIMPORT-URL" = Proto.parse1 RETRIEVEIMPORT_URL
parseCommand "DELEGATE" = Proto.parseList DELEGATE
parseCommand "UNSUPPORTED-REQUEST" = Proto.parse0 UNSUPPORTED_REQUEST
parseCommand _ = Proto.parseFail
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 440276de59..56eb250d92 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -503,6 +503,8 @@ These protocol extensions are currently supported.
* `TRANSFER-RETRIEVE-URL`
This allows the `TRANSFER-RETRIEVE-URL` response to be used
in reply to `TRANSFER` and `TRANSFEREXPORT`.
+ It also allows the `RETRIEVEIMPORT-URL` response to be used
+ in reply to `RETRIEVEIMPORT`.
* `CHECKPRESENT-URL`
This allows the `CHECKPRESENT-URL` response to be used
in reply to `CHECKPRESENT` and `CHECKPRESENTEXPORT`.
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index d759111ab5..1eeaad7190 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -201,10 +201,15 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
- * `TRANSFER-SUCCESS`
+ * `RETRIEVEIMPORT-SUCCESS`
Indicates that the retrieve was successful.
- * `TRANSFER-FAILURE ErrorMsg`
+ * `RETRIEVEIMPORT-FAILURE ErrorMsg`
Indicates the retrieve failed.
+ * `RETRIEVEIMPORT-URL Url`
+ Rather than the special remote downloading content from an url itself,
+ this lets it offload that work to git-annex. This response is a
+ protocol extension; it's only safe to send it to git-annex after
+ it sent an `EXTENSIONS` that included `TRANSFER-RETRIEVE-URL`.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
@@ -220,6 +225,11 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
* `CHECKPRESENT-UNKNOWN Key ErrorMsg`
Indicates that it is not currently possible to verify if content is
present in the remote. (Perhaps the remote cannot be contacted.)
+ * `CHECKPRESENT-URL Key Url`
+ Rather than the special remote checking an url itself,
+ this lets it offload that work to git-annex. This response is a protocol
+ extension; it's only safe to send it to git-annex after it sent an
+ `EXTENSIONS` that included `CHECKPRESENT-URL`.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
clarify
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 770ef5e206..d759111ab5 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -196,8 +196,8 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
whitespace, and other special characters.
No response is made to this message.
* `RETRIEVEIMPORT File`
- Retrieves the content of a file with the previously provided Name
- from the special remote to the File on local disk.
+ Retrieves the content of a file with the previously provided
+ `IMPORT` Name from the special remote to the File on local disk.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
@@ -209,7 +209,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
* `CHECKPRESENTIMPORT Key`
- Requests the remote to check if the previously provided Name
+ Requests the remote to check if the previously provided `IMPORT` Name
is present in it.
* `CHECKPRESENT-SUCCESS Key`
Indicates that a content has been positively verified to be present
branch
diff --git a/doc/todo/importtree_only_remotes/comment_4_7823d9526111213e991343a2285c2a23._comment b/doc/todo/importtree_only_remotes/comment_4_7823d9526111213e991343a2285c2a23._comment index aa32982938..04d78c9791 100644 --- a/doc/todo/importtree_only_remotes/comment_4_7823d9526111213e991343a2285c2a23._comment +++ b/doc/todo/importtree_only_remotes/comment_4_7823d9526111213e991343a2285c2a23._comment @@ -11,5 +11,5 @@ can safely do that. I know it will be much easier and safer to implement importtree only external special remotes. The external special remote interface for this still needs to be -implemented. +implemented. Work has begun in the `importonly_external` branch.) """]]
use "IMPORT" for consistency with "EXPORT"
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 42cecebb56..770ef5e206 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -189,7 +189,7 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
* `DELEGATE type=value [params]`
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
-* `LOCATION Name`
+* `IMPORT Name`
Comes before each of the following requests,
specifying the name of the file on the remote. It will be in the
form of a relative path, and may contain path separators,
@@ -253,7 +253,7 @@ Which responds:
Next git-annex asks for the content of a file to be retrieved.
- LOCATION foo
+ IMPORT foo
RETRIEVEIMPORTED tmpfile
If the file no longer exists, the response would be:
improved
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index 9f4ac6503e..42cecebb56 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -175,17 +175,20 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
* `LISTIMPORTABLECONTENTS`
Used to get a list of all the files that are stored in the special
remote. A block of responses
- can be made to this, which must always end with `END`.
- * `CONTENT Size Name`
+ can be made to this, which must always end with `IMPORTABLECONTENTEND`.
+ * `IMPORTABLECONTENT Size Name`
A file stored in the special remote. The Size is its size
in bytes. The Name is the name of the file on the remote,
in the form of a relative path, and may contain path separators,
whitespace, and other special characters.
Always followed by `CONTENTIDENTIFIER`.
- * `CONTENTIDENTIFIER ContentIdentifier`
+ * `IMPORTABLECONTENTIDENTIFIER ContentIdentifier`
Provide the ContentIdentifier for the previous `CONTENT`.
- * `END`
+ * `IMPORTABLECONTENTEND`
Indicates the end of the block of responses.
+ * `DELEGATE type=value [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `LOCATION Name`
Comes before each of the following requests,
specifying the name of the file on the remote. It will be in the
@@ -198,10 +201,13 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
- * `RETRIEVE-SUCCESS`
+ * `TRANSFER-SUCCESS`
Indicates that the retrieve was successful.
- * `RETRIEVE-FAILURE ErrorMsg`
+ * `TRANSFER-FAILURE ErrorMsg`
Indicates the retrieve failed.
+ * `DELEGATE type=value [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
* `CHECKPRESENTIMPORT Key`
Requests the remote to check if the previously provided Name
is present in it.
@@ -214,6 +220,9 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
* `CHECKPRESENT-UNKNOWN Key ErrorMsg`
Indicates that it is not currently possible to verify if content is
present in the remote. (Perhaps the remote cannot be contacted.)
+ * `DELEGATE type=value [params]`
+ Delegate this request to a different type of special remote.
+ See [[delegate_appendix]].
### import example
Support importtree=yes with rsync special remotes
It works fine, but this first importtree-only special remote exposed some
bugs with the implementation:
* initremote accepts importtree=yes exporttree=yes
* git-annex drop --from the remote is not blocked, does not actually remove
content
It works fine, but this first importtree-only special remote exposed some
bugs with the implementation:
* initremote accepts importtree=yes exporttree=yes
* git-annex drop --from the remote is not blocked, does not actually remove
content
diff --git a/CHANGELOG b/CHANGELOG
index 70b97f13d3..1a61501996 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -6,6 +6,7 @@ git-annex (10.20260625) UNRELEASED; urgency=medium
See https://git-annex.branchable.com/no_llm_code/
* git-annex.cabal: Added NoLLMDependencies build flag.
* Added stack-NoLLMDependencies.yaml
+ * Support importtree=yes with rsync special remotes.
* Fix build with time-1.15.
-- Joey Hess <id@joeyh.name> Wed, 01 Jul 2026 14:12:59 -0400
diff --git a/Remote/Rsync.hs b/Remote/Rsync.hs
index 9e8f1d04a7..824bdfd567 100644
--- a/Remote/Rsync.hs
+++ b/Remote/Rsync.hs
@@ -1,11 +1,11 @@
{- A remote that is only accessible by rsync.
-
- - Copyright 2011-2020 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2026 Joey Hess <id@joeyh.name>
-
- Licensed under the GNU AGPL version 3 or higher.
-}
-{-# LANGUAGE CPP, OverloadedStrings #-}
+{-# LANGUAGE OverloadedStrings #-}
module Remote.Rsync (
remote,
@@ -32,6 +32,7 @@ import Annex.Perms
import Remote.Helper.Special
import Remote.Helper.ExportImport
import Remote.Helper.Path
+import Types.Import
import Types.Export
import Types.ProposedAccepted
import Remote.Rsync.RsyncUrl
@@ -52,6 +53,7 @@ import qualified Utility.RawFilePath as R
import qualified Data.Map as M
import qualified Data.List.NonEmpty as NE
+import Text.Read
remote :: RemoteType
remote = specialRemoteType $ RemoteType
@@ -64,7 +66,7 @@ remote = specialRemoteType $ RemoteType
]
, setup = rsyncSetup
, exportSupported = exportIsSupported
- , importSupported = importUnsupported
+ , importSupported = importIsSupported
, exportImportSupported = exportImportUnsupported
, thirdPartyPopulated = False
}
@@ -106,11 +108,15 @@ gen r u rc gc rs = do
{ storeExport = storeExportM o
, retrieveExport = retrieveExportM o
, removeExport = removeExportM o
- , checkPresentExport = checkPresentExportM o
+ , checkPresentExport = checkPresentImportExportM o
, removeExportDirectory = Nothing
, renameExport = Just $ renameExportM o
}
- , importActions = importUnsupported
+ , importActions = ImportActions
+ { listImportableContents = listImportableContentsM o
+ , retrieveImport = retrieveImportM o
+ , checkPresentImport = checkPresentImportExportM o
+ }
, exportImportActions = exportImportUnsupported
, whereisKey = Nothing
, remoteFsck = Nothing
@@ -332,8 +338,8 @@ retrieveExportM o k loc dest p =
where
rsyncurl = mkRsyncUrl o (fromOsPath (fromExportLocation loc))
-checkPresentExportM :: RsyncOpts -> Key -> ExportLocation -> Annex Bool
-checkPresentExportM o _k loc = checkPresentGeneric o [rsyncurl]
+checkPresentImportExportM :: RsyncOpts -> Key -> ExportLocation -> Annex Bool
+checkPresentImportExportM o _k loc = checkPresentGeneric o [rsyncurl]
where
rsyncurl = mkRsyncUrl o (fromOsPath (fromExportLocation loc))
@@ -348,6 +354,59 @@ removeExportM o _k loc =
renameExportM :: RsyncOpts -> Key -> ExportLocation -> ExportLocation -> Annex (Maybe ())
renameExportM _ _ _ _ = return Nothing
+listImportableContentsM :: RsyncOpts -> Annex (Maybe (ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)))
+listImportableContentsM o =
+ withRsyncScratchDir $ \tmp -> do
+ opts <- rsyncOptions o
+ let p = rsyncCreateProcess $ opts ++
+ [ Param "--recursive"
+ , Param "--dry-run"
+ , Param $ "--out-format=" ++ formatstring
+ , Param url
+ , Param $ fromOsPath tmp
+ ]
+ l <- mapMaybe parse . lines <$> liftIO (readProcess' p)
+ return $ Just $ ImportableContentsComplete $ ImportableContents
+ { importableContents = l
+ , importableHistory = []
+ }
+ where
+ -- Make the url end in a slash so rsync will avoid prefixing
+ -- filenames it outputs with part of the url.
+ url = fromOsPath $ addTrailingPathSeparator $ toOsPath $ rsyncUrl o
+
+ formatstring = "%l|%M|%L|%n"
+
+ parse s
+ | "/" `isSuffixOf` s = Nothing
+ | otherwise = case splitc '|' s of
+ (ssz:sdate:ssymlink:rest)
+ | not (null ssymlink) -> Nothing
+ | otherwise -> do
+ sz <- readMaybe ssz
+ let loc = mkImportLocation $ toOsPath $
+ rsyncPathUnescape $
+ intercalate "|" rest
+ let cid = ContentIdentifier $ encodeBS $
+ ssz ++ "|" ++ sdate
+ Just (loc, (cid, sz))
+ _ -> Nothing
+
+retrieveImportM :: RsyncOpts -> ImportLocation -> [ContentIdentifier] -> OsPath -> Either Key (Annex Key) -> MeterUpdate -> Annex (Key, Verification)
+retrieveImportM o loc _ dest gk p =
+ case gk of
+ Right mkkey -> do
+ go Nothing
+ k <- mkkey
+ return (k, UnVerified)
+ Left k -> do
+ v <- verifyKeyContentIncrementally AlwaysVerify k go
+ return (k, v)
+ where
+ go iv = tailVerify iv dest $
+ rsyncRetrieve o [rsyncurl] dest (Just p)
+ rsyncurl = mkRsyncUrl o (fromOsPath (fromImportLocation loc))
+
{- Rsync params to enable resumes of sending files safely,
- ensure that files are only moved into place once complete
-}
diff --git a/Utility/Rsync.hs b/Utility/Rsync.hs
index 1a35aca09c..3bdeeda05c 100644
--- a/Utility/Rsync.hs
+++ b/Utility/Rsync.hs
@@ -1,6 +1,6 @@
{- various rsync stuff
-
- - Copyright 2010-2013 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2026 Joey Hess <id@joeyh.name>
-
- License: BSD-2-clause
-}
@@ -13,10 +13,12 @@ module Utility.Rsync (
rsyncServerReceive,
rsyncUseDestinationPermissions,
rsync,
+ rsyncCreateProcess,
rsyncUrlIsShell,
rsyncUrlIsPath,
rsyncProgress,
filterRsyncSafeOptions,
+ rsyncPathUnescape,
) where
import Common
@@ -69,6 +71,9 @@ rsyncUseDestinationPermissions = Param "--chmod=ugo=rwX"
rsync :: [CommandParam] -> IO Bool
rsync = boolSystem "rsync" . rsyncParamsFixup
+rsyncCreateProcess :: [CommandParam] -> CreateProcess
+rsyncCreateProcess = proc "rsync" . toCommand . rsyncParamsFixup
+
{- On Windows, rsync is from msys2, and expects to get msys2 formatted
- paths to files. (It thinks that C:foo refers to a host named "C").
- Fix up the Params appropriately. -}
@@ -187,3 +192,17 @@ toMSYS2Path p
| otherwise = s
#endif
+{- When listing files with eg --dry-run, rsync escapes some characters
+ - to 3 octal digits. Eg, "\#012" is '\n'
+ -}
+rsyncPathUnescape :: String -> FilePath
+rsyncPathUnescape = go
+ where
+ go [] = []
+ go ('\\':'#':d1:d2:d3:cs)
+ | isOctDigit d1 && isOctDigit d2 && isOctDigit d3 =
+ case (readish [d1], readish [d2], readish [d3]) of
+ (Just n1, Just n2, Just n3) ->
+ chr (n3+8*n2+8*8*n1) : go cs
+ _ -> error "internal"
+ go (c:cs) = c : go cs
diff --git a/doc/special_remotes/rsync.mdwn b/doc/special_remotes/rsync.mdwn
index 8ba16c998f..ad46fab75f 100644
--- a/doc/special_remotes/rsync.mdwn
(Diff truncated)
external special remote import only interface design
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index e85333ae9f..440276de59 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -54,9 +54,9 @@ any extensions it wants to use.
EXTENSIONS
Next, git-annex will generally send a message telling the special
-remote to start up. (Or it might send an INITREMOTE or EXPORTSUPPORTED or
-LISTCONFIGS, or perhaps other things in the future, so don't
-hardcode this order.)
+remote to start up. (Or it might send an INITREMOTE or EXPORTSUPPORTED
+or IMPORTSUPPORTED or LISTCONFIGS, or perhaps other things in the future,
+so don't hardcode this order.)
PREPARE
@@ -118,7 +118,7 @@ The following requests *must* all be supported by the special remote.
* `PREPARE`
Tells the remote that it's time to prepare itself to be used.
Only a few requests for details about the remote can come before this
- (EXTENSIONS, INITREMOTE, EXPORTSUPPORTED and LISTCONFIGS,
+ (EXTENSIONS, INITREMOTE, EXPORTSUPPORTED, IMPORTSUPPORTED and LISTCONFIGS,
but others may be added later).
* `PREPARE-SUCCESS`
Sent as a response to PREPARE once the special remote is ready for use.
@@ -184,7 +184,7 @@ The following requests *must* all be supported by the special remote.
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
-Special remotes can optionally support tree exports and imports,
+Special remotes can optionally support tree exports or imports,
which makes the [[git-annex-export]] and [[git-annex-import]] commands
work with them. See the [[export_and_import_appendix]] for
additional requests that git-annex will make when using special remotes in
diff --git a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
index ce1ec411bf..9f4ac6503e 100644
--- a/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
+++ b/doc/design/external_special_remote_protocol/export_and_import_appendix.mdwn
@@ -4,20 +4,28 @@ Some special remotes interface to a key/value datastore using keys that are
eg hashes, and it won't make sense for them to implement any of this.
When a special remote interfaces with something that looks like a directory
-of files, the simple export interface can be implemented to allow the
+of files, it can either or both of the interfaces described here.
+
+The export interface can be implemented to allow the
[[git-annex-export]] command to be used to store file trees on the special
remote.
-If other tools can write files to the special remote too, the import/export
-interface can be implemented. This allows for both [[git-annex-import]] and
-[[git-annex-export]] to be used with the special remote.
+The import interface can be implemented to allow the
+[[git-annex-import]] command to be used to import file trees from the special
+remote that are written by some other tool or process.
+
+Note that, when a special remote implements both interfaces, the user will
+need to choose at `initremote` time whether to use `importtree=yes` or
+`exporttree=yes`. They won't be allowed to use both, since this interface
+is not designed to safely allow import and export with the same special
+remote.
[[!toc]]
-## simple export interface
+## export interface
git-annex will use this when the special remote is initialized with
-exporttree=yes but without importtree=yes and indicates that it supports
+exporttree=yes and without importtree=yes and indicates that it supports
exports.
These are requests git-annex sends to the special remote program.
@@ -122,13 +130,13 @@ a request, it can reply with `UNSUPPORTED-REQUEST`.
Delegate this request to a different type of special remote.
See [[delegate_appendix]].
-## import/export interface
+## import interface
(This part is a draft, not implemented yet.)
git-annex will use this interface when the special remote is
-initialized with both exporttree=yes and importtree=yes and indicates
-that it supports both imports and exports.
+initialized with importtree=yes and indicates
+that it supports imports.
### content identifiers
@@ -141,7 +149,8 @@ file. This is called a ContentIdentifier. A good ContentIdentifier needs to:
* Be as unique as possible, but not necessarily fully unique.
A hash of the content would be ideal.
A (size, mtime, inode) tuple is as good a content identifier as git uses in
- its index.
+ its index. A (size, mtime) tuple will detect most modifications to files,
+ and is usually good enough.
* Be reasonably short, since it will be stored in the git-annex branch.
It's up to the implementor of a external special remote program what
@@ -156,14 +165,6 @@ program. Once the special remote has finished performing a request,
it should send one of the listed replies. Or, if it does not
support a request, it can reply with `UNSUPPORTED-REQUEST`.
-* `EXPORTSUPPORTED`
- Used to check if a special remote supports exports.
- Note that this request may be made before or after `PREPARE`.
- * `EXPORTSUPPORTED-SUCCESS`
- Indicates that it makes sense to use this special remote as an export.
- * `EXPORTSUPPORTED-FAILURE`
- Indicates that it does not make sense to use this special remote as an
- export.
* `IMPORTSUPPORTED`
Used to check if a special remote supports imports.
Note that this request may be made before or after `PREPARE`.
@@ -171,13 +172,6 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that it makes sense to import from this special remote.
* `IMPORTSUPPORTED-FAILURE`
Indicates that it does not make sense to import from this special remote.
-* `IMPORTKEYSUPPORTED`
- Used to check if a special remote supports `IMPORTKEY`.
- Note that this request may be made before or after `PREPARE`.
- * `IMPORTKEYSUPPORTED-SUCCESS`
- Indicates that `IMPORTKEY` can be used.
- * `IMPORTKEYSUPPORTED-FAILURE`
- Indicates that `IMPORTKEY` cannot be used.
* `LISTIMPORTABLECONTENTS`
Used to get a list of all the files that are stored in the special
remote. A block of responses
@@ -190,56 +184,17 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
Always followed by `CONTENTIDENTIFIER`.
* `CONTENTIDENTIFIER ContentIdentifier`
Provide the ContentIdentifier for the previous `CONTENT`.
- * `HISTORY`
- When a special remote stores historical versions of files,
- this can be used to list those versions. It opens a new
- block of responses. This can be repeated any number of times
- (indicating a branching history), and histories can also
- be nested multiple levels deep.
- This should only be a response when the remote supports using
- "TRANSFER RECEIVE Key" to retrieve historical versions of files,
- and when "GETCONFIG versioning" yields "VALUE TRUE".
* `END`
- Indicates the end of a block of responses.
+ Indicates the end of the block of responses.
* `LOCATION Name`
- Comes before each of the following requests (except for
- REMOVEEXPORTDIRECTORYWHENEMPTY),
+ Comes before each of the following requests,
specifying the name of the file on the remote. It will be in the
form of a relative path, and may contain path separators,
whitespace, and other special characters.
No response is made to this message.
-* `EXPECTED ContentIdentifier`
- Comes before each of the following requests (except
- for REMOVEEXPORTDIRECTORYWHENEMPTY), specifying the
- ContentIdentifier that is expected to be present on the remote.
-* `NOTHINGEXPECTED`
- If no ContentIdentifier is expected to be present, this is sent
- rather than `EXPECTED`.
-* `IMPORTKEY File`
- This only needs to be implemented if IMPORTKEYSUPPORTED indicates
- it is supported. And if a remote did not support it before, adding
- it will make importing the same content as before generate a likely
- different tree, which can lead to merge conflicts. So be careful
- implementing this.
- Generates a key by querying the remote for eg, a checksum.
- (See [[internals/key_format]] for details of how to format a key.)
- Any kind of key can be generated, depending on what the remote
- can support.
- The user expects this to be reasonably fast and not use a lot of disk
- space. It should not download the whole content of the file from the
- remote.
- Must take care to generate a key for the same content as the
- ContentIdentifier specified by `EXPECTED`, or otherwise fail.
- Replies:
- * `IMPORTKEY-SUCCESS Key`
- * `IMPORTKEY-SKIP`
- This causes nothing to be imported for this file.
- * `IMPORTKEY-FAILURE ErrorMsg`
-* `RETRIEVEEXPORTEXPECTED File`
- Retrieves the content of a file from the special remote
- to the File on local disk. Must take care to only retrieve
- content that has the ContentIdentifier specified by
- `EXPECTED`.
+* `RETRIEVEIMPORT File`
+ Retrieves the content of a file with the previously provided Name
+ from the special remote to the File on local disk.
While the transfer is running, the remote can send any number of
`PROGRESS` messages. Once the transfer is complete, it finishes by
sending one of these replies:
@@ -247,63 +202,24 @@ support a request, it can reply with `UNSUPPORTED-REQUEST`.
Indicates that the retrieve was successful.
* `RETRIEVE-FAILURE ErrorMsg`
Indicates the retrieve failed.
-* `STOREEXPORTEXPECTED Key File`
- Stores the content of File on the local disk to the previously
- provided Name on the remote. If the Name already exists on the remote,
- must take care to only overwrite it when it has the
(Diff truncated)
comment
diff --git a/doc/todo/import_tree_from_rsync_special_remote/comment_9_ae02364bc36d738a6b049e84560246fd._comment b/doc/todo/import_tree_from_rsync_special_remote/comment_9_ae02364bc36d738a6b049e84560246fd._comment new file mode 100644 index 0000000000..5243126696 --- /dev/null +++ b/doc/todo/import_tree_from_rsync_special_remote/comment_9_ae02364bc36d738a6b049e84560246fd._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-07-13T17:52:28Z" + content=""" +This is now possible to implement, since git-annex has an interface for +importtree-only special remotes. + +To list, it could avoid checksums and use: + + rsync -a --dry-run --out-format='%l|%M|%L|%n' $repo empty-directory + +The '%l' field is the file size and combined with the '%M' field's mtime, +would constitute the ContentIdentifier. Rsync uses a hardcoded mtime +format like "2026/07/13-12:49:50" which git-annex could parse and convert +to seconds-from-epoch to save a little bit of space. + +The '%L' field will be non-empty when it's a symlink, and those would be skipped. + +The '%n' field is the filename, which may be prefixed with a common top-level directory or module name. +Note that rsync escapes some characters, eg `\#012` is `\n`. + +--- + +Size+mtime as a ContentIdentifier is good enough, it only would prevent +git-annex from noticing and updating the import when: + +1. Two files with the same size and mtime got swapped with one-another. +2. A file is modified in place without updating the mtime or changing its size. +"""]]
support for import-only remotes
This will allow making remotes support importtree=yes that cannot safely
support exporttree=yes at the same time.
Note that adjustExportImport takes care of populating importActions
for a remote that has exportImportActions populated but does not
populate importActions itself. This allows for eg Remote.Directory
to not bother with separately populating importActions, which would be a
lot of redundant code. Annex.Import relies on that when it uses eg
listImportableContents.
This will allow making remotes support importtree=yes that cannot safely
support exporttree=yes at the same time.
Note that adjustExportImport takes care of populating importActions
for a remote that has exportImportActions populated but does not
populate importActions itself. This allows for eg Remote.Directory
to not bother with separately populating importActions, which would be a
lot of redundant code. Annex.Import relies on that when it uses eg
listImportableContents.
diff --git a/Annex/Export.hs b/Annex/Export.hs
index 6d2a6226e9..738ed07170 100644
--- a/Annex/Export.hs
+++ b/Annex/Export.hs
@@ -31,10 +31,12 @@ exportKey sha = mk <$> catKey sha
warnExportImportConflict :: Remote -> Annex ()
warnExportImportConflict r = do
isexport <- Remote.isExportSupported r
+ isimport <- Remote.isImportSupported r
isexportimport <- Remote.isExportImportSupported r
- let (ops, resolvcmd) = case (isexport, isexportimport) of
- (False, True) -> ("exported to and/or imported from", "git-annex import")
- (True, False) -> ("exported to", "git-annex export")
+ let (ops, resolvcmd) = case (isexport, isimport, isexportimport) of
+ (False, _, True) -> ("exported to and/or imported from", "git-annex import")
+ (True, _, False) -> ("exported to", "git-annex export")
+ (False, True, False) -> ("imported from", "git-annex import")
_ -> ("exported to and/or imported from", "git-annex export")
toplevelWarning True $ UnquotedString $ unwords
[ "Conflict detected. Different trees have been"
diff --git a/Annex/Import.hs b/Annex/Import.hs
index a48f0bdacb..ea5393225c 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -807,7 +807,7 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
return (Right job)
thirdpartypopulatedimport db (loc, (cid, sz)) =
- case Remote.importKey ia of
+ case Remote.importKey (Remote.exportImportActions remote) of
Nothing -> return Nothing
Just importkey ->
tryNonAsync (importkey loc cid sz nullMeterUpdate) >>= \case
@@ -827,7 +827,7 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
-- than downloading and retrieving a key, to avoid
-- generating trees with different keys for the same content.
let act = if importcontent
- then case Remote.importKey ia of
+ then case Remote.importKey (Remote.exportImportActions remote) of
Nothing -> dodownload
Just _ -> if Utility.Matcher.introspect matchNeedsFileContent (fst matcher)
then dodownload
@@ -836,7 +836,7 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
act cidmap (loc, (cid, sz)) f matcher
doimport cidmap (loc, (cid, sz)) f matcher =
- case Remote.importKey ia of
+ case Remote.importKey (Remote.exportImportActions remote) of
Nothing -> error "internal" -- checked earlier
Just importkey -> do
when (Utility.Matcher.introspect matchNeedsFileContent (fst matcher)) $
@@ -883,8 +883,9 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
getcontent k = do
let af = AssociatedFile (Just f)
let downloader p' tmpfile = do
- _ <- Remote.retrieveExportWithContentIdentifier
- ia loc [cid] tmpfile
+ _ <- Remote.retrieveImport
+ (Remote.importActions remote)
+ loc [cid] tmpfile
(Left k)
(combineMeterUpdate p' p)
ok <- moveAnnex k tmpfile
@@ -901,8 +902,9 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
-- need to retrieve this file.
doimportsmall cidmap loc cid sz p = do
let downloader tmpfile = do
- (k, _) <- Remote.retrieveExportWithContentIdentifier
- ia loc [cid] tmpfile
+ (k, _) <- Remote.retrieveImport
+ (Remote.importActions remote)
+ loc [cid] tmpfile
(Right (mkkey tmpfile))
p
case keyGitSha k of
@@ -924,8 +926,9 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
dodownload cidmap (loc, (cid, sz)) f matcher = do
let af = AssociatedFile (Just f)
let downloader tmpfile p = do
- (k, _) <- Remote.retrieveExportWithContentIdentifier
- ia loc [cid] tmpfile
+ (k, _) <- Remote.retrieveImport
+ (Remote.importActions remote)
+ loc [cid] tmpfile
(Right (mkkey tmpfile))
p
case keyGitSha k of
@@ -971,8 +974,6 @@ importKeys remote importtreeconfig importcontent thirdpartypopulated importablec
}
fst <$> genKey ks nullMeterUpdate backend
else gitShaKey <$> hashFile tmpfile
-
- ia = Remote.exportImportActions remote
bwlimit = remoteAnnexBwLimitDownload (Remote.gitconfig remote)
<|> remoteAnnexBwLimit (Remote.gitconfig remote)
@@ -1096,7 +1097,7 @@ pruneImportMatcher = Utility.Matcher.pruneMatcher matchNeedsKey
-}
getImportableContents :: Remote -> ImportTreeConfig -> CheckGitIgnore -> FileMatcher Annex -> Annex (Maybe (ImportableContentsChunkable Annex (ContentIdentifier, ByteSize)))
getImportableContents r importtreeconfig ci matcher = do
- Remote.listImportableOrExportedContents (Remote.exportImportActions r) >>= \case
+ Remote.listImportableContents (Remote.importActions r) >>= \case
Just (ImportableContentsComplete ic) -> do
dbhandle <- opendbhandle
Just . ImportableContentsComplete
diff --git a/Remote/Adb.hs b/Remote/Adb.hs
index c7aa32d9ac..a784ae1232 100644
--- a/Remote/Adb.hs
+++ b/Remote/Adb.hs
@@ -54,6 +54,7 @@ remote = specialRemoteType $ RemoteType
]
, setup = adbSetup
, exportSupported = exportIsSupported
+ , importSupported = importUnsupported
, exportImportSupported = exportImportIsSupported
, thirdPartyPopulated = False
}
@@ -97,6 +98,7 @@ gen r u rc gc rs = do
, removeExportDirectory = Just $ removeExportDirectoryM serial adir
, renameExport = Just $ renameExportM serial adir
}
+ , importActions = importUnsupported
, exportImportActions = ExportImportActions
{ listImportableOrExportedContents = listImportableOrExportedContentsM serial adir c
, importKey = Nothing
diff --git a/Remote/BitTorrent.hs b/Remote/BitTorrent.hs
index d500f5c00b..0923fc9205 100644
--- a/Remote/BitTorrent.hs
+++ b/Remote/BitTorrent.hs
@@ -47,6 +47,7 @@ remote = RemoteType
, configParser = mkRemoteConfigParser []
, setup = giveup "not supported"
, exportSupported = exportUnsupported
+ , importSupported = importUnsupported
, exportImportSupported = exportImportUnsupported
, thirdPartyPopulated = False
}
@@ -78,6 +79,7 @@ gen r _ rc gc rs = do
, checkPresent = checkKey
, checkPresentCheap = False
, exportActions = exportUnsupported
+ , importActions = importUnsupported
, exportImportActions = exportImportUnsupported
, whereisKey = Nothing
, remoteFsck = Nothing
diff --git a/Remote/Borg.hs b/Remote/Borg.hs
index 506d4366c7..c242c310b6 100644
--- a/Remote/Borg.hs
+++ b/Remote/Borg.hs
@@ -59,6 +59,7 @@ remote = RemoteType
]
, setup = borgSetup
, exportSupported = exportUnsupported
+ , importSupported = importUnsupported
, exportImportSupported = exportImportIsSupported
, thirdPartyPopulated = True
}
@@ -94,6 +95,7 @@ gen r u rc gc rs = do
, checkPresent = checkPresentDummy
, checkPresentCheap = borgLocal borgrepo
, exportActions = exportUnsupported
+ , importActions = importUnsupported
, exportImportActions = ExportImportActions
{ listImportableOrExportedContents = listImportableOrExportedContentsM u borgrepo c
, importKey = Just ThirdPartyPopulated.importKey
diff --git a/Remote/Bup.hs b/Remote/Bup.hs
index d221ab3c4a..5ff440ff45 100644
--- a/Remote/Bup.hs
+++ b/Remote/Bup.hs
@@ -54,6 +54,7 @@ remote = specialRemoteType $ RemoteType
]
, setup = bupSetup
, exportSupported = exportUnsupported
+ , importSupported = importUnsupported
, exportImportSupported = exportImportUnsupported
, thirdPartyPopulated = False
}
@@ -87,6 +88,7 @@ gen r u rc gc rs = do
, checkPresent = checkPresentDummy
, checkPresentCheap = bupLocal buprepo
, exportActions = exportUnsupported
+ , importActions = importUnsupported
, exportImportActions = exportImportUnsupported
, whereisKey = Nothing
, remoteFsck = Nothing
diff --git a/Remote/Compute.hs b/Remote/Compute.hs
index a21fd2f7af..e443f94643 100644
--- a/Remote/Compute.hs
+++ b/Remote/Compute.hs
@@ -85,6 +85,7 @@ remote = RemoteType
, configParser = computeConfigParser
, setup = setupInstance
, exportSupported = exportUnsupported
+ , importSupported = importUnsupported
, exportImportSupported = exportImportUnsupported
, thirdPartyPopulated = False
}
@@ -119,6 +120,7 @@ gen r u rc gc rs = case getComputeProgram' rc of
, checkPresent = checkKey rs
, checkPresentCheap = False
(Diff truncated)
despam
diff --git a/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment b/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment deleted file mode 100644 index 46aa03452c..0000000000 --- a/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="nanalyly" - avatar="http://cdn.libravatar.org/avatar/e9307232a8b6942ccf35b73d26fa2ee3" - subject="comment 3" - date="2026-07-13T01:43:14Z" - content=""" -I also like the idea of having the laptop avoid storing annexed files locally, since it keeps the system lightweight while still allowing access when needed. Although <a href=\"https://sloperidergame.com\">Slope Rider</a> is unrelated to file management, I appreciate how both require thoughtful planning and organization to achieve the best results. I would also make sure the backup process is tested regularly so I can be confident that every important file can be restored if necessary. -"""]]
update
diff --git a/doc/todo/importtree_only_remotes.mdwn b/doc/todo/importtree_only_remotes.mdwn index 1728406876..5d518feab1 100644 --- a/doc/todo/importtree_only_remotes.mdwn +++ b/doc/todo/importtree_only_remotes.mdwn @@ -51,24 +51,31 @@ What is needed in such an interface? -- strongly verified if this is used to download a particular Key -- that was at one point stored on the remote, since the content -- of the remote could change at any time. - -- (The MeterUpdate does not need to be used if - -- sequentially to the file.) + -- (The MeterUpdate does not need to be used if it writes + -- sequentially to the file.) -- Throws exception on failure. retrieveImport :: ImportLocation -> FilePath -> MeterUpdate -> Annex () -- Checks if anything is present on the remote at the specified -- ImportLocation. It may check the size or other characteristics - -- of the Key, but does not need to guarantee that the content on - -- the remote is the same as the Key's content. - -- Throws an exception if the remote cannot be accessed. + -- of the Key, but does not need to guarantee that the content on + -- the remote is the same as the Key's content. + -- Throws an exception if the remote cannot be accessed. checkPresentImport :: Key -> ImportLocation -> Annex Bool listImportableContents is unchanged, and checkPresentImport above -is identical to checkPresentExport. retrieveImport is very similar -to retrieveExport, except that the content retrieved is not guaranteed -to be the same as the content of any key. Actually, it may be an identical -interface; the only thing I can find that uses retrieveExport forces -verification of the content retrived since it could have been changed by -another writer. +is identical to checkPresentExport. + +retrieveImport is very similar to retrieveExport, except that the content +retrieved is not a particular Key, and that it does not provide any +Verification. + +An alternative to the above `retrieveImport` would be to use +`retrieveExport` for getting known keys from the remote, and use +`retrieveExportWithContentIdentifier` with its current interface for +importing new ContentIdentifiers from the remote. It would not need +to ensure that the file really has the ContentIdentifier, but this leaves +open the possibility of importtree only remotes that do at least some +verification of the ContentIdentifier. The similarity with interface that we already have suggests that perhaps this does not need changes to Types.Remote to implement. diff --git a/doc/todo/importtree_only_remotes/comment_2_ca4f23bcb20bafec2f4d72049b4ed339._comment b/doc/todo/importtree_only_remotes/comment_2_ca4f23bcb20bafec2f4d72049b4ed339._comment new file mode 100644 index 0000000000..04d0a71a12 --- /dev/null +++ b/doc/todo/importtree_only_remotes/comment_2_ca4f23bcb20bafec2f4d72049b4ed339._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2026-07-13T13:08:53Z" + content=""" +I think it would actually be better to modify Types.Remote and add a new +interface specifically for this, than to try to jam it into an existing +interface it was not designed for. +"""]]
Added a comment
diff --git a/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment b/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment new file mode 100644 index 0000000000..46aa03452c --- /dev/null +++ b/doc/forum/only_keep_current_version_of_files_in_repo/comment_3_afd16a2b05d3e6fda6078fc4178ac3c2._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="nanalyly" + avatar="http://cdn.libravatar.org/avatar/e9307232a8b6942ccf35b73d26fa2ee3" + subject="comment 3" + date="2026-07-13T01:43:14Z" + content=""" +I also like the idea of having the laptop avoid storing annexed files locally, since it keeps the system lightweight while still allowing access when needed. Although <a href=\"https://sloperidergame.com\">Slope Rider</a> is unrelated to file management, I appreciate how both require thoughtful planning and organization to achieve the best results. I would also make sure the backup process is tested regularly so I can be confident that every important file can be restored if necessary. +"""]]
2026
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index 0ddf71eaaf..8c7a4e2343 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -19,7 +19,7 @@ from [[!inline raw=yes pages="thanks/contributor-list"]] and many others who have [[contributed|contribute]] documentation fixes, tips, user support, etc. John Lawrence made the logo. -## financial support, 2024-2025 +## financial support, 2024-2026 <img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> <img alt="Stanford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200>
fzj logo hotlink broke
Could not find a good svg one on their website,
although I did find a lot of png ones here
https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg/@@search?SearchableText=Julich%20Forschungszentrum%20logo&active=relevance&allow_local=false&group_select=4&hl=true&local=false&metadata_fields=effective&metadata_fields=UID&metadata_fields=start&path_prefix=%2Fstatic%2Fmedia%2FLogo.2ceb35fc.svg%2F&use_site_search_settings=1
Hotlinking to wikimedia commons feels a bit naughty, but the traffic on
this page is probably very low.
Could not find a good svg one on their website,
although I did find a lot of png ones here
https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg/@@search?SearchableText=Julich%20Forschungszentrum%20logo&active=relevance&allow_local=false&group_select=4&hl=true&local=false&metadata_fields=effective&metadata_fields=UID&metadata_fields=start&path_prefix=%2Fstatic%2Fmedia%2FLogo.2ceb35fc.svg%2F&use_site_search_settings=1
Hotlinking to wikimedia commons feels a bit naughty, but the traffic on
this page is probably very low.
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index 89e20d22f7..0ddf71eaaf 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -23,7 +23,7 @@ tips, user support, etc. John Lawrence made the logo. <img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> <img alt="Stanford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> -<img alt="Jülich Forschungszentrum logo" src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width=200> +<img alt="Jülich Forschungszentrum logo" src="https://upload.wikimedia.org/wikipedia/commons/4/40/Logo_des_Forschungszentrums_J%C3%BClich_seit_2018.svg" width=200> <img alt="DANDI logo" src="https://dandiarchive.org/assets/logo-DbYqjGgV.svg" width=150> <img alt="ReproNim logo" src="https://repronim.org/images/square-512T2.png" width=70>
git-annex-browser
diff --git a/doc/related_software.mdwn b/doc/related_software.mdwn index 44580bbf30..353941f2d9 100644 --- a/doc/related_software.mdwn +++ b/doc/related_software.mdwn @@ -71,4 +71,7 @@ designed to interoperate with it. * [forgejo-aneksajo](https://codeberg.org/matrss/forgejo-aneksajo) is a soft-fork of Forgejo (a git forge) that integrates support for git-annex. +* [git-annex-browser](https://github.com/janttsu/git-annex-browser) is a + terminal UI for exploring git-annex repositories. + See also [[not]] for software that is *not* related to git-annex, but similar.
fixed
diff --git a/doc/bugs/auth.mdwn b/doc/bugs/auth.mdwn index c45cea90ba..f49438eb73 100644 --- a/doc/bugs/auth.mdwn +++ b/doc/bugs/auth.mdwn @@ -69,3 +69,5 @@ My expectation would be that a git-annex push would be able to trigger the oauth `git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05` on Debian forky/sid [[!tag projects/INM7]] + +> [[fixed]] in forgejo-aneksajo --[[Joey]] diff --git a/doc/bugs/auth/comment_11_bfc8f3d5be99e86efd79dfe3b55e395e._comment b/doc/bugs/auth/comment_11_bfc8f3d5be99e86efd79dfe3b55e395e._comment new file mode 100644 index 0000000000..1a91e48a45 --- /dev/null +++ b/doc/bugs/auth/comment_11_bfc8f3d5be99e86efd79dfe3b55e395e._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 11""" + date="2026-07-02T15:58:22Z" + content=""" +<https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/pulls/121> +"""]]
generalize
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 0b579a15fa..349ed1f501 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -9,8 +9,9 @@ important aspect of its design. So, git-annex does not contain code generated by LLMs and guarantees it will never do so. -However, libraries that git-annex depends on do not generally have such -guarantees. Although it would be very much appreciated if they did. +However, libraries and other things that git-annex depends on do not +generally have such guarantees. Although it would be very much appreciated +if they did. git-annex currently supports being built with versions of dependencies that pre-date any introduction of LLM generated code.
git
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 66d9ce7a62..0b579a15fa 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -99,3 +99,11 @@ and 10,000+ lines of changes. Cabal is needed to build git-annex, but is not linked into it. There is a risk that a new version of Cabal could need changes to git-annex.cabal that prevent an old version building it. + +### git + +Since 2.53 + +[First LLM generated code](https://github.com/git/git/commit/d7971544fe17378f44f49983010dbfc1834f7bef) + +git-annex supports git back to 2.22.
verified fixed
diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials.mdwn b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials.mdwn index d9e2ac1002..332de67489 100644 --- a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials.mdwn +++ b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials.mdwn @@ -106,3 +106,4 @@ $ ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +> [[fixed|done]] at some point after 10.20251029. --[[Joey]] diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_5_2eb904c8c743cffd704e4d785d5c453a._comment b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_5_2eb904c8c743cffd704e4d785d5c453a._comment new file mode 100644 index 0000000000..cb54da6541 --- /dev/null +++ b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_5_2eb904c8c743cffd704e4d785d5c453a._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-07-02T15:36:07Z" + content=""" +I am not able to reproduce this with current git-annex. + +But, I was able to reproduce it with git-annex 10.20251029. + +I have not bisected to find the fix, but will close this. +"""]]
link
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 1cc0fdd426..66d9ce7a62 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -1,6 +1,7 @@ -LLM generated code in free software is a potential landmine. The copyright -of such code is an open question, and any current answer to that question -risks changing at some point in the future. +[LLM](https://en.wikipedia.org/wiki/Large_language_model) generated code in +free software is a potential landmine. The copyright of such code is an +open question, and any current answer to that question risks changing at +some point in the future. This is a particular problem for git-annex since [[future proofing]] is an important aspect of its design.
typo
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 2d3d53caea..1cc0fdd426 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -13,7 +13,7 @@ guarantees. Although it would be very much appreciated if they did. git-annex currently supports being built with versions of dependencies that pre-date any introduction of LLM generated code. -To do do, turn on the `NoLLMDependencies` build flag. +To do so, turn on the `NoLLMDependencies` build flag. When building with stack, use `stack-NoLLMDependencies.yaml`. (It is not currently built that way by default, but such builds are welcome.)
Added stack-NoLLMDependencies.yaml
diff --git a/CHANGELOG b/CHANGELOG
index c3e93c2e0a..70b97f13d3 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,7 @@ git-annex (10.20260625) UNRELEASED; urgency=medium
that predate the addition of any LLM generated code.
See https://git-annex.branchable.com/no_llm_code/
* git-annex.cabal: Added NoLLMDependencies build flag.
+ * Added stack-NoLLMDependencies.yaml
* Fix build with time-1.15.
-- Joey Hess <id@joeyh.name> Wed, 01 Jul 2026 14:12:59 -0400
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn
index 31bf0fff9a..2d3d53caea 100644
--- a/doc/no_llm_code.mdwn
+++ b/doc/no_llm_code.mdwn
@@ -14,6 +14,7 @@ guarantees. Although it would be very much appreciated if they did.
git-annex currently supports being built with versions of
dependencies that pre-date any introduction of LLM generated code.
To do do, turn on the `NoLLMDependencies` build flag.
+When building with stack, use `stack-NoLLMDependencies.yaml`.
(It is not currently built that way by default, but such builds are welcome.)
Unfortunately, it's not possible to guarantee that will continue to work
diff --git a/git-annex.cabal b/git-annex.cabal
index 1a010768e1..6a920de2e0 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -36,6 +36,7 @@ Description:
Extra-Source-Files:
stack.yaml
stack-botan.yaml
+ stack-NoLLMDependencies.yaml
README
CHANGELOG
NEWS
diff --git a/stack-NoLLMDependencies.yaml b/stack-NoLLMDependencies.yaml
new file mode 100644
index 0000000000..3fcb296182
--- /dev/null
+++ b/stack-NoLLMDependencies.yaml
@@ -0,0 +1,30 @@
+flags:
+ git-annex:
+ NoLLMDependencies: true
+ production: true
+ parallelbuild: true
+ assistant: true
+ torrentparser: true
+ magicmime: false
+ dbus: false
+ debuglocks: false
+ benchmark: true
+ ospath: true
+ botan: false
+ blake3: true
+ xxh3: true
+ file-io:
+ os-string: true
+ xxhash-ffi:
+ pkg-config: false
+packages:
+- '.'
+resolver: lts-24.26
+extra-deps:
+- aws-0.25.2
+- file-io-0.2.0
+- blake3-0.3
+- xxhash-ffi-0.3.1
+- ram-0.20.1
+- persistent-2.14.6.3
+- persistent-sqlite-2.13.3.0
update
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index b3b0717e41..31bf0fff9a 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -18,11 +18,13 @@ To do do, turn on the `NoLLMDependencies` build flag. Unfortunately, it's not possible to guarantee that will continue to work in new versions of git-annex. That's the goal, but it may become -untenable. It's already the case that building git-annex with the older -versions of dependencies leaves at least one security hole in a dependency -open. See below for details about this and other possible future problems +untenable. See below for details about possible future problems with specific dependencies. +Note that if a security hole is only fixed by a newer version of a +dependency, the `NoLLMDependencies` build flag will still build with the +older, insecure version. + Additional work needs to be done on an ongoing basis to review git-annex's dependencies to detect the addition of LLM generated code.
correction
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index e0ed75f24e..b3b0717e41 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -56,9 +56,7 @@ with apparently broken (how?) changes in 0.21.0 being reverted in 0.21.1. Rather than use ram, git-annex continues to use the unmaintained -[memory](//hackage.haskell.org/package/memory). -Since git-annex's use of this package is fairly incidental, this -dependency could be removed if it becomes a problem. +[memory](//hackage.haskell.org/package/memory) that ram was forked from. But ram is an dependency of other dependencies, and these in particular depend on 0.21.0 or newer:
simplify NoLLMDependencies by depending on older version of ram
That prevents cabal from picking dependencies that need a newer version
of ram which contains LLM generated code.
Note that git-annex still depends on memory. And so all imports from
memory need to be PackageImports qualified now.
It would be easier if git-annex could use the old version of ram.
The problem is that it needs the ByteArrayAccess instance for Digest.
When crypton switched to ram, it depended immediately on the newer
version. So there is no available instance for git-annex to use.
That prevents cabal from picking dependencies that need a newer version
of ram which contains LLM generated code.
Note that git-annex still depends on memory. And so all imports from
memory need to be PackageImports qualified now.
It would be easier if git-annex could use the old version of ram.
The problem is that it needs the ByteArrayAccess instance for Digest.
When crypton switched to ram, it depended immediately on the newer
version. So there is no available instance for git-annex to use.
diff --git a/Annex/Balanced.hs b/Annex/Balanced.hs
index 394cb56e00..9005440ad6 100644
--- a/Annex/Balanced.hs
+++ b/Annex/Balanced.hs
@@ -5,6 +5,8 @@
- Licensed under the GNU AGPL version 3 or higher.
-}
+{-# LANGUAGE PackageImports #-}
+
module Annex.Balanced where
import Key
@@ -15,7 +17,7 @@ import Data.Maybe
import qualified Data.List as L
import Data.Bits (shiftL)
import qualified Data.Set as S
-import qualified Data.ByteArray as BA
+import qualified "memory" Data.ByteArray as BA
-- The Int is how many UUIDs to pick.
type BalancedPicker = S.Set UUID -> Key -> Int -> [UUID]
diff --git a/Annex/DirHashes.hs b/Annex/DirHashes.hs
index 21756dea37..e2fb08d5e8 100644
--- a/Annex/DirHashes.hs
+++ b/Annex/DirHashes.hs
@@ -5,6 +5,8 @@
- Licensed under the GNU AGPL version 3 or higher.
-}
+{-# LANGUAGE PackageImports #-}
+
module Annex.DirHashes (
Hasher,
HashLevels(..),
@@ -20,7 +22,7 @@ module Annex.DirHashes (
import Data.Default
import Data.Bits
import qualified Data.List.NonEmpty as NE
-import qualified Data.ByteArray as BA
+import qualified "memory" Data.ByteArray as BA
import qualified Data.ByteString as S
import Common
diff --git a/Annex/Import.hs b/Annex/Import.hs
index 67b845ddd5..170bf4aeeb 100644
--- a/Annex/Import.hs
+++ b/Annex/Import.hs
@@ -6,6 +6,7 @@
-}
{-# LANGUAGE OverloadedStrings #-}
+{-# LANGUAGE PackageImports #-}
{-# LANGUAGE CPP #-}
module Annex.Import (
@@ -71,7 +72,7 @@ import Backend.Utilities
import Control.Concurrent.STM
import qualified Data.Map.Strict as M
import qualified Data.Set as S
-import qualified Data.ByteArray.Encoding as BA
+import qualified "memory" Data.ByteArray.Encoding as BA
#ifdef mingw32_HOST_OS
import qualified System.FilePath.Posix as Posix
#endif
diff --git a/Utility/AuthToken.hs b/Utility/AuthToken.hs
index 4f94c984a1..46a7c4a7b6 100644
--- a/Utility/AuthToken.hs
+++ b/Utility/AuthToken.hs
@@ -25,7 +25,7 @@ import Utility.Exception
import Data.Maybe
import Data.Char
-import qualified Data.ByteArray as BA
+import qualified "memory" Data.ByteArray as BA
import qualified Data.Text as T
import qualified Data.Text.Encoding as TE
import qualified Data.ByteString.Lazy as L
diff --git a/Utility/Hash/Crypton.hs b/Utility/Hash/Crypton.hs
index bb49a7a2bc..4d290d0150 100644
--- a/Utility/Hash/Crypton.hs
+++ b/Utility/Hash/Crypton.hs
@@ -69,7 +69,7 @@ module Utility.Hash.Crypton (
import qualified Data.ByteString as S
import qualified Data.ByteString.Lazy as L
import Data.IORef
-import qualified Data.ByteArray as BA
+import qualified "memory" Data.ByteArray as BA
import "crypton" Crypto.Hash
import Utility.Hash.Types
diff --git a/Utility/Hash/Types.hs b/Utility/Hash/Types.hs
index acc3ef97e7..9705876013 100644
--- a/Utility/Hash/Types.hs
+++ b/Utility/Hash/Types.hs
@@ -6,12 +6,13 @@
-}
{-# LANGUAGE DeriveGeneric #-}
+{-# LANGUAGE PackageImports #-}
module Utility.Hash.Types where
import qualified Data.ByteString as S
-import Data.ByteArray
-import qualified Data.ByteArray.Encoding as BAE
+import "memory" Data.ByteArray
+import qualified "memory" Data.ByteArray.Encoding as BAE
import Data.String
import Control.DeepSeq
import GHC.Generics
diff --git a/Utility/IPAddress.hs b/Utility/IPAddress.hs
index e19cecac1d..cde446b642 100644
--- a/Utility/IPAddress.hs
+++ b/Utility/IPAddress.hs
@@ -6,6 +6,7 @@
-}
{-# LANGUAGE BinaryLiterals #-}
+{-# LANGUAGE PackageImports #-}
-- Note that some extensions are necessary for reasons outlined in
-- my July 2021 blog post. -- JEH
@@ -21,7 +22,7 @@ import Utility.Exception
import Network.Socket
import Data.Word
-import Data.Memory.Endian
+import "memory" Data.Memory.Endian
import Data.List
import Text.Printf
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn
index 258b1d0337..e0ed75f24e 100644
--- a/doc/no_llm_code.mdwn
+++ b/doc/no_llm_code.mdwn
@@ -55,23 +55,22 @@ Note particularly [large LLM generated code churn](https://github.com/jappeace/r
with apparently broken (how?) changes in 0.21.0 being reverted in
0.21.1.
-Rather than depend on ram, git-annex continues to depend on the unmaintained
+Rather than use ram, git-annex continues to use the unmaintained
[memory](//hackage.haskell.org/package/memory).
Since git-annex's use of this package is fairly incidental, this
dependency could be removed if it becomes a problem.
-But ram is an dependency of other dependencies:
+But ram is an dependency of other dependencies, and these in particular
+depend on 0.21.0 or newer:
* crypton since 1.1.0
-* crypton-x509-validation since 1.9.0. Note that 1.9.1 fixes a security
- hole, [CVE-2026-9648](https://www.cve.org/CVERecord?id=CVE-2026-9648).
- This security hole is not expected to affect many git-annex users.
-* aws since 0.25.3 (needed to get the CVE-2026-9648 fix)
-* tls since 2.3.0
-* blake3 since 0.3.1 (optional git-annex dependency)
+* tls since 2.3.1
[Reverse dependencies of ram](https://packdeps.haskellers.com/reverse/ram)
+The `NoLLMDependencies` build flag depends on an older version of ram
+in order to prevent such dependencies using the newer version.
+
### persistent
[persistent](https://hackage.haskell.org/package/persistent) since 2.15.0.0
diff --git a/git-annex.cabal b/git-annex.cabal
index d627e0f14c..1a010768e1 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -292,8 +292,11 @@ Executable git-annex
servant-client-core,
warp (>= 3.2.8),
warp-tls (>= 3.2.2),
+ crypton,
crypton-connection (>= 0.4.3),
- crypton-x509-store
+ crypton-x509-store,
+ tls,
+ aws (>= 0.24.1)
CC-Options: -Wall
GHC-Options: -Wall -fno-warn-tabs -Wincomplete-uni-patterns
Default-Language: Haskell2010
@@ -305,17 +308,11 @@ Executable git-annex
if flag(NoLLMDependencies)
Build-Depends:
base (>= 4.18.2.1 && < 4.23),
- crypton (< 1.1.0),
- crypton-x509-validation (< 1.9.0),
- aws (>= 0.24.1) && (< 2.25.3),
- tls (< 2.3.0),
+ ram (< 0.21.0),
persistent (>= 2.13.3) && (< 2.15.0.0)
else
Build-Depends:
base (>= 4.18.2.1 && < 5),
- crypton,
- aws (>= 0.24.1),
- tls,
(Diff truncated)
update
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index a11f18e350..258b1d0337 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -20,11 +20,11 @@ Unfortunately, it's not possible to guarantee that will continue to work in new versions of git-annex. That's the goal, but it may become untenable. It's already the case that building git-annex with the older versions of dependencies leaves at least one security hole in a dependency -open. +open. See below for details about this and other possible future problems +with specific dependencies. -This means that additional work needs to be done to review git-annex's -dependencies on an ongoing basis to detect the addition of LLM generated -code. +Additional work needs to be done on an ongoing basis to review +git-annex's dependencies to detect the addition of LLM generated code. Help with finding these is welcome. Please edit this page and/or file bug reports on git-annex if it cannot be built without LLM generated code. @@ -63,8 +63,9 @@ dependency could be removed if it becomes a problem. But ram is an dependency of other dependencies: * crypton since 1.1.0 -* crypton-x509-validation since 1.9.0. Note that 1.9.1 fixes - [CVE-2026-9648](https://www.cve.org/CVERecord?id=CVE-2026-9648). +* crypton-x509-validation since 1.9.0. Note that 1.9.1 fixes a security + hole, [CVE-2026-9648](https://www.cve.org/CVERecord?id=CVE-2026-9648). + This security hole is not expected to affect many git-annex users. * aws since 0.25.3 (needed to get the CVE-2026-9648 fix) * tls since 2.3.0 * blake3 since 0.3.1 (optional git-annex dependency)
git-annex.cabal: Added NoLLMDependencies build flag
diff --git a/CHANGELOG b/CHANGELOG
index 2f70663abc..c3e93c2e0a 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -4,6 +4,7 @@ git-annex (10.20260625) UNRELEASED; urgency=medium
and will attempt to remain buildable with versions of dependencies
that predate the addition of any LLM generated code.
See https://git-annex.branchable.com/no_llm_code/
+ * git-annex.cabal: Added NoLLMDependencies build flag.
* Fix build with time-1.15.
-- Joey Hess <id@joeyh.name> Wed, 01 Jul 2026 14:12:59 -0400
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn
index a8aa82ec71..a11f18e350 100644
--- a/doc/no_llm_code.mdwn
+++ b/doc/no_llm_code.mdwn
@@ -13,14 +13,18 @@ guarantees. Although it would be very much appreciated if they did.
git-annex currently supports being built with versions of
dependencies that pre-date any introduction of LLM generated code.
-(It is not currently built that way by default but such builds are welcome.)
-Unfortunately, it's not possible to guarantee that will continue to be the
-case in new versions of git-annex. That's the goal, but it may become
-untenable.
-
-Unfortunately, this means that additional work needs to be done to review
-git-annex's dependencies on an ongoing basis to detect the addition of LLM
-generated code.
+To do do, turn on the `NoLLMDependencies` build flag.
+(It is not currently built that way by default, but such builds are welcome.)
+
+Unfortunately, it's not possible to guarantee that will continue to work
+in new versions of git-annex. That's the goal, but it may become
+untenable. It's already the case that building git-annex with the older
+versions of dependencies leaves at least one security hole in a dependency
+open.
+
+This means that additional work needs to be done to review git-annex's
+dependencies on an ongoing basis to detect the addition of LLM generated
+code.
Help with finding these is welcome. Please edit this page and/or file
bug reports on git-annex if it cannot be built without LLM generated code.
diff --git a/git-annex.cabal b/git-annex.cabal
index 2fef9ae934..d627e0f14c 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -146,6 +146,12 @@ Extra-Source-Files:
templates/notifications/longpolling.julius
Utility/libkqueue.h
+Flag NoLLMDependencies
+ Description: Avoid building with versions of dependencies that contain
+ LLM generated code <https://git-annex.branchable.com/no_llm_code/>
+ Default: False
+ Manual: True
+
Flag Assistant
Description: Enable git-annex assistant, webapp, and watch command
Default: True
@@ -213,7 +219,6 @@ custom-setup
Executable git-annex
Main-Is: git-annex.hs
Build-Depends:
- base (>= 4.18.2.1 && < 5),
network-uri (>= 2.6),
optparse-applicative (>= 0.14.2),
containers (>= 0.5.8),
@@ -252,7 +257,6 @@ Executable git-annex
conduit,
time (>= 1.9.1),
persistent-sqlite (>= 2.13.3),
- persistent (>= 2.13.3),
persistent-template (>= 2.8.0),
unliftio-core,
microlens,
@@ -277,22 +281,19 @@ Executable git-annex
tasty-quickcheck,
tasty-rerun,
ansi-terminal >= 0.9,
- aws (>= 0.24.1),
DAV (>= 1.0),
network (>= 3.0.0.0),
network-bsd,
git-lfs (>= 1.2.0),
clock (>= 0.3.0),
- crypton,
- crypton-connection (>= 0.4.3),
- crypton-x509-store,
- tls,
servant,
servant-server,
servant-client,
servant-client-core,
warp (>= 3.2.8),
- warp-tls (>= 3.2.2)
+ warp-tls (>= 3.2.2),
+ crypton-connection (>= 0.4.3),
+ crypton-x509-store
CC-Options: -Wall
GHC-Options: -Wall -fno-warn-tabs -Wincomplete-uni-patterns
Default-Language: Haskell2010
@@ -300,6 +301,22 @@ Executable git-annex
Other-Extensions: TemplateHaskell
-- Some things don't work with the non-threaded RTS.
GHC-Options: -threaded
+
+ if flag(NoLLMDependencies)
+ Build-Depends:
+ base (>= 4.18.2.1 && < 4.23),
+ crypton (< 1.1.0),
+ crypton-x509-validation (< 1.9.0),
+ aws (>= 0.24.1) && (< 2.25.3),
+ tls (< 2.3.0),
+ persistent (>= 2.13.3) && (< 2.15.0.0)
+ else
+ Build-Depends:
+ base (>= 4.18.2.1 && < 5),
+ crypton,
+ aws (>= 0.24.1),
+ tls,
+ persistent (>= 2.13.3)
-- Fully optimize for production.
if flag(Production)
@@ -340,7 +357,10 @@ Executable git-annex
-- Disabled on arm until this issue is resolved:
-- https://github.com/k0001/hs-blake3/issues/8
if flag(Blake3) && (! arch(aarch64) && ! arch(arm))
- Build-Depends: blake3 (>= 0.3)
+ if flag(NoLLMDependencies)
+ Build-Depends: blake3 (>= 0.3) && (< 0.3.1)
+ else
+ Build-Depends: blake3 (>= 0.3)
CPP-Options: -DWITH_BLAKE3
Other-Modules:
Utility.Hash.Blake3
@@ -366,10 +386,6 @@ Executable git-annex
CPP-Options: -DWITH_ASSISTANT -DWITH_WEBAPP
Build-Depends:
mountpoints,
- yesod (>= 1.4.3),
- yesod-static (>= 1.5.1),
- yesod-form (>= 1.4.8),
- yesod-core (>= 1.6.0),
path-pieces (>= 0.2.1),
wai,
wai-extra,
@@ -377,6 +393,18 @@ Executable git-annex
clientsession,
template-haskell,
shakespeare (>= 2.0.11)
+ if flag(NoLLMDependencies)
+ Build-Depends:
+ yesod (>= 1.4.3) && (< 1.7.0.0),
+ yesod-static (>= 1.5.1) && (<1.6.1.3),
+ yesod-form (>= 1.4.8) && (< 1.7.9.3),
+ yesod-core (>= 1.6.0) && (< 1.7.0.0)
+ else
+ Build-Depends:
+ yesod (>= 1.4.3),
+ yesod-static (>= 1.5.1),
+ yesod-form (>= 1.4.8),
+ yesod-core (>= 1.6.0)
Other-Modules:
Assistant
Assistant.Alert
diff --git a/stack-botan.yaml b/stack-botan.yaml
index d3e6d2987a..dfa29f07b1 100644
--- a/stack-botan.yaml
+++ b/stack-botan.yaml
@@ -1,5 +1,6 @@
flags:
git-annex:
+ NoLLMDependencies: false
production: true
parallelbuild: true
assistant: true
diff --git a/stack.yaml b/stack.yaml
index 6399b77762..323e2aa300 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -1,5 +1,6 @@
flags:
git-annex:
+ NoLLMDependencies: false
production: true
parallelbuild: true
assistant: true
toc
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 6da9209c58..a8aa82ec71 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -27,7 +27,9 @@ bug reports on git-annex if it cannot be built without LLM generated code. ## known dependencies that contain LLM generated code -## ghc +[[!toc startlevel=3]] + +### ghc [This commit](https://github.com/ghc/ghc/commit/a5ec467ee3d4e77c026437a545981269acde3434) is probably the first, and will be released in the upcoming ghc 9.15. @@ -65,7 +67,7 @@ But ram is an dependency of other dependencies: [Reverse dependencies of ram](https://packdeps.haskellers.com/reverse/ram) -## persistent +### persistent [persistent](https://hackage.haskell.org/package/persistent) since 2.15.0.0 @@ -73,7 +75,7 @@ But ram is an dependency of other dependencies: git-annex supports being built with older versions. -## yesod +### yesod [yesod](https://hackage.haskell.org/package/yesod-core) since 1.7.0.0 @@ -84,7 +86,7 @@ and 10,000+ lines of changes. (See [[todo/ditch_yesod]]) -## Cabal +### Cabal [First LLM generated code](https://github.com/haskell/cabal/commit/da8b314563feb15a3df7bc1baeef4b7aa08f7578)
cabal
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 586d2fb285..6da9209c58 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -83,3 +83,11 @@ git-annex supports being built with older versions. and 10,000+ lines of changes. (See [[todo/ditch_yesod]]) + +## Cabal + +[First LLM generated code](https://github.com/haskell/cabal/commit/da8b314563feb15a3df7bc1baeef4b7aa08f7578) + +Cabal is needed to build git-annex, but is not linked into it. +There is a risk that a new version of Cabal could need changes to +git-annex.cabal that prevent an old version building it.
yesod-core version
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index cffabf62ef..586d2fb285 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -75,7 +75,7 @@ git-annex supports being built with older versions. ## yesod -[yesod](https://hackage.haskell.org/package/yesod), not yet in a released version +[yesod](https://hackage.haskell.org/package/yesod-core) since 1.7.0.0 [First LLM generated code](https://github.com/yesodweb/yesod/commit/1b033c741ce81d01070de993b285a17e71178156)
yesod
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 1fc22aff29..cffabf62ef 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -27,6 +27,20 @@ bug reports on git-annex if it cannot be built without LLM generated code. ## known dependencies that contain LLM generated code +## ghc + +[This commit](https://github.com/ghc/ghc/commit/a5ec467ee3d4e77c026437a545981269acde3434) +is probably the first, and will be released in the upcoming ghc 9.15. + +git-annex remains buildable with older versions of ghc back to 9.6.6. + +This will probably prevent git-annex from taking advantage of +most new improvements to the Haskell language going forward. +That is deeply unfortunate. This is the main reason why git-annex is not +guaranteed to never change to depend on LLM generated code, because cutting +it off from all future Haskell language improvements may be worse than the +alternative. + ### ram and its reverse dependencies [ram](https://hackage.haskell.org/package/ram) since 0.21.0. @@ -59,16 +73,13 @@ But ram is an dependency of other dependencies: git-annex supports being built with older versions. -## ghc +## yesod -[This commit](https://github.com/ghc/ghc/commit/a5ec467ee3d4e77c026437a545981269acde3434) -is probably the first, and will be released in the upcoming ghc 9.15. +[yesod](https://hackage.haskell.org/package/yesod), not yet in a released version -git-annex remains buildable with older versions of ghc back to 9.6.6. +[First LLM generated code](https://github.com/yesodweb/yesod/commit/1b033c741ce81d01070de993b285a17e71178156) -This will probably prevent git-annex from taking advantage of -most new improvements to the Haskell language going forward. -That is deeply unfortunate. This is the main reason why git-annex is not -guaranteed to never change to depend on LLM generated code, because cutting -it off from all future Haskell language improvements may be worse than the -alternative. +[LLM generated commit with a 1489 line commit message](https://github.com/yesodweb/yesod/commit/1ee25122d82f8f94136bf1496a825c6c00b74fcf) +and 10,000+ lines of changes. + +(See [[todo/ditch_yesod]]) diff --git a/doc/todo/ditch_yesod.mdwn b/doc/todo/ditch_yesod.mdwn index d8ef722ae5..e6c4a4d6e2 100644 --- a/doc/todo/ditch_yesod.mdwn +++ b/doc/todo/ditch_yesod.mdwn @@ -14,6 +14,8 @@ reasons: * I find that Hamlet does not lend itself to being refactored, everything is essentially an IO action with side effects of generating html, so can't really bring proper FP tools to bear. +* Its current maintainer is on the good LLM drugs. + <https://github.com/yesodweb/yesod/commit/1ee25122d82f8f94136bf1496a825c6c00b74fcf> At the moment, servant seems like the nicest place to end up. Just as type safe as yesod afaik, and very lightweight and simple and rather awesome.
weaken
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index 5848705677..1fc22aff29 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -11,9 +11,12 @@ will never do so. However, libraries that git-annex depends on do not generally have such guarantees. Although it would be very much appreciated if they did. -So git-annex is also guaranteed to support being built with versions of +git-annex currently supports being built with versions of dependencies that pre-date any introduction of LLM generated code. (It is not currently built that way by default but such builds are welcome.) +Unfortunately, it's not possible to guarantee that will continue to be the +case in new versions of git-annex. That's the goal, but it may become +untenable. Unfortunately, this means that additional work needs to be done to review git-annex's dependencies on an ongoing basis to detect the addition of LLM @@ -65,4 +68,7 @@ git-annex remains buildable with older versions of ghc back to 9.6.6. This will probably prevent git-annex from taking advantage of most new improvements to the Haskell language going forward. -That is deeply unfortunate. +That is deeply unfortunate. This is the main reason why git-annex is not +guaranteed to never change to depend on LLM generated code, because cutting +it off from all future Haskell language improvements may be worse than the +alternative.
reword
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index c41fd6e185..5848705677 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -63,5 +63,6 @@ is probably the first, and will be released in the upcoming ghc 9.15. git-annex remains buildable with older versions of ghc back to 9.6.6. -It's deeply unfortunate that git-annex will not be able to take advantage -of any new improvements to the Haskell language going forward. +This will probably prevent git-annex from taking advantage of +most new improvements to the Haskell language going forward. +That is deeply unfortunate.
sadness
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index fbc54586cf..c41fd6e185 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -62,3 +62,6 @@ git-annex supports being built with older versions. is probably the first, and will be released in the upcoming ghc 9.15. git-annex remains buildable with older versions of ghc back to 9.6.6. + +It's deeply unfortunate that git-annex will not be able to take advantage +of any new improvements to the Haskell language going forward.
ghc 9.15
diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn index da50b14a47..fbc54586cf 100644 --- a/doc/no_llm_code.mdwn +++ b/doc/no_llm_code.mdwn @@ -58,5 +58,7 @@ git-annex supports being built with older versions. ## ghc -Need to look up the version, but in any case git-annex remains buildable -with older versions of ghc back to 9.6.6. +[This commit](https://github.com/ghc/ghc/commit/a5ec467ee3d4e77c026437a545981269acde3434) +is probably the first, and will be released in the upcoming ghc 9.15. + +git-annex remains buildable with older versions of ghc back to 9.6.6.
LLM dependency tracking
diff --git a/CHANGELOG b/CHANGELOG index b85b0ba223..24f2187984 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,5 +1,9 @@ git-annex (10.20260625) UNRELEASED; urgency=medium + * git-annex is guaranteed to not contain LLM generated code, + and to remain buildable with versions of dependencies that + predate the addition of any LLM generated code. + See https://git-annex.branchable.com/no_llm_code/ * Fix build with time-1.15. -- Joey Hess <id@joeyh.name> Wed, 01 Jul 2026 14:12:59 -0400 diff --git a/doc/future_proofing.mdwn b/doc/future_proofing.mdwn index 84883d060f..8be7197eb4 100644 --- a/doc/future_proofing.mdwn +++ b/doc/future_proofing.mdwn @@ -51,3 +51,5 @@ problem: a remote, as long as you have access to the encryption keys (which for some types of encryption are stored in the git-annex branch of the repository, sometimes encrypted with your gpg key). + +* [[No LLM code]] diff --git a/doc/no_llm_code.mdwn b/doc/no_llm_code.mdwn new file mode 100644 index 0000000000..da50b14a47 --- /dev/null +++ b/doc/no_llm_code.mdwn @@ -0,0 +1,62 @@ +LLM generated code in free software is a potential landmine. The copyright +of such code is an open question, and any current answer to that question +risks changing at some point in the future. + +This is a particular problem for git-annex since [[future proofing]] is an +important aspect of its design. + +So, git-annex does not contain code generated by LLMs and guarantees it +will never do so. + +However, libraries that git-annex depends on do not generally have such +guarantees. Although it would be very much appreciated if they did. + +So git-annex is also guaranteed to support being built with versions of +dependencies that pre-date any introduction of LLM generated code. +(It is not currently built that way by default but such builds are welcome.) + +Unfortunately, this means that additional work needs to be done to review +git-annex's dependencies on an ongoing basis to detect the addition of LLM +generated code. + +Help with finding these is welcome. Please edit this page and/or file +bug reports on git-annex if it cannot be built without LLM generated code. + +## known dependencies that contain LLM generated code + +### ram and its reverse dependencies + +[ram](https://hackage.haskell.org/package/ram) since 0.21.0. + +Note particularly [large LLM generated code churn](https://github.com/jappeace/ram/commit/3a0c034648f1cb7e60e96a681fd74066ff5944fe) +with apparently broken (how?) changes in 0.21.0 being reverted in +0.21.1. + +Rather than depend on ram, git-annex continues to depend on the unmaintained +[memory](//hackage.haskell.org/package/memory). +Since git-annex's use of this package is fairly incidental, this +dependency could be removed if it becomes a problem. + +But ram is an dependency of other dependencies: + +* crypton since 1.1.0 +* crypton-x509-validation since 1.9.0. Note that 1.9.1 fixes + [CVE-2026-9648](https://www.cve.org/CVERecord?id=CVE-2026-9648). +* aws since 0.25.3 (needed to get the CVE-2026-9648 fix) +* tls since 2.3.0 +* blake3 since 0.3.1 (optional git-annex dependency) + +[Reverse dependencies of ram](https://packdeps.haskellers.com/reverse/ram) + +## persistent + +[persistent](https://hackage.haskell.org/package/persistent) since 2.15.0.0 + +[First LLM generated code](https://github.com/yesodweb/persistent/commit/ac0a8698f38ae3b07acdacf0ce236d7fc73bb077) + +git-annex supports being built with older versions. + +## ghc + +Need to look up the version, but in any case git-annex remains buildable +with older versions of ghc back to 9.6.6.
comment
diff --git a/doc/bugs/auth/comment_10_5cfac94f32b17336a81deabe2508c129._comment b/doc/bugs/auth/comment_10_5cfac94f32b17336a81deabe2508c129._comment new file mode 100644 index 0000000000..50c81dfbb9 --- /dev/null +++ b/doc/bugs/auth/comment_10_5cfac94f32b17336a81deabe2508c129._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2026-07-01T20:12:11Z" + content=""" +`git-annex push` won't always trigger this problem because often it has +things to `git push` to the server before sending content. A more reliable +way is to use `git-annex copy --to`. +"""]]
comment
diff --git a/doc/bugs/auth.mdwn b/doc/bugs/auth.mdwn index 170aa3be31..c45cea90ba 100644 --- a/doc/bugs/auth.mdwn +++ b/doc/bugs/auth.mdwn @@ -1,3 +1,6 @@ +[[!meta title="git-credential-oauth does not work with git-annex push to +git-annex p2phttp behind forgejo-aneksajo unless git push primes the credential cache first"]] + ### Please describe the problem. `git annex push` does not trigger git credential retrieval reliably, but forces manual credential entry. diff --git a/doc/bugs/auth/comment_9_c5b7872812b6aa29224f45b266a7e87e._comment b/doc/bugs/auth/comment_9_c5b7872812b6aa29224f45b266a7e87e._comment new file mode 100644 index 0000000000..08738e4aa6 --- /dev/null +++ b/doc/bugs/auth/comment_9_c5b7872812b6aa29224f45b266a7e87e._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 9""" + date="2026-07-01T20:00:36Z" + content=""" +`git-annex push` is talking to `git-annex p2phttp` which is on a +`Forgejo-aneksajo` server. + +In this case, the server responds with HTTP 401, and I have verified that +response does *not* include the `www-authenticate` header. + +So, git-annex's code to send that header on to `git-credential` seems fine, +and this needs to be fixed in either p2phttp or Forgejo-aneksajo to make +the header be sent. I suspect it will need to be done in Forgejo-aneksajo. +"""]]
comment
diff --git a/doc/bugs/auth/comment_8_d6b94d6eef80a8711899f008bb31609b._comment b/doc/bugs/auth/comment_8_d6b94d6eef80a8711899f008bb31609b._comment new file mode 100644 index 0000000000..e56517d8ac --- /dev/null +++ b/doc/bugs/auth/comment_8_d6b94d6eef80a8711899f008bb31609b._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 8""" + date="2026-07-01T19:40:42Z" + content=""" +The wwwauth field is the relevant one. + +There was a past bug about a very similar problem: +<https://git-annex.branchable.com/bugs/__96__git_annex_push__96___does_not_use_git-credential-oauth/> + +[[!commit 32fac0b103db1982b748f15def825e1490d34f50]] was supposed to fix +that, by passing the www-authenticate http header in to git credential. + +This bug must be a case where that doesn't happen. +"""]]
comment
diff --git a/doc/bugs/auth/comment_7_fcb6f9dacdae34ee522d0efd53038eed._comment b/doc/bugs/auth/comment_7_fcb6f9dacdae34ee522d0efd53038eed._comment new file mode 100644 index 0000000000..54f2b67f4e --- /dev/null +++ b/doc/bugs/auth/comment_7_fcb6f9dacdae34ee522d0efd53038eed._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2026-07-01T19:34:30Z" + content=""" +@mih told me the url privately. (There is only 1.) + +It is indeed a host which when passed to `git-credential-oauth get` +it decides not to do anything. + +But, `git push` to the same url uses `git-credential-oauth get` and it +*does* do oauth. + +Dumping what git sends to it, there are additional fields beyond what +git-annex sends: + + capability[]=authtype + capability[]=state + protocol=https + host=<obscured> + wwwauth[]=Basic realm="Gitea" +"""]]
Fix build with time-1.15
It added a Seconds pattern, so rename Seconds to SecondsDelay to avoid
conflict. Also renamed Microseconds to MicrosecondsDelay; although time
does not have Microseconds, it has Picoseconds, so might add Microseconds
later..
It added a Seconds pattern, so rename Seconds to SecondsDelay to avoid
conflict. Also renamed Microseconds to MicrosecondsDelay; although time
does not have Microseconds, it has Picoseconds, so might add Microseconds
later..
diff --git a/Annex/Init.hs b/Annex/Init.hs
index e610fbce00..7966ce7595 100644
--- a/Annex/Init.hs
+++ b/Annex/Init.hs
@@ -428,7 +428,7 @@ probeLockSupport = withEventuallyCleanedOtherTmp $ \tmp -> do
return ok
warnstall annexrunner = do
- threadDelaySeconds (Seconds 10)
+ threadDelaySeconds (SecondsDelay 10)
annexrunner $ do
warning "Probing the filesystem for POSIX fcntl lock support is taking a long time."
warning "(Setting annex.pidlock will avoid this probe.)"
diff --git a/Annex/StallDetection.hs b/Annex/StallDetection.hs
index 9b885c2ecf..9e6f67ba76 100644
--- a/Annex/StallDetection.hs
+++ b/Annex/StallDetection.hs
@@ -55,7 +55,7 @@ detectStalls (Just (StallDetection bwrate@(BwRate _minsz duration))) metervar on
let BwRate scaledminsz scaledduration = upscale bwrate timepassed
detectStalls' scaledminsz scaledduration metervar onstall v
where
- minwaitsecs = Seconds $
+ minwaitsecs = SecondsDelay $
min 60 (fromIntegral (durationSeconds duration))
waitforfirstupdate startval = do
liftIO $ threadDelaySeconds minwaitsecs
@@ -75,7 +75,7 @@ detectStalls (Just ProbeStallDetection) metervar onstall = do
where
duration = Duration 60
- delay = Seconds (fromIntegral (durationSeconds duration) `div` 2)
+ delay = SecondsDelay (fromIntegral (durationSeconds duration) `div` 2)
waitforfirstupdate startval = do
liftIO $ threadDelaySeconds delay
@@ -115,7 +115,7 @@ detectStalls' minsz duration metervar onstall st = do
| sofar - prev < minsz -> onstall
| otherwise -> cont
where
- delay = Seconds (fromIntegral (durationSeconds duration))
+ delay = SecondsDelay (fromIntegral (durationSeconds duration))
readMeterVar
:: MonadIO m
diff --git a/Annex/Transfer.hs b/Annex/Transfer.hs
index 7ec629e442..191c7061fa 100644
--- a/Annex/Transfer.hs
+++ b/Annex/Transfer.hs
@@ -360,19 +360,19 @@ forwardRetry numretries old new
- by git configuration. -}
configuredRetry :: RetryDecider
configuredRetry numretries _old new = do
- (maxretries, Seconds initretrydelay) <- getcfg $
+ (maxretries, SecondsDelay initretrydelay) <- getcfg $
Remote.gitconfig <$> transferRemote new
if numretries < maxretries
then do
- let retrydelay = Seconds (initretrydelay * 2^(numretries-1))
- showSideAction $ UnquotedString $ "Delaying " ++ show (fromSeconds retrydelay) ++ "s before retrying."
+ let retrydelay = SecondsDelay (initretrydelay * 2^(numretries-1))
+ showSideAction $ UnquotedString $ "Delaying " ++ show (fromSecondsDelay retrydelay) ++ "s before retrying."
liftIO $ threadDelaySeconds retrydelay
return True
else return False
where
globalretrycfg = fromMaybe 0 . annexRetry
<$> Annex.getGitConfig
- globalretrydelaycfg = fromMaybe (Seconds 1) . annexRetryDelay
+ globalretrydelaycfg = fromMaybe (SecondsDelay 1) . annexRetryDelay
<$> Annex.getGitConfig
getcfg Nothing = (,) <$> globalretrycfg <*> globalretrydelaycfg
getcfg (Just gc) = (,)
diff --git a/Assistant/CredPairCache.hs b/Assistant/CredPairCache.hs
index eba3d2b779..a645840b15 100644
--- a/Assistant/CredPairCache.hs
+++ b/Assistant/CredPairCache.hs
@@ -27,7 +27,7 @@ import Control.Concurrent
- Note that repeatedly caching the same CredPair
- does not reset its expiry time.
-}
-cacheCred :: CredPair -> Seconds -> Assistant ()
+cacheCred :: CredPair -> SecondsDelay -> Assistant ()
cacheCred (login, password) expireafter = do
cache <- getAssistant credPairCache
liftIO $ do
diff --git a/Assistant/Repair.hs b/Assistant/Repair.hs
index caa02138e2..978d59b080 100644
--- a/Assistant/Repair.hs
+++ b/Assistant/Repair.hs
@@ -149,7 +149,7 @@ repairStaleLocks lockfiles = go =<< getsizes
go [] = return ()
go l = whenM (liftIO $ null <$> Lsof.query ("--" : map (fromOsPath . fst) l)) $ do
debug ["Waiting for 60 seconds to check stale git lock file"]
- liftIO $ threadDelaySeconds $ Seconds 60
+ liftIO $ threadDelaySeconds $ SecondsDelay 60
l' <- getsizes
if l' == l
then liftIO $ mapM_ (removeWhenExistsWith removeFile . fst) l
diff --git a/Assistant/Restart.hs b/Assistant/Restart.hs
index 58a17c12a2..5a3f03b943 100644
--- a/Assistant/Restart.hs
+++ b/Assistant/Restart.hs
@@ -52,7 +52,7 @@ postRestart url = do
modifyDaemonStatus_ $ \status -> status { globalRedirUrl = Just url }
liftIO . sendNotification . globalRedirNotifier =<< getDaemonStatus
void $ liftIO $ forkIO $ do
- threadDelaySeconds (Seconds 120)
+ threadDelaySeconds (SecondsDelay 120)
terminateSelf
terminateSelf :: IO ()
diff --git a/Assistant/Threads/Committer.hs b/Assistant/Threads/Committer.hs
index 7c11ecc05c..210f096a21 100644
--- a/Assistant/Threads/Committer.hs
+++ b/Assistant/Threads/Committer.hs
@@ -59,7 +59,7 @@ commitThread :: NamedThread
commitThread = namedThread "Committer" $ do
havelsof <- liftIO $ inSearchPath "lsof"
delayadd <- liftAnnex $
- fmap Seconds . annexDelayAdd <$> Annex.getGitConfig
+ fmap SecondsDelay . annexDelayAdd <$> Annex.getGitConfig
largefilematcher <- liftAnnex largeFilesMatcher
annexdotfiles <- liftAnnex $ getGitConfigVal annexDotFiles
addunlockedmatcher <- liftAnnex $
@@ -111,7 +111,7 @@ waitChangeTime a = waitchanges 0
where
waitchanges lastcommitsize = do
-- Wait one one second as a simple rate limiter.
- liftIO $ threadDelaySeconds (Seconds 1)
+ liftIO $ threadDelaySeconds (SecondsDelay 1)
-- Now, wait until at least one change is available for
-- processing.
cs <- getChanges
@@ -193,7 +193,7 @@ waitChangeTime a = waitchanges 0
loop 0 = continue oldchanges
loop n = do
liftAnnex noop -- ensure Annex state is free
- liftIO $ threadDelaySeconds (Seconds 1)
+ liftIO $ threadDelaySeconds (SecondsDelay 1)
changes <- getAnyChanges
if null changes
then loop (n - 1)
@@ -280,7 +280,7 @@ commitStaged msg = do
- Any pending adds that are not ready yet are put back into the ChangeChan,
- where they will be retried later.
-}
-handleAdds :: OsPath -> Bool -> GetFileMatcher -> Bool -> Maybe AddUnlockedMatcher -> Maybe Seconds -> [Change] -> Assistant [Change]
+handleAdds :: OsPath -> Bool -> GetFileMatcher -> Bool -> Maybe AddUnlockedMatcher -> Maybe SecondsDelay -> [Change] -> Assistant [Change]
handleAdds lockdowndir havelsof largefilematcher annexdotfiles addunlockedmatcher delayadd cs = returnWhen (null incomplete) $ do
let (pending, inprocess) = partition isPendingAddChange incomplete
let lockdownconfig = LockDownConfig
@@ -467,7 +467,7 @@ handleAdds lockdowndir havelsof largefilematcher annexdotfiles addunlockedmatche
-
- Check by running lsof on the repository.
-}
-safeToAdd :: OsPath -> LockDownConfig -> Bool -> Maybe Seconds -> [Change] -> [Change] -> Assistant [Either Change Change]
+safeToAdd :: OsPath -> LockDownConfig -> Bool -> Maybe SecondsDelay -> [Change] -> [Change] -> Assistant [Either Change Change]
safeToAdd _ _ _ _ [] [] = return []
safeToAdd lockdowndir lockdownconfig havelsof delayadd pending inprocess = do
maybe noop (liftIO . threadDelaySeconds) delayadd
diff --git a/Assistant/Threads/ConfigMonitor.hs b/Assistant/Threads/ConfigMonitor.hs
index 97cd4af8bb..9a251616c5 100644
--- a/Assistant/Threads/ConfigMonitor.hs
+++ b/Assistant/Threads/ConfigMonitor.hs
@@ -50,7 +50,7 @@ configMonitorThread = namedThread "ConfigMonitor" $ loop =<< getConfigs
{- Record a commit to get this config
- change pushed out to remotes. -}
recordCommit
- liftIO $ threadDelaySeconds (Seconds 60)
+ liftIO $ threadDelaySeconds (SecondsDelay 60)
loop new
{- Config files, and their checksums. -}
diff --git a/Assistant/Threads/Cronner.hs b/Assistant/Threads/Cronner.hs
index 9b063b5882..21282e0291 100644
--- a/Assistant/Threads/Cronner.hs
+++ b/Assistant/Threads/Cronner.hs
@@ -116,8 +116,8 @@ sleepingActivityThread urlrenderer activity lasttime = go lasttime =<< getnextti
desc = fromScheduledActivity activity
schedule = getSchedule activity
waitrun l t mmaxt = do
- seconds <- liftIO $ secondsUntilLocalTime t
- when (seconds > Seconds 0) $ do
+ seconds <- liftIO $ secondsDelayUntilLocalTime t
+ when (seconds > SecondsDelay 0) $ do
debug ["waiting", show seconds, "for next scheduled", desc]
liftIO $ threadDelaySeconds seconds
now <- liftIO getCurrentTime
@@ -161,14 +161,14 @@ remoteActivityThread urlrenderer mvar activity lasttime = do
go _ = noop -- running at exact time not handled here
loop = remoteActivityThread urlrenderer mvar activity
-secondsUntilLocalTime :: LocalTime -> IO Seconds
-secondsUntilLocalTime t = do
+secondsDelayUntilLocalTime :: LocalTime -> IO SecondsDelay
+secondsDelayUntilLocalTime t = do
now <- getCurrentTime
tz <- getTimeZone now
let secs = truncate $ diffUTCTime (localTimeToUTC tz t) now
return $ if secs > 0
(Diff truncated)
formatting
diff --git a/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment b/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment index b6eabb917d..1aaf96e8de 100644 --- a/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment +++ b/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment @@ -4,8 +4,8 @@ date="2026-07-01T17:00:06Z" content=""" In your bug report I noticed that the password prompt is for -"https://oauth2@<host>" but when you git push it's to -"https://<host>/<repo>.git". I'm not clear on if these were +`"https://oauth2@<host>"` but when you git push it's to +`"https://<host>/<repo>.git"`. I'm not clear on if these were really different urls or if it was a mistake in your sanitization. If there are different urls, `git-credential-oauth` could have acted
comment
diff --git a/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment b/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment new file mode 100644 index 0000000000..b6eabb917d --- /dev/null +++ b/doc/bugs/auth/comment_6_896c29667ae2774d65caafd2a4472e60._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2026-07-01T17:00:06Z" + content=""" +In your bug report I noticed that the password prompt is for +"https://oauth2@<host>" but when you git push it's to +"https://<host>/<repo>.git". I'm not clear on if these were +really different urls or if it was a mistake in your sanitization. + +If there are different urls, `git-credential-oauth` could have acted +on one and not on the other. Although I don't understand why running +`git pull` would then cache a credential that would be used on a different url. + +Assuming you're using `git-credential-oauth` from eg the Debian package of +that name, here's an example of it not acting on an url: + + joey@darkstar:~/tmp/x>(echo 'protocol=https'; echo 'host=example.com'; echo 'username=oauth2') | git-credential-oauth get + joey@darkstar:~/tmp/x> + +Running it the same way with github.com doesn't behave that way: + + joey@darkstar:~/tmp/x>(echo 'protocol=https'; echo 'host=github.com'; echo 'username=oauth2') | git-credential-oauth get + Please complete authentication in your browser... + +This behavior is not network dependent. In fact, `git-credential-oauth` has a +pile of special casing for different hosts and different servers, including a +looksLikeForgejo case. If it does not recognize the server name, it doesn't do +anything without further configuration. +"""]]
comment
diff --git a/doc/bugs/auth/comment_5_d8f1045d7fe42f74e10766d791bcbfd4._comment b/doc/bugs/auth/comment_5_d8f1045d7fe42f74e10766d791bcbfd4._comment new file mode 100644 index 0000000000..9298a5fbf1 --- /dev/null +++ b/doc/bugs/auth/comment_5_d8f1045d7fe42f74e10766d791bcbfd4._comment @@ -0,0 +1,36 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2026-07-01T15:52:14Z" + content=""" +The "Password for" prompt comes from git-credential, not git-annex. + +I was able to reproduce that behavior using a `git-credential-foo` +"helper" which just exits successfully but does not output any credentials. +With the git config adjusted to use that (as well as cache), git runs +it but then falls back to the password prompt: + + joey@darkstar:~/tmp/a1>GIT_TRACE=2 git push foo + 12:08:19.869469 git.c:502 trace: built-in: git push foo + 12:08:19.869942 run-command.c:673 trace: run_command: GIT_DIR=.git git remote-https foo https://github.com/joeyh/gitannexbuilder + 12:08:19.869965 run-command.c:765 trace: start_command: /usr/lib/git-core/git remote-https foo https://github.com/joeyh/gitannexbuilder + 12:08:19.873067 git.c:808 trace: exec: git-remote-https foo https://github.com/joeyh/gitannexbuilder + 12:08:19.873107 run-command.c:673 trace: run_command: git-remote-https foo https://github.com/joeyh/gitannexbuilder + 12:08:19.873121 run-command.c:765 trace: start_command: /usr/lib/git-core/git-remote-https foo https://github.com/joeyh/gitannexbuilder + 12:08:20.549858 run-command.c:673 trace: run_command: 'git credential-cache get' + 12:08:20.549927 run-command.c:765 trace: start_command: /bin/sh -c 'git credential-cache get' 'git credential-cache get' + 12:08:20.555753 git.c:502 trace: built-in: git credential-cache get + 12:08:20.556612 run-command.c:673 trace: run_command: 'git credential-foo get' + 12:08:20.556635 run-command.c:765 trace: start_command: /bin/sh -c 'git credential-foo get' 'git credential-foo get' + 12:08:20.561465 git.c:808 trace: exec: git-credential-foo get + 12:08:20.561506 run-command.c:673 trace: run_command: git-credential-foo get + 12:08:20.561530 run-command.c:765 trace: start_command: /home/joey/bin/git-credential-foo get + Username for 'https://github.com': + +So it seems likely that `git-credential-oauth` is just choosing not to do +anything when git-annex runs it, and then git is falling back like that. + +Why it would choose to do that I don't know, but I suppose it could be a +difference in that is sent to it on stdin when called by git-annex vs +by git. +"""]]
comment
diff --git a/doc/bugs/auth/comment_4_d62d27ba12c59f1f0b49a2140276b5b4._comment b/doc/bugs/auth/comment_4_d62d27ba12c59f1f0b49a2140276b5b4._comment new file mode 100644 index 0000000000..74b0b13458 --- /dev/null +++ b/doc/bugs/auth/comment_4_d62d27ba12c59f1f0b49a2140276b5b4._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2026-07-01T15:43:12Z" + content=""" +At the risk of introducing a second red herring, this seems similar to +[[bugs/git_annex_checkpresentkey_removes_git_credentials]]. But I don't +think the same, because here `git-annex push` works after `git push` has +cached a credential, so git-annex is apparently not removing the cached +credential. + +Could you link to the forgejo-aneksajo issue that is/was being dealt with? +"""]]
diff --git a/doc/bugs/time_1.15__58___Ambiguous_occurrence___8216__Seconds__8217__.mdwn b/doc/bugs/time_1.15__58___Ambiguous_occurrence___8216__Seconds__8217__.mdwn index de5c58ee7c..4812411c79 100644 --- a/doc/bugs/time_1.15__58___Ambiguous_occurrence___8216__Seconds__8217__.mdwn +++ b/doc/bugs/time_1.15__58___Ambiguous_occurrence___8216__Seconds__8217__.mdwn @@ -1,6 +1,13 @@ ## Please describe the problem. -Building git-annex with GHC 9.14/time 1.15 fails with error "Ambiguous occurrence ‘Seconds’." in Annex/StallDetection.hs +Building git-annex with GHC 9.14/time 1.15 fails with error "Ambiguous occurrence ‘Seconds’." in + +- Annex/StallDetection.hs, +- Assistant/Threads/Committer.hs, +- Assistant/Threads/Cronner.hs, +- Assistant/Threads/Exporter.hs +- Assistant/Threads/Watcher.hs +- Command/FuzzTest.hs time 1.15 adds "Seconds" to Data.Time.Clock, leading to ambiguity. Easiest fix would probably be to hide the export, alternatively use qualified import.