Recent changes to this wiki:
Added a comment: Exporting a synced/* branch works now! 🥳
diff --git a/doc/bugs/Can__39__t_export_synced__47___branches/comment_2_6f8792b4b31f49712f6707d57a2c3115._comment b/doc/bugs/Can__39__t_export_synced__47___branches/comment_2_6f8792b4b31f49712f6707d57a2c3115._comment new file mode 100644 index 0000000000..8effe18dd5 --- /dev/null +++ b/doc/bugs/Can__39__t_export_synced__47___branches/comment_2_6f8792b4b31f49712f6707d57a2c3115._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="nobodyinperson" + avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5" + subject="Exporting a synced/* branch works now! 🥳" + date="2025-01-21T18:30:50Z" + content=""" +Thank you very much, I can confirm that it works with git-annex 10.20250116-g8d80b07f5aee368579e04b2acba56a8821eeaab0. 👍 +"""]]
RawFilePath conversion of System.Directory
By using System.Directory.OsPath, which takes and returns OsString,
which is a ShortByteString. So, things like dirContents currently have the
overhead of copying that to a ByteString, but that should be less than
the overhead of using Strings which often in turn were converted to
RawFilePaths.
Added Utility.OsString and the OsString build flag. That flag is turned
on in the stack.yaml, and will be turned on automatically by cabal when
built with new enough libraries. The stack.yaml change is a bit ugly,
and that could be reverted for now if it causes any problems.
Note that Utility.OsString.toOsString on windows is avoiding only a
check of encoding that is documented as being unlikely to fail. I don't
think it can fail in git-annex; if it could, git-annex didn't contain
such an encoding check before, so at worst that should be a wash.
By using System.Directory.OsPath, which takes and returns OsString,
which is a ShortByteString. So, things like dirContents currently have the
overhead of copying that to a ByteString, but that should be less than
the overhead of using Strings which often in turn were converted to
RawFilePaths.
Added Utility.OsString and the OsString build flag. That flag is turned
on in the stack.yaml, and will be turned on automatically by cabal when
built with new enough libraries. The stack.yaml change is a bit ugly,
and that could be reverted for now if it causes any problems.
Note that Utility.OsString.toOsString on windows is avoiding only a
check of encoding that is documented as being unlikely to fail. I don't
think it can fail in git-annex; if it could, git-annex didn't contain
such an encoding check before, so at worst that should be a wash.
diff --git a/Annex/AdjustedBranch/Merge.hs b/Annex/AdjustedBranch/Merge.hs index 904f4ee412..5a88a8e79f 100644 --- a/Annex/AdjustedBranch/Merge.hs +++ b/Annex/AdjustedBranch/Merge.hs @@ -29,6 +29,7 @@ import Annex.GitOverlay import Utility.Tmp.Dir import Utility.CopyFile import Utility.Directory.Create +import qualified Utility.RawFilePath as R import qualified Data.ByteString as S import qualified System.FilePath.ByteString as P @@ -72,7 +73,6 @@ mergeToAdjustedBranch tomerge (origbranch, adj) mergeconfig canresolvemerge comm -} changestomerge (Just updatedorig) = withOtherTmp $ \othertmpdir -> do git_dir <- fromRepo Git.localGitDir - let git_dir' = fromRawFilePath git_dir tmpwt <- fromRepo gitAnnexMergeDir withTmpDirIn (fromRawFilePath othertmpdir) "git" $ \tmpgit -> withWorkTreeRelated tmpgit $ withemptydir git_dir tmpwt $ withWorkTree tmpwt $ do @@ -82,16 +82,15 @@ mergeToAdjustedBranch tomerge (origbranch, adj) mergeconfig canresolvemerge comm -- causes it not to look in GIT_DIR for refs. refs <- liftIO $ emptyWhenDoesNotExist $ dirContentsRecursive $ - git_dir' </> "refs" - let refs' = (git_dir' </> "packed-refs") : refs + git_dir P.</> "refs" + let refs' = (git_dir P.</> "packed-refs") : refs liftIO $ forM_ refs' $ \src -> do - let src' = toRawFilePath src - whenM (doesFileExist src) $ do - dest <- relPathDirToFile git_dir src' + whenM (R.doesPathExist src) $ do + dest <- relPathDirToFile git_dir src let dest' = toRawFilePath tmpgit P.</> dest createDirectoryUnder [git_dir] (P.takeDirectory dest') - void $ createLinkOrCopy src' dest' + void $ createLinkOrCopy src dest' -- This reset makes git merge not care -- that the work tree is empty; otherwise -- it will think that all the files have diff --git a/Annex/Branch.hs b/Annex/Branch.hs index ce4c3ad85e..2474b2ae13 100644 --- a/Annex/Branch.hs +++ b/Annex/Branch.hs @@ -753,7 +753,7 @@ stageJournal jl commitindex = withIndex $ withOtherTmp $ \tmpdir -> do Nothing -> return () Just file -> do let path = dir P.</> toRawFilePath file - unless (dirCruft file) $ whenM (isfile path) $ do + unless (dirCruft (toRawFilePath file)) $ whenM (isfile path) $ do sha <- Git.HashObject.hashFile h path hPutStrLn jlogh file streamer $ Git.UpdateIndex.updateIndexLine diff --git a/Annex/Content.hs b/Annex/Content.hs index aba53add7b..40f13e7ea5 100644 --- a/Annex/Content.hs +++ b/Annex/Content.hs @@ -817,7 +817,7 @@ listKeys' keyloc want = do s <- Annex.getState id r <- Annex.getRead id depth <- gitAnnexLocationDepth <$> Annex.getGitConfig - liftIO $ walk (s, r) depth (fromRawFilePath dir) + liftIO $ walk (s, r) depth dir where walk s depth dir = do contents <- catchDefaultIO [] (dirContents dir) @@ -825,7 +825,7 @@ listKeys' keyloc want = do then do contents' <- filterM present contents keys <- filterM (Annex.eval s . want) $ - mapMaybe (fileKey . P.takeFileName . toRawFilePath) contents' + mapMaybe (fileKey . P.takeFileName) contents' continue keys [] else do let deeper = walk s (depth - 1) @@ -843,8 +843,8 @@ listKeys' keyloc want = do present _ | inanywhere = pure True present d = presentInAnnex d - presentInAnnex = doesFileExist . contentfile - contentfile d = d </> takeFileName d + presentInAnnex = R.doesPathExist . contentfile + contentfile d = d P.</> P.takeFileName d {- Things to do to record changes to content when shutting down. - diff --git a/Annex/RepoSize/LiveUpdate.hs b/Annex/RepoSize/LiveUpdate.hs index a792b42597..8710282999 100644 --- a/Annex/RepoSize/LiveUpdate.hs +++ b/Annex/RepoSize/LiveUpdate.hs @@ -161,7 +161,7 @@ checkStaleSizeChanges h@(RepoSizeHandle (Just _) livev) = do where go livedir lck pidlockfile now = do void $ tryNonAsync $ do - lockfiles <- liftIO $ filter (not . dirCruft) + lockfiles <- liftIO $ filter (not . dirCruft . toRawFilePath) <$> getDirectoryContents (fromRawFilePath livedir) stale <- forM lockfiles $ \lockfile -> if (lockfile /= pidlockfile) diff --git a/Annex/Ssh.hs b/Annex/Ssh.hs index 90d462f7be..6cdfba7b02 100644 --- a/Annex/Ssh.hs +++ b/Annex/Ssh.hs @@ -5,6 +5,7 @@ - Licensed under the GNU AGPL version 3 or higher. -} +{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE CPP #-} module Annex.Ssh ( @@ -100,15 +101,16 @@ consumeStdinParams NoConsumeStdin = [Param "-n"] {- Returns a filename to use for a ssh connection caching socket, and - parameters to enable ssh connection caching. -} -sshCachingInfo :: (SshHost, Maybe Integer) -> Annex (Maybe FilePath, [CommandParam]) +sshCachingInfo :: (SshHost, Maybe Integer) -> Annex (Maybe RawFilePath, [CommandParam]) sshCachingInfo (host, port) = go =<< sshCacheDir' where go (Right dir) = liftIO (bestSocketPath $ dir P.</> hostport2socket host port) >>= return . \case Nothing -> (Nothing, []) Just socketfile -> - let socketfile' = fromRawFilePath socketfile - in (Just socketfile', sshConnectionCachingParams socketfile') + (Just socketfile + , sshConnectionCachingParams (fromRawFilePath socketfile) + ) -- No connection caching with concurrency is not a good -- combination, so warn the user. go (Left whynocaching) = do @@ -214,7 +216,7 @@ portParams (Just port) = [Param "-p", Param $ show port] - Locks the socket lock file to prevent other git-annex processes from - stopping the ssh multiplexer on this socket. -} -prepSocket :: FilePath -> SshHost -> [CommandParam] -> Annex () +prepSocket :: RawFilePath -> SshHost -> [CommandParam] -> Annex () prepSocket socketfile sshhost sshparams = do -- There could be stale ssh connections hanging around -- from a previous git-annex run that was interrupted. @@ -286,13 +288,13 @@ prepSocket socketfile sshhost sshparams = do - and this check makes such files be skipped since the corresponding lock - file won't exist. -} -enumSocketFiles :: Annex [FilePath] +enumSocketFiles :: Annex [RawFilePath] enumSocketFiles = liftIO . go =<< sshCacheDir where go Nothing = return [] go (Just dir) = filterM (R.doesPathExist . socket2lock) =<< filter (not . isLock) - <$> catchDefaultIO [] (dirContents (fromRawFilePath dir)) + <$> catchDefaultIO [] (dirContents dir) {- Stop any unused ssh connection caching processes. -} sshCleanup :: Annex () @@ -324,9 +326,9 @@ sshCleanup = mapM_ cleanup =<< enumSocketFiles forceSshCleanup :: Annex () forceSshCleanup = mapM_ forceStopSsh =<< enumSocketFiles -forceStopSsh :: FilePath -> Annex () +forceStopSsh :: RawFilePath -> Annex () forceStopSsh socketfile = withNullHandle $ \nullh -> do - let (dir, base) = splitFileName socketfile + let (dir, base) = splitFileName (fromRawFilePath socketfile) let p = (proc "ssh" $ toCommand $ [ Param "-O", Param "stop" ] ++ sshConnectionCachingParams base ++ @@ -338,7 +340,7 @@ forceStopSsh socketfile = withNullHandle $ \nullh -> do } void $ liftIO $ catchMaybeIO $ withCreateProcess p $ \_ _ _ pid -> forceSuccessProcess p pid - liftIO $ removeWhenExistsWith R.removeLink (toRawFilePath socketfile) + liftIO $ removeWhenExistsWith R.removeLink socketfile {- This needs to be as short as possible, due to limitations on the length - of the path to a socket file. At the same time, it needs to be unique @@ -355,13 +357,13 @@ hostport2socket' s where lengthofmd5s = 32 -socket2lock :: FilePath -> RawFilePath -socket2lock socket = toRawFilePath (socket ++ lockExt) +socket2lock :: RawFilePath -> RawFilePath +socket2lock socket = socket <> lockExt -isLock :: FilePath -> Bool -isLock f = lockExt `isSuffixOf` f +isLock :: RawFilePath -> Bool +isLock f = lockExt `S.isSuffixOf` f -lockExt :: String +lockExt :: S.ByteString lockExt = ".lock" {- This is the size of the sun_path component of sockaddr_un, which diff --git a/Annex/Tmp.hs b/Annex/Tmp.hs index 2bbebd6388..6f9f28b8b6 100644 (Diff truncated)
update
diff --git a/doc/todo/RawFilePath_conversion.mdwn b/doc/todo/RawFilePath_conversion.mdwn index efb32e5e06..a8b9af18d2 100644 --- a/doc/todo/RawFilePath_conversion.mdwn +++ b/doc/todo/RawFilePath_conversion.mdwn @@ -12,10 +12,11 @@ status. libraries like directory now have versions that operate on OSPath. That could be used in git-annex eg for things like getDirectoryContents, when built against those versions. + (OSPath uses ShortByteString, while RawFilePath is ByteString, so + conversion still entails a copy, eg using + `System.OsString.Internal.fromBytes`) * unix has modules that operate on RawFilePath but no OSPath versions yet. See https://github.com/haskell/unix/issues/240 - (OSPath uses ShortByteString, while RawFilePath is ByteString, so - conversion can entail a copy.) * filepath-1.4.100 implements support for OSPath. It is bundled with ghc-9.6.1 and above. Will need to switch from filepath-bytestring to this, and to avoid a lot of ifdefs, probably only after git-annex no
update
diff --git a/doc/todo/RawFilePath_conversion.mdwn b/doc/todo/RawFilePath_conversion.mdwn index b1e1d3e9cd..efb32e5e06 100644 --- a/doc/todo/RawFilePath_conversion.mdwn +++ b/doc/todo/RawFilePath_conversion.mdwn @@ -9,14 +9,20 @@ But this conversion is not yet complete. This is a todo to keep track of the status. * The Abstract FilePath proposal (AFPP) has been implemented, and so a number of - libraries like unix and directory now have versions that operate on + libraries like directory now have versions that operate on OSPath. That could be used in git-annex eg for things like getDirectoryContents, when built against those versions. - (But OSPath uses ShortByteString, while RawFilePath is ByteString, so - conversion still entails a copy.) +* unix has modules that operate on RawFilePath but no OSPath versions yet. + See https://github.com/haskell/unix/issues/240 + (OSPath uses ShortByteString, while RawFilePath is ByteString, so + conversion can entail a copy.) +* filepath-1.4.100 implements support for OSPath. It is bundled with + ghc-9.6.1 and above. Will need to switch from filepath-bytestring to + this, and to avoid a lot of ifdefs, probably only after git-annex no + longers supports building with older ghc versions. * withFile remains to be converted, and is used in several important code paths, including Annex.Journal and Annex.Link. - There is a RawFilePath version in file-io library, but that is + There is a OSPath version in file-io library, but that is not currently a git-annex dependency. (withFile is in base, and base is unlikely to convert to AFPP soon)
relax annex-tracking-branch to allow "/"
Allow setting remote.foo.annex-tracking-branch to a branch name that
contains "/", as long as it's not a remote tracking branch.
Allow setting remote.foo.annex-tracking-branch to a branch name that
contains "/", as long as it's not a remote tracking branch.
diff --git a/CHANGELOG b/CHANGELOG index 15833a80c1..fa11259b2b 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,6 +1,8 @@ git-annex (10.20250116) UNRELEASED; urgency=medium * Support help.autocorrect settings "prompt", "never", and "immediate". + * Allow setting remote.foo.annex-tracking-branch to a branch name + that contains "/", as long as it's not a remote tracking branch. -- Joey Hess <id@joeyh.name> Mon, 20 Jan 2025 10:24:51 -0400 diff --git a/Command/Export.hs b/Command/Export.hs index ebbcb51ea8..4e87323bf3 100644 --- a/Command/Export.hs +++ b/Command/Export.hs @@ -120,15 +120,21 @@ seekExport r tree mtbcommitsha srcrs = do -- branch. getExportCommit :: Remote -> Git.Ref -> Annex (Maybe (RemoteTrackingBranch, Sha)) getExportCommit r treeish - | '/' `notElem` fromRef baseref = do - let tb = mkRemoteTrackingBranch r baseref - commitsha <- inRepo $ Git.Ref.sha $ Git.Ref.underBase refsheads baseref - return (fmap (tb, ) commitsha) - | otherwise = return Nothing + | '/' `notElem` fromRef baseref = go + | otherwise = ifM isremoteref + ( return Nothing + , go + ) where baseref = Ref $ S8.takeWhile (/= ':') $ fromRef' $ Git.Ref.removeBase refsheads treeish refsheads = "refs/heads" + isremoteref = inRepo $ Git.Ref.exists $ + Git.Ref.underBase "refs/remotes" baseref + go = do + let tb = mkRemoteTrackingBranch r baseref + commitsha <- inRepo $ Git.Ref.sha $ Git.Ref.underBase refsheads baseref + return (fmap (tb, ) commitsha) -- | Changes what's exported to the remote. Does not upload any new -- files, but does delete and rename files already exported to the remote. diff --git a/doc/bugs/Can__39__t_export_synced__47___branches.mdwn b/doc/bugs/Can__39__t_export_synced__47___branches.mdwn index 19085dbe88..20385272b1 100644 --- a/doc/bugs/Can__39__t_export_synced__47___branches.mdwn +++ b/doc/bugs/Can__39__t_export_synced__47___branches.mdwn @@ -66,3 +66,5 @@ export myexport bla ok ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) git-annex is amazing, thank you! 🤩 + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Can__39__t_export_synced__47___branches/comment_1_19ef18864eef6d08dae48df6ddb7152d._comment b/doc/bugs/Can__39__t_export_synced__47___branches/comment_1_19ef18864eef6d08dae48df6ddb7152d._comment new file mode 100644 index 0000000000..19f60bc6a4 --- /dev/null +++ b/doc/bugs/Can__39__t_export_synced__47___branches/comment_1_19ef18864eef6d08dae48df6ddb7152d._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-20T15:13:05Z" + content=""" +The difficulty here is that it needs to come up with the name of a +corresponding remote tracking branch to update. It could use +"myexport/synced/main" for that. But, setting `annex-tracking-branch` to +a remote ref like "origin/main" is not supported, and it would not be good +if it allowed it and used a tracking branch with a name like +"myexport/origin/main". + +And well, we know that "synced/main" is not a remote ref, but "$foo/main" +generally may or may not be one. + +So, I think to support this, it would have only allow using "$foo/main" +when the ref "refs/remotes/$foo/main" does not exist. I've implemented +that. +"""]]
Report bug that one can't export synced/ branches
diff --git a/doc/bugs/Can__39__t_export_synced__47___branches.mdwn b/doc/bugs/Can__39__t_export_synced__47___branches.mdwn new file mode 100644 index 0000000000..19085dbe88 --- /dev/null +++ b/doc/bugs/Can__39__t_export_synced__47___branches.mdwn @@ -0,0 +1,68 @@ +### Please describe the problem. + +It is not possible to export e.g. the `synced/main` branch to an exporttree=yes special remote. + +### What steps will reproduce the problem? + +see transcript below, but basically: + + +[[!format sh """ +> git annex initremote myexport type=directory exporttree=yes encryption=none directory=/path/to/dir +> git config remote.myexport.annex-tracking-branch synced/main +> git annex push myexport +# Not updating export to myexport because tracking branch name is not valid. (As configured by remote.myexport.annex-tracking-branch) +"""]] + + +### What version of git-annex are you using? On what operating system? + +10.20240927 on NixOS + +### Please provide any additional information below. + +[[!format sh """ +yann in 🌐 yann-desktop-nixos in …/code/git-annex-test/export-synced-branch +🐟 ❯ git init +Leeres Git-Repository in /home/yann/code/git-annex-test/export-synced-branch/.git/ initialisiert +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ git annex init +init ok +(recording state in git...) +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ touch bla +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main [?] +🐟 ❯ git annex assist -m "Add file" +add bla ok +(recording state in git...) +commit (recording state in git...) +ok +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main took 2s644ms +🐟 ❯ git annex initremote myexport type=directory exporttree=yes encryption=none directory=../exported-synced-branch +initremote myexport +git-annex: Directory does not exist: /home/yann/code/git-annex-test/exported-synced-branch +failed +initremote: 1 failed +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +❌1 🐟 ❯ mkdir ../exported-synced-branch/ +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ git annex initremote myexport type=directory exporttree=yes encryption=none directory=../exported-synced-branch +initremote myexport ok +(recording state in git...) +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ git config remote.myexport.annex-tracking-branch synced/main +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ git annex push myexport + + Not updating export to myexport because tracking branch name is not valid. (As configured by remote.myexport.annex-tracking-branch) +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❮ git config remote.myexport.annex-tracking-branch main +yann in 🌐 yann-desktop-nixos in …/export-synced-branch on main +🐟 ❯ git annex push myexport +export myexport bla ok +(recording state in git...) +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +git-annex is amazing, thank you! 🤩
Added a comment: Feature idea
diff --git a/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file/comment_1_ad2fc3f81a714dde9a39a27948ae0e5f._comment b/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file/comment_1_ad2fc3f81a714dde9a39a27948ae0e5f._comment new file mode 100644 index 0000000000..d51ad42610 --- /dev/null +++ b/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file/comment_1_ad2fc3f81a714dde9a39a27948ae0e5f._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="hello@da0030bba070302e85904b4d73db61fb4af7bced" + nickname="hello" + subject="Feature idea" + date="2025-01-17T11:14:55Z" + content=""" +This could be detected and handled directly by `git annex repair` or the `git annex assistant` daemon. +I unfortunately have no Haskell knowledge and the barrier to entry seems to big for me to contribute right now. +"""]]
diff --git a/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file.mdwn b/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file.mdwn new file mode 100644 index 0000000000..d3d175b0af --- /dev/null +++ b/doc/tips/Recovering_from_corrupted_encrypted_GITBUNDLE_file.mdwn @@ -0,0 +1,63 @@ +Yesterday upon setting up an `hybrid` encrypted rsync backend (with `git-remote-annex`) support, I stumbled around this bug. The annex remote would corrupt itself and no amount of `git annex repair` would fix it, unfortunately. + +``` +$ git annex push +push <remote> +Full remote url: annex::<remote-uuid>?encryption=hybrid&rsyncurl=<url>&type=rsync +gpg: zlib inflate problem: invalid block type + user error (gpg ["--quiet","--trust-model","always","--batch","--passphrase-fd","13","--decrypt"] exited 2) +git-annex: Failed to download GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49 +Full remote url: annex::<remote-uuid>?encryption=hybrid&rsyncurl=<url>&type=rsync +gpg: zlib inflate problem: invalid block type + user error (gpg ["--quiet","--trust-model","always","--batch","--passphrase-fd","13","--decrypt"] exited 2) +git-annex: Failed to download GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49 + + Pushing to <remote> failed. +failed +push: 1 failed +``` + +I then started to investigate the temporary files in `.git/annex/tmp` and found the aforementionned **GITBUNDLE**, as well as it's encrypted counterpart. + +``` +$ tree .git/annex/tmp/ +.git/annex/tmp/ +├── GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49 +└── GPGHMACSHA1--f2d78638494030b34841f7a30ffb2800816ef839 +``` + +I wanted to confirm that the file was corrupted for real, so obtained the cipher for my hyrid repository like so: + +``` +$ cipher=$(git show git-annex:remote.log | grep 'name=<remote>' | grep -oP 'cipher\=.*? ' | sed 's/cipher=//') +``` + +And confirmed the encrypted file was indeed corrupted by decrypting it manually: + +``` +$ echo $cipher | base64 -d | gpg -d | tail -c +257 | gpg --batch --passphrase-fd 0 --decrypt .git/annex/tmp/GPGHMACSHA1--f2d78638494030b34841f7a30ffb2800816ef839 > /dev/null +gpg: encrypted with rsa2048 key, ID <gpg-id>, created 1970-01-01 + "... <...>" +gpg: encrypted with cv25519 key, ID <gpg-id>, created 1970-01-01 + "... <...>" +gpg: AES256.CFB encrypted data +gpg: encrypted with 1 passphrase +gpg: zlib inflate problem: invalid block type +``` + +I then searched in the annex's directory for the uncorrupted **GITBUNDLE** to find out that it indeed existed: + +``` +$ find . -type f -name "GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49" +./.git/annex/objects/Vg/Mm/GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49/GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49 +``` + +Now that I have a way to get the symmetric cipher and the source file, as well as it's encrypted filename, I can re-encrypt it: + +``` +$ echo $cipher | base64 -d | gpg -d | tail -c +257 | gpg --batch --passphrase-fd 0 --output /tmp/GPGHMACSHA1--f2d78638494030b34841f7a30ffb2800816ef839 --symmetric ./.git/annex/objects/Vg/Mm/GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49/GITBUNDLE-s<id>--<remote-uuid>-77e0ff1580b971da4e39b15ba22439d66e3c5729adea2d7df8643438ef900c49 +``` + +And then upload it to replace the corruped one on the remote. + +I then confirmed it was fixed by issuing a `git annex sync --content`.
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn index e4d448f828..5530cd570f 100644 --- a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -10,7 +10,7 @@ The problem seems to be reproducible with any repository with a lot of files in The deadlock described makes higher-level commands like git annex sync also block indefinitely when checkout-ing the unlocked branch for any reason. Also, because the filtering is not completely applied, the index is pretty scrambled, its easier to clone the repo and move the annex than fix it, for me at least. -I call the behavior "deadlock" due to the absence of outpout and low cpu usage on the process when in that state. This seems to indicate some kind of multiprocessing deadlock to me. +I call the behavior "deadlock" due to the absence of debug log output and low cpu usage on the process when in that state. This seems to indicate some kind of multiprocessing deadlock to me. ### What steps will reproduce the problem?
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn index e6ba8e5bb1..e4d448f828 100644 --- a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -5,11 +5,13 @@ I wanted to unlock all those files from that branch on a machine, so I tried to Sadly, the command do not seems to finish, ever. Executing the command with debug from a clone(to avoid interacting with the broken index from the first), it seems to deadlock after executing between 10000 and 20000 "thawing" processes when executing the filter-process logic over the files in the worktree. -The problems seems to be reproducible with any repository with a lot of files in the worktree as far as I can tell, independant of file size. +The problem seems to be reproducible with any repository with a lot of files in the worktree as far as I can tell, independant of file size. -The infinite loop make higher-level commands like git annex sync also deadlock when checkout-ing the unlocked branch for any reason. +The deadlock described makes higher-level commands like git annex sync also block indefinitely when checkout-ing the unlocked branch for any reason. Also, because the filtering is not completely applied, the index is pretty scrambled, its easier to clone the repo and move the annex than fix it, for me at least. +I call the behavior "deadlock" due to the absence of outpout and low cpu usage on the process when in that state. This seems to indicate some kind of multiprocessing deadlock to me. + ### What steps will reproduce the problem? Here is a minimum set of bash commands that generate the deadlock on my end:
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn index 1516481f44..e6ba8e5bb1 100644 --- a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -70,6 +70,7 @@ Excerpt of the last lines from the huge debug log: Given the huge debug log produced, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generate one as required. Repeatedly calling this(and ctrl-c it when it inevitably get stuck) seems to eventually unlock the files, but its not really a valid solution in my case. + git annex smudge --update --debug ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn index 4a5022d67f..1516481f44 100644 --- a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -64,13 +64,13 @@ Debian Bookworm [Compiled via "building from source on Debian"] Excerpt of the last lines from the huge debug log: - a blocking[2025-01-16 23:30:27.913022014] (Utility.Process) process [493397] done ExitSuccess + [2025-01-16 23:30:27.913022014] (Utility.Process) process [493397] done ExitSuccess [2025-01-16 23:30:27.91309169] (Annex.Perms) thawing content .git/annex/othertmp/BKQKGR.0/BKQKGR -Given the huge debug log produced, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generated one as required. +Given the huge debug log produced, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generate one as required. Repeatedly calling this(and ctrl-c it when it inevitably get stuck) seems to eventually unlock the files, but its not really a valid solution in my case. -git annex smudge --update --debug + git annex smudge --update --debug ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) I really like git-annex, it allowed me to deduplicate the files in the big repository described above without much issues except for this bug.
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn index 40b1994e6a..4a5022d67f 100644 --- a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -27,7 +27,7 @@ Here is a minimum set of bash commands that generate the deadlock on my end: # This will get stuck after around ~10000-20000 processes from Utility.Process in the debug log while the git annex thaws files into unlocked files # The deadlock seems to happens after outputing the start of a new thawing, ctrl-c seems to be the only end state for this - git adjust --unlock --debug + git annex adjust --unlock --debug ### What version of git-annex are you using? On what operating system?
diff --git a/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn new file mode 100644 index 0000000000..40b1994e6a --- /dev/null +++ b/doc/bugs/Unlock_filter_seems_to_deadlock_for_huge_worktree.mdwn @@ -0,0 +1,77 @@ +### Please describe the problem. + +I have a pretty big repository with around 300 000 files in the workdir of a branch. +I wanted to unlock all those files from that branch on a machine, so I tried to use git-annex-adjust --unlock. +Sadly, the command do not seems to finish, ever. + +Executing the command with debug from a clone(to avoid interacting with the broken index from the first), it seems to deadlock after executing between 10000 and 20000 "thawing" processes when executing the filter-process logic over the files in the worktree. +The problems seems to be reproducible with any repository with a lot of files in the worktree as far as I can tell, independant of file size. + +The infinite loop make higher-level commands like git annex sync also deadlock when checkout-ing the unlocked branch for any reason. +Also, because the filtering is not completely applied, the index is pretty scrambled, its easier to clone the repo and move the annex than fix it, for me at least. + +### What steps will reproduce the problem? + +Here is a minimum set of bash commands that generate the deadlock on my end: +(https://github.com/klieret/RandomFileTree for the randomfiletree python command used) + + mkdir test_data + # Create about 280 000 empty and random files(Can still happens with non-empty files) + randomfiletree test_data -d 30 -f 250 -r 3 + cd test_data + git init + git annex init + git commit -m "empty commit" --allow-empty + git annex add + git commit -m "add all empty files" + + # This will get stuck after around ~10000-20000 processes from Utility.Process in the debug log while the git annex thaws files into unlocked files + # The deadlock seems to happens after outputing the start of a new thawing, ctrl-c seems to be the only end state for this + git adjust --unlock --debug + + +### What version of git-annex are you using? On what operating system? + +Happens on both: +Archlinux [normal package] + + git-annex version: 10.20240831-g9d29b99ac4074884d33fd25ef81baed5a11d0244 + build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Servant Benchmark Feeds Testsuite S3 WebDAV + dependency versions: aws-0.24.2 bloomfilter-2.0.1.2 crypton-1.0.0 DAV-1.3.4 feed-1.3.2.1 ghc-9.8.2 http-client-0.7.17 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external + operating system: linux x86_64 + supported repository versions: 8 9 10 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 + local repository version: 10 + +and + +Debian Bookworm [Compiled via "building from source on Debian"] + + git-annex version: 10.20250102-gaba8ee1ca1d571cada979ef47becb2a75379d63b + build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV + dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 + torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external + operating system: linux x86_64 + supported repository versions: 8 9 10 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 + local repository version: 10 + +### Please provide any additional information below. + +Excerpt of the last lines from the huge debug log: + + a blocking[2025-01-16 23:30:27.913022014] (Utility.Process) process [493397] done ExitSuccess + [2025-01-16 23:30:27.91309169] (Annex.Perms) thawing content .git/annex/othertmp/BKQKGR.0/BKQKGR + +Given the huge debug log produced, it may be easier to reproduce the bug to have it than copying it here. If wanted, I can generated one as required. + +Repeatedly calling this(and ctrl-c it when it inevitably get stuck) seems to eventually unlock the files, but its not really a valid solution in my case. +git annex smudge --update --debug + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +I really like git-annex, it allowed me to deduplicate the files in the big repository described above without much issues except for this bug. +
Added a comment: Still happening, managed to get a reproduction (maybe ?)
diff --git a/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_18_6032081d9f96b09e5eed32dc28ec7738._comment b/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_18_6032081d9f96b09e5eed32dc28ec7738._comment new file mode 100644 index 0000000000..d69c21e45e --- /dev/null +++ b/doc/bugs/Packfile_does_not_match_digest__58___gcrypt_with_assistant/comment_18_6032081d9f96b09e5eed32dc28ec7738._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="hello@da0030bba070302e85904b4d73db61fb4af7bced" + nickname="hello" + subject="Still happening, managed to get a reproduction (maybe ?)" + date="2025-01-16T18:17:51Z" + content=""" +Hi, I have stumbled upon this specific bug today too. + +I use `git-annex`: `10.20240927-3` (latest on Manjaro) with `git-remote-gcrypt`: `1.5-1` (latest at the time) and I now have the same issue. + + $ git push <remote> master + gcrypt: Decrypting manifest + gpg: Signature made jeu. 16 janv. 2025 18:56:55 CET + gpg: using EDDSA key *** + gpg: Good signature from \"*** <***>\" [ultimate] + gcrypt: Due to a longstanding bug, this push implicitly has --force. + gcrypt: Consider explicitly passing --force, and setting + gcrypt: gcrypt's require-explicit-force-push git config key. + gcrypt: Repacking remote <remote>, ... + gcrypt: Packfile *** does not match digest! + fatal: early EOF + error: failed to push some refs to 'gcrypt::ssh://librarian@<ip>:<port>/~/library.git' + +I tried to reproduce the issue and it seems that it is easy to force this to happen if +- you have a `git annex assistant` running in the annex +- copy a large directory (I used a `.flac` music album, 2.1Gi) to the annex so that it is uploaded to the `gcrypt` remote automatically +- issue a `git annex sync --content` while the assistant is trying to upload the content to the remote + + +"""]]
comment
diff --git a/doc/todo/Size_of_unused_files/comment_1_59d7d7c1d156d45aac6442cf9ee5f5c8._comment b/doc/todo/Size_of_unused_files/comment_1_59d7d7c1d156d45aac6442cf9ee5f5c8._comment new file mode 100644 index 0000000000..8c871e8b86 --- /dev/null +++ b/doc/todo/Size_of_unused_files/comment_1_59d7d7c1d156d45aac6442cf9ee5f5c8._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-15T23:11:33Z" + content=""" +I found myself wanting this today, wondering if a `git-annex move --unused` +would fit into the destination. + +I think it would make sense to add it to `git-annex unused`. Making +`git-annex info` output vary depending on when and how unused was run last +I don't like. +"""]]
add news item for git-annex 10.20250115
diff --git a/doc/news/version_10.20240831.mdwn b/doc/news/version_10.20240831.mdwn deleted file mode 100644 index 39b752e198..0000000000 --- a/doc/news/version_10.20240831.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -git-annex 10.20240831 released with [[!toggle text="these changes"]] -[[!toggleable text=""" * Special remotes configured with exporttree=yes annexobjects=yes - can store objects in .git/annex/objects, as well as an exported tree. - * Support proxying to special remotes configured with - exporttree=yes annexobjects=yes, and allow such remotes to be used as - cluster nodes. - * post-retrieve: When proxying is enabled for an exporttree=yes - special remote (or it is a cluster node) and the configured - remote.name.annex-tracking-branch is received, the tree is - exported to the special remote. - * Support "balanced=", "fullybalanced=", "sizebalanced=" and - "fullysizebalanced=" in preferred content expressions. - * Added --rebalance option. - * Added the annex.fullybalancedthreshhold git config. - * maxsize: New command to tell git-annex how large the expected maximum - size of a repository is, and to display repository sizes. - * vicfg: Include maxsize configuration. - * info: Improved speed by using new repository size tracking. - * lookupkey: Allow using --ref in a bare repository. - * export: Added --from option. - * git-remote-annex: Store objects in exportree=yes special remotes - in the same paths used by annexobjects=yes. This is a backwards - compatible change. - * updateproxy, updatecluster: Prevent using an exporttree=yes special - remote that does not have annexobjects=yes, since it will not work. - * The config versioning=true is now reserved for use by versioned special - remotes. External special remotes should not use that config for their - own purposes."""]] \ No newline at end of file diff --git a/doc/news/version_10.20250115.mdwn b/doc/news/version_10.20250115.mdwn new file mode 100644 index 0000000000..c6b56c47d6 --- /dev/null +++ b/doc/news/version_10.20250115.mdwn @@ -0,0 +1,26 @@ +git-annex 10.20250115 released with [[!toggle text="these changes"]] +[[!toggleable text=""" * Improve handing of ssh connection problems during + remote annex.uuid discovery. + * log: Support --key, as well as --branch and --unused. + * Avoid verification error when addurl --verifiable is used + with an url claimed by a special remote other than the web. + * Fix installation on Android. + * Allow enableremote of an existing webdav special remote that has + read-only access. + * git-remote-annex: Use enableremote rather than initremote. + * Windows: Fix permission denied error when dropping files that + have the readonly attribute set. + * Added freezecontent-annex and thawcontent-annex hooks that + correspond to the git configs annex.freezecontent and + annex.thawcontent. + * Added secure-erase-annex hook that corresponds to the git config + annex.secure-erase-command. + * Added commitmessage-annex hook that corresponds to the git config + annex.commitmessage-command. + * Added http-headers-annex hook that corresponds to the git config + annex.http-headers-command. + * Added git configs annex.post-update-command and annex.pre-commit-command + that correspond to the post-update-annex and pre-commit-annex hooks. + * Added annex.pre-init-command git config and pre-init-annex hook + that is run before git-annex repository initialization. + * Linux standalone builds' bundled rsync updated to fix security holes."""]] \ No newline at end of file
arm autobuild moved to sparrow
diff --git a/doc/install/Linux_standalone.mdwn b/doc/install/Linux_standalone.mdwn index 85ce6ed74b..57bcf5bfae 100644 --- a/doc/install/Linux_standalone.mdwn +++ b/doc/install/Linux_standalone.mdwn @@ -37,9 +37,6 @@ An hourly autobuild is also available, hosted by [[Joey]]: * x86-32: [download tarball](https://downloads.kitenet.net/git-annex/autobuild/i386/git-annex-standalone-i386.tar.gz) ([build logs](https://downloads.kitenet.net/git-annex/autobuild/i386/)) * arm64: [download tarball](https://downloads.kitenet.net/git-annex/autobuild/arm64/git-annex-standalone-arm64.tar.gz) ([build logs](https://downloads.kitenet.net/git-annex/autobuild/arm64/)) * arm64, for ancient kernels: [download tarball](https://downloads.kitenet.net/git-annex/autobuild/arm64-ancient/git-annex-standalone-arm64-ancient.tar.gz) ([build logs](https://downloads.kitenet.net/git-annex/autobuild/arm64-ancient/)) - -The arm autobuilder runs daily (sun permitting), and is hosted by [[Joey]]. - * arm: [download tarball](https://downloads.kitenet.net/git-annex/autobuild/armel/git-annex-standalone-armel.tar.gz) ([build logs](https://downloads.kitenet.net/git-annex/autobuild/armel/)) ## download security
pre-init config and hook
Added annex.pre-init-command git config and pre-init-annex hook that is run
before git-annex repository initialization.
This can block initialization. Or it can preform pre-initialization
configuration or tweaking.
I left stdio connected while it's running, so it could also be used for
interactive prompting conceivably, although that would want to use /dev/tty
anyway probably in order to not pollute the stdout of a command when
automatic initialization is done.
Sponsored-by: Dartmouth College's OpenNeuro project
Added annex.pre-init-command git config and pre-init-annex hook that is run
before git-annex repository initialization.
This can block initialization. Or it can preform pre-initialization
configuration or tweaking.
I left stdio connected while it's running, so it could also be used for
interactive prompting conceivably, although that would want to use /dev/tty
anyway probably in order to not pollute the stdout of a command when
automatic initialization is done.
Sponsored-by: Dartmouth College's OpenNeuro project
diff --git a/Annex/Hook.hs b/Annex/Hook.hs index e4264ce9d5..0496094be8 100644 --- a/Annex/Hook.hs +++ b/Annex/Hook.hs @@ -48,6 +48,9 @@ preCommitAnnexHook = Git.Hook "pre-commit-annex" "" [] postUpdateAnnexHook :: Git.Hook postUpdateAnnexHook = Git.Hook "post-update-annex" "" [] +preInitAnnexHook :: Git.Hook +preInitAnnexHook = Git.Hook "pre-init-annex" "" [] + freezeContentAnnexHook :: Git.Hook freezeContentAnnexHook = Git.Hook "freezecontent-annex" "" [] @@ -98,20 +101,33 @@ doesAnnexHookExist hook = do return exists runAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex () -runAnnexHook hook commandcfg = ifM (doesAnnexHookExist hook) +runAnnexHook hook commandcfg = runAnnexHook' hook commandcfg >>= \case + Nothing -> noop + Just failedcommanddesc -> + warning $ UnquotedString $ failedcommanddesc ++ " failed" + +-- Returns Nothing if the hook or GitConfig command succeeded, or a +-- description of what failed. +runAnnexHook' :: Git.Hook -> (GitConfig -> Maybe String) -> Annex (Maybe String) +runAnnexHook' hook commandcfg = ifM (doesAnnexHookExist hook) ( runhook , runcommandcfg ) where - runhook = unlessM (inRepo $ Git.runHook boolSystem hook []) $ do - h <- fromRepo $ Git.hookFile hook - commandfailed h + runhook = ifM (inRepo $ Git.runHook boolSystem hook []) + ( return Nothing + , do + h <- fromRepo (Git.hookFile hook) + commandfailed h + ) runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Nothing -> return Nothing Just command -> - unlessM (liftIO $ boolSystem "sh" [Param "-c", Param command]) $ - commandfailed command - Nothing -> noop - commandfailed c = warning $ UnquotedString $ c ++ " failed" + ifM (liftIO $ boolSystem "sh" [Param "-c", Param command]) + ( return Nothing + , commandfailed $ "git configured command '" ++ command ++ "'" + ) + commandfailed c = return $ Just c runAnnexPathHook :: String -> Git.Hook -> (GitConfig -> Maybe String) -> RawFilePath -> Annex Bool runAnnexPathHook pathtoken hook commandcfg p = ifM (doesAnnexHookExist hook) @@ -121,9 +137,9 @@ runAnnexPathHook pathtoken hook commandcfg p = ifM (doesAnnexHookExist hook) where runhook = inRepo $ Git.runHook boolSystem hook [ File (fromRawFilePath p) ] runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Nothing -> return True Just basecmd -> liftIO $ boolSystem "sh" [Param "-c", Param $ gencmd basecmd] - Nothing -> return True gencmd = massReplace [ (pathtoken, shellEscape (fromRawFilePath p)) ] outputOfAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex (Maybe String) @@ -135,6 +151,6 @@ outputOfAnnexHook hook commandcfg = ifM (doesAnnexHookExist hook) runhook = inRepo (Git.runHook runhook' hook []) runhook' c ps = Just <$> readProcess c (toCommand ps) runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Nothing -> return Nothing Just command -> liftIO $ Just <$> readProcess "sh" ["-c", command] - Nothing -> return Nothing diff --git a/Annex/Init.hs b/Annex/Init.hs index 0cb2e09019..ea7cd09765 100644 --- a/Annex/Init.hs +++ b/Annex/Init.hs @@ -1,6 +1,6 @@ {- git-annex repository initialization - - - Copyright 2011-2024 Joey Hess <id@joeyh.name> + - Copyright 2011-2025 Joey Hess <id@joeyh.name> - - Licensed under the GNU AGPL version 3 or higher. -} @@ -74,17 +74,29 @@ data InitializeAllowed = InitializeAllowed checkInitializeAllowed :: (InitializeAllowed -> Annex a) -> Annex a checkInitializeAllowed a = guardSafeToUseRepo $ noAnnexFileContent' >>= \case - Nothing -> do - checkSqliteWorks - a InitializeAllowed + Nothing -> runAnnexHook' preInitAnnexHook annexPreInitCommand >>= \case + Nothing -> do + checkSqliteWorks + a InitializeAllowed + Just failedcommanddesc -> do + initpreventedby failedcommanddesc + notinitialized Just noannexmsg -> do - warning "Initialization prevented by .noannex file (remove the file to override)" + initpreventedby ".noannex file (remove the file to override)" unless (null noannexmsg) $ warning (UnquotedString noannexmsg) - giveup "Not initialized." + notinitialized + where + initpreventedby r = warning $ UnquotedString $ + "Initialization prevented by " ++ r + notinitialized = giveup "Not initialized." initializeAllowed :: Annex Bool -initializeAllowed = isNothing <$> noAnnexFileContent' +initializeAllowed = noAnnexFileContent' >>= \case + Nothing -> runAnnexHook' preInitAnnexHook annexPreInitCommand >>= \case + Nothing -> return True + Just _ -> return False + Just _ -> return False noAnnexFileContent' :: Annex (Maybe String) noAnnexFileContent' = inRepo $ @@ -268,7 +280,7 @@ autoInitialize' check startupannex remotelist = getInitializedVersion >>= maybe needsinit checkUpgrade where needsinit = - whenM (initializeAllowed <&&> check) $ do + whenM (check <&&> initializeAllowed) $ do initialize startupannex Nothing Nothing autoEnableSpecialRemotes remotelist diff --git a/CHANGELOG b/CHANGELOG index 101fb7d0e2..d7ca6e3d15 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -22,6 +22,8 @@ git-annex (10.20250103) UNRELEASED; urgency=medium annex.http-headers-command. * Added git configs annex.post-update-command and annex.pre-commit-command that correspond to the post-update-annex and pre-commit-annex hooks. + * Added annex.pre-init-command git config and pre-init-annex hook + that is run before git-annex repository initialization. -- Joey Hess <id@joeyh.name> Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs index 81904653f8..053a9c8c66 100644 --- a/Types/GitConfig.hs +++ b/Types/GitConfig.hs @@ -97,6 +97,7 @@ data GitConfig = GitConfig , annexAlwaysCompact :: Bool , annexCommitMessage :: Maybe String , annexCommitMessageCommand :: Maybe String + , annexPreInitCommand :: Maybe String , annexPreCommitCommand :: Maybe String , annexPostUpdateCommand :: Maybe String , annexMergeAnnexBranches :: Bool @@ -192,6 +193,7 @@ extractGitConfig configsource r = GitConfig , annexAlwaysCompact = getbool (annexConfig "alwayscompact") True , annexCommitMessage = getmaybe (annexConfig "commitmessage") , annexCommitMessageCommand = getmaybe (annexConfig "commitmessage-command") + , annexPreInitCommand = getmaybe (annexConfig "pre-init-command") , annexPreCommitCommand = getmaybe (annexConfig "pre-commit-command") , annexPostUpdateCommand = getmaybe (annexConfig "post-update-command") , annexMergeAnnexBranches = getbool (annexConfig "merge-annex-branches") True diff --git a/doc/git-annex-init.mdwn b/doc/git-annex-init.mdwn index aa0e418639..dcbfd64102 100644 --- a/doc/git-annex-init.mdwn +++ b/doc/git-annex-init.mdwn @@ -27,7 +27,11 @@ already initialized git-annex repository. A top-level `.noannex` file will prevent git-annex init from being used in a repository. This is useful for repositories that have a policy reason not to use git-annex. The content of the file will be displayed -to the user who tries to run git-annex init. +to the user who tries to run git-annex init. + +The annex.pre-init-command git configuration or pre-init-annex hook +is run before initialization and can exit nonzero as well to prevent +initialization, as well as doing other setup. # EXAMPLES diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn index db305956f3..869686883a 100644 --- a/doc/git-annex.mdwn +++ b/doc/git-annex.mdwn @@ -1176,6 +1176,16 @@ repository, using [[git-annex-config]]. See its man page for a list.) Alternatively, a hook script can be installed in `.git/hooks/pre-commit-annex` +* `annex.pre-init-command` + + This command is run before the repository is initialized, either by + `git-annex init`, or automatic initialization. It can configure + the repository in any way needed. If it exits nonzero, the repository + initialization will fail. + + Alternatively, a hook script can be installed in + `.git/hooks/pre-init-annex` + (Diff truncated)
Added a comment
diff --git a/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_5_66a581d0032a73bdba72c502e0ac5117._comment b/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_5_66a581d0032a73bdba72c502e0ac5117._comment new file mode 100644 index 0000000000..eb7279fd45 --- /dev/null +++ b/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_5_66a581d0032a73bdba72c502e0ac5117._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="lemondata" + avatar="http://cdn.libravatar.org/avatar/481bf8163b175e5d0bdbef7d8c7fb3f4" + subject="comment 5" + date="2025-01-13T17:43:25Z" + content=""" +I appreciate you following up. The improvements you made do seem to allow the import to run for longer before resulting in the same error. Unfortunately, the source location has only gained a lot more files since the original issue. + +Given that is only going to continue, I am thinking I will not be able to import in this way on a location like this without being able to filter what contents to include prior to, or on the fly with the import. + +I don't know how feasible that actually would be to implement from your work on `importtreefilter`, so in the meantime I am going to try and think of some alternatives to my workflow to make this work. +"""]]
comment
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment index b04bfbab97..f57ea2c56c 100644 --- a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment @@ -22,6 +22,4 @@ Worth bearing in mind that if such a hook output to stdout, or read from stdin, or for that matter was interactive on /dev/tty, it could badly mess up things that use git-annex as plumbing. Including git, in the case of the smudge filter. - -(Should this todo be tagged repronim btw?) """]] diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_4_c62cc81e02c0d1906a747426e73102af._comment b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_4_c62cc81e02c0d1906a747426e73102af._comment new file mode 100644 index 0000000000..4a7d8456e5 --- /dev/null +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_4_c62cc81e02c0d1906a747426e73102af._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2025-01-13T17:20:28Z" + content=""" +Yes, a single pre-hook that gets run by every command would not be great +for performance. Especially when git is eg sometimes running `git-annex +smudge` repeatedly. + +You earlier spoke of "overall hook for git-annex invocation within a +repo". Did you just mean a hook that runs before initialization, and not +a hook that runs before every git-annex command? + +I feel more comfortable with adding a pre-init hook than pre-everything +hooks. For one thing, it makes sense for that hook to prevent +initialization if there is a reason to want to avoid that. It makes a lot +less sense for a `pre-smudge` hook to prevent that bit of git-annex +plumbing from working, which can have very different effects depending on +what git is doing, or for a `pre-find` hook to output something to +stdout. +"""]]
update
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_7_8fd25d5d2b802c6ff6ee0a3802d6f1cb._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_7_8fd25d5d2b802c6ff6ee0a3802d6f1cb._comment index a40e6ee0f7..2bd8eb1d4d 100644 --- a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_7_8fd25d5d2b802c6ff6ee0a3802d6f1cb._comment +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_7_8fd25d5d2b802c6ff6ee0a3802d6f1cb._comment @@ -9,4 +9,10 @@ secure-erase-annex, commitmessage-annex, http-headers-annex That leaves only `remote.name.annex-cost-command` and similar git configs that don't have hooks. And a few like annex.youtube-dl-command that are not really equivilant to hooks. + +I think I will wait on adding hooks for remote git configs, I'd rather talk +with someone who has a use case for that than make up something for +completeness. Am not currently liking the idea of including a remote name +in the hook for those, but perhaps someone would have a use case that +argues otherwise. """]] diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment index 4005c2228f..064594db49 100644 --- a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment @@ -3,5 +3,8 @@ subject="""Re: comment 8""" date="2025-01-13T17:12:40Z" content=""" -Seems right. +That seems like it would work. I suppose you could instead always install +the freeze/thaw hooks, and just make them do nothing when ACL is not used. +If probing for that is expensive or better to only do once for some other +reason, having the pre-init set up the hooks would make sense. """]]
comment
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment new file mode 100644 index 0000000000..4005c2228f --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_9_c61485293631d8d82b8573f7a938dc90._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""Re: comment 8""" + date="2025-01-13T17:12:40Z" + content=""" +Seems right. +"""]]
document files
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn index 81b6dc1727..db305956f3 100644 --- a/doc/git-annex.mdwn +++ b/doc/git-annex.mdwn @@ -1427,7 +1427,8 @@ repository, using [[git-annex-config]]. See its man page for a list.) For example, to use the wipe command, set it to `wipe -f %file`. Alternatively to setting the git config, a hook script can be installed - in `.git/hooks/secure-erase-annex` + in `.git/hooks/secure-erase-annex`. The hook is passed the file to be + erased as its only parameter. * `annex.freezecontent-command`, `annex.thawcontent-command` @@ -1447,6 +1448,8 @@ repository, using [[git-annex-config]]. See its man page for a list.) Alternatively to setting the git config, hook scripts can be installed in `.git/hooks/freezecontent-annex` and `.git/hooks/thawcontent-annex`. + The hook is passed the file or directory to operate on as its only + parameter. * `annex.tune.objecthash1`, `annex.tune.objecthashlower`, `annex.tune.branchhash1`
todo
kind of a bug, but I'm not sure if it can be fixed
kind of a bug, but I'm not sure if it can be fixed
diff --git a/doc/todo/avoid_sync_unncessary_transfers.mdwn b/doc/todo/avoid_sync_unncessary_transfers.mdwn new file mode 100644 index 0000000000..52ad595b74 --- /dev/null +++ b/doc/todo/avoid_sync_unncessary_transfers.mdwn @@ -0,0 +1,57 @@ +`git-annex sync --content` can sometimes do unncessary transfers +when syncing with multiple remotes at the same time. Here is an example: + + git-annex sync --content r2 r3 + ... + copy foo (to r2...) + ok + copy foo (to r3...) + ok + drop foo (from r2...) ok + + git-annex wanted r3 + anything + git-annex wanted r2 + not copies=foo:1 + git-annex group r3 + foo + +At the time sync decides to copy to the first repository, it legitimately +wants the content. But once the content is also copied to the second +repository, the first no longer wants it. + +Perhaps this could be improved by sync simulating that the second copy +has already happened, and seeing if the first repository still wants the +content then. + +But, bear in mind that the second repository may not be accessible +currently. And indeed it may currently be located somewhere such that content +can only reach it via the first repository. So if it decides to send to the +second repository but not then to the first, it should still send to the +first repository if it is unable to send to the second. + +So essentially, this would reorder the list of repositories to work on from +"r2 r3" to "r3 r2". + +But, it's also possible to have this situation: + + git-annex wanted r3 + not copies=foo:1 + git-annex wanted r2 + not copies=foo:1 + git-annex group r3 + foo + git-annex group r2 + foo + +In that situation, `git-annex sync --content r2 r3` will currently send +content to r2 when it's accessible. Then r3 doesn't get a copy, and the +copy remains on r2. If the list of repositories were reordered, that would +change which repository ends up with the file. Which would be surprising +since that configuration and order of repositories passed to sync +is clear in its desired effect. + +With the config above, there would be no need to reorder the list of +repositories since it does not avoid excess work. Are there +configs where reordering would avoid excess work, but would also change +desired behavior (for the same file)? --[[Joey]]
update
diff --git a/doc/thanks/list b/doc/thanks/list index e493dc436b..1f590c5572 100644 --- a/doc/thanks/list +++ b/doc/thanks/list @@ -124,3 +124,4 @@ schodet, oz, Lilia.Nanne, Dusty Mabe, +mpol,
Added a comment
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_8_ce1f98226ab84acc847d4b370278e081._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_8_ce1f98226ab84acc847d4b370278e081._comment new file mode 100644 index 0000000000..8d27f42235 --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_8_ce1f98226ab84acc847d4b370278e081._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 8" + date="2025-01-10T20:27:48Z" + content=""" +Coolio! So my ACLs use-case should need/entail + +- Setup [`annex.pre-init-hook`](https://git-annex.branchable.com/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/#comment-17018d2e1d9f26df13aa54c6f3969406) globally to point to the script which would check if ACL is used on a specific folder where `git annex init` is ran. If it is ran, it would then either + - Under `.git/hooks` either place or remove (if not required) `freezecontent-annex`, `thawcontent-annex` which would take a single path (file or a directory) which would need to be frozen or thawed. + +Is that correct/complete picture or am I missing anything I would need to do for my use case? +"""]]
added hooks corresponding to annex.*-command
* Added freezecontent-annex and thawcontent-annex hooks that
correspond to the git configs annex.freezecontent and
annex.thawcontent.
* Added secure-erase-annex hook that corresponds to the git config
annex.secure-erase-command.
* Added commitmessage-annex hook that corresponds to the git config
annex.commitmessage-command.
* Added http-headers-annex hook that corresponds to the git config
annex.http-headers-command.
that correspond to the post-update-annex and pre-commit-annex hooks.
The use case for these is eg, setting up a git repository that is run in a
container, where the easiest way to provide a script is by putting it in
.git/hooks/, rather than copying it into the container in a way that puts
it in PATH.
This is all the ones that make sense to add for annex.*-config git configs.
annex.youtube-dl-command is not a hook, it's telling git-annex what command
to run. So is annex.shared-sop-command. So omitted those.
May later also want to add hooks corresponding to
`remote.<name>.annex-cost-command` etc.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
* Added freezecontent-annex and thawcontent-annex hooks that
correspond to the git configs annex.freezecontent and
annex.thawcontent.
* Added secure-erase-annex hook that corresponds to the git config
annex.secure-erase-command.
* Added commitmessage-annex hook that corresponds to the git config
annex.commitmessage-command.
* Added http-headers-annex hook that corresponds to the git config
annex.http-headers-command.
that correspond to the post-update-annex and pre-commit-annex hooks.
The use case for these is eg, setting up a git repository that is run in a
container, where the easiest way to provide a script is by putting it in
.git/hooks/, rather than copying it into the container in a way that puts
it in PATH.
This is all the ones that make sense to add for annex.*-config git configs.
annex.youtube-dl-command is not a hook, it's telling git-annex what command
to run. So is annex.shared-sop-command. So omitted those.
May later also want to add hooks corresponding to
`remote.<name>.annex-cost-command` etc.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
diff --git a/Annex/Branch.hs b/Annex/Branch.hs index 03adce0886..ce4c3ad85e 100644 --- a/Annex/Branch.hs +++ b/Annex/Branch.hs @@ -521,12 +521,10 @@ createMessage :: Annex String createMessage = fromMaybe "branch created" <$> getCommitMessage getCommitMessage :: Annex (Maybe String) -getCommitMessage = do - config <- Annex.getGitConfig - case annexCommitMessageCommand config of - Nothing -> return (annexCommitMessage config) - Just cmd -> catchDefaultIO (annexCommitMessage config) $ - Just <$> liftIO (readProcess "sh" ["-c", cmd]) +getCommitMessage = + outputOfAnnexHook commitMessageAnnexHook annexCommitMessageCommand + <|> + (annexCommitMessage <$> Annex.getGitConfig) {- Stages the journal, and commits staged changes to the branch. -} commit :: String -> Annex () diff --git a/Annex/Content/LowLevel.hs b/Annex/Content/LowLevel.hs index 9d732f6a6e..69baf19957 100644 --- a/Annex/Content/LowLevel.hs +++ b/Annex/Content/LowLevel.hs @@ -10,6 +10,7 @@ module Annex.Content.LowLevel where import Annex.Common +import Annex.Hook import Logs.Transfer import qualified Annex import Utility.DiskFree @@ -25,11 +26,8 @@ import System.PosixCompat.Files (linkCount) - File may or may not be deleted at the end; caller is responsible for - making sure it's deleted. -} secureErase :: RawFilePath -> Annex () -secureErase file = maybe noop go =<< annexSecureEraseCommand <$> Annex.getGitConfig - where - go basecmd = void $ liftIO $ - boolSystem "sh" [Param "-c", Param $ gencmd basecmd] - gencmd = massReplace [ ("%file", shellEscape (fromRawFilePath file)) ] +secureErase = void . runAnnexPathHook "%file" + secureEraseAnnexHook annexSecureEraseCommand data LinkedOrCopied = Linked | Copied diff --git a/Annex/Hook.hs b/Annex/Hook.hs index deadf871ed..e4264ce9d5 100644 --- a/Annex/Hook.hs +++ b/Annex/Hook.hs @@ -4,7 +4,7 @@ - git-annex not change, otherwise removing old hooks using an old - version of the script would fail. - - - Copyright 2013-2019 Joey Hess <id@joeyh.name> + - Copyright 2013-2025 Joey Hess <id@joeyh.name> - - Licensed under the GNU AGPL version 3 or higher. -} @@ -48,6 +48,21 @@ preCommitAnnexHook = Git.Hook "pre-commit-annex" "" [] postUpdateAnnexHook :: Git.Hook postUpdateAnnexHook = Git.Hook "post-update-annex" "" [] +freezeContentAnnexHook :: Git.Hook +freezeContentAnnexHook = Git.Hook "freezecontent-annex" "" [] + +thawContentAnnexHook :: Git.Hook +thawContentAnnexHook = Git.Hook "thawcontent-annex" "" [] + +secureEraseAnnexHook :: Git.Hook +secureEraseAnnexHook = Git.Hook "secure-erase-annex" "" [] + +commitMessageAnnexHook :: Git.Hook +commitMessageAnnexHook = Git.Hook "commitmessage-annex" "" [] + +httpHeadersAnnexHook :: Git.Hook +httpHeadersAnnexHook = Git.Hook "http-headers-annex" "" [] + mkHookScript :: String -> String mkHookScript s = unlines [ shebang @@ -69,23 +84,26 @@ hookWarning h msg = do warning $ UnquotedString $ Git.hookName h ++ " hook (" ++ Git.hookFile h r ++ ") " ++ msg -{- Runs a hook. To avoid checking if the hook exists every time, - - the existing hooks are cached. -} -runAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex () -runAnnexHook hook commandcfg = do +{- To avoid checking if the hook exists every time, the existing hooks + - are cached. -} +doesAnnexHookExist :: Git.Hook -> Annex Bool +doesAnnexHookExist hook = do m <- Annex.getState Annex.existinghooks case M.lookup hook m of - Just True -> runhook - Just False -> runcommandcfg + Just exists -> return exists Nothing -> do exists <- inRepo $ Git.hookExists hook Annex.changeState $ \s -> s { Annex.existinghooks = M.insert hook exists m } - if exists - then runhook - else runcommandcfg + return exists + +runAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex () +runAnnexHook hook commandcfg = ifM (doesAnnexHookExist hook) + ( runhook + , runcommandcfg + ) where - runhook = unlessM (inRepo $ Git.runHook hook) $ do + runhook = unlessM (inRepo $ Git.runHook boolSystem hook []) $ do h <- fromRepo $ Git.hookFile hook commandfailed h runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case @@ -94,3 +112,29 @@ runAnnexHook hook commandcfg = do commandfailed command Nothing -> noop commandfailed c = warning $ UnquotedString $ c ++ " failed" + +runAnnexPathHook :: String -> Git.Hook -> (GitConfig -> Maybe String) -> RawFilePath -> Annex Bool +runAnnexPathHook pathtoken hook commandcfg p = ifM (doesAnnexHookExist hook) + ( runhook + , runcommandcfg + ) + where + runhook = inRepo $ Git.runHook boolSystem hook [ File (fromRawFilePath p) ] + runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Just basecmd -> liftIO $ + boolSystem "sh" [Param "-c", Param $ gencmd basecmd] + Nothing -> return True + gencmd = massReplace [ (pathtoken, shellEscape (fromRawFilePath p)) ] + +outputOfAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex (Maybe String) +outputOfAnnexHook hook commandcfg = ifM (doesAnnexHookExist hook) + ( runhook + , runcommandcfg + ) + where + runhook = inRepo (Git.runHook runhook' hook []) + runhook' c ps = Just <$> readProcess c (toCommand ps) + runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Just command -> liftIO $ + Just <$> readProcess "sh" ["-c", command] + Nothing -> return Nothing diff --git a/Annex/Perms.hs b/Annex/Perms.hs index 56bd35190e..03bce4fe83 100644 --- a/Annex/Perms.hs +++ b/Annex/Perms.hs @@ -33,6 +33,7 @@ module Annex.Perms ( ) where import Annex.Common +import Annex.Hook import Utility.FileMode import Git import Git.ConfigTypes @@ -340,24 +341,24 @@ modifyContentDirWhenExists f a = do either throwM return v hasFreezeHook :: Annex Bool -hasFreezeHook = isJust . annexFreezeContentCommand <$> Annex.getGitConfig +hasFreezeHook = + (isJust . annexFreezeContentCommand <$> Annex.getGitConfig) + <||> + (doesAnnexHookExist freezeContentAnnexHook) hasThawHook :: Annex Bool -hasThawHook = isJust . annexThawContentCommand <$> Annex.getGitConfig +hasThawHook = + (isJust . annexThawContentCommand <$> Annex.getGitConfig) + <||> + (doesAnnexHookExist thawContentAnnexHook) freezeHook :: RawFilePath -> Annex () -freezeHook p = maybe noop go =<< annexFreezeContentCommand <$> Annex.getGitConfig - where - go basecmd = void $ liftIO $ - boolSystem "sh" [Param "-c", Param $ gencmd basecmd] - gencmd = massReplace [ ("%path", shellEscape (fromRawFilePath p)) ] +freezeHook = void . runAnnexPathHook "%path" + freezeContentAnnexHook annexFreezeContentCommand thawHook :: RawFilePath -> Annex () -thawHook p = maybe noop go =<< annexThawContentCommand <$> Annex.getGitConfig - where - go basecmd = void $ liftIO $ - boolSystem "sh" [Param "-c", Param $ gencmd basecmd] - gencmd = massReplace [ ("%path", shellEscape (fromRawFilePath p)) ] +thawHook = void . runAnnexPathHook "%path" + thawContentAnnexHook annexThawContentCommand {- Calculate mode to use for a directory from the mode to use for a file. - diff --git a/Annex/Url.hs b/Annex/Url.hs index 2f12a10768..e796b314b9 100644 (Diff truncated)
Added a comment: on the init hook
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_3_1dcbe1b1287f7bad262995b72ab0660b._comment b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_3_1dcbe1b1287f7bad262995b72ab0660b._comment new file mode 100644 index 0000000000..f5d4fe42ee --- /dev/null +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_3_1dcbe1b1287f7bad262995b72ab0660b._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="on the init hook" + date="2025-01-10T17:33:14Z" + content=""" +> There could be a generalized hook config, eg `git config annex.pre-hook my-hook` could make `git-annex init run my-hook git-annex init` and same for other git-annex commands. + +I feel that having such a generic gateway hook would be + +- not aligned to how `.git/hooks` operate (separate scripts for different operations) +- make it harder to provide CLI options handling given that different command hooks might have different options (via env var or not) +- add increasing performance hit with increasing number of commands potentially calling such a generic hook + +If someone wanted such one, they could create a \"symlink\" driven generic hook which would have resolved its action upon `$0` name as some tools etc do. + +I really feel that having `annex.pre-{command}-hook` set of configurations would be cleaner. + +> It sounds like you would want a failure of such a hook to prevent the git-annex command from running. Is that right? + +I think so... and I think that's how git hooks perform? (less git-annex is special the better) +"""]]
configs annex.post-update-command and annex.pre-commit-command
Added git configs annex.post-update-command and annex.pre-commit-command
that correspond to the git-annex hook scripts post-update-annex and
pre-commit-annex.
Note that the hook files take precience over the git config, since the git
config can includ global config which should be overridden by local config.
These new git configs are probably not super useful. Especially the
pre-commit-annex hook is there to install scripts to instead of the
pre-commit hook, since git-annex installs that hook itself. So why would
someone want to use a git config for that? Only reason I can think of would
be in a global git config. Or possibly because it's easier to set a git
config than write a hook script, on an OS like Windows.
The real reason I'm adding these is as groundwork for making other
annex.*-command git configs also be available as hook scripts. I want
to avoid having some things available as only git hooks and others as
both gitconfigs and git hooks. (It seems that some annex.*-command configs
don't translate to git hooks though.)
In the man page, moved documentation of the hooks to be next to the
documentation of the git configs. This is to avoid repitition.
Added git configs annex.post-update-command and annex.pre-commit-command
that correspond to the git-annex hook scripts post-update-annex and
pre-commit-annex.
Note that the hook files take precience over the git config, since the git
config can includ global config which should be overridden by local config.
These new git configs are probably not super useful. Especially the
pre-commit-annex hook is there to install scripts to instead of the
pre-commit hook, since git-annex installs that hook itself. So why would
someone want to use a git config for that? Only reason I can think of would
be in a global git config. Or possibly because it's easier to set a git
config than write a hook script, on an OS like Windows.
The real reason I'm adding these is as groundwork for making other
annex.*-command git configs also be available as hook scripts. I want
to avoid having some things available as only git hooks and others as
both gitconfigs and git hooks. (It seems that some annex.*-command configs
don't translate to git hooks though.)
In the man page, moved documentation of the hooks to be next to the
documentation of the git configs. This is to avoid repitition.
diff --git a/Annex/Branch.hs b/Annex/Branch.hs index 945acf724b..03adce0886 100644 --- a/Annex/Branch.hs +++ b/Annex/Branch.hs @@ -257,7 +257,7 @@ updateTo' pairs = do mergeIndex jl refs let commitrefs = nub $ fullname:refs ifM (handleTransitions jl localtransitions commitrefs) - ( runAnnexHook postUpdateAnnexHook + ( runAnnexHook postUpdateAnnexHook annexPostUpdateCommand , do ff <- if dirty then return False @@ -724,7 +724,7 @@ setIndexSha :: Git.Sha -> Annex () setIndexSha ref = do f <- fromRepo gitAnnexIndexStatus writeLogFile f $ fromRef ref ++ "\n" - runAnnexHook postUpdateAnnexHook + runAnnexHook postUpdateAnnexHook annexPostUpdateCommand {- Stages the journal into the index, and runs an action that - commits the index to the branch. Note that the action is run diff --git a/Annex/Hook.hs b/Annex/Hook.hs index 8c6d648fb0..deadf871ed 100644 --- a/Annex/Hook.hs +++ b/Annex/Hook.hs @@ -71,18 +71,26 @@ hookWarning h msg = do {- Runs a hook. To avoid checking if the hook exists every time, - the existing hooks are cached. -} -runAnnexHook :: Git.Hook -> Annex () -runAnnexHook hook = do +runAnnexHook :: Git.Hook -> (GitConfig -> Maybe String) -> Annex () +runAnnexHook hook commandcfg = do m <- Annex.getState Annex.existinghooks case M.lookup hook m of - Just True -> run - Just False -> noop + Just True -> runhook + Just False -> runcommandcfg Nothing -> do exists <- inRepo $ Git.hookExists hook Annex.changeState $ \s -> s { Annex.existinghooks = M.insert hook exists m } - when exists run + if exists + then runhook + else runcommandcfg where - run = unlessM (inRepo $ Git.runHook hook) $ do + runhook = unlessM (inRepo $ Git.runHook hook) $ do h <- fromRepo $ Git.hookFile hook - warning $ UnquotedString $ h ++ " failed" + commandfailed h + runcommandcfg = commandcfg <$> Annex.getGitConfig >>= \case + Just command -> + unlessM (liftIO $ boolSystem "sh" [Param "-c", Param command]) $ + commandfailed command + Nothing -> noop + commandfailed c = warning $ UnquotedString $ c ++ " failed" diff --git a/CHANGELOG b/CHANGELOG index b412a20602..42bcfa6fb1 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -11,6 +11,9 @@ git-annex (10.20250103) UNRELEASED; urgency=medium * git-remote-annex: Use enableremote rather than initremote. * Windows: Fix permission denied error when dropping files that have the readonly attribute set. + * Added git configs annex.post-update-command and annex.pre-commit-command + that correspond to the git-annex hook scripts post-update-annex and + pre-commit-annex. -- Joey Hess <id@joeyh.name> Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/Command/PreCommit.hs b/Command/PreCommit.hs index bc69e4a210..204a5fa8e2 100644 --- a/Command/PreCommit.hs +++ b/Command/PreCommit.hs @@ -44,7 +44,7 @@ seek ps = do -- files in the worktree won't be populated, so populate them here Command.Smudge.updateSmudged (Restage False) - runAnnexHook preCommitAnnexHook + runAnnexHook preCommitAnnexHook annexPreCommitCommand -- committing changes to a view updates metadata currentView >>= \case diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs index 2ab5de3ea6..81904653f8 100644 --- a/Types/GitConfig.hs +++ b/Types/GitConfig.hs @@ -97,6 +97,8 @@ data GitConfig = GitConfig , annexAlwaysCompact :: Bool , annexCommitMessage :: Maybe String , annexCommitMessageCommand :: Maybe String + , annexPreCommitCommand :: Maybe String + , annexPostUpdateCommand :: Maybe String , annexMergeAnnexBranches :: Bool , annexDelayAdd :: Maybe Int , annexHttpHeaders :: [String] @@ -190,6 +192,8 @@ extractGitConfig configsource r = GitConfig , annexAlwaysCompact = getbool (annexConfig "alwayscompact") True , annexCommitMessage = getmaybe (annexConfig "commitmessage") , annexCommitMessageCommand = getmaybe (annexConfig "commitmessage-command") + , annexPreCommitCommand = getmaybe (annexConfig "pre-commit-command") + , annexPostUpdateCommand = getmaybe (annexConfig "post-update-command") , annexMergeAnnexBranches = getbool (annexConfig "merge-annex-branches") True , annexDelayAdd = getmayberead (annexConfig "delayadd") , annexHttpHeaders = getlist (annexConfig "http-headers") diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn index 5e4c9f4777..1f499cb315 100644 --- a/doc/git-annex.mdwn +++ b/doc/git-annex.mdwn @@ -1155,6 +1155,24 @@ repository, using [[git-annex-config]]. See its man page for a list.) This command is run and its output is used as the commit message to the git-annex branch. +* `annex.post-update-command` + + This command is run after git-annex updates the git-annex branch. + + Alternatively, a hook script can be installed in + `.git/hooks/post-update-annex` + + When publishing a git-annex repository by http, this can be used to run + `git update-server-info` + +* `annex.pre-commit-command` + + This command is run whenever a commit is made to the HEAD branch of + the git repository, either by git commit, or by git-annex. + + Alternatively, a hook script can be installed in + `.git/hooks/pre-commit-annex` + * `annex.alwayscompact` By default, git-annex compacts data it records in the git-annex branch. @@ -2322,14 +2340,6 @@ used by git-annex. `~/.config/git-annex/autostart` is a list of git repositories to start the git-annex assistant in. -`.git/hooks/pre-commit-annex` in your git repository will be run whenever -a commit is made to the HEAD branch, either by git commit, git-annex -sync, or the git-annex assistant. - -`.git/hooks/post-update-annex` in your git repository will be run -whenever the git-annex branch is updated. You can make this hook run -`git update-server-info` when publishing a git-annex repository by http. - # SEE ALSO More git-annex documentation is available on its web site,
allocate this development to openneuro - we need to curate/process datasets from/for openneuro on discovery
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn index b5ba7190af..95cd81a9c1 100644 --- a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn @@ -3,3 +3,6 @@ The idea is stemmed from discussions/problems with using freeze/thaw hooks, and - test e.g. if custom freeze/thaw are needed (before even git annex decides to switch to adjusted branches mode) and setup that repo accordingly for git-annex to proceed without flipping out into adjusted branch mode - may be improved/custom pidlock detection (on one of my servers I remember needing to just hardcode in ~/.gitconfig to use pidlock although that was relevant only for some paths). - similarly do some other testing which could allow or disallow some git-annex decision such as e.g. use of adjusted unlocked branch + +[[!meta author=yoh]] +[[!tag projects/openneuro]]
comment
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_6_6bda77480362b38d6984d38430a1dc17._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_6_6bda77480362b38d6984d38430a1dc17._comment new file mode 100644 index 0000000000..5b5c5d12fe --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_6_6bda77480362b38d6984d38430a1dc17._comment @@ -0,0 +1,34 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2025-01-10T16:13:44Z" + content=""" +After sleeping on it, I concur that PATH changes feel unwise. + +Also, it turns out that git-annex already actually caches existing hooks, +so adding new hook scripts to .git/hooks/ even for things that run +frequently is not a performance problem. + +So, my plan is to add .git/hooks that are run in preference to some +`annex.*-command` git configs. Eg, .git/hooks/freezecontent-annex +corresponds to annex.freezecontent-command. + +The "-annex" prefix matches the current pre-commit-annex and post-update-annex +hooks. Also considering adding git configs corresponding to the existing +hooks. I doubt that there would be much use case for configuring +annex.pre-commit-command rather than the pre-commit-annex hook, since the +hook is there only to let users who would usually install a pre-commit hook +to install their hook script without getting in the way of the pre-commit +hook that git-annex writes. But, it seems worth having the git config just +for consistency. + +There are some things like annex.youtube-dl-command and +annex.http-headers-command that are configuring commands for git-annex to +run, and are not really hooks per se. + +And it does not make sense to have hook scripts that a specific to a given +remote corresponding to configs like `remote.name.annex-cost-command`. +Instead it might make sense to have a `.git/hooks/remote-cost-annex` that +is passed the name of the remote, but that bridge can be crossed if we +come to it. +"""]]
comment
diff --git a/doc/bugs/Trust_but_Verify__58___RClone/comment_6_8e3533a9f784eb9001d01b85227aa8d4._comment b/doc/bugs/Trust_but_Verify__58___RClone/comment_6_8e3533a9f784eb9001d01b85227aa8d4._comment new file mode 100644 index 0000000000..a0602b6005 --- /dev/null +++ b/doc/bugs/Trust_but_Verify__58___RClone/comment_6_8e3533a9f784eb9001d01b85227aa8d4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2025-01-10T16:07:55Z" + content=""" +I see Spencer closed this bug recently. Spencer, did you determine this is +not actually a problem? +"""]]
fixed
diff --git a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn index f0b5b049bf..f96f1e54fe 100644 --- a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn +++ b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn @@ -43,3 +43,8 @@ Approximate round trip times in milli-seconds: Sure. It's always rockin'! [[!meta author=jkniiv]] + +> This was git's safe.directory changes which uncessarily broke git-daemon +> for no added security (something I've told the git devs about but they +> have not fixed), +> now worked around in the server config. [[fixed|done]]
Added a comment
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_5_3383606878295faefd913f8a5173c99e._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_5_3383606878295faefd913f8a5173c99e._comment new file mode 100644 index 0000000000..cf4f5e55bb --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_5_3383606878295faefd913f8a5173c99e._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2025-01-10T03:20:31Z" + content=""" +It could be `%gitdir` or any other you like. It could be that any relative path - relative to that folder since it makes no sense relative to current working dir etc + +I really feel odd about changing PATH for this purpose, especially to point to .git/hooks which want intended to be added to PATH. I think it if it was `.git/annex/hooks` , with documentation about such act happening, it would be a but more kosher, but I still feel better about some more explicit path specification +"""]]
repo is probably not technically offline, just out of reach
diff --git a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn index 010d826aad..f0b5b049bf 100644 --- a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn +++ b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn @@ -1,6 +1,6 @@ ### Please describe the problem. -The main repository `git://git-annex.branchable.com/` seems to be offline at the moment. +The main repository `git://git-annex.branchable.com/` seems to be out of reach at the moment. The mirror `git://git.joeyh.name/git-annex` seems to work. ### What steps will reproduce the problem?
Added a comment: same problem on a VPS
diff --git a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07/comment_1_766e975f08dacca4e27eb91b3f8ec238._comment b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07/comment_1_766e975f08dacca4e27eb91b3f8ec238._comment new file mode 100644 index 0000000000..46952cadf6 --- /dev/null +++ b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07/comment_1_766e975f08dacca4e27eb91b3f8ec238._comment @@ -0,0 +1,34 @@ +[[!comment format=mdwn + username="jkniiv" + avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" + subject="same problem on a VPS" + date="2025-01-09T21:14:06Z" + content=""" +This doesn't seem to be merely a problem with my home internet but I also get the same error +on a VPS hosted by Hetzner (Helsinki data center): + +``` +jkniiv@jibun-ubuntu-2gb-hel1-1:~$ git clone git://git-annex.branchable.com/ git-annex-clone-TEST +Cloning into 'git-annex-clone-TEST'... +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +``` + +Yet: + +``` +jkniiv@jibun-ubuntu-2gb-hel1-1:~$ git clone git://git.joeyh.name/git-annex git-annex-clone-TEST +Cloning into 'git-annex-clone-TEST'... +remote: Enumerating objects: 281839, done. +remote: Counting objects: 100% (18490/18490), done. +remote: Compressing objects: 100% (6909/6909), done. +remote: Total 281839 (delta 12887), reused 16462 (delta 11469), pack-reused 263349 (from 1) +Receiving objects: 100% (281839/281839), 59.49 MiB | 20.45 MiB/s, done. +Resolving deltas: 100% (201606/201606), done. +``` + +So what is going on? + +"""]]
comment
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment new file mode 100644 index 0000000000..b04bfbab97 --- /dev/null +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_2_0015660a3ede2c3181f960b9c2bac81c._comment @@ -0,0 +1,27 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2025-01-09T17:38:25Z" + content=""" +Seems like a reasonable idea to have a hook that runs before `git-annex +init` does any of its probing or configuration. + +As for generalizing to a hook that is run before each git-annex command, +putting it in .git/hooks seems like it could cause a fair bit of extra work +probing if the hook(s) exist, especially in cases where git-annex runs git +which might run git-annex again repeatedly. + +There could be a generalized hook config, eg `git config annex.pre-hook +my-hook` could make `git-annex init` run `my-hook git-annex init` +and same for other git-annex commands. + +It sounds like you would want a failure of such a hook to prevent the +git-annex command from running. Is that right? + +Worth bearing in mind that if such a hook output to stdout, or read from +stdin, or for that matter was interactive on /dev/tty, it could badly +mess up things that use git-annex as plumbing. Including git, in the case +of the smudge filter. + +(Should this todo be tagged repronim btw?) +"""]] diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_5a510abfe992a1aa4d7e2edcd9f6844b._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_5a510abfe992a1aa4d7e2edcd9f6844b._comment new file mode 100644 index 0000000000..a97654da01 --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_5a510abfe992a1aa4d7e2edcd9f6844b._comment @@ -0,0 +1,35 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2025-01-09T17:57:05Z" + content=""" +I can see how it would be simpler for you to just be able to have those +hooks in .git/hooks/ along with the rest of the repository. + +I don't think that a special casing of "./" is a good idea, that would be +pretty confusing and for all I know some user might really want git-annex +to run a hook in the current directory of their git repository. + +I am meh on "%dotgit", for one thing in a bare repository it's not `.git`. + +What if git-annex just added the git hooks directory to the end of PATH +when running configured `annex.*-command`s? Then you could: `git config +annex.thawcontent-command annex-thawcontent` and put your script in +.git/hooks/annex-thawcontent + +This nicely avoids git-annex doing any extra work in general to check if +.git/hooks/ exist. + +The reason I think it would need to be at the end of PATH rather than the +front is that there are some git hooks with names like "update" and "applypatch" +that I can imagine might have the same names as a user's own programs +in their PATH. For example, if `annex.commitmessage-command=foo` and the +script foo runs "update", the user would be surprised if that ran the git +hook rather than their `~/bin/update`. + +On the other hand, when configuring a `annex.*-command`, it does not seem +likely that the user would set it to "update" or any of the other names +of git hooks. Especially if they didn't have an "update" in PATH. +So using the git hook directory for this, rather than some +other special directory under .git seems ok. +"""]]
comment
diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_4_926f3d02aa5044497af4fc88a8ecefa6._comment b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_4_926f3d02aa5044497af4fc88a8ecefa6._comment new file mode 100644 index 0000000000..c54196e4d7 --- /dev/null +++ b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_4_926f3d02aa5044497af4fc88a8ecefa6._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2025-01-09T15:58:54Z" + content=""" +Getting a file into this state can be done like this: + + echo bar > foo + git-annex add + echo '.git/annex/objects/g7/9v/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730' > foo + git add foo + git commit -m foo + git reset --hard + git status + git diff + +How the symlink bit might have gotten lost in real life, I don't know +though. +"""]]
retitle
diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn b/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn index d584c9d11c..9dfb941483 100644 --- a/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn +++ b/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn @@ -90,3 +90,5 @@ Not sure if relates to the following issue under current investigation (but not [[!meta author=yoh]] [[!tag projects/repronim]] + +[[!meta title="pointer file in the form of annex symlink target gets restaged modified"]]
comment
diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_3_1a6120c92f6104623432a60886cc628e._comment b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_3_1a6120c92f6104623432a60886cc628e._comment new file mode 100644 index 0000000000..7419dc18b2 --- /dev/null +++ b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_3_1a6120c92f6104623432a60886cc628e._comment @@ -0,0 +1,41 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-09T14:46:37Z" + content=""" +Ok, thanks for the test case. I can reproduce it with that. + +The file is actually unlocked already in the first commit of it to git, +as can be seen in its file permissions: + + joey@darkstar:~/tmp/open-brain-consent>git ls-tree -r f50cc382241662613b86073a2c001398e30c120f|grep samples/UK_gla_3T_fMRI_consent_form_v3.0.docx + 100644 blob 3215574fde295d53dc0032ec65e826236a2f2758 samples/UK_gla_3T_fMRI_consent_form_v3.0.docx + +Somewhat weirdly, while that file is not a symlink, it has the *form* of a +git-annex symlink target rather than an unlocked pointer file: + + joey@darkstar:~/tmp/open-brain-consent>cat samples/UK_gla_3T_fMRI_consent_form_v3.0.docx; echo + ../.git/annex/objects/5M/wv/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx + +That unusual content of a pointer file is still something git-annex +can handle, because it happens to use the same code path to parse a link +target and a pointer file. + +This explains why the file gets a modification staged: +The file's annexed content is present, so the smudge clean filter and +restage happens. In this case, the smudge clean filter emits +a usual annex link, which is different than the file's previous content. +Which is why it ends up with a modification staged. + +What you should probably do is just commit the change that git-annex staged, +converting it to a usual unlocked file. Or lock the file is you want it locked. +Once it's in a usual state, this will stop being a problem. + +Perhaps git-annex could probably avoid staging a modification in this case. +Although there are several code paths where a pointer file can be written +and/or staged, and all of them would have to take care to handle this case, +which I think would mean extra work and slow it down for everyone. + +And, I don't know how the file got into this state in the first place, +perhaps some other bug or weird behavior? +"""]]
diff --git a/doc/forum/b2_rclone_initremote_failed__58___empty_folders.mdwn b/doc/forum/b2_rclone_initremote_failed__58___empty_folders.mdwn new file mode 100644 index 0000000000..d1beddcf62 --- /dev/null +++ b/doc/forum/b2_rclone_initremote_failed__58___empty_folders.mdwn @@ -0,0 +1,19 @@ +Trying to add a Backblaze b2 special remote using rclone. + +Have tested the remote using rclone so I know it is set up properly and connecting. + +Running + + git annex initremote remotename type=rclone encryption=none rcloneremotename=remotename + +to initiate the remote in git-annex. + + +It fails with: + + initremote remotename + git-annex: this rclone remote does not support empty directories + failed + initremote: 1 failed + +Can not figure out what that means, searching does not help. What have I missed?
layout
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index a597d92ff5..d99a2b43be 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -34,7 +34,7 @@ contributed good bug reports and great ideas. ## financial support, 2024-2025 <img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> -<img alt="Standford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> &bbsp; +<img alt="Standford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> <img alt="Jülich Forschungszentrum logo" src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width=200> git-annex development is supported in large part by:
layout
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index b42a9cce26..a597d92ff5 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -33,9 +33,9 @@ contributed good bug reports and great ideas. ## financial support, 2024-2025 -<img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> -<img alt="Standford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> -<img alt="Jülich Forschungszentrum logo" src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width=200> +<img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> +<img alt="Standford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> &bbsp; +<img alt="Jülich Forschungszentrum logo" src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width=200> git-annex development is supported in large part by:
update
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index 9b06435e90..b42a9cce26 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -31,17 +31,19 @@ tips, user support, etc. Two names that come to mind are Anarcat and CandyAngel. John Lawrence made the logo. And many others have contributed good bug reports and great ideas. -## financial support, 2024 +## financial support, 2024-2025 <img alt="OpenNeuro logo" src="https://raw.githubusercontent.com/OpenNeuroOrg/openneuro/1c1e0d3b2a2032729727702eb65b1b563eadce1d/packages/openneuro-components/src/assets/on-dark.svg" width=100> <img alt="Standford wordmark" src="https://poldracklab.org/images/stanford-line2-8.png" width=200> - +<img alt="Jülich Forschungszentrum logo" src="https://www.fz-juelich.de/static/media/Logo.2ceb35fc.svg" width=200> git-annex development is supported in large part by: * [OpenNeuro](https://openneuro.org/), funded by a [BRAIN Initiative NIH grant](//reporter.nih.gov/project-details/10881961) awarded to The Poldrack Lab at Stanford University. +* Institute of Neuroscience and Medicine (Brain and Behavior; INM-7) + at [Forschungszentrum Jülich](https://www.fz-juelich.de/en/inm/inm-7). * [DANDI](https://www.dandiarchive.org/), funded by [a NIH grant](https://projectreporter.nih.gov/project_info_description.cfm?aid=9981835) awarded to MIT, Dartmouth College, and Kitware.
Added a comment: failing with `Operation not petmitted`
diff --git a/doc/special_remotes/adb/comment_7_347f59e0d8b982171528b267decc7ecf._comment b/doc/special_remotes/adb/comment_7_347f59e0d8b982171528b267decc7ecf._comment new file mode 100644 index 0000000000..d469a6b06a --- /dev/null +++ b/doc/special_remotes/adb/comment_7_347f59e0d8b982171528b267decc7ecf._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="octvs@17a99a7aaeb0c0e0a2375e14807b138740ba34e9" + nickname="octvs" + avatar="http://cdn.libravatar.org/avatar/e31212348b6392eb67daddf78fadfb1b" + subject="failing with `Operation not petmitted`" + date="2025-01-08T08:25:04Z" + content=""" +I recently faced with a fail + +``` +... +export pixel podcasts/<redacted>.mp3 +adb: error: failed to copy '../../.git/annex/objects/17/vK/SHA256E-s57638506--06c92a06db07fa511271e5426c9447cb0fab551d7ba2b9c9d26d3c7e3f975b5c.mp3/SHA256E-s57638506--06c92a06db07fa511271e5426c9447cb0fab551d7ba2b9c9d26d3c7e3f975b5c.mp3' to '/storage/emulated/0/home/media/podcasts/not-related/<redacted>.mp3': remote couldn't create file: Operation not permitted +../../.git/annex/objects/17/vK/SHA256E-s57638506--06c92a06db07...1 file pushed, 0 skipped. 37.0 MB/s (57638506 bytes in 1.487s + + adb failed to store file +failed +(recording state in git...) +export: 2 failed +``` + +Everything is done as usual on the phone, barring two all other files are +succesfully sent. Any clues what might be the cause? + +"""]]
Added a comment
diff --git a/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__/comment_1_a7f24e5001606edcedaedc2a9f3f654c._comment b/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__/comment_1_a7f24e5001606edcedaedc2a9f3f654c._comment new file mode 100644 index 0000000000..2286b2f59e --- /dev/null +++ b/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__/comment_1_a7f24e5001606edcedaedc2a9f3f654c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Atemu" + avatar="http://cdn.libravatar.org/avatar/6ac9c136a74bb8760c66f422d3d6dc32" + subject="comment 1" + date="2025-01-07T22:28:16Z" + content=""" +It appears git-annex find has commands that look like they can help debug wanted expressions (`--explain`, `--want-get`, `--want-drop`) but I couldn't figure out how to hold these flags correctly. +"""]]
Added a comment
diff --git a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_4_145622307d038925777800645e0db7aa._comment b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_4_145622307d038925777800645e0db7aa._comment new file mode 100644 index 0000000000..4521744d4d --- /dev/null +++ b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_4_145622307d038925777800645e0db7aa._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 4" + date="2025-01-07T22:01:20Z" + content=""" +**yes**: remote would need to *indicate to git-annex* that \"feature\". ATM git-annex already does that via `EXTENSIONS` to announce what it can do, and it seems that + +> The special remote can respond to that with its own EXTENSIONS message + +so it could have announced + + +``` +EXTENSIONS ABSENTIFNOURL +``` + +or alike. + +**yes** -- it is \"unusual\" as not every special remote would be \"URL-only\" remote. But there is AFAIK a growing number of custom remotes which are like that at least in `datalad` land: `datalad`, `datalad-archives`, [`datalad-uncurl`](https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.annexremotes.uncurl.html) and likely others. Typically they are \"read-only\" remotes, and URL is used as the identifier for custom \"downloader\" support. + +It is interesting that bittorrent remote doesn't make them visible... never used, but wondered how my life would have been if I wanted to manage collection of torrent urls per each key... +"""]]
Added a comment
diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_2_1f45d12d1dcf3e42148d0400165d3c3d._comment b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_2_1f45d12d1dcf3e42148d0400165d3c3d._comment new file mode 100644 index 0000000000..ef408f971c --- /dev/null +++ b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_2_1f45d12d1dcf3e42148d0400165d3c3d._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2025-01-07T21:23:26Z" + content=""" +FWIW +- no assistant running +- reproduces still with 10.20250102+git5-ge8e36a90c1-1~ndall+1 +- reproduces on a full copy. I will send email with a URL (not sure if there are any secrets there, so do not want to send publicly) +"""]]
diff --git a/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__.mdwn b/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__.mdwn new file mode 100644 index 0000000000..8d418c9cf7 --- /dev/null +++ b/doc/forum/How_to_figure_out_why_files_aren__39__t_being_dropped__63__.mdwn @@ -0,0 +1,31 @@ +I've got a repo on my laptop to which I copied some files in order to store it in the cold storage. All repos of the cold storage are in the same group and wanted/required expressions are set up such that the group wants exactly one copy. + +This has been working fine but now after I copied the keys, I'm in the situation where running `git annex sync --content --all` in the repo on the laptop doesn't drop the keys that now have a copy in the cold storage group. + +numcopies=3 and there are 4 trusted copies (including the laptop), so that should work. + +The laptop repo wants `groupwanted or not copies=cold:1`. + +groupwanted expressions: + +``` +groupwanted hot = ((not copies=hot:1) or (not copies=cold:1) or (not copies=cold:1 and unused)) or ((not copies=lukewarm:1) or (approxlackingcopies=1)) +groupwanted client = (approxlackingcopies=1) +groupwanted cold = (not copies=cold:1) or (not copies=cold:1 and unused) +groupwanted lukewarm = (not copies=cold:1) or (not copies=lukewarm:1) or (approxlackingcopies=1) +``` + +(I need to clean these up, I know.) + +There's one copy in client (laptop), hot, cold, and lukewarm and all of them are either reachable or trusted. + +I could drop stuff manually of course but the next sync would try to get them again, so that's no good. + +How can I figure out *why* a certain repo wants a certain key? I currently don't even know whether it's numcopies somehow or the wanted expression and then also which expression causes it. + +One way I just thought of for the latter part would be to evaluate the individual expressions for some key to see whether they match. I've done that in my head of course and came to the conclusion that it shouldn't be wanted but I'd like the logic inside git-annex to tell me what actually happens. I effectively need a way to debug wanted expressions. +Is there a way to evaluate an arbitrary wanted expression to a boolean given a key? + +``` +git-annex version: 10.20240927 +```
Added a comment: odd system
diff --git a/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_3_764dae20267ed67c188fe414821b296b._comment b/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_3_764dae20267ed67c188fe414821b296b._comment new file mode 100644 index 0000000000..2477910be1 --- /dev/null +++ b/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_3_764dae20267ed67c188fe414821b296b._comment @@ -0,0 +1,65 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="odd system" + date="2025-01-07T21:06:35Z" + content=""" +``` +~ $ find git-annex.linux/ -iname xargs -ls + 13206 4 drwx------ 2 u0_a102 u0_a102 3488 Jul 2 2024 git-annex.linux/shimmed/xargs + 13207 68 -rwx------ 1 u0_a102 u0_a102 67856 Jul 2 2024 git-annex.linux/shimmed/xargs/xargs + 14834 0 lrwxrwxrwx 1 u0_a102 u0_a102 34 Jul 2 2024 git-annex.linux/exe/xargs -> ../lib/ld-linux-aarch64 +.so.1 + 14339 4 -rwx------ 1 u0_a102 u0_a102 128 Jul 2 2024 git-annex.linux/bin/xargs +~ $ echo 123 | git-annex.linux/bin/xargs echo +git-annex.linux/bin/xargs: 2: exec: /exe/xargs: Permission denied +~ $ file git-annex.linux/lib/ld-linux-aarch64.so.1 +git-annex.linux/lib/ld-linux-aarch64.so.1: ELF shared object, 64-bit LSB arm64, BuildID=c955aebf7720fc28ce7443a0599cde62b194 +e7b7, stripped +~ $ ls -ld git-annex.linux/lib/ld-linux-aarch64.so.1 +-rwx------ 1 u0_a102 u0_a102 145328 Apr 30 2024 git-annex.linux/lib/ld-linux-aarch64.so.1 +~ $ git-annex.linux/lib/ld-linux-aarch64.so.1 +Could not find a PHDR: broken executable? +Aborted + +~ $ ldd git-annex.linux/lib/ld-linux-aarch64.so.1 +git-annex.linux/lib/ld-linux-aarch64.so.1[1]: syntax error: unexpected '(' +~ $ file git-annex.linux/lib/ld-linux-aarch64.so.1 +git-annex.linux/lib/ld-linux-aarch64.so.1: ELF shared object, 64-bit LSB arm64, BuildID=c955aebf7720fc28ce7443a0599cde62b194 +e7b7, stripped +``` + +so architecture is right but something \"not-good-enough\" for Android in it? + +interestingly that `ldd` fiasco since seems to be that the + +``` +/bin $ nl -ba `which ldd` + 1 #!/system/bin/sh + 2 + 3 # Rather than have ldd and ldd64, this script does the right thing depending + 4 # on the argument. + 5 + 6 function error() { + 7 echo \"$1\" + 8 exit 1 + 9 } +``` + +and that `system/sh` does not support this syntax! (removal of `function ` seems to not help ..) but that seems to be how many copies of such file distributed [around](https://github.com/search?q=%22rather+than+have+ldd+and+ldd64%22&type=code) . `ldd` starts to work if I remove com.termux's `/bin/` from PATH and start system's `sh`... + +But overall that system feels really odd: + +``` +~ $ ls -ld / +drwxr-xr-x 29 root root 4096 Dec 31 2008 / +~ $ ls -l / +ls: cannot open directory '/': Permission denied +~ $ /wrongpath +bash: /wrongpath: Permission denied +``` + +so - even through permissions there, `ls` cannot list it with Permissions denied. The same error if just using non-existing path. But it is more appropriate if I am in the system shell, not termux's bash. + +full transcript of my meanderings around the boox system is [here](http://www.oneukrainian.com/tmp/boox-transcript-20250107-1.txt). +"""]]
comment
diff --git a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_3_9d724cfc7248319dcad3b57db24118fd._comment b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_3_9d724cfc7248319dcad3b57db24118fd._comment new file mode 100644 index 0000000000..1423c8377f --- /dev/null +++ b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_3_9d724cfc7248319dcad3b57db24118fd._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T20:40:02Z" + content=""" +It would need some way for a special remote to indicate to git-annex that +it is in this unusual class of remotes where not having an url is the same as +content no longer being present in it. + +Implementing that would just make some more remotes have a special case, +which seems even harder to remember. I'd rather remove the special case, +but of course that will break existing workflows. + +And it is unusual. Consider a S3 remote. It can have an url recorded for a +object stored in it, but forgetting the url doesn't mean that the S3 bucket +no longer contains the file. If git-annex behaved this way for S3, it would +be broken in a way that could be expensive to the user. + +(The special case is not currently implemented for bittorrent special +remote. But it also doesn't record urls in a user-visible way actually.) +"""]]
windows permissions fix
Windows: Fix permission denied error when dropping files that have the
readonly attribute set.
Files coming from a special remote may have had write permission removed
from them. The directory special remote does that. And there are
probably others. So rather than fixing it on the special remote side,
made moveAnnex, on Windows, add back the write bit. This apparently
removes the readonly attribute. See Remote.Directory.removeDirGeneric
which already did the same on windows to allow removing files from the
directory special remote.
The reason that cleanObjectLoc also calls allowWrite is to handle
situations where files have already gotten into git-annex repositories on
Windows with the write bit set. Eg, an older git-annex put them there.
Or perhaps the git-annex repository was populated on some other OS.
Windows: Fix permission denied error when dropping files that have the
readonly attribute set.
Files coming from a special remote may have had write permission removed
from them. The directory special remote does that. And there are
probably others. So rather than fixing it on the special remote side,
made moveAnnex, on Windows, add back the write bit. This apparently
removes the readonly attribute. See Remote.Directory.removeDirGeneric
which already did the same on windows to allow removing files from the
directory special remote.
The reason that cleanObjectLoc also calls allowWrite is to handle
situations where files have already gotten into git-annex repositories on
Windows with the write bit set. Eg, an older git-annex put them there.
Or perhaps the git-annex repository was populated on some other OS.
diff --git a/Annex/Content.hs b/Annex/Content.hs index 93d111140f..63746298d9 100644 --- a/Annex/Content.hs +++ b/Annex/Content.hs @@ -512,6 +512,12 @@ withTmp key action = do moveAnnex :: Key -> AssociatedFile -> RawFilePath -> Annex Bool moveAnnex key af src = ifM (checkSecureHashes' key) ( do +#ifdef mingw32_HOST_OS + {- Windows prevents deletion of files that are not + - writable, and the file could have such a mode. + - So avoid problems with deleting the file, now or later. -} + void $ liftIO $ tryIO $ allowWrite src +#endif withObjectLoc key storeobject return True , return False @@ -733,7 +739,11 @@ cleanObjectLoc key cleaner = do -} whenM hasThawHook $ void $ tryIO $ thawContent file - +#ifdef mingw32_HOST_OS + {- Windows prevents deletion of files that are not writable. -} + void $ liftIO $ tryIO $ allowWrite file +#endif + cleaner cleanObjectDirs file diff --git a/CHANGELOG b/CHANGELOG index 70c26b518b..b412a20602 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -9,6 +9,8 @@ git-annex (10.20250103) UNRELEASED; urgency=medium * Allow enableremote of an existing webdav special remote that has read-only access. * git-remote-annex: Use enableremote rather than initremote. + * Windows: Fix permission denied error when dropping files that + have the readonly attribute set. -- Joey Hess <id@joeyh.name> Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn index 87a0c37e61..939c979a33 100644 --- a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn +++ b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn @@ -131,3 +131,5 @@ This observation comes from an effort to implement a git-remote-helper that uses [[!tag projects/datalad]] [[!tag projects/INM7]] + +> I think this is [[fixed|done]] now.. --[[Joey]] diff --git a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_4_8a139e5a44ee9bae5195f46a7a67cfaf._comment b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_4_8a139e5a44ee9bae5195f46a7a67cfaf._comment new file mode 100644 index 0000000000..08aa9e4831 --- /dev/null +++ b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_4_8a139e5a44ee9bae5195f46a7a67cfaf._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2025-01-07T20:33:22Z" + content=""" +I see that the directory special remote, on windows, adds back the write +permission on files before removing them. That was done way back in +[[!commit 38022f4f493588505896635703c34afa6c2e2406]]. + +I've applied the same fix now to the annex objects side, so this should be +fixed. +"""]] diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn index 10a823cca6..8e38f1c17d 100644 --- a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn @@ -498,3 +498,5 @@ say I'm a believer. :) [[!meta author=jkniiv]] [[!tag projects/INM7]] + +> I think this is [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_5_9a591141f1105490f359b58a3b29cd91._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_5_9a591141f1105490f359b58a3b29cd91._comment new file mode 100644 index 0000000000..0b3b531fa7 --- /dev/null +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_5_9a591141f1105490f359b58a3b29cd91._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2025-01-07T20:31:07Z" + content=""" +I see that the directory special remote, on windows, adds back the write +permission on files before removing them. That was done way back in +[[!commit 38022f4f493588505896635703c34afa6c2e2406]]. + +I've applied the same fix now to the annex objects side, so this should be +fixed. +"""]]
Added a comment
diff --git a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_2_95085b4d43916cbc3c8ff437fc90256b._comment b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_2_95085b4d43916cbc3c8ff437fc90256b._comment new file mode 100644 index 0000000000..302af3fca0 --- /dev/null +++ b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_2_95085b4d43916cbc3c8ff437fc90256b._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2025-01-07T20:12:31Z" + content=""" +Well, it then seem to me that \"web\" special remote is not that special and it would be useful to extend its *special case* to be supported by other special remotes exhibiting similar behavior of being useless if no URLs are registered for a key. + +BTW, isn't it not just `web` but also `bittorrent` (do not have any URL handy to check ATM) since I expect it also to be a special remote needing to know a URL? + +Anyways, sorry for the noise, but as it happens with *special cases* and aging brains, I keep forgetting some of them. Hopefully I will make a better mental note about this one ;-) +"""]]
close
diff --git a/doc/todo/Read-only_support_for_webdav.mdwn b/doc/todo/Read-only_support_for_webdav.mdwn index cb7ca2e87c..4e11a8763e 100644 --- a/doc/todo/Read-only_support_for_webdav.mdwn +++ b/doc/todo/Read-only_support_for_webdav.mdwn @@ -7,3 +7,5 @@ Weighing the two approaches (read-only `webdav` vs authenticated `httpalso`), it It would be great to have authenticated, read-only access to webdav shares. Thanks in advance for considering! [[!tag projects/INM7]] + +> [[done]] --[[Joey]]
git-remote-annex enableremote to support readonly webdav
* Allow enableremote of an existing webdav special remote that has
read-only access.
* git-remote-annex: Use enableremote rather than initremote.
* Allow enableremote of an existing webdav special remote that has
read-only access.
* git-remote-annex: Use enableremote rather than initremote.
diff --git a/CHANGELOG b/CHANGELOG index b401848477..70c26b518b 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -6,6 +6,9 @@ git-annex (10.20250103) UNRELEASED; urgency=medium * Avoid verification error when addurl --verifiable is used with an url claimed by a special remote other than the web. * Fix installation on Android. + * Allow enableremote of an existing webdav special remote that has + read-only access. + * git-remote-annex: Use enableremote rather than initremote. -- Joey Hess <id@joeyh.name> Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/CmdLine/GitRemoteAnnex.hs b/CmdLine/GitRemoteAnnex.hs index ef1801dd94..da2a61b34b 100644 --- a/CmdLine/GitRemoteAnnex.hs +++ b/CmdLine/GitRemoteAnnex.hs @@ -586,14 +586,15 @@ withSpecialRemote cfg@(SpecialRemoteConfig {}) sab a = case specialRemoteName cf Nothing -> specialRemoteFromUrl sab inittempremote where -- Initialize a new special remote with the provided configuration - -- and name. + -- and name. This actually does a Remote.Enable, because the + -- special remote has already been initialized somewhere before. initremote remotename = do let c = M.insert SpecialRemote.nameField (Proposed remotename) $ M.delete (Accepted "config-uuid") $ specialRemoteConfig cfg t <- either giveup return (SpecialRemote.findType c) dummycfg <- liftIO dummyRemoteGitConfig - (c', u) <- Remote.setup t Remote.Init (Just (specialRemoteUUID cfg)) + (c', u) <- Remote.setup t (Remote.Enable c) (Just (specialRemoteUUID cfg)) Nothing c dummycfg `onException` cleanupremote remotename Logs.Remote.configSet u c' diff --git a/Remote/WebDAV.hs b/Remote/WebDAV.hs index d9984d3387..aaf8b8f059 100644 --- a/Remote/WebDAV.hs +++ b/Remote/WebDAV.hs @@ -137,7 +137,9 @@ webdavSetup ss mu mcreds c gc = do (c', encsetup) <- encryptionSetup c gc pc <- either giveup return . parseRemoteConfig c' =<< configParser remote c' creds <- maybe (getCreds pc gc u) (return . Just) mcreds - testDav url creds + case ss of + Init -> testDav url creds + _ -> noop gitConfigSpecialRemote u c' [("webdav", "true")] c'' <- setRemoteCredPair ss encsetup pc gc (davCreds u) creds return (c'', u) diff --git a/doc/special_remotes/webdav/comment_24_6c16a7c9abb257be4d046e27e11bacdf._comment b/doc/special_remotes/webdav/comment_24_6c16a7c9abb257be4d046e27e11bacdf._comment index 094399f650..0b305c38a0 100644 --- a/doc/special_remotes/webdav/comment_24_6c16a7c9abb257be4d046e27e11bacdf._comment +++ b/doc/special_remotes/webdav/comment_24_6c16a7c9abb257be4d046e27e11bacdf._comment @@ -1,10 +1,9 @@ [[!comment format=mdwn username="joey" - subject="""comment 24""" - date="2023-06-20T17:25:27Z" + subject="""Re: read-only webdav access""" + date="2025-01-07T18:51:26Z" content=""" -httpalso now supports chunking. So I think there's no need to add readonly -support to webdav, probably. But if you disagree, I do think it would be -possible to add. Just probably not useful.. after all webdav minus writing -is little different than http. If you disagree, open a [[todo]]. +Update: Since I posted this comment, enableremote of an existing webdav +special remote has been made to work even if you cannot write to it. So +there is no need to use httpalso and no need for a readonly=true. """]] diff --git a/doc/todo/Read-only_support_for_webdav/comment_1_1fd87a5ab19a81a1b094efea7e70e581._comment b/doc/todo/Read-only_support_for_webdav/comment_1_1fd87a5ab19a81a1b094efea7e70e581._comment new file mode 100644 index 0000000000..063f08805b --- /dev/null +++ b/doc/todo/Read-only_support_for_webdav/comment_1_1fd87a5ab19a81a1b094efea7e70e581._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-07T18:51:26Z" + content=""" +I'm not sure I'd trust an old comment of mine to reflect whether httpalso +supports http authentication. From what I can see, without trying it, it +will use git-credential to prompt for password when receiving a 401 +response, the same as all other parts of git-annex that download urls +do nowadays. + +Re git-remote-annex, I see your point though. Since it does set up the +remote when cloning, and webdav remote setup calls involves creating and +deleting a test file, the user needs write access currently. (Whether that +test file is worth it doing at all, I'm on the fence about. It seemed like +a good idea way back when.) + +I think that, rather than needing readonly=true, enableremote of a webdav +special remote should just assume it exists and not test write access to +it. Then it will work with readonly access. I've implemented that. + +And I've made git-remote-annex on git clone do an enableremote, rather than +an initremote. That should make it work with a readonly webdav special +remote. +"""]]
same bug
diff --git a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_3_a13fc099ba358138dfc4f03656232bd1._comment b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_3_a13fc099ba358138dfc4f03656232bd1._comment new file mode 100644 index 0000000000..4c2ce22c3d --- /dev/null +++ b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/comment_3_a13fc099ba358138dfc4f03656232bd1._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T19:46:35Z" + content=""" +This seems to be the same bug as +<https://git-annex.branchable.com/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/#comment-129fb966abab410a08d919cf1dc09625> +and over there we have some progress happening. +"""]] diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_4_b950f9d58ac292a1e318aaf0bc0754c0._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_4_b950f9d58ac292a1e318aaf0bc0754c0._comment new file mode 100644 index 0000000000..326d72097e --- /dev/null +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_4_b950f9d58ac292a1e318aaf0bc0754c0._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2025-01-07T19:48:29Z" + content=""" +Seems to be the same problem as +<https://git-annex.branchable.com/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__/> +which also involves directory special remote, but does not involve +git-remote-annex. + +A comment there tried what I suggested here, and indeed removing the +ReadOnly attribute from the file in the directory special remote avoids the +problem. +"""]]
tag INM7
based on mih filing or commenting on things and/or on
git-remote-annex being used
based on mih filing or commenting on things and/or on
git-remote-annex being used
diff --git a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn index 664a808bfa..87a0c37e61 100644 --- a/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn +++ b/doc/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__.mdwn @@ -130,3 +130,4 @@ And here is the diff between the two sequences (failure(-) vs success(+) This observation comes from an effort to implement a git-remote-helper that uses git-annex to enable fetch/push to and from any location that can be reached by any special remote. The fact that git-annex can be used to build things like this is just crazy cool! Thx much! [[!tag projects/datalad]] +[[!tag projects/INM7]] diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn index df2d612a32..10a823cca6 100644 --- a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__.mdwn @@ -496,3 +496,5 @@ backups, where it gives structure to my image-based backup routines, so you coul say I'm a believer. :) [[!meta author=jkniiv]] + +[[!tag projects/INM7]] diff --git a/doc/todo/Read-only_support_for_webdav.mdwn b/doc/todo/Read-only_support_for_webdav.mdwn index c09271775f..cb7ca2e87c 100644 --- a/doc/todo/Read-only_support_for_webdav.mdwn +++ b/doc/todo/Read-only_support_for_webdav.mdwn @@ -5,3 +5,5 @@ A use case not possible with that approach is *authenticated* read-only access. Weighing the two approaches (read-only `webdav` vs authenticated `httpalso`), it appears that only the read-only `webdav` is compatible with [git-remote-annex](https://git-annex.branchable.com/git-remote-annex/), because a user would need to declare *one* special remote (configuration) to use for cloning. It would be great to have authenticated, read-only access to webdav shares. Thanks in advance for considering! + +[[!tag projects/INM7]] diff --git a/doc/todo/compute_special_remote.mdwn b/doc/todo/compute_special_remote.mdwn index a4c57c0b27..5b9fa5bca3 100644 --- a/doc/todo/compute_special_remote.mdwn +++ b/doc/todo/compute_special_remote.mdwn @@ -60,3 +60,5 @@ I believe that no particular handling of annex key that are declared inputs to c ### Trust We would need a way for users to indicate that they trust a particular compute introduction or the entity that provided it. Even if git-annex does not implement tooling for that, it would be good to settle on a concept that can be interpreted/implemented by such special remotes. + +[[!tag projects/INM7]]
close, behaving as documented
diff --git a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered.mdwn b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered.mdwn index 35448a9709..907fefe443 100644 --- a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered.mdwn +++ b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered.mdwn @@ -26,3 +26,5 @@ I think that git-annex should have completely dissociated that remote from the k [[!meta author=yoh]] [[!tag projects/dandi]] + +> Behaving as documented. [[done]] --[[Joey]] diff --git a/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_1_a1c532b22645802a85eec35a05c2b1b0._comment b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_1_a1c532b22645802a85eec35a05c2b1b0._comment new file mode 100644 index 0000000000..b17a77ad11 --- /dev/null +++ b/doc/bugs/keeps_ext_remote_after_all_urls_unregistered/comment_1_a1c532b22645802a85eec35a05c2b1b0._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-07T18:41:20Z" + content=""" +Removing the last url from the web special remote makes it treat +content as no longer present in that remote. That is a documented special +case. + +For all other special remotes, git-anenx does not have any reason +to expect that removing a record of an url will mean that the special +remote will not be able to still retrieve content that was stored on it. + +reregisterurl is no different than unregisterurl in this respect. Both document +this special case. You've filed multiple bugs about rmurl having the same +behavior in the past, IIRC. And IIRC those all got closed, so I guess I'll +close this one too. +"""]]
comment
diff --git a/doc/todo/generic_p2p_socket_transport/comment_5_bb9d36eae53e3b1e806f8ae089aaf91a._comment b/doc/todo/generic_p2p_socket_transport/comment_5_bb9d36eae53e3b1e806f8ae089aaf91a._comment new file mode 100644 index 0000000000..b87c43dac3 --- /dev/null +++ b/doc/todo/generic_p2p_socket_transport/comment_5_bb9d36eae53e3b1e806f8ae089aaf91a._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="joey" + subject="""Re: comment 4""" + date="2025-01-07T18:29:44Z" + content=""" +> If the PSK were fully contained in the remote string then a third-party getting hold of that string could pretend to be the server + +I agree this would be a problem, but how would a third-party get ahold of +the string though? Remote urls don't usually get stored in the git +repository, perhaps you were thinking of some other way. +"""]]
comment
diff --git a/doc/todo/Peer_to_peer_connection_purely_over_magic-wormhole/comment_3_b0079bc09b5643d7c8ce37fa39f6ef19._comment b/doc/todo/Peer_to_peer_connection_purely_over_magic-wormhole/comment_3_b0079bc09b5643d7c8ce37fa39f6ef19._comment new file mode 100644 index 0000000000..8c0f1965b5 --- /dev/null +++ b/doc/todo/Peer_to_peer_connection_purely_over_magic-wormhole/comment_3_b0079bc09b5643d7c8ce37fa39f6ef19._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T18:27:11Z" + content=""" +[[generic_p2p_socket_transport]] is discussing adding something that could +perhaps be used to implement this. +"""]]
comment
diff --git a/doc/forum/import_and_export_treeish_for_rsync_and_webdav/comment_3_3d5b5044dd0906638e4f162816a19606._comment b/doc/forum/import_and_export_treeish_for_rsync_and_webdav/comment_3_3d5b5044dd0906638e4f162816a19606._comment new file mode 100644 index 0000000000..491ff278f1 --- /dev/null +++ b/doc/forum/import_and_export_treeish_for_rsync_and_webdav/comment_3_3d5b5044dd0906638e4f162816a19606._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T18:21:38Z" + content=""" +> Maybe this was improved after this forum post happened? + +No. All remotes that currently support import+export have protections in +place to make that guarantee that exporting will not overwrite. My comment +concerned the ability to implement exactly those protections. +"""]]
Fix installation on Android
Unfortunately, commit ce95cac19501bfbffd610203083510888a6b9646 broke
android installation way back in May of last year.
Unfortunately, commit ce95cac19501bfbffd610203083510888a6b9646 broke
android installation way back in May of last year.
diff --git a/CHANGELOG b/CHANGELOG index cdef9716d2..b401848477 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -5,6 +5,7 @@ git-annex (10.20250103) UNRELEASED; urgency=medium * log: Support --key, as well as --branch and --unused. * Avoid verification error when addurl --verifiable is used with an url claimed by a special remote other than the web. + * Fix installation on Android. -- Joey Hess <id@joeyh.name> Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/doc/bugs/Android_Installation_Error.mdwn b/doc/bugs/Android_Installation_Error.mdwn index bf16d41eac..1536276f74 100644 --- a/doc/bugs/Android_Installation_Error.mdwn +++ b/doc/bugs/Android_Installation_Error.mdwn @@ -28,3 +28,6 @@ I have tried downloading the `.tar.gz` file and extracting it manually. Inside I ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) Works great on all of my Fedora, Ubuntu, and Rocky machines! + +> Thanks for reporting, I have now fixed the bug in runshell. [[done]] +> --[[Joey]] diff --git a/doc/bugs/Installation_error_on_android.mdwn b/doc/bugs/Installation_error_on_android.mdwn index 3e1fbc1638..21102fc2ab 100644 --- a/doc/bugs/Installation_error_on_android.mdwn +++ b/doc/bugs/Installation_error_on_android.mdwn @@ -73,4 +73,4 @@ sed: can't read /data/data/com.termux/files/home/git-annex.linux/git-remote-anne ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Installation_error_on_android/comment_3_15c32517b85b6e91b7388d4f861eaf37._comment b/doc/bugs/Installation_error_on_android/comment_3_15c32517b85b6e91b7388d4f861eaf37._comment new file mode 100644 index 0000000000..a8c6e7f075 --- /dev/null +++ b/doc/bugs/Installation_error_on_android/comment_3_15c32517b85b6e91b7388d4f861eaf37._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T18:11:59Z" + content=""" +Indeed that is the same bug. And I've fixed it now, the next daily build +and release will have the fix. +"""]] diff --git a/standalone/linux/skel/runshell b/standalone/linux/skel/runshell index bbf2aad271..1360d31fe8 100755 --- a/standalone/linux/skel/runshell +++ b/standalone/linux/skel/runshell @@ -206,7 +206,7 @@ case "$os" in # The bundled git does not work well on sdcard, so delete # it and use termux's git which works better. cd "$base" - find . | grep git | grep -v git-annex | grep -v git-remote-tor-annex | grep -v git-remote-gcrypt | xargs rm -rf + find . | grep git | grep -v git-annex | grep -v git-remote-annex | grep -v git-remote-tor-annex | grep -v git-remote-gcrypt | xargs rm -rf # Use termux's uname, which knows it's on android, # not the bundled one. rm -f bin/uname
remove multiple dup comments
diff --git a/doc/bugs/Installation_error_on_android/comment_2_308a74b7a4ce5a8c8e4352c56f7e5663._comment b/doc/bugs/Installation_error_on_android/comment_2_308a74b7a4ce5a8c8e4352c56f7e5663._comment deleted file mode 100644 index 14a328e041..0000000000 --- a/doc/bugs/Installation_error_on_android/comment_2_308a74b7a4ce5a8c8e4352c56f7e5663._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="brendan.ward@a2e11ad27f6b2fa2c556aea6811496e0d95dd0da" - nickname="brendan.ward" - avatar="http://cdn.libravatar.org/avatar/bc7899f1aa523365cdfadc2ba3f7913d" - subject="comment 2" - date="2025-01-04T04:35:23Z" - content=""" -The same bug report was made about three months ago, as seen [here](https://git-annex.branchable.com/bugs/Android_Installation_Error/). -"""]] diff --git a/doc/bugs/Installation_error_on_android/comment_3_5373c0641034fddad22bd40d63ad439b._comment b/doc/bugs/Installation_error_on_android/comment_3_5373c0641034fddad22bd40d63ad439b._comment deleted file mode 100644 index 72173a3b0c..0000000000 --- a/doc/bugs/Installation_error_on_android/comment_3_5373c0641034fddad22bd40d63ad439b._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="brendan.ward@a2e11ad27f6b2fa2c556aea6811496e0d95dd0da" - nickname="brendan.ward" - avatar="http://cdn.libravatar.org/avatar/bc7899f1aa523365cdfadc2ba3f7913d" - subject="comment 3" - date="2025-01-04T04:39:20Z" - content=""" -The same bug report was made about three months ago, as seen [here](../Android_Installation_Error). -"""]] diff --git a/doc/bugs/Installation_error_on_android/comment_4_c5446ff939e6afd185fad952b2db8b46._comment b/doc/bugs/Installation_error_on_android/comment_4_c5446ff939e6afd185fad952b2db8b46._comment deleted file mode 100644 index fc7e85681f..0000000000 --- a/doc/bugs/Installation_error_on_android/comment_4_c5446ff939e6afd185fad952b2db8b46._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="brendan.ward@a2e11ad27f6b2fa2c556aea6811496e0d95dd0da" - nickname="brendan.ward" - avatar="http://cdn.libravatar.org/avatar/bc7899f1aa523365cdfadc2ba3f7913d" - subject="comment 4" - date="2025-01-04T04:41:14Z" - content=""" -The same bug report was made about three months ago, as seen here: [[Android Installation Error]]. -"""]] diff --git a/doc/bugs/Installation_error_on_android/comment_5_5f6be7a1d841e71aa4d1f160308027f7._comment b/doc/bugs/Installation_error_on_android/comment_5_5f6be7a1d841e71aa4d1f160308027f7._comment deleted file mode 100644 index d6d2ac66af..0000000000 --- a/doc/bugs/Installation_error_on_android/comment_5_5f6be7a1d841e71aa4d1f160308027f7._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Brendan" - avatar="http://cdn.libravatar.org/avatar/bc7899f1aa523365cdfadc2ba3f7913d" - subject="Same bug" - date="2025-01-04T04:46:11Z" - content=""" -The same bug was posted about 3 months ago in this report: [[Android Installation Error]]. -"""]]
comment
diff --git a/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_2_6aa84b45e90adceec69f4f413d325e62._comment b/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_2_6aa84b45e90adceec69f4f413d325e62._comment new file mode 100644 index 0000000000..d13c52fe08 --- /dev/null +++ b/doc/bugs/install_on_android_boox__58___xargs_Permission_denied/comment_2_6aa84b45e90adceec69f4f413d325e62._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2025-01-07T17:57:37Z" + content=""" +My guess would be that, since xargs is the first bundled binary from the +git-annex tarball that gets run, that the binary is either the wrong +architecture, or that something about the Android system prevents it from +running ELF binaries from termux's home directory. +"""]]
comment
diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_1_e5a1341612d4a45e89cc3f895ec266eb._comment b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_1_e5a1341612d4a45e89cc3f895ec266eb._comment new file mode 100644 index 0000000000..240319503a --- /dev/null +++ b/doc/bugs/keeps_trying_to_commit_file_unlocked/comment_1_e5a1341612d4a45e89cc3f895ec266eb._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-07T17:51:20Z" + content=""" +All I'm seeing here is a file that was locked and has gotten unlocked +somehow. + +The new and old versions of the file in `git diff` are the same key, +so this is not the same as the "inception" bug. + +Was the assistant running in this repository? + +If you're using the assistant, it's certainly possible that something +that it observed happening on disk prompted it to restage the file. +And the assistant always stages files unlocked. +"""]]
close, not a bug
mrisse posted a comment earlier explaining the user's confusion to them
mrisse posted a comment earlier explaining the user's confusion to them
diff --git a/doc/bugs/git_annex_copy_just_does_not_copy_sometimes.mdwn b/doc/bugs/git_annex_copy_just_does_not_copy_sometimes.mdwn index b86b1521e9..e881f412e8 100644 --- a/doc/bugs/git_annex_copy_just_does_not_copy_sometimes.mdwn +++ b/doc/bugs/git_annex_copy_just_does_not_copy_sometimes.mdwn @@ -33,3 +33,4 @@ Rocky OS 9 git-annex version: 10.20230926-g44a7b4c9734adfda5912dd82c1aa97c615689f57 +> Not a bug, [[done]] --[[Joey]]
retitle
diff --git a/doc/bugs/S3_bucket_not_configured.mdwn b/doc/bugs/S3_bucket_not_configured.mdwn index 25b04fd09b..b3f8d959ec 100644 --- a/doc/bugs/S3_bucket_not_configured.mdwn +++ b/doc/bugs/S3_bucket_not_configured.mdwn @@ -43,3 +43,4 @@ ConfigMonitor crashed: S3 bucket not configured ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) I've successfully used git-annex (on and off) since you ran the kickstarter to develop the assistant :) I really like the project. +[[!meta title="assistant: ConfigMonitor crashed: S3 bucket not configured"]]
comment
diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_fcd6d9b8a9c569710ffdbd7f9be74352._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_fcd6d9b8a9c569710ffdbd7f9be74352._comment new file mode 100644 index 0000000000..0e9bb9f44b --- /dev/null +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_fcd6d9b8a9c569710ffdbd7f9be74352._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T17:29:57Z" + content=""" +Only way I can see thawing removing write perms if is you have +core.sharedrepository configured to a particular umask value. + +But it certainly is possible that some other part of git-annex removes +write perms. + +And in particular the directory special remote does, when content is stored +in it. And you're using that. + +So, new theory: The GITBUNDLE object is stored on the directory special +remote. When it later gets pulled from it, the file has readonly attribute +set. + +Can you check if the directory special remote contains files with the +readonly attribute set, and see if clearing that attribute prevents the +problem from happening? +"""]]
remove empty comment
diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_3_158e63e75a95f186dbfbf9216c8e2102._comment b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_158e63e75a95f186dbfbf9216c8e2102._comment similarity index 100% rename from doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_3_158e63e75a95f186dbfbf9216c8e2102._comment rename to doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_158e63e75a95f186dbfbf9216c8e2102._comment diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment deleted file mode 100644 index 497f2e8fe9..0000000000 --- a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2025-01-07T17:06:46Z" - content=""" - -"""]]
reponse
diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment new file mode 100644 index 0000000000..497f2e8fe9 --- /dev/null +++ b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_2_48515c3eb5725dac082e750812a981d8._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2025-01-07T17:06:46Z" + content=""" + +"""]] diff --git a/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_3_158e63e75a95f186dbfbf9216c8e2102._comment b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_3_158e63e75a95f186dbfbf9216c8e2102._comment new file mode 100644 index 0000000000..5ceed5aea0 --- /dev/null +++ b/doc/bugs/git_annex_checkpresentkey_removes_git_credentials/comment_3_158e63e75a95f186dbfbf9216c8e2102._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2025-01-07T17:17:14Z" + content=""" +AFAICS it only does it on a 401. I cannot find a code path where +rejectUrlCredential is called with any other HTTP status. + +Are you still seeing this problem? +"""]]
todo from forum post
diff --git a/doc/forum/Automatic_fallback_to_special_remote_with_sameas/comment_1_4d5a3369f020e7df3231491b680602d3._comment b/doc/forum/Automatic_fallback_to_special_remote_with_sameas/comment_1_4d5a3369f020e7df3231491b680602d3._comment new file mode 100644 index 0000000000..73f6fe210d --- /dev/null +++ b/doc/forum/Automatic_fallback_to_special_remote_with_sameas/comment_1_4d5a3369f020e7df3231491b680602d3._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-07T16:28:03Z" + content=""" +Interesting idea. Even without --sameas, it's possible to have 2 remotes +that connect to the same repository in different ways, with one being +readonly and the other supporting write as well. So I don't think this is +really specific to --sameas. + +git-annex also has limited knowledge about whether a given special remote +is readonly or not. That would make any kind of automated fallback unlikely +to work well. + +I think there would need to be some config option to enable this. +Otherwise, the user might be surprised if git-annex used a remote that they +didn't want it to use for whatever reason. + +I've opened a todo, [[todo/config_different_remote_to_use_for_write_operations]]. +"""]] diff --git a/doc/todo/config_different_remote_to_use_for_write_operations.mdwn b/doc/todo/config_different_remote_to_use_for_write_operations.mdwn new file mode 100644 index 0000000000..b2d6a4b849 --- /dev/null +++ b/doc/todo/config_different_remote_to_use_for_write_operations.mdwn @@ -0,0 +1,21 @@ +Sometimes the same repository can be accessed via two remotes. One remote +might be readonly, but fast (or not needing a password) while the other one +supports writing. The idea is to have a config setting that makes git-annex +use the latter remote for write operations when the user tells it to +operate on the former remote. + +This is somewhat similar to `remote.<name>.annexUrl`, but that is for +different urls that access the same type of remote, eg http and ssh access +to a git repository. + +This would also support --sameas remotes where a data store can be +accessed via two different types of special remotes. + +The config could be something like `remote.<name>.annexWrite`. + +Implementation would need to somehow handle cases where a command can both +read and write. Eg, `git-annex sync --content foo` with +remote.foo.annexWrite=bar should use foo for reads, but bar for writes. +So the implementation can't just replace the remote at parameter parse +time. Instead, it seems it would need to modify the Remote object for foo, +making write actions call to the Remote object for bar. diff --git a/doc/todo/copy__47__move_support_for_pushinsteadOf_/comment_10_084e824d6fdace591409bda8a71ed411._comment b/doc/todo/copy__47__move_support_for_pushinsteadOf_/comment_10_084e824d6fdace591409bda8a71ed411._comment new file mode 100644 index 0000000000..cc4666e7d4 --- /dev/null +++ b/doc/todo/copy__47__move_support_for_pushinsteadOf_/comment_10_084e824d6fdace591409bda8a71ed411._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2025-01-07T16:57:04Z" + content=""" +Note that I've opened a related todo, +[[config_different_remote_to_use_for_write_operations]] +which might be a better approach to the `pushInsteadOf` +type of thing. +"""]]
Added a comment
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_1_60c69b1e48aba5afa6fd81f285321e89._comment b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_1_60c69b1e48aba5afa6fd81f285321e89._comment new file mode 100644 index 0000000000..970610d043 --- /dev/null +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks/comment_1_60c69b1e48aba5afa6fd81f285321e89._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 1" + date="2025-01-07T16:09:31Z" + content=""" +I even wonder if there could be some easy way to setup overall hook for `git-annex` invocation within a repo (hence within `.git/` of a repo) to e.g. safe-guard invocations of git-annex and prevent use within container environments (we often run into various problems). So smells to me like developing some kind of `.git/annex/hooks/` support analog to `.git/hooks`. Then whatever global `annex init` hook would setup for a repo within its `.git/annex/hooks` would be in effect for subsequent `annex` invocations within containers or natively. + +Other desired use-cases could be to prevent invocation of git-annex with known critical for that deployment defects/absent features (e.g. absent support of freeze/thaw scripts or some other recent feature). +"""]]
initial idea on `git annex init` hooks
diff --git a/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn new file mode 100644 index 0000000000..b5ba7190af --- /dev/null +++ b/doc/todo/Support___40__globally_configured__41_____34__annex_init__34___hooks.mdwn @@ -0,0 +1,5 @@ +The idea is stemmed from discussions/problems with using freeze/thaw hooks, and in particular [line of thinking in the comment on specify_freeze__47__thaw_scripts_relative_to_topdir](https://git-annex.branchable.com/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/#comment-c71b25bbd0e3f018e07812965bd6a5b1). ATM `git-annex` does analysis of either repository needs any special handling (adjusted branch, pidlock, etc) during `annex init` and otherwise does not bother. It would make sense to make it possible for a user to also be able similarly to + +- test e.g. if custom freeze/thaw are needed (before even git annex decides to switch to adjusted branches mode) and setup that repo accordingly for git-annex to proceed without flipping out into adjusted branch mode +- may be improved/custom pidlock detection (on one of my servers I remember needing to just hardcode in ~/.gitconfig to use pidlock although that was relevant only for some paths). +- similarly do some other testing which could allow or disallow some git-annex decision such as e.g. use of adjusted unlocked branch
report on the main repo being offline
diff --git a/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn new file mode 100644 index 0000000000..010d826aad --- /dev/null +++ b/doc/bugs/__40__site__47__server__41___main_repository_offline_2025-01-07.mdwn @@ -0,0 +1,45 @@ +### Please describe the problem. + +The main repository `git://git-annex.branchable.com/` seems to be offline at the moment. +The mirror `git://git.joeyh.name/git-annex` seems to work. + +### What steps will reproduce the problem? + +A git clone of the main repo gives an error message "fatal: Could not read from remote repository.". + +### What version of git-annex are you using? On what operating system? + +. + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log +C:\Users\jkniiv\Projektit\git-annex.branchable.com> git clone git://git-annex.branchable.com/ git-annex-clone-TEST +Cloning into 'git-annex-clone-TEST'... +fatal: Could not read from remote repository. + +Please make sure you have the correct access rights +and the repository exists. +C:\Users\jkniiv\Projektit\git-annex.branchable.com> ping git-annex.branchable.com + +Pinging git-annex.branchable.com [2600:3c03::f03c:91ff:fedf:c0e5] with 32 bytes of data: +Reply from 2600:3c03::f03c:91ff:fedf:c0e5: time=126ms +Reply from 2600:3c03::f03c:91ff:fedf:c0e5: time=116ms +Reply from 2600:3c03::f03c:91ff:fedf:c0e5: time=118ms +Reply from 2600:3c03::f03c:91ff:fedf:c0e5: time=117ms + +Ping statistics for 2600:3c03::f03c:91ff:fedf:c0e5: + Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), +Approximate round trip times in milli-seconds: + Minimum = 116ms, Maximum = 126ms, Average = 119ms + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Sure. It's always rockin'! + +[[!meta author=jkniiv]]
removed
diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment deleted file mode 100644 index 19f9f0d06a..0000000000 --- a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment +++ /dev/null @@ -1,78 +0,0 @@ -[[!comment format=mdwn - username="jkniiv" - avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" - subject="comment 3" - date="2025-01-06T08:54:05Z" - content=""" -> AFAIK, git-remote-annex is not installed on windows. I assume you set up the link to git-annex yourself. - -That's right. Knowing that some of the functionality of git-remote-annex was already present I made -a symlink in a directory in my `$env:PATH` with `cmd /c mklink git-remote-annex ..\path-to\git-annex.exe` -(I have developer mode active in Windows settings) and that made git-remote-annex available to me/git-annex. - -> This is a puzzling problem to me. I don't know anything about windows readonly attributes. But I don't think git-annex would ever set them. -> ->Indeed, it never freezes content on windows at all. That can be seen in the debug output you posted, where it does say it's \"thawing content\", but never \"freezing content\". - -That is puzzling. Could the thawing do the wrong/opposite thing in some cases? Also, remember -that anything that does `chmod u-w` by way of Git Bash's own shell (sh.exe/bash.exe), e.g. via `sh -c 'chmod u-w file'`, will effect -a Windows readonly attribute on that file (thanks to MSYS2 behind the scenes): - -``` -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> touch ddd -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls - - Directory: e:\git-annex-tests\test-git-remote-annex\annex-c - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 1.6.2024 23:28 5 a-1 --a--- 1.6.2024 23:28 5 b-2 --a--- 1.6.2024 23:28 7 c-3 --a--- 6.1.2025 10:31 0 ddd - -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'chmod u-w ddd' -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls - - Directory: e:\git-annex-tests\test-git-remote-annex\annex-c - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 1.6.2024 23:28 5 a-1 --a--- 1.6.2024 23:28 5 b-2 --a--- 1.6.2024 23:28 7 c-3 --ar-- 6.1.2025 10:31 0 ddd - -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> rm ddd -Remove-Item: You do not have sufficient access rights to perform this operation or the item is hidden, system, or read only. -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type chmod' -chmod is /usr/bin/chmod -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type mount' -mount is /usr/bin/mount -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c mount -C:/scoop/apps/git/2.47.1 on / type ntfs (binary,noacl,auto) -C:/scoop/apps/git/2.47.1/usr/bin on /bin type ntfs (binary,noacl,auto) -C:/Users/jkniiv/AppData/Local/Temp on /tmp type ntfs (binary,noacl,posix=0,usertemp) -C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto) -D: on /d type ntfs (binary,noacl,posix=0,user,noumount,auto) -E: on /e type ntfs (binary,noacl,posix=0,user,noumount,auto) - -``` - -But I guess git-annex isn't calling out to external `chmod` in these cases. - -> ->If you had somehow configured a freeze hook that set the readonly attribute, it would run it on windows. I suppose you would have thought to mention if that was the case though. - -Nope. No freeze/thaw hooks set. I'm not that brave nor cognizant of how to script them in a fool proof manner. :) - - -``` -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.freezecontent-command -e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.thawcontent-command -``` - -> ->Also rather puzzling is that this is a temp object file, and not a .git/annex/objects/ file. So the failure is apparently happening in the middle of downloading the GITBUNDLE object, before it gets moved to that location. But the same code is run at that point as by any download of any git-annex object. - -"""]]
Added a comment
diff --git a/doc/todo/git-annex-cat/comment_6_6d6fd092fd58c59d633b9ee381264328._comment b/doc/todo/git-annex-cat/comment_6_6d6fd092fd58c59d633b9ee381264328._comment new file mode 100644 index 0000000000..d86ddc3454 --- /dev/null +++ b/doc/todo/git-annex-cat/comment_6_6d6fd092fd58c59d633b9ee381264328._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="Doable8234" + avatar="http://cdn.libravatar.org/avatar/b0d5fea745f92c3b8cc8ecc3dafa6278" + subject="comment 6" + date="2025-01-07T02:11:33Z" + content=""" +Joey, I recently came across this same usecase. There are some intermediate files I store using git annex safely in the cloud and I want to fetch it. + +Doing a `git annex get` and a drop seems like the wrong solution. Why am I unnecessarily adding risk when I know I don't care about whether the file currently exists in my repo? I then have to think about various cases like if I already had the file in my repo or not and be very careful. I can't just do a `git annex get; cat; git annex drop`. + +I could use a pull-only-clone of my git annex repo, but that comes with many issues and usage hassles like reconfiguring everything. On top of this, I'd sometimes need to do a `git annex drop --force` in my clones since they may not have access to everything that the main repo does which is even more scary. + +Your concerns [here](http://git-annex.branchable.com/todo/git-annex-cat/#comment-8ca717fcdeadb1c2413da1f82d3659c6) make sense to me. However, streaming vs downloading is just an optimization. I'm HAPPY to pay the performance cost which is much better than the safety cost I'm currently facing with my hacky solutions to this problem. All we need (from my meager understanding of git annex internals) is to have the `git annex cat` command download the contents on to a temporary file (in the literal `/tmp` directory) instead of the `annex/objects` directory, and then `cat` that at the end. That's pretty much all I (we?) am asking for. + +I do know that you like to do things perfectly and I'm sure there'll be lots of issues with the proposal here that you can see that the rest of us cannot. But that's true of solutions too. Really really hoping you can figure out a solution for this. I'm happy to try and help with the code changes too if that helps. I have never used haskell before but very happy to take that challenge if we can settle on a design. +"""]]
Added a comment
diff --git a/doc/bugs/git-annex_proceeds_forward_if_freeze_script_fails/comment_2_ee183af32340ef0d5811ec746779dfa5._comment b/doc/bugs/git-annex_proceeds_forward_if_freeze_script_fails/comment_2_ee183af32340ef0d5811ec746779dfa5._comment new file mode 100644 index 0000000000..537f41293c --- /dev/null +++ b/doc/bugs/git-annex_proceeds_forward_if_freeze_script_fails/comment_2_ee183af32340ef0d5811ec746779dfa5._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2025-01-07T01:06:57Z" + content=""" +2nd demonstration on [add_config_var_preventing_adjusted_branch_mode/#comment-...](https://git-annex.branchable.com/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/#comment-d1335f67352cc698862464515363f061) demonstrates another scenario with undesired effect of failing to freeze/thaw -- git-annex just switches to adjusted branches mode. In that case there is likely no harm yet done to unwind and it would have better errored out altogether. +"""]]
Added a comment
diff --git a/doc/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/comment_5_69b52c6385ac2220575e27913ec916f8._comment b/doc/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/comment_5_69b52c6385ac2220575e27913ec916f8._comment new file mode 100644 index 0000000000..cc8c0c79f4 --- /dev/null +++ b/doc/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/comment_5_69b52c6385ac2220575e27913ec916f8._comment @@ -0,0 +1,129 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2025-01-07T01:03:24Z" + content=""" +well, `git-annex` is calling `init` upon initial `get` if it was not `init`ed before. +In our use case I think user cloned repository and then invoked `git annex get` within a container environment which did either not have access to original ~/.gitconfig or just to those thaw/freeze scripts. Result is the same as git-annex does not care if configured to be executed scripts fail to execute and plows forward switching to adjusted branches mode instead of errorring out. + +<details> +<summary>execution where `~/.gitconfig` is not bound at all - switches since there is no global configuration for thaw/freeze. Having some global variable preventing switching to adjusted mode also would have been of no help </summary> + +``` +[d31548v@discovery-01 tmp]$ git config annex.thawcontent-command +/dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content %path +[d31548v@discovery-01 tmp]$ singularity exec -B $PWD -c --cleanenv /dartfs/rc/lab/D/DBIC/DBIC/archive/containers/images/nipy/nipy-heudiconv--1.3.2.sing git config annex.thawcontent-command +[d31548v@discovery-01 tmp]$ ls +acl-with-separate-fd-aces facl hello.txt now testdir yohdir +[d31548v@discovery-01 tmp]$ git clone https://github.com/dandisets/000027 +Cloning into '000027'... +remote: Enumerating objects: 198, done. +remote: Counting objects: 100% (198/198), done. +remote: Compressing objects: 100% (121/121), done. +remote: Total 198 (delta 79), reused 171 (delta 52), pack-reused 0 (from 0) +Receiving objects: 100% (198/198), 24.36 KiB | 1.06 MiB/s, done. +Resolving deltas: 100% (79/79), done. +[d31548v@discovery-01 tmp]$ singularity exec -B $PWD -c --cleanenv /dartfs/rc/lab/D/DBIC/DBIC/archive/containers/images/nipy/nipy-heudiconv--1.3.2.sing git -C $PWD/000027 annex get sub-RAT123/sub-RAT123.nwb + + Filesystem does not allow removing write bit from files. + + Detected a crippled filesystem. + + Disabling core.symlinks. + + Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. + +Switched to branch 'adjusted/draft(unlocked)' +hint: The '.git/hooks/post-checkout' hook was ignored because it's not set as executable. +hint: You can disable this warning with `git config advice.ignoredHook false`. + + Remote origin not usable by git-annex; setting annex-ignore + + https://github.com/dandisets/000027/config download failed: Not Found +get sub-RAT123/sub-RAT123.nwb (from web...) +ok +(recording state in git...) + +``` +</details> + +<details> +<summary>and execution where I do bind ~/.gitconfig but then scripts themselves are not available. Here if git-annex just failed -- used might have had a better chance to understand the issue and would have not ended up in adjusted branches mode</summary> + +```shell + +Cloning into '000027'... +remote: Enumerating objects: 198, done. +remote: Counting objects: 100% (198/198), done. +remote: Compressing objects: 100% (121/121), done. +remote: Total 198 (delta 79), reused 171 (delta 52), pack-reused 0 (from 0) +Receiving objects: 100% (198/198), 24.36 KiB | 891.00 KiB/s, done. +Resolving deltas: 100% (79/79), done. +[d31548v@discovery-01 tmp]$ singularity exec -B $PWD -B $HOME/.gitconfig -c --cleanenv /dartfs/rc/lab/D/DBIC/DBIC/archive/containers/images/nipy/nipy-heudiconv--1.3.2.sing git -C $PWD/000027 annex get sub-RAT123/sub-RAT123.nwb +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content: not found +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content: not found + + Filesystem allows writing to files whose write bit is not set. + + Detected a crippled filesystem. + + Disabling core.symlinks. + + Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. + +Switched to branch 'adjusted/draft(unlocked)' +hint: The '.git/hooks/post-checkout' hook was ignored because it's not set as executable. +hint: You can disable this warning with `git config advice.ignoredHook false`. + + Remote origin not usable by git-annex; setting annex-ignore + + https://github.com/dandisets/000027/config download failed: Not Found +get sub-RAT123/sub-RAT123.nwb (from web...) +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content: not found +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content: not found +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content: not found +/usr/lib/git-annex.linux/shimmed/sh/sh: 1: /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content: not found +ok +(recording state in git...) + +``` +</details> + + +So, indeed -- likely having a config variable would have been of lesser value than if git-annex just errorred out as soon as configured scripts failed to execute since they are not available. + +*Might be worth a separate issue*: another \"bad\" thing happens is when `git annex init` is ran with freeze/thaw available, so it does not switch to `adjusted` branch mode, but user later invokes `annex get` without scripts being configured -- files silently become unprotected and writeable. + +<details> +<summary>demonstration</summary> + +```shell +[d31548v@discovery-01 tmp]$ git clone https://github.com/dandisets/000027 +Cloning into '000027'... +gremote: Enumerating objects: 198, done. +remote: Counting objects: 100% (198/198), done. +remote: Compressing objects: 100% (121/121), done. +remote: Total 198 (delta 79), reused 171 (delta 52), pack-reused 0 (from 0) +Receiving objects: 100% (198/198), 24.36 KiB | 831.00 KiB/s, done. +Resolving deltas: 100% (79/79), done. +t -[d31548v@discovery-01 tmp]$ git -C 000027 annex init +init + Remote origin not usable by git-annex; setting annex-ignore + + https://github.com/dandisets/000027/config download failed: Not Found +ok +(recording state in git...) +[d31548v@discovery-01 tmp]$ singularity exec -B $PWD -c --cleanenv /dartfs/rc/lab/D/DBIC/DBIC/archive/containers/images/nipy/nipy-heudiconv--1.3.2.sing git -C $PWD/000027 annex get sub-RAT123/sub-RAT123.nwb +get sub-RAT123/sub-RAT123.nwb (from web...) +ok +(recording state in git...) +[d31548v@discovery-01 tmp]$ ls -lL 000027/sub-RAT123/sub-RAT123.nwb +-rw-rw----+ 1 d31548v rc-DBIC 18792 Jan 6 19:54 000027/sub-RAT123/sub-RAT123.nwb +[d31548v@discovery-01 tmp]$ echo 123 >> 000027/sub-RAT123/sub-RAT123.nwb + +``` +</details> + +Hence overall suggesting that there should be some repository `.git/config` setting which would be set \"permanently\" upon `git annex init` to reliably enforce consistent use of the same thaw/freeze logic -- either it be built-in, or via specific tandem of freeze/thaw scripts (potentially related issue [on relative paths for configs](https://git-annex.branchable.com/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/?updated)). +"""]]
removed
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment deleted file mode 100644 index 494ac90786..0000000000 --- a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="may be %dotgit?" - date="2025-01-06T23:38:54Z" - content=""" -Original line of my thought was expressed in [this issue on github](https://github.com/dbic/handbook/issues/27). -One of the recent cases which made git-annex \"flip out\" into adjusted branch mode (yet to try to reproduce and follow up on [add_config_var_preventing_adjusted_branch_mode](https://git-annex.branchable.com/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/)), which happened when user executed datalad with git-annex inside a singularity container. -To facilitate reproducibility etc, we are aiming to minimize effects of outside elements on execution within container so bind mount only current dataset and transfer only some [git / git-annex settings](https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L88). -We could also check on paths for those scripts and bind mount them too. Also if relying on PATH, we would need somehow to ensure that inside the container PATH would point to them too (might be overridden by container's startup script since after all outside PATH might have little to do with inside -- think about running docker container on OSX). - -I think it would have been clean(er) if some initial invocation of current global git-annex freeze/thaw script which would potentially determine either it is needed or not at all (since some partitions might not need it, some need one kind, another - some other one), would instantiate in a given repository a copy of the specific freeze/thaw scripts tandem. But inability to specify relative path hinders that. May be similarly to `%path` , it could have some `%dotgit` or alike variable to point to location of `.git` folder, and our \"freeze/thaw\" installation script populating values like `thawcontent-command = %dotgit/bin-annex/thaw-content %path`? I guess also could simply add treatment of leading `./` to signal being relative to `.git/` folder. Such susbstitution would need to be done once upon reading that config setting per repo, there is no need to sense if script is there or not. Since if not -- it better error out instead of proceeding forward with \"default\" behavior (which seems to be \"switch to adjusted branch\"). - - -"""]]
encfs testing issue on v10 locked
diff --git a/doc/bugs/few_tests_fail_under_encfs.mdwn b/doc/bugs/few_tests_fail_under_encfs.mdwn new file mode 100644 index 0000000000..40ea013297 --- /dev/null +++ b/doc/bugs/few_tests_fail_under_encfs.mdwn @@ -0,0 +1,73 @@ +### Please describe the problem. + +`encfs` I think in general is supported but there are some minor glitches (submitting one about ssh separately) which seems complicate use of git-annex under it. + +### What steps will reproduce the problem? + +Running `git annex test` causes a few tests to FAIL. After `git-annex` gets to operate "nominally" under encfs, we better add testing under `encfs` to [datalad/git-annex daily tests](https://github.com/datalad/git-annex/blob/master/.github/workflows/build-ubuntu.yaml#L247) . + +Sample fail: + +``` +Tests + Repo Tests v10 locked + Init Tests + init: OK (0.97s) + add: OK (2.44s) + addurl: OK (2.65s) + conflict resolution (mixed locked and unlocked file): OK (9.30s) + version: OK (0.99s) + fix: FAIL (0.11s) + ./Test/Framework.hs:86: + git clone failed with unexpected exit code (transcript follows) + fatal: hardlink different from source at 'tmprepo4/.git/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904' + + Use -p '/fix/' to rerun this test only. + +``` + +and overall it is seems v10 locked specific tests + +``` +❯ ~/proj/CON/utils/bin/show-paths -f full-lines -e FAIL < .duct/logs/2025.01.04T19.33.23-680497_stdout +1350 Tests +1351 Repo Tests v10 locked +1358: fix: FAIL (0.11s) +1403 Tests +1404 Repo Tests v10 locked +1411: partial commit: FAIL (0.06s) +1417: reinject: FAIL (0.06s) +1449 Tests +1450 Repo Tests v10 locked +1457: edit (no pre-commit): FAIL (0.02s) +1463: magic: FAIL (0.03s) +``` + +### What version of git-annex are you using? On what operating system? + +```shell +❯ git annex version +git-annex version: 10.20241031 +build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Servant Benchmark Feeds Testsuite S3 WebDAV +dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.6 http-client-0.7.17 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +``` + + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +
Added a comment: may be %dotgit?
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment new file mode 100644 index 0000000000..494ac90786 --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_4_c6523cd4ee7c50d13f641c40e022ab8d._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="may be %dotgit?" + date="2025-01-06T23:38:54Z" + content=""" +Original line of my thought was expressed in [this issue on github](https://github.com/dbic/handbook/issues/27). +One of the recent cases which made git-annex \"flip out\" into adjusted branch mode (yet to try to reproduce and follow up on [add_config_var_preventing_adjusted_branch_mode](https://git-annex.branchable.com/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/)), which happened when user executed datalad with git-annex inside a singularity container. +To facilitate reproducibility etc, we are aiming to minimize effects of outside elements on execution within container so bind mount only current dataset and transfer only some [git / git-annex settings](https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L88). +We could also check on paths for those scripts and bind mount them too. Also if relying on PATH, we would need somehow to ensure that inside the container PATH would point to them too (might be overridden by container's startup script since after all outside PATH might have little to do with inside -- think about running docker container on OSX). + +I think it would have been clean(er) if some initial invocation of current global git-annex freeze/thaw script which would potentially determine either it is needed or not at all (since some partitions might not need it, some need one kind, another - some other one), would instantiate in a given repository a copy of the specific freeze/thaw scripts tandem. But inability to specify relative path hinders that. May be similarly to `%path` , it could have some `%dotgit` or alike variable to point to location of `.git` folder, and our \"freeze/thaw\" installation script populating values like `thawcontent-command = %dotgit/bin-annex/thaw-content %path`? I guess also could simply add treatment of leading `./` to signal being relative to `.git/` folder. Such susbstitution would need to be done once upon reading that config setting per repo, there is no need to sense if script is there or not. Since if not -- it better error out instead of proceeding forward with \"default\" behavior (which seems to be \"switch to adjusted branch\"). + + +"""]]
Added a comment: may be %dotgit?
diff --git a/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_3_389e75ae6df3a379f0ae79f38ff444f9._comment b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_3_389e75ae6df3a379f0ae79f38ff444f9._comment new file mode 100644 index 0000000000..2288d80867 --- /dev/null +++ b/doc/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/comment_3_389e75ae6df3a379f0ae79f38ff444f9._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="may be %dotgit?" + date="2025-01-06T23:38:39Z" + content=""" +Original line of my thought was expressed in [this issue on github](https://github.com/dbic/handbook/issues/27). +One of the recent cases which made git-annex \"flip out\" into adjusted branch mode (yet to try to reproduce and follow up on [add_config_var_preventing_adjusted_branch_mode](https://git-annex.branchable.com/projects/datalad/bugs-done/add_config_var_preventing_adjusted_branch_mode/)), which happened when user executed datalad with git-annex inside a singularity container. +To facilitate reproducibility etc, we are aiming to minimize effects of outside elements on execution within container so bind mount only current dataset and transfer only some [git / git-annex settings](https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L88). +We could also check on paths for those scripts and bind mount them too. Also if relying on PATH, we would need somehow to ensure that inside the container PATH would point to them too (might be overridden by container's startup script since after all outside PATH might have little to do with inside -- think about running docker container on OSX). + +I think it would have been clean(er) if some initial invocation of current global git-annex freeze/thaw script which would potentially determine either it is needed or not at all (since some partitions might not need it, some need one kind, another - some other one), would instantiate in a given repository a copy of the specific freeze/thaw scripts tandem. But inability to specify relative path hinders that. May be similarly to `%path` , it could have some `%dotgit` or alike variable to point to location of `.git` folder, and our \"freeze/thaw\" installation script populating values like `thawcontent-command = %dotgit/bin-annex/thaw-content %path`? I guess also could simply add treatment of leading `./` to signal being relative to `.git/` folder. Such susbstitution would need to be done once upon reading that config setting per repo, there is no need to sense if script is there or not. Since if not -- it better error out instead of proceeding forward with \"default\" behavior (which seems to be \"switch to adjusted branch\"). + + +"""]]
forgot to add this comment earlier
diff --git a/doc/bugs/VURL_verification_failure_on_first_download/comment_2_eca6d2515be111728b71fb19fd3557e1._comment b/doc/bugs/VURL_verification_failure_on_first_download/comment_2_eca6d2515be111728b71fb19fd3557e1._comment new file mode 100644 index 0000000000..3349141073 --- /dev/null +++ b/doc/bugs/VURL_verification_failure_on_first_download/comment_2_eca6d2515be111728b71fb19fd3557e1._comment @@ -0,0 +1,45 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2025-01-03T20:59:36Z" + content=""" +Thinking about consequences of generalizing this from the web +special remote to all special remotes that claim urls some more, I came up +with this scenario: + +A special remote claims some urls. But it can also store arbitrary +keys that are sent to it by git-annex. + +At first, `git-annex addurl --verifiable --relaxed` is used to download one +of the urls that the special remote claims. + +Later, that key gets copied back to the special remote. + +Then the special remote *corrupts* the content that was stored on it. + +Then, `git-annex get` is used to download the corrupted +content from the special remote. Let's say that the special remote, in this +case, sends the object file that was stored in it, rather than looking up +the url and retrieving that. + +This special remote doesn't do checksum verification itself, so +retreiveKeyFile succeeds despite the corruption, and returns UnVerified. + +Then git-annex verifies the content. And it fails verification. +But, since `--relaxed` was used when the VURL was generated, it has no +size. Which means any content from the special remote should be accepted. +Even though it's corrupted! + +The web special remote doesn't have this problem because it doesn't store +arbitrary git-annex objects. My conclusion is that a special remote that +does support storing arbitrary objects in it, and also supports claimUrl, +cannot work properly with `--relaxed` for VURLs. They could support +`--fast` still, but this is making me wonder if learning equivilant key +checksums for VURLs really should be generalized beyond the web special +remote. + +Maybe special remotes where that does make sense should do +the same kind of thing that the web special remote does. Then we would be +looking at an extension to the external special remote protocol, or some +utility command to use to register the content of a VURL key. +"""]]
Added a comment
diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment new file mode 100644 index 0000000000..19f9f0d06a --- /dev/null +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_3_520eaedda15b3a55f4c1f6d724734ff9._comment @@ -0,0 +1,78 @@ +[[!comment format=mdwn + username="jkniiv" + avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" + subject="comment 3" + date="2025-01-06T08:54:05Z" + content=""" +> AFAIK, git-remote-annex is not installed on windows. I assume you set up the link to git-annex yourself. + +That's right. Knowing that some of the functionality of git-remote-annex was already present I made +a symlink in a directory in my `$env:PATH` with `cmd /c mklink git-remote-annex ..\path-to\git-annex.exe` +(I have developer mode active in Windows settings) and that made git-remote-annex available to me/git-annex. + +> This is a puzzling problem to me. I don't know anything about windows readonly attributes. But I don't think git-annex would ever set them. +> +>Indeed, it never freezes content on windows at all. That can be seen in the debug output you posted, where it does say it's \"thawing content\", but never \"freezing content\". + +That is puzzling. Could the thawing do the wrong/opposite thing in some cases? Also, remember +that anything that does `chmod u-w` by way of Git Bash's own shell (sh.exe/bash.exe), e.g. via `sh -c 'chmod u-w file'`, will effect +a Windows readonly attribute on that file (thanks to MSYS2 behind the scenes): + +``` +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> touch ddd +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls + + Directory: e:\git-annex-tests\test-git-remote-annex\annex-c + +Mode LastWriteTime Length Name +---- ------------- ------ ---- +-a--- 1.6.2024 23:28 5 a-1 +-a--- 1.6.2024 23:28 5 b-2 +-a--- 1.6.2024 23:28 7 c-3 +-a--- 6.1.2025 10:31 0 ddd + +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'chmod u-w ddd' +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls + + Directory: e:\git-annex-tests\test-git-remote-annex\annex-c + +Mode LastWriteTime Length Name +---- ------------- ------ ---- +-a--- 1.6.2024 23:28 5 a-1 +-a--- 1.6.2024 23:28 5 b-2 +-a--- 1.6.2024 23:28 7 c-3 +-ar-- 6.1.2025 10:31 0 ddd + +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> rm ddd +Remove-Item: You do not have sufficient access rights to perform this operation or the item is hidden, system, or read only. +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type chmod' +chmod is /usr/bin/chmod +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type mount' +mount is /usr/bin/mount +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c mount +C:/scoop/apps/git/2.47.1 on / type ntfs (binary,noacl,auto) +C:/scoop/apps/git/2.47.1/usr/bin on /bin type ntfs (binary,noacl,auto) +C:/Users/jkniiv/AppData/Local/Temp on /tmp type ntfs (binary,noacl,posix=0,usertemp) +C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto) +D: on /d type ntfs (binary,noacl,posix=0,user,noumount,auto) +E: on /e type ntfs (binary,noacl,posix=0,user,noumount,auto) + +``` + +But I guess git-annex isn't calling out to external `chmod` in these cases. + +> +>If you had somehow configured a freeze hook that set the readonly attribute, it would run it on windows. I suppose you would have thought to mention if that was the case though. + +Nope. No freeze/thaw hooks set. I'm not that brave nor cognizant of how to script them in a fool proof manner. :) + + +``` +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.freezecontent-command +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.thawcontent-command +``` + +> +>Also rather puzzling is that this is a temp object file, and not a .git/annex/objects/ file. So the failure is apparently happening in the middle of downloading the GITBUNDLE object, before it gets moved to that location. But the same code is run at that point as by any download of any git-annex object. + +"""]]
Added a comment
diff --git a/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_2_c3f0dcb9b22fdebe22c5e7b5237b25d2._comment b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_2_c3f0dcb9b22fdebe22c5e7b5237b25d2._comment new file mode 100644 index 0000000000..e3bc5cdb05 --- /dev/null +++ b/doc/bugs/git-remote-annex_doesn__39__t_work_on_Windows___40__perms__41__/comment_2_c3f0dcb9b22fdebe22c5e7b5237b25d2._comment @@ -0,0 +1,77 @@ +[[!comment format=mdwn + username="jkniiv" + avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" + subject="comment 2" + date="2025-01-06T08:52:04Z" + content=""" +> AFAIK, git-remote-annex is not installed on windows. I assume you set up the link to git-annex yourself. + +That's right. Knowing that some of the functionality of git-remote-annex was already present I made +a symlink in a directory in my `$env:PATH` with `cmd /c mklink git-remote-annex ..\path-to\git-annex.exe` +(I have developer mode active in Windows settings) and that made git-remote-annex available to me/git-annex. + +> This is a puzzling problem to me. I don't know anything about windows readonly attributes. But I don't think git-annex would ever set them. +> +>Indeed, it never freezes content on windows at all. That can be seen in the debug output you posted, where it does say it's \"thawing content\", but never \"freezing content\". + +That is puzzling. Could the thawing do the wrong/opposite thing in some cases? Also, remember +that anything that does `chmod u-w` by way of Git Bash's own shell (sh.exe/bash.exe), e.g. via `sh -c 'chmod u-w file'`, will effect +a Windows readonly attribute on that file (thanks to MSYS2 behind the scenes): + +``` +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> touch ddd +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls + + Directory: e:\git-annex-tests\test-git-remote-annex\annex-c + +Mode LastWriteTime Length Name +---- ------------- ------ ---- +-a--- 1.6.2024 23:28 5 a-1 +-a--- 1.6.2024 23:28 5 b-2 +-a--- 1.6.2024 23:28 7 c-3 +-a--- 6.1.2025 10:31 0 ddd + +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'chmod u-w ddd' +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> ls + + Directory: e:\git-annex-tests\test-git-remote-annex\annex-c + +Mode LastWriteTime Length Name +---- ------------- ------ ---- +-a--- 1.6.2024 23:28 5 a-1 +-a--- 1.6.2024 23:28 5 b-2 +-a--- 1.6.2024 23:28 7 c-3 +-ar-- 6.1.2025 10:31 0 ddd + +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> rm ddd +Remove-Item: You do not have sufficient access rights to perform this operation or the item is hidden, system, or read only. +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type chmod' +chmod is /usr/bin/chmod +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c 'type mount' +mount is /usr/bin/mount +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked) +1 ~0 -0 !]> sh -c mount +C:/scoop/apps/git/2.47.1 on / type ntfs (binary,noacl,auto) +C:/scoop/apps/git/2.47.1/usr/bin on /bin type ntfs (binary,noacl,auto) +C:/Users/jkniiv/AppData/Local/Temp on /tmp type ntfs (binary,noacl,posix=0,usertemp) +C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto) +D: on /d type ntfs (binary,noacl,posix=0,user,noumount,auto) +E: on /e type ntfs (binary,noacl,posix=0,user,noumount,auto) + +``` + +But I guess git-annex isn't calling out to external `chmod` in these cases. + +> +>If you had somehow configured a freeze hook that set the readonly attribute, it would run it on windows. I suppose you would have thought to mention if that was the case though. + +Nope. No freeze/thaw hooks set. I'm not that brave nor cognizant of how to script them in a fool proof manner. :) + + +``` +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.freezecontent-command +e:\git-annex-tests\test-git-remote-annex\annex-c [adjusted/master(unlocked)]> git config get annex.thawcontent-command +``` + +> +>Also rather puzzling is that this is a temp object file, and not a .git/annex/objects/ file. So the failure is apparently happening in the middle of downloading the GITBUNDLE object, before it gets moved to that location. But the same code is run at that point as by any download of any git-annex object. +"""]]
Added a comment
diff --git a/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_3_2db0c65df19f6fb34d6291cbe59a3111._comment b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_3_2db0c65df19f6fb34d6291cbe59a3111._comment new file mode 100644 index 0000000000..add9fb932f --- /dev/null +++ b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_3_2db0c65df19f6fb34d6291cbe59a3111._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Doable8234" + avatar="http://cdn.libravatar.org/avatar/b0d5fea745f92c3b8cc8ecc3dafa6278" + subject="comment 3" + date="2025-01-06T05:03:22Z" + content=""" +Looking back, it looks like this has already been asked and phrased much better [here](https://git-annex.branchable.com/todo/git-annex-cat/). It's the TODO section and marked as `needsthought`. I'm going to add a comment to that instead of here and hopefully have a more fruitful discussion. +"""]]
Added a comment
diff --git a/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_2_1d16bd2e75011f55e8c337a4e603e346._comment b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_2_1d16bd2e75011f55e8c337a4e603e346._comment new file mode 100644 index 0000000000..62a257511d --- /dev/null +++ b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_2_1d16bd2e75011f55e8c337a4e603e346._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Doable8234" + avatar="http://cdn.libravatar.org/avatar/b0d5fea745f92c3b8cc8ecc3dafa6278" + subject="comment 2" + date="2025-01-06T05:02:21Z" + content=""" +Looking back, it looks like this has already been asked and phrased much better [here](https://git-annex.branchable.com/todo/git-annex-cat/). It's the TODO section and marked as `needsthought`. I'm going to add a comment to that instead of here and hopefully have a more fruitful discussion. +"""]]
Added a comment
diff --git a/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_1_670b93dd6fbed3aa9d16b915bd6274bc._comment b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_1_670b93dd6fbed3aa9d16b915bd6274bc._comment new file mode 100644 index 0000000000..5e6ea50226 --- /dev/null +++ b/doc/forum/How_can_I_ONLY_fetch_a_file_from_a_special_remote__63__/comment_1_670b93dd6fbed3aa9d16b915bd6274bc._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Doable8234" + avatar="http://cdn.libravatar.org/avatar/b0d5fea745f92c3b8cc8ecc3dafa6278" + subject="comment 1" + date="2025-01-06T05:00:47Z" + content=""" +I did a simple search and it looks like it's already been asked and phrased much better [here](https://git-annex.branchable.com/todo/git-annex-cat/). It's in the todo forum and tagged with `needsthought`. I'm going to add a comment with some of my thoughts and hopefully have a more fruitful discussion there. +"""]]
initial report on encfs and per-repository ssh caching
diff --git a/doc/todo/either_support_or_fail___34__more_gracefully__34___on_encfs.mdwn b/doc/todo/either_support_or_fail___34__more_gracefully__34___on_encfs.mdwn index d6d332d542..4b33938e00 100644 --- a/doc/todo/either_support_or_fail___34__more_gracefully__34___on_encfs.mdwn +++ b/doc/todo/either_support_or_fail___34__more_gracefully__34___on_encfs.mdwn @@ -56,3 +56,64 @@ git-annex: cannot determine uuid for origin (perhaps you need to run "git annex ``` which is simply due to the fact that git-annex does not only unable to parse, it is unable to connect. But if so, IMHO ideally it should avoid claiming anything about git annex installation there. + + + +After spending some time in troubleshooting the situation that `git-annex` for some reason claims that `origin` remote does not support git-annex, I have ran into [11 year old discussion](https://git-annex.branchable.com/forum/ssh__95__exchange__95__identification__58___read__58___Connection_reset_by_peer/#comment-1776b4d3b175c77cdb44fc8656e3352c) which boils down to + +> It would be a good idea to `git config annex.sshcaching` false in repositories stored on encfs. + +As AFAIK encfs is generally working ok (seems to test ok with bleeding edge [ref](https://github.com/datalad/git-annex/issues/216)), and overall if there is something wrong, it better be fixed and not "avoided", I think that git-annex should either inform user about necessity to not do internal caching within the same repository (e.g. to have it configured to be shared somewhere under ~/.cache/` instead or a like) if file system does not support the ssh caching, or even somehow automagically handle the case, but it should not mislead in its reporting stating that remote does not support git-annex whenever it does. + +With this minimalistic reproducer where origin on a regular filesystem and a clone is under `encfs`, we receive a misleading error: + +<details> +<summary>reproducer</summary> + +``` +#!/bin/bash +export PS4='> ' +set -x +set -eu + +cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)" + +topd=$PWD +mkdir -p denc{r,}; encfs --standard --extpass="echo 123" $PWD/denc{r,} + +cleanup () { + # to possible kill/troubleshoot + if ! fusermount -u $topd; then + lsof $topd/denc + echo "ERROR: failed to unmount" + exit 1 + fi +} +trap cleanup SIGINT SIGHUP SIGABRT EXIT + + +mkdir d +(cd d; git init; git annex init; echo 1 > 1.dat; git annex add 1.dat; git commit -m 1 1.dat) + +echo "cloning locally and ssh works ok, testing" +git clone localhost:$topd/d d-clone +git -C d-clone annex wanted origin + +echo "but would not be happy under encfs with misleading error:" +cd denc +git clone localhost:$topd/d d-clone +git -C d-clone annex wanted origin + +``` +</details> + + +``` +> git -C d-clone annex wanted origin + Unable to parse git config from origin + Remote origin does not have git-annex installed; setting annex-ignore + This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote origin +git-annex: cannot determine uuid for origin (perhaps you need to run "git annex sync"?) +``` + +which is simply due to the fact that git-annex does not only unable to parse, it is unable to connect. But if so, IMHO ideally it should avoid claiming anything about git annex installation there.
initial report on encfs and per-repository ssh caching
diff --git a/doc/todo/fail_on_encfs_more_informatively_or_do_not_fail_.mdwn b/doc/todo/fail_on_encfs_more_informatively_or_do_not_fail_.mdwn new file mode 100644 index 0000000000..22722b5a53 --- /dev/null +++ b/doc/todo/fail_on_encfs_more_informatively_or_do_not_fail_.mdwn @@ -0,0 +1,60 @@ +reposting since initial post seems has failed with odd https://www.oneukrainian.com/tmp/annex-blew-1.png . + +After spending some time in troubleshooting the situation that `git-annex` for some reason claims that `origin` remote does not support git-annex, I have ran into [11 year old discussion](https://git-annex.branchable.com/forum/ssh__95__exchange__95__identification__58___read__58___Connection_reset_by_peer/#comment-1776b4d3b175c77cdb44fc8656e3352c) which boils down to + +> It would be a good idea to `git config annex.sshcaching` false in repositories stored on encfs. + +As AFAIK encfs is generally working ok (seems to test ok with bleeding edge [ref](https://github.com/datalad/git-annex/issues/216)), and overall if there is something wrong, it better be fixed and not "avoided", I think that git-annex should either inform user about necessity to not do internal caching within the same repository (e.g. to have it configured to be shared somewhere under ~/.cache/` instead or a like) if file system does not support the ssh caching, or even somehow automagically handle the case, but it should not mislead in its reporting stating that remote does not support git-annex whenever it does. + +With this minimalistic reproducer where origin on a regular filesystem and a clone is under `encfs`, we receive a misleading error: + +<details> +<summary>reproducer</summary> + +``` +#!/bin/bash +export PS4='> ' +set -x +set -eu + +cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)" + +topd=$PWD +mkdir -p denc{r,}; encfs --standard --extpass="echo 123" $PWD/denc{r,} + +cleanup () { + # to possible kill/troubleshoot + if ! fusermount -u $topd; then + lsof $topd/denc + echo "ERROR: failed to unmount" + exit 1 + fi +} +trap cleanup SIGINT SIGHUP SIGABRT EXIT + + +mkdir d +(cd d; git init; git annex init; echo 1 > 1.dat; git annex add 1.dat; git commit -m 1 1.dat) + +echo "cloning locally and ssh works ok, testing" +git clone localhost:$topd/d d-clone +git -C d-clone annex wanted origin + +echo "but would not be happy under encfs with misleading error:" +cd denc +git clone localhost:$topd/d d-clone +git -C d-clone annex wanted origin + +``` +</details> + + +``` +> git -C d-clone annex wanted origin + Unable to parse git config from origin + Remote origin does not have git-annex installed; setting annex-ignore + This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote origin +git-annex: cannot determine uuid for origin (perhaps you need to run "git annex sync"?) +``` + +which is simply due to the fact that git-annex does not only unable to parse, it is unable to connect. But if so, IMHO ideally it should avoid claiming anything about git annex installation there.