forumgit-annexhttp://git-annex.branchable.com/forum/git-annexikiwiki2024-03-04T17:36:21ZNew external special remote for rclonehttp://git-annex.branchable.com/forum/New_external_special_remote_for_rclone/2024-03-04T17:36:21Z2024-03-04T17:36:21Z
<p>Hi folks, I'm working on a new external special remote for rclone named "git-annex-remote-rclone-goyle", or "garrgoyle" for short. (I chose a unique name to disambiguate from the preexisting <a href="https://github.com/git-annex-remote-rclone/git-annex-remote-rclone">git-annex-remote-rclone</a> project.)</p>
<p>Like git-annex-remote-rclone, garrgoyle only supports the <a href="https://git-annex.branchable.com/design/external_special_remote_protocol/">external special remote protocol</a>. In the near future, I'd like to add support for the <em>simple export interface</em> (see <a href="https://git-annex.branchable.com/design/external_special_remote_protocol/export_and_import_appendix/">export_and_import_appendix</a>).</p>
<p>Garrgoyle seems to be faster than git-annex-remote-rclone. Anecdotally, it was 10x faster when I asked to copy ~20 files that were already present on a "drive" remote. I think the difference is that garrgoyle pays the rclone startup cost once, rather than once per action.</p>
<p>I've uploaded a PR to rclone's GitHub repo <a href="https://github.com/rclone/rclone/pull/7654">https://github.com/rclone/rclone/pull/7654</a>. If anyone is willing to alpha test, I would greatly appreciate it! Feedback on the PR is also welcome.</p>
Possible to restore from an encrypted s3 remote?http://git-annex.branchable.com/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/2024-03-04T17:03:28Z2024-03-04T17:03:28Z
<p>I tried creating a new git annex repo on my desktop and adding the remote but I think I got an error about both repos not having the same idea. I might be wrong.</p>
<p>Is it possible to init a git annex repo on my desktop using an encrypted s3 remote?</p>
<p>thanks</p>
git-annex unused on directory special remote?http://git-annex.branchable.com/forum/git-annex_unused_on_directory_special_remote__63__/2024-02-27T02:23:33Z2024-02-27T02:23:33Z
<p>Should git-annex unused work on a directory special remote? I tried the following:</p>
<pre><code>$ git-annex unused --from SanDisk64GB4Torrented
unused SanDisk64GB4Torrented (checking for unused data...) ok
$
</code></pre>
<p>However, despite git-annex list --in SanDisk64GB4Torrented demonstrating that there were no annex files in it, the size of the directory was still large. And I can see a file in tmp/ of that annex's directory:</p>
<pre><code>$ ls tmp/
SHA256E-s2351316992--bdfba5df0bd72cffdb398fe885d9e36d052617647c0ae4fd0579a8fc785c95ba.iso
$
</code></pre>
<p>Am I right to think that the file being present in tmp/ should cause git-annex unused to list it?</p>
<p>I'm assuming it's fine to just remove the file. But I guess this might be a bug... so I'm putting it out there just in case.</p>
rsync test remote failhttp://git-annex.branchable.com/forum/rsync_test_remote_fail/2024-02-25T00:55:19Z2024-02-25T00:55:19Z
<p>Hi,</p>
<p>I am running into a bizarre testremote failure (with an rsync+encryption=shared special remote).</p>
<p>Here are the exact steps to replicate:</p>
<pre><code>$ mkdir test
$ cd test/
$ git init
$ git annex init
$ git annex initremote newremote type=rsync rsyncurl=10.0.0.100:/backup/annex-enc/test encryption=shared
$ git annex testremote newremote
Here is what I will get
testremote newremote (generating test keys...) Remote Tests
unavailable remote
removeKey: OK
storeKey: OK
checkPresent: OK
retrieveKeyFile: OK
retrieveKeyFileCheap: OK
key size Just 1048576; remote chunksize=0 encryption=none
removeKey when not present: OK (1.51s)
present False: OK (0.62s)
storeKey: OK (1.10s)
present True: FAIL (0.62s)
./Command/TestRemote.hs:290:
failed
storeKey when already present: OK (0.37s)
present True: FAIL (0.63s)
./Command/TestRemote.hs:290:
failed
retrieveKeyFile: rsync: change_dir "/backup/annex-enc/test/242/793/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
rsync: [Receiver] write error: Broken pipe (32)
rsync exited 23
rsync: change_dir "/backup/annex-enc/test/W4/4g/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
rsync: [Receiver] write error: Broken pipe (32)
rsync exited 23
FAIL (0.61s)
./Command/TestRemote.hs:290:
failed
fsck downloaded object: OK
retrieveKeyFile resume from 33%: FAIL
Exception: .git/annex/objects/W4/4g/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key: openBinaryFile: does not exist (No such file or directory)
fsck downloaded object: OK
retrieveKeyFile resume from 0: rsync: change_dir "/backup/annex-enc/test/242/793/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
rsync: [Receiver] write error: Broken pipe (32)
rsync exited 23
rsync: change_dir "/backup/annex-enc/test/W4/4g/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
rsync: [Receiver] write error: Broken pipe (32)
rsync exited 23
FAIL (0.61s)
./Command/TestRemote.hs:290:
failed
fsck downloaded object: OK
retrieveKeyFile resume from end: cp: cannot stat '.git/annex/objects/W4/4g/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key': No such file or directory
</code></pre>
<p>This will repeat again and again (with different remote chunksize?) until finally it hangs where I would have to Ctrl+C it to force it to abort.</p>
<p>I am using Ubuntu 22.04.3 LTS (Jammy) with the following git annex version:</p>
<pre><code>$ git annex version
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
</code></pre>
<p>I can't see why git-annex rsync would be failing. I can do normal rsync via command line w/o any issues.</p>
<p>Any help would be appreciated.</p>
copying annex between remotes manually?http://git-annex.branchable.com/forum/copying_annex_between_remotes_manually__63__/2024-02-20T20:51:02Z2024-02-20T20:51:02Z
<p>I'm currently trying to migrate a git-annex repository to a new machine, one of whose remotes is the one from <a href="https://git-annex.branchable.com/bugs/apparent_hang_in_git-annex-smudge/">this issue</a>. I determined that the root cause there seemed to be under git rather than git-annex; in particular, any whole-repository operation would take multiple days to execute, for unclear reasons. Pulling the commit data to a new repository seems to fix this.</p>
<p>I'm now trying to move all the annexed data from the original, broken remote to the new one. The default option here would be <code>git annex move</code>. However, when I run this, it apparently does some git operation for every key moved, taking hours to days; there are tens of thousands of keys, so this is obviously unworkable.</p>
<p>Is there a way to simply mass-move the annexed data into the new repo, via rsync or similar, and then update the new repo's metadata all at once? The state of the old repository does not matter, since I intend to discard it as soon as this migration is done.</p>
git annex lock/unlock/fix on non-committed fileshttp://git-annex.branchable.com/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files/2024-02-16T09:25:49Z2024-02-16T09:25:01Z
<p>In an ideal world, one could just use <code>git annex diffdriver</code> to compare annex'ed files. However, sometimes one might need an approach that is constructed simpler. For example, I want to run a comparison utility that is in a container which translates my paths for better reproducibility of scientific results. Or I have a colleage who is not familiar with all the intricacies of git annex and knows git just enough <code>git checkout</code> version 1, move it, the <code>git checkout</code> the second version and compare these two. Then the situation can arise that one has a symlink to <code>.git/annex/objects/...</code> which is named differently than what is saved in the tree.</p>
<p>In that case, <code>git annex unlock</code> refuses to work:</p>
<pre><code>mv file file_version1
git annex unlock file_version1
error: pathspec 'file' did not match any file(s) known to git
Did you forget to 'git add'?
unlock: 1 failed
</code></pre>
<p>Similar story goes with <code>git lock</code> and <code>git fix</code>. I think it would be a nice behaviour if these commands would just do their work on the new file name. For example, git unlock would replace a symlink with its content even if the file is not committed with this name.</p>
<p>Is it possible to implement this in git annex?</p>
Git Annex shirts at hellotux.comhttp://git-annex.branchable.com/forum/Git_Annex_shirts_at_hellotux.com/2024-02-13T19:36:10Z2024-02-13T19:36:10Z
<p>The fine people over at https://hellotux.com would love to make git-annex branded shirts, hoodies and currently also backpacks with git-annex branding. We recently started a cooperation for DataLad shirts, which are now available <a href="https://www.hellotux.com/datalad">here</a>. I have one of those and they look and feel very good. I would love to have the same with git annex's logo embroidered on it. Especially as the distribits meeting is upcoming, now would be a good time to give Gabor from hellotux a quick go that he may start. It won't cost git-annex anything, for datalad the process was very simple: send Gabor the SVG, they make a test stitch and if that looks fine, he'll add it to the webpage.</p>
<p>@joey if you're against this for some reason, you may just say so, then I'll stop bringing this up but I'd like to know what you think of it. 😃</p>
move files under git-annex and graft history?http://git-annex.branchable.com/forum/move_files_under_git-annex_and_graft_history__63__/yoh2024-02-09T17:54:00Z2024-02-09T17:54:00Z
<p>I do not know yet if that it possible since I have only cursory knowledge of former grafts in git and what new came to replace them.</p>
<p>The use case: we have a git-annex repo (heudiconv converted MRI data) where some files with sensitive information were added directly to git not git-annex. We do not want to rewrite the entire history since that repository already saw a good number of clones and forks. I wondered if we could</p>
<ol>
<li>move those files under git-annex</li>
<li>establish a new history from that new tree object</li>
<li>"graft" new history commit into the old "sensitive" one as to establish correspondence between the two
<ul>
<li>I would expect then something like "git pull --ff-only" to work for people seamlessly jumping between those points</li>
</ul>
</li>
<li>git push only new history to github thus not revealing old history with sensitive data under git</li>
</ol>
<p>Joey, WDYT? any words of wisdom?</p>
agent keep deleting local filehttp://git-annex.branchable.com/forum/agent_keep_deleting_local_file/2024-02-05T20:39:59Z2024-02-05T20:39:59Z
<p>I realize this is git 101, and I apologize in advance, but I honestly have no idea how to debug this.</p>
<p>I have two Macs, with git annex assistant running headless on both. I renamed a file on one system somewhere along the line, and now whenever the assistant syncs changes from system B to system A, I always find myself in a situation like this:</p>
<pre><code>On branch master
Your branch is behind 'mbp/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
deleted: myfile.md
</code></pre>
<p>This file was originally myfile-1234.md or whatever, but I renamed it on system A, and those changes were synced. Today, on system B, that file exists as myfile.md as I'd expect. On system A, however, it always gets deleted whenever I make changes made to any file on system B that the assistant then syncs over.</p>
<p>I've tried re-adding the file and committing, but it continues to be deleted every time system B syncs changes to system A.</p>
<p>I don't know what to do except abandon this file path completely and just use a different filename that git doesn't think is deleted on one of the systems; the correct solution to this eludes me.</p>
<p>I appreciate any suggestions that you might have.</p>
<p>Thanks.</p>
`git annex assistant` command replace all files with pointer file.http://git-annex.branchable.com/forum/__96__git_annex_assistant__96___command_replace_all_files_with_pointer_file./2024-02-04T10:56:30Z2024-02-04T10:47:59Z
<h2>Expect behaviors</h2>
<p><code>git annex assistant</code> doesn't annex the old files and annex "non-large files". According to the below quote from the <a href="https://git-annex.branchable.com/git-annex-assistant/"><code>git-annex assistant</code></a> document:</p>
<blockquote><p>By default, all files added are added to the annex, the same as when you run git annex add. If you configure annex.largefiles, files that it does not match will instead be added with git add.</p></blockquote>
<h2>Actual behaviors</h2>
<p>Git annex replaces all old files (file is already stored by git before the <code>git annex</code> command) and newly added files with pointer files after using the <code>git annex assistant</code> command. Even I config <code>annex.largefiles</code> to include all files.</p>
<p><a href="https://github.com/BilderLoong/test">Example repo</a></p>
<h2>Context</h2>
<p>Output of <code>git annex version</code>:</p>
<pre><code>git-annex version: 10.20240129
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.3 http-client-0.7.16 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
</code></pre>
<p>System: macOS Monterey Version 12.6 (21G115)</p>