only pass unlocked files through the clean/smudge filter

Right now, non-annexed files get passed through the annex clean/smudge filter (see Adding files to git: Very long "recording state in git" phase). It would be better if git-annex configure the filter only for the annexed unlocked files, in the .gitattributes file at the root of the repository.

not a viable solution, done --Joey

RSS Atom

comment 1

It immediately occurs to me that the proposal would break this:

git annex add foo
git annex add bar
git annex unlock bar
git mv bar foo
git commit -m add

Since foo was a locked file, gitattributes would prevent from being smudged, so the large content that was in bar gets committed directly to git.

The right solution is to improve the smudge/clean filter interface to it's not so slow, which there is copious discussion of elsewhere.

Comment by joey — Fri Nov 22 16:01:26 2019

Remove comment

moving unlocked file onto locked file isn't possible

git mv won't move an unlocked file onto a locked file (trace below).

"The right solution is to improve the smudge/clean filter interface" -- of course, but realistically, do you think git devs can be persuaded to do this sometime soon? Even if yes, it still seems better to avoid adding a step to common git workflows, than to make the step fast.

(master_env_v164_py36) 11:14  [t1] $ ls
bar  foo
(master_env_v164_py36) 11:14  [t1] $ git init
Initialized empty Git repository in /tmp/t1/.git/
(master_env_v164_py36) 11:14  [t1] $ git annex init
init  (scanning for unlocked files...)
ok
(recording state in git...)
(master_env_v164_py36) 11:14  [t1] $ git annex add foo
add foo ok
(recording state in git...)
(master_env_v164_py36) 11:14  [t1] $ git annex add bar
add bar ok
(recording state in git...)
(master_env_v164_py36) 11:14  [t1] $ ls -alt
total 0
drwxrwxr-x  8 ilya ilya 141 Nov 24 11:14 .git
drwxrwxr-x  3 ilya ilya  40 Nov 24 11:14 .
lrwxrwxrwx  1 ilya ilya 108 Nov 24 11:14 bar -> .git/annex/objects/jx/MV/MD5E-s4--c157a79031e1c40f85931829bc5fc552/MD5E-s4--c157a79031\
e1c40f85931829bc5fc552
lrwxrwxrwx  1 ilya ilya 108 Nov 24 11:14 foo -> .git/annex/objects/00/zZ/MD5E-s4--d3b07384d113edec49eaa6238ad5ff00/MD5E-s4--d3b07384d1\
13edec49eaa6238ad5ff00
drwxrwxrwt 12 root root 282 Nov 24 11:14 ..
(master_env_v164_py36) 11:14  [t1] $ git annex unlock bar
unlock bar ok
(recording state in git...)
(master_env_v164_py36) 11:16  [t1] $ git mv bar foo
fatal: destination exists, source=bar, destination=foo
(master_env_v164_py36) 11:17  [t1] $

Comment by Ilya_Shlyakhter — Sun Nov 24 16:36:24 2019

Remove comment

even git mv -f seems to work correctly

Also, git mv seems to reuse the already-smudged object contents of the source file for the target file, so even with git mv -f only the checksum gets checked into git:

+ cat ./test-git-mv
#!/bin/bash

set -eu -o pipefail -x

cat $0

TEST_DIR=/tmp/test_dir
mkdir -p $TEST_DIR
chmod -R u+w $TEST_DIR
rm -rf $TEST_DIR
mkdir -p $TEST_DIR
pushd $TEST_DIR

git init
git annex init

git --version
git annex version

rm .git/info/attributes
echo foo > foo
echo bar > bar
git annex add foo bar
git check-attr -a foo
git check-attr -a bar
echo 'bar filter=annex' > .gitattributes
git add .gitattributes
git check-attr -a foo
git check-attr -a bar

git annex unlock bar
git mv bar foo  || true
git mv -f bar foo
git commit -m add
git log -p


+ TEST_DIR=/tmp/test_dir
+ mkdir -p /tmp/test_dir
+ chmod -R u+w /tmp/test_dir
+ rm -rf /tmp/test_dir
+ mkdir -p /tmp/test_dir
+ pushd /tmp/test_dir
/tmp/test_dir /tmp
+ git init
Initialized empty Git repository in /tmp/test_dir/.git/
+ git annex init
init  (scanning for unlocked files...)
ok
(recording state in git...)
+ git --version
git version 2.20.1
+ git annex version
git-annex version: 7.20191024-g6dc2272
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: linux x86_64
supported repository versions: 7
upgrade supported from repository versions: 0 1 2 3 4 5 6
local repository version: 7
+ rm .git/info/attributes
+ echo foo
+ echo bar
+ git annex add foo bar
add foo ok
add bar ok
(recording state in git...)
+ git check-attr -a foo
+ git check-attr -a bar
+ echo 'bar filter=annex'
+ git add .gitattributes
+ git check-attr -a foo
+ git check-attr -a bar
bar: filter: annex
+ git annex unlock bar
unlock bar ok
(recording state in git...)
+ git mv bar foo
fatal: destination exists, source=bar, destination=foo
+ true
+ git mv -f bar foo
+ git commit -m add
[master (root-commit) 8610c0d] add
 2 files changed, 2 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 foo
+ git log -p
commit 8610c0d8f327140608e71dc229f167731552d284
Author: Ilya Shlyakhter <ilya_shl@alum.mit.edu>
Date:   Sun Nov 24 12:24:28 2019 -0500

    add

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..649f07e
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1 @@
+bar filter=annex
diff --git a/foo b/foo
new file mode 100644
index 0000000..266ae50
--- /dev/null
+++ b/foo
@@ -0,0 +1 @@
+/annex/objects/MD5E-s4--c157a79031e1c40f85931829bc5fc552

Comment by Ilya_Shlyakhter — Sun Nov 24 17:25:32 2019

Remove comment

installing clean/smudge filter lazily

"the proposal would break this" -- suppose git-annex-unlock was changed to install the clean/smudge filter for * if not installed yet?

Comment by Ilya_Shlyakhter — Fri Mar 19 02:30:13 2021

Remove comment

comment 5

"the proposal would break this" -- suppose git-annex-unlock was changed to install the clean/smudge filter for * if not installed yet?

git-annex unlock is not the only way unlocked files can appear in your tree. Consider git pull.

Comment by joey — Tue Mar 23 16:02:47 2021

Remove comment

Add a comment