git annex move --unused not moving some files

Please describe the problem.

git annex move --unused is not moving some of the files git annex unused reports:

Calibre Library % git annex unused 
unused . (checking for unused data...) (checking master...) (checking refs/annex/last-index...) 
  Some annexed data is no longer used by any files:
    NUMBER  KEY
    1       SHA256E-s182401--1c42279d1ea477f2a1831a3f7543481e7d5c72cbbeffdd0442918253b4f3ecc9.jpg
    2       SHA256E-s127374--17bfedf7453382df04a58d9c508d82285970148e49a84c488a728f7f770d8cc3.jpg
[...]
    56      SHA256E-s427315--3617fae344fb4fbc825963ee0ddfc5a31f1000c8d1c61ce3955bc16ae625e207.epub
  (To see where this data was previously used, run: git annex whereused --historical --unused

  To remove unwanted data: git-annex dropunused NUMBER

ok
Calibre Library % git annex move --unused -t bims 
Calibre Library %

Is there anything that explains this behavior?

What steps will reproduce the problem?

I am not sure how to reproduce this in another repo.

What version of git-annex are you using? On what operating system?

git-annex version: 8.20210803

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

Sure, I use it every day for various tasks! Great software!

RSS Atom

comment 1

May be fixed in 8.20210903 ("unused: Skip the refs/annex/last-index ref that git-annex recently started creating.") - I'm compiling it to confirm.

Comment by mike — Fri Sep 10 14:28:23 2021

Remove comment

comment 2

It's not fixed in 8.20210903, it just isn't (checking refs/annex/last-index...) anymore.

Comment by mike — Fri Sep 10 14:41:17 2021

Remove comment

comment 3

I have not been able to reproduce this, it works for me.

joey@darkstar:~/tmp>git init a1
Initialized empty Git repository in /home/joey/tmp/a1/.git/
joey@darkstar:~/tmp>cd a1
joey@darkstar:~/tmp/a1>git annex init
init  ok
(recording state in git...)
joey@darkstar:~/tmp/a1>date > foo
joey@darkstar:~/tmp/a1>date > bar
joey@darkstar:~/tmp/a1>git annex add
add bar
ok
add foo
ok
(recording state in git...)
joey@darkstar:~/tmp/a1>git commit -m add
[master (root-commit) fdd215f] add
 2 files changed, 2 insertions(+)
 create mode 120000 bar
 create mode 120000 foo
joey@darkstar:~/tmp/a1>rm foo
joey@darkstar:~/tmp/a1>git commit -a -m rm
[master 1f84de1] rm
 1 file changed, 1 deletion(-)
 delete mode 120000 foo
joey@darkstar:~/tmp/a1>git annex unused
unused . (checking for unused data...) (checking master...) 
  Some annexed data is no longer used by any files:
    NUMBER  KEY
    1       SHA256E-s30--11de88d41da5962620f4a0590a577710e067d72e0f24220c4a4d2e81d594388a
  (To see where this data was previously used, run: git annex whereused --historical --unused

  To remove unwanted data: git-annex dropunused NUMBER

ok
joey@darkstar:~/tmp/a1>cd ..
joey@darkstar:~/tmp>git clone a1 a2
Cloning into 'a2'...
done.
joey@darkstar:~/tmp>cd a1
joey@darkstar:~/tmp/a1>git remote add a2 ../a2
joey@darkstar:~/tmp/a1>git annex move --unused --to a2
move SHA256E-s30--11de88d41da5962620f4a0590a577710e067d72e0f24220c4a4d2e81d594388a (to a2...) 
ok

You did say it's not moving "some files" which suggests maybe it is moving other ones? Do you see it move any unused files at all?

Do other commands that use the same --unused option work? Eg, does git annex whereis --unused list them?

The only way I can reproduce this behavior is if the remote has the same uuid as the current repository. Then any move is a no-op and it avoids operating on the files at all, the same as your output shows. So it seems possible that could be your real problem.

Comment by joey — Tue Sep 21 17:34:08 2021

Remove comment

comment 4

You did say it's not moving "some files" which suggests maybe it is moving other ones? Do you see it move any unused files at all?

It's moving most of it, just not all of them!

Do other commands that use the same --unused option work? Eg, does git annex whereis --unused list them?

git annex whereis --unused lists the files. git annex copy --unused -t bims OTOH does not do anything. I also tried git annex drop --unused --in=bims in an attempt to emulate git annex move --unused -t bims, but this also does not do anything.

% git annex whereis --unused                                                                                            
whereis SHA256E-s24643--167c39a6ca9aacb7d4a111ce58cb65405e9f43e0b0f686108fbbaf201e0561aa.jpg (2 copies)                 
  ▹‧87f95204-e2d7-41bc-bf24-29c43c4a95d4 -- orange@leguin:~/Calibre Library [here]                                      
   ▹994901a8-670e-403e-9a72-aff9ce07556a -- bims:/srv/annex/Calibre Library.git [bims]                                  
[... more files ...]

The only way I can reproduce this behavior is if the remote has the same uuid
as the current repository. Then any move is a no-op and it avoids operating on
the files at all, the same as your output shows. So it seems possible that
could be your real problem.

I don't think that's the issue: If you look above, here and bims have different UUIDs. (I also notice the files are already on the bims remote.)

Comment by mike — Wed Sep 22 21:18:10 2021

Remove comment

comment 5

Well it's not surprising that git annex copy -t bims does nothing when the files are in the remote. Although it would normally display each unused key and just say "ok" when it verifies that the remote has the content.

But move and drop doing nothing while whereis does is strange. It must not be a problem with getting the list of unused keys, since whereis works.

You showed git annex move not displaying any output, which is what's really weird. Does git annex drop --unused --in bims display any output? What about git annex copy -t bims?

Comment by joey — Mon Sep 27 16:20:19 2021

Remove comment

Add a comment