files got replaced by broken symblinks, how to get them back?

Basically, this is how it looks now (this is part of my music collection, but there are other directories with the same symptoms):

simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » ls -l
insgesamt 4
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 01 So Help Me God.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 02 Sore Loser.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 03 How The Wind Must Feel.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 04 Sold Out.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 05 In The Dark.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 06 Breaking Bad Habits.mp3
-rwxrwxrwx 1 simon simon 196  7. Dez 13:37 AmyKuneyEP.jpg
simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » cat 01\ So\ Help\ Me\ God.mp3 
../../.git/annex/objects/M5/Pq/SHA256E-s213--96db10197b716998d35576c39fc2468bd197b5a4ba2bec14a78454b278d96498.mp3/SHA256E-s213--96db10197b716998d35576c39fc2468bd197b5a4ba2bec14a78454b278d96498.mp3%
simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » ls ../../.git/annex/objects/M5
ls: Zugriff auf ../../.git/annex/objects/M5 nicht möglich: Datei oder Verzeichnis nicht gefunden
simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » find ../../.git/annex | grep 96db10197b716998d3557                               2 ↵
../../.git/annex/objects/a5b/e78/SHA256E-s213--96db10197b716998d35576c39fc2468bd197b5a4ba2bec14a78454b278d96498.mp3
../../.git/annex/objects/a5b/e78/SHA256E-s213--96db10197b716998d35576c39fc2468bd197b5a4ba2bec14a78454b278d96498.mp3/SHA256E-s213--96db10197b716998d35576c39fc2468bd197b5a4ba2bec14a78454b278d96498.mp3.map
simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » git annex info  
repository mode: direct

So, I only have a bunch of text files containing file paths. The actual files are in .git/annex/objects somewhere, but not exactly in the path specified in those text files. I'm not sure how it ended up that way, I'm in direct mode after all. For starters, I would like to know how I can repair this mess.

RSS Atom

comment 1

git-annex uses broken symlinks for files in the git repository whose content is not currently present.

On filesystems not supporting symlinks, git represents them by putting the symlink target in a regular file. This seems to be what you have here.

Your find command shows that the content is not located in .git/annex/objects. It only found a .map file, not the content file.

So, git annex get seems like the most likely thing that will get the content of these files from wherever it's stored.

Comment by joey — Fri Jan 15 19:01:22 2016

Remove comment

comment 2

Well here is where it gets weird. I used "git annex get" to retrieve the file and it did something.

simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » git annex get 01\ So\ Help\ Me\ God.mp3
get 01 So Help Me God.mp3 (from desktop-linux...) 
01 So Help Me God.mp3
             213 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)
(checksum...) ok
(recording state in git...)

simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » ls -l
insgesamt 4
-rwxrwxrwx 1 simon simon 213 15. Jän 20:30 01 So Help Me God.mp3
....

simon@simon-desktop /run/media/simon/TOSHIBA EXT/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » cat 01\ So\ Help\ Me\ God.mp3 
..\..\..\..\..\.git\annex\objects\K8\xx\SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3\SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3%

Yet the file still is only a fake symlink. On the other computer (desktop-linux) where it got the file from it's also only a symlink, but the actual file is somewhere in annex/objects

simon@simon-desktop /d/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » git annex whereis 01\ So\ Help\ Me\ God.mp3 
whereis 01 So Help Me God.mp3 (1 copy) 
    5e0f083c-d81f-4a4b-b1a5-35b45e057dea -- desktop pc [here]
ok
simon@simon-desktop /d/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » cat 01\ So\ Help\ Me\ God.mp3 
..\..\..\..\..\.git\annex\objects\K8\xx\SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3\SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3%
simon@simon-desktop /d/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » find /d/Stuff/.git/annex/objects | grep a6247eb640b2d8c0ca48480
/d/Stuff/.git/annex/objects/48d/eb4/SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3
/d/Stuff/.git/annex/objects/48d/eb4/SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3/SHA256E-s3460908--ad74d891fd20ef771f933d3fea6247eb640b2d8c0ca4848065735992a07a4eb7.mp3.map

Comment by simon.parzer — Fri Jan 15 19:37:57 2016

Remove comment

comment 3

Try running git-annex fsck on the file; this may fix it up to make the content actually available. It might also output something useful to debug what went wrong.

Comment by joey — Wed Jan 20 19:06:00 2016

Remove comment

comment 4

I already tried git annex fsck, but it does not report any errors and the files remain unchanged. Here is the output.

simon@simon-desktop /d/Stuff/Music/Amy Kuney - Breaking Bad Habits EP (2006) ±annex/direct/master » git annex fsck 01\ So\ Help\ Me\ God.mp3
fsck 01 So Help Me God.mp3 (checksum...) ok
(recording state in git...)

Comment by simon.parzer — Thu Jan 21 12:54:57 2016

Remove comment

comment 5

Does git annex info show this repository as being in direct mode, or indirect mode? (It's the first line of its output.)

I'm thinking you might somehow have a repository that is not really in direct mode despite the current branch being annex/direct/master.

Comment by joey — Fri Jan 29 17:49:47 2016

Remove comment

comment 6

To sort of follow up on this after 3 months, git annex info displays repository mode: direct

I have given up on git annex in the meantime, it is a nightmare to sync between different operating systems/file systems. What might be "wrong" with my setup is that I have at least two machines that are dual-boot. So I'm using the same folder both in Windows and Linux. I also have an external HD with an annex repository that I plug in to different machines, and it apparently causes problems. Now I have to write a python script to recover my lost files from the bottomless abyss that is the .git/annex folder.

Comment by simon.parzer — Thu May 5 11:26:03 2016

Remove comment

Python script to recover git annex' lost files

Maybe someone else needs this at some point

Use it at your own risk of course.

#!python3

import os
import re
from shutil import copyfile

targets = {}

for path, dirs, files in os.walk('.'):
    for f in files:
        fp = os.path.join(path, f)
        if fp.endswith('.py'): continue
        fp_size = os.stat(fp).st_size
        if fp_size < 72 or fp_size > 256: continue
        with open(fp, 'r', encoding='latin1') as stream:
            l = stream.readline()
            hashmatch = re.match('.*SHA256E\-\w+\-\-(\w{16})', l)
            if hashmatch:
                targets[hashmatch.group(1)] = os.path.abspath(fp)

#print(targets)

target = os.path.abspath('.')
while not os.path.exists('.git/annex/objects'):
    os.chdir('..')

for path, dirs, files in os.walk('.git/annex/objects'):
    for f in files:
        fp = os.path.join(path, f)
        fp_size = os.stat(fp).st_size
        if fp_size > 512:
            hashmatch = re.match('.*SHA256E\-\w+\-\-(\w{16})', fp)
            if hashmatch:
                hash = hashmatch.group(1)
                if hash in targets:
                    print(fp, '->', targets[hash])
                    copyfile(fp, targets[hash])

Comment by simon.parzer — Thu May 5 12:01:46 2016

Remove comment

Add a comment