Currently the hashed directories in .git-annex allow for upper and lower case directory names... on linux (or any case sensitive filesystem) the directory names such as 'Gg' and 'GG' are different and unique. However on systems like OSX (and probably windows if it is ever supported) the directory names 'Gg' is the same as 'GG'

In one of the annex'd repos that I have this has occured...

$ git add -i                                                                                          
           staged     unstaged path
  1:    unchanged        +1/-1 .git-annex/GM/GV/WORM-s183630166-m1301072171--somefile.log
  2:    unchanged        +1/-1 .git-annex/Gm/GV/WORM-s183630166-m1301072171--somefile.log

this has somewhat confused git when it tries to stage/merge files, I didn't notice this at first, but it is definately a problem for someone using case insensitive filesystems like the default OSX HFS+ formats or vfat/fat32.

I feel a bit stupid to not have considered case-insensative filesystems. They are just so far from where I have lived for 20 years that it's hard to keep them in mind.

I guess that git-annex has issues with git when staging/commiting logs is somehow a consequence (or cause?) of this, but I don't quite understand how this is causing git to fail to stage files, or stage the same file twice under different capitalizations. git-annex always will run git add on the path with the "correct" capitalization. So unless something else has added the path with the other capitalization (perhaps git add .git-annex manually?) I don't understand how you get to this state. --Joey

I think I got myself into this situation when I copied some files over from a HFS+ partition to a GPFS network share (which is pretty posix compliant) over samba. It probably is related to the git-annex has issues with git when staging/commiting logs. I thought they were unique enough to have two bug reports logged as one is a git behavioural thing and the other is git-annex specific.

If you copied .git/ over, perhaps you got a git repo without core.ignorecase set right for the filesystem it landed on?

I usually git clone or do a fresh repository and pull things in, I was also unaware of this ignorecase setting as well.

Something like this might reproduce it:

# mkdir test; cd test; git init
# git config core.ignorecase false
# mkdir Foo
# touch Foo/bar
# git add Foo/bar
# git add foo/bar
# git add fOo/bar
# git status
# touch foo/other
# git add fOo/other
# git status

And then either git commit or git clone would probably get confused if it thought 3 distinct files had been committed. --Joey

Doing the above test on a HFS+ partition yields this

## with ignorecase=false
commit bb024c6fd7482b2d10f60ae899cb7a949aca1ad8
Author: Jimmy Tang 
Date:   Sun Mar 27 18:40:24 2011 +0100

    commit

diff --git a/Foo/bar b/Foo/bar
new file mode 100644
index 0000000..e69de29
diff --git a/fOo/bar b/fOo/bar
new file mode 100644
index 0000000..e69de29
diff --git a/fOo/other b/fOo/other
new file mode 100644
index 0000000..e69de29
diff --git a/foo/bar b/foo/bar
new file mode 100644
index 0000000..e69de29

and without changing ignorecase

commit 909a089158ffb98f8e91f98905e2bfdc7234666f
Author: Jimmy Tang 
Date:   Sun Mar 27 18:46:57 2011 +0100

    commit

diff --git a/Foo/bar b/Foo/bar
new file mode 100644
index 0000000..e69de29
diff --git a/Foo/other b/Foo/other
new file mode 100644
index 0000000..e69de29

Closing this bug, as it seems I have dealt with it adequately now. done --Joey

One possible work around is to just create a loopback file system with a case sensitive filesystem. I think I might do that for anything that I really care about for now.
Comment by Jimmy Mon Mar 28 07:23:41 2011

I think I know how I got myself into this mess... I was on my mac workstation and I had just pulled in a change set from another repo on a linux workstation after I had a made a bunch of moves. here's a bit of a log of what happened...

jtang@x00:~/sources $ git pull cports-devel master
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
remote: Counting objects: 4195, done.
remote: Compressing objects: 100% (1135/1135), done.
remote: Total 2582 (delta 866), reused 2576 (delta 860)
Receiving objects: 100% (2582/2582), 229.42 KiB | 111 KiB/s, done.
Resolving deltas: 100% (866/866), completed with 9 local objects.
From cports-devel:/home/people/jtang/sources
 * branch            master     -> FETCH_HEAD
Updating 319df99..ab0a98c
error: Your local changes to the following files would be overwritten by merge:
    .git-annex/09/5X/WORM-s361516678-m1301310614--l_fcompxe_intel64_2011.2.137.tgz.log
    .git-annex/43/2g/WORM-s19509673-m1301310496--l_fcompxe_2011.2.137_redist.tgz.log
    .git-annex/4J/qF/WORM-s18891115-m1301310934--w_flm_p_1.0.011_ia64.zip.log
    .git-annex/87/w1/WORM-s12212473-m1301310909--w_flm_p_1.0.011_ia32.zip.log
    .git-annex/99/Jq/WORM-s194345957-m1301310926--l_mkl_10.3.2.137_ia32.log
    .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log
    .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log
    .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log
    .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log
    .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log
    .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log
    .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log
    .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log
    .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log
    .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log
    .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log
Please, commit your changes or stash them before you can merge.
error: Your local changes to the following files would be overwritten by merge:
    .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log
    .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log
    .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log
    .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log
    .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log
    .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log
    .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log
    .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log
    .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log
    .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log
    .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log
    .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log
    .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log
    .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log
    .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log
    .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log
    .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log
    .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log
    .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log
    .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log
    .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log
    .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log
    .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log
    .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log
    .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log
    .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log
    .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log
    .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log
    .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log
    .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log
Please, commit your changes or stash them before you can merge.
error: The following untracked working tree files would be overwritten by merge:
    .git-annex/1g/X3/WORM-s309910751-m1301311322--l_fcompxe_ia32_2011.2.137.tgz.log
    .git-annex/3w/Xf/WORM-s805764902-m1301312756--l_cproc_p_11.1.075_intel64.log
    .git-annex/9Q/Wz/WORM-s1234430253-m1301311891--l_ccompxe_2011.2.137.log
    .git-annex/FQ/4z/WORM-s318168323-m1301310848--l_cprof_p_11.1.075_ia64.log
    .git-annex/FV/0P/WORM-s710135470-m1301311835--l_ccompxe_intel64_2011.2.137.log
    .git-annex/Jx/qM/WORM-s599386592-m1301310731--l_fcompxe_2011.2.137.tgz.log
    .git-annex/KX/w1/WORM-s35976002-m1301312193--l_tbb_3.0.6.174.log
    .git-annex/Vw/jK/WORM-s15795178-m1301310913--w_flm_p_1.0.011_intel64.zip.log
    .git-annex/jK/zK/WORM-s374617670-m1301312705--l_ipp_7.0.2.137_intel64.log
    .git-annex/vK/kv/WORM-s584342291-m1301312669--l_cproc_p_11.1.075_ia64.log
    .git-annex/vw/v1/WORM-s736986678-m1301312794--l_cproc_p_11.1.075_ia32.log
    .git-annex/zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
Please move or remove them before you can merge.
Aborting
1|jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits.
#
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#   modified:   .git-annex/09/5X/WORM-s361516678-m1301310614--l_fcompxe_intel64_2011.2.137.tgz.log
#   modified:   .git-annex/43/2g/WORM-s19509673-m1301310496--l_fcompxe_2011.2.137_redist.tgz.log
#   modified:   .git-annex/4J/qF/WORM-s18891115-m1301310934--w_flm_p_1.0.011_ia64.zip.log
#   modified:   .git-annex/87/w1/WORM-s12212473-m1301310909--w_flm_p_1.0.011_ia32.zip.log
#   modified:   .git-annex/99/Jq/WORM-s194345957-m1301310926--l_mkl_10.3.2.137_ia32.log
#   modified:   .git-annex/99/kf/WORM-s9784531-m1301311680--l_ccompxe_2011.2.137_redist.log
#   modified:   .git-annex/FF/f3/WORM-s93033394-m1301311706--l_gen_ipp_7.0.2.137.log
#   modified:   .git-annex/MF/xZ/WORM-s515140733-m1301310936--l_cprof_p_11.1.075.log
#   modified:   .git-annex/XW/X8/WORM-s355559731-m1301310797--l_mkl_10.3.2.137.log
#   modified:   .git-annex/fJ/mZ/WORM-s1372886477-m1301313368--l_cproc_p_11.1.075.log
#   modified:   .git-annex/j7/Q9/WORM-s44423202-m1301310622--l_cprof_p_11.1.075_redist.log
#   modified:   .git-annex/k4/K7/WORM-s239539070-m1301310760--l_mkl_10.3.2.137_intel64.log
#   modified:   .git-annex/kz/01/WORM-s279573314-m1301310783--l_cprof_p_11.1.075_ia32.log
#   modified:   .git-annex/p6/Kq/WORM-s31199343-m1301311829--l_cproc_p_11.1.075_redist.log
#   modified:   .git-annex/pz/J5/WORM-s626995277-m1301312301--l_ccompxe_ia32_2011.2.137.log
#   modified:   .git-annex/v3/kX/WORM-s339693045-m1301310851--l_cprof_p_11.1.075_intel64.log
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   .git-annex/12/3W/WORM-s3058814-m1276699694--Botan-1.8.9.tgz.log
#   modified:   .git-annex/1G/qV/WORM-s9122-m1251558854--Array-Compare-2.01.tar.gz.log
#   modified:   .git-annex/3W/W5/WORM-s231523-m1270740744--DBD-Pg-2.17.1.tar.gz.log
#   modified:   .git-annex/3x/PX/WORM-s380310-m1293025187--HTSeq-0.4.7.tar.gz.log
#   modified:   .git-annex/45/gk/WORM-s67337-m1248732018--ExtUtils-Install-1.54.tar.gz.log
#   modified:   .git-annex/4J/7Q/WORM-s8608-m1224694862--Algorithm-Munkres-0.08.tar.gz.log
#   modified:   .git-annex/4g/XQ/WORM-s89208-m1278682033--HTML-Parser-3.66.tar.gz.log
#   modified:   .git-annex/54/jw/WORM-s300163-m1226422051--AcePerl-1.92.tar.gz.log
#   modified:   .git-annex/63/kj/WORM-s1213460-m1262942058--DBD-SQLite-1.29.tar.gz.log
#   modified:   .git-annex/6Z/42/WORM-s4074-m943766010--File-Sync-0.09.tar.gz.log
#   modified:   .git-annex/8F/M5/WORM-s6989-m1263161127--Digest-HMAC-1.02.tar.gz.log
#   modified:   .git-annex/G2/FK/WORM-s3309-m1163872981--Bundle-BioPerl-2.1.8.tar.gz.log
#   modified:   .git-annex/Gk/XF/WORM-s23572243-m1279546902--EMBOSS-6.3.1.tar.gz.log
#   modified:   .git-annex/Jk/X6/WORM-s566429-m1279309002--DBI-1.612.tar.gz.log
#   modified:   .git-annex/K6/fV/WORM-s1561451-m1240055295--Convert-Binary-C-0.74.tar.gz.log
#   modified:   .git-annex/KM/4q/WORM-s146959-m1268515086--Graph-0.94.tar.gz.log
#   modified:   .git-annex/MF/m2/WORM-s425766-m1212514609--Data-Stag-0.11.tar.gz.log
#   modified:   .git-annex/QJ/P6/WORM-s1045868-m1282215033--9base-6.tar.gz.log
#   modified:   .git-annex/Qm/WG/WORM-s39078-m1278163547--Digest-SHA1-2.13.tar.gz.log
#   modified:   .git-annex/Wq/Fj/WORM-s45680640-m1297862101--BclConverter-1.7.1.tar.log
#   modified:   .git-annex/Wq/Wm/WORM-s263536640-m1295025537--CASAVA_v1.7.0.tar.log
#   modified:   .git-annex/XW/qm/WORM-s36609-m1276050470--Bio-ASN1-EntrezGene-1.10-withoutworldwriteables.tar.gz.log
#   modified:   .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
#   modified:   .git-annex/f7/g0/WORM-s40872-m1278273227--ExtUtils-ParseXS-2.2206.tar.gz.log
#   modified:   .git-annex/j3/JF/WORM-s11753-m1232427595--Clone-0.31.tar.gz.log
#   modified:   .git-annex/kX/9g/WORM-s84690-m1229117599--GraphViz-2.04.tar.gz.log
#   modified:   .git-annex/km/z5/WORM-s44634-m1275505134--Authen-SASL-2.15.tar.gz.log
#   modified:   .git-annex/kw/J3/WORM-s132396-m1278780649--DBD-mysql-4.016.tar.gz.log
#   modified:   .git-annex/p5/1P/WORM-s53736-m1278673485--Archive-Tar-1.64.tar.gz.log
#   modified:   .git-annex/wv/zG/WORM-s30584-m1268774021--ExtUtils-CBuilder-0.2703.tar.gz.log
#   modified:   .git-annex/x5/7v/WORM-s10462526-m1254242591--BioPerl-1.6.1.tar.gz.log
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   .git-annex/1G/X3/
#   .git-annex/3W/Xf/
#   .git-annex/9q/Wz/
#   .git-annex/Fq/4z/
#   .git-annex/Jk/zK/
#   .git-annex/Kx/w1/
#   .git-annex/VK/kv/
#   .git-annex/fv/0P/
#   .git-annex/jX/qM/
#   .git-annex/vW/jK/
#   .git-annex/vW/v1/
jtang@x00:~/sources $ git commit -a -m "snap"
[master 45f254a] snap
 47 files changed, 64 insertions(+), 30 deletions(-)
jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 3 commits.
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   .git-annex/1G/X3/
#   .git-annex/3W/Xf/
#   .git-annex/9q/Wz/
#   .git-annex/Fq/4z/
#   .git-annex/Jk/zK/
#   .git-annex/Kx/w1/
#   .git-annex/VK/kv/
#   .git-annex/fv/0P/
#   .git-annex/jX/qM/
#   .git-annex/vW/jK/
#   .git-annex/vW/v1/
nothing added to commit but untracked files present (use "git add" to track)
jtang@x00:~/sources $ git pull
Comment by Jimmy Mon Mar 28 15:09:45 2011

So, there is evidence here of a circumstance caused by the ?other bug, as I suspected.

I don't think that manual git commit -a caused the problem. I suspect it was a subsequent git add that caused git to follow the wrong case paths and add the files in the wrong place. Ie, when you run "git add .git-annex", it recurses into .git-annex/Gm/, and adds files using that case, that were previously added from .git-annex/GM/.

For completeness, can you verify this repo's core.ignorecase setting?


I hate that you are stuck using loop filesystems to work around this bug. If my guess is correct, you don't need to, as long as you avoid manually running "git add .git-annex". I take this bug seriously. While I'm currently very involved in adding Amazon S3 support to git-annex (which will take days more of solid work), I do plan to make a loop filesystem of my own, probably vfat, so I can try and reproduce this on a case-insensative filesystem. If you could confirm my above hypothesis, that would speed things up for me.

It's possible I will have to tweak the hash directories. Hopefully if so, I will only tweak them for new keys; if I had to do a v3 backend just to fix this stupid thing, I'd be sad -- upgrading all my offline disks from v1 to v2 took me many days.

Comment by joey Mon Mar 28 15:25:18 2011
In my "sources" repo on x00, the current setting is this "ignorecase = true" it was the first repo that I created before I clone it elsewhere and pull my changes back, it is on a HFS+ partition which is case insensitive and it is replicated on a portable hdd with a bare repo on a exfat partition. I wonder if my portable disk has a partially borked repo :P
Comment by Jimmy Mon Mar 28 15:41:56 2011

I also failed to mention, that in the case when i have stray log files after what has happened in comment 2, I get this left over after a commit when git is confused...

jtang@x00:~/sources $ git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   .git-annex/1G/X3/WORM-s309910751-m1301311322--l_fcompxe_ia32_2011.2.137.tgz.log
#   modified:   .git-annex/3W/Xf/WORM-s805764902-m1301312756--l_cproc_p_11.1.075_intel64.log
#   modified:   .git-annex/9Q/Wz/WORM-s1234430253-m1301311891--l_ccompxe_2011.2.137.log
#   modified:   .git-annex/FQ/4z/WORM-s318168323-m1301310848--l_cprof_p_11.1.075_ia64.log
#   modified:   .git-annex/FV/0P/WORM-s710135470-m1301311835--l_ccompxe_intel64_2011.2.137.log
#   modified:   .git-annex/Jk/zK/WORM-s374617670-m1301312705--l_ipp_7.0.2.137_intel64.log
#   modified:   .git-annex/Jx/qM/WORM-s599386592-m1301310731--l_fcompxe_2011.2.137.tgz.log
#   modified:   .git-annex/KX/w1/WORM-s35976002-m1301312193--l_tbb_3.0.6.174.log
#   modified:   .git-annex/VK/kv/WORM-s584342291-m1301312669--l_cproc_p_11.1.075_ia64.log
#   modified:   .git-annex/Vw/jK/WORM-s15795178-m1301310913--w_flm_p_1.0.011_intel64.zip.log
#   modified:   .git-annex/Zq/7X/WORM-s343075585-m1301312233--l_ipp_7.0.2.137_ia32.log
#   modified:   .git-annex/vW/v1/WORM-s736986678-m1301312794--l_cproc_p_11.1.075_ia32.log
#
no changes added to commit (use "git add" and/or "git commit -a")

Up until now I have just been updating the status of the staged files by hand and commiting it on my mac x00, this probably isn't helping. I'd rather not lose the tracking information.

Comment by Jimmy Mon Mar 28 15:51:11 2011

Alright, I have created a case-insensative HFS+ filesystem here on my linux laptop.

I have not been able to trick git into staging the same file with 2 different capitalizations yet.

It might be helpful if you can send me a copy of a git repository where 'git add -i' shows the same file staged with two capitalizations. Leaving out .git/annex of course. (joey@kitenet.net; a tarball would probably work)

It seems that git add only started properly working on case insensative filesystems quite recently. The commit in question is 5e738ae820ec53c45895b029baa3a1f63e654b1b, "Support case folding for git add when core.ignorecase=true", which was first released in git 1.7.4, January 30, 2011. If you don't yet have that version, that could explain the problem entirely. In about half an hour (dialup!) I will have downloaded an older git and will see if I can reproduce the problem with it.

Comment by joey Thu Mar 31 18:02:42 2011

git 1.7.4 does not make things better. With it, if I add first "X/foo" and then "x/bar", it commits "X/bar".

That will certainly cause problems when interoperating with a repo clone on a case-sensative filesystem, since git-annex there will not see the location log that git committed to the wrong case directory.

It's possible there is some interoperability problem when pulling from linux like you did, onto HFS+, too. I am not quite sure. Ah, I did find one.. if I clone the repo with "X/foo" in it to a case-sensative filesystem, and add a "x/foo" there, and pull that commit back to HFS+, git says:

 * branch            master     -> FETCH_HEAD
Updating 8754149..e3d4640
Fast-forward
 x/foo |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 x/foo
joey@gnu:/mnt/r4>ls
X/
joey@gnu:/mnt/r4>git st
# On branch master
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory

#   modified:   X/foo

Aha -- that lets me reproduce your problem with the same file being staged twice with different capitalizations, too:

joey@gnu:/mnt/r4>echo haaai >| x/foo
joey@gnu:/mnt/r4>git st
# On branch master
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   X/bar
#   modified:   X/foo
#   modified:   x/foo
#
joey@gnu:/mnt/r4>git commit -a
fatal: Will not add file alias 'X/Bar' ('x/Bar' already exists in index)

And modified files that git refuses to commit, which entirely explains ?strong>commiting logs.

joey@gnu:/mnt/r4>git add X/foo
joey@gnu:/mnt/r4>git commit X/foo
# On branch master
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#   modified:   X/bar
#   modified:   X/foo
#
no changes added to commit (use "git add" and/or "git commit -a")

I think git is frankly, buggy. It seems I will need to work around this by stopping using mixed case hashing for location logs.

Comment by joey Thu Mar 31 19:08:01 2011

I've posted about this on the git mailing list. It's possible that these bugs, which can be shown to affect things other than just git-annex, will be fixed in git.

I will wait a while to see. But am considering making git-annex use all-lowercase hash dirs for the log files. Maybe it could first look for .git-annex/aaaa/bbbb/foo.log, but also look for, read, and merge in any info from .git-annex/Aa/Bb/foo.log. And always write to the new style filenames. This would avoid confusing git with changes to mixed-case files, and avoid another massive transition.

Comment by joey Thu Mar 31 19:28:02 2011
I'm was running git 1.7.4.1 at the time when I came across it, I have just upgraded to 1.7.4.2. I've also just moved to using a loopback fs for the stuff i care about. Do you still want a repo that exhibits the problem (excluding the .git/annex data) ??? I'm also not sure if 1.7.4.2 has corrected the problem yet as I haven't done much with my repos since. I suspect just making all the .git-annex hashed directories seems to be lower case might be better in the long run.
Comment by Jimmy Thu Mar 31 21:32:10 2011
No, I don't need a copy of your repo now.
Comment by joey Fri Apr 1 16:11:52 2011

I have pushed out a preliminary fix. The old mixed-case directories will be left where they are, and still read from by git-annex. New data will be written to new, lower-case directories. I think that once git stops seeing changes being made to mixed-case, colliding directories, the bugs you ran into won't manifest any more.

You will need to find a way to get your git repository out of the state where it complains about uncommitted files (and won't let you commit them). I have not found a reliable way to do that; git reset --hard worked in one case but not in another. May need to clone a fresh git repository.

Let me know how it works out.

Comment by joey Sat Apr 2 17:53:58 2011
Also, you can delete .git-annex/?? if you want to, then running git annex fsck --fast in each of your clones would regenerate the data using only the lower-case hash directories.
Comment by joey Sat Apr 2 17:58:24 2011
Ok, thanks for the fix. It seems the fix isn't too reliable with my repos, I get different numbers of "** No known copies of..." in the various cloned repos that I have. After all the "messing" that I have done to my repos I think git-annex has gotten very confused. I will just leave things as they are and let git-annex slowly migrate over to the new format or re-clone from a linux source and see how things go. I will report back on this issue in abit after I use it more to see.
Comment by Jimmy Sun Apr 3 07:43:37 2011

I meant to say in it wasn't reliable when I was following the instructions for "Comment 12". I did find that just doing a "git annex copy -t externalusb ." then a "git annex drop ." from the root of my cloned and "none trusted" annexed repos to be more reliable, it just means I temporarily need a load of space to get myself out of my earlier mess.

On testing this bug fix, I found a minor behavioural issue with git annex copy -f REMOTE . doesn't work as expected

Comment by Jimmy Sun Apr 3 08:24:17 2011

I also ran into problems on a case-insensitive HFS+ file system, it seems. I tried following the instructions in comment 12:

1. Remove everything in .git-annex besides uuid.log and trust.log
2. git annex fsck --fast
3. Commit

However, I still see upper and lower case directories in .git-annex. Did I misunderstand that they should all be lower case now?

Comment by gernot Sun Apr 3 15:41:00 2011

I think the correct steps should be, make a backup first :) then ...

  1. git pull # update your clone, and commit everything so you don't lose anything
  2. git annex fsck --fast # check the repo first, just in case
  3. rm -rf .git-annex/?? # remove the old metadata
  4. git annex fsck --fast # get git annex to regenerate it all
  5. push your changes out to your other repos, you will need to make sure git-annex is updated everywhere if there are remotes in your setup.

I eventually migrated all of my own annex'd repos and I no longer have the old hashed directories but the new ones in the form

.git/annex/aaa/bbb/foo.log

I did lose some tracking information but not data (as far as I can see for now), but that was quickly fixed by pushing and pulling to my bare repo which tracks most of my data.

I also found that it worked a bit more reliably for me on the copies of repos that were located on case sensitive filesystems, but I guess that was expected.

Comment by Jimmy Sun Apr 3 16:02:33 2011
@gernot step 0 is to upgrade git-annex to current git, on all systems where you use it, in case that wasn't clear.
Comment by joey Sun Apr 3 16:53:51 2011

Joey, sorry, I got it wrong. I thought upgrading git didn't help and you adjusted things in git-annex instead.

Anyway, can I get around upgrading on all hosts by reformatting the drive to case-sensitive HFS+? Or will I have to upgrade git (currently version 1.7.2.5) eventually anyway?

Comment by gernot Sun Apr 3 19:46:16 2011
Git does not need to be upgraded. Git-annex needs to be upgraded to git rev 616e6f8a840ef4d99632d12a2e7ea15c3cfb1805 or newer, on all machines.
Comment by joey Sun Apr 3 19:53:44 2011

Hi,

(I'm new to git and git annex, so please forgive any mistakes I make...)

My repo is messed up right now. The fact that I copied the repo with rsync -a back and forth from a case insensitive filesystem to a case sensitive one, probably didn't help.

I believe the annexed files in .git/annex/objects/ are still using a mixed case directory hashing scheme. That's the problem I'm having. The symlinks point to the wrong case and are now broken. I don't think the latest versions of git-annex changed that (it only changed the hashing under .git-annex, right?).

Even if I clean up my repo, I think I'm still going to have a problem because I have one repo on an OS X case insensitive filesystem and my other repos on case sensitive Linux filesystems. Potentially the directory name under .git/annex/objects will have a different case. Then the symlink might have a different case than my Linux FS. Does git-annex track changes in git by the contents of the symlink? In which case the case difference would show up as a change even though there is no change?

Is it possible to change the directory hashing scheme under .git/annex/objects to use lowercase names?

Comment by ssqq Thu Jun 2 20:31:55 2011

@seqq git-annex always uses the same case when creating and accessing the files pointed to by the symlinks. So it will not matter if it's used on a case-insensative, or case-insensative but preserving system like OSX.

You need to fix up the cases of the files in .git/annex/objects to what it expects. I'm not sure what would be the best way to do that. The method described in ?recover data from lost+found might work well.

Comment by joey Fri Jun 10 16:46:03 2011
I moved an external HDD formatted with NTFS from Mac (case-insensitive) to Linux (case sensitive), and half of the links are broken now... What can I do to fix this?
Comment by Jaen Tue Dec 25 17:58:53 2012
(to be clear, Mac put eg. hashes "Gg" and "gg" into the same directory, while Linux expects them to be in separate dirs)
Comment by Jaen Tue Dec 25 20:21:15 2012