Recent comments posted to this site:

Sorry, just saw your comment...

Yes, I'm using the assistant, as I pointed out in my original posting. Just right now, I'm having the same situation:

florian@horus ~/.synced_configuration (git)-[annex/direct/master] % git annex whereis .emacs
whereis .emacs (2 copies) 
        c7fc42ca-49f7-4688-b81d-c9cfd4a42564 -- Asaru
        d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
ok
florian@horus ~/.synced_configuration (git)-[annex/direct/master] % git annex log .emacs    
+ Mon, 29 May 2017 20:04:11 CEST .emacs | c7fc42ca-49f7-4688-b81d-c9cfd4a42564 -- Asaru
- Mon, 29 May 2017 20:04:10 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
+ Mon, 29 May 2017 20:02:37 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
+ Mon, 29 May 2017 20:02:37 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
- Mon, 29 May 2017 20:02:36 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus
- Mon, 29 May 2017 20:02:35 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
+ Mon, 29 May 2017 20:02:35 CEST .emacs | d5b0586f-32dd-4aae-a7a2-b23d9edd0c32 -- Marduk [marduk]
+ Mon, 29 May 2017 20:02:33 CEST .emacs | 69902190-d4c1-4abb-a000-f0712769f1cd -- Horus

Thanks! Florian

Comment by Horus Mon May 29 18:10:55 2017

Hi,

I'm not sure what exactly has changed, but it's working again!

Until now, I was using git-annex version: 6.20170321-gf3dee9d65 from the standalone-amd64.tar.gz. In the meantime I convinced the administrator to install git-annex system wide. She installed git-annex version: 5.20140221 from the packet-system.

Now, the repository works again with both, git-annex 5 and 6.

However, I cannot exactly tell if anything else has changed in the meantime, since I am not the administrator of the system.

Thanks for your help, anyway.

Comment by mario Mon May 29 15:19:18 2017

Not sure how I did it, but I have 2 repos that give me the same error. (Perhaps it happened when killing the assistant in the middle of a sync? No idea…)

However I "solved" the issue for one repo:

  1. cloned the gcrypt remote to a temporary directory
  2. removed all the (encrypted) files with git rm, committed that, and pushed it (the repo is hosted on Gitlab and I can't just remove the branch or do something like that)
  3. removed the gcrypt-id| from my.git/config`
  4. ran git annex sync again

And voilà! It works :)

I don't have any content in this remote, only metadata, so it's easy to do. But I still don't know what caused this.

Comment by Schnouki Mon May 29 09:56:59 2017

For the benefit of future readers, I did manage to get the stand alone git-annex running on my Synology DS216+. With a little bit of sorting out details, including adding some symlinks in /usr/bin, it runs like git-annex on any other Linux-based system. At this time that is definitely much easier than trying to use a SMB share if you are able to enable ssh access -- but note that ssh access to the Synology NAS requires an administrator account, so may not be available to everyone. (Plus of course this work around is useful mainly to relatively open Linux-based NAS solutions.)

Ewen

Comment by ewen Sun May 28 01:24:43 2017

I used a similar approach, with a bit of refinement to not require a custom ssh key/user, to get git-annex running on a Synology DS216+ NAS, which is based around a Celeron chip (and thus needs the x86-64 build). Once all the PATH related issues are taken care of (which some symlinks into /usr/bin) it appears to work like any other Linux/Unix-based git-annex install. (Definitely much more successful than trying to use git-annex via a SMB share at this point.)

Ewen

Comment by ewen Sun May 28 01:19:16 2017

Thank you Joey! It seems to work very nice now! Not a single one lost out of 1550!

$> datalad get -J5 38*
[INFO   ] Getting 100 items of dataset <Dataset path=/tmp/travis-buildlogs> ... 
[INFO   ] Actually getting 1550 files 
Tried to get 1550 files. Got 1550. 
Comment by yarikoptic Fri May 26 02:03:57 2017

The .git/config concurrent access happens because the remote list is only generated on demand, and nothing demands it when running with -J until all the threads are spun up and each thread has its own state then, so each generates the remote list.

There don't look to be any other git-config settings that would cause problems for concurrency other than ones run while generating the remote list.

So, generating the remote list before starting concurrent threads would avoid that problem, and also leads to a slightly faster startup since the remote git config only has to be read once, etc.

The only risk in doing that would be if generating a Remote opens some kind of handle, which can't be used concurrently, or is less efficient if only opened once and then used by multiple threads.

I've audited all the Remote.*.gen methods, and they all seem ok. For example, Remote.External.gen sets up a worker pool of external special remote processes, and new ones are allocated as needed. And Remote.P2P.chainGen sets up a connection pool.

Ok, gone ahead with this fix.

Comment by joey Thu May 25 21:52:37 2017

That looks like concurrent git config setting remote.origin.annex-uuid are failing.

I have not reproduced the .git/config error, but with a local clone of a repository, I have been able to reproduce some intermittent "transfer already in progress, or unable to take transfer lock" failures with git annex get -J5, happening after remote.origin.annex-uuid has been cached.

So, two distinct bugs I think..


Debugging, the lock it fails to take always seems to be the lock on the remote side, which points to the local clone being involved somehow.

Debugging further, Utility.LockPool.STM.tryTakeLock is what's failing. That's supposed to only fail when another thread holds a conflicting lock, but as it's implemented with orElse, if the main STM transaction retries due to other STM activity on the same TVar, it will give up when it shouldn't.

That's probably why this is happening under heavier concurrency loads; it makes that failure case much more likely. And with a local clone, twice as much locking is done.

I've fixed this part of it!


The concurrent git config part remains. Since git-annex can potentially have multiple threads doing different git config for their own reasons concurrently, it seems it will need to add its own locking around that.

Comment by joey Thu May 25 19:08:59 2017

didn't need to go far ;)

$> git annex get -J5 
(merging origin/git-annex into git-annex...)
(recording state in git...)
get 10/10.1-None.txt.gz get 1000/1000.2-0.txt.gz get 1000/1000.1-0.txt.gz get 1000/1000.3-0.txt.gz get 1000/1000.4-1.txt.gz error: could not lock config file .git/config: File exists
(from origin...) (from origin...) (from origin...) 

(transfer already in progress, or unable to take transfer lock) 
  Unable to access these remotes: origin
(from origin...) 

  Try making some of these repositories available:
    3db23446-8c40-441e-97ec-55ffc86b4fc0 -- yoh@smaug:~/proj/datalad/datalad/.git/travis-ci/origin-annex [origin]
failed
SHA256E-s55194--48621840b3a869ac27dfdb2a201c202ee1382aa0114337563da2c38262a0c9e8.txt.gz
         55,194 100%   21.39MB/s    0:00:00 (xfr#1, to-chk=0/1)
SHA256E-s56291--48af9865017aca9c8931c3b91c4f0ff9954273d21007ff25de1a2c73ca0df95cSHA256E-s7813--7861750b475c620012f1a76635749d89b86b7777130352b6adb844753d3822c2.txt.gz
.txt.gz
         32,768  58%    0.00kB/s    0:00:00  (checksum...)    0:00:00 (xfr#1, to-chk=0/1)
         56,291 100%   22.43MB/s    0:00:00 (xfr#1, to-chk=0/1)
(checksum...) (checksum...) ok
ok
ok
git-annex: git [Param "config",Param "remote.origin.annex-uuid",Param "3db23446-8c40-441e-97ec-55ffc86b4fc0"] failed
CallStack (from HasCallStack):
  error, called at ./Git/Command.hs:39:17 in main:Git.Command

$> git annex get 2>&1 | head
get 1000/1000.2-0.txt.gz (from origin...) 
SHA256E-s56206--1fa38270b23b4c700c4a998712549c65782613d4567b8650b1ebaa3fcfc884f4.txt.gz
         56,206 100%   22.35MB/s    0:00:00 (xfr#1, to-chk=0/1)
(checksum...) ok
get 1000/1000.3-0.txt.gz (from origin...) 
SHA256E-s55880--dea4463f7e42c51463e236d29c4c40486b56754cbd8340c260e481bdd506115b.txt.gz
         55,880 100%   22.04MB/s    0:00:00 (xfr#1, to-chk=0/1)
(checksum...) ok
get 1000/1000.5-1.txt.gz (from origin...) 
SHA256E-s55028--f6f8d5f25bed2bb51d9ba17275a956a97276cdcc74c6538c7e590919426e97df.txt.gz

FWIW, the repository in question is this one: http://datasets.datalad.org/devel/travis-buildlogs/.git/

Comment by yarikoptic Thu May 25 18:39:03 2017

I checked the file system, it's nfs.

git config --get annex.pidlock gives nothing, so I guess it's not enabled. Do I have to enable it, if the repo is on an nfs mount?

Comment by mario Thu May 25 18:25:59 2017