local pairing git-annex-shell issue when using standalone tarballgit-annexhttp://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/git-annexikiwiki2013-11-27T22:47:37Zcomment 1http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_1_69eafc4201e3014ef1b5d74fe319e462/joeyh.name2013-11-27T22:47:37Z2013-07-10T16:31:40Z
<p>Everything you've described is expected behavior, when the content of the file has not yet been transferred to the repository.</p>
<p>There should be something in daemon.log about the transfer of the file. If there's an error message for the transfer, it will certainly show up there. You can enable debugging to get more information in daemon.log about file transfers.</p>
debug loghttp://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_2_b7a64db9abe006af8c30169ad849efe9/Drew Perttula2013-11-27T22:47:37Z2013-07-16T04:02:40Z
<p>receiver side logs again, with --debug mode on. I don't see anything weird.</p>
<pre><code>[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..a7cb1b3f2a3f2a7d24827e10f9f3ac5848fd11d9","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..4a2771e15cb5c7a0a8d70443e76b65c12115a8cd","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..ddf6aa74d771521b11657edfae558858b60e8368","--oneline","-n1"]
(merging synced/git-annex into git-annex...)
[2013-07-15 20:53:02 PDT] feed: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","update-index","-z","--index-info"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","4a2771e15cb5c7a0a8d70443e76b65c12115a8cd"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","4a2771e15cb5c7a0a8d70443e76b65c12115a8cd..refs/heads/git-annex","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] call: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","update-ref","refs/heads/git-annex","4a2771e15cb5c7a0a8d70443e76b65c12115a8cd"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","symbolic-ref","HEAD"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","refs/heads/master"]
[2013-07-15 20:53:02 PDT] Merger: merging refs/heads/synced/master into refs/heads/master
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","symbolic-ref","HEAD"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","refs/heads/master"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/master"]
[2013-07-15 20:53:02 PDT] call: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex/.git/annex/merge/","merge","--no-edit","refs/heads/synced/master"]
Updating 6f8cbe0..18cea18
Fast-forward
test-2013-07-15e | 1 +
1 file changed, 1 insertion(+)
create mode 120000 test-2013-07-15e
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/master"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","diff-tree","-z","--raw","--no-renames","-l0","-r","6f8cbe0a34d70e70a6365385bbd4338d97047d4b","18cea18855143095a872a463eb0a7cf5dd81de4c"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..4a2771e15cb5c7a0a8d70443e76b65c12115a8cd","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..ddf6aa74d771521b11657edfae558858b60e8368","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..a7cb1b3f2a3f2a7d24827e10f9f3ac5848fd11d9","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] feed: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","update-index","-z","--index-info"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex"]
[2013-07-15 20:53:02 PDT] Watcher: add symlink test-2013-07-15e
[2013-07-15 20:53:02 PDT] chat: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","hash-object","-t","blob","-w","--stdin","--no-filters"]
[2013-07-15 20:53:02 PDT] Committer: committing 1 changes
[2013-07-15 20:53:02 PDT] Committer: Committing changes to git
(Recording state in git...)
[2013-07-15 20:53:02 PDT] feed: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","update-index","-z","--index-info"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","commit","--allow-empty-message","--no-edit","-m","","--quiet","--no-verify"]
[2013-07-15 20:53:02 PDT] Pusher: Syncing with host1.local_annex
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","symbolic-ref","HEAD"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","refs/heads/master"]
[2013-07-15 20:53:02 PDT] Pusher: pushing to [Remote { name ="host1.local_annex" }]
[2013-07-15 20:53:02 PDT] call: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","branch","-f","synced/master"]
[2013-07-15 20:53:02 PDT] call: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","push","host1.local_annex","git-annex:synced/git-annex","master:synced/master"]
To ssh://drewp@git-annex-host1.local-drewp_annex/~/annex/
a7cb1b3..4a2771e git-annex -> synced/git-annex
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","push","host1.local_annex","git-annex:synced/git-annex","master"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/git-annex"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..4a2771e15cb5c7a0a8d70443e76b65c12115a8cd","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","log","refs/heads/git-annex..ddf6aa74d771521b11657edfae558858b60e8368","--oneline","-n1"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","symbolic-ref","HEAD"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","refs/heads/master"]
[2013-07-15 20:53:02 PDT] Merger: merging refs/remotes/host1.local_annex/synced/master into refs/heads/master
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","symbolic-ref","HEAD"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","refs/heads/master"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/master"]
[2013-07-15 20:53:02 PDT] call: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex/.git/annex/merge/","merge","--no-edit","refs/remotes/host1.local_annex/synced/master"]
Already up-to-date.
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","show-ref","--hash","refs/heads/master"]
[2013-07-15 20:53:02 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","diff-tree","-z","--raw","--no-renames","-l0","-r","18cea18855143095a872a463eb0a7cf5dd81de4c","18cea18855143095a872a463eb0a7cf5dd81de4c"]
[2013-07-15 20:53:28 PDT] read: git ["--git-dir=/home/drewp/annex/.git","--work-tree=/home/drewp/annex","ls-tree","-z","--","refs/heads/git-annex","uuid.log","remote.log","trust.log","group.log","preferred-content.log"]
</code></pre>
<p>I also noticed on host1 where the file was made, there is a git node in gitk that says 'Local uncommitted changes, not checked in to index' and this node appears to have all my file contents as diffs. So the problem is maybe not the transfer, but the failure of the originating side to commit the file contents properly?</p>
comment 3http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_3_197ac6070f256131c6e18a07aa3834fa/joeyh.name2013-11-27T22:47:37Z2013-07-16T18:03:40Z
<p>It's normal for git diff and gitk to look like that in a repository that's using direct mode.</p>
<p>In your log, I see you added a file "test-2013-07-15e". And I don't see any indication that it tried to transfer it.</p>
<p>There are a few reasons that could happen. If you've "paused" syncing with a remote in the webapp, it could do that.</p>
<p>I think more likely would be if you've put the remote repository in a repository group that doesn't want the file. If it's configured to be in the Manual group or the Source group (or the Transfer, Public, or Small Archive group for that matter), the assistant won't try to send files to it. You can check this by editing the repository in the webapp. It should probably be set to be in the Client group.</p>
comment 4http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_4_fe07832333b536c71b7dcb46a4a44bd0/Drew Perttula2013-11-27T22:47:37Z2013-07-25T05:02:16Z
<p>I saw the 'syncing enabled'/'syncing disabled' switch in the webapp, which I think is what you're calling pause. That was always enabled.</p>
<p>I never adjusted the groups, but I see in the webapp that my remote is busted in other ways. It has no title, and settings->edit makes an error page. The edit link is like this:</p>
<p>http://127.0.0.1:38187/config/repository/edit/NoUUID?auth=4294a...333c</p>
<p>and the error is</p>
<pre><code>Internal Server Error
Unknown UUID
</code></pre>
<p>I'll try re-configuring that remote. I wish the webapp could tell me more about what is broken with the remote repo and where I might go to repair it.</p>
comment 5http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_5_540bca4e6fdfc10eeab875ecc0f2b3f3/Drew Perttula2013-11-27T22:47:37Z2013-07-25T05:14:12Z
<p>BTW "go enter it into the computer you want to pair with" is a very frustrating instruction to read. Every time I try to do a pairing and I encounter that, it's telling me "now it's time for you to guess what to do, so this is probably where you'll screw things up." I have no idea where to enter that secret phrase. I would also like to know if 'asdfasdf' is a good secret phrase or not, presuming the two computers are both on my internal network. Does the secret matter later or is it just used for a minute?</p>
<p>So far, I keep getting blank repo names with the NoUUID edit url, or I fail to accomplish a pairing at all.</p>
comment 6http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_6_3f236b35e9820cd88bb77fcd57d6975e/Drew Perttula2013-11-27T22:47:37Z2013-07-25T05:36:07Z
<p>Here's how little it takes for things to go wrong:</p>
<hr />
<p>First host:</p>
<p>bang(pts/17):~/annex% git init<br />
Initialized empty Git repository in /home/drewp/annex/.git/</p>
<p>bang(pts/17):~/annex% git-annex init<br />
init ok<br />
(Recording state in git...)</p>
<p>bang(pts/17):~/annex% git-annex webapp --listen 10.1.0.1:9999 <br />
http://10.1.0.1:9999/?auth=6f9a8b...</p>
<hr />
<p>Second host:</p>
<p>dash(pts/34):~/annex% git init<br />
Initialized empty Git repository in /home/drewp/annex/.git/</p>
<p>dash(pts/34):~/annex% git-annex init<br />
init ok<br />
(Recording state in git...)</p>
<p>dash(pts/34):~/annex% git-annex webapp --listen 10.1.0.229:9999<br />
http://10.1.0.229:9999/?auth=f28bd56b456....</p>
<p>Browse to first URI. Add another repo. Local computer. Enter 'qwerty'. <br />
Browse to second URI. Respond to pairing request. Enter 'qwerty' again.<br />
View both dashboards: http://bigasterisk.com/post/git-annex-bad-pair.png</p>
comment 7http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_7_3cc5dae0351201522711a7caeecd60d5/joeyh.name2013-11-27T22:47:37Z2013-07-25T20:02:18Z
<p>What settings does <code>.git/config</code> have for the git remote created by the local pairing process?</p>
<p>Since I can't reproduce any problems with local pairing -- it works great on this network -- you will need to enable debugging, reproduce the bug, and send a debug log for me to debug this any further.</p>
~/annex/.git/config files (after running the steps above)http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_8_3c3883cb66d02a15d5de84d22aa113da/Drew Perttula2013-11-27T22:47:37Z2013-07-26T13:20:29Z
<pre><code>dash(pts/32):~/annex% cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[annex]
uuid = ddbb5112-3cff-46d3-b42a-f3a6826827a5
version = 3
[remote "bang.local_annex"]
url = ssh://drewp@git-annex-bang.local-drewp_annex/~/annex/
fetch = +refs/heads/*:refs/remotes/bang.local_annex/*
annex-ignore = true
annex-cost = 175.0
bang(pts/16):~/annex% cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[annex]
uuid = c8a6c420-0567-4f33-8abb-e44b2012ad55
version = 3
[remote "dash.local_annex"]
url = ssh://drewp@git-annex-dash.local-drewp_annex/~/annex/
fetch = +refs/heads/*:refs/remotes/dash.local_annex/*
annex-ignore = true
annex-cost = 175.0
</code></pre>
comment 9http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_9_c8cece9559bd2dc6154cd28772369e48/joeyh.name2013-11-27T22:47:37Z2013-07-26T16:55:08Z
<p>Like I sort of expected, this is missing a remote.$foo.annex-uuid setting. But I don't know why yet. I need a debug log to figure this out.</p>
<p>Can you enable debugging (run git-annex webapp --debug), reproduce the problem again, and send me the log?</p>
comment 10http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_10_8a1d16b2aaba224e94be3d9dcc036d91/Drew Perttula2013-11-27T22:47:37Z2013-07-27T07:42:26Z
<p>Sorry, I meant to get those logs earlier but I got distracted.</p>
<p>One confounding thing in the previous runs is that I may have had accidental instances of git-annex assistant running on at least one of the machines. I killed them all for this next attempt, and named the dir annex2 just in case.</p>
<p>daemon.log files are now at http://bigasterisk.com/post/git-annex/logs1/</p>
comment 11http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_11_434ed328a22a6657dba3b2929a56e499/joeyh.name2013-11-27T22:47:37Z2013-07-28T00:06:53Z
<p>From the log on bang:</p>
<pre>
[2013-07-27 00:33:15 PDT] read: ssh ["-S","/home/drewp/annex2/.git/annex/ssh/drewp@git-annex-dash.local-drewp_annex2","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","drewp@git-annex-dash.local-drewp_annex2","git-annex-shell 'configlist' '/~/annex2/'"]
zsh:1: command not found: git-annex-shell
Remote dash.local_annex2 does not have git-annex installed; setting remote.dash.local_annex2.annex-ignore
</pre>
<p>You said you're running from the standalone tarball. I think that's the problem, probably if you get git-annex-shell into your path it'll just work.</p>
<p>Can you paste the line git-annex added to "dash"'s .ssh/authorized_keys?</p>
comment 12http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_12_1837b70ace42882db3ab82e001680934/Drew Perttula2013-11-27T22:47:37Z2013-07-28T05:36:39Z
<p>Thanks for finding that. I added git-annex-shell to my path and ran again. The repo lists look good now.</p>
<p>For the record, here's what I am getting in authorized_keys on dash:</p>
<pre><code>command="GIT_ANNEX_SHELL_DIRECTORY='/home/drewp/annex2' ~/.ssh/git-annex-shell",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa AAAAB3NzaC1y.......0c2ilnh drewp@bang
</code></pre>
<p>When I went to the settings on each box, all 4 repository group select boxes were blank. I set them all to 'client'.</p>
<p>How do we make this long ticket worth your time by improving the code so people don't hit the same trap in the future? I think I would have looked in the right direction if, instead of saying nothing and not having a uuid to make the links work, the web ui said something like this:</p>
<pre><code>'command not found: git-annex-shell' on remote host. Sync disabled with this host.
</code></pre>
<hr />
<p>Next, I ran "date > annex2/file1" on bang. The web consoles said they synced, but no file appeared on dash. There was a ~/annex2/.git/annex/objects/../../SHA256.../SHA256... file on dash with the right contents, but no symlink showed up in dash:~/annex2.</p>
<p>I ran "date > annex2/file2" on dash, and then I got both file1 and file2 on both boxes.</p>
<p>In the new logs at http://bigasterisk.com/post/git-annex/logs2/, the surprise is at [2013-07-27 22:10:49 PDT] where I thought dash would make a symlink called 'file1' but instead it reports no attempt or error. Later at [2013-07-27 22:12:08 PDT], while dealing with the upload of file2, dash finally makes the symlink for file1.</p>
comment 13http://git-annex.branchable.com/bugs/remote_files_appear_but_are_unreadable_because_their_symlink_targets_don__39__t_exist/comment_13_ca9c87a10f29e41572540edeb99652f2/joeyh.name2013-11-27T22:47:37Z2013-07-30T18:51:38Z
<p>I'm more interested in fixing core problems, like why it was not able to find git-annex-shell, than in fixing bugs in the web UI that follow on from those problems.</p>
<p>When you're using the standalone tarball, it's supposed to set up <code>~/.ssh/git-annex-shell</code> to point to wherever you've unpacked the tarball. The assistant does this the first time you run it, and any time it sees the file is out of date.
I've tested this, and it appears to be working ok. I don't understand what happened in your case to prevent this from happening.</p>