Today I built the UI in the webapp to set up a ssh or rsync remote.
This is the most generic type of remote, and so it's surely got the most complex description. I've tried to word it as clearly as I can; suggestions most appreciated. Perhaps I should put in a diagram?
The idea is that this will probe the server, using ssh. If git-annex-shell
is available there, it'll go on to set up a full git remote. If not, it'll
fall back to setting up a rsync special remote. It'll even fall all the way
back to using rsync://
protocol if it can't connect by ssh. So the user
can just point it at a server and let it take care of the details,
generally.
The trickiest part of this will be authentication, of course. I'm relying
on ssh using ssh-askpass
to prompt for any passwords, etc, when there's
no controlling terminal. But beyond passwords, this has to deal with ssh
keys.
I'm planning to make it check if you have a ssh key configured already. If you do, it doesn't touch your ssh configuration. I don't want to get in the way of people who have a manual configuration or are using MonkeySphere.
But for a user who has never set up a ssh key, it will prompt asking if
they'd like a key to be set up. If so, it'll generate a key and configure
ssh to only use it with the server.. and as part of its ssh probe, that key
will be added to authorized_keys
.
(Obviously, advanced users can skip this entirely; git remote add
ssh://...
still works..)
Also today, fixed more UI glitches in the transfer display. I think I have them all fixed now, except for the one that needs lots of javascript to be written to fix it.
Amusingly, while I was working on UI glitches, it turned out that all the fixes involved 100% pure code that has nothing to do with UI. The UI was actually just exposing bugs.
For example, closing a running transfer had a bug that weirdly reordered the queue. This turned out to be due to the transfer queue actually maintaining two versions of the queue, one in a TChan and one a list. Some unknown bugs caused these to get out of sync. That was fixed very handily by deleting the TChan, so there's only one copy of the data.
I had only been using that TChan because I wanted a way to block while the queue was empty. But now that I'm more comfortable with STM, I know how to do that easily using a list:
getQueuedTransfer q = atomically $ do
sz <- readTVar (queuesize q)
if sz < 1
then retry -- blocks until size changes
else ...
Ah, the times before STM were dark times indeed. I'm writing more and more STM code lately, building up more and more complicated and useful transactions. If you use threads and don't know about STM, it's a great thing to learn, to get out of the dark ages of dealing with priority inversions, deadlocks, and races.
I've been playing around with the Assistant in the last few days.
And besides for gpg encryption it works great.
But whenever i set up a special remote(done it manually since you hadn't made this yet) it will break, bad.
Some files will transfer fine, but at some point it will still(the stdout from git-annex webapp shows gpg being stupid).
I get a huge amount of [git] in my ps auxw. And two git-annex transferkey --to host --file
It seems to consistently fall on a 9mb file i have. I have found no way to recover from this(killing and restarting git-annex will say 'transfer already in progress'. Deleting the file was to no avail, it still tries to transfer).
I have done a lot of testing and i've only seen this on special remotes when gpg encryption is enabled(i haven't tried encryption=shared)
Besides for this issue, the Assistant is looking great.
Updated to the latest trunk, and did some more testing.
I'll retract the "gpg being stupid" comment. it just gives the normal "(gpg)".
There really isn't anything interesting in the stdout(but a lot of defunct git's in ps auxw), but i still can't get any files larger than ~710kbyte to transfer. They just stall.
http://pastebin.com/ZA8dzxZD <- stdout/stderr http://pastebin.com/f1J1T79E <- ps auxw | grep git
So, i've let the assistant run since i posted my last message, while i slept.
But 8 hours later, it still hasn't made any progress.
Again, only see this with gpg encryption enabled, and on files >700kbyte.
Can you try running the transfer command by hand? The command line is:
/home/alansmithee/bin/git-annex transferkey SHA256-s806100--ad2c33a88a665c56d97393fad99b37529afc5947694eab7d9c27f4d178d182f4 --to host1free --file 787kbyte.txt
Paste me the full output until it seems to hang (if it hangs this way), and if it does hang you may also need to strace it and send me that..
No joy.
strace: http://pastebin.com/UYSrZZAg
strace -f
output for the same command you straced before?I've tried with rsync and directory(real, mountpoint, and fuse mount), no S3. Same result on all of them.
Here is an strace -f http://pastebin.com/YqCbi7Ue
Funny thing is that now i hitting the failure on smaller files(<500kbyte).
I even did a restart because of that. To see if a clean boot would increate it back to the ~700kbyte, but it made no difference.
The annex hasn't been idle since last time, but i haven't done anything major in it. In total the annex is 204mb(up from 185mb at the time of my first post about this)
Someone else reported what looks like the same problem, here: ?transferkey fails due to gpg
I'mm be following up to that bug report. In fact, I think I just managed to replicate the bug!
That's great news.
I suggest we disregard this comment thread, i'll just follow the proper bug report(which i guess i should have made to begin with).
Thanks for your good work.