megaannex

Hook program for gitannex to use mega.co.nz as backend

Requirements:

requests>=0.10
pycrypto

Credit for the mega api interface goes to: https://github.com/richardasaurus/mega.py

Install

Clone the git repository in your home folder.

git clone git://github.com/TobiasTheViking/megaannex.git 

This should make a ~/megannex folder

Setup

Run the program once to make an empty config file

cd ~/megaannex; python2 megaannex.py

Edit the megaannex.conf file. Add your mega.co.nz username and password

Note: The folder option in the megaannex.conf file isn't yet used.

Commands for gitannex:

git config annex.mega-store-hook '/usr/bin/python2 ~/megaannex/megaannex.py store --subject $ANNEX_KEY --file $ANNEX_FILE'
git config annex.mega-retrieve-hook '/usr/bin/python2 ~/megaannex/megaannex.py  getfile --subject $ANNEX_KEY --file $ANNEX_FILE'
git config annex.mega-checkpresent-hook '/usr/bin/python2 ~/megaannex/megaannex.py fileexists --subject $ANNEX_KEY'
git config annex.mega-remove-hook '/usr/bin/python2 ~/megaannex/megaannex.py delete --subject $ANNEX_KEY'
git annex initremote mega type=hook hooktype=mega encryption=shared
git annex describe mega "the mega.co.nz library"

Based on the log, it appears that you have a ~/.config/git-annex/program file that contains "/home/pedrocr/software/git-annex/git-annex.linux/git-annex", but that is either not where git-annex is actually installed, or running that program (which git-annex does when it needs to transfer a file contents) fails.

I am able to exactly replicate the failure to transfer file content, and the log output, when I set things up in that way.

This would be consistent with you, for example, having previously installed git-annex from the standalone tarball, used it at least once, and then deleted that installation, and installed it from, say, a Ubuntu repository.

I've put in a fix so if the programfile is wrong, git-annex just tries PATH.

(BTW, I do not advocate storing config files in the git annex. Small files that you want to have fully versioned are best stored in git. The git-annex assistant can still be used for syncing files that are checked into git in the regular way. See replacing Sparkleshare or dvcs-autosync with the assistant.)

(git annex assistant --foreground intentionally logs to the log file, because otherwise the "view logs" page in the webapp can't show any logs.)

Here's the same example you posted being followed by me and showing the problem

$mkdir 1 2
$cd 1; git init; git-annex init; git annex direct; echo "file added to 1" > file_from_1; cd ..
Initialized empty Git repository in /home/pedrocr/Hacks/test-git-annex/1/.git/
init  ok
(Recording state in git...)
commit  
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)
ok
direct  ok
$cd 2; git init; git-annex init; git annex direct; echo "file added to 2" > file_from_2; cd ..
Initialized empty Git repository in /home/pedrocr/Hacks/test-git-annex/2/.git/
init  ok
(Recording state in git...)
commit  
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)
ok
direct  ok
$cd 1; git remote add 2 ssh://localhost/~pedrocr/Hacks/test-git-annex/2; git annex assistant; cd ..
$cd 2; git remote add 1 ssh://localhost/~pedrocr/Hacks/test-git-annex/1; git annex assistant; cd ..
(merging synced/git-annex into git-annex...)
$cd 1
$ls
file_from_1  file_from_2
$ls -la
total 20
drwxrwxr-x 3 pedrocr pedrocr 4096 May 20 20:57 .
drwxrwxr-x 4 pedrocr pedrocr 4096 May 20 20:55 ..
-rw-r--r-- 1 pedrocr pedrocr   16 May 20 20:56 file_from_1
lrwxrwxrwx 1 pedrocr pedrocr  180 May 20 20:57 file_from_2 -> .git/annex/objects/1P/8w/SHA256E-s16--b651aaa274225b617cb4d3033047ac6aee29dd6f2465f94ec38dc6630b7d48c8/SHA256E-s16--b651aaa274225b617cb4d3033047ac6aee29dd6f2465f94ec38dc6630b7d48c8
drwxrwxr-x 9 pedrocr pedrocr 4096 May 20 20:57 .git
$cd ..
$cd 2
$ls -la
total 20
drwxrwxr-x 3 pedrocr pedrocr 4096 May 20 20:57 .
drwxrwxr-x 4 pedrocr pedrocr 4096 May 20 20:55 ..
lrwxrwxrwx 1 pedrocr pedrocr  180 May 20 20:57 file_from_1 -> .git/annex/objects/qQ/x9/SHA256E-s16--cca8b6c2db480aa680e12c48f471a351de69978c7665fac5b63d9a765f4c16f4/SHA256E-s16--cca8b6c2db480aa680e12c48f471a351de69978c7665fac5b63d9a765f4c16f4
-rw-r--r-- 1 pedrocr pedrocr   16 May 20 20:56 file_from_2
drwxrwxr-x 9 pedrocr pedrocr 4096 May 20 20:57 .git
$

Your example is pretty much what I've done. I'll enable a debug log as soon as possible. By the away "git annex assistant --foreground" still logs to .git/annex/daemon.log instead of stdout/stderr, which I assume was not the intended behavior.

This bug was indeed about the assistant not syncing, the comments about the broken symlinks were just in response to edheil's comment. I can post a separate bug report about this but the reason I don't think this should be default behavior is that it breaks several simple uses of git-annex. One simple example is using it to store your configs across machines (things like .bashrc). If at any point the git-annex can't sync a file (a network problem or ssh breakage or whatever) but becomes aware of a file change (for example through xmpp) you now have a broken login shell. In general it breaks the user expectations of having a "folder that just happens to also sync" for something where his files randomly get replaced with strange broken things for odd technical reasons.

Erm, I missed that you want to use direct mode. All this fun with git won't really work in direct mode, and indeed direct mode is not able to guarantee that old versions of modified files are retained.

Direct mode is nice for some applications involving syncing and less than ideal devices, but not for this.

Here is a simple example of setting up 2 repositories in the way the bug reporter describes here and on IRC. As you can see, the assistant syncs file content without any configuration:

joey@gnu:~/tmp/test>mkdir 1 2
joey@gnu:~/tmp/test>cd 1; git init; git-annex init; git annex direct; echo "file added to 1" > file_from_1; cd ..
Initialized empty Git repository in /home/joey/tmp/test/1/.git/
init  ok
(Recording state in git...)
commit  
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)
ok
direct  ok
joey@gnu:~/tmp/test>cd 2; git init; git-annex init; git annex direct; echo "file added to 2" > file_from_2; cd ..
Initialized empty Git repository in /home/joey/tmp/test/2/.git/
init  ok
(Recording state in git...)
commit  
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)
ok
direct  ok
joey@gnu:~/tmp/test>cd 1; git remote add 2 ssh://localhost/~joey/tmp/test/2; git annex assistant; cd ..
joey@gnu:~/tmp/test>cd 2; git remote add 1 ssh://localhost/~joey/tmp/test/1; git annex assistant; cd ..
(merging synced/git-annex into git-annex...)
(Recording state in git...)
joey@gnu:~/tmp/test>cd 1
joey@gnu:~/tmp/test/1>ls
file_from_1
joey@gnu:~/tmp/test/1>ls
file_from_1  file_from_2
joey@gnu:~/tmp/test/1>cat file_from_2
file added to 2
joey@gnu:~/tmp/test/1>cd ..
joey@gnu:~/tmp/test>cd 2
joey@gnu:~/tmp/test/2>cat file_from_1
file added to 1
joey@gnu:~/tmp/test/2>rm file_from_2
joey@gnu:~/tmp/test/2>cd ..
joey@gnu:~/tmp/test>cd 1
joey@gnu:~/tmp/test/1>ls
file_from_1
joey@gnu:~/tmp/test/1>date > newfile
joey@gnu:~/tmp/test/1>cd ..
joey@gnu:~/tmp/test>cd 2
joey@gnu:~/tmp/test/2>cat newfile
Mon May 20 12:20:24 JEST 2013

"What I want to do is to tell the assistant "sync the contents of every file into this repository as soon as you can from whatever remote you happen to be able to copy it from". I couldn't find a setting that would do that though."

The reason you cannot find a setting that does this, is because that is the default behavior of the assistent, when correctly configured.

Since it syncs files for everyone else, I conclude there must be an error in your configuration. You need to descibe it in detail and/or enable debugging and paste a debug log.

If git-annex did not update its working tree until all files referenced in it had been downloaded, then it would be possible, if you have a lot of files, for changes to take a long time, or forever, to show up. I prefer the current behavior.

In any case, is this bug report about the mechanics of how the assistant choses to update its working tree, or is it about your configuration of two repositories that are not syncing with one-another? Conflating two entirely separate issues in one bug report is a good way to add so much noise to it that nothing gets done.