This is work in progress, since there is now a hook for users to plug in whatever they want as a remote, here's my recipe for using tahoe-lafs as a remote, this is a copy and paste the relavent section from my .git/config file
tahoe-store-hook = tahoe put $ANNEX_FILE tahoe:$ANNEX_KEY
tahoe-retrieve-hook = tahoe get tahoe:$ANNEX_KEY $ANNEX_FILE
tahoe-remove-hook = tahoe rm tahoe:$ANNEX_KEY
tahoe-checkpresent-hook = tahoe ls tahoe:$ANNEX_KEY 2>&1 || echo FAIL
Where tahoe: is a tahoe-lafs alias, ideally you should create a new alias (DIR-CAP or whatever the terminolgy is) to store your files, I just used the default tahoe: alias for testing.
The only quirk I've noticed is this...
$ git annex whereis .
whereis frink.jar (2 copies)
084603a8-7243-11e0-b1f5-83102bcd7953 -- here (testtest)
1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a
ok
1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a is my [[!google tahoe-lafs]] remote, but there is no label/description on it. The checkpresent-hook was a little confusing when I was setting it up, I'm currently unsure if I am doing the right thing or not with my hook. My get and put commands are a little verbose for now, i might redirect it to /dev/null once I am happier with the overall performance/behaviour my setup.
Other than the quirks above, I am able to put and get files from my tahoe-lafs remote. The only thing that I have not figured out is how to "remove a file" on the remote to free up space on the remote.
You should be able to fix the missing label by editing .git-annex/uuid.log and adding
If
tahoe lsoutputs only the key, on its own line, and exits nonzero if it's not present, then I think you did the right thing.To remove a file, use
git annex move file --from tahoeand then you can drop it locally.@justin, I discovered that "git annex describe" did what I wanted
@joey, yep that is the behaviour of "tahoe ls", thanks for the tip on removing the file from the remote.
It seems to be working okay for now, the only concern is that on the remote everything is dumped into the same directory, but I can live with that, since I want to track biggish blobs and not lots of small little files.
I've just tried to use the ANNEX_HASH_ variables, example of my configuration
git config annex.tahoe-store-hook 'tahoe mkdir $ANNEX_HASH_1 && tahoe put $ANNEX_FILE tahoe:$ANNEX_HASH_1/$ANNEX_KEY' git config annex.tahoe-retrieve-hook 'tahoe get tahoe:$ANNEX_HASH_1/$ANNEX_KEY $ANNEX_FILE' git config annex.tahoe-remove-hook 'tahoe rm tahoe:$ANNEX_HASH_1/$ANNEX_KEY' git config annex.tahoe-checkpresent-hook 'tahoe ls tahoe:$ANNEX_HASH_1/$ANNEX_KEY 2>&1 || echo FAIL' git annex initremote library type=hook hooktype=tahoe encryption=none git annex describe 1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a libraryIt's seems to work quite well for me now, I did run across this when I tried to drop a file locally, leaving the file on my remote
jtang@x00:/tmp/annex3 $ git annex drop . drop frink.sh (checking library...) (unsafe) Could only verify the existence of 0 out of 1 necessary copies Try making some of these repositories available: 1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a -- library (Use --force to override this check, or adjust annex.numcopies.) failed drop t/frink.jar (checking library...) (unsafe) Could only verify the existence of 0 out of 1 necessary copies Try making some of these repositories available: 1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a -- library (Use --force to override this check, or adjust annex.numcopies.) failed git-annex: 2 failed 1|jtang@x00:/tmp/annex3 $I do know that the files exist in my library as I have just inserted them, it seemed to work when I didnt have the hashing, it appears that the checkpresent doesn't seem to pass the ANNEX_HASH_* variables (from the limited debugging I did)
ANNEX_HASH_*oversight. (It also affected removal, btw.)Cool, that seems to make things work as expected, here's an updated recipe
I just needs some of the output redirected to /dev/null.
(I updated this comment to fix a bug. --Joey)
Hey Jimmy: how's this working for you now? I would expect it to go slower and slower since Tahoe-LAFS has an O(N) algorithm for reading or updating directories.
Of course, if it is still fast enough for your uses then that's okay.
(We're working on optimizations of this for future releases of Tahoe-LAFS.)
I'd like to understand the desired behavior of store-hook and retrieve-hook better, in order to see if there is a more efficient way to use Tahoe-LAFS for this.
Off to look for docs.
Regards,
Zooko
@joey thanks for the update in the previous comment, I had forgotten about updating it.
@zooko it's working okay for me right now, since I'm only putting fairly big blogs on stuff on to it and only things that I really care about. On the performance side, if it ran faster then it would be nicer