devblog/day 219 catching up and looking backgit-annexhttp://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/git-annexikiwiki2014-09-17T20:53:01ZHard linking on local clonehttp://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/comment_1_16b13b2510183a9da5f960ae5765e581/Michael2014-09-13T06:28:01Z2014-09-13T06:28:01Z
<p>Thanks for this feature. It will save a lot of space when working on one-off projects with big scientific datasets.</p>
<p>Unfortunately, there is probably no easy solution to achieve similar savings across file systems. On our shared cluster individual labs have their data in separate ZFS volumes (to ease individual backup handling), but data is often shared (i.e. copied) across volumes when cloning an annex. We need expensive de-duplication on the backup-server to, at least, prevent this kind of waste to hit the backups -- but the master file server still suffers (de-duplication ratio sometimes approaching a factor of 2.0).</p>
comment 2http://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/comment_2_460c064bebb5061fcba2a6c79f039362/anarcat [id.koumbit.net]2014-09-14T17:38:34Z2014-09-14T17:38:34Z
<p>thanks so much for your work on git-annex, joeyh. it's hard to imagine that just 4 years ago, we didn't have anything even close to this tool and how far it went since then.</p>
<pre><code>anarcat@marcos:git-annex$ pepper age
Loading cache index... done
git-annex is 3 years old
git-annex's birthday is in about 1 month (October 19th)
</code></pre>
<p>birthday is coming soon! <img src="http://git-annex.branchable.com/smileys/smile.png" alt=":)" /></p>
<p>it's also quite impressive how much work can be done in a single year with some (fairly minimal) funding to dedicate a full dev on a project. very inspiring - keep up the good work! -- <a href="http://git-annex.branchable.com/users/anarcat/">anarcat</a></p>
camlistorehttp://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/comment_3_9699d5a9de5ea64fbc876352e20261c4/anarcat2014-09-17T20:18:56Z2014-09-17T20:18:56Z
have you looked at <a href="http://camlistore.org/">camlistore</a> at all? it's a fairly new project, but it seems like it could be an interesting backend, or at least inspiration for git-annex's design. --<a href="http://git-annex.branchable.com/users/anarcat/">anarcat</a>
Camlistorehttp://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/comment_4_23c4ede3db0ea8165311466881cfa6a2/Giovanni2014-09-17T20:36:43Z2014-09-17T20:36:43Z
<p>anarcat, have you used it? I tried, but it is buggy. They seem to be at <a href="http://git-annex.branchable.com/">"the Archivist"</a> group of people, but if you don't have a hard drive to store the things, everything breaks up. I paid a lot of money to Amazon because I believed I could use Camlistore to organize data stored at S3, but apparently S3 is "just for backup", and if it is your only storage, then Camlistore will keep fetching data over and over "to index it" and in the end you pay.</p>
<p>Yes, it keeps working, so you need some server online at all times, with Camlistore running.</p>
comment 5http://git-annex.branchable.com/devblog/day_219__catching_up_and_looking_back/comment_5_7997305d7ec7db072b78dd0c31ecd824/anarcat2014-09-17T20:53:01Z2014-09-17T20:53:01Z
i haven't tried it at all - only looked at <a href="https://www.youtube.com/watch?v=yxSzQIwXM1k">this one demo video</a> that reminded me of git-annex a lot...