help running git-annex on top of existing repo

Question

Can git-annex run passively with an existing repo?

Scenario

We have an existing web application with large binary assets spread throughout modules in the repo. The project is in constant development with weekly deploys to production and there are many developers working on the project.

Goal

We need to maintain the directory structure for these assets without actually committing large binaries to the main remote (hosted on GitHub). I need a solution that has a low barrier of entry when on-boarding new developers. I was thinking maybe a script the is executed on every commit that would filter binaries files based on extensions and commit them to the git-annex remote instead. We are flexible on the type of remote storage type (SSH, S3, etc)

Notes

I have gone through the last couple months of forum posts and done a bit of Googling but have come up empty. If anyone can point me in the right direction that would be great. Thanks!

RSS Atom

comment 1

Mixing files controlled by git directly and files controlled by git annex is possible, simply 'git annex add' the large files and 'git add' the rest. You will not be able to use direct mode and you should not run the git annex assisstant, as it will add all files to the annex.

Comment by Karsten — Sat Jun 1 09:21:54 2013

Remove comment

what about with git-svn?

I'm trying to wrap my head around a similar situation. I've tested this by git cloning my repo; the symlinks are copied, and end up broken because the annex directory under .git doesn't exist in the new repo.

So to be specific: can I conclude that when I use git to copy my repo, as long as I don't explicitly use git-annex in the process, I end up with a 'bare' git repo and I don't have to worry about my annexed files coming along?

Also: anyone know of anything that would be different about using git-svn? (i.e. git svn dcommit to push my changes to an svn repo)?

Comment by Hans — Tue Jun 4 08:00:15 2013

Remove comment

comment 3

As long as you're not using Direct Mode, then a git-annex repo is just an absolutely ordinary git repo, which happens to contain some symlinks that point to .git/annex/objects/*

Comment by edheil [wordpress.com] — Tue Jun 4 12:29:04 2013

Remove comment

comment 4

I'm trying to find a way to prevent developers from adding certain file types to the main repo. Is there something like a pre-add hook that could be used?

I found this in another forum but I'm not sure how it was intended to be implemented.

git check-attr addtoannex "$FILE" | grep -q ": set$"  
if [ $? -eq 0 ]; then  
    git annex add "$FILE"  
else  
    git add "$FILE"  
fi

http://git-annex.branchable.com/forum/Let_watch_selectively_annex_files/

Comment by Daniel — Fri Jun 7 23:02:29 2013

Remove comment

comment 5

I think the most useful thing for people in this thread to know about is replacing Sparkleshare or dvcs-autosync with the assistant.

This doesn't solve it at the command line, where you still need to choose between git add and git annex add, but you can use git annex watch to automatically commit small files to git, and large files to the git annex.

Comment by joey — Tue Jun 11 14:30:08 2013

Remove comment

comment 6

This is what I ended up doing.
https://gist.github.com/ifnull/5761255

Basically you just add the extensions of the files you want to exclude to .gitignore_large_binaries and run "git a ." instead of "git add ."

#######################
# Setup
#######################
mkdir annex-test
cd annex-test
git init
git annex init master

#######################
# Fab setup task
#######################
git config --local core.excludesfile ./.gitignore_large_binaries
git config --local alias.a '! sh ./git-add.sh'

#######################
# git a (git-add.sh)
#######################

# Generate annex include arg from .gitignore_large_binaries
include_str="--include='.lazy'";

while read line
do
    if [[ "$line" != *"#"* ]] && ? "$line" != "" ; then 
        include_str="$include_str --or --include=${line}";
    fi
done < "./.gitignore_large_binaries"

# git annex add
git config --local core.excludesfile ./.gitignore;
git annex add $1 $include_str;

# git add
git config --local core.excludesfile ./.gitignore_large_binaries;
git add $1

Comment by Daniel — Tue Jun 11 22:28:52 2013

Remove comment

comment 7

Is there a way just to sync git-annex branch with the git annex sync command? As we have an already existing git branch and want to push/pull master manually.

Comment by Torkaly — Fri Oct 17 11:00:06 2014

Remove comment

comment 8

@Torkaly just git pull as usual, and then run git annex merge to auto-merge the git-anne branches. Then git push origin git-annex to push the git-annex branch.

Comment by joey — Mon Oct 20 15:25:13 2014

Remove comment

Add a comment