on your own server

This tutorial shows how to set up a centralized git repository hosted on your own git server, which can be any unix system with ssh and git and git-annex installed. A VPS, a home server, etc.

This sets up a very simple git server. More complex setups are possible. See for example using gitolite with git-annex.

set up the server

On the server, you'll want to install git, and git-annex, if you haven't already. If possible, install it using your distribution's package manager:

server# sudo apt-get install git git-annex

Note that git-annex-shell needs to be located somewhere in the PATH, so that a client can successfully run "ssh yourserver git-annex-shell". Installing git-annex using a package manager will take care of this for you. But if you're not root or otherwise can't install git-annex that way, you may need to do more work; see get git-annex-shell into PATH.

Decide where to put the repository on the server, and create a bare git repo there. In your home directory is a simple choice:

server# cd
server# git init annex.git --bare --shared
server# cd annex.git
server# git annex init
init ok

That's the server setup done!

make a checkout

Now on your laptop, clone the git repository from the server:

laptop# git clone ssh://example.com/~/annex.git
Cloning into 'annex'...
warning: You appear to have cloned an empty repository.
Checking connectivity... done.

Tell git-annex to use the repository:

laptop# cd annex
laptop# git annex init
init ok

add files to the repository

Add some files, obtained however.

# git annex add *.mp4
add Haskell_Amuse_Bouche-b9OVqxmI.mp4 (checksum) ok
(Recording state in git...)
# git commit -m "added a video. I have not watched it yet but it sounds interesting"

Feel free to rename the files, etc, using normal git commands:

# git mv Haskell_Amuse_Bouche-b9OVqxmI.mp4 Haskell_Amuse_Bouche.mp4
# git commit -m 'better filenames'

Now push your changes back to the central repository on your server. As well as pushing the master branch, remember to push the git-annex branch, which is used to track the file contents.

# git push origin master git-annex
To git@github.com:joeyh/techtalks.git
 * [new branch]      master -> master
 * [new branch]      git-annex -> git-annex

That push went fast, because it didn't upload large videos to the server.

So, to finish up, tell git-annex to sync all the data in the repository to your server:

# git annex sync origin --content
...

make more checkouts

So far you have a central repository on your server, and a checkout on a laptop. Let's make another checkout elsewhere. Clone the central repository as before.

elsewhere# git clone ssh://example.com/~/annex.git
elsewhere# cd annex

Notice that your clone does not have the contents of any of the files yet. If you run ls, you'll see broken symlinks. It's easy to download them from your server either by running git annex sync --content, or by asking git-annex to download individual files:

# git annex get Haskell_Amuse_Bouche.mp4
get Haskell_Amuse_Bouche.mp4 (from origin...)
12877824   2%  255.11kB/s    00:00
ok

RSS Atom

git annex using the "wrong" ssh socket

To avoid frequent typing of pin + RSA passcode + password, we typically establish an ssh control master just once. This works fine with regular git commands, but the git-annex command apparently try to create a different socket. Even that would be ok, except that apparently it is a new socket each time we enter a command.

With sufficient "-vvvv" we see things like:

...
debug1: Executing proxy command: exec ssh -l fred proxy.xxx.yyy direct host
...

(Note I have eliminated references to the actual machine names and userid's.)

If the command had instead been:

 exec ssh -l fred proxy.xxx.yyy direct /home/fred/.ssh/master_host:22

everything would have worked fine. In fact, we are now using:

 git config remote.origin.annex-ssh-options '-S /home/fred/.ssh/master_host:22'

and this eliminates the issue. But it would be nice if git annex could somehow automatically use the pre-existing connection. Is there a better way to achieve this?

Comment by tom_clune — Wed Feb 24 19:23:13 2016

Remove comment

comment 2

git-annex sets up its own ssh connection caching because this makes it a lot faster.

To disable this feature, you can set annex.sshcaching=false, or set remote.origin.annex-ssh-options as you have.

git-annex has no way to know if you have another ssh socket to use, so it seems fine for you to need to configure it if you want it to use one.

Comment by joey — Thu Apr 14 19:40:32 2016

Remove comment

Is it essential to clone from the server?

Now on your laptop, clone the git repository from the server:

Is it possible to init git-annex repo on your local machine, add git-remote there and push local data to the server? Won't there be any problems with this 'non-official' approach?

Comment by ynikitenko — Fri Nov 3 11:28:01 2017

Remove comment

tutorial up to date?

When I follow the tutorial and run

git push origin master git-annex

it replies

error: src refspec master does not match any.
error: failed to push some refs to 'vds:~/test.git'

However one can use git-annex sync instead and that works fine.

Comment by ynikitenko — Fri Nov 3 20:23:12 2017

Remove comment

clone to the server

As usually, create a directory on the server, git init, then git-annex init there.

Add that locally: git remote add my-server-name my-server:~/my-repo

git-annex sync locally seems to work fine and pushes data to the server.

I needed to have this workaround before, because I could not get data from my laptop while on the server (I wasn't sure I had an open IP address for my laptop). This is mostly a basic thing in git, but I had errors with git-annex earlier and I try to be cautious now.

Comment by metst13 — Tue Nov 6 14:58:39 2018

Remove comment

comment 6

I must add that after the previous commands finished, content of the repository was not shown on the server (it was all in .git). I made

git checkout synced/master

and files appeared. Unfortunately it seems all timestamps were new, which I didn't like, and it was already asked here (http://git-annex.branchable.com/todo/does_not_preserve_timestamps/).

Comment by metst13 — Tue Nov 6 16:10:11 2018

Remove comment

problems with init after clone

I'm trying to following the instructions here, but keep running into the following problem whenever I try to clone the remote repo:

0$ git clone ssh://remote.example.com/~/tmp/archive/git
Cloning into 'git'... warning: You appear to have cloned an empty repository.
0$ cd git/
0$ git annex init local
init servo (scanning for unlocked files...)

  Unable to parse git config from origin

  Remote origin does not have git-annex installed; setting annex-ignore

  This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote origin
ok
(recording state in git...)
0$

What am I doing wrong? The remote repo is setup bare, exactly as suggested.

Comment by jrollins — Sun Jan 31 20:34:31 2021

Remove comment

re: problems with init after clone

@jrollins Well, it looks like you have maybe not installed git-annex on your server. If it is installed and you still have this problem, you may need to consult get git-annex-shell into PATH.

Comment by joey — Mon Feb 1 16:03:04 2021

Remove comment

centralized repository workflow

Hi, what would be a recommended setup and the working procedures for the following scenario:

using git-annex version: 8.20210223, which is the one in ubuntu-22.04 (can't upgrade easily)
a central server as a mutable central archive, many users (over ssh)
users are all trusted
the server shall keep all annexed files, but only the HEAD version is relevant, that is: if the file is removed by the user, it shall eventually be permanently removed from the central server too, to save space.
users would tipically not need all the files, but only some, so git annex get files... would do
users would also add or remove annexed files (and push them to the central repository)
a user might remove his/her local repository at any time, so the central server shall not keep track about clones or at least shall not care if any or all clones get removed

I have created central repository like this (please correct me):

git init test --bare                                                                                       
cd test                                                                                                    
git annex init                                                                                             
git annex required . "include=*"

On the user site

git clone ssh://some.server/repo/test test
cd test
dd if=/dev/random of=./bigfile bs=1M count=10
git annex add bigfile
# how to sync (push only)?
# how to permanently remove big file?
cd ..
# done with the task
chmod -R 777 test
rm -rf test

What I am looking for is the sequence of commands for the users, to:

sync to the latest state (without fetching the content)
add new annexed file to the repository and push it
permanently delete annexed file

There are several issues I am facing at the moment. I was expecting to push the new file with git annex sync --content --no-pull, but this command also pulls the contents of all annexd files, which I don't want. The server does not want to remove the old content. It looks like I am doing something wrong. Appreciate your suggestions about this scenario.

Comment by zoran.bosnjak — Tue Feb 6 12:33:20 2024

Remove comment

Add a comment