Using git-annex as a library

I am looking to use git-annex as a library/tool to bring syncing to another application. Is there any documentation or open examples of doing something like this?

RSS Atom

comment 1

I should say specifically interfacing with git-annex

Comment by George — Sat Jun 29 16:07:45 2013

Remove comment

comment 2

I think there are two approaches to doing this.

Use the git-annex command line interface from your program. You can use the --json switch to enable machine-parsable output of many git-annex commands. If something needs work to be more suitable to be used as "plumbing" in this way, we can improve it to meet your needs.
Use the git-annex Haskell code as a library for your program. The git-annex assistant is a great example of how far you can take this. It has the benefit that by accessing git-annex's internals, you can sometimes do things more efficiently than by using the CLI. Much of git-annex's code is already well modularized and suitable for use as a library in this way. The build system doesn't currently spit out git-annex libraries, but it would not be hard to make it do so. Of course this would entail writing at least some of your program in Haskell.

Comment by joeyh.name — Sun Jun 30 18:08:52 2013

Remove comment

comment 3

With version 5.20140127 first approach became impossible. Why did you decide to remove json supported by the majority of command?

Comment by Alexcei — Tue Mar 25 05:43:36 2014

Remove comment

comment 4

I didn't. I removed the --json options from commands that didn't necessarily output valid json when used with it. There are numerous commands that support --json (find, whereis, info, status, metadata, examinekey), and I'm open to adding it to other commands -- but it's probably not feasible to make commands like get output json, since their output includes progress info printed by rsync etc.

Comment by joeyh.name — Wed Mar 26 18:31:30 2014

Remove comment

Would you accept a patch?

Dear Joey,

I am also interested in using git-annex as a Haskell library, would you accept a patch to the .cabal file then?

Thanks, Emilio

Comment by Emilio Jesús — Sun Aug 3 01:18:54 2014

Remove comment

comment 6

There are a couple of problems with using the haskell code as a library that would need to be addressed:

I can't guarantee much about providing every interface in a compatible way going forward. It might make sense to pick out some key interfaces and make those stable, but I don't know what the right choices would be.
If all of git-annex is a library, cabal build will build everything a second time. This is annoying when trying to do a fast edit/build/test cycle, but I don't know a way to make cabal not do it. AFAIK cabal build flags cannot be used to disable building a library.

Comment by joeyh.name — Tue Aug 12 17:48:34 2014

Remove comment

comment 7

Hmm, cabal build does not build the library 2x in the case of propellor, which has a similar split between the propellor library and the executables that depend on it. Perhaps cabal has improved that since I posted my comment.

Comment by joey — Tue Nov 21 20:23:19 2023

Remove comment

comment 8

A lot of git-annex's end user functionality is in the Command/* modules. Much of the code in those is not in a form that would be useful in a library, unless the library was structured to run a Command.

If the goal is say, to add a file to git-annex, and then to be able to get and drop the file, and Command.Add, Command.Get, and Command.Drop were not in the library, you'd have to put together the eqivilant of what those commands do out of the internal library code:

Command.Add essentially uses Annex.Ingest followed by calling into Logs.Location to update the location log.
Command.Drop uses Annex.NumCopies to construct a drop proof and passes it off to Annex.Drop.
Command.Get handles trying different remotes itself, calling into Annex.Transfer.

So the library structure works for git-annex as a thing for Command modules to use, but not great for other things. The assistant actually imports some Command modules, eg it uses Command.Add.addFile. Similarly it would be possible to call Command.Get.getKey.

Maybe it would be better to have a library interface that does mirror the git-annex command line, so you could run eg:

runCommand (Command.Add.cmd) (AddOptions {...}) "somefile"
runCommand (Command.Drop.cmd) (DropOptions {...}) "somefile"

That would necessarily have output like those commands too (unless quiet mode were enabled).

It would take some work to get there from the current state.

Comment by joey — Tue Nov 21 20:25:55 2023

Remove comment

comment 9

A belated thanks for the follow-up from https://git-annex.branchable.com/forum/Using_git_annex_as_a_library/

Not that its useful for anyone here, but just FWIW after some more playing around I decided to keep things minimal and not depend on git-annex as a library. Although what I'm making naturally pairs well with git-annex it doesn't really require it, so adding the dependency just complicates things.

Comment by oadams — Sat Dec 16 21:37:21 2023

Remove comment

comment 10

For my two cents, I have found git-annex to be a simple enough format that I have only needed basic helper scripts.

But many operations can be done with one or a few lines of code.

Git can do much of the heavy lifting for you in terms of looking stuff up from the git-annex branch, and I find the formats to be quite regular and easy to parse.