Recent comments posted to this site:

Running the command above gives me the same error on Xubuntu 16.04, using git-annex-standalone package from NeuroDebian repositories.

git-annex version: 6.20161122+gitg9f179ae-1~ndall+1
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64

I encountered other commands that fail as well:

$ touch u.txt ü.txt
$ git annex add

$ git-annex calckey ü.txt
# prints key

$ git-annex calckey --batch
ü.txt
# dies

$ git-annex lookupkey ü.txt
# prints key

$ git-annex lookupkey --batch
ü.txt
# dies

$ git-annex metadata --batch --json
{"file":"ü.txt"}
# dies

$ git-annex metadata --batch --json
{"file":"u.txt","fields":{"ü":["b"]}}
# dies

$ git-annex metadata --batch --json
{"file":"u.txt","fields":{"a":["ü"]}}
# dies

All those die without output, all $? are 0. No values were recorded to metadata. Also:

$ git-annex-metadata --json
# entry for "ü.txt" has "file":"��.txt"
Comment by alpernebbi Mon Dec 5 20:46:07 2016

For a while things were working, but now it's not working again, same problem as before.

Do you think maybe it's a timestamp bug in the signature or something? That could explain this "mysteriously works then stops working" behavior.

Comment by justin.lebar Sun Dec 4 04:30:38 2016

What I'd really like to have is an interface that displays a one-time-use phrase of five to ten words, that can be read over the phone or across the room. Exchange phrases with a friend, and get your repositories securely linked together with tor.

I already mentionned the project in telehash, but magic-wormhole does exactly that:

% wormhole send README.md
Sending 7924 byte file named 'README.md'
On the other computer, please run: wormhole receive
Wormhole code is: 7-crossover-clockwork

Sending (<-10.0.1.43:58988)..
100%|=========================| 7.92K/7.92K [00:00<00:00, 6.02MB/s]
File sent.. waiting for confirmation
Confirmation received. Transfer complete.

Receiver:

% wormhole receive
Enter receive wormhole code: 7-crossover-clockwork
Receiving file (7924 bytes) into: README.md
ok? (y/n): y
Receiving (->tcp:10.0.1.43:58986)..
100%|===========================| 7.92K/7.92K [00:00<00:00, 120KB/s]
Received file written to README.md

While that example shows a file transfer, arbitrary data can be transfered this way. There's a documented protocol, and it's not completely peer-to-peer: there are relay servers to deal with NAT'd machines. But the PAKE protocol (basically SPAKE2) could be a good inspiration here.

Otherwise, I must say that, as a user, I don't mind copy-pasting a hidden service string (if that's what it's about): i can do that over a secure medium (email + OpenPGP or IM + OTR) easily... But I understand it can be difficult to do for new users.

Comment by anarcat Wed Nov 30 22:16:19 2016

The homebrew formula doesn't yet this fix, but you can get around the problem in the meantime by getting a newer SSH via homebrew:

brew install homebrew/dupes/openssh

You can then choose to keep that or get rid of it when the formula for git annex is later updated.

Comment by palday Tue Nov 29 02:17:52 2016

aws has merged a PR to support http-conduit 2.2, but git-annex itself doesn't build with the new component yet:

[ 95 of 544] Compiling Utility.Url      ( Utility/Url.hs, dist/build/git-annex/git-annex-tmp/Utility/Url.o )

Utility/Url.hs:354:34: error:
    * The constructor `StatusCodeException' should have 2 arguments, but has been given 3
    * In the pattern: StatusCodeException s _ _
      In an equation for `matchStatusCodeException':
          matchStatusCodeException want e@(StatusCodeException s _ _)
            | want s = Just e
            | otherwise = Nothing

Utility/Url.hs:354:34: error:
    * Couldn't match expected type `HttpException'
                  with actual type `HttpExceptionContent'
    * In the pattern: StatusCodeException s _ _
      In an equation for `matchStatusCodeException':
          matchStatusCodeException want e@(StatusCodeException s _ _)
            | want s = Just e
            | otherwise = Nothing
Comment by felixonmars Mon Nov 28 04:17:12 2016

Seems as if the problem still exists in 6.20161118 (Debian).

I have three repositories (among others), jolla, sts-3xx, and here. jolla and here are in group manual, sts-3xx is backup; here and sts-3xx have assistants running, jolla not. jolla and sts-3xx have slightly older versions of git-annex installed.

Now, when I copy a file from here to jolla like this

git annex copy real_programmers.png -t jolla

the file is subsequently dropped by the assistant:

drop real_programmers.png (locking jolla...) [2016-11-27 13:00:02.667376556] chat: ssh ["-S",".git/annex/ssh/jolla","-o","ControlMaster
=auto","-o","ControlPersist=yes","-F",".git/annex/ssh.config","-T","jolla","git-annex-shell 'lockcontent' '/~/Music/media/' '--debug' '
SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png' --uuid 5298e3ce-1106-4d5e-b052-0aee4b27a344"]
(locking sts-3xx...) [2016-11-27 13:00:03.252473676] chat: ssh [..., "git-annex-shell 'lockcontent' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d 36380d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec"]
(lockcontent failed) [2016-11-27 13:00:03.486158016] process done ExitFailure 1
(checking sts-3xx...) [2016-11-27 13:00:03.487047149] call: ssh [..., "git-annex-shell 'inannex' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d363 80d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec"]
[2016-11-27 13:00:03.76435136] process done ExitSuccess
[2016-11-27 13:00:03.764705754] Dropping from here proof: Just (SafeDropProof (NumCopies 2) [RecentlyVerifiedCopy UUID "1fec6253-171d-4 f86-885b-e233be2d65ec",LockedCopy UUID "5298e3ce-1106-4d5e-b052-0aee4b27a344"] (Just (ContentRemovalLock (Key {keyName = "ff98a733cc012 2858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png", keyBackendName = "SHA256E", keySize = Just 84499, keyMtime = Nothing, keyChun kSize = Nothing, keyChunkNum = Nothing}))))
[2016-11-27 13:00:04.24333081] process done ExitFailure 1
ok
[2016-11-27 13:00:04.251232455] dropped real_programmers.png (from here) (copies now 4) : drop wanted after Upload UUID "5298e3ce-1106- 4d5e-b052-0aee4b27a344" real_programmers.png Just 84499

However, I failed to reproduce the problem by replicating my setup with fresh repositories …

Please let me know if you need more information, and so many thanks for git-annex!

Comment by boh Sun Nov 27 12:23:20 2016

I really don't know what to say. I can't even figure out which computer I updated git-annex on to test if it was still happening.. let alone reproduce it anymore. It does work fine.

I'm so sorry to bother you with this, I've done something stupid! This is exactly why you ask for a transcript of bugs occurring. (Feel free to use this as an example for why you ask for them, so some good can come of it at least..).

Comment by CandyAngel Fri Nov 25 20:27:07 2016
I was wondering if it is possible to share a rsync special remote between repository which are not parented in any way. The use case would be that even if these repositories are not related at all they still may contains the same binary file. It would be useful to have a single rsync remote in order to reduce space usage. I think it could work as the object names are based on their checksum, but I wonder if anyone has already try that ?
Comment by davidriod Thu Nov 24 19:23:42 2016

Been using the one-liner. Despite the warning, I'm not dead yet.

There's much more to do than the one-liner.

This post offers instructions.

First simple try: slow

Was slow (estimated >600s for 189 commits).

In tmpfs: about 6 times faster

I have cloned repository into /run/user/1000/rewrite-git, which is a tmpfs mount point. (Machine has plenty of RAM.)

There I also did git annex init, git-annex found its state branches.

On second try I also did

git checkout -t remotes/origin/synced/master

So that filter-branch would clean that, too.

There, filter-branch operation finished in 90s first try, 149s second try.

.git/objects wasn't smaller.

Practicing reduction on clone

This produced no visible benefit:

time git gc --aggressive time git repack -a -d

Even cloning and retrying on clone. Oh, but I should have done git clone file:///path as said on git-filter-branch man page's section titled "CHECKLIST FOR SHRINKING A REPOSITORY"

This (as seen on https://rtyley.github.io/bfg-repo-cleaner/ ) was efficient:

git reflog expire --expire=now --all && git gc --prune=now --aggressive

.git/objects shrunk from 148M to 58M

All this was on a clone of the repo in tmpfs.

Propagating cleaned up branches to origin

This confirmed that filter-branch did not change last tree:

git diff remotes/origin/master..master
git diff remotes/origin/synced/master synced/master

This, expectedly, was refused:

git push origin master
git push origin synced/master

On origin, I checked out the hash of current master, then on tmpfs clone

git push -f origin master
git push -f origin synced/master

Looks good.

I'm not doing the aggressive shrink now, because of the "two orders of magnitude more caution than normal filter-branch" recommended by arand.

Now what? Check if precious not broken

I'm planning to do the same operation on the other repos, then :

  • if everything seems right,
  • if git annex sync works between all those fellows
  • etc,
  • then I would perform the reflog expire, gc prune on some then all of them, etc.

Joey, does this seem okay? Any comment?

Comment by StephaneGourichon Thu Nov 24 11:27:59 2016

As noted, include appears to not work on a mac at the moment. This means git-annex silently ignores the included configs, which may be required to ssh to the remotes of interest. This is happening to me.

My understanding is that ssh aliases are the recommended way of juggling multiple private keys amongst multiple hosts, so it is a required part of many git workflows. In this particular case, I have set up git annex on a NAS which does not allow multiple ssh users (QNAP) and the authentication is done only via key identity, not username. Thus, host aliases are necessary.

If one config can't include another, I would prefer an early failure indicating a problem with the config file, or better, a solution where git-annex doesn't require a config. In this scenario, git fetch remote_name and git annex copy --to remotename do not resolve to the same alias definitions (the latter is missing because of the ignored config!).

I got my setup to work only by finding and manually editing /.git/annex/ssh_config, which to my knowledge is undocumented (ie when is it written? do any commands change it?); manual mucking around inside .git to me is not a good practice, and for now I have two different alias's defined (in repo and in ~/.ssh/config)

Comment by scottgorlin Wed Nov 23 20:07:45 2016