git-annex-test failures sometimes reflect failures not of git-annex but of externals utils on which it relies. E.g. when my installation or configuration of gpg has problems, git-annex test suite fails due to the tests that rely on gpg. (And there doesn't seem to be a simple way to skip tests that match a regexp.) git-annex could avoid that by running some simple sanity checks (beyond just existence) on gpg or other optional dependencies, and skipping tests if these checks fail. E.g. if simple test commands to encrypt/sign a small file with gpg fail, then skip gpg-based tests (and warn the user).
I kind of like that the test suite will fail if git-annex is being used in some broken environment, because it gives an easy way to check if git-annex is able to fully work.
Note that, the test suite does try a self-test of its ability to use gpg before proceeding with other gpg tests; if that self-test fails it does abort the other tests in that test block. That self-test does not currently test encryption or decrpryption, only that gpg can access the test key pair.