provide TAP protocol logging for 'annex test'

ATM outputs from git annex test are custom. I personally find it difficult to find results of failure etc. But what is more important is that it is difficult to script deterministic analytics on top of such results (e.g. as xfailing some specific tests in some specific contexts). I know of TAP protocol which I believe a number of generic testing harnesses support. May be could be some form of --output-format text,tap options for git annex test. There is also a somewhat related SARIF (Static Analysis Results Interchange Format), which was suggested in another scope, but not familiar with it at all in practice.

implemented TastyTap build flag and --tap, done --Joey

RSS Atom

comment 1

https://hackage.haskell.org/package/tasty-tap can be used to do this.

A proof of concept patch is below.

It always uses TAP format, which I would not want because personally I find it hard to read and scan. Using TAP needs to be an option.

Note that every concurrent test job produces its own TAP output. To avoid that problem, in TAP mode the test suite would need to not run concurrent tests.

The added dependency will need a build flag as tasty-tap is not packaged in eg, Debian.

diff --git a/Test/Framework.hs b/Test/Framework.hs
index f47de2c91d..8729bbc531 100644
--- a/Test/Framework.hs
+++ b/Test/Framework.hs
@@ -20,6 +20,7 @@ import Test.Tasty.HUnit
 import Test.Tasty.Options
 import Test.Tasty.Ingredients.Rerun
 import Test.Tasty.Ingredients.ConsoleReporter
+import Test.Tasty.Runners.TAP
 import qualified Test.Tasty.Patterns.Types as TP
 import Options.Applicative.Types
 import Control.Concurrent
@@ -855,9 +856,7 @@ parallelTestRunner' numjobs opts mkts
        args <- getArgs
        pp <- fromOsPath <$> Annex.Path.programPath
        termcolor <- hSupportsANSIColor stdout
-       let ps = if useColor (lookupOption tastyopts) termcolor
-           then "--color=always":args
-           else "--color=never":args
+       let ps = args
        let runone n = do
            let subdir = fromOsPath $ toOsPath tmpdir </> toOsPath (show n)
            ensuredir subdir
@@ -907,6 +906,7 @@ tastyParser ts = suiteOptionParser ingredients (topLevelTestGroup ts)
 ingredients :: [Ingredient]
 ingredients =
    [ listingTests
+   , tapRunner
    , rerunningTests [consoleTestReporter]
    ]

diff --git a/git-annex.cabal b/git-annex.cabal
index d4970cfea5..b48f2323de 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -277,6 +277,7 @@ Executable git-annex
    tasty-hunit,
    tasty-quickcheck,
    tasty-rerun,
+   tasty-tap,
    ansi-terminal >= 0.9,
    DAV (>= 1.0),
    network (>= 3.0.0.0),

Comment by joey — Sat Sep 26 16:45:53 2026

comment 2

I have thought of several different ways to support concurrent tests with TAP output:

Include a TAP parser (or at least a parser for the subset of TAP that tasty-tap uses, which does not include YAML blocks), so subtest process output is gathered, parsed, and re-emitted. (Also needs a TAP generator.)
Add an option like --test-portion=1/3 so the caller can run multiple processes in parallel and do whatever combining they like. This is close to the way subtest processes are implemented currently, so probably the easiest option. Also more flexible since it can be used without using TAP.
Since test numbers are actually optional in TAP output. modify tasty-tap to not include test numbers, and then the outputs of concurrent processes can be concacenated easily. This would probably entail forking tasty-tap, but it is a small amount of code, so total code would probably be less than option #1.
Clean up git-annex's hacks around parallelism and tasty by making tasty itself support running the sub-test processes. That would leave tasty-tap reporting on the collection of all the tasty test results at the top level. This would be the best solution.

However, I think this should be deferred to a later todo if it turns out to be needed.

Comment by joey — Fri Oct 2 14:33:10 2026

Add a comment