Running an SSH server on a phone is quite some setup and could perhaps even be considered insecure.
KDE-connect provides a an easy to set up method of creating a transport tunnel between phone and computer. Many users might actually have it set up already.
Apparently, it can even be used to connect multiple desktops with another but I haven't tested that. This could be an alternative (or perhaps even replacement?) to git-annex' current pairing mechanism.
KDE-connect offers remote command execution and file sharing, so it should be possible to use it for git-annex' purposes.
From what I can see, kde-connect lets the phone be used to run specific, pre-selected commands on the linux computer it's paired with. Not the other way around. https://userbase.kde.org/KDE_Connect/Tutorials/Adding_commands Also being able to run a command on a phone is rather a long way from what git-annex actually needs.
It seems like it would be possible to use the remote filesystem via sftp part of it. https://rafaelc.org/tech/p/mounting-kde-connect-filesystem-via-cli/ That would be a useful alternative to adb. I don't think git-annex needs any changes to use it that way, just mount it with sshfs and initremote a directory special remote with exporttree=yes and importtree=yes.
See also add sftp special remote which if it were implemented would avoid needing to use sshfs.