Is there a way to only add binary files with git annex add command?
If you can configure annex.largefiles to match only binary files, then git annex add will respect it, and only add those files.
git annex add
For example, if you were working on a game written in C, and wanted to use git-annex only for the game art, but not the source code, you could configure it:
git config annex.largefiles 'largerthan=100kb and not (include=*.c or include=*.h)'
This doesn't currently support looking at the file content to determine eg, its MIME type. That's been suggested as an added feature before.
More simply, if you git add the non-binary files yourself first, git annex add will skip over those files and only add the other files.