Right now, non-annexed files get passed through the annex clean/smudge filter (see Adding files to git: Very long "recording state in git" phase). It would be better if git-annex configure the filter only for the annexed unlocked files, in the .gitattributes file at the root of the repository.