mimeencoding detection is not working

Please describe the problem.

Mimeencoding detection doesn't work for files with Cyrillic filenaes.

What steps will reproduce the problem?

I have

git annex config --set annex.largefiles 'mimeencoding=binary and largerthan=0'

So I expect all binary files to be annexed.

But I have some jpg file with Cyrillic letters in filename: привет.jpg

$ file --mime-encoding привет.jpg
привет.jpg: binary

$ git annex add привет.jpg --verbose
add привет.jpg (non-large file; adding content to git repository) ok
(recording state in git...)

$ mv привет.jpg hello.jpg

$ git annex add hello.jpg --verbose
add hello.jpg
ok
(recording state in git...)

What version of git-annex are you using? On what operating system?

git-annex 10.20220223-g8f6b52b77

Windows 11

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

I just started using it and I love it. I like it more than git LFS

RSS Atom

comment 1 dbcdb6ae6fb1d06f4d5cf27cba4c69be

[!comment format=mdwn username="joey" subject="""comment 1""" date="2022-05-05T14:50:13Z" content=""" Is your git-annex built with support for mime type detection? Post the output of `git-annex version` I don't think it's possible that a filename can affect this. I'd only believe that if you showed me the same content in a file without a cryllic filename being treated differently. Also, when I try feeding all the data into `git-annex matchexpression`, it behaves as expected: joey@darkstar:~/lib/tmp> if git-annex matchexpression --file=привет.jpg --size=100 --mimeencoding=binary --largefiles 'mimeencoding=binary and largerthan=0'; then echo match; fi match You could try the same command to see if your git-annex behaves differently. """]]

— Thu May 5 14:57:20 2022

comment 2

Thanks for your response.

Your test for matchexpression doesn't show anything because you put --mimeencoding=binary part in it. So obviously expression will match.

$ git annex version
git-annex version: 10.20220223-g8f6b52b77
build flags: Assistant Webapp Pairing TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: mingw32 x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 2 3 4 5 6 7 8 9 10
local repository version: 10

I prepared a simple script to reproduce the bug

mkdir bug
cd bug
git init
git annex init --version=10
git annex config --set annex.largefiles 'mimeencoding=binary and largerthan=0'
wget https://picsum.photos/id/237/200/300.jpg -O hello.jpg
wget https://picsum.photos/id/237/200/300.jpg -O привет.jpg
git annex add hello.jpg --verbose
git annex add привет.jpg --verbose

It behaves as I described before. The file with Cyrillic letters is added as non-binary.

I think it proves that something is not right with MagicMime library dealing with Cyrillic filenames

Comment by mnaoumov — Thu May 5 22:44:01 2022

Remove comment

comment 3

Thanks, I think you must be right that there is some form of mojibake involved here, that is preventing cryllic filenames being sent through to libmagic, but only on Windows.

windows support talks about filename encoding problems on windows, and this is probably one of those.

Comment by joey — Mon May 9 15:21:59 2022

Remove comment

Add a comment