Please describe the problem.

I'm using git-annex-remote-googledrive to sync a large music library. Some files contain special characters like: chillgressive/[Astropilot\ Music]/Unusual\ Cosmic\ Process\ -\ Spacetime/Unusual\ Cosmic\ Process\ -\ Spacetime\ -\ 03\ Souvenirs\ Éphémère.flac

When running: "git annex sync google --content -v -d"

What steps will reproduce the problem?

The problem seems to be, that git annex sends non utf-8 data to the remote script which crashes due to decoding errors. This seems to be git annex bug as the remote should not have to guess which encoding is used but should be able to use utf-8 always.

environment: LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8

What version of git-annex are you using? On what operating system?

nixos

git-annex version: 6.20180807 build flags: Assistant Pairing WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite dependency versions: bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.4.3 http-client-0.5.13.1 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external operating system: linux x86_64 supported repository versions: 3 5 6 upgrade supported from repository versions: 0 1 2 3 4 5 local repository version: 5

Please provide any additional information below.

# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
git annex sync google --content -v -d                                                                                                    1.8m  Tue 18 Sep 2018 12:29:32 PM CEST
[2018-09-18 12:29:41.854290811] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-09-18 12:29:41.856338835] process done ExitSuccess
[2018-09-18 12:29:41.85643436] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-09-18 12:29:41.858259966] process done ExitSuccess
[2018-09-18 12:29:41.859926397] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..97d762166d735b12c5393a21d5d2eeb27b33eb8d","--pretty=%H","-n1"]
[2018-09-18 12:29:41.861562154] process done ExitSuccess
[2018-09-18 12:29:41.864078093] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-09-18 12:29:41.864393271] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2018-09-18 12:29:41.866339921] read: git ["config","--null","--list"]
[2018-09-18 12:29:41.867406295] process done ExitSuccess
[2018-09-18 12:29:41.867536122] read: git ["config","--null","--list"]
[2018-09-18 12:29:41.86768986] read: git ["config","--null","--list"]
[2018-09-18 12:29:41.86875252] chat: /home/poelzi/.nix-profile/bin/git-annex-remote-googledrive []
[2018-09-18 12:29:42.10211189] git-annex-remote-googledrive[1] --> VERSION 1
[2018-09-18 12:29:42.102224585] git-annex-remote-googledrive[1] <-- EXTENSIONS INFO
[2018-09-18 12:29:42.10234806] git-annex-remote-googledrive[1] --> EXTENSIONS
[2018-09-18 12:29:42.10240219] git-annex-remote-googledrive[1] <-- EXPORTSUPPORTED
[2018-09-18 12:29:42.102579476] git-annex-remote-googledrive[1] --> EXPORTSUPPORTED-SUCCESS
commit 
[2018-09-18 12:29:42.106079317] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in poelzi@galaxy:/run/media/poelzi/backup-1/Music"]
On branch master
Your branch is ahead of 'origin/master' by 3 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
[2018-09-18 12:29:42.325422783] process done ExitFailure 1
ok
[2018-09-18 12:29:42.325517478] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2018-09-18 12:29:42.326719154] process done ExitSuccess
[2018-09-18 12:29:42.326780286] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2018-09-18 12:29:42.328496909] process done ExitSuccess
[2018-09-18 12:29:42.328549028] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"]
[2018-09-18 12:29:42.329769099] process done ExitSuccess
[2018-09-18 12:29:42.329841385] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"]
[2018-09-18 12:29:42.33163463] process done ExitSuccess
[2018-09-18 12:29:42.331711233] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-09-18 12:29:42.333577663] process done ExitSuccess
[2018-09-18 12:29:42.333638691] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-09-18 12:29:42.335399941] process done ExitSuccess
[2018-09-18 12:29:42.337137236] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..97d762166d735b12c5393a21d5d2eeb27b33eb8d","--pretty=%H","-n1"]
[2018-09-18 12:29:42.338812954] process done ExitSuccess
[2018-09-18 12:29:42.338954942] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--"]
[2018-09-18 12:29:42.349381649] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2018-09-18 12:29:42.349788044] read: git ["--version"]
[2018-09-18 12:29:42.351035475] process done ExitSuccess
[2018-09-18 12:30:46.421010356] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","rev-parse","master:"]
[2018-09-18 12:30:46.423208056] process done ExitSuccess
[2018-09-18 12:30:46.426223794] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-09-18 12:30:46.428877128] process done ExitSuccess
[2018-09-18 12:30:46.428974324] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-tree","--full-tree","-z","-r","-t","--","97d762166d735b12c5393a21d5d2eeb27b33eb8d"]
[2018-09-18 12:30:50.326843882] process done ExitSuccess
[2018-09-18 12:30:50.326943526] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","mktree","--batch","-z"]
[2018-09-18 12:30:50.357438031] process done ExitSuccess
[2018-09-18 12:30:50.357526145] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","ee08aadd5604249699ce157df47426c70f62312b","--no-gpg-sign","-p","97d762166d735b12c5393a21d5d2eeb27b33eb8d"]
[2018-09-18 12:30:50.359994305] process done ExitSuccess
[2018-09-18 12:30:50.360058172] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","mktree","--batch","-z"]
[2018-09-18 12:30:50.383728957] process done ExitSuccess
[2018-09-18 12:30:50.383984605] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","acf111be942ccf1cfda391d7ecffc4a9c6a27d6f","--no-gpg-sign","-p","46bf4e6b14c01c587d5eb1be2db75c66479b6e15"]
[2018-09-18 12:30:50.388013862] process done ExitSuccess
[2018-09-18 12:30:50.388101083] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","update-ref","refs/heads/git-annex","125e9dfe8129591025117cc9dc0c752265be7e70"]
[2018-09-18 12:30:50.391058496] process done ExitSuccess
[2018-09-18 12:30:50.391473142] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff-tree","-z","--raw","--no-renames","-l0","-r","9557efae9721f725d73f2b2776a2070aad7b0a39","9557efae9721f725d73f2b2776a2070aad7b0a39","--"]
[2018-09-18 12:30:50.402405642] process done ExitSuccess
[2018-09-18 12:30:50.402521039] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff-tree","-z","--raw","--no-renames","-l0","-r","33bf55cdf1f58109ee31097308693f833c4cca97","9557efae9721f725d73f2b2776a2070aad7b0a39","--"]
[2018-09-18 12:30:50.415691771] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-09-18 12:30:50.41669191] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2018-09-18 12:30:50.4222699] process done ExitSuccess
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs �ph�m�re.flac -> .git-annex-tmp-content-SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac [2018-09-18 12:30:50.422844753] git-annex-remote-googledrive[1] <-- PREPARE
[2018-09-18 12:30:50.423065052] git-annex-remote-googledrive[1] --> DEBUG Running .git-annex-remote-googledrive-wrapped version 0.11.1
[2018-09-18 12:30:50.423111151] Running .git-annex-remote-googledrive-wrapped version 0.11.1
[2018-09-18 12:30:50.423145446] git-annex-remote-googledrive[1] --> DEBUG Using AnnexRemote version 1.2.0
[2018-09-18 12:30:50.423239295] Using AnnexRemote version 1.2.0
[2018-09-18 12:30:50.423278034] git-annex-remote-googledrive[1] --> GETCONFIG prefix
[2018-09-18 12:30:50.423319342] git-annex-remote-googledrive[1] <-- VALUE Music
[2018-09-18 12:30:50.423424438] git-annex-remote-googledrive[1] --> GETCONFIG root_id
[2018-09-18 12:30:50.423486989] git-annex-remote-googledrive[1] <-- VALUE 1vnnTRDDMJWqTjm5hc0BGzxhCW4S7-j7b
[2018-09-18 12:30:50.42358159] git-annex-remote-googledrive[1] --> GETCREDS credentials
[2018-09-18 12:30:50.423727488] git-annex-remote-googledrive[1] <-- CREDS {"access_token":"ya...8Q","client_id":"...","client_secret":"...","refresh_token":"..."} 
[2018-09-18 12:30:51.007447341] git-annex-remote-googledrive[1] --> SETCREDS credentials {"access_token":...} 
[2018-09-18 12:30:51.348855667] git-annex-remote-googledrive[1] --> PREPARE-SUCCESS
[2018-09-18 12:30:51.348957671] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs �ph�m�re.flac
[2018-09-18 12:30:51.34902213] git-annex-remote-googledrive[1] <-- RENAMEEXPORT SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac .git-annex-tmp-content-SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac
Traceback (most recent call last):
  File "/nix/store/478kqwy2yfrx61swlsnhjajna7sl8lhh-git-annex-remote-googledrive-0.11.1/bin/.git-annex-remote-googledrive-wrapped", line 584, in <module>
    main()
  File "/nix/store/478kqwy2yfrx61swlsnhjajna7sl8lhh-git-annex-remote-googledrive-0.11.1/bin/.git-annex-remote-googledrive-wrapped", line 580, in main
    master.Listen()
  File "/nix/store/5wncs360nzq0fgqmv30p8qn0cga8zi0d-python3.6-annexremote-1.2.0/lib/python3.6/site-packages/annexremote/annexremote.py", line 394, in Listen
    for line in self.input:
  File "/nix/store/hy65mn4wjswqih75gfr6g4q3xgqdm325-python3-3.6.6/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 125: invalid continuation byte
git-annex: external special remote protocol error, unexpectedly received "" (unable to parse command)

  rename failed; deleting instead
[2018-09-18 12:30:51.384823525] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs �ph�m�re.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs �ph�m�re (Trance Version).flac -> .git-annex-tmp-content-SHA256E-s90158047--578d406e3db07835d3c4045096c7f8333c5adbbfe1003d37121d5b0656f45688.flac [2018-09-18 12:30:51.385017728] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs �ph�m�re (Trance Version).flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)

  rename failed; deleting instead
[2018-09-18 12:30:51.385128278] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs �ph�m�re (Trance Version).flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
rename google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem�sia.flac -> .git-annex-tmp-content-SHA256E-s41053932--7bb9b80766e1d445bb39442e39e3e727ba7329d00186cccb5b2b1e2b28b493b1.flac [2018-09-18 12:30:51.38521185] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem�sia.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)

  rename failed; deleting instead
[2018-09-18 12:30:51.385281886] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem�sia.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
rename google psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L�t� - Lullaby.flac -> .git-annex-tmp-content-SHA256E-s43901186--9d51034d81247e3e0890d81c1e1c20c06149aac3bf4451d880523dcc89c3de82.flac [2018-09-18 12:30:51.385360183] git-annex-remote-googledrive[1] <-- EXPORT psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L�t� - Lullaby.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)

  rename failed; deleting instead
[2018-09-18 12:30:51.385429424] git-annex-remote-googledrive[1] <-- EXPORT psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L�t� - Lullaby.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
rename google chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac�o.flac -> .git-annex-tmp-content-SHA256E-s89899720--afbc07b561f058db6da5849f728d665d14cc6ea42e2768fecf6396d0009e2861.flac [2018-09-18 12:30:51.385513597] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac�o.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)

  rename failed; deleting instead
[2018-09-18 12:30:51.385594622] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac�o.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs �ph�m�re (Ambient Version).flac -> .git-annex-tmp-content-SHA256E-s81128227--c4222d79979c8562c44847869c20d172f51a8d4670192d860aa2dae1ba3d54c8.flac [2018-09-18 12:30:51.385768001] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs �ph�m�re (Ambient Version).flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)

  rename failed; deleting instead
[2018-09-18 12:30:51.385830411] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs �ph�m�re (Ambient Version).flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
[2018-09-18 12:30:51.38631869] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-tree","--full-tree","-z","-r","--","9557efae9721f725d73f2b2776a2070aad7b0a39"]
export google .gitignore (not available) failed
export google bin/bandcamp-rip-album.variant-5bf5 (not available) failed
export google bin/flac2opus.variant-1201.py (not available) failed
export google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artemísia.flac 
[2018-09-18 12:30:59.661174242] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artemísia.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed
export google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 02 Marwell.flac 
[2018-09-18 12:30:59.663165427] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 02 Marwell.flac
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
failed


# End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

I love git annex, using it very day :)

closing since this is not a bug in git-annex but in the external remote. done --Joey