The git-annex Windows port is beta, but rapidly becoming polished and usable!
status
Local pairing seems to fail, after acking on Linux box, it stalls. (Also, of course, the Windows box is unlikely to have a ssh server, so only pairing with a !Windows box will work.)
gcrypt is not ported to windows (and as a shell script, may need to be rewritten)
Deleting a git repository from inside the webapp fails "RemoveDirectory permision denied ... file is being used by another process"
Tor remotes are not supported yet. Should not be very hard to get it working.
potential encoding problems
?Unicode file names ignored on Windows is fixed, but some potential problems remain, since the FileSystemEncoding that git-annex relies on seems unreliable/broken on Windows.
When git-annex displays a filename that it's acting on, there can be mojibake on Windows. For example, "háčky.txt" displays the accented characters as instead the pairs of bytes making up the utf-8. Tried doing various things to the stdout handle to avoid this, but only ended up with encoding crashes, or worse mojibake than this.
This may be use to windows actually using utf-16, but git-annex uses utf-8 for filename encoding when on windows.
If interoperating with a git-annex repository from a unix system, it's possible for a key to contain some invalid utf-8, which means its filename cannot even be represented on Windows, so who knows what will happen in that case -- probably it will fail in some way when adding the object file to the Windows repo.
If data from the git repo does not have a unicode encoding, it will be mangled in various places on Windows, which can lead to undefined behavior.
minor problems
- webapp lets user choose to encrypt repo, and generate gpg key, before checking that gcrypt is not installed
- Ssh connection caching does not work on Windows, so
git annex gethas to connect twice to the remote system over ssh per file, which is much slower than on systems supporting connection caching. - glacier-cli is not easily available (probably)
stuff needing testing
- test that adding a repo on a removable drive works; that git is synced to it and files can be transferred to it and back
- Does stopping in progress transfers work in the webapp?
do we need this port anymore?
See http://blog.dustinkirkland.com/2016/03/ubuntu-on-windows.html
If windows has transparent support for running linux executables, and those executables can access files in "." which are on the windows system, then you could just use this to run linux git-annex on windows. No port needed.
That would be great!
Seems like this would need Windows 10.
The latest builds of Windows 10 (build 15063) can run git-annex in the Windows Subsystem for Linux. After following the instructions at https://msdn.microsoft.com/en-us/commandline/wsl/about, run:
sudo apt-get install git-annexgit-annex in WSL passes its full test suite, and it avoids all the problems discussed in sections above.
git-annex can access Windows files in eg
/mnt/c, so a git-annex repository can be stored there. However, if the git-annex repository uses indirect mode, the symlinks used by git-annex won't be usable by Windows programs. Use either direct mode, or v6 mode to avoid the symlink problem.Also, see this important caveat: https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/
March 2017: WSL is currently rather annoying to enable. If it became easy enough to enable, note that "bash -c git-annex" works from a windows command prompt, and would probably work in a .bat file as well, so git-annex from the WSL could be transparently used on the windows side.
The webapp does not currently work. It doesn't know how to open a web browser from the linux side. There are also what look like some emulation problems around the daemonization code.
git annex assistant --foregrounddoes run, but while it notices when new files are added, it does not notice when existing files get modified. Probably an inotify emulation bug. --JoeyUpdate May 2018: With the latest MS Edge VirtualBox image, enabling WSL was much easier than before, just a matter of checking a single box and a reboot. No need to enable developer mode first. (Don't know if this will apply to commercial Windows 10 or only this VM image.)
Then bash was available in the menu, but it said no distro installed, and gave an url to a "Store" where Ubuntu, Debian, etc could be installed.
git-annex test had a failure this time, something to do with disk IO error and sqlite.
The assistant can run as a daemon now. It still doesn't notice some changes. Eg with "rm foo; echo new > foo", it got an inotify event for the removal, but missed the new creation.
Running
git annex assistantwhen the assistant is running complains of a stale pid file, with the wrong pid number, but it doesn't start a second one.webapp still can't open a web browser, but there's a way to do it from within WSL that it should be possible for it to use:
powershell.exe Start http://google.com("powershell.exe" is in PATH)--Joey
The sqlite problem may be fixed in git-annex 7. I fixed some similar, less frequent crashes on Linux. Need to test on windows. --Joey
But here's a bug about sqlite in WSL: WSL adjusted braches: smudge fails with sqlite thread crashed - locking protocol --Joey
update Sept 2022:
Git-annex has become more reliable on WSL1 recently, see Using git-annex on NTFS with WSL1.
What about Cygwin? It emulates POSIX fairly well under Windows (including signals, forking, fs (also things like /dev/null, /proc), unix file permissions), has all standard gnu utilities. It also emulates symlinks, but they are unfortunately incompatible with NTFS symlinks introduced in Vista due to some stupid restrictions on Windows.
If git-annex could be modified to not require symlinks to work, the it would be a pretty neat solution (and you get a real shell, not some command.com on drugs (aka cmd.exe))
Windows support is a must. In my experience, binary file means proprietary editor, which means Windows.
Unfortunately, there's not much overlap between people who use graphical editors in Windows all day vs. people who are willing to tolerate Cygwin's setup.exe, compile a Haskell program, learn git and git-annex's 90-odd subcommands, and use a mintty terminal to manage their repository, especially now that there's a sexy GitHub app for Windows.
That aside, I think Windows-based content producers are still the audience for git-annex. First Windows support, then a GUI, then the world.
Haskell has C++ binding, so it should be possible to port it to .net/Mono with a VB GUI for Windows users. Windows has a primitive form of symlinks called shortcuts. Perhaps shortcut support in Windows could replace the use of symlinks. I've used shortcuts since XP to put my home Windows directory on another partition and never had a hitch...
If anyone is interested in working on this, hit me up. I would like to use this in my XP vbox to have access to files on my host OS...I have a student edition of Visual Studio 2005 to do an open source port...
It should be easy to fix whatever's wrong the the rsync special remote. Just a matter of debugging that.
Adding encryption support on Windows is stuck at a roadblock I don't know the way around. To drive gpg, git-annex uses the
--passphrase-fdoption, and sends the "passphrase" (really a big block of binary foo!) over a file descriptor of a pipe that it set up.Windows, AFAIK, doesn't have file descriptors, or at least there is no equivilant to them that I have access to in the Haskell POSIX compatability layer for Windows. I am reluctant to fall back to using
--passphrase-fileon Windows, since that would be a massive security hole (as would passing the passphrase as a parameter via--passphrase=).Hello.
I volunteer to test the Windows port. I can only do so on the work computer.
I know how to produce a Minimal Reproducible Example. Are there some areas that need attention?
I've enabled the 'email replies to me' ikiwiki feature. (Nice plugin.)
Cheers,
--Dave
Hi, what’s the recommended way to manage a repo accessed from both a Linux and a Windows box? Has anyone had success with that?
I’m dual-booting Fedora 22 and Windows 8.1. I do most of my work on Linux, but I need Windows programs to edit some of my files. I’ve had a go at two approaches:
1) Have a “central” repo on an NTFS external drive which also holds the contents of files. This fails because Windows fails to check out the contents of most files. It isn’t random – specific files at specific points of the commit history fail to check out. Perhaps it has something to do with filenames? Characters in filenames?
2) Keep a “direct” clone of my repo on a Windows NTFS partition. I do my work on Windows, and after rebooting to Linux I mount the partition and commit my changes directly in the repo. Here Windows seems to outsmart me. git-annex sometimes takes many files without content as modified – and commits them as empty.
Ping!
I’m trying to organize my Linux + Windows workflow again. Has anyone had success managing an annex repo from Linux and Windows?
Currently there is a bug that stops all Haskell programs from running on Bash on Windows: https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/13572504-add-support-for-timer-create
https://github.com/Microsoft/BashOnWindows/issues/307
https://blogs.msdn.microsoft.com/commandline/2016/10/07/wsl-adds-inotify-filesystem-change-notification-support/
So it should work now, right?
I find installation on windows unnecessarily complicated with current demand of using x86 git. Not that it is impossible but it makes it more elaborate, even for old timers. Would be nice if it was streamlined more. I even wondered if there is a way to provide (additional) bundles which would come with git itself. This way it could be just the git-annex installer to install all needed for working with git-annex (instead of this multistep process)
Is there any chance of getting ssh connection caching supported on Windows?
I didn’t realize the full impact of this until I started syncing a larger repo, but now I understand if you start a sync you have to hand around and be ready with a passphrase when asked, because the sync will pause multiple times waiting for that input. For testing that’s fine but in production that’s too inconvenient.
Enabling the ssh agent avoids password prompts, but not the latency of making a ssh connection per transfer.
You can try setting the environment variable
GIT_ANNEX_SSH_SOCKET_DIRto point to a directory where ssh can try to put its socket. I don't know if ssh on windows is able to use a socket or not. It might just work.User asakurareiko added some instructions to this page how to use git-annex in Ubuntu 20 (presumably 20.04) in WSL1 and I tested them out in my older Ubuntu 18.04 installation and found out that they amazingly work although my Windows apps couldn't access any of the files that were still locked. Somehow the symlinks were only in a form that only WSL1 and Cygwin/MSYS2/Git Bash could access (mind you I have developer mode active and my Windows 10 is also version 21H1 so the environment was otherwise similar). An interesting use case nonetheless. I edited the section with my corrections -- I hope asakurareiko doesn't mind.
For me I tried git on windows at first but when checking out the working tree after a clone it slowed down so much that it would not complete in a reasonable amount of time. So that's why I decided to try using WSL1.
In my opinion it's not necessary to use
case=dir.case=dirwas at one point the default but was removed as the default due to the potential to cause problems with windows programs (https://devblogs.microsoft.com/commandline/improved-per-directory-case-sensitivity-support-in-wsl). But if you do havecase=dirthen it is not necessary to set the attribute.If your symlinks are not working, make sure to have deleted and recreated the symlinks after doing
git annex getand that the NT symlink requirements listed here have been met. If the symlink target has changed from file to directory or vice versa the symlink also has to be recreated. However there are other reports of symlinks not working despite following these requirements:@asakurareiko: Oh, based on https://docs.microsoft.com/en-us/windows/wsl/case-sensitivity#case-sensitivity-options-for-mounting-a-drive-in-wsl-configuration-file I got the impression that
case=dirwas a prerequisite to make the attributesystem.wsl_case_sensitivework as per the heading "Default setting: dir for enabling case sensitivity per directory". I guess I was wrong,case=offworks too. I'll removecase=dirit from the instructions once again.Also the NT symlink requirements are fulfilled in my case simply by way of me having developer mode enabled in 21H1 (Windows 10 Pro). I create symlinks all the time with "cmd /c mklink ..." in Powershell without elevation. Git-annex also creates only relative symlinks which was also a requirement in the WSL release news you mentioned. No, this must be one of those miscellanous problems mentioned in WSL issue 353.
Edit: Oh, in fact recreating the symlink after getting the file with
git-annex getby deleting the symlink with rm and then checking it out again withgit checkout -- <file>seems to allow me to access the file in Windows just fine. Interesting. Maybe it's git-annex itself that creates the wrong kind of symlink that a later call to plain git can repair. Quite convoluted it seems.One of the requirements in particular is that the link target must exist. In a freshly cloned repo the link targets do not exist so it's important to recreate the symlinks after you get the annexed files.
NT symlinks must have either file or directory target type. If the link target does not exist at the time when the symlink is created, it's not possible to determine the target type.
That's useful info.
Many thanks for your input regarding WSL1 use! For me that opens up new avenues by way of
borg special remotes (borg support on Windows is very lacking and there are no recent binaries for it).
I still think git -- and to a lesser extent git-annex -- is performant enough on native Windows, so I'm
going to continue using it that way for those repos that need to be foolproof and in a supported configuration.
But for more experimental uses WSL1 still holds quite some promise wrt git-annex. DrvFs is still many times faster
than 9p of WSL2 for Windows native files on NTFS which is an important use case for exchanging data between
Windows and POSIXland. In fact while WSL2 boots incredibly fast for a Hyper-V virtual machine, in many file-level
use cases it can't hold candle to the almost seamless nature of WSL1. The latter has its warts in API compatibility
but in my mind it's quite a undervalued little performer. I hope Microsoft doesn't remove it too soon in favor of WSL2.
I've been adding stuff to git annex on my linux server and now I'm thinking that I need to uninit and re-init with
annex.tune.objecthashlower=true. I'll want to use git-annex from windows as well.Should I use
annex.tune.objecthashlower=true? Also, what's the advice around the other tuning optionsobjecthash1andbranchhash1?Thanks!