When working on Android in termux, user has limited choices in where to check out an annex:
- in the home directory (
/data/data/com.termux/files/home): well-behaved ext4 file system that allow symlinks, but is only accessible from inside termux. - in shared storage (
/storage/emulated/0, symlinked as~/storage/...): accessible to media apps, but does not support symlinks and is therefore a crippled file system with all its drawbacks. The file system seems to be backed by the same ext4 partition as above, but mounted through some fuse mechanism which I can only assume is responsible for enforcing Android's permissions model.
It would be nice to find a way to have the best of both; here's what I tried so far:
- running git-annex as root for experimental purposes: Does not help, the fuse file system does not support symlinks (
ln -s foo bar→ln: cannot create symbolic link from 'foo' to 'bar': Function not implemented).- If that had worked, I would have had a look at whether it's just a matter of permissions that termux could obtain, but it's obviously a matter of principle.
- running git-annex in home, but ro-bind-mounting a view of it into emulated storage: Only made the files visible to root, not even to termux let alone other processes (I tried adding
--make-sharedwithout knowing what it'd do exactly) - I saw there'd be a way to allow symlinks on Android, but honestly that appears too scary, and given the root ln issue, I think it does not apply any more.
The use case I have in mind is syncing photos to an Android tablet in a v5 repository.
Are there other ideas around on how git-annex can be used with classical symlinks on Android?
Hmmm. Could you talk more about your use case “The use case I have in mind is syncing photos to an Android tablet in a v5 repository.”?
Lets say I have a nice collection of family photos on my desktop computer annexed in a folder called
family. Lets say we havenumcopiesset at 1. I keep only the best photos, so it always takes up about 1gb. In the family folder I have nice organized sub-folders and I add new images occasionally. I have an Android tablet that sits on my coffee table and I want visitors to be able to open the tablet, run some gallery app and view all my family photos. Sometimes i'll grab the tablet and take it with me to family gatherings, so I want all the photos on in, ready for offline viewing.With this use case, perhaps I run assistant inside termux. I clone family into
/data/data/com.termux/files/homeset it as standard group transfer and call itt, I also clone it into/storage/emulated/0, set it as standard group client, and call this crippled client repoc. Ourcfolder should now be accessible to any Android photo gallery app. As we make changes tofamily, on our desktop, assistant on the Android tablet brings them in, when we have wifi, it will probably copy them directly toc?If I take a photo on my tablet, put it in the
cdirectory (which is accessible from my photo app), then this photo should be copied directly to family (if I have wifi). If I don't have have wifi, this photo should be immediately copied totin which case I have two copies of the file on my tablet (I think)? If I delete the photo from the crippled filesystemcI still have the one that was already copied tot. If I re-connect to wifi, perhaps the photo will get sent tofamilyfromt. It might stay intsince it is missing fromc, not sure how this would work (but anyhow, see archive folders in the next paragraph).Maybe I want to save some space on my tablet (deleting from photo app might work?, but archiving is more likely to work). So I open up my photo app, browse to my
cfolder and start putting a bunch of files in a sub-folder namedarchive. They are then immediately deleted from my tablet (unless they were never synced tofamilyin which case I have two copies of them, so I would still have a copy sitting in transfert). If I archive files when I have no wifi, presumably they will just get moved totby assistant since git-annex cannot verify the copy atfamily?Would something like that work for your use case? I don't actually use Android termux, so I am sure others might have some insight into use of the various directories.
—Andrew
The particular repository I'm looking at for a first stage is a wedding photo repository. It contains both the working directories of the photographers involved (some JPEG, some RAW-based with git-versioned Makefiles and sidecars to get the JPEGs which are then checked in with numcopies=0), where some photographers have their files in multiples copies to reflect their pre-sorting. The final photos are also copied to a release directory in various combinations ("complete couple's selection", "pictures released to guests", "other pictures of guests") which were then distributed by a process outside of git-annex' control (zip files were created and distributed over a web server).
This copying workflow usually works greatly with git-annex v5 (or v6 when files are always kept locked): even if files were checked in by different contributors with different default hash settings, copying a file means copying a symlink.
The whole repository amounts to 125gb, with 3gb in the selection.
On the tablet, when I clone the repository on the crippled side of the file system, the initial checkout takes quite some time, and the 3gb get blown to 8gb due to the file being copied not only to the "complete couple's selection", but also to the other selections and the occasional picture in the photographers' directories.
Note that on this repository, the tablet does not even have write access to the repository. It is set to untrusted and "wants" the selection folder.
The next repository I'd like to do this with is our family photo repository, containing an average of four persons' digitized analog and digital photos starting in 2002 and contains about 700gb of images; that repository has not that much duplication, but the interesting files (often a "Selection" folder per event that contains copies with appropriately named files for a sequence) are still checked in in duplicate.
Hmmm, yes, my solution above wouldn't handle de-duplication of the same file in multiple folders. I have a one thought that might work (I can't test right now):
Use proot inside termux to bind a path
/data/data/com.termux/files/home/annex-photosto a mount point~/storage/pictures/annex. Then perhaps symlinks within annex-photos will be presented as files at~/storage/pictures/annexwithout increasing actual storage.proot would only change the view of programs launched under it. I could probably do that (if git annex works under proot on the tablet in question at all; I had to disable it to get it running on Android 7.1 (compatible; LineageOS 14.1)).
The gallery and similar apps would still only see the fuse file system.
Aaaah. OK, that makes sense.
So, when you call termux-setup-storage in Termux it creates a bunch of symlinks to the standard Android directories, calling setupStorageSymlinks. But yeah, the new folders sitting in $HOME are just symlinks to Android's FAT32 view of these folders. I was thinking maybe they were something else…
So after reading Diving into SDCardFS: How Google’s FUSE Replacement Will Reduce I/O Overhead it appears that FAT32 is very much the baked-in standard. The article also has some nice information on the naming conventions in Android of various storage types. Some Google-ing does imply that people have changed the underlying filesystem of the root OS, with rooting, but the solutions seems to now, at least, be device specific.
I can't find any elegant, filesystem level solution for de-duplication of files that would be visible as media to all Android apps in
external storage.Maybe it is possible that with Android's new Adoptable Storage one could create symlinks, but I haven't tried this and I am not optimistic. Adoptable storage is not widely supported, but essentially it fully encrypts your device and re-formats your “external storage” as ext4 or f2fs. But I would imagine that any public facing views of external storage would probably still be FAT32 so their wouldn't be any way to create symlinks.
My other thought would be to create an app specific solution to your issue. For example you are trying to make photos visible to Gallery apps and you have a lot of duplicate photos present in multiple folders. One solution would be to use albums, which I am pretty sure already support having the same photo in multiple albums (at least in Google Photos), while only taking up storage for a single photo. So, you could copy a single copy of each of your photos to
~/storage/pictures, then using Android APIs add the photos into multiple albums. You could probably automate this process with scripts or perhaps a Termux addon? Or, perhaps there is some other album supporting Android Gallery app that uses a simpler album format, like a JSON file or something.ad SDCardFS / FAT32: It seems that both the fuse wrapper around the SD card (be it actually formatted FAT32 because it's the external one, or ext4 as is the encrypted internal one on my device) and SDCardFS (which, it appears to me, is just Google moving the fuse wrapper into the kernel) won't expose the functionality we want.
ad adoptable storage: I think that's what's happening on my device (but am not sure), so if it is, the there's still the fuse wrapper / SDCardFS inbetween.
ad specific app: That'd still mean that git-annex needs to operate on crippled file system, and I don't want to rely on special solutions for each app.
What I've seen when just setting up a new device is that the Storage Access Framework (looks like Android's idea of gvfs / kvfs with a spike of flatpak-portals to me: a userland file system that operates on
saf:/andcontent://URIs rather than files, probably does file operations by means of IPC and receiving file handlers) might help:In the LineageOS built-in file manager, I don't see two "virtual SD cards" (the internal and the external one), but also a "Termux" "device" with proper logo and "0B free", which allows me to peek into the termux home directory (but not its usr directory). From that file manager I even see the photos inside a regular (symlinked) checkout, and can open them with the Simple Gallery app. (Provided they end with '.jpg' and not '.JPG', but that's probably a detail of the file manager).
I did not yet find another app that sees this "Termux" device; Ghost Commander only sees "Documents" and the internal and external SD cards; the "Dir" file manager exposes a SAF selection only when creating directories on external SD (and then shows the same selection as Ghost Commander), the Simple tools (Simple File Manager, Simple gallery) show "Internal", "SD Card" and "Root" in their own storage selection (might be pre-SAF; similar situation also for the Amaze file manager).
If we could reproduce what the builtin file manager does to access the files, that might be a solution here. Termux obviously already helps in this by exporting the home directory (since 0.34), but maybe they also can do more (like exporting XDG user directories for quick access?).
I've looked around for how this whole SAF thing works and where it is supported; it seems to me that it is indeed the way to go:
Oh, Awesome, thanks! Great find. I'll write out some of the things I have tried:
Create a text file called
hi.txtfrom Termux in/data/data/com.termux/files/homeAKA$HOME. LaunchTurbo EditorenableUse the Storage Access Frameworkin Preferences menu. ClickOpen a file|Open fromTermux, clickhi.txt, edit file, save. Now I can see changes in Termux. Nice.From Termux create a new soft link,
cd,ln -s hi.txt hilink.txt, then in Turbo Editor I can see both files, I can edit and save either, and see changes correctly in Termux.I wasn't able to find the Termux
$HOMEfolder in any File Manager I tried, nor any Gallery App.