Hi,
I have a lot of files which are around 80MB and can be easily compressed down to ~55MB. I did some tests with brotli and decompression was reasonable fast, at least fast enough that I would probably not notice given my current transfer speeds. In order to save disk space I would like to able to transparently compress my files. That means, a file is stored compressed in git-annex's blob store and decompressed when I get
it.
I understand that gpg does compression, but I don't want to deal with encryption, all my repos are local. I've looked at the code and from what I could see the Hash-Backends are rather simple. However, that's probably not the right place. Is this a planned feature? Would it be hard to implement? Of course, ideally the compression algorithm should be configurable. E.g. by just doing a syscall to brotli
or gzip
.
Thanks for your hint, I appretiate that and I think it could be done that way.
However, on closer thought I was wondering if git-annex is the right tool for job. I had the impression that my idea came more from a hammer and nail situation. So, FUSE came to my mind. I popped it into google and found this: https://github.com/FS-make-simple/fusecompress Unfortunately this does not look very active though.
Now, since I'm already at the FS layer I can look into ZFS compression. My repos are already on ZFS but I haven't looked at the built-in compression yet. I think I'll evaluate that first. If none of that is satisfactory I'll turn to git-annex again
Just a quick update:
I've enabled zfs compression on my backup storage and couldn't be happier. Same goes for btrfs volumes. If one would like to narrow it down it's easy to create separate volumes (for zfs and btrfs) with different compression settings. I think that's the way to go, adding compression to btrfs would be just re-inventing the wheel.