This special remote stores file contents in ipfs.
Warning: ipfs is written in Go, a language controlled by Google, which collects user's data with its compiler. If installing ipfs from source you should consider investigating how to disable Go's data collection. Or install precompiled ipfs from a software distributor you trust.
prerequisites
- Install git-annex-remote-ipfs somewhere in PATH
and
chmod +xthe script. - Install go-ipfs somewhere in PATH.
- Run
ipfs initand start theipfs daemon
(Note that this special remote does not use ipfs's FUSE support; it
communicates with ipfs using the ipfs command-line utility.)
configuration
These parameters can be passed to git annex initremote to configure the
remote:
encryption- One of "none", "hybrid", "shared", or "pubkey". See encryption. Note that this is git-annex's encryption, not ipfs's encryption.keyid- Specifies the gpg key to use for encryption.
Setup example:
# git annex initremote ipfs type=external externaltype=ipfs encryption=none
content distribution
After git annex copy --to ipfs, a file will typically only have
been copied to your computer's local ipfs object store. It will not reach
other ipfs nodes on the network until they request the content.
If you set up a clone of your repository on another computer, and install ipfs and enable the ipfs remote there, you can proceed with using it to get files that have been stored in ipfs:
# git annex sync
# git annex enableremote ipfs
# git annex copy --from ipfs
content removal
Removing content from ipfs requires all nodes that have a copy to decide to
delete it. This is not something git-annex can arrange to happen, or
reliably tell has happened, so git annex drop --from ipfs will always fail.
using ipfs addresses
Once a file has been copied to ipfs, you can use git annex whereis
to look up the ipfs address of the file:
# git annex whereis somefile
whereis somefile
ed1c811d-fe42-4436-aa75-56566c990aa8 -- ipfs
ipfs: ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W
In the example above, the ipfs address for the file is
QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W. You can give this
address to any other ipfs user and they can use it to download the file!
You can also use ipfs addresses with git annex addurl. For example:
# git annex addurl ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W --file somefile
That's a real file; try it!
future directions
While perhaps useful, this is just a proof of concept. It's particularly lacking in that it doesn't integrate well git-annex's location tracking with ipfs.
Tracking which ipfs nodes have a copy of an annexed object
would make this special remote work better. In particular, git-annex does
not currently trust ipfs to contain a copy of an object, since it has no
way of keeping track of which which ipfs nodes might contain it. So, eg,
git annex drop will refuse to trust ipfs.
I'm using git-annex version: 5.20150522-gb199d65
The example above gives the ipfs hash for the file, but when I run whereis, the hash is not reported:
Is there a way for git-annex to report the file's ipfs hash ID?
@rob.syme, the reason that does not work when using encryption is that git-annex internally generates an encrypted key, and stores that on IPFS. The ipfs special remote records the IPFS address that can be used for a key, but it's recorded on the encrypted key, which is not the one you're querying with whereis.
This is a general problem with git-annex; ipfs is probably the first special remote that exposes the problem.
Anyway, I'm not sure what benefit knowing the IPFS address of a file encrypted by git-annex is. You need to use git-annex to decrypt it, in general. You might be able to use your gpg key to decrypt it, if the special remote was set up using pubkey encryption, but that's not going to help anyone else you give the IPFS address to access the encrypted data..
Does your use case involve pinning the IPFS address, or something like that?
The use case is only very silly - I just wanted to look directly at the data stored at that IPFS hash - it was not important and certainly not worth changing the way git-annex stores its keys.
On a new machine with a new version of git-annex, I can no longer initialize a new IFPS remote:
It looks like I'm missing a binary somewhere, but I'm not sure which one is missing.
@rob.syme I think that error message is pretty clear: git-annex-remote-ipfs is not present in your PATH or perhaps is not executable.
(Still, the error message could be improved a bit, doing that now.)
IPFS now has an experimental feature for not duplicating file content in the ipfs store, which could be very helpful for this use. See description at https://github.com/ipfs/go-ipfs/issues/3397#issuecomment-284337564 .
To use this, git-annex needs to pass the
--nocopyflag when runningipfs add.ipfs appears to be running some free storage services for large files on ipfs backed by filecoin. https://nft.storage/ and htps://web3.storage/ are both presently ostensibly free and permanent and have cli interfaces to upload to them.
@xloem I'm not sure how services that have CLI interfaces to upload files to them, and then presumably pin them in IPFS are relevant to this IPFS special remote?
Your history of good comments is keeping me from deleting the above 2 as spam, but...
https://nft.storage/terms/ claims this is limited to NFTs and not other data. (Whatever that means; why can't I mint a NFT for a nice gpg encrypted blob?)
I also doubt it actually will store anything for longer than the current NFT mania; the terms have a loophole that allow them to provide 90 days notice before ending the service.
It does have an API if someone wants to build a special remote and use it for general data storage. But that's out of scope for this IPFS special remote AFAICS.
sorry for the comments that could come across as marketing or spam
i am often in a situation where i have nowhere to back files up, so simply appreciated the free services and thought others might need something similar as well
their terms may have changed by now, not sure. they helped me a lot at the time. i'm now using arweave for doing similar free things, which has a custom protocol but is intended to never kick data out. node2.bundlr.network provides free "bundlr" arweave storage for objects 100,000 bytes or smaller.
i had written a hacky special remote for web3.storage at https://github.com/xloem/gitlakepy/blob/master/scripts/git-annex-remote-w3-subprocess but it was kind of flaky, wrapped the cli tool.
i guess arweave isn't relevant here for ipfs. the support for ipfs is nice with w3.storage because you can download things with the ipfs url without requiring the special remote installed.
i have cognitive decline, haven't found recovery from political trafficking, so depending on how my life goes i may make more and more spammy expressions. feel free to clean them up, and thank you for commenting on this one.