Most special remotes have support for breaking large files up into chunks that are stored on the remote.

This can be useful to work around limitations on the size of files on the remote.

Chunking also allows for resuming interrupted downloads and uploads.

Note that git-annex has to buffer chunks in memory before they are sent to a remote. So, using a large chunk size will make it use more memory.

To enable chunking, pass a chunk=nnMiB parameter to git annex initremote, specifying the chunk size.

Good chunk sizes will depend on the remote, but a good starting place is probably 1MiB. Very large chunks are problematic, both because git-annex needs to buffer one chunk in memory when uploading, and because a larger chunk will make resuming interrupted transfers less efficient. On the other hand, when a file is split into a great many chunks, there can be increased overhead of making many requests to the remote.

To disable chunking of a remote that was using chunking, pass chunk=0 to git annex enableremote. Any content already stored on the remote using chunks will continue to be accessed via chunks, this just prevents using chunks when storing new content.

To change the chunk size, pass a chunk=nnMiB parameter to git annex enableremote. This only affects the chunk sized used when storing new content.

old-style chunking

Note that older versions of git-annex used a different chunk method, which was configured by passing chunksize=nnMib when initializing a remote.

The old-style chunking had a number of problems, including being less efficient, and not allowing resumes of encrypted uploads.

It's not possible to change a remote using that old chunking method to the new one, but git-annex continues to support the old-style chunking to support such remotes.

See also: design document