Init a directory special remote with already existing files

Hi,

I've been a git-annex user for few years now, and I'm progressively migrating old rsync backups into git-annex. Most of the time I create new fresh repos or special remotes and re-upload all data into it. But I'm now facing an issue with this workflow: one of the remote disk I use has a very low connection. Since it has already a complete up-to-date plain copy of all files, I'd like to avoid the "re-upload" phase.

I was thinking of using a directory/rsync special remote, and feed it with the existing local content. But the file names & paths are not the usual plain ones.

is this a good idea ?
if yes, what is the way to retrieve the "special remote" path of each file ? I'm not against scripting a little if necessary.
if no, what can I do ?

Few additional notes:

I can't ssh to the remote, this is a windows share with a FAT fs underneath.
I think I can assume all the remote files are good, they are transfered and checked by rsync.

RSS Atom

comment 1

I think if you don't mind scripting checkout Joey's comment in this thread: "Preseeding" a special remote.

Otherwise maybe the following will work. You'll still need to download all of the files to hash them.

UNTESTED: Lets say your Windows share data is at mnt/share/data

create a new repo on your Windows share at mnt/share/repo1
git-annex-import all of your files into the new repo. This probably would download all of the files to generate the hash, then hopefully do a file rename avoiding the re-upload step.
then create a new directory remote for repo1 at say, mnt/share/remote1
then do a git-annex-move of all the files from mnt/share/repo1 to your new empty remote
you could then make a clone of that repo locally, now you have a nice looking working tree locally
then add mnt/share/remote1 as a new remote to your local clone

If you don't care about hashing you could use the WORM backend avoiding the initial download.

I think, If you don't care about having a nice working tree you could probably use the web special remote and just leave all the files in their messy rsynced state. IE create a local repo, then do git annex addurl --fast <path> for each file on your Windows share. See joey's comment on addurl in this DVD thread

Comment by andrew — Wed Mar 28 00:59:41 2018

Remove comment

comment 2

Thks for the answer, I found many interesting info. I ended up scripting my own crappy stuff, please see below:

# Populate a special remote directory with files from the input source
# The current repository is used to find out keys & file names,
# but is not used directly to copy/move the files from
# WHERE selects which files & repo to look for
# MOVE=1 moves files instead of copying them

alias annex_populate='MOVE= _annex_populate'

alias annex_populatem='MOVE=1 _annex_populate'

_annex_populate() {
  local DST="${1:?No dst directory specified...}"
  local SRC="${2:-$PWD}"
  local WHERE="${3:-${WHERE:---include '*'}}"
  eval git annex find "$WHERE" --format='\${file}\\000\${hashdirlower}\${key}/\${key}\\000' | xargs -r0 -n2 sh -c '
    MOVE="$1"; SRC="$2/$4"; DST="$3/$5"
    if [ -r "$SRC" ]; then
      echo "$SRC -> $DST"
      mkdir -p "$(dirname "$DST")"
      if [ -n "$MOVE" -a ! -h "$SRC" ]; then
        mv -f -T "$SRC" "$DST"
      else
        rsync -K -L --protect-args "$SRC" "$DST"
      fi
    fi
  ' _ "$MOVE" "$SRC" "$DST"
}

Basic usage: annex_populate(m) /my/destination/folder/ /my/source/folder

Principle: we get each file key from the current repository, then build the special remote backend filename and launch the copy/move operation.

Warning:

before using it, make a backup of your data !
it was very little tested: it worked for me and I stopped there. Like many scripts in my toolbox, it may malfunction in different cases than mine.
it should copy files unless you define MOVE=X (or have it defined in your shell ! beware).

Comment by oliv5kta — Wed Apr 25 17:49:48 2018

Remove comment

Add a comment