GIN is not recognized as a git-annex enabled repos

Hi, git-annex Team.

I want to ask you this issue. GIN is not recognized as a git-annex enabled repository when using the HTTPS protocol.
I cloned a GIN repository with HTTPS protocol and then did git annex init on the cloned local repository, but it is not recognized as a git annex-enabled remote.
I checked the .git/config file and it says annex-ignore=true.

Because of this, git annex copy --to [GIN remote repository] is not possible with HTTPS protocol.
Is git-annex capable of copying content over the HTTPS protocol?
If there are any mistakes in my understanding please let me know.

sincerely yours,
Rino Mizuguchi

notabug --Joey

RSS Atom

comment 1

If you're using GIN, perhaps you're also using Datalad. Have you read their documentation on using it with git-annex?

Part of that documentation that seems likely to be especially relevant is here:

Important: Take the URL in the browser, not the copy-paste URL Please note that you need to use the browser URL of the repository, not the copy-paste URL on the upper right hand side of the repository if you want to get anonymous HTTPS access! The two URLs differ only by a .git extension:

Brower bar: https://gin.g-node.org/<user>/<repo>

Copy-paste “HTTPS clone”: https://gin.g-node.org/<user>/<repo>.git

A dataset cloned from https://gin.g-node.org/<user>/<repoy>.git, however, can not retrieve annexed files!

The reason is that git-annex needs to construct urls to download files from the repo, starting with git's config file. Looking at the sample repo https://gin.g-node.org/adswa/DataLad-101, an url based on that works https://gin.g-node.org/adswa/DataLad-101/config, but its web page suggests cloning the url https://gin.g-node.org/adswa/DataLad-101.git, and https://gin.g-node.org/adswa/DataLad-101.git/config does not work.

How does git manage to clone the latter url then? Well, the http server is speaking the git smart http protocol. So git makes requests like https://gin.g-node.org/adswa/DataLad-101.git/info/refs?service=git-upload-pack which do work. But that protocol is not useful for git-annex, so it needs the url where the actual files in the repository can be downloaded, not the url where the smart http protocol is spoken.

There is not generally a way to discover one url from the other url (removing ".git" might work in this case, but not in other cases). So unfortunately you will need to remember to use the right url.

Comment by joey — Thu Aug 4 16:22:09 2022

Remove comment

Additional Questions: git annex copy failed

Thanks for the advice, joey.

I understand that the URL in the browser bar needs to be set to remote.
Then, after modifying the remote URL, I am trying to sync the contents of the git annex file to GIN using the https protocol.
But git annex copy command failed.

(base) mizuguchi@ubuntu:~/https$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed

I checked the origin remote has annex-uuid, and it's the same as uuid of [annex] in https://gin.g-node.org/ivis-mizuguchi/https/config.
I would like to know why git annex cannot find test2.txt.

The version of the tool used in my environment is below.

git version 2.24.0
git-annex version: 8.20211123+git12-g02e3756bd-1~ndall+1

The repository in GIN

https://gin.g-node.org/ivis-mizuguchi/https

I have done the following process and failed.

Create a repository named https in GIN with initialization. And set the repository to public.
git clone https://gin.g-node.org/ivis-mizuguchi/https
cd https
git annex init
echo "test" > test.txt
git annex add test.txt
git commit -m "test"
git annex sync origin
echo "test" > test2.txt
git annex add 'test2.txt'
git commit -m 'test2'
git push origin master

git annex copy --to=origin test2.txt

(base) mizuguchi@ubuntu:~/https$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed

cat .git/config

(base) mizuguchi@ubuntu:~/https$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gin.g-node.org/ivis-mizuguchi/https
fetch = +refs/heads/:refs/remotes/origin/
annex-uuid = c7a50190-9146-44de-8f1d-288551fa32e3
[branch "master"]
remote = origin
merge = refs/heads/master
[annex]
uuid = 80f0adc3-3144-4bba-a288-09a9ab009cef
version = 8
[filter "annex"]
smudge = git-annex smudge -- %f
clean = git-annex smudge --clean -- %f

Any advice would be appreciated.
Sincerely,
Rino Mizuguchi

Comment by rinomizu5 — Mon Aug 8 10:31:04 2022

Remove comment

Additional Questions: git annex copy failed

Thanks for the advice, joey.

(base) mizuguchi@ubuntu:~/https$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed

I checked the origin remote has annex-uuid, and it's the same as uuid of [annex] in https://gin.g-node.org/ivis-mizuguchi/https/config.
I would like to know why git annex cannot find test2.txt.

The version of the tool used in my environment is below.

git version 2.24.0
git-annex version: 8.20211123+git12-g02e3756bd-1~ndall+1

The repository in GIN

https://gin.g-node.org/ivis-mizuguchi/https

I have done the following process and failed.

Create a repository named https in GIN with initialization. And set the repository to public.
git clone https://gin.g-node.org/ivis-mizuguchi/https
cd https
git annex init
echo "test" > test.txt
git annex add test.txt
git commit -m "test"
git annex sync origin
echo "test" > test2.txt
git annex add 'test2.txt'
git commit -m 'test2'
git push origin master

git annex copy --to=origin test2.txt

(base) mizuguchi@ubuntu:~/https$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed

cat .git/config

(base) mizuguchi@ubuntu:~/https$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://gin.g-node.org/ivis-mizuguchi/https
fetch = +refs/heads/:refs/remotes/origin/
annex-uuid = c7a50190-9146-44de-8f1d-288551fa32e3
[branch "master"]
remote = origin
merge = refs/heads/master
[annex]
uuid = 80f0adc3-3144-4bba-a288-09a9ab009cef
version = 8
[filter "annex"]
smudge = git-annex smudge -- %f
clean = git-annex smudge --clean -- %f

Any advice would be appreciated.
Sincerely,
Rino Mizuguchi

Comment by rinomizu5 — Mon Aug 8 10:31:26 2022

Remove comment

comment 4

Well, that is a different problem than the problem you reported, isn't it? You seem to have shown that git-annex does in fact recognize your GIN remote as a git-annex repo.

git-annex does not support uploading files to a git repo on GIN using http.

The datalad documentation says to add a remote using the ssh url of the GIN repository, which git-annex can store files on. See http://handbook.datalad.org/en/0.13_a/basics/101-139-gin.html#publishing-your-dataset-to-gin

Without using datalad, the equivilant commands would be something like:

git remote add gin git@gin.g-node.org:/ivis-mizuguchi/https.git
git-annex copy --to gin

So this leaves the question of why git-annex tries to do a http request when you ask it to copy the content to the http repo. That http request failing is what results in the "(not found)" message.

Looks like that is due to git-annex copy checking to see if the content exists on the remote. It does not. So it ought to try to send it, which would fail with a better "copying to non-ssh repo not supported" error message. However, intstead it displays the error.

I've fixed this bug, and now it fails with this message:

copy test3.txt (to origin...) 
  copying to non-ssh repo not supported

Comment by joey — Mon Aug 8 15:27:21 2022

Remove comment

comment 5

Thank you joey for the additional help.

I see that I need to use the SSH protocol to upload the file contents to GIN with git annex.
And I also understand why it showed (not found).

I appreciate your advice. RIno Mizuguchi

Comment by rinomizu5 — Tue Aug 9 00:33:55 2022

Remove comment

Add a comment