credential.<url>.annex-ignore seems like the right track, and I suppose there could also be something like http.<url>.sslCAInfo.annex-ignore etc too. Such configs would need to be clearly documented as only being supported by the web special remote and not other special remotes though.

FWIW I don't want DELEGATE to supplant every other use of http in special remotes. Any applicable web API library should be able to be used by a special remote, when that's the lowest friction way to implement one.

@yoh what do you think about that idea?

I might be missing the point for extra parameter (urlinclude) for web remote itself as it might complicate handling multiple <url> settings, and overall feels more of a user/system wide configuration and not remote.

As for the need of support by all special remotes -- now that there is a DELEGATE functionality for special remotes, I think special remotes should generally be advised to just delegate HTTP downloads to git-annex for robust/centralized operation overall where feasible. Then it would fall under centralized credential.<url>. handling by git-annex.

Related: just an idea, what about adding credential.<url>.annex-ignore defaulting to True, just to let folks where needed explicitly set to False so to tell credential provider to be used for that <url>? Since manual configuration of a credential. is anyways required, IMHO it is ok to request explicitly specify if it should apply to git-annex or not, with default being "not".

If that is a security hole, then autoenabled git remotes could already be used for a similar attack. (The user needs to run eg git pull --all rather than git-annex get in order to be exploited, at least until some of these todos get implemented, but that's not a significant difference.)

Now, I do expect that a git-annex user needs to be familiar with the fact that it can autoenable remotes. So I could say that the user who sets a too-broad git config wildcard of sensative information is responsible for the consequences.

But, if I had considered this problem in 2015, I may have thought twice about implementing autoenable. (FWIW, git started supporting url wildcards in configs in 2013.)

Anyway, if that is a security hole then autoenabling is at the root of it, and it would need to be fixed by changing the default autoenable behavior. Eg, git-annex could default to not allow autoenable, and the config to allow it could discuss using config wildcards combined with autoenable.

However, that could lead to a security hole...

For example, suppose a user has read the git documentation above, and knows that http.<url>.extraheader will only be used for urls that they give to a git command. They know that the only git repos that they'll be using under sharedhosting.com are under foo.sharedhosting.com and bar.sharedhosting.com, which are both sites that they control. So they decide to set http.https://*.sharedhosting.com/.extraheader to an authentication token to use for both. That's a shortcut, but it's fine.

... Until they clone a git-annex repo from elsewhere that has an autoenabled web special remote configured with urlinclude=https://baz.sharedhosting.com/, which is a site controlled by an attacker, who intercepts the extraheader that git-annex sends.

I think that credential.<url>.* configs could also be subject to that kind of security hole, if a credential helper had a config that contained sensitive information. (I guess that even credential.<url>.username could be considered sensative information by someone.)

I also think you're pretty far down a slippery slope with this repository. It just happens to have a S3 proxy endpoint under it. Any HTTP resource could be put under a repo that way, and if that were used to argue that these configs should apply to it, then every git-annex special remote that uses HTTP would need to support applying those configs to every url that it accesses. No matter what third-party library might be used for a cloud service protocol. Even external special remotes. That is clearly taking it too far.

There could be room for a middle ground, and it might be something like a separate web special remote that is configured with urlinclude=https://datalad-test.local.lan inheriting your git configs for that url.

@yoh what do you think about that idea?

There are some fairly related todos that I have been thinking about recently.

• extraheader config wants http.<url>.extraheader to be supported at least for P2P over HTTP urls
• Support per SITE http.SITE.sslCAInfo etc for http.<url>.sslCAInfo and http.<url>.sslCAPath.

Each of those has their own considerations around things like security and what is able to be implemented sanely.

Those are about http.<url>.*, while this is about credential.<url>.*. git has different matching rules for those. For http.<url>.* git's documentation explicitly limits it to

The URLs that are matched against are those given directly to Git com‐
mands. This means any URLs visited as a result of a redirection do not
participate in matching.

For credential.<url>.* git's documentation is less explicit about exactly which urls it applies to. (I also noticed that at least the design document technical/bundle-uri.adoc talks about using git credential for accessing bundle urls that are provided by the git server. I don't know if that is implemented in git though.)

I think it's important that whatever git-annex does about these per-url configs, it does it consistently. A small divergence from git's documented behavior would be ok as long as it's documented and doesn't open security cans of worms.

I do think that "present and foo" will work, with the right "foo". I'm just having trouble coming up with it, and it may be that it's not expressible with current preferred content syntax.

What's needed is something that only matches when there are too many copies.

"lackingcopies=groupname:0" matches when there are too many or the desired numcopies, which is not quite right.

Reproduced a problem with that filename:

joey@darkstar:~>sudo mount -t msdos -o loop,uid=1000,gid=1000,check=strict fs mnt
joey@darkstar:~>touch 'mnt/foobar 100%.bin'
touch: cannot touch 'mnt/foobar 100%.bin': Invalid argument

(Without check=strict, it does not EINVAL but ls shows that the filename created is "foobar 1.bin")

But, the '%' is not responsible for the problem really:

joey@darkstar:~>touch 'mnt/foobar 1.bin'
touch: cannot touch 'mnt/foobar 1.bin': Invalid argument

The problem is just that FAT with check=strict doesn't allow any spaces in the filename. And I think there's just nothing git-annex can do about exporting trees with filenames with spaces to filesystems that don't support spaces in filenames.

Perhaps you are using some other mount options though that have a different behavior where '%' is really responsible for the problem. If so, I think you should file a new bug report with enough details to reproduce the problem.

I tried implementing this (patch below) and found that it made objects be moved amoung nodes of the balanced group to balance, despite --rebalance not being used.

This patch actually makes it like "(present and lackingcopies=1)" but that is equvilant to "(present and not lackingcopies=0)" .. I think?

diff --git a/Limit.hs b/Limit.hs
index 11a94a5ced..7cf7e00e83 100644
--- a/Limit.hs
+++ b/Limit.hs
@@ -626,17 +626,19 @@ limitBalanced' termname fullybalanced mu want = do
        else limitCopies $ if ':' `elem` want
            then want
            else want ++ ":1"
-   let checkenoughcopies = if checklackingcopies then id else not
    let present = limitPresent mu
    let combo f = f present || f fullybalanced || f limitcopies
    let matchaction lu a i =
        let match f = matchAction f lu a i
        in ifM (Annex.getRead Annex.rebalance)
            ( match fullybalanced
-           , match present <||>
-               ((checkenoughcopies <$> match limitcopies)
-                   <&&> match fullybalanced
-               )
+           , if checklackingcopies
+               then (match present <&&> match limitcopies)
+                   <||> match fullybalanced
+               else match present <||>
+                   ((not <$> match limitcopies)
+                       <&&> match fullybalanced
+                   )
            )
    Right $ MatchFiles
        { matchAction = matchaction

I ran into this error during the export of a view to FAT32. At least some of the problems were caused by how git-annex constructed the view's filenames (or temporary transfer filenames), I believe. The filenames are constructed from path elements, which may get truncated, and so the final filenames could end up with trailing spaces. Upgrading to the current version has fixed almost all of those errors.

What appears to continue to be a problem are filenames with percentage signs in them, e.g., foobar 100%.bin. As far as the filesystem is concerned, % characters themselves are obviously fine, because git-annex uses them for all view branch filenames, and the export generally works for them. (For now, I've renamed files to, e.g., foo 100 percent.bin, and it works 100% now. )