Please describe the problem.
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) git annex initremote s3-origin type=S3 importtree=yes encryption=none autoenable=true bucket=dandiarchive fileprefix=zarr-checksums/2ac71edb-738c-40ac-bd8c-8ca985adaa12/
initremote s3-origin (checking bucket...) (creating bucket in US...)
git-annex: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidRequest", s3ErrorMessage = "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.", s3ErrorResource = Nothing, s3ErrorHostId = Just "????=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
failed
initremote: 1 failed
(dandisets) dandi@drogon:/mnt/backup/dandi/tmp/dandisets/test-importtree-s3$ git annex version
git-annex version: 10.20220525-gf1fdc90
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.7 http-client-0.7.9 persistent-sqlite-2.13.0.3 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.1.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
what exactly does it want from me for AWS4-HMAC-SHA256 ? here is how those secrets look like (after slight obfuscation):
$ echo AWS_ACCESS_KEY_ID=$(awk '/^access_key/{print $3}' ~/.s3cfg-dandi-backup) AWS_SECRET_ACCESS_KEY=$(awk '/^secret_key/{print $3}' ~/.s3cfg-dandi-backup) | tr 'a-z0-9' '?-?'
AWS_ACCESS_KEY_ID=AKIA?GIMZPVVE??Y?NKH AWS_SECRET_ACCESS_KEY=/??U??TV?LH?L???KZJ??RF?G???Y+?S??????LM
That seems to be Signature Version 4. Try adding this to the initremote: signature=v4
Some AWS regions need V4, others still work with V2, which is what it used by default. I have not seen the default US region require V4 before. Could be something about your AWS access key that needs v4?
signature=v4brought a different error which mentioneds3ErrorMessage = "The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'"region=us-east-2while running older10.20241031-1~ndall+1version of git-annex seems resulted in no change10.20250721-g8867e7590a3a70afa8a93d2fefab94adc9a176d0installed from pypi seems took that into account and workedand then
importworked.I can replicate that behavior here with the command given, leaving off "region=us-east-2", which is handy for testing (needs AWS credentials in the environment).
It would be possible for git-annex to use GetBucketLocation to determine the location of an existing bucket, rather than requiring it to be manually specified. I don't see much downside in making it do that. It would of course fail for a new bucket, and it might be that some existing buckets are not configured to allow that API call. Or another S3 server might not support it. But as long as it falls back to current behavior it seems it would only avoid a problem.
As for determining which signature version to use, the problem is that a S3 implementation might not support signature v4 yet. So using it by default could break existing setups. It seems likely that there are plenty of third party S3 servers out there that only support v2. According to AWS documentation, every region supports v4 now, while some regions do not support v2. So git-annex could have a special case for AWS to use v4 by default.
While AWS tried and failed to deprecate v2 entirely around 2019, they ended up keeping it for those regions. That could still change I suppose, and so it might be good for git-annex to proactively use v4 for all AWS buckets.
Made signature=v4 be used by default for AWS endpoints. Note that, if you want a special remote to be able to be used by an older version of git-annex, it would still make sense to explicitly specify signature=v4.
I got GetBucketLocation to work, although only when git-annex is built with aws-0.23 or newer.