standalone build is known to be slower than "proper" one and one of the reasons we identified in the past was result in excessive lookups for dynamic libraries due to all the shims and LD* tuneups iirc. So we added a test for git annex init to not exceed 130 lookups for libpcre.*so here. Well, today that test failed with 131. It seems that number of those lookups was growing
was about 110 in Jan
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/01[master]git
$> git grep '> test .* -lt 130' | grep ubuntu.*Seek
cron-20220104/build-ubuntu.yaml-532-280a49ed-success/test-annex-more/6_Seek of dynlibs.txt:2022-01-04T02:59:06.2777397Z > test 109 -lt 130
cron-20220105/build-ubuntu.yaml-534-d2f63758-success/test-annex-more/6_Seek of dynlibs.txt:2022-01-05T02:59:31.3381111Z > test 110 -lt 130
...
and was in >120 in Aug:
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/08[master]git
$> git grep '> test .* -lt 130' | grep ubuntu.*Seek
cron-20220801/build-ubuntu.yaml-780-841b46fb-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-01T03:34:08.8366192Z > test 122 -lt 130
cron-20220802/build-ubuntu.yaml-781-841b46fb-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-02T03:35:33.8514595Z > test 120 -lt 130
cron-20220803/build-ubuntu.yaml-782-841b46fb-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-03T03:32:27.6511541Z > test 124 -lt 130
cron-20220804/build-ubuntu.yaml-783-841b46fb-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-04T03:27:48.0776987Z > test 120 -lt 130
cron-20220805/build-ubuntu.yaml-784-841b46fb-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-05T03:31:10.4380896Z > test 126 -lt 130
cron-20220806/build-ubuntu.yaml-785-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-06T03:18:23.3003314Z > test 128 -lt 130
cron-20220807/build-ubuntu.yaml-786-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-07T03:22:13.8922352Z > test 127 -lt 130
cron-20220808/build-ubuntu.yaml-787-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-08T03:27:25.2816456Z > test 126 -lt 130
cron-20220809/build-ubuntu.yaml-788-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-09T03:29:40.2487978Z > test 127 -lt 130
cron-20220810/build-ubuntu.yaml-789-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-10T03:10:28.9067625Z > test 126 -lt 130
cron-20220811/build-ubuntu.yaml-790-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-11T03:20:27.1936625Z > test 127 -lt 130
cron-20220812/build-ubuntu.yaml-791-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-12T03:25:42.9069031Z > test 127 -lt 130
cron-20220813/build-ubuntu.yaml-792-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-13T03:19:17.5307096Z > test 124 -lt 130
cron-20220814/build-ubuntu.yaml-793-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-14T03:28:08.1891666Z > test 128 -lt 130
cron-20220815/build-ubuntu.yaml-794-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-15T03:36:22.6735643Z > test 127 -lt 130
cron-20220816/build-ubuntu.yaml-795-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-16T03:31:06.9664034Z > test 127 -lt 130
cron-20220817/build-ubuntu.yaml-797-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-17T03:34:55.5031993Z > test 128 -lt 130
cron-20220818/build-ubuntu.yaml-798-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-18T03:34:18.7936294Z > test 128 -lt 130
cron-20220819/build-ubuntu.yaml-800-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-19T03:31:21.6241453Z > test 126 -lt 130
cron-20220820/build-ubuntu.yaml-801-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-20T03:28:08.2721924Z > test 126 -lt 130
cron-20220821/build-ubuntu.yaml-802-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-21T03:29:24.9739245Z > test 125 -lt 130
cron-20220822/build-ubuntu.yaml-803-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-22T03:34:06.7771514Z > test 126 -lt 130
cron-20220823/build-ubuntu.yaml-805-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-23T03:35:41.5831076Z > test 127 -lt 130
cron-20220824/build-ubuntu.yaml-806-41b4226d-success/test-annex-more/6_Seek of dynlibs.txt:2022-08-24T03:41:06.5345006Z > test 126 -lt 130
cron-20220825/build-ubuntu.yaml-807-41b4226d-success/test-annex-more/6_Seek of dynlibs.txt:2022-08-25T03:36:47.4347632Z > test 125 -lt 130
cron-20220826/build-ubuntu.yaml-808-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-26T03:39:23.5398288Z > test 131 -lt 130
manual-20220822/build-ubuntu.yaml-804-41b4226d-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-22T18:16:44.8256532Z > test 129 -lt 130
pr-132/build-ubuntu.yaml-796-e482990a-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-16T19:03:13.1484274Z > test 128 -lt 130
pr-55/build-ubuntu.yaml-799-32886238-failed/test-annex-more/6_Seek of dynlibs.txt:2022-08-18T18:11:38.9535820Z > test 127 -lt 130
anything to be done or should we just raise the bar to 140 in that test?
I compared current master with Nov 21, and the total number of git processes run by init increased from 29 to 34.
One is that it now runs
git config --local --listto handle a change in git's security model. (For some reason init runs that twice.)Another such change is it now runs
git config filter.annex.process.And then there's a
git update-indexthat I had to add to work around a (still unfixed) segfault in git.I think that
git-annex initis more likely than other git-annex commands to get such added calls to git, since it has to detect and work around problems. You might be better off looking at a different command. For example, the number of libpcre opens bygit-annex versionhas increased by only 1 since 2020.That one is because the dynamic linker has started looking in yet another hwcap directory,
x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/. As well as the other 5 or so it already looked in. (Forgit-annex init, that adds another 34 or so opens..) There seems to be no way to prevent the linker from doing that excess work, even though it's already settingLD_HWCAP_MASK, as was previously observed over here)Anyway, I think you should just raise the bar in your test..
I was able to eliminate one of the calls to
git config --local --listanyway..ok, did in datalad/git-annex/commit/e26581b6
Had an idea: ld-linux.so hardcodes some search paths like x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/ and x86_64-linux-gnu/x86_64/. There is no way to prevent it doing that short of a recompile. And no way to make it use a custom ld.so.cache either.
But.. If those directories are made symlinks back to x86_64-linux-gnu, then it will find the library in the first path it looks in, every time!
On my laptop, it searches x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/ first, but this will vary depending on the hardware capabilities. I made that a symlink, and that reduces the number of ENOENT by some 300 when running git-annex init. And reduces the libpcre opens back down to 35.
Implemented that idea now, so you may want to drop your test back down to 40 or 50. Or even better, make the test only look at the number of ENOENT opens of libpcre.. which should be 0 now.
So far it's only implemented for amd64, i386, and arm. Other architectures might also have such directories that could be symlinked, I have not investigated.