forum/View performance with 7,000 files
git-annex
http://git-annex.branchable.com/forum/View_performance_with_7__44__000_files/
git-annex
ikiwiki
2014-06-16T18:19:20Z
debug output
http://git-annex.branchable.com/forum/View_performance_with_7__44__000_files/comment_1_e45ea752100d09d29efb6136a722eab3/
Michael
2014-06-16T14:17:06Z
2014-06-16T14:17:06Z
<p>So after looking for a debug flag, I see that it's spending all its time in cat-file, which I guess is probably not a surprise:</p>
<pre><code>% /usr/bin/time -p git annex view "Year=*" "Address=*" -d
view [2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","symbolic-ref","HEAD"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","refs/heads/master"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","symbolic-ref","HEAD"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","refs/heads/master"]
(searching...)
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","ls-files","--cached","-z","--","/Volumes/Four TB Backup/Photos"]
[2014-06-16 06:29:12 PDT] feed: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","update-index","-z","--index-info"]
[2014-06-16 06:29:12 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","hash-object","-w","--stdin-paths","--no-filters"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","git-annex"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","--hash","refs/heads/git-annex"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","log","refs/heads/git-annex..4e590d433e01886e2823c3316d18b7e3cbafe227","--oneline","-n1"]
[2014-06-16 06:29:12 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","cat-file","--batch"]
[2014-06-16 06:34:54 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","write-tree"]
[2014-06-16 06:34:55 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","commit-tree","c0fd1ea9ccae70a7353a2e787089b88115a42b7a"]
[2014-06-16 06:34:55 PDT] call: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","update-ref","refs/heads/views/Year=_;Address=_","249c37d78e45d1fccfcb00f40337e05cb69fb64c"]
[2014-06-16 06:34:55 PDT] call: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","checkout","views/Year=_;Address=_"]
Checking out files: 100% (12789/12789), done.
Switched to branch 'views/Year=_;Address=_'
ok
real 362.14
user 18.16
sys 7.70
</code></pre>
comment 2
http://git-annex.branchable.com/forum/View_performance_with_7__44__000_files/comment_2_b1942eed65e9b5c046095a094191a38c/
joeyh.name
2014-06-16T17:52:34Z
2014-06-16T17:52:34Z
View's don't scale to lots of files yet. Discussed in <a href="http://git-annex.branchable.com/design/caching_database/">caching database</a>
comment 3
http://git-annex.branchable.com/forum/View_performance_with_7__44__000_files/comment_3_e6e19339c9d72cf8eaae32ef4269e850/
Michael
2014-06-16T18:19:20Z
2014-06-16T18:19:20Z
<p>@joeyh, thanks - is that code all up to date here: https://github.com/joeyh/git-annex/tree/database ?</p>
<p>I'd be interested in helping test it.</p>