View performance with 7,000 files

I've imported about half of my photos into an annex on an external HDD, using metadata extensively for EXIF info, including place names.

Checking out a new view is slower than I expected, at ~6 minutes. Is this expected behavior, am I pushing the limits of file count already?

Is there anything I can do to speed things up?

% ls -1 | wc -l
    7050

% git branch -vv
  git-annex                      4e590d4 update
* master                         985ba54 add jen's phone backups
  views/Year=_;Month=_;Address=_ 795a58b refs/heads/views/Year=_;Month=_;Address=_

% /usr/bin/time -p git annex view "Year=*" "Address=*" 
view  (searching...)

Checking out files: 100% (12789/12789), done.
Switched to branch 'views/Year=_;Address=_'
ok
real       376.80
user        17.98
sys          9.66

RSS Atom

debug output

So after looking for a debug flag, I see that it's spending all its time in cat-file, which I guess is probably not a surprise:

% /usr/bin/time -p git annex view "Year=*" "Address=*" -d
view  [2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","symbolic-ref","HEAD"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","refs/heads/master"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","symbolic-ref","HEAD"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","refs/heads/master"]
(searching...)
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","ls-files","--cached","-z","--","/Volumes/Four TB Backup/Photos"]
[2014-06-16 06:29:12 PDT] feed: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","update-index","-z","--index-info"]
[2014-06-16 06:29:12 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","hash-object","-w","--stdin-paths","--no-filters"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","git-annex"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","show-ref","--hash","refs/heads/git-annex"]
[2014-06-16 06:29:12 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","log","refs/heads/git-annex..4e590d433e01886e2823c3316d18b7e3cbafe227","--oneline","-n1"]
[2014-06-16 06:29:12 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","cat-file","--batch"]
[2014-06-16 06:34:54 PDT] read: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","write-tree"]
[2014-06-16 06:34:55 PDT] chat: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","commit-tree","c0fd1ea9ccae70a7353a2e787089b88115a42b7a"]
[2014-06-16 06:34:55 PDT] call: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","update-ref","refs/heads/views/Year=_;Address=_","249c37d78e45d1fccfcb00f40337e05cb69fb64c"]

[2014-06-16 06:34:55 PDT] call: git ["--git-dir=/Volumes/Four TB Backup/Photos/.git","--work-tree=/Volumes/Four TB Backup/Photos","checkout","views/Year=_;Address=_"]
Checking out files: 100% (12789/12789), done.
Switched to branch 'views/Year=_;Address=_'
ok
real       362.14
user        18.16
sys          7.70

Comment by Michael — Mon Jun 16 14:17:06 2014

Remove comment

comment 2

View's don't scale to lots of files yet. Discussed in caching database

Comment by joeyh.name — Mon Jun 16 17:52:34 2014

Remove comment

comment 3

@joeyh, thanks - is that code all up to date here: https://github.com/joeyh/git-annex/tree/database ?

I'd be interested in helping test it.

Comment by Michael — Mon Jun 16 18:19:20 2014

Remove comment

Add a comment