forum/s3 vs ssh Performance Problemsgit-annexhttp://git-annex.branchable.com/forum/s3_vs_ssh_Performance_Problems/git-annexikiwiki2013-12-12T20:14:02Zcomment 1http://git-annex.branchable.com/forum/s3_vs_ssh_Performance_Problems/comment_1_65f064f09d7850abecab97007b0d30f0/Justin2013-11-27T22:47:37Z2013-11-21T04:09:15Z
<p>Is</p>
<pre><code>git annex copy --not --in mys3 --to mys3 .
</code></pre>
<p>any faster?</p>
comment 2http://git-annex.branchable.com/forum/s3_vs_ssh_Performance_Problems/comment_2_baaf2384d9196077268e9ca9bbe3b871/Hamza2013-11-27T22:47:37Z2013-11-21T17:47:57Z
No difference. Both take roughly the same time but I did time a couple of runs with both commands they usually take 3 to 5 minutes to complete. Maybe git did something behind the scenes like gc? But still slower than it used to be. My other repo (one with 42k files) still takes a 2 hours.
comment 3http://git-annex.branchable.com/forum/s3_vs_ssh_Performance_Problems/comment_3_dc44be42070c073d150c476406e9b425/Justin2013-11-27T22:47:37Z2013-11-22T20:21:11Z
<p>Well it's not related to s3... that copy command won't even do any network traffic if there is nothing to copy. I have a similarly configured annex with 4500 files and that command takes 10 seconds to run.</p>
<p>I do remember there being a recent fix that reduced the algorithmic complexity of an operation, but I forget which.</p>
comment 4http://git-annex.branchable.com/forum/s3_vs_ssh_Performance_Problems/comment_4_f9c3ef3b1b44bfb29125acb6ec621f38/joeyh.name2013-12-12T20:14:02Z2013-12-12T20:14:01Z
<p>You mentioned something about high memory usage when copying. How much memory are we talking about?</p>
<p>Have you run <code>git annex forget</code> in this repository before? It kind of sounds like you have, and it might be possible that it's repeatedly trying to forget old history for some reason.</p>