forum/building the standalone distribution from sourcegit-annexhttp://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/git-annexikiwiki2020-08-28T01:02:48Zcomment 1http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_1_204eda1dc4b935ee357a6675ee14317b/joey2020-08-10T19:38:51Z2020-08-10T19:08:20Z
<p>I've added some docs to the page about how to build it.</p>
<p>BUILDER=stack will build it with stack.</p>
full source tarballs for specific releases?http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_2_ff968f9f9d6644b9e8607c8e76f45c1e/Ilya_Shlyakhter2020-08-10T21:14:04Z2020-08-10T21:14:04Z
Thanks a lot! Do I understand correctly that this can't be done from the <a href="https://hackage.haskell.org/package/git-annex">tarballs on Hackage</a>, and requires <a href="http://git-annex.branchable.com/install/fromsource/">checking out the full source tree</a>? (And that there is URL from which the full source repo tarball for a given released version can be downloaded?)
comment 3http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_3_e94f70e19b77c9ad6b6349fa65dcc60f/Ilya_Shlyakhter2020-08-10T21:52:30Z2020-08-10T21:52:29Z
(I meant, there is <em>no</em> URL from which the full source repo tarball for a given released version can be downloaded?)
comment 4http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_4_0b7824acaf031d8ba69700674c82f472/joey2020-08-10T23:22:17Z2020-08-10T22:43:19Z
<p>Right, the hackage tarball is only useful for cabal, it would not be
practical to include everything else in it. Arguably there's small enough
value in publishing it there that I ought to just drop it.</p>
<p>I don't release other tarballs, but it is possible to get a stable tarball
from git, eg
<a href="https://git.joeyh.name/index.cgi/git-annex.git/snapshot/git-annex-8.20200720.1.tar.gz">https://git.joeyh.name/index.cgi/git-annex.git/snapshot/git-annex-8.20200720.1.tar.gz</a></p>
please keep the hackage tarballshttp://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_5_bfec6091cc312a9d4c87163b845454f6/Ilya_Shlyakhter2020-08-11T00:43:47Z2020-08-11T00:43:46Z
" Arguably there's small enough value in publishing it there that I ought to just drop it" -- the <a href="https://github.com/conda-forge/git-annex-feedstock/blob/master/recipe/meta.yaml#L48">conda-forge recipe</a> is based on the hackage tarballs, as is the <a href="https://github.com/Homebrew/homebrew-core/blob/master/Formula/git-annex.rb#L4">homebrew formula</a>.
Great tip about conda.http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_6_044da90b6cc18534de74ee5d38d6becb/cardoso-neto2020-08-11T13:22:58Z2020-08-11T13:22:58Z
<p>I had no idea git-annex was on a conda channel even though it is clearly stated on the <a href="https://git-annex.branchable.com/install/">install page</a>.</p>
<p>I just installed it with <code>conda install git-annex -c conda-forge</code> and, to my amazement, instead of <code>v8.20200226</code> (which was available on my focal fossa standard apt repos) being installed, <code>v8.20200810-g693f2093a</code> was installed.</p>
<p>It was so easy to install and it is a version that was built just yesterday. I loved it.</p>
<p>Please keep the plumbing that made this possible alive.</p>
checking the conda-forge recipehttp://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_7_5187f5f40bc47b5b079cfdd3eb5f1fc3/Ilya_Shlyakhter2020-08-11T15:51:28Z2020-08-11T15:51:27Z
<p>@cardoso-neto "v8.20200810-g693f2093a was installed" -- for the purpose of checking that the conda-forge installation worked correctly, can you tell me which exact package got installed, i.e. what is output by running</p>
<p><code>conda list --explicit | grep git-annex</code></p>
<p>?
Thanks!</p>
comment 8http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_8_eb0c565043164c17b2a6d2c6361b6e35/cardoso-neto2020-08-11T19:57:03Z2020-08-11T19:57:03Z
<p>This was the tarball conda used:</p>
<p><code>https://conda.anaconda.org/conda-forge/linux-64/git-annex-8.20200810-nodep_h1234567_0.tar.bz2</code></p>
<p>I noticed you updated the (/install/conda)[https://git-annex.branchable.com/install/conda/] with possible package conflict tips.</p>
<p>The only dependency that was installed when I conda-installed git-annex was this:</p>
<p><code>python_abi conda-forge/linux-64::python_abi-3.8-1_cp38</code></p>
<p>Which struck me as odd, because I'm running python 3.8.5, not 3.8.1.
Maybe they just have perfectly compatible ABIs, or maybe it's related to the problem you mentioned.
I don't know.</p>
<p>If there's any other test you want me to run, no problem.</p>
strange conda install behaviorhttp://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_9_edb3a79772f81dd3ffd00b81e79f67e5/Ilya_Shlyakhter2020-08-11T20:19:47Z2020-08-11T20:19:46Z
<p>What happens if you request the installation of the standard package with</p>
<p><code>conda install -c conda-forge git-annex=*=alldep*</code></p>
<p>?</p>
comment 10http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_10_ae23bbd2da15e8ae115b4030994502cc/cardoso-neto2020-08-11T22:10:25Z2020-08-11T22:10:25Z
<p>After running the command you request this was the package plan conda came up with:</p>
<pre><code> added / updated specs:
- git-annex[build=alldep*]
The following packages will be downloaded:
package | build
---------------------------|-----------------
c-ares-1.16.1 | h516909a_0 108 KB conda-forge
curl-7.71.1 | he644dc0_4 139 KB conda-forge
dbus-1.13.6 | he372182_0 602 KB conda-forge
expat-2.2.9 | he1b5a44_2 191 KB conda-forge
gettext-0.19.8.1 | hc5be6a0_1002 3.6 MB conda-forge
git-2.28.0 | pl526h5e3e691_1 15.9 MB conda-forge
git-annex-8.20200810 |alldep_h988281c_100 12.2 MB conda-forge
glib-2.65.0 | h6f030ca_0 3.5 MB conda-forge
gmp-6.2.0 | he1b5a44_2 811 KB conda-forge
krb5-1.17.1 | hfafb76e_2 1.5 MB conda-forge
ld_impl_linux-64-2.34 | hc38a660_9 612 KB conda-forge
libcurl-7.71.1 | hcdd3856_4 312 KB conda-forge
libev-4.33 | h516909a_0 105 KB conda-forge
libffi-3.2.1 | he1b5a44_1007 47 KB conda-forge
libiconv-1.15 | h516909a_1006 2.0 MB conda-forge
libmagic-5.39 | hed695b0_0 280 KB conda-forge
libnghttp2-1.41.0 | hab1572f_1 709 KB conda-forge
libssh2-1.9.0 | hab1572f_5 225 KB conda-forge
lsof-4.89 | 0 84 KB conda-forge
openssh-8.3p1 | h5957347_0 1.3 MB conda-forge
pcre-8.44 | he1b5a44_0 261 KB conda-forge
perl-5.26.2 | h516909a_1006 15.4 MB conda-forge
popt-1.16 | h299ea2f_2002 184 KB conda-forge
python-3.8.5 |h4d41432_2_cpython 71.0 MB conda-forge
rsync-3.1.3 | hed695b0_1002 287 KB conda-forge
------------------------------------------------------------
Total: 131.4 MB
The following NEW packages will be INSTALLED:
c-ares conda-forge/linux-64::c-ares-1.16.1-h516909a_0
curl conda-forge/linux-64::curl-7.71.1-he644dc0_4
dbus conda-forge/linux-64::dbus-1.13.6-he372182_0
expat conda-forge/linux-64::expat-2.2.9-he1b5a44_2
gettext conda-forge/linux-64::gettext-0.19.8.1-hc5be6a0_1002
git conda-forge/linux-64::git-2.28.0-pl526h5e3e691_1
git-annex conda-forge/linux-64::git-annex-8.20200810-alldep_h988281c_100
glib conda-forge/linux-64::glib-2.65.0-h6f030ca_0
gmp conda-forge/linux-64::gmp-6.2.0-he1b5a44_2
krb5 conda-forge/linux-64::krb5-1.17.1-hfafb76e_2
libcurl conda-forge/linux-64::libcurl-7.71.1-hcdd3856_4
libev conda-forge/linux-64::libev-4.33-h516909a_0
libiconv conda-forge/linux-64::libiconv-1.15-h516909a_1006
libmagic conda-forge/linux-64::libmagic-5.39-hed695b0_0
libnghttp2 conda-forge/linux-64::libnghttp2-1.41.0-hab1572f_1
libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_5
lsof conda-forge/linux-64::lsof-4.89-0
openssh conda-forge/linux-64::openssh-8.3p1-h5957347_0
pcre conda-forge/linux-64::pcre-8.44-he1b5a44_0
perl conda-forge/linux-64::perl-5.26.2-h516909a_1006
popt conda-forge/linux-64::popt-1.16-h299ea2f_2002
python_abi conda-forge/linux-64::python_abi-3.8-1_cp38
rsync conda-forge/linux-64::rsync-3.1.3-hed695b0_1002
The following packages will be UPDATED:
ld_impl_linux-64 pkgs/main::ld_impl_linux-64-2.33.1-h5~ --> conda-forge::ld_impl_linux-64-2.34-hc38a660_9
openssl pkgs/main::openssl-1.1.1g-h7b6447c_0 --> conda-forge::openssl-1.1.1g-h516909a_1
python pkgs/main::python-3.8.5-hcff3b4d_0 --> conda-forge::python-3.8.5-h4d41432_2_cpython
The following packages will be SUPERSEDED by a higher-priority channel:
ca-certificates pkgs/main::ca-certificates-2020.6.24-0 --> conda-forge::ca-certificates-2020.6.20-hecda079_0
certifi pkgs/main::certifi-2020.6.20-py38_0 --> conda-forge::certifi-2020.6.20-py38h32f6830_0
libffi pkgs/main::libffi-3.3-he6710b0_2 --> conda-forge::libffi-3.2.1-he1b5a44_1007
</code></pre>
<p>As a user, there were no noticeable differences.</p>
<p>After running <code>git annex version</code> on this env, I noticed I got slightly older dependencies compared to the normal installation (<code>conda install -c conda-forge git-annex</code>).</p>
<pre><code>$ diff git-annex git-annex=*=alldep*
-git-annex version: 8.20200810-g693f2093a
+git-annex version: 8.20200810-g8b113bc
-dependency versions: cryptonite-0.26 feed-1.3.0.1 ghc-8.8.3 persistent-sqlite-2.10.6.2
+dependency versions: cryptonite-0.25 feed-1.2.0.1 ghc-8.6.5 persistent-sqlite-2.9.3
</code></pre>
<p>If the dependencies were as up-to-date as the normal installation, I think I'd actually prefer this method, since it keeps things more contained and independent.</p>
Having cutting edge packages with conda is not that straighforward.http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_11_5c580c8fce388b9808c2afc778da853b/cardoso-neto2020-08-14T18:55:17Z2020-08-14T18:55:17Z
<p>I am always inclined to "contain" my dependencies inside separate environments, so I want to do what you recommended and use <code>git-annex=*=alldep*</code>.
But when I install it, I get outdated versions of really important programs (although only slightly outdated).</p>
<pre><code>perl; 5.26.2; 5.30.0
git; 2.23.0; 2.28.0
</code></pre>
<p>I also got some downgrades of stuff that was already inside the env:</p>
<pre><code>The following packages will be SUPERSEDED by a higher-priority channel:
ca-certificates pkgs/main::ca-certificates-2020.6.24-0 --> conda-forge::ca-certificates-2020.6.20-hecda079_0
libffi pkgs/main::libffi-3.3-he6710b0_2 --> conda-forge::libffi-3.2.1-he1b5a44_1007
python pkgs/main::python-3.8.5-hcff3b4d_0 --> conda-forge::python-3.8.3-cpython_he5300dc_0
</code></pre>
<p>Some googling later and I found this <a href="https://github.com/conda-forge/git-annex-feedstock/blob/master/.ci_support/linux_64_nodepFalse.yaml">github/conda-forge/git-annex-feedstock/.ci_support/linux_64_nodepFalse.yaml</a>, which does have <code>perl: - 5.26.2</code> <em>hard-coded</em>.
However I also found the <a href="https://github.com/conda-forge/git-annex-feedstock/blob/master/.ci_support/README">README</a> saying that file was auto-generated based on this one <a href="https://github.com/conda-forge/git-annex-feedstock/blob/master/recipe/meta.yaml">feedstock/recipe/meta.yaml</a>, which furthers the mystery as this <code>meta.yaml</code> does not have any specified version for <code>perl</code> and git only has a minimum version (<code>git >=2.22</code>).
So it seems <code>conda</code> is grabbing older (than the most recent) versions, because those are the latest it has.</p>
<p><code>git 2.28.0</code> is available on <code>conda-forge</code>, though. So I'm using that now. And since <code>conda-forge</code> is overall more up-to-date...</p>
<h2>This is how I decided to create my env:</h2>
<p><code>conda create --name main --channel conda-forge python=3.8 git=2.28 git-annex=*=alldep*</code></p>
<p>Still no idea on how to update the <em>outdated</em> dependencies I stated on my previous post, though. They (cryptonite, feed, ghc, and persistent-sqlite) aren't even mentioned in the <code>git-annex-feedstock</code> repo.</p>
understanding why standalone conda package gets preferredhttp://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_12_9b0a7c108855d9db63092dd5f93bc5e5/Ilya_Shlyakhter2020-08-27T15:32:06Z2020-08-27T15:32:05Z
@cardoso-neto "The only dependency that was installed when I conda-installed git-annex was this" -- do you by chance remember the exact commands used to create that conda environment, before you tried installing git-annex into it? (Or if you have the environment, can you do <code>conda list --explicit</code> in it?) I'd like to better understand why <code>conda install</code> prefers the standalone package <a href="https://anaconda.org/conda-forge/git-annex/files">so often</a>. I'll ask conda gurus, but first need a reproducible case where that happens seemingly unnecessarily.
comment 13http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_13_27f987a16d931a02f5a4ab3f80bf327a/cardoso-neto2020-08-28T00:20:30Z2020-08-28T00:20:30Z
<p>I checked my .bash_history and it was <code>conda create -n $myname python=3.8</code> the command used.</p>
<p>I just reran that and got the same result when I installed git-annex with <code>conda install -c conda-forge git-annex</code>.</p>
<p>I also noticed you made the standalone dist warning message much prettier. Thanks for that. However, you forgot the <code>--channel</code> argument in your suggested install script. Here is a copypasta of your current message:</p>
<pre><code>Executing transaction: /
##############################################################################
# #
# Standalone distribution of git-annex was installed, instead of the #
# standard distribution, likely due to package conflicts in the target #
# environment. The standalone distribution may have issues (e.g. be slower, #
# or not pass the expected environment to some external programs); #
# the standard distribution should be used when possible. #
# You can force installation of the standard version by adding =alldep* to #
# the build string, e.g. #
# conda install git-annex=*=alldep* #
# #
# For more info on the standalone git-annex distribution see #
# https://git-annex.branchable.com/install/Linux_standalone/ #
# #
##############################################################################
done
</code></pre>
comment 14http://git-annex.branchable.com/forum/building_the_standalone_distribution_from_source/comment_14_44606ec8db82f75f37aabcd7df5c24bb/Ilya_Shlyakhter2020-08-28T01:02:48Z2020-08-28T01:02:48Z
<p>Thanks for point out the channel omission in the message.</p>
<p>I was able to reproduce the issue, but only with <code>conda create --override-channels -c defaults -n new-test-env-03 python=3.8</code> . When I instead create the initial environment with <code>-c conda-forge</code>, the issue doesn't recur. I guess it makes sense: packages in conda-forge are pinned to be more compatible with each other, but not necessarily with packages in <code>defaults</code>.</p>