8.9 KiB
| title | date | tags | ||
|---|---|---|---|---|
| Building a Crystal Project with Nix, Revisited | 2020-04-26T18:37:22-07:00 |
|
As I've described in my [previous post]({{< relref "crystal_nix.md" >}}), the process for compiling a Crystal project with Nix is a fairly straightforward one. As is standard within the Nix ecosystem, the project's dependencies, as specified by the source language's build system (shards, in Crystal's case), are converted into a Nix expression (shards.nix). These dependencies are then used in a derivation, which, in Crystal's case, can take advantage of buildCrystalPackage to reduce boilerplate build scripts. All is well.
Things start to fall apart a little bit when the Crystal project being built is more complex. The predefined infrastructure (like buildCrystalPackage)
{{< sidenote "right" "versatility-note" "is not written with versatility in mind," >}}
This is not a bad thing at all; it's much better to get something working for the practical case, rather than concoct an overcomplicated solution that covers all theoretically possible cases.
{{< /sidenote >}} though it seems to work exceptionally in the common case. Additionally, I discovered that the compiler itself has some quirks, and have killed a few hours of my time trying to figure out some unexpected behaviors.
This post will cover the extra, more obscure steps I had to take to build an HTTPS-enabled Crystal project.
First Problem: Git-Based Dependencies
A lot of my projects use Crystal libraries that are not hosted on GitHub at all; I use a private Git server, and most of my non-public code resides on it. The Crystal people within Nix don't seem to like this: let's look at the code for crystal2nix.cr file in the nixpkgs repository. In particular, consider lines 18 and 19:
18
19
yaml.shards.each do |key, value|
owner, repo = value["github"].split("/")
Ouch! If you as much as mention a non-GitHub repository in your shards.lock file, you will experience a good old uncaught exception. Things don't end there, either. Nix provides a convenient fetchFromGitHub function, which only requires a repository name and its enclosing namespace (user or group). crystal2nix uses this, by generating a file with that information:
34
35
36
37
38
39
file.puts %( #{key} = {)
file.puts %( owner = "#{owner}";)
file.puts %( repo = "#{repo}";)
file.puts %( rev = "#{rev}";)
file.puts %( sha256 = "#{sha256}";)
file.puts %( };)
And, of course, build-package.nix (of which this is the version at the time of writing) uses this to declare dependencies:
26
27
28
29
crystalLib = linkFarm "crystal-lib" (lib.mapAttrsToList (name: value: {
inherit name;
path = fetchFromGitHub value;
}) (import shardsFile));
This effectively creates a folder of dependencies cloned from GitHub, which is then placed into lib as if shards was run:
37
38
39
40
41
configurePhase = args.configurePhase or lib.concatStringsSep "\n" ([
"runHook preConfigure"
] ++ lib.optional (lockFile != null) "ln -s ${lockFile} ./shard.lock"
++ lib.optional (shardsFile != null) "ln -s ${crystalLib} lib"
++ [ "runHook postConfigure "]);
Sleek, except that there's no place in this flow for dependencies based only on Git! crystalLib is declared locally in a let/in expression, and we don't have access to it; neither can we call linkFarm again, since this results in a derivation, which, with different inputs, will be created at a different path. To work around this, I made my own Nix package, called customCrystal, and had it pass several modifications to buildCrystalPackage:
{ stdenv, lib, linkFarm, fetchgit, fetchFromGitHub }:
{ crystal,
gitShardsFile ? null,
lockFile ? null,
shardsFile ? null, ...}@args:
let
buildArgs = builtins.removeAttrs args [ "crystal" ];
githubLinks = lib.mapAttrsToList (name: value: {
inherit name;
path = fetchFromGitHub value;
}) (import shardsFile);
gitLinks = lib.mapAttrsToList (name: value: {
inherit name;
path = fetchgit { inherit (value) url rev sha256; };
}) (import gitShardsFile);
crystalLib = linkFarm "crystal-lib" (githubLinks ++ gitLinks);
configurePhase = args.configurePhase or lib.concatStringsSep "\n" ([
"runHook preConfigure"
] ++ lib.optional (lockFile != null) "ln -s ${lockFile} ./shard.lock"
++ lib.optional (shardsFile != null) "ln -s ${crystalLib} lib"
++ [ "runHook postConfigure "]);
in
crystal.buildCrystalPackage (buildArgs // { inherit configurePhase; })
This does pretty much the equivalent of what buildCrystalPackage does (indeed, it does the heavy lifting). However, this snippet also retrieves Git repositories from the gitShardsFile, and creates the lib folder using both Git and GitHub dependencies. I didn't bother writing a crystal2nix equivalent for this, since I only had a couple of dependencies. I invoked my new function like buildCrystalPackage, with the addition of passing in the Crystal package, and that problem was solved.
Second Problem: OpenSSL
The package I was trying to build used Crystal's built-in HTTP client, which, in turn, required OpenSSL. This, I thought, would be rather straightforward: add openssl to my package's buildInputs, and be done with it. It was not as simple, though, and I was greeted with a wall of errors like this one:
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: _main.o: in function `__crystal_main':
main_module:(.text+0x6f0): undefined reference to `SSL_library_init'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6f5): undefined reference to `SSL_load_error_strings'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6fa): undefined reference to `OPENSSL_add_all_algorithms_noconf'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6ff): undefined reference to `ERR_load_crypto_strings'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: _main.o: in function `*HTTP::Client::new<String, (Int32 | Nil), Bool>:HTTP::Client':
Some snooping led me to discover that these symbols were part of OpenSSL 1.0.2, support for which ended in 2019. OpenSSL 1.1.0 has these symbols deprecated, and from what I can tell, they might be missing from the .so file altogether. I tried changing the package to specifically accept OpenSSL 1.0.2, but that didn't work, either: for some reason, the Crystal kept running the gcc command with -L...openssl-1.1.0. It also seemed like the compiler itself was built against the most recent version of OpenSSL, so what's the issue? I discovered this is a problem in the compiler itself. Consider the following line from Crystal's openssl/lib_ssl.cr source file:
8
{% ssl_version = `hash pkg-config 2> /dev/null && pkg-config --silence-errors --modversion libssl || printf %s 0.0.0`.split.last.gsub(/[^0-9.]/, "") %}
Excuse me? If pkg-config is not found (which, in Nix, it won't be by default), Crystal assumes that it's using the least up-to-date version of OpenSSL,
{{< sidenote "right" "version-note" "indicated by version code 0.0.0." >}}
The Crystal compiler compares version numbers based on semantic versioning, it seems, and 0.0.0 will always compare to be less than any other version of OpenSSL. Thus, code 0.0.0 indicates that Crystal should assume it's dealing with an extremely old version of OpenSSL.
{{< /sidenote >}} This matters, because later on in the file, we get this beauty:
215
216
217
218
219
220
221
{% if compare_versions(OPENSSL_VERSION, "1.1.0") >= 0 %}
fun tls_method = TLS_method : SSLMethod
{% else %}
fun ssl_library_init = SSL_library_init
fun ssl_load_error_strings = SSL_load_error_strings
fun sslv23_method = SSLv23_method : SSLMethod
{% end %}
That would be where the linker errors are coming from. Adding pkg-configto buildInputs along with openssl fixes the issue, and my package builds without problems.
Conclusion
Crystal is a rather obscure language, and Nix is a rather obscure build system. I'm grateful that the infrastructure I'm using exists, and that using it is as streamlined as it is. There is, however, always room for improvement. If I have time, I will be opening pull requests for the crystal2nix tool on GitHub (to allow Git-based repositories), and perhaps on the Crystal compiler as well (to try figure out what to do about pkg-config). If someone else wants to do it themselves, I'd be happy to hear how it goes! Otherwise, I hope you found this post useful.