Web-Projects
/
blog-static
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add article about Crystal and Nix with OpenSSL.
continuous-integration/drone/push Build is failing Details

This commit is contained in:
Danila Fedorin 2020-04-26 19:56:15 -07:00
parent eda9bbb191
commit a441280812
1 changed files with 112 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

112
content/blog/crystal_nix_revisited.md Normal file
View File

@ -0,0 +1,112 @@
---
title: Building a Crystal Project with Nix, Revisited
date: 2020-04-26T18:37:22-07:00
tags: ["Crystal", "Nix"]
---
As I've described in my [previous post]({{< relref "crystal_nix.md" >}}), the process for compiling a Crystal project with Nix is a fairly straightforward one. As is standard within the Nix ecosystem, the project's dependencies, as specified by the source language's build system (shards, in Crystal's case), are converted into a Nix expression (`shards.nix`). These dependencies are then used in a derivation, which, in Crystal's case, can take advantage of `buildCrystalPackage` to reduce boilerplate build scripts. All is well.
Things start to fall apart a little bit when the Crystal project being built is more complex. The predefined infrastructure (like `buildCrystalPackage`)
{{< sidenote "right" "versatility-note" "is not written with versatility in mind," >}}
This is not a bad thing at all; it's much better to get something working for the practical case, rather than concoct an overcomplicated solution that covers all theoretically possible cases.
{{< /sidenote >}} though it seems to work exceptionally in the common case. Additionally, I discovered that the compiler itself has some quirks, and have killed a few hours of my time trying to figure out some unexpected behaviors.
This post will cover the extra, more obscure steps I had to take to build an HTTPS-enabled Crystal project.
### First Problem: Git-Based Dependencies
A lot of my projects use Crystal libraries that are not hosted on GitHub at all; I use a private Git server, and most of my non-public code resides on it. The Crystal people within Nix don't seem to like this: let's look at the code for `crystal2nix.cr` file in the [nixpkgs repository](https://github.com/NixOS/nixpkgs/blob/1ffdf01777360f548cc7c10ef5b168cbe78fd183/pkgs/development/compilers/crystal/crystal2nix.cr). In particular, consider lines 18 and 19:
```Crystal {linenos=table,linenostart=18}
yaml.shards.each do |key, value|
owner, repo = value["github"].split("/")
```
Ouch! If you as much as mention a non-GitHub repository in your `shards.lock` file, you will experience a good old uncaught exception. Things don't end there, either. Nix provides a convenient `fetchFromGitHub` function, which only requires a repository name and its enclosing namespace (user or group). `crystal2nix` uses this, by generating a file with that information:
```Crystal {linenos=table,linenostart=34}
file.puts %( #{key} = {)
file.puts %( owner = "#{owner}";)
file.puts %( repo = "#{repo}";)
file.puts %( rev = "#{rev}";)
file.puts %( sha256 = "#{sha256}";)
file.puts %( };)
```
And, of course, `build-package.nix` (of which [this is the version at the time of writing](https://github.com/NixOS/nixpkgs/blob/912eb6b120eba15237ff053eafc4b5d90577685b/pkgs/development/compilers/crystal/build-package.nix)) uses this to declare dependencies:
```Nix {linenos=table,linenostart=26}
crystalLib = linkFarm "crystal-lib" (lib.mapAttrsToList (name: value: {
inherit name;
path = fetchFromGitHub value;
}) (import shardsFile));
```
This effectively creates a folder of dependencies cloned from GitHub, which is then placed into `lib` as if `shards` was run:
```Nix {linenos=table,linenostart=37}
configurePhase = args.configurePhase or lib.concatStringsSep "\n" ([
"runHook preConfigure"
] ++ lib.optional (lockFile != null) "ln -s ${lockFile} ./shard.lock"
++ lib.optional (shardsFile != null) "ln -s ${crystalLib} lib"
++ [ "runHook postConfigure "]);
```
Sleek, except that there's no place in this flow for dependencies based _only_ on Git! `crystalLib` is declared locally in a `let/in` expression, and we don't have access to it; neither can we call `linkFarm` again, since this results in a derivation, which, with different inputs, will be created at a different path. To work around this, I made my own Nix package, called `customCrystal`, and had it pass several modifications to `buildCrystalPackage`:
```Nix
{ stdenv, lib, linkFarm, fetchgit, fetchFromGitHub }:
{ crystal,
gitShardsFile ? null,
lockFile ? null,
shardsFile ? null, ...}@args:
let
buildArgs = builtins.removeAttrs args [ "crystal" ];
githubLinks = lib.mapAttrsToList (name: value: {
inherit name;
path = fetchFromGitHub value;
}) (import shardsFile);
gitLinks = lib.mapAttrsToList (name: value: {
inherit name;
path = fetchgit { inherit (value) url rev sha256; };
}) (import gitShardsFile);
crystalLib = linkFarm "crystal-lib" (githubLinks ++ gitLinks);
configurePhase = args.configurePhase or lib.concatStringsSep "\n" ([
"runHook preConfigure"
] ++ lib.optional (lockFile != null) "ln -s ${lockFile} ./shard.lock"
++ lib.optional (shardsFile != null) "ln -s ${crystalLib} lib"
++ [ "runHook postConfigure "]);
in
crystal.buildCrystalPackage (buildArgs // { inherit configurePhase; })
```
This does pretty much the equivalent of what `buildCrystalPackage` does (indeed, it does the heavy lifting). However, this snippet also retrieves Git repositories from the `gitShardsFile`, and creates the `lib` folder using both Git and GitHub dependencies. I didn't bother writing a `crystal2nix` equivalent for this, since I only had a couple of dependencies. I invoked my new function like `buildCrystalPackage`, with the addition of passing in the Crystal package, and that problem was solved.
### Second Problem: OpenSSL
The package I was trying to build used Crystal's built-in HTTP client, which, in turn, required OpenSSL. This, I thought, would be rather straightforward: add `openssl` to my package's `buildInputs`, and be done with it. It was not as simple, though, and I was greeted with a wall of errors like this one:
```
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: _main.o: in function `__crystal_main':
main_module:(.text+0x6f0): undefined reference to `SSL_library_init'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6f5): undefined reference to `SSL_load_error_strings'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6fa): undefined reference to `OPENSSL_add_all_algorithms_noconf'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: main_module:(.text+0x6ff): undefined reference to `ERR_load_crypto_strings'
/nix/store/sq2b0dqlq243mqn4ql5h36xmpplyy20k-binutils-2.31.1/bin/ld: _main.o: in function `*HTTP::Client::new<String, (Int32 | Nil), Bool>:HTTP::Client':
```
Some snooping led me to discover that these symbols were part of OpenSSL 1.0.2, support for which ended in 2019. OpenSSL 1.1.0 has these symbols deprecated, and from what I can tell, they might be missing from the `.so` file altogether. I tried changing the package to specifically accept OpenSSL 1.0.2, but that didn't work, either: for some reason, the Crystal kept running the `gcc` command with `-L...openssl-1.1.0`. It also seemed like the compiler itself was built against the most recent version of OpenSSL, so what's the issue? I discovered this is a problem in the compiler itself. Consider the following line from Crystal's `openssl/lib_ssl.cr` [source file](https://github.com/crystal-lang/crystal/blob/0.34.0/src/openssl/lib_ssl.cr):
```Crystal {linenos=table,linenostart=8}
{% ssl_version = `hash pkg-config 2> /dev/null && pkg-config --silence-errors --modversion libssl || printf %s 0.0.0`.split.last.gsub(/[^0-9.]/, "") %}
```
Excuse me? If `pkg-config` is not found (which, in Nix, it won't be by default), Crystal assumes that it's using the _least_ up-to-date version of OpenSSL,
{{< sidenote "right" "version-note" "indicated by version code 0.0.0." >}}
The Crystal compiler compares version numbers based on semantic versioning, it seems, and 0.0.0 will always compare to be less than any other version of OpenSSL. Thus, code 0.0.0 indicates that Crystal should assume it's dealing with an extremely old version of OpenSSL.
{{< /sidenote >}} This matters, because later on in the file, we get this beauty:
```Crystal {linenos=table,linenostart=215}
{% if compare_versions(OPENSSL_VERSION, "1.1.0") >= 0 %}
fun tls_method = TLS_method : SSLMethod
{% else %}
fun ssl_library_init = SSL_library_init
fun ssl_load_error_strings = SSL_load_error_strings
fun sslv23_method = SSLv23_method : SSLMethod
{% end %}
```
That would be where the linker errors are coming from. Adding `pkg-config`to `buildInputs` along with `openssl` fixes the issue, and my package builds without problems.
### Conclusion
Crystal is a rather obscure language, and Nix is a rather obscure build system. I'm grateful that the infrastructure I'm using exists, and that using it is as streamlined as it is. There is, however, always room for improvement. If I have time, I will be opening pull requests for the `crystal2nix` tool on GitHub (to allow Git-based repositories), and perhaps on the Crystal compiler as well (to try figure out what to do about `pkg-config`). If someone else wants to do it themselves, I'd be happy to hear how it goes! Otherwise, I hope you found this post useful.