Compiling Elixir+Mix projects that use Rustler with Nix

Some background

There’s this really cool project called Rustler which lets you easily run Rust code inside Erlang/Elixir. Erlang has support for something called Native Implemented Functions , which means that it can load a shared library and make some calls to code from it, all within Erlang code. Elixir inherits this capability.

Rustler makes it easy to build and use a shared library following the structure that Erlang’s NIF expects. To do so, it provides some Rust code to help build the shared library. It also provides some Erlang/Elixir code to help load and use the shared library.

I’ll stop referring to Erlang/Elixir now, because what follows comes from me trying to do things within Elixir and Mix. It might be different for Erlang.

By default, the Elixir code provided by Rustler builds the Rust code. This means that as the Elixir code is being compiled, when it gets to the Rustler-specific bits, it will trigger a build of the Rust code. This is pretty cool behaviour for development, but is a bit of a nightmare scenario when doing things within Nix.

Nix sandboxes builds, which means that unless you do some magic with nativeBuildInputs and other things, it will fail to compile your Elixir code because it can’t find cargo. Worse, even if you do make it find cargo, the default cargo behaviour will be to download dependencies of your code, which will fail within Nix since the sandbox disables network access by default.

Making Rustler work in Nix

Luckily, Rustler’s Elixir code let you control its Rust-compiling behaviour by preventing any compilation from happening at all.

With Nix, it would be preferrable to build the Rust code separately, and then provide the shared library from the Rust code to the Elixir code. This becomes relevant when you want to have more control over how the Rust code is built, by changing the target to musl or something like that.

To accomplish this, I’m assuming you already have the following set up:

• A derivation that builds your Elixir code without any Rustler/native bits at all.
• A derivation that builds your Rust code.

This is roughly how I had these things in my project (a few details omitted to focus on the important bits). My Elixir derivation uses mixRelease :

# This comes from mix2nix.
mixNixDeps = import ./mix_deps.nix { inherit beamPackages lib; };

elixir_project = beamPackages.mixRelease {
  inherit src pname version mixNixDeps;
};

And my Rust derivation uses crane, compiling for a musl target:

rustToolchain = pkgs.rust-bin.stable.latest.default.override {
  targets = [ "x86_64-unknown-linux-musl" ];
};
craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;

commonArgs = {
  inherit src;
  CARGO_BUILD_TARGET = "x86_64-unknown-linux-musl";
  CARGO_BUILD_RUSTFLAGS = "-C target-feature=-crt-static";
};

cargoArtifacts = craneLib.buildDepsOnly commonArgs;
myCrate = craneLib.buildPackage (commonArgs // {
  inherit cargoArtifacts;
});

For this post, I’ll assume the name of the crate is mycrate, which means it will generate a shared library called libmycrate.so.

To support the pattern of “use default Rustler behaviour during development” and “use Nix tooling during deployment”, I had to come up with different ways to use the Rustler-specific Elixir code based on whether I was developing or deploying with Nix.

There might be a better way to do this, but I used a macro like the following:

defmodule MyRustlerMacro do
  defmacro rustler_use() do
    use_precompiled = System.get_env("PRECOMPILED_NIF", "false")

    cond do
      use_precompiled == "true" or use_precompiled == "1" ->
        quote do
          use Rustler,
            otp_app: :my_app,
            skip_compilation?: true,
            load_from: {:my_app, "priv/native/libmycrate"}
        end

      true ->
        quote do
          use Rustler,
            otp_app: :my_app,
            crate: "mycrate",
            path: "../mycrate",
            target: "x86_64-unknown-linux-musl"
        end
    end
  end
end

And then in the module that has the functions to be replaced by Rustler:

defmodule MyRustler do
  require MyRustlerMacro
  MyRustlerMacro.rustler_use()

  def function_from_rustler1, do: :erlang.nif_error(:nif_not_loaded)
  def function_from_rustler2, do: :erlang.nif_error(:nif_not_loaded)
end

With the Elixir code properly configured, some extra bits are necessary in the Elixir derivation, because we’re not telling it about the Rust code at all yet. Let’s fix that:

elixir_project = beamPackages.mixRelease {
  inherit src pname version mixNixDeps;

  PRECOMPILED_NIF = true;

  preBuild = ''
    mkdir -p priv/native
    cp ${myCrate}/lib/libmycrate.so priv/native
  '';
};

With this done, the Elixir code should properly build and run now.

Bonuses

• Switching compile-time behaviour based on an environment variable like that is a Bad Idea™ in most cases, because Mix won’t recompile your code if the only thing that changed was the value of the environment variable. This works here because it’s assumed we’ll never set that environment variable during development, and Nix will force the derivation to be rebuilt if the environment variable changes.

  Since Nix always builds things in sandboxes, it won’t ever reuse previous artifacts from the Elixir compilation, so Elixir will always compile everything again should the environment variable change. This shouldn’t be an issue because within Nix we always want that environment variable to be set (otherwise the build won’t work at all!).

• It might not be necessary to copy the shared library in the preBuild phase. I only do this because, as I mentioned, I’m using Phoenix, which will end up packing the entire priv directory in the outputs for me. From what I could gather, if the shared library doesn’t exist during the Elixir compilation, there’ll be a warning emitted, but it’ll still proceed. The shared library only needs to exist when the Elixir project runs. You probably can get away with copying the shared library at some later phase in the derivation if you want/need to. This is all a guess though, because I haven’t explored it.