Declarative NixOS nspawn containers

This is a work in progress proof of concept for a simple alternative to NixOS containers with an opinionated minimal feature set.

The idea is to provide a minimal layer around systemd's existing nspawn facilities and simple networking using networkd on both sides. In contrast to standard NixOS containers, the containers are by design ephemeral. No state is being kept across restarts. Directories can be bind mounted into the container if state is explicitly needed.

Imperative containers are not in scope of this project for now. At some point, a nixos-nspawn executable could be added that does only supports networkd based default networking. This would provide a simple way to spin up containers imperatively and keep things simple without the need to retain any configuration state.

The goal is to upstream this to nixpkgs at some point. Either as a new module or a replacement for NixOS containers.

Use cases

  • Run services in different network namespaces for custom routing
  • Run multiple instances of a NixOS service on the same machine
  • Provide more isolation by default than the systemd service hardening options between services on the same machine
  • To accelerate NixOS VM tests that don't need multiple virtual machines to mock multiple nodes

How it works

The project provides a NixOS module for a host machine that create nspawn units and uses systemd's systemd-nspawn@ service to launch the containers. Only the nix store is bind mounted into the container and the nix daemon from the host is not passed into the container. User namespaces with dynamic UID/GID allocation are enabled by default.

Networking

By default, a veth link is created between the host and the container and set up with networkd's default DHCP-based configuration. Additionally, LinkLocalAddressing and MDNS is enabled by default. The networkd network units can be overridden easily to configure custom networking instead.

Operation

Most machinectl commands can be used to manage these declarative containers like start, stop,shell and other commands not involving images work as expected. Using the -M flag tools like systemctl or journalctl can access containers from the host.

Open Issues

  • the whole host nix store is being bind mounted into the container
    • explore if only needed store paths could be bind mounted instead
    • maybe create an option to make a separate nix daemon instance available in the container
  • explore how to pass credentials into the container and provide an interface

How to use this

You can consume this flake and use the provided NixOS modules. See the simple-container check in checks.nix for an example. If you are not using flakes, the NixOS modules are located in host.nix and container.nix.

Example: Simple Container

Simple container called mycontainer running a plain NixOS instance with htop installed:

# NixOS configuration of host machine
{
  # import the module on the host
  imports = [
    # with flakes
    inputs.nixos-nspawn.nixosModules.host
    # OR
    # without flakes
    "${builtins.fetchTarball "https://github.com/fpletz/nixos-nspawn/archive/main.tar.gz"}/host.nix"
  ];

  nixos-nspawn.containers = {
    mycontainer.config = { pkgs,... }: {
      environment.systemPackages = [ pkgs.htop ];
    };
  };
}

You can use machinectl shell mycontainer to access a root shell in the container and run htop.

Example: Reverse Proxy on the host for container

The following NixOS configuration creates a container host with an nginx configured to reverse proxy to a container named backend with another nginx instance.

{
  # reverse proxy on the host
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    virtualHosts."_".locations."/".proxyPass = "http://backend";
  };

  nixos-nspawn.containers = {
    backend = {
      config = {
        networking.firewall.allowedTCPPorts = [ 80 ];
        services.nginx = {
          enable = true;
          virtualHosts."backend".locations."/".return = ''200 "hack the planet"'';
        };
      };
    };
  };
}

Example: Custom network configuration

Static network configuration is also possible:

{
  nixos-nspawn.containers = {
    testcontainer = {
      config = { };
      network.veth.config = {
        # networkd network unit configs for host and container side
        host = {
          networkConfig = {
            DHCPServer = false;
            Address = [
              "fc42::1/64"
              "192.168.42.1/24"
            ];
          };
        };
        container = {
          networkConfig = {
            DHCP = false;
            Address = [
              "fc42::2/64"
              "192.168.42.2/24"
            ];
            Gateway = [
              "fc42::1"
              "192.168.42.1"
            ];
          };
        };
      };
    };
  };
}

nixos-nspawn.containers

Attribute set of containers that are configured by this module.

Type: attribute set of (submodule)

Default: { }

Example:

{
  webserver = {
    config = {
      networking.firewall.allowedTCPPorts = [ 80 ];
      services.nginx.enable = true;
    };
  };
}

nixos-nspawn.containers.<name>.autoStart

Whether to start the container by default with machines.target.

Type: boolean

Default: true

Example: false

nixos-nspawn.containers.<name>.binds

Read-Write bind mounts from the host. Keys are paths in the container.

Type: attribute set of (submodule)

Default: { }

Example:

{
  "/var/lib/example" = { };
  "/var/lib/postgresql" = {
    hostPath = "/mnt/data/postgresql";
    options = [
      "idmap"
    ];
  };
}

nixos-nspawn.containers.<name>.binds.<name>.hostPath

If not null, path on the host. Defaults to the same path as in the container.

Type: null or string

Default: null

nixos-nspawn.containers.<name>.binds.<name>.options

Options to pass to the bind mount. See systemd-nspawn(1) for possible values.

Type: list of string

Default: [ ]

nixos-nspawn.containers.<name>.binds.<name>.readOnly

Whether to enable Mount read-only.

Type: boolean

Default: false

Example: true

nixos-nspawn.containers.<name>.config

A specification of the desired configuration of this container, as a NixOS module.

Type: Toplevel NixOS config

Example:

{ pkgs, ... }: {
  networking.hostName = "foobar";
  services.openssh.enable = true;
  environment.systemPackages = [ pkgs.htop ];
}

nixos-nspawn.containers.<name>.network.veth.enable

Enable default veth link between host and container.

Type: boolean

Default: true

Example: false

nixos-nspawn.containers.<name>.network.veth.config.container

Networkd network config merged with the systemd.network.networks unit on the container side. Interface match config is already prepopulated.

Type: null or (attribute set)

Default: null

Example:

{
  networkConfig = {
    Address = [
      "fd42::2/64"
      "10.23.42.2/28"
    ];
  };
}

nixos-nspawn.containers.<name>.network.veth.config.host

Networkd network config merged with the systemd.network.networks unit on the host side. Interface match config is already prepopulated.

Type: null or (attribute set)

Default: null

Example:

{
  networkConfig = {
    Address = [
      "fd42::1/64"
      "10.23.42.1/28"
    ];
  };
}

nixos-nspawn.containers.<name>.network.veth.zone

Name of the zone to attach the veth on the host. The Interface name will be prefixed with “vz-”.

Type: null or string

Default: null

nixos-nspawn.containers.<name>.path

As an alternative to specifying config, you can specify the path to the evaluated NixOS system configuration, typically a symlink to a system profile.

Type: path

Example: "/nix/var/nix/profiles/my-container"

nixos-nspawn.imports

List of NixOS modules to be imported in every system evaluation when containers.*.config is being used.

Type: list of module

Default: [ ]

Example:

[
  { services.getty.helpLine = "Hello world! I'm a nspawn container!"; }
  inputs.lix-module.nixosModules.default
]