Setting Up NixOps On Mac OS X With VirtualBox

Disclaimer

I am a new user of nixops, so I cannot guarantee these directions work for everyone. I have successfully set it up on two machines.

Preamble

The following directions describe how to setup nixops on a Mac OS X machine in VirtualBox. By the end of this you should be able to spawn as many NixOS instances in VirtualBox as your machine can handle. NixOps is similar to vagrant, except it deploys NixOS instances. It can deploy them locally, using VirtualBox, or remotely using EC2. It allows you to deploy clusters of machines, automatically allowing them to communicate with each other. At a high-level, nixops deploys an instance by doing the following:

1. It builds the environment you ask for on another NixOS instance. This could be your local machine or a build server.
2. It creates a VM on the service or system you defined (VirtualBox, EC2, etc).
3. It uploads the environment you've defined to the machine.

The main problem is that nixops must build the environment on the same OS and arch it is deploying. NixOS is a linux distro, that means you cannot built the environment on your Mac. The minor problem is that, by default, the OS X filesystem that everyone gets is case insensitive and that doesn't play well with nix, the package manager.

This post will accomplish the following:

1. Install and setup VirtualBox.
2. If your OS X file system is case insensitive (assume it is if you haven't done anything to change it), we will create a loopback mount to install nix on.
3. Install nix on OS X.
4. An initial NixOS VirtualBox instance will be created to bootstrap the process and act as a distributed build server.
5. Create a user on the build system.
6. Setup up signing keys, so we can copy environments between build server, host, and deployed VM.
7. Setup local nix to use this VM as a build server.
8. Deploy a VM.

1. Install VirtualBox

Download VirtualBox and install it. Just follow the directions. The only interesting thing you have to do is make sure you have the vboxnet0 adapter setup in networking. To do this:

1. Start VirtualBox.
2. Go to preferences (Cmd-,).
3. Click on Network.
4. If vboxnet0 is not present, add it by clicking the green +.
5. Edit vboxnet0 and make sure DHCP Server is turned on. The settings I use are below.

• Server Address: 192.168.56.100
• Server Mask: 255.255.255.0
• Lower Address Bound: 192.168.56.101
• Upper Address Bound: 192.168.56.254

2. Creating a case-sensitive file system

Unless you have explicitly changed it, your OS X machine likely has a case insensitive file system. This means nix build some packages. The method I have chosen to get around this is to create a loopback filesystem and mount that.

1. Create a image. I have been using one 5GB successfully, but if you plan on being a heavy user of nix, you should make it larger.
   hdiutil create ~/nix-loopback -megabytes 5000 -ov -type UDIF
2. Load it but do not mount:
   hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount ~/nix-loopback.dmg
3. Determine which disk and partition your newly created image corresponds to. Specifically you want to find the image that corresponds to the Apple_HFS entry you just created. It will probably be something like disk2s2, but could be anything.
   diskutil list
4. Create a case-sensitive file system on this partition:
   newfs_hfs -s /dev/disk2s2
5. Make the mountpoint:
   sudo mkdir /nix
6. Mount it:
   sudo mount -t hfs /dev/disk2s2 /nix

At this point if you run mount you should see something mounted on /nix.

NOTE: I don't know how to make this point on reboot, which you will need to do if you want to use nix after restarting your system.

3. Install Nix

1. Download the binary nix darwin package from nixos.org.
2. Go to root:
   cd /
3. Untar nix:
   sudo tar -jxvf /path/to/nix-1.5.2-x86_64-darwin.tar.bz2
4. Chown it to your user:
   sudo chown -R your-user /nix
5. Finish the install:
   nix-finish-install
6. nix-finish-install will print out some instructions, you should copy the 'source' to your ~/.profile and run it in your current shell (and any other shell you plan on not restarting but using nix in).
7. Delete the installer:
   sudo rm /usr/bin/nix-finish-install

4. Setup Nix

1. Add the nixos channel:
   nix-channel --add http://nixos.org/releases/nixos/channels/nixos-unstable
2. Update:
   nix-channel --update
3. Install something:
   nix-env -i tmux

5. Install NixOps

1. Set NIX_PATH:
   export NIX_PATH=/nix/var/nix/profiles/per-user/`whoami`/channels/nixos
2. Get nixops:
   git clone git://github.com/NixOS/nixops.git
3. cd nixops
4. Install:
   nix-env -f . -i nixops
5. Verify it is installed:
   nixops --version

5. Setup Distributed Builds

When deploying an instance, nixops needs to build the environment somewhere then it will transfer it to the instance. In order to do this, it needs an already existing NixOS instance to build on. If you were running NixOS already, this would be the machine you are deploying from. To accomplish this, you need a a NixOS running in a VM. Eventually nixops will probably accomplish this for you, but for now it needs to be done manually. Luckily, installing NixOS on VirtualBox is pretty straight forward.

1. Install a NixOS on VirtualBox from the directions here. This doesn't need any special settings, just SSH.
2. Setup a port forward so you can SSH into the machine. I'll assume this port forward is 3223.
3. Make a user called 'nix' on the VM. This is the user that we will SSH through for building. The name of the user doesn't matter, but these directions will assume its name is 'nix'.
4. On OS X, create two pairs of passwordless SSH keys. One pair will be the login for the nix user. The other will be signing keys.
5. Install the login public key.
6. On OS X, create /etc/nix/ (mkdir /etc/nix)
7. Copy the private signing key to /etc/nix/signing-key.sec. Make sure this is owned by the user you'll be running nixops as and is readable only by that user.
8. Create a public signing key from your private signing key using openssl. This needs to be in whatever format openssl produces which is not the same as what ssh-keygen created. This output should be in /etc/nix/signing-key.pub. The owner and permissions don't matter as long as the user you'll run nixops as can read it.
   openssl rsa -in /etc/nix/signing-key.sec -pubout > /etc/nix/signing-key.pub
9. Copy the signing keys to the build server, putting them in the same location. Make sure the nix user owns the private key and is the only one that can read it.
10. Tell nix to do distributed builds:
    export NIX_BUILD_HOOK=$HOME/.nix-profile/libexec/nix/build-remote.pl
11. Tell the distributed builder where to store load content:
    export NIX_CURRENT_LOAD=/tmp/current-load
    mkdir /tmp/current-load
12. Go into a directory you can create files in:
    

    cat <<EOF > remote-systems.conf
    nix@nix-build-server x86_64-linux /Users/`whoami`/.ssh/id_rsa 1 1
    EOF

13. Tell the remote builder where to find machine information:
    export NIX_REMOTE_SYSTEMS=$PWD/remote-systems.conf
14. Add an entry to ~/.ssh/config the fake host 'nix-build-server' turns into your actual VM:
    

    Host nix-build-server
        HostName localhost
        Port 3223

6. Start An Instance

1. Create your machine's nix expression:
   

   cat <<EOF > test-vbox.nix
   {
     test = 
       { config, pkgs, ... }:
       { deployment.targetEnv = "virtualbox";
         deployment.virtualbox.memorySize = 512; # megabytes
       };
   }
   EOF

2. Create a machine instance named test:
   nixops create ./test-vbox.nix --name test
3. Deploy it:
   nixops deploy -d test

This could take awhile, and at some points it might not seem like it's doing anything because it's waiting for a build or a transfer. It will push around a fair amount of data. After all is said and done you should be able to do nixops ssh -d test test to connect to it.

Troubleshooting

• I do a deploy and it sits forever waiting for SSH - You probably forgot to setup your vboxnet0 adapter properly. See Section 1.
• It dies while building saying a store isn't signed - Only root an import unsigned stores, this means your signing keys aren't stup properly. Double check your permissions.

Other problems? Post them in the comments and I'll add them to the list.

Known Bugs

• nixops stop -d test never returns - I've only experienced this on one of my installations. It is okay, though. Wait a bit and exit out of the command, then you can do any command as if stop succeeded
• My Mac grey-screens of death! - This has happened to be once. I update my version of VirtualBox and installed any updates from Apple and I have not experienced it again.

Further Reading

• NixOps Manual