[remark] We need deterministic installs, not just immutable OSs

by Ciprian Dorin Craciun (https://volution.ro/ciprian) on 

Immutable OSs are just a minor step towards reliable OS installations. However, for a complete solution we also need reproducible and thus deterministic installations, which implies cleaning-up and modernizing our package managers.

// permanent-link // Lobsters // HackerNews // index // RSS

These days I see a lot of talk about immutable Linux distributions (like various Fedora, OpenSUSE or Ubuntu spinoffs), some targeting desktops, some servers, while others containers. All the talk is about reliability and maintability, because any updates are done atomically, any failures can be easily rollbacked, and user-state is kept outside the OS file-system -- except for /etc which is "merged", and except for /var which is completely ignored. (A good recent article on the subject is Introduction to immutable Linux systems.)

Which I would say it sounds very nice, and I'm glad to see some positive progress in the unices landscape (Linux and BSD alike).

Because I still remember how Windows used to (perhaps still needs to?) be reinstalled every few months because cruft would pile on, and the whole thing crawled to a halt or crashed constantly. And I still struggle with Linux deployments that slowly drift away from a pristine installation with each new update and with each new configuration change (with or without Ansible-like tools).

However I'm not overly enthusiastic as I'll shortly describe.

Let me start first with a parallel: my favorite topic of containers. The technology is great in principle, however it is terrible in practice.

Instead of the container technology nudging people into creating small deployment bases with only the minimum required files to support the use-case at hand, it has helped "devops engineers" bundle even the proverbial kitchen sink with their applications.

(I am aware that there are initiatives to create small language tailored containerized runtimes; however, nothing beats in practice the good old apt install this-package that-package the-whole-internet-meta-package...)

Thus, my enthusiasm might be flamed by the possibilities enabled by the technology, but I'm quickly reminded by history that we don't always use the tools at our disposal in the most proper manner...

Getting back to our immutable OSs, my main issue is mainly with how they are assembled. (And my objections apply to all intended scenarios, from desktop, to container, to embedded.)

Namely, regardless of the chosen distribution (perhaps except for NixOS, but I'll have to learn more about NixOS to be sure) they all basically snapshot the outcome of a regular distribution package install and call the outcome the immutable OS "image" or "snapshot" or "layer" or "whatever-your-fancy".

Sure, to have an immutable OS there must be a lot of other supporting infrastructure (both in code and in services), there are certainly a lot of changes to the OS internals (especially bootup scripts), however ignoring all that, they are plain snapshots of regular distributions with some magic sprinkled here and there.

(I won't even mention that they make the usage of "application containers" like Flatpack or Snap mandatory for any kind of desktop application, or "proper containers", like Podman or Docker or any of their wrappers, for any development work...)

The only differentiator, besides which Linux distribution they are based on, is how that snapshot is made, how it's customized (by example for installing new packages, if even at all possible), how it's updated and how it's distributed and installed. Most use OSTree or rpm-ostree, others use BTRFS snapshots, meanwhile others use plain disk images.

What would I like instead?

I think it is past the time for distributions to take a leap forward into the future and change the way software packages are deployed. (I would have preferred to also leap forward into the future with regard to how these packages are built, but that's a much tougher battle...)

But first, I need to make a detour and describe where we are today within the landscape of software packaging and deployment.

For the sake of this article, I'm just going to ignore how the packages are actually built. I would just assume that there is a magic tool that enables us to build-package --package tool-v42 --output ./directory, and in a blink of an eye, we get our build artifacts nicely tucked into ./directory.

At the moment all package managers, from rpm to dpkg to apk to pacman, do the following (I'll be generalizing a bit):

Then at install time all package managers do the following (again, I'll be generalizing):

I would be very glad if anyone that writes me an email and points out some package manager that doesn't follow exactly this pattern. Sure, there is Clear's swupd which works with multiple packages as "bundles", and perhaps NixOS which I'll have to check if it supports hooks.

A reader pointed out to me that Haiku uses disk images instead of archive packages that are mounted read-only.

So what is the problem with any of the above?

Not much, with the exception of those hooks...

Because you see, these hooks are plain sh scripts that can do basically anything on your system, which means:

So, to wrap things up, the mere existence of such package hooks has the following consequences for an installation:

Here are a few links about the kinds of hooks in various distributions:

Thus, getting back to our immutable OSs, because all of them are the outcome of a classical installation process that relies on hooks, they are by consequence: unreliable, unreproducible, nondeterministic...

That can't be so, right?

Indeed, it isn't so for the simple reason of release engineering. Namely, when releasing a new snapshot of an immutable OS one doesn't just blindly run the install and snapshot that; instead one runs a battery of tests to see if the outcome is sane, and if all "tastes and smells right" it is then released.

So, how can we fix this?

(And as a direct consequence, not only fix the immutable OS release quality, but also improve the quality of container-based deployments, and even classical package based distributions, which could benefit greatly from all of this.)

First of all, let's see what use-cases these hooks serve, and see if we can't provide a better way to handle these.

To my knowledge (based on my experience) these hooks solve the following broad functions:

Thus, by removing the concept of package hooks, and improving package conflict detection (which shouldn't really happen, because that is why our package managers are full-blown SAT solvers capable of solving Sudoku puzzles), we can truly have immutable OSs.

Not only immutable in the sense of "we always boot from the same immutable snapshot", but also immutable in the stronger sense of "regardless of how many times we try to create that immutable snapshot, we always deterministically get the same outcome".

What prompted me to write this article?

I've been experimenting with OSTree and various distributions (Alpine and OpenSUSE) to see how easy it is to create (as a non-root user) extremely lightweight containers to be run with bwrap. And obviously, because I do want to have deterministic outcomes, I've opted out from executing package hooks...

As a consequence I've failed with Alpine, because without running busybox's post install hook I get a completely broken system, because many basic UNIX tools (like basename and such) are only installed as symlinks to busybox after running the post install hook. (Obviously these symlinks could have been part of the package itself, or perhaps an additional package, but unfortunately this is not the case...)

Thus, I couldn't manage to have Erlang or Elixir running in such a container, because their bootstrap scripts are actually sh scripts with a lot of magic in them... But this is a topic for the next article...

What others have to say on the subject: