Now while many options exists, for me, I been debating on three. Dive so deep I hit bedrock with Nix and NixOS. Accept Kubernetes as the one true OS through Talos. Or drink the orange glowing Kool-Aid of snaps in Ubuntu Core. Bare with me. There's logic in these.

The choices

I been debating the choice on what to use as my de-facto immutable OS for a while now.

The three I settled (for now) are ones I think make the most sense. To set and forget. Distributions where I can realistically automate the entire thing. Without much fear of the ground falling apart. Because unlike normal Linux distributions, immutable OS versions are the wild west of Linux distros. All choose radically different ways of achieving same goals. Choice is great for ecosystem growth. Terrible for a foundation.

Ubuntu Core

The one I'm sure gets most debate is Ubuntu Core. Even though it's the most systematically grounded version of an immutable OS. The design is fairly simple. Use snaps. But the genius of this is how Canonical made every part of the OS into an isolated snap. Snaps you can update and change without impacting the 'core' of the OS.

You can update the entire OS with changing the snap version of the core you are based from. So jumping from Ubuntu core 20 to Ubuntu core 22 was doing something like this. And as Canonical has matured with how Ubuntu core operates, they have also made different layers to this design. Using the atomic principal.

So now there are snaps for the linux kernel. And configuration based ones called gadget snaps. Responsible for handling bootstrapping on a specific hardware. All following the same everything is a snap.

You can still run any application you want. The difference here is you have to package them differently. Or at least differently from the norm.

The con with Ubuntu core is that you have to accept their way of packaging software with snaps. There is no alternative. You must have Canonical as your overseer here. The pro is everything stated before benefits your ability to have a stable immutable system. It should not be understated. The ease that comes with having maintainability so low, with ubuntu core, is pretty remarkable.

Talos

Talos differs in a very simple way. The ENTIRE operating system. It is Linux technically because Talos uses the Linux kernel. It immediately deviates from there. You don't use an OS with Talos. What you use is Kubernetes.

Kubernetes is infamous at being complex. I even wrote about my annoyance years ago. After years, that article is still true for self hosted Kubernetes cluster. So is Kubernetes worth self hosting? I can wholeheartedly say NO. But with an exception. The exception here is if you are using Talos.

You see, Talos strips away practically everything of the OS. But it does it in a way that makes a lot of sense. You don't use ssh, because there is no shell to ssh to! Instead you use their gRPC API component; apid. Interfacing with the operating system. Talos has no systemd. Instead, replaced with their own PID 1. An init system called machined. With the whole purpose running what kubernetes needs and the gRPC interface to define the OS.

In practice, Talos is actually dead simple to use and administer. It makes using and maintaining kubernetes strikingly easy. New version release? Run talosctl to upgrade:

talosctl upgrade-k8s --to 1.28.0

Same is true for updating Talos itself. Because the OS is atomic, there is very little thought process required to handling failures. You rollback like nothing happened. Here, you simply use kubernetes.

The con with talos isn't actually the use of talos. It's the principal of only using kubernetes for everything. If you not comfortable with a job orchestration like it, DO NOT USE. If you somehow like ssh or want to install other things directly on the OS, DO NOT USE. If you don't want to do literally everything as code, definitely DO NOT USE.

But if you do value what Talos offers, it's immensely difficult not to choose it. So many problems are simply non-existent on Talos. The OS makes you question why even bother with the old ways.

NixOS

The prophecy is written. NixOS is the answer to our Linux administration ways. It will solve all our problems with packaging software. We will know of history as before and after NixOS. And quite frankly, it really does feel this way.

NixOS shines in the same ways the others in this list shine. It rethinks what a Linux is and could be. If you make absolutely everything atomic, to the core of how you package and run software, then do you even need to care if your OS breaks? NixOS potency is the nix programming language. Not to be confused with the nix as a package manager. Or the OS who is also called nix.

Unlike the other options on this list, the con with NixOS is quite immediately apparent. It's the difficulty you first have learning the ways of then learning the ways of using nix. Documentation is quite difficult to come by for nix. Much of your time will be left questioning how anything even functions. There's also then the full upgrade to nix design called nix flakes. All enough to really put incredible friction. Friction on nix usage and nixos adoption.

However, the moment you passed the hurdles of learning nix, nothing comes even close to its versatility. I personally have migrated most of my software to run with nix. Or at the very least, build with nix. For work, I have development environments strictly nix based. And the list goes on.

With NixOS, it's the same principal. You write your closure and you can be assured it will work. No matter what mess you do to the machine. You can roll back like nothing ever occurred. There's nothing really like Nix and NixOS. The principal is that you handle the hurdle of defining how your software is built. From then on, it will just run.

No more conflicts with versions of python. No issues with running two independently different versions of the same software. Because of the closure design, there's no need to containerize your applications. They 'just' work. With zero conflict running on the same host. Optimally, NixOS gets you the closest to the promise of Gentoo, but entirely atomic and immutable.

So what to choose?

I don't know yet. The reality is, all three options serve very similar ideas of running an immutable OS. They simply attack the problems differently. Talos packages software in kubernetes manifest essentially. Ubuntu Core is snaps. And NixOS is nix closures. All with different tradeoffs that are far too long to add to this never ending post.

For security reasons, I would probably go for Talos. Because of the stripped purpose of the OS, there's a smaller footprint to a security issue. Yes. Even with Kubernetes.

For maintainability, I think Ubuntu Core is prime. Canonical has been doing Linux distributions for decades now. They know what it means to make something function. Every Ubuntu core release has a maintenance window of up to ten years. Meaning, if I want to just run my thing, with no fuss, this will be it.

For customization, nothing gets even close to what NixOS promises and delivers. I would be able to take all the Nix flakes I been writing for myself and run straight on NixOS. True “it works on my machine” on all machines.

#linux #immutableos #ubuntucore #talos #nixos #atomic #ubuntu #kubernetes

Not too long ago Kelsey Hightower posted a tweet about what you may require to run an application platform. While his list may look large, it's not all encompassing. It's just an example of how ridiculous our abstractions have gotten. Even taking one or two service on each category in the landscape.cncf.io page, would not give the full list of dependencies. So I started thinking. What would a full encompassing list of the 'recommended stack' look like?

I've been running a few posts now on striking a balance between what you do to run your software in a sustainable and manageable way. This post is not that. The design here is the A-Z stack. A platform of everything I could think of, from on top of my head, you may need for a modern application platform. Please don't try to implement the stack here at work. If you already have, I'm sorry and I know your pain. There's consoling we can probably get for our insanity.

First, you need a cloud provider. If we want to be realistic, this would certainly be AWS. A high chance you and your team are already AWS Architects with decades of experience collectively.

Next, you need an infrastructure as code software. The list can get quite large here on implementation. For the purpose of the A-Z stack list, let's just say you use Hashicorp's Terraform. With a team of N > 1, you most certainly will be working on the code in terraform together. In other words, you need two parts to make terraform work properly here. One, you must be using version control. The famous single options are git and github. The second part is you need to have a backend that's not local for your terraform state. Since we using AWS, might as well use S3 with DynamoDB for state locking.

With your cloud provider and IaC ready, the next section is the OS. You should be running Linux, let's say CentOS. Make sure you have SELinux enabled and actually enforcing here. While the trend is to disable, don't. There are too many reasons to count why not. If you doing it to be fast, you are hurting yourself and your company far more later down the road. Moreover the OS security, you need something to actually provision the Linux of choice. Let's stay with Red Hat and go with Ansible. The provisioning alone can be its own series on lists of things you need, but we'll just keep it as is.

Ok, here we go with the container orchestrator. Sticking to the norm and the standard being Red Hat, let's say openshift. Openshift comes with a lot of batteries included, so the list would be much larger with what isn't set by default. However, for peace of mind, bringing up cri-o for CRI, Flannel for CNI, and CoreDNS for service discovery. For CSI here means probably Ceph with Rook. Secret management is a must and should be Vault.

Now comes networking. Already gave Flannel for CNI and CoreDNS, but you still need an ingress controller. The popular one here would probably be ambassador. Don't forget your services in this platform need to be able to communicate with one another. That means a service mesh is necessary. Here I'm going to cheat and use Linkerd. By cheat, I mean it covers the service proxy sidecar and the controller. Otherwise, we most certainly would then have to add Envoy. However, if your service does have proxy requirements not covered by the linkered sidecar, then Envoy is still required.

Here comes the home stretch! The actual application requirements. First is the Container registry, harbor. With it comes, Notary and Clair for build image security. Next, you need the deployments to Kubernetes using something structured like Helm. If not already brought up before, you need a key value store like etcd. You also need a CI/CD solution for your applications. So staying with the trends, Argo would suffice. Don't forget to monitor your application and platform. Most likely means Prometheus for metrics, Loki for logs, and Jaeger for tracing.

Now you can finally begin to write your application. Just so the point is made clear, here's the list of all that you must know, like the creators of the software themselves, to get this type of modern day application platform working:

1. Cloud Provider : AWS
2. IaC : Terraform
3. Version Control: Git
4. Version Control Host: Github
5. S3 and dynamodb for state locking on terraform
6. Linux Distribution: CentOS
7. SELinux enabled and enforcing
8. Linux Provisioner: Ansible
9. Container Orchestrator: Kubernetes, Openshift
10. CRI: CRI-O
11. CNI: Flannel
12. Service Discovery: CoreDNS
13. CSI: Ceph with Rook
14. Secret Management: Vault
15. Ingress Controller: Ambassador
16. Service Mesh: Linkerd
17. Service Proxy: Envoy
18. Container Registry: Harbor
19. Version Security Management: Notary
20. Build container image security: Clair
21. Kubernetes deployment: Helm
22. Key Value store: etcd
23. Continuous Integration and Delivery: Argo
24. Metrics Observability: Prometheus
25. Logs Observability: Loki
26. Tracing Observability: Jaeger

#100DaysToOffload #Day12 #Kubernetes #PaaS