In the following post, continuing the series of #StrikingABalance, we will explore how we would run a service, in legacy infrastructure. Brightening the shadow we've made using app containers and the actual simplicity container orchestration brings.

Container orchestrations are an answer to a problem created from the technological introduction of app containers. App containers are an excellent form of making reproducible artifacts. However, with its use, a requirement for something to run these new artifacts arises. While running locally is quite trivial. Especially with tools like docker-compose, running a container image in any form on the cloud will most certainly not be. The reason is fairly easy to describe, but not so easy to implement. You need a way to be able to make your deployment ephemeral and idempotent.

Let's take the approach of running an app container without a container orchestration.

The first you have to account for is how your app container is going to run. Let's say you will use a VM instance to run the container. Yet, before you can even answer how you will set up that instance, you need to figure out what is going to run the container image consistently. The easiest approach here probably be to use an init daemon. A simple systemd service unit file to keep the app container running can suffice. Allowing you to retrieve logs and status of the service, fairly quickly for the app container's runtime.

Now, back to the how of the app container's runtime will function. You now need to provision the VM instance before you can reach the stage of running the app container on systemd. A simple BASH script could work here, but remember the end goal here is for something that's idempotent and ephemeral. If your VM instance shuts down, you need a way to get back the setup you had prior exactly as it was before. Or if you introduce changes, you need a way to have proper configuration drift resolution. Writing an idempotent BASH script is non-trivial. Probably can make any grown man cry at the sight of its existence.

Most certainly, the second complexity introduced is some configuration manager like Ansible, Chef, or Salt to cover the provisioning of the instance. Take note, once you've completed your provision, you are now half way to your end goal of running an app container. The next stage here is now how are you going to retrieve the app container image for your runtime. The options can grow quite large. However, to keep it as simple, while skipping massive chunks of the implementation details, you can create a continuous deliver pipeline to run your configuration manager.

The continuous delivery pipeline would run a configuration manager, which fetches your container image, applies the systemd unit file, and starts up the service. One of the requirements to make the systemd runtime work is you need to run a container registry and another VM instance running said container registry. You will also need a CI/CD service as hinted before, if you don't already have one.

Lastly, you need to manage all of the VM instances you spun up for that one single app container you want to run on the cloud. You are now managing both an entire operating system to run that single app container and a fleet of VM instances to manage that app container runtime. The complexity doesn't stop there. You also need to track many other portions, like ssh privilege, security group isolation, system resource management, and service management.

The past way worked when we required only a few instances to run for our services. It becomes completely unmanageable when you have full fleet you require to run. Container orchestration allows you to take all of the complexity described here and apply it to a single standard structure on how you define an app container to run. Allowing for better abstractions, but also keeping the level of complexity built prior at a hopeful minimum.

#Day9 #100DaysToOffload #StrikingABalance

Modern day software design always tries to make the software we run close to stable as possible. Deadlines and limitations the teams will always have make it next to impossible to ship something bug free. Most likely than not, a 1.0.0 release will be a 1.0.1 in the same week or even day the version is available. The infrastructure that runs said software is no different.

In this post of #StrikingABalance, I'll be focusing on the work that goes into the infrastructure design and what can be done to make it more manageable. Infrastructure design should always be thought of as something you want to let crash. The whole idea of treat your services as as cattle rather than pets is with this precise notion. Letting things die, but having automation to bring it back alive.

Designing to fail means designing your software and infrastructure to gracefully manage itself. Ever had the moment that you get a call at 2:00 AM, because something is on fire on production? If you design your system to fail, that call would come by with the commonality of lightning striking twice.

So how do you go about in designing to fail? You could go with the route of always adding tests. Similar to unit and integration tests in software design, you add unit and integration tests to your infrastructure design. But the test then need to record what goes wrong. A way track events of your infrastructure's failures. Those tracked events would then trigger some automation tooling, you most likely have made, to do the response you want it to do for your infrastructure.

At the same time, due to the almost certain countless of moving parts in your infrastructure, you probably cannot test the whole platform in a silo. It may simply take too long, cost too much, or just not be feasible with what you have available. Meaning, you need to learn how to let infrastructure fail in production. You end up having load balancers, canary deployments, roll back releases, injected load tests, structured automated service rerouting, event triggered automation processing, database vertical scaling, horizontal service scaling, and countless other practices and paradigms.

No right way for how to design to fail exists. What you need above all else is to be comfortable with letting the infrastructure crash. Have a way to learn from those crashes. Then, make practices that can catch these crashes and recover before you have to be involved. So when those bugs creep up, you don't need to be ready, because your infrastructure is built to fail.

#100DaysToOffload #StrikingABalance #Day5

In the first of the #StrikingABalance series, I'll discuss over what is the most common ways to do a repeatable build. Then touch a little on what we all could do better, to do more with less.

It is almost a given, but the most common way of doing a repeatable build that should be laid out is none other than an app container. An app container in the following context is one described by the open container initiative (OCI) image-spec. I don't believe I need to explain what they are, with their commonality in the space thanks to Docker and Kubernetes. However, there is one thing I would like to point out in its implementation. With an app container, the way to actually build has stayed using practices of old. Essentially we make snapshots of a chroot environment with a set of bash scripts in the hopes of making our software run how we required it to run. It may sound like its reproducible and it can be, but the trend tends to be far from the goal.

The interesting part of creating an app container image though, is in what you contain for your app container image. It's the tools you add to the container image, which will become the container you run. Ideally, you only add applications that are required to run your software and that's it. But in practice, rarely do individuals take into make an app container, an app container. Instead, trending to garnishing an app container to be no different than that of a system container like LXD. Not to say system containers don't have their place. They very much do and play infinitesimally more important role as we move further into the cloud and abstractions. But a system container, is not what an app container should be.

There's a reason why members of the world are still reluctant to jump on doing everything in app containers. The benefit don't necessarily resolve the problem of making a repeatable and isolated build all the time. In many cases, an app container ends up becoming more complex than that of using the works of package management technologies like DEB and RPMs.

An app container should only run and contain the software you require it to run and that's it. Best case scenario is that you build an app container in a way that the source and its dependencies are the source of truth for the app container. A way to describe the app container's snapshots in a way that are completely identified by the source of the software you are trying to run. There is a trend for this, and it's definitely growing.

In the past couple of years, strides of experimentation have been made to get us to a place where the source is the truth for how an app container is built and what it runs within that container. Two projects that have works doing so are Nix and Habitat. Both projects approach building an app container similarly. In essence, making an app container by assigning an artifact from builds, which are instrumented using a source dependencies for said software. In doing so, an app container's image, holds only the source and libraries required to run the software, while also using extremely advance dependency tree mapping available from a full package manager.

We still have time to grow on creating app container images. if used properly we can have our abstractions for builds to be manageable. So we can focus on what matters, the code we want to run.

#100DaysToOffload #Day4 #StrikingABalance

We spend quite a bit of time developing software to run software.

It is is not to say we can't run software proficiently. There's a reason why a trillion dollar company like Alphabet spent time writing container orchestration tools like Borg and Kubernetes. But the level of complexity that we adhere to make an abstraction for something we want repeatable and automated, can get ridiculous.

I spend my days writing code to run code. The effort is non-trivial and at times can feel quite daunting. You spend hours or even days optimizing a set of abstractions just to keep things just slightly more sane than the last set of changes. However the practice can also be grandly satisfying. When you do get those optimizations in place, your software tends to become more performant and more stable. Meaning you get to sleep more and enjoy life a little bit more. Always looking for ways to lessen the workload by decreasing complexity, but also increasing abstractions.

I think a strong a balance on what need to run for your software and tooling you write to run that software. With that said, will write a small series focusing mostly on software development builds and their runtime.

#100DaysToOffload #Day3 #StrikingABalance