Anyone who has written software for multiple architectures know it is never plainly, “compile for your target and it will just work.” The reality is, there are always some exception you must take account for. It makes sense why. Any difference in architecture means entirely different ISA or even revision. Which translates to different rules for memory, instruction sets available, operational cost analyst, and much more. So taking your C code and just compiling it for AArch64 wouldn't cut it.

The same can be said for a process virtual machine used in Java. In theory, using the virtual machine meant you would target only the virtual machine and not have have to worry about the architecture the virtual machine would run under. However, in practice it has not been the case. Almost always, you still have to be aware of the architecture you are targeting due to bound operational differences like the list given prior.

This is where Web Assembly (wasm) is starting to shine. Wasm initially started as a way to run bytecode faster on the browser. Allowing for running heavier logic not possible with ECMAScript. To accomplish this, wasm was set up to use a process virtual machine like the JVM, but that targets a very specific ISA. So the VM can execute the ISA differently, but still requires to be able to run without modification. This results in the bytecode having exactly the same assembly definition regardless of physical architecture. Which means the VM can execute the bytecode on how it would be better optimized for the architecture. Giving way to focusing on only writing for a single target and only optimizing for that target. Leaving the VM that will run the wasm binary to take care of the heavy lifting as it should.

Right now, the prospect of wasm use in running on everything is not there just yet. It was only December 2019 the wasm specification was even agreed upon by W3C. Yet, you can see the ability starting to creep up. Projects like WASI from Mozilla for portability and security, Krustlet for kubelet running wasm instead of containers on kubernetes, and Cloudflare's Workers for running on the edge with GEO distribution using wasm.

I'm excited to see web assembly's potential. Its progress may result in a saner stack. Maybe even only requiring one runtime and nothing else, web assembly.

#Day14 #100DaysToOffload #Wasm #WebAssembly

If you haven't, then you haven't had a codebase at the multi thousand lines of boilerplate. You just naturally end up writing so much logic gates, just to get around the limitations of what you are using. This is true in software development, but it is especially true in a Domain Specific Language (DSL) for Infrastructure as Code (IaC). My setup has always been to use the declarative nature of Hashicorp's Terraform and AWS Cloudformation for their respective jobs. The IaC tools have worked great. I've been a heavy user of both. But it can get pretty ridiculous how much you have to do to get one single resource up and running (correctly anyway).

Moreover, if you try to do the same with a provisioner or ochestrator, then you are definitely in for multiples levels of hell. Not to say you can't go the route of using Kubernetes for your IaC implementation or Ansible in a declarative fashion. The problem is, you end up writing too much boilerplate before you ever get to what you wanted to do. Your goal ends being forever further away from your intentions.

What terraform and cloudformation get right is you need a full set of primitives to your infrastructure. It makes declarative state the goal of what you want, rather than the how you get to that state. If you try to use libraries for cloud providers directly, like boto or godo, with a programming language of choice, you end essentially building an entire in-house IaC provisioner. Requiring no difference in the level of boilerplate you write. Probably more so as you now need code to define basic primitives in a declarative fashion before you can create what you want.

However, using a generic programming language for IaC can have some strong upsides. You can get precise logic and structures that are very well defined to what you need the infrastructure to be. You can also do some classical test driven development to better define your logic. So in the past few months, I have been thinking on how to resolve the problems of having to constantly write so much boilerplate, make maintenance more manageable, and have better abstractions for core primitives I want to create on a cloud provider. Instead of trying to do this again with cloudformation and terraform, I have begun working with pulumi and AWS CDK for the ambitious goal.

I'm still early in my venture, but so far I have learned both provide much simpler definitions for resources to create. The boilerplate is extremely minimal as the tools are designed for you to create modules or packages that you extend for your needs. With both, my codebases have gone done considerably. Making maintenance actually feasible. I'm still discovering of their usage. Still, I'm really liking the ability to use a full generic programming language to do everything I need to in my IaC.

#100DaysToOffload #Day13 #Infrastructure #IaC #Declarative

Not too long ago Kelsey Hightower posted a tweet about what you may require to run an application platform. While his list may look large, it's not all encompassing. It's just an example of how ridiculous our abstractions have gotten. Even taking one or two service on each category in the landscape.cncf.io page, would not give the full list of dependencies. So I started thinking. What would a full encompassing list of the 'recommended stack' look like?

I've been running a few posts now on striking a balance between what you do to run your software in a sustainable and manageable way. This post is not that. The design here is the A-Z stack. A platform of everything I could think of, from on top of my head, you may need for a modern application platform. Please don't try to implement the stack here at work. If you already have, I'm sorry and I know your pain. There's consoling we can probably get for our insanity.

First, you need a cloud provider. If we want to be realistic, this would certainly be AWS. A high chance you and your team are already AWS Architects with decades of experience collectively.

Next, you need an infrastructure as code software. The list can get quite large here on implementation. For the purpose of the A-Z stack list, let's just say you use Hashicorp's Terraform. With a team of N > 1, you most certainly will be working on the code in terraform together. In other words, you need two parts to make terraform work properly here. One, you must be using version control. The famous single options are git and github. The second part is you need to have a backend that's not local for your terraform state. Since we using AWS, might as well use S3 with DynamoDB for state locking.

With your cloud provider and IaC ready, the next section is the OS. You should be running Linux, let's say CentOS. Make sure you have SELinux enabled and actually enforcing here. While the trend is to disable, don't. There are too many reasons to count why not. If you doing it to be fast, you are hurting yourself and your company far more later down the road. Moreover the OS security, you need something to actually provision the Linux of choice. Let's stay with Red Hat and go with Ansible. The provisioning alone can be its own series on lists of things you need, but we'll just keep it as is.

Ok, here we go with the container orchestrator. Sticking to the norm and the standard being Red Hat, let's say openshift. Openshift comes with a lot of batteries included, so the list would be much larger with what isn't set by default. However, for peace of mind, bringing up cri-o for CRI, Flannel for CNI, and CoreDNS for service discovery. For CSI here means probably Ceph with Rook. Secret management is a must and should be Vault.

Now comes networking. Already gave Flannel for CNI and CoreDNS, but you still need an ingress controller. The popular one here would probably be ambassador. Don't forget your services in this platform need to be able to communicate with one another. That means a service mesh is necessary. Here I'm going to cheat and use Linkerd. By cheat, I mean it covers the service proxy sidecar and the controller. Otherwise, we most certainly would then have to add Envoy. However, if your service does have proxy requirements not covered by the linkered sidecar, then Envoy is still required.

Here comes the home stretch! The actual application requirements. First is the Container registry, harbor. With it comes, Notary and Clair for build image security. Next, you need the deployments to Kubernetes using something structured like Helm. If not already brought up before, you need a key value store like etcd. You also need a CI/CD solution for your applications. So staying with the trends, Argo would suffice. Don't forget to monitor your application and platform. Most likely means Prometheus for metrics, Loki for logs, and Jaeger for tracing.

Now you can finally begin to write your application. Just so the point is made clear, here's the list of all that you must know, like the creators of the software themselves, to get this type of modern day application platform working:

1. Cloud Provider : AWS
2. IaC : Terraform
3. Version Control: Git
4. Version Control Host: Github
5. S3 and dynamodb for state locking on terraform
6. Linux Distribution: CentOS
7. SELinux enabled and enforcing
8. Linux Provisioner: Ansible
9. Container Orchestrator: Kubernetes, Openshift
10. CRI: CRI-O
11. CNI: Flannel
12. Service Discovery: CoreDNS
13. CSI: Ceph with Rook
14. Secret Management: Vault
15. Ingress Controller: Ambassador
16. Service Mesh: Linkerd
17. Service Proxy: Envoy
18. Container Registry: Harbor
19. Version Security Management: Notary
20. Build container image security: Clair
21. Kubernetes deployment: Helm
22. Key Value store: etcd
23. Continuous Integration and Delivery: Argo
24. Metrics Observability: Prometheus
25. Logs Observability: Loki
26. Tracing Observability: Jaeger

#100DaysToOffload #Day12 #Kubernetes #PaaS

Back in '05 when Linus Torvalds decided to release git for the Linux kernel, a week later mercurial was released with the same purpose. Git was built in a way that you would expect a man like Linus Torvalds would make version control system. Something utterly complex, but absolutely brilliant if you understand everything about its internals. Mercurial was designed instead to be user friendly first and simple enough to to get the job done.

Mercurial most certainly lost to git due to its own simplicity. You can get up and running faster on mercurial than you ever could/can with git. It has very intuitive verb commands that do precisely what you would believe they would do. Mercurial also has extremely powerful search capabilities with revsets expressions. It has built in safeguards, which makes sure you don't shoot yourself in the foot in every command. Mercurial even has its own web server baked in. And that's the problem.

The tool was so correctly designed, it didn't give way for something like github to be required for it to work. You only needed someone with an IP and port, and you can get a running host. Have a team near you? Run hg serve and you can collaborate immediately. No need to sign up to some site now owned by a trillion dollar company. You can just work and do what you need to do. This is precisely the problem. The tool actually solved the problems you needed. So there just simply was no real reason why you would need to spend money on a web host or forge for mercurial.

So now we are all stuck with three options: git (github), git (gitlab), and git (bitbucket). Good job, mercurial. You beat git so well, you kicked yourself out the fight.

However, there is some hope. In recent years, there's been a slow but steady resurgence of the tool. Facebook still very much uses it for their terabyte sized repository. A full rust rewrite of mercurial's internals have been in the works to fully make it just that much better. New hosting forges like heptapod and sourcehut have come into the picture.

Mercurial will probably never reach the heights of git. But for people who want to get their work done, rather than fight with their tool, it will find a nice and sane home.

#Mercurial #Git #100DaysToOffload #Day11

Asynchronous messaging is everywhere and you most certainly already are familiar with. Especially with the most common one, your email. While, there are many examples of its abuse (looking at you Google), email is still prevalent everywhere for a reason. Not only is email decentralized and thankfully not yet owned by single a entity (again, looking at you Google), but it is also excellent for working remote. With an email, you can reply to conversations at your own leisure, rather than someone's else. That empowering moment to reflect of what you will say is very much important to how you collect your thoughts.

Even with the way people communicate with one another on the internet changes due to these delays between messages. That's not to say asynchronous messaging is the only route to use. Sometimes you do need that instant mode of sensory input. But those type of modes of communication should be used after you've expressed a thread of a conversation in an async manner. Which can also mean your instantaneous mode of communication become more fruitful, as both you and your colleagues would have more insight into what the conversation would be about.

The best part of asynchronous messaging only shines brighter when you have a team that is distributed. The further you are between co-workers, the more it is vital you understand that your teammates are not available at your time. Meaning, you can recognize their time is just as precious as your own. They are only available, when its their work hours hopefully. Sometimes not even the same region or even a near timezone as you. So being mindful and expressive in your mode of communication is key. So you can go and spend less time trying communicate what you need and actually communicate what you need.

So here's to using asynchronous messaging. Use it first when you can. Only after do you then leave the spam that is instant messaging to come in, if needed.

#100DaysToOffload #Day10 #Email

In the following post, continuing the series of #StrikingABalance, we will explore how we would run a service, in legacy infrastructure. Brightening the shadow we've made using app containers and the actual simplicity container orchestration brings.

Container orchestrations are an answer to a problem created from the technological introduction of app containers. App containers are an excellent form of making reproducible artifacts. However, with its use, a requirement for something to run these new artifacts arises. While running locally is quite trivial. Especially with tools like docker-compose, running a container image in any form on the cloud will most certainly not be. The reason is fairly easy to describe, but not so easy to implement. You need a way to be able to make your deployment ephemeral and idempotent.

Let's take the approach of running an app container without a container orchestration.

The first you have to account for is how your app container is going to run. Let's say you will use a VM instance to run the container. Yet, before you can even answer how you will set up that instance, you need to figure out what is going to run the container image consistently. The easiest approach here probably be to use an init daemon. A simple systemd service unit file to keep the app container running can suffice. Allowing you to retrieve logs and status of the service, fairly quickly for the app container's runtime.

Now, back to the how of the app container's runtime will function. You now need to provision the VM instance before you can reach the stage of running the app container on systemd. A simple BASH script could work here, but remember the end goal here is for something that's idempotent and ephemeral. If your VM instance shuts down, you need a way to get back the setup you had prior exactly as it was before. Or if you introduce changes, you need a way to have proper configuration drift resolution. Writing an idempotent BASH script is non-trivial. Probably can make any grown man cry at the sight of its existence.

Most certainly, the second complexity introduced is some configuration manager like Ansible, Chef, or Salt to cover the provisioning of the instance. Take note, once you've completed your provision, you are now half way to your end goal of running an app container. The next stage here is now how are you going to retrieve the app container image for your runtime. The options can grow quite large. However, to keep it as simple, while skipping massive chunks of the implementation details, you can create a continuous deliver pipeline to run your configuration manager.

The continuous delivery pipeline would run a configuration manager, which fetches your container image, applies the systemd unit file, and starts up the service. One of the requirements to make the systemd runtime work is you need to run a container registry and another VM instance running said container registry. You will also need a CI/CD service as hinted before, if you don't already have one.

Lastly, you need to manage all of the VM instances you spun up for that one single app container you want to run on the cloud. You are now managing both an entire operating system to run that single app container and a fleet of VM instances to manage that app container runtime. The complexity doesn't stop there. You also need to track many other portions, like ssh privilege, security group isolation, system resource management, and service management.

The past way worked when we required only a few instances to run for our services. It becomes completely unmanageable when you have full fleet you require to run. Container orchestration allows you to take all of the complexity described here and apply it to a single standard structure on how you define an app container to run. Allowing for better abstractions, but also keeping the level of complexity built prior at a hopeful minimum.

#Day9 #100DaysToOffload #StrikingABalance

I didn't know about the recovery recovery partition until I ended up purchasing a System76 laptop and using the clean install procedure. Pop!_OS has its own Recovery mode. The recovery mode is essentially a partition with a live install version of the beautiful distribution. It allows you to do full install changes, upgrades, and any other recovery requirements you may have.

If you like me, fellow reader, you may be using a more custom partition setup. Unfortunately, anything other than the default clean install on Pop!_OS means you do not get the automatically generated recovery partition of goodness. Here's a small guide over how to get the setup on an already existing install.

First, make sure you have all the tools required here:

• mkfs.vfat : required for creating the recovery filesystem.
• parted or gdisk : resize and create partition.
• pop-upgrade : available by default on POP!_OS.
• a text editor of choice.

Make the recovery partition

You now need to make sure you have 4.4 GiB of unallocated storage available on your primary disk drive. The unallocated storage space will be used to make a new FAT32 partition on your partition table.

Note, if you are using btrfs, make sure to resize the filesystem before shrinking the storage available. A required step as you may have allocations used in locations you will be removing. Hence a small rebalance for those data blocks:

btrfs filesystem resize -4.4g /

Use your favorite tool to resize the disk. Personally, I tend to just use gdisk, but you can use parted or any other tool for the task. Once you have resized, make the 4.4 GiB partition, if you haven't already, and create the FAT32 filesystem on the new partition:

# the label RECOVERY is to make easier to define later
mkfs.vfat -n RECOVERY /dev/{{ recovery_parition_id }}

Next, you need the new partition mounted on /recovery.

mount -L RECOVERY /recovery

Now comes the tricky part. You need to run pop-upgrade tool to install the Pop!_OS ISO into the recovery partition. Before you do, make sure you have open on a tail of the logs for pop-upgrade:

journalctl -flu pop-upgrade

The pop-upgrade tool has a tendency to fail and the error messages are not exactly descriptive. Having the logs available can be quite helpful in debugging what went wrong. Once you do have the logs tailed, on a separate terminal run the following:

# use 20.04 as of this writing for Focal version
pop-upgrade recovery upgrade from-release 20.04

If you have a /tmp directory with less storage available than the full ISO image size of POP!_OS, you will have troubles installing. Before this becomes an issue, make sure you have available at least 2.5 GiB of disk storage on /tmp. If you don't, you can always do a bind mount to filesystem that does temporarily. Allowing you to download the ISO. Once pop-upgrade procedure is complete, simply unbind the bind mount.

If by any chance your recovery installation fails on pop-upgrade, head over to your /etc/fstab and comment out the auto generated entry for you /recovery partition. The tool currently has an issue where it will fail if the entry exists on your filesystem table file.

Add the recovery.conf file

Ok, so now that you have the recovery partition, you need to add the recovery.conf file. First run the command below to create the file in /recovery/recovery.conf filepath with the following template:

 EFI_UUID={{ boot_efi_uuid }}
HOSTNAME=pop-os
KBD_LAYOUT=us
KBD_MODEL=
KBD_VARIANT=
LANG=en_US.UTF-8
LUKS_UUID=
OEM_MODE=0
RECOVERY_UUID={{ pop_recovery_uuid }}
ROOT_UUID=UUID={{ pop_root_uuid }}
UPGRADE=1

Edit all the {{ }} with the correct partition's UUID. To keep this easier to identify, use the PARTUUID of your partitions for all of the entries like the example below:

ROOT_UUID=PARTUUID=6fee8edb-1e18-485b-95aa-4e36f1abaa4e

Create the systemd-boot entry

Congrats, you now up to the final stretch. You only need two things here, the actual ID for your recovery partition and the partuuid of the partition. Make an entry on your systemd-boot loader entry:

title Pop!_OS recovery
linux /EFI/Recovery-{{ recovery_partition_uuid }}/vmlinuz.efi
initrd /EFI/Recovery-{{ recovery_partition_uuid }}/initrd.gz
options boot=casper hostname=recovery userfullname=Recovery username=recovery live-media-path=/casper-{{ recovery_partition_uuid }} live-media=/dev/disk/by-partuuid/{{ recovery_partition_partuuid }} noprompt

All that's left is to test the new boot entry. Simply do a restart and see if you can actually boot into the new Pop!OS recovery entry. If all went well, you now you should have a fully functioning Pop!OS recovery mode available on boot. If you had any troubles during this setup, you can look through the journal logs for pop-upgrade to see if you can resolve.

While this may seem like a lot, it is due to the fact pop-upgrade tool doesn't yet support setting up post install. If you had the luxury of doing your custom setup before any of this, you can simply add a 4.4 GiB FAT32 partition and Pop!_OS install will take care of the rest.

#100DaysToOffload #Day8 #PopOS #System76

I don't even know how I convinced myself to try to use a machine without a Tiling manager, but that was the case back when I first tried out System76's Pop!_OS. Before I get to the why for Pop, let me give a little back story.

I been using Tiling Managers for almost a full decade. It all started with an obsession of a small, but definitely strong, embedded language called Lua.

Since the moment I learned of the language, I became absolutely obsessed with it. So much so, I wanted my whole software suite to be based on Lua. Everything from what I wrote personally, to what I wrote professionally, was in this language in one shape or another. It became such a large part of my way of being, to this day, Lua is the first thing I have installed on every machine that I can install it to. Be it work, home, servers, or things that compute, Lua is always there ready to help.

So when I was looking for some sort of desktop environment using the language, I learned of a tiling manager written in Lua; Awesome. The Awesome tiling manager is what would happen if you tried to mix ease of use with customization you can die for. You end up being in a place where you can make your desktop experience actually function as your desktop experience. A truly custom tailored tiling manager to your precise needs.

I was happy with Awesome. But being a polyglot by nature, I became in love with a different language in the coming years; Rust. Very much like my fascination with Lua, I ended up having a Rust written something, everywhere. Thus, when System76 announced Pop!_OS, they also announced it would have parts of its internals written in Rust. Nuff said. I ended up installing it on my machine and wait for it... actually like Gnome 3.

With the new love, my heart still ached from missing the tiling manager life of old. Ended up trying all the Gnome tiling manager extensions, but they just didn't feel right. Still, I stuck with gTile as the extension of choice for imitative Tiling management on Gnome 3.

However, with Pop's new release of 20.04, a full integrator level extension to gnome 3 has been made by the lovely lovely team in system76 for tiling management. Now it's not like Awesome, but I mean how could it be? Awesome is awesome for a reason. No, this released extension for Pop!_OS is based on i3. But boy is it outstanding. I definitely have a home with System76's work for years to come. Still learning the key bindings and tricks, yet you can very much see there was so much thought put in the tiling manager work. For crying out loud, the crazy team are even making a keyboard designed slightly different to augment the tiling manager use with their keyboard shortcuts. I'm absolutely sold.

If you still deciding whether you want to run a Tiling manager or not, stop what you doing. Install Pop!_OS. You'll Thank me later.

#100DaysToOffload #Day7

On average, humans spend about 26 years sleeping. However, that's only the average. Not necessarily the norm. In fields with high stress work loads, sleep deprivation tends to escalate, along with degradation of time and focus. Sometimes these environments' workload may put you, in a position to remove your moments of rest, to finish work set out for the day. You not only are sacrificing the circadian rhythm for sleep, you end up becoming void of your memory of that day.

It turns out sleep plays a forever growing role in the ability to record memory and learn. When you remove your time of rest, you end up removing your ability to learn and most importantly remember.

From an anecdotal perspective, I have been able to adapt and learn more with succumbing to sleep than when I've stripped away. The realization of its importance has made me appreciate the necessity of the hours I lay on my bed.

The interesting part of sleep is that there are multiple different patterns towards getting a good rest in the night. It's not all just one block of eight hours in your bed. Learning above the different types, I have become adapted to the biphasal segmented sleep pattern. Essentially the idea is you sleep in two 3.5 hour cycles. For each cycle, you wake up and do something of light brain activity. So when you enter the next core sleep cycle, you end allowing yourself to have four cycles total of uninterrupted REM sleep for the night. In doing so, waking up gives sort of jolt of energy for the day ahead.

Biphasal segmented sleep has helped me immensely absorb, learn, and recover from the activity I've had in the day past. Yet, that's not to say the pattern is for everyone. Others may end having better rest with single blocks of sleep. Some may even require to have a siesta in their day to truly feel energized.

Whatever is the pattern for your sleep, it is what helps you actually achieve the goals you set out to do. Not the overwork or deprivation of bodily necessities.

So take the rest as important as you should, and sleep.

#100DaysToOffload #Day6

Modern day software design always tries to make the software we run close to stable as possible. Deadlines and limitations the teams will always have make it next to impossible to ship something bug free. Most likely than not, a 1.0.0 release will be a 1.0.1 in the same week or even day the version is available. The infrastructure that runs said software is no different.

In this post of #StrikingABalance, I'll be focusing on the work that goes into the infrastructure design and what can be done to make it more manageable. Infrastructure design should always be thought of as something you want to let crash. The whole idea of treat your services as as cattle rather than pets is with this precise notion. Letting things die, but having automation to bring it back alive.

Designing to fail means designing your software and infrastructure to gracefully manage itself. Ever had the moment that you get a call at 2:00 AM, because something is on fire on production? If you design your system to fail, that call would come by with the commonality of lightning striking twice.

So how do you go about in designing to fail? You could go with the route of always adding tests. Similar to unit and integration tests in software design, you add unit and integration tests to your infrastructure design. But the test then need to record what goes wrong. A way track events of your infrastructure's failures. Those tracked events would then trigger some automation tooling, you most likely have made, to do the response you want it to do for your infrastructure.

At the same time, due to the almost certain countless of moving parts in your infrastructure, you probably cannot test the whole platform in a silo. It may simply take too long, cost too much, or just not be feasible with what you have available. Meaning, you need to learn how to let infrastructure fail in production. You end up having load balancers, canary deployments, roll back releases, injected load tests, structured automated service rerouting, event triggered automation processing, database vertical scaling, horizontal service scaling, and countless other practices and paradigms.

No right way for how to design to fail exists. What you need above all else is to be comfortable with letting the infrastructure crash. Have a way to learn from those crashes. Then, make practices that can catch these crashes and recover before you have to be involved. So when those bugs creep up, you don't need to be ready, because your infrastructure is built to fail.

#100DaysToOffload #StrikingABalance #Day5