everything as code

Sometimes the abstraction of abstraction of abstraction with yet another layer of abstraction is headache inducing and you just wish there would be a bash script somewhere in this repo that would tell you the exact sequence of commands you need to run to get the infrastructure for this app working. Today was one of those days. But let me go back to the beginning.

For the past three years, which is pretty much the bulk of my whole career in software from university graduation to today, I have been a permanent occupier of Layer 7 ( shoutout to the OSI model diagrams they showed us in CISSP training!). As a citizen of this layer, the code I type up deals with user features and relies on abstractions. Files to me are objects with attributes and methods - not bytes and buffers or memory pages. Once my file handling code exits the context manager, case closed, finito, data written to disk (which is of course a convenient abstraction pipe dream and not at all what actually happens). Once the feature is tested and shipped and the CI/CD system signs off with a green light of approval, the other six layers that make up the abstraction layer cake are usually the problem of another team (read Charity Majors’ blog if you want to find out why ‘I just ship it, someone else runs it’ is generally a bad strategy) or another member of my team.

Last October I stumbled on a cool thing called Kubernetes and started poking around the infrastructure tech space: Docker, Terraform and friends, which lead to me accepting a consulting job with a team building out some serious infrastructure on AWS. All of a sudden, this poor Layer 7 lurker is thrust into a world of bash scripts and infrastructure as code. If you haven’t heard of it before IAAC is a ‘movement’ (for lack of a better word) to define infrastructure through configuration files that can stored centrally and versioned instead of the good ol’ way of manually provisioning and setting up servers or running custom bash scripts. For one, this makes it easier to tear down and spin up new infrastructure really quickly. In particular if you are managing large scale infras in a cloud provider, you do not want to spend your day clicking around the console or ssh’ing into every single instance you have to run package manager commands and installation scripts. Second, if anything does go wrong, you can use version control to quickly checkout a previous copy of your infra, run the deployment commands and restore it to a previous state. With manual intervention, it’s harder to rollback changes, especially when production is on fire.

All of these are great benefits in particular if the scale of infra you are dealing with forces the ‘my servers are cattle, not pets’ paradigm. Increased abstraction in the form of tools with specialised configuration languages (ie. HCL for Terraform), can sometimes mean an initial hit in the velocity, especially if someone new to the tool is making changes. I came face to face with this reality while trying to untangle a Jenkins pipeline that was generating Terraform tvars files to be used in downstream jobs. Although, I had previously used Terraform to provision new infrastructure components, working with someone else’s infra code and associated scripts can be tedious. In particular, sometimes untangling what is happening depends on one’s knowledge of all the knobs and levers in the tool. Another problem here is that it is rarely possible to test these changes locally. A third problem is (as Rick Altherr mentioned in a reply to one of my tweets) small syntactic issues can quickly lead to hours of headscratching and painful debugging by trial and error. So good linting and formatting tools are a must (hi, from the developer who has spent hours trying to untangle tabs vs spaces issues in YAML and hunting down an errant comma in a JMESPath expression).

I suppose at the end of the day, we cannot have our abstraction cake and eat it.