James Quigley's Blog

Kubernetes Autoscaling Overview

Tue, 15 Dec 2020 00:00:00 GMT

Summary
Autoscalers
Autoscaling Caveats

Summary

Kubernetes is often touted as the solution for modern cloud native deployments of applications. It provides an agnostic platform for running code in a fashion that can tick all the “-ility” boxes (i.e. reliability, scalability, observability, manageability etc).

However when working with Kubernetes, by default nothing autoscales!

Autoscaling is a common tool when considering the scalability of your application. After all having to scale up a production application manually is time consuming and error prone. In Kubernetes you can specify a deployment with multiple replicas which can help with high availability, but that doesn’t address any sort of autoscaling concerns. Autoscaling can help your application keep up with increased load and save you money in times of reduced load.

While they don’t come setup by default, there are three main types of autoscaling that you can utilize when it comes to Kubernetes. They are:

Horizontal Pod Autoscaler (HPA)
Vertical Pod Autoscaler (VPA)
Cluster Autoscaler

As implied by the names, HPA and VPA work on the pod level inside the cluster, whereas cluster autoscaler will add entire nodes to your worker fleet, dynamically scaling the entire cluster. Note that both HPA and VPA require Metrics Server installed in your cluster in order to function.

HPA and VPA should not be used together on CPU/Memory for a deployment. You can however combine them if using custom metrics on your HPA definition. That would allow you to scale up if you hit a CPU threshold, and scale out if you hit another limit, like request throughput, or queue length if you are autoscaling some kind of task processing pod.

Autoscalers

Horizontal Pod Autoscaler (HPA)

Horizontal Pod Autoscaler is a part of the “autoscaling” api group in Kubernetes. As of right now, the v1 version of this API only supports autoscaling based on CPU, but the beta version supports autoscaling based on memory or custom metrics.

HPA is a resource in your cluster that you can create with a definition like so:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  targetCPUUtilizationPercentage: 80

It behaves pretty simply. You set a target utilization, minimum and maximum number of pods, and what deployment to target, and HPA does the rest. It will scale your deployment up/down based on the CPU utilization and try to keep the CPU metric as close to the target as possible, respecting the min and max replica values. If overnight you get a lot less traffic, now your deployment can scale down so as not to be using as many resources during that time. Then in the morning when traffic picks back up, it will automatically scale up to keep up with the increased load.

Vertical Pod Autoscaler (VPA)

Whereas HPA deals with scaling horizontally, or adding/removing pods, Vertical Pod Autoscaler scales vertically, which adjusts the resources allocated to pods in order to give them more CPU/Memory.

Also unlike HPA, VPA is not baked into one of the default APIs, and must be installed into the cluster with a Custom Resource Definition (CRD).

A sample VPA definition might look like this:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind:       Deployment
    name:       my-app
  updatePolicy:
    updateMode: "Auto"

VPA will examine your pods, and if they are maxing out their resources, will update the pods to give them more resources. There’s some nuance to the “updateMode”, in that “Auto” by default will evict running pods to adjust their resources. This gives you a more dynamic scaling, but also means that pods can be evicted by VPA, potentially causing downtime if your configuration doesn’t have multiple replicas and/or you haven’t configured Pod Disruption Budgets.

There is no current way for VPA to update resources on a pod in place, but if/when that becomes available in the future, it is likely that it will become the default updateMode when selecting “Auto”.

Cluster Autoscaler

Cluster Autoscaler behaves much like HPA, but instead of scaling pods it adds/removes worker nodes in your cluster. When combined with HPA and/or VPA, this allows for a truly dynamic cluster experience, allowing you to go from a single worker node with a single pod all the way up to a large cluster with tons of pods without having to scale anything manually.

Installing and using cluster autoscaler takes a little bit more work than HPA though, since it needs to be aware of where and how you are running your cluster. For example, if you’re running a cluster in AWS, cluster autoscaler needs IAM permissions to manage the autoscaling groups for your worker nodes, where as if you’re running in Azure, you’ll need to create a service principal in your Subscription with at least the “Contributor” role.

Cluster autoscaler works by looking at the resources of all the nodes and all the pods, and determining if nodes need to be added to support the current number of pods (or unschedulable pods because of a lack of resources), or if entire nodes can be removed because they are either unused or their workload could be moved to other nodes.

Cluster Autoscaler is powerful because it doesn’t simply scale on the CPU Utilization % on the worker nodes, but performs pod scheduling simulations to see where pods could be scheduled if it were to add or remove nodes.

Autoscaling Caveats

Cost

Since your infrastructure usage can fluctuate day to day or even minute to minute, your costs could vary greatly. Likely for any company already running applications in the cloud, you are used to the pay-as-you-go model, but its still something to take into consideration. Month to month your cloud bill could look quite different

Reasonable Limits

When setting up autoscaling, you are likely operating in an elastic cloud environment that could scale up “infinitely”. Due to this, an issue with your autoscaling definitions, a bug, or a force majeure could cause your cloud bill to sky rocket unexpectedly. To prevent this, you should always place reasonable maximums on your autoscaling policies to prevent situations like these. What is “reasonable” will differ between companies or even between applications at the same company, but don’t get caught with thousands of dollars in extra cloud spend because you let your cluster scale up to 100 nodes on accident!

Resource constraints

In the last paragraph, I put “infinitely” in quotes. This is more than me just saying that obviously there isn’t infinite computing power. Within your cloud provider, there will be limits that could prevent you from autoscaling, even if you wanted to. I speak mostly in an AWS context, but these are sure to apply to other clouds. Here is a non-comprehensive list of some limits you could run into:

IP Address exhaustion (Make sure your VPC/subnet CIDRs have enough IPs available!)
- Every pod in Kubernetes gets its own IP address. Depending on your networking setup, a cluster can easily eat up all the available IPs in your CIDR.
Availability Zone EC2 Capacity
- As hinted at before, the number of available EC2s running in a given Availability Zone is not infinite. It is possible to try to spin up an EC2 and have AWS reject your request because there isn’t capacity at the current moment.
- It can be useful to be able to use multiple instance types, as sometimes only a specific instance type can be at capacity
AWS Service limits
- In part to protect you from unexpected spend, and in part to protect them from people eating up a lot of resources unnecessarily, AWS has Service Quotas. If you hit limits on things like Network Interfaces or EC2 Instances, you will have to request a quota increase from AWS.
CPU/Memory
- If either you can’t or don’t autoscale your cluster, but do autoscale applications within the cluster, you can quickly exhaust the available cluster resources. In that scenario you might then be unable to schedule new pods and/or lower priority pods will be killed so that higher priority pods can continue to run smoothly.
Network Throughput
- If your applications scales up and is serving more and more traffic, you will need to ensure that the path that the traffic takes is also elastic enough to keep up with demand. This might mean needing beefier instances, scaling up load balancers etc so you don’t flood your own network.

I’m sure there are other nuances and caveats that I haven’t addressed, but these are some major ones that I am aware of or have personally experienced in the past. Keep them in consideration when planning out for a Kubernetes cluster that will involve autoscaling!

My CLI has been getting a little Rusty

Mon, 21 Sep 2020 00:00:00 GMT

I also made a video version of this post, if you’d rather watch than read.

Over the past few years I’ve adopted a handful of various CLI tools that I lean on frequently. Some of them I discovered on my own, some from coworkers, some from discussion topics during NYC CoffeeOps, a weekly meetup that I help co-organize. However one thing that I’ve noticed is that all of these tools happen to be written in Rust! Take from that what you will, but regardless of what language they’re written in, these tools are pretty neat, and I’d highly recommend them!

Note: I blatantly stole this idea from someone in the meetup who said this would make a good blog post. Thank you for your wonderful idea.

The tools

starship

Starship is the prompt I use for my shell. I love that basically with zero configuration, I get a really nice looking shell that integrates with all the tools/languages I use regularly. Alongside oh-my-zsh and iTerm2, it makes for a great terminal aesthetic.

lsd

Lsd is an alternative to ls. I like it because it includes colors and icons to more easily distinguish between types of files and directories

insert requisite drug joke here

ripgrep

Ripgrep is an alternative to grep. It searches text, but does it lightning fast. I once was trying to search a ~200GB dump of hacked usernames/passwords to ensure that none of my accounts had been compromised, and other methods of searching would have taken literal days, but ripgrep did the search in less than a minute.

Some other pros of ripgrep that I enjoy:

Obeys .gitignore files
Easily searches gzip archives

I use the latter feature occasionally to search logs automatically put into S3 by AWS. They’re usually categorized by date, so I can pull all the gzipped files for a specific day, search the archives, and only unzip and go into detail on files that match. Big time saver!

bat

bat bills itself as a ”cat clone with wings” The TL;DR on this one is:

Built in syntax highlighting and line numbers
Will automatically pipe to less if the output is large
Integration with git to show you if lines have been modified or added in the index

Need I say more?

fd

fd is an alternative to find. As with basically all of these Rust based tools, one of its biggest pros is that it is super fast. But in addition to that, I like using it because it has:

Sane defaults. I can just do fd foo instead of having to remember find -name "*foo*"
Color output
Like ripgrep, it ignores things in .gitignore by default

mcfly

This one is a hidden gem in my opinion. mcfly is an alternative to the default history search in your terminal when you press ctrl-r. I didn’t even know that was a thing until recently, and had always just used $ history | grep foo.

As with basically all the others, it has a nice color output, and shows a list of options instead of the single one that matches closest to what you’ve typed. Saves you a lot of time compared to grepping history, finding what you’re looking for, and then either copy/pasting, or using the !{num} shortcut. (Every entry in your history has a number next to it. If you type ! and then the number, your shell will rerun the command that was at that number in your history)

Dev Lessons Learned From Factorio

Sat, 15 Aug 2020 00:00:00 GMT

Factorio is one of my all time favorites games all about automation. You create an ever growing factory, with increasing complexity, dependencies, and threats. It probably shouldn’t be surprising, and some of these may seem fairly obvious but this game taught me many lessons about development. There is a lot of shared language between this game and development work in general. The lessons I’ve learned are:

Refactoring is inevitable
Modular code and shared libraries will save you a lot of time
Don’t let perfect be the enemy of the good
Prioritize your work and eliminate bottlenecks

Refactoring is inevitable

In Factorio, at some point or another, you’re going to refactor the work you’ve done before. You might tear it all up and start fresh or replace it with something else. Or maybe you’ll just upgrade your belts, inserters, and furnaces to the next tier. Theoretically some of your original factory might survive until you win or stop playing, but likely it will be contributing very little, and is probably just in the way.

In many ways the same can be said of code in large projects. Almost all parts of it will be refactored, rewritten, or upgraded, and that’s okay. It’s even necessary. Keeping code fresh can help improve performance, fix unknown issues, and keep developers familiar with different parts of the code base. I wasn’t able to find a source, but I swear I once saw that at Google, a project’s codebase gets completely rewritten every three to five years. Refactoring for refactoring sake is obviously not what you should be doing, but the point is to not put off refactoring if it is necessary or could help.

Modular code and shared libraries will save you a lot of time

In Factorio, there is a blueprint system that allows you to copy/paste parts of your factory. If you’ve managed to research and make construction robots, they’ll even place all the objects for you when you paste. What’s even better is when you create or find blueprints that are “tileable”, meaning you can place them side by side and keep expanding the factory with no additional modifications. As you can imagine, this saves you a lot of time and clicking when constructing your factory.

There are many ways this lesson applies to code as well. You might write a quick script to automate a step of your regular workflow that saves your team a lot of time. You also might create or use a loosely coupled interface for your logging that allows you to easily use/swap out different logging outputs, such as console output, writing to a file, or dumping logs into Elasticsearch. Small productivity wins can really compound to save you a lot of time. If you spend two hours automating a task that normally takes you 30 seconds, but it’s something everyone on your team of five does twice a day, you’ll make up the time in less than a month. If it’s a task that takes five minutes a day, you’ll make up the time in 3 days.

Don’t let perfect be the enemy of the good

So often in Factorio I found myself trying to make a perfect ratio setup that would be optimized to consume 100% of the input and ensure a constant steady flow of maximum output. However I’d take so long trying to make this perfect setup that a different part of my factory would turn into a larger issue and get neglected, eventually causing more problems. If instead I had just gotten the new setup working 50%, I could have dealt with the other parts of the factory and come back to optimize later on.

This pops up all the time when you have competing priorities when working on a product or feature. Someone will want to do the best job and make it the most efficient they possibly can. Someone else will want the feature out the door ASAP. Find your happy medium. Don’t deliver something broken, but don’t hold back others or neglect larger problems by trying to squeeze every ounce of performance possible out of your code, unless that’s something that’s explicitly needed. Come back and refactor in the future (see point one about refactoring).

Prioritize your work and eliminate bottlenecks

In line with the last point, in Factorio it’s really important to find what your current biggest issue or bottleneck is and address that first. It doesn’t make sense to scale up the production of green circuits when you’re not even maximizing current production because you don’t have enough copper wire to support the throughput needed. If you do, you’ll just have a bunch of assembling machines sitting idle, waiting for input resources.

This applies to so many things in software. I’ve been guilty of this myself, where I spend way too much time optimizing a system that doesn’t even need it, or making a feature that nobody wants. That time would be much better spent working on an issue that is actually affecting users, or on making a troubled system more scalable. This is where having monitoring of various kinds comes in handy. What actions are your users taking most often? Where is your application having performance problems? Spend time there instead of elsewhere.

Finding bottlenecks can have huge benefits to your team. If you don’t eliminate bottlenecks, eventually there will be a backup of work in front of the bottleneck. The more you keep things the same, the more things will backup. Map out your value stream and figure out where things are getting held up. Stop piling more things into the backup, and instead spend time helping things get through the bottleneck, or even improving the throughput of the systems. As an example, maybe you have a QA team that manually reviews every ticket that gets completed. When they first started, most tickets got completed in the same day, but over time, a larger and larger backlog of tickets has piled up. Now it can take upwards of two to three weeks for tickets to get reviewed. If the status quo continues, things will only continue to get worse. You may feel really productive, since you “complete” and pass on more tickets to QA than the rest of your team, but really you’re just exacerbating the problem! Your team should prioritize changing or improving the processes around QA to improve throughput. Can you automate tasks? Have developers rotate in to help out on QA?

Shamir's Secret Sharing

Fri, 15 May 2020 00:00:00 GMT

When I was a kid, I had the opportunity to tour the Titan II Missile Museum in Arizona. It’s the last of the 54 silos across the U.S. that hasn’t been destroyed or filled in, and instead was turned into a museum for the public to learn about the history of the Cold War and how these silos operated. In part of the tour of the museum, they show you the control room, and mock out what would have happened if the silo had ever received the command to launch. One step of the process is that two separate individuals have to insert a key into panels on opposite sides of the room and turn them at the same time, ensuring that a launch could not be completed by a single individual (unless you’re Mrs. Incredible).

There are a few situations where the stakes are so high that we don’t want to trust a single person to have the full power to make a decision or act unilaterally. Broadly, this is referred to as the Two-person rule, and has lots of uses. The nuclear launch two key setup is just a single example of that rule in action, but it is used in banking, hazardous materials handling, aircraft piloting, and more.

In cryptography, the two-person rule can be enforced using Shamir’s Secret Sharing (SSS), a mathematical algorithm that reconstructs a secret by bringing together multiple parts (Adi Shamir is a famous cryptographer known for creating SSS and being one of the three creators of the RSA public key encryption scheme). SSS even has the ability to not just split into two, but can split a secret into any number of parts and can require a different number of those parts to be present to recreate the whole. Hashicorp’s Vault, a solution for securing and storing secrets, tokens, passwords, etc., supports using SSS to unseal the Vault. By default, Vault will generate 5 keys and requires at least 3 to be provided in order to unseal the Vault. This means that if you give each of the keys to a different person, it would require 3/5ths of the group to agree to unseal the Vault. No one or even two people could do it on their own.

The general principle of SSS relies on some basic concepts in math. To define a line, you need two points, (1,2) (3,7) for example. With only one of these points, there are an infinite number of lines that pass through the single point. For a parabola, you need three points. Four points for a cubic curve and so on. For a secret split into n parts where you require k pieces to be present to recreate the secret, SSS essentially generates a polynomial of degree k - 1, and then chooses n random points on the curve to distribute as the parts.

Let’s say like Vault, you want to generate n keys (in our case 5), and require k (in our case 3) of them to be present to recreate the original secret. SSS would use the original Vault encryption key as part of the curve, and then generate a random parabola (ax^2 + bx + c). Then it will randomly pick 5 points on that curve. This is a simplification of the math, especially since it doesn’t actually use basic integers, but rather a finite field. But since that’s over my head, it’s much easier to conceptualize using the simpler algebra you might have learned in school.

I find this algorithm insanely cool. I don’t know if there are any mystery novels out there that have SSS as a basis, but you could easily have a Knives Out style drama where the genius computer millionaire dies, but secretly leaves their children each a part of the secret, and enough of them have to agree to unlock their inheritance. Or maybe a spy novel where the latitude and longitude coordinates of spots on the globe represent the coordinates of the polynomial curve and can be used to save the world.

One downside to SSS is that when contributing the parts of the secret to try to recreate the whole, there is no way to verify that a part is valid. Meaning someone could contribute fake data and either prevent the reconstruction of the original secret, or even make the reconstructed data be something else entirely. I think this plays well in my spy novel idea, where an unexpected betrayal happens and a double agent is revealed. This whole concept is a Dan Brown novel just waiting to happen.

Node 14 New JavaScript Feature Spotlight

Mon, 27 Apr 2020 00:00:00 GMT

Node.js version 14 was released recently, and will become active under Long Term Support (LTS) version in fall of 2020 (October 20th to be specific). There are a handful of changes, but to the average developer, I think a few are worth highlighting as they might change the JavaScript that gets read and written.

Nullish Coalescing

Nullish Coalescing is similar to the || operator, but it only returns the right hand value when the left hand side is nullish, instead of falsy.

/**
 * Zero is considered falsy and therefore
 * the right hand side value is returned
 * when using the `||` operator
 */
const foo = 0 || "oops" // "oops"
const bar = null || "oops" // "oops"

/**
 * Zero is not considered nullish
 * and therefore is returned when on
 * the left hand side of the `??` operator
 */
const baz = 0 ?? "oops" // 0
const buzz = null ?? "oops" // "oops"

Falsy

Falsy is a term that means a value that is considered to be false when treated as a boolean. In JavaScript, the following values are falsy:

null
undefined
NaN (Not a number)
0
-0 (yes that’s right, negative zero)
0n (BigInt zero)
"" (the empty string)
false (You’d really hope)

These values being falsy can be simultaneously convenient, but also a pain if forgotten. It is common in JavaScript to pass a non-boolean value to an if statement, and let JavaScript check if the value is truthy or falsy. But if you unintentionally pass in 0 or "", you might encounter unexpected behavior.

if (1) // true
if (0) // false

This is why passing unsanitized input directly to an if to check for the presence of value is an issue

Nullish

Similar to falsy, there are multiple values that are considered nullish. These values are:

null
undefined

This means that these values are both nullish and falsy.

Optional Chaining

Optional Chaining is a new feature that allows you to reference a field nested within several objects, but without having to check that each object isn’t nullish.

Let’s take the following object as an example:

const users = {
  Jack: {
    phone: "(123) 456-7890",
    address: {
      street: "6732 Main St",
      zip: "12345",
    },
  },
  // Notice jill has no address
  Jill: {
    phone: "(349) 233-2348",
  },
}

In the past, if we wanted to get the zip code, we would have done something like:

const jack = users.Jack
let zip = null
if (jack) {
  const address = jack.address
  if (address) {
    zip = address.zip // "12345"
  }
}

let zip = null
if (users.Jack && users.Jack.address) {
  zip = users.Jack.address.zip // "12345"
}

or if you really want to do it in one line:

const zip = users.Jack && users.Jack.address && users.Jack.address.zip

All of these are valid, but pretty verbose. With optional chaining you can now do the same thing still in one line, and without the use of if, &&, and without repeating any fields:

const zip = users.Jack?.address?.zip // "12345"

?. replaces plain old . and short circuits if the value is nullish. In the above example, Jack has an address and a zip, but Jill doesn’t. To check for Jill’s zip code, its the same thing:

const zip = users.Jill?.address?.zip // undefined

If you tried this line without the ?. operator, it would throw an error!

One of the great things about Optional Chaining is that it works with just about everything. Not just object properties, but also arrays and function calls too.

function asyncHelloCallback(name, callback) {
  const helloString = `Hello, ${name}!`
  setTimeout(() => {
    console.log(helloString)
    // If callback is undefined, there's no error
    callback?.(helloString)
  })
}

function getFifthElement(arr) {
  // Doesn't throw an error if arr is undefined
  return arr?.[4]
}

Combining Optional Chaining and Nullish Coalescing

Combined, these two features can be quite the powerhouse.

// Check for value and provide a default if Nullish
const middleName = users.Jack?.middleName ?? ""
const quantity = order.items?.[0].quantity ?? 3

/**
 * Use multiple nullish coalescing and
 * optional chaining operators to check
 * several values before providing a default
 */
const userNames = [
  {
    name: {
      first: "John",
      middle: "B",
      last: "Doe",
    },
  },
  {
    name: {
      last: "Smith",
    },
  },
  {
    name: {
      middle: "A",
    },
  },
  {},
]
const defaultName = user =>
  user?.name?.first ?? user?.name?.last ?? user?.name?.middle ?? "Unknown"

// [ 'John', 'Smith', 'A', 'Unknown' ]
const defaultNames = userNames.map(defaultName)

Frustrations without static types

Mon, 13 Apr 2020 00:00:00 GMT

Recently I was working on a piece of my company’s Python code base that I hadn’t touched before. It was part of an automation script that makes some API calls to GitHub using the Requests library.

I needed to check some of the parameters on the response, run some different functions based on the response code and various other checks. But without static types, there was no easy way to know exactly what parameters the response object had. Surely it had a status code of some kind right? But was it:

response.code
response.status
response.statusCode
response.status_code
response.statuscode

Without static types there was no easy way for me to know. Even if I happened to remember that for this specific library the answer is response.status_code, is it a number? Is it a string? What about other values on the response? In order to figure all this out, I had to dig into the docs and source code of the requests library.

Python has a lot of benefits, and I use it daily, but static typing is not definitely not one of them. With one of any of the many statically typed languages and a decent IDE or text editor, I would have been able to write the checks that I needed without having to do any external investigation. In my spare time I’ve been working on a little bit of Go, and the presence of static types is a breath of fresh air from the Python that I write at work, and the JavaScript that I used at my previous job.

Clearly static types are not the only thing to consider when picking a language, but this is a clear, albeit simple, example of some frustrations of when you don’t have them.

Environment Variables

Fri, 03 Apr 2020 00:00:00 GMT

Inspiration

I was inspired to write the post by an acquaintance who is new to the software industry. They couldn’t find any good resources on environment variables that covered everything in one place. They wanted to know things like:

What are environment variables?
What kind of things should be in environment variables?
How can I set environment variables?
How can I read environment variables?
Do only secret things go in them?
How do I securely share them?

Hopefully this post can become a kind of one stop shop for an introduction into environment variables.

What are Environment Variables?

Environment variables, commonly referred to as env vars, are variables available to a process that are a part of the environment where the process runs. Lazy to define the word with the word, so let’s break it down a little bit more.

A variable represents a value that can change. An environment variable typically represents a value set from outside the application, and exists as a part of system that is running the application. Even if you’re not aware of it, every time you use your computer, environment variables are impacting the applications you run. The browser that you are reading this blog post on probably uses hundreds of them. Some common examples are:

PATH (When you open a terminal and type a command, where should it look for the executable you are trying to run)
HOME (The home directory for the current user)
ENVIRONMENT (commonly used to say either “development” or “production” to specify where the app is running.)

Env vars keys by convention are written in all capital snake case, meaning multiple words in the key are separated by underscores. The value is a string. E.g.

THIS_IS_A_VAR_NAME=foobar

Environment variables exist within your system whether or not you use them in your applications. If you open a terminal and type the following, you’ll see the content of the environment variable PATH.

Mac OS/Linux

echo $PATH

Windows

echo %PATH%

What kind of things should be in environment variables?

There are many environment variables that will be set automatically by the system, but you can also set environment variables yourself specific to your application. Commonly you would set things that change based on where the app is running. These might be things such as:

Database credentials
Whether your app is running in Dev/Prod
What port your web application should run on

Some environment variables are secret and sensitive, like your database credentials. Others, like the port or dev/prod flag are not.

How can I set environment variables?

Ah the tried and true answer: “It depends”

In Unix based systems, like Linux or Mac OS, environment variables can be set for every shell session, the current shell session only, or exclusively for a single command:

Can be set for the shell session automatically (.bashrc/.zshrc etc)
Can be set temporarily manually export MY_VAR=foo
Can be set for a single command MY_VAR=foo ls

On Windows things are a little bit different. You can set in the environment variables settings for either the whole system or just current user, or in the terminal, use the command SET MY_VAR=foo. It’s different again if you’re using Powershell.

It is important to note that processes inherit the environment variables of their parent processes. However if all you need is to set some environment variables for your application, a very common convention is to put key value pairs in a a file called .env (pronounced: “dot ee en vee”. Or at least that’s how I say it) that lives alongside your project files, but is not committed to git. Many frameworks will automatically read from this file if it exists and set the environment variables for your process. If you aren’t using a framework that does this, many languages have libraries (usually called ”dotenv” or something similar) that will do the same. Some teams will have multiple .env files with different content, and might prefix them with their intended use, like sample.env, test.env, or dev.env.

A common question is how to properly set environment variables when running in CI systems like GitHub Actions or CircleCI. These systems run your build/test/deploy steps in their own isolated, ephemeral (temporary) container, and therefore they might lack some of the same environment variables that you have on your local development environment, and have many extras that you don’t. You should read into the docs of your CI system to see how to properly set environment variables and figure out which ones are already made available to you.

How can I read environment variables?

Here’s a smattering of examples of how to read environment variables in a few programming languages:

JavaScript:

process.env.MY_VAR

Using JavaScript fun, to provide a default you can use

process.env.MY_VAR || "some default value"

Python:

import os
os.environ["MY_VAR"]

os.environ returns a dict. Python dicts raise a KeyError if you try to access a key that doesn’t exist, so it tends to be better to actually do something like os.environ.get("MY_VAR", "some default value") to prevent that situation.

import "os"
os.Getenv("MY_VAR")

Unlike other languages, Go returns an empty string if the key doesn’t exist. Make sure to take that into consideration!

Java

System.getenv().get("MY_VAR")

getenv() returns a Map<String, String>, and therefore a get for a key that doesn’t exist returns null

Rust

env::var("MY_VAR")

Because it’s Rust, env::var returns Result that could contain an Ok variant or an Err variant. You’ll need to handle that with either an expect, unwrap, or is_err on the Result.

If the value doesn’t exist, languages do different things. Some return undefined (JavaScript). Others throw an error. Whatever language you are using, make sure you properly handle the case where the value does not exist!

Environment Variables and Secrets

Not all environment variables are secrets! Many of them are totally fine to share, post publicly etc. However, some of them inevitably will be sensitive. As soon as you introduce a database, the credentials to that database will likely be set as environment variables and need to be kept secret. However secrets don’t have to be environment variables. There are other options too:

Encrypted files (fun fact: Kubernetes “secrets” aren’t actually encrypted by default. Just base64 encoded. You have to add the encryption at rest by yourself.)
Fetched from some external secret store (Hashicorp Vault, AWS Secrets Manager, etc)

You will need to find the option that best suits you and your organization while still maintaining security best practices.

Don’t share secrets in Slack. Please. Just don’t. Use one of the myriad of options for sharing things securely:

GPG
Password Managers (1Password, LastPass, BitWarden, etc)
Firefox Send
Send Safely
Signal
Use one of the above secret stores (Vault, Secrets Manager)

I’m sure there are a million other options I didn’t include. This is just a few one of the ones I’m familiar with.

What if I really want to commit my secrets to git?

I really don’t recommend it, but if you absolutely must, you should at least use git-crypt or something similar so the file is encrypted!

What if I really want to just put the secret in a Slack message?

It’s just so easy to quickly send it over right? And you’ll definitely delete the message right after the person has a chance to copy it, right? There’s no way anybody might somehow get a hold of that secret, and you totally trust Slack that your messages will never be lost or read by someone else!

BAD!! STOP IT!

Seriously people, don’t do this!! There are so many good ways to send things securely, so stop posting plain text secrets in Slack, no matter how badly you want to. No matter how much of a rush you’re in. No matter how lazy you are.

Right now go download Signal and make an account. Go generate a GPG key pair and post your public key so people can send you encrypted messages. Do something encrypted. Anything! Just please don’t post it in plain text in Slack.

Merging and Overriding IAM Policies in Terraform

Mon, 30 Mar 2020 00:00:00 GMT

TL;DR

Use the source_json and/or override_json attributes on a Terraform aws_iam_policy_document.

data "aws_iam_policy_document" "foo" {
    source_json = data.aws_iam_policy_document.bar.json
}

# OR

data "aws_iam_policy_document" "buzz" {
    override_json = data.aws_iam_policy_document.fizz.json
}

Summary

When creating IAM policies in AWS, it can be really easy to either:

Give things way too many permissions because you’re lazy and don’t want to repeat yourself
End up repeating things a lot.

When using Terraform, you can get the best of both worlds by merging disparate policy documents to both avoid repeating yourself and limit permissions. This capability is especially useful when creating modules or useful base policies. You could create a base policy that enforces the best practices of your organization and use that as the source_json on every aws_iam_policy_document. Or if there is already a policy document in place, but you want to override specific statements, you can do that too!

You could accomplish some of the same things with plain JSON in a heredoc and some string interpolation, but I’ve found it to be much simpler to actually use the policy document data source.

Source vs. Override

The difference between source_json and override_json is a question of which of the two policies you want to take priority. When using source_json, the source is imported, and the current policy document will override anything in the source with a matching statement ID. When using override_json, the current policy document is overridden by the imported policy.

Note that statement IDs (sid) have to match in order to actually override any statements. Otherwise they will be merged.

Examples

Given the following policy document, let’s look at a practical example of source_json versus override_json.

data "aws_iam_policy_document" "a" {
    # Allows reading from all buckets
    statement {
        sid = "1"
        actions = ["s3:GetObject"]
        resources = ["arn:aws:s3:::*"]
    }

    # Allow put object in "some-bucket"
    statement {
        sid = "2"
        actions = ["s3:PutObject"]
        resources = ["arn:aws:s3:::some-bucket/*"]
    }
}

Using source_json

data "aws_iam_policy_document" "b" {

    source_json = data.aws_iam_policy_document.a.json
    # Allows reading from a specific bucket
    statement {
        sid = "1"
        actions = ["s3:GetObject"]
        resources = ["arn:aws:s3:::some-bucket/*"]
    }

    # Allows put object in "a-different-bucket"
    statement {
        sid = "3"
        actions = ["s3:PutObject"]
        resources = ["arn:aws:s3:::a-different-bucket/*"]
    }
}

# Final output for "b" is GetObject only for "some-bucket", but PutObject on both "some-bucket" and "a-different-bucket"

Using override_json

data "aws_iam_policy_document" "b" {

    override_json = data.aws_iam_policy_document.a.json
    # Allows reading from a specific bucket
    statement {
        sid = "1"
        actions = ["s3:GetObject"]
        resources = ["arn:aws:s3:::some-bucket/*"]
    }

    # Allows put object in "a-different-bucket"
    statement {
        sid = "3"
        actions = ["s3:PutObject"]
        resources = ["arn:aws:s3:::a-different-bucket/*"]
    }
}

# Final output for "b" is GetObject for all buckets, and PutObject on both "some-bucket" and "a-different-bucket"

See the Terraform Docs for more information.

Rotating Docker Logs — Keeping your overlay folder small

Tue, 14 Aug 2018 00:00:00 GMT

TL;DR

Learn from my mistakes. If you are using the default json-file driver, configure max-file and max-size options to have Docker automatically rotate and remove old logs.

The Problem

4:30 pm. Time to go home, figure something out for dinner, relax and — — THE BUILDS ARE FAILING. Classic.

Can’t push or pull to our self-hosted registry. The container is running, so why would it not be able to read/write to the disk unless…the disk is full?

After some quick digging, turns out the Docker overlay folder is big. Like almost 200 GB big. But a quick glance shows that we only have 4 images and a few small volumes, so why would that overlay folder be so big?

The answer: logs. a LOT of logs.

The Fix

By default, Docker uses the json-file log driver and saves those logs in the overlay folder. Docker doesn’t limit the size of the files or how many log files there can be for a single container. If you continue to run that container, the log file will grow as long as it has drive space to do so. Thankfully, it’s pretty easy to set these limits.

If you are running from the CLI, your option looks like:

docker run --log-opt max-size=10m --log-opt max-file=5 my-app:latest

This sets a maximum of five log files with a max size of 10 Mb each. So at most 50 Mb of logs for that container. Tune those numbers as you see fit.

If you are using docker-compose, your options should look like:

my-app:
    image: my-app:latest
    logging:
        driver: "json-file"
        options:
            max-file: 5
            max-size: 10m

You can also configure this as a default in your daemon settings https://docs.docker.com/config/containers/logging/configure/#configure-the-logging-driver-for-a-container

MongoDB Atlas Is Awesome

Mon, 28 Aug 2017 00:00:00 GMT

MongoDB is really fast and easy to get setup with, and therefore is a favorite for many developers who want to get a running start into their new application. However, because of this, many developers don’t take the time to set up replica sets (that provide high availability (HA) and automatic failover) or role based access control (RBAC). For local development, this might not be necessary, but when working in a team environment, or whenever you have a publicly accessible MongoDB instance, these should be required. And there really is no excuse to NOT have it.

Atlas is a service by the creators of MongoDB that provisions and creates a MongoDB cluster with a replica set, users with RBAC and encryption at the click of a button for free. It also provides metrics and alerts for your cluster. Now the free tier only gets you 512 MB of storage, but this should be plenty for your development environment. Once you get to production, upgrade that same cluster to handle the load of your production application. You might decide to setup your own production environment in order to avoid the premium cost, but for the rapidly working developer, the free tier is amazing.

I just recently discovered Atlas, and I will never waste my time trying to setup my own development environment again. So far it has “just worked”, and when trying to develop quickly, that goes a long way.

Specifying npm Version in Dockerfile

Mon, 21 Aug 2017 00:00:00 GMT

TL;DR

RUN npm i npm
RUN mv node_modules/npm /usr/local/lib/node_modules/npm
RUN rm -rf /usr/local/lib/node_modules/npm

Details

Each Node.js version ships with npm. Currently the LTS Node version (v6.11.2) ships with npm v3.10.10. But what if you want to use the latest features of npm 5.x.x with an older version of Node?

Outside of a container, this is simple. You install the Node version that you want (use nvm to easily swap between them) and then specify npm version.

npm i -g npm@5.3.0 //specific version
npm i -g npm       //for latest

Doing this inside a container, while possible, is a little bit trickier. You declare the version of Node you want to use with the “FROM” command.

FROM node:boron

But this comes with an old version of npm. If you try to upgrade as you normally would, you’ll run into an issue where the package “semver” can’t be found. The npm installation is bad. The solution:

RUN npm install npm@5.3.0
RUN rm -rf /usr/local/lib/node_modules/npm
RUN mv node_modules/npm /usr/local/lib/node_modules/npm

Install npm in the project, remove the global package, then move the local package to your global location. Now you have the Node version specified at the top, but the npm version you want, not the one bundled with Node.

Everything You Wanted To Know About package-lock.json But Were Too Afraid To Ask

Fri, 11 Aug 2017 00:00:00 GMT

Introduction

So you’ve updated Node Package Manager (npm) to v5.x.x, and everything seems to be going fine. But wait, what’s this? A new file was created automatically. package-lock.json. If you open it, it looks sort of like the dependencies in package.json, but more verbose. You decide to ignore it and go along your way developing your project. Eventually, you run into problems with a dependency. It can’t be found or the wrong version seems to be installed. Most people just end up deleting the package-lock.json and running npm install. So why even have it? What is it supposed to do? What does it actually do?

Summary

If you’re using npm ^5.x.x, by default a package-lock.json will be generated for you
You should use package-lock to ensure a consistent install and compatible dependencies
You SHOULD commit your package-lock to source control
As of npm ^5.1.x, package.json is now able to trump package-lock.json, so you should experience much less of a headache
No more deleting that package-lock just to run npm install and regenerate it;
Use semver if your app offers an API, and adhere to the rules of semver.

Background

Semantic Versioning

Before you can understand the package-lock and even package.json, you have to understand semantic versioning (semver). It’s the genius behind npm, and what has made it even more successful. You can read more about how npm uses it here. In a nutshell, if you are building an application with which other applications interface, you should communicate how the changes that you make will affect the third party’s ability to interact with your application. This is done via semantic versioning. A version is made up of three parts: X,Y,Z where those are major, minor and patch versions respectively. An example would be 1.2.3, or major version 1, minor version 2, patch 3. A change in patch represents a bugfix that doesn’t break anything. A change in minor version represents a new functionality that doesn’t break anything. A change in major version represents a large change that breaks compatibility. If users don’t adapt to a major version change, stuff won’t work.

Managing Packages

npm exists to make managing packages easy. Your projects might have hundreds of dependencies, each of those with a hundred others. To keep your mind away from dependency hell, npm was created so that with some simple commands, you could install and manage these dependencies and hardly ever have to think about them.

When you install a package with npm (and save it), an entry is added to your package.json containing the package name, and the semver that should be used. npm supports some wildcards in this semver definition however. By default, npm installs the latest version, and prepends a caret e.g. “^1.2.12”. This signifies that at a minimum, version 1.2.12 should be used, but any version higher than that is OK, as long as it has the same major version, 1. Since minor and patch numbers only represent bugfixes and non-breaking additions, you should be safe to use any higher same-major version. Read more about semver wildcards and play with npm’s cool semver calculator here.

Shared Projects

The real benefit to having dependencies defined like this in package.json, is that anybody who has access to the package.json could create a dependency folder that contains the modules needed to run your application. But let’s take a look at a specific way where things might go wrong.

Let’s say we create a new project that is going to use express. After running npm init, we install express: npm install express — save. At the time of writing, the latest express version is 4.15.4. So “express”: “^4.15.4” is added as a dependency within my package.json and that exact version is installed on my machine. Now maybe tomorrow, the maintainers of express release a bug fix, and so the latest version becomes 4.15.5. Then if someone were to want to contribute to my project, they would clone it, and run `npm install.’ Since 4.15.5 is a higher version with the same major version, that is installed for them. We both have express, but we have two different versions. Theoretically, they should still be compatible, but maybe that bugfix affected functionality that we are using, and our application would produce different results when run with express version 4.15.4 compared to 4.15.5.

Package-lock

The Goal

The purpose of the package-lock is to avoid the situation described above, where installing modules from the same package.json results in two different installs. Package-lock.json was added in npm version 5.x.x, so if you are using major version 5 or higher, you will see it generated unless you disabled it.

The Format

Package-lock is a large list of each dependency listed in your package.json, the specific version that should be installed, the location of the module (URI), a hash that verifies the integrity of the module, the list of packages it requires, and a list of dependencies. Let’s take a look at what the entry for express is:

"express": {
      "version": "4.15.4",
      "resolved": "https://registry.npmjs.org/express/-/express-4.15.4.tgz",
      "integrity": "sha1-Ay4iU0ic+PzgJma+yj0R7XotrtE=",
      "requires": {
        "accepts": "1.3.3",
        "array-flatten": "1.1.1",
        "content-disposition": "0.5.2",
        "content-type": "1.0.2",
        "cookie": "0.3.1",
        "cookie-signature": "1.0.6",
        "debug": "2.6.8",
        "depd": "1.1.1",
        "encodeurl": "1.0.1",
        "escape-html": "1.0.3",
        "etag": "1.8.0",
        "finalhandler": "1.0.4",
        "fresh": "0.5.0",
        "merge-descriptors": "1.0.1",
        "methods": "1.1.2",
        "on-finished": "2.3.0",
        "parseurl": "1.3.1",
        "path-to-regexp": "0.1.7",
        "proxy-addr": "1.1.5",
        "qs": "6.5.0",
        "range-parser": "1.2.0",
        "send": "0.15.4",
        "serve-static": "1.12.4",
        "setprototypeof": "1.0.3",
        "statuses": "1.3.1",
        "type-is": "1.6.15",
        "utils-merge": "1.0.0",
        "vary": "1.1.1"
      }
    },

Equivalent entries can be found for every package listed in the “requires” section.

The idea then becomes that instead of using package.json to resolve and install modules, npm will use the package-lock.json. Because the package-lock specifies a version, location and integrity hash for every module and each of its dependencies, the install it creates will be the same, every single time. It won’t matter what device you are on, or when in the future you install, it should give you the same result every time, which is very useful.

The Controversy

So if package-lock is supposed to solve a common problem, why are the top search results for it (other than npm documentation) all about disabling it or questioning the role that it plays?

Before npm 5.x.x, package.json was the source of truth for a project. What lived in package.json was law. npm users liked this model and grew very accustomed to maintaining their package file. However, when package-lock was first introduced, it acted contrary to how many people expected it to. Given a pre-existing package and package-lock, a change to the package.json (what many users considered the source of truth) was not reflected in the package-lock.

Example: Package A, version 1.0.0 is in the package and package-lock. In package.json, A is manually edited to version 1.1.0. If a user who considers package.json to be the source of truth runs npm install, they would expect version 1.1.0 to be installed. However, version 1.0.0 is installed, despite the fact that v1.1.0 is listed is the package.json.

Example: A module does not exist in the package-lock, but it does exist in the package.json. As a user who looks to package.json as the source of truth, I would expect for my module to be installed. However since the module is not present in package-lock, it isn’t installed, and my code fails because it cannot find the module.

Much of the time, because they couldn’t figure out why their dependencies weren’t being installed correctly, users either deleted package-lock and reinstalled, or would disable the package-lock altogether.

This conflict between expect and real behavior sparked a very interesting issue thread in the npm repo. Some people thought that the package.json should be the source of truth, some people thought that since package-lock is what npm uses to create the install, that should be considered the source of truth. The resolution to this controversy lies in npm PR #17508. Npm maintainers added a change that causes package.json to overrule the package-lock if package.json has been updated. Now in both above scenarios, the packages that a user would expect to be installed are installed correctly. This change was released as a part of npm v5.1.0, which went live on July 5th, 2017.

James Quigley's Blog

Kubernetes Autoscaling Overview

Table of Contents

Summary

Autoscalers

Horizontal Pod Autoscaler (HPA)

Vertical Pod Autoscaler (VPA)

Cluster Autoscaler

Autoscaling Caveats

Cost

Reasonable Limits

Resource constraints

My CLI has been getting a little Rusty

The tools

starship

lsd

ripgrep

bat

fd

mcfly

Dev Lessons Learned From Factorio

Refactoring is inevitable

Modular code and shared libraries will save you a lot of time

Don’t let perfect be the enemy of the good

Prioritize your work and eliminate bottlenecks

Shamir's Secret Sharing

Node 14 New JavaScript Feature Spotlight

Nullish Coalescing

Falsy

Nullish

Optional Chaining

Combining Optional Chaining and Nullish Coalescing

Frustrations without static types

Environment Variables

Inspiration

What are Environment Variables?

What kind of things should be in environment variables?

How can I set environment variables?

How can I read environment variables?

Environment Variables and Secrets

What if I really want to commit my secrets to git?

What if I really want to just put the secret in a Slack message?

Merging and Overriding IAM Policies in Terraform

TL;DR

Summary

Source vs. Override

Examples

Using source_json

Using override_json

Rotating Docker Logs — Keeping your overlay folder small

TL;DR

The Problem

The Fix

MongoDB Atlas Is Awesome

Specifying npm Version in Dockerfile

TL;DR

Details

Everything You Wanted To Know About package-lock.json But Were Too Afraid To Ask

Introduction

Summary

Background

Semantic Versioning

Managing Packages

Shared Projects

Package-lock

The Goal

The Format

The Controversy