Will Murphy's personal home page

I Guess I'm Both?

Recently, fly.io announced that adding GPUs to their platform isn’t really paying off1, and, unsurprisingly, there was much discussion on Hacker News2. In said discussion, the top comment divides developers into two camps:

I’m increasingly coming to the view that there is a big split among “software developers” and AI is exacerbating it. There’s an (increasingly small) group of software developers who don’t like “magic” and want to understand where their code is running and what it’s doing. These developers gravitate toward open source solutions like Kubernetes, and often just want to rent a VPS or at most a managed K8s solution. The other group (increasingly large) just wants to git push and be done with it, and they’re willing to spend a lot of (usually their employer’s) money to have that experience. They don’t want to have to understand DNS, linux, or anything else beyond whatever framework they are using.3

There’s a little bit of typical, “real men write assembly and like it” Hacker News BS in here, but what I want to talk about today is that I am both kinds of developer:

  1. I want to know why the stuff works. I’ve worked on CloudFoundry and on internal developer tooling at AWS. That’s way behind the back-end. When I’m understanding a new system, I like to eventually get it down to the level of processes and system calls. Like, at some point some Linux server is going to write this JSON to a socket. Where’s that happen?
  2. I hate futzing around when I’m trying to deploy. My litmus test for whether you’re doing CI/CD right is two questions: Can you deploy in one step? Is that step git push?

This comment has some grains of truth to it, but I still feel like the “developers who like magic” contingency is a little slighted. First, the grains of truth:

Magical Thinking can hurt software engineering

I almost wrote a blog post on this, but Eric Lippert said it first and better:

https://ericlippert.com/2015/06/04/its-not-magic/

Although I admit that fly.io feels pretty magical4, let’s (speculatively) peel back the curtain here:

  1. Fly.io has a subdomain under fly.dev for many projects. That means there’s a key-value store somewhere that’s like, “hey, when I hit taskq.fly.dev, this OCI image is the one that gets the request.” I don’t know exactly how they built this, but it’s not magic. Maybe it’s Redis? Or etcd?
  2. Even paused fly.io machines come back really fast, and they’ve mentioned they use firecracker. So there’s some snapshotting of a running process and then re-inflating it on demand. This is really cool, and it involves solving some tricky problems like making sure that recently rehydrated snapshots don’t issue the same random numbers or timestamps they would have before snapshotting.
  3. They must have some cool process isolation, probably also from firecracker.

So this isn’t magic. It’s good software engineering. But what I like about it is that it feels magical.

Some developers don’t want to pull back the curtain

freedomben is also right that sometimes folks don’t want to (or can’t) peel back the curtain. I’ve interviewed candidates where it was abundantly clear that, although they could make the .NET framework sing and dance, if we took it away, they wouldn’t know how to build much. And to be fair, this is what Microsoft is selling when they get you on .NET: your company can deploy website with custom functionality without having to hire for the broad skillset that someone like a Google SRE needs.

I want to understand the magic. But I also want to use it.

Where I disagree with freedomben is in the claim that both camps are separate. I’m in both camps. I love knowing how things actually work. Getting to peer behind the curtain on various infrastructure and platform products is something I really loved about my time at both CloudFoundry and AWS.

But man when I want to go fast and ship features, the magic helps. I love using magic. I just want to understand it to.

Anyway, I guess I won’t take a hard dependency on fly.io GPUs any time soon. RIP.

Till next time, happy learning!
–Will

  1. https://fly.io/blog/wrong-about-gpu/ ↩︎

  2. https://news.ycombinator.com/item?id=43053844 ↩︎

  3. https://news.ycombinator.com/item?id=43054389 ↩︎

  4. fly.io feels pretty magical ↩︎

Discussion

Love this post? Hate it? Is someone wrong on the internet? Is it me? Please feel free to @me on mastodon

I used to host comments on the blog, but have recently stopped. Previously posted comments will still appear, but for new ones please use the mastodon link above.