Featuring

Sponsors

Namespace – Speed up your development and testing workflows using your existing tools. (Much) faster GitHub actions, Docker builds, and more. At an unbeatable price.

Depot – 10x faster builds? Yes please. Build faster. Waste less time. Accelerate Docker image builds, and GitHub Actions workflows. Easily integrate with your existing CI provider and dev workflows to save hours of build time.

Squarespace – A website makes it real! Use code CHANGELOG to save 10% on your first website purchase.

Notes & Links

📝 Edit Notes

Chapters

1 00:00 Let's Kaizen! 00:38
2 00:38 Sponsor: Namespace 01:41
3 02:18 Killer starts & Friends 05:23
4 07:41 MacTuner and Mole 04:04
5 11:46 What's next? 00:33
6 12:19 Erlang in Anger 04:29
7 16:48 Pipedream crashing 03:16
8 20:04 Quiz! How many crashes? 01:58
9 22:02 Quiz! Why the crashes? 05:36
10 27:38 Sponsor: Depot 02:20
11 29:58 Quiz! Why more in traffic than out? 02:31
12 32:29 90s jean shopping 00:43
13 33:12 What on earth is going on 03:20
14 36:32 Hot vs cold regions 02:30
15 39:02 What improved? 01:39
16 40:41 Examining an instance 02:14
17 42:55 Gerhard's prompt 02:38
18 45:33 Claude's analysis 07:09
19 52:42 Gemini's analysis 04:10
20 56:52 Sponsor: Squarespace 01:56
21 58:48 When all the "fun" begins 04:41
22 1:03:29 Checking in hourly 01:31
23 1:05:00 Thanks mayailrus! 01:09
24 1:06:09 Checking in locally 06:43
25 1:12:52 All sorts of issues 01:03
26 1:13:55 That was the easy stuff... 00:19
27 1:14:13 "It's complicated" 01:16
28 1:15:29 It's just too much 02:44
29 1:18:13 Knock on some doors 02:08
30 1:20:21 Mitigation 03:16
31 1:23:37 Food for thought 01:04
32 1:24:41 More mitigation ideas 04:06
33 1:28:47 How good systems become bad systems 00:48
34 1:29:35 Jerod's cudgel 04:57
35 1:34:33 Robust observability 00:20
36 1:34:53 One last thing 04:40
37 1:39:33 Bye, friends 00:40
38 1:40:13 Next week on the pod 00:53

Transcript

📝 Edit Transcript

How else would you learn? Let it crash.

Exactly. The best things happen when things fail… [laughter] Seriously. If it’s in a controlled way, right? I think that’s something which isn’t said. It’s implied. It has to be a controlled failure, where you have the boundary, and things will not blow up. I mean, they’ll blow up, but the fireworks sort of blowing up, where it’s a controlled explosion.

Right. Tiny little crashes to learn from. Welcome, everyone, to Kaizen #22, with the incomparable Gerhard Lazu… He’s here to let us know how he lets it crash. It’s that song, “Let it snow, let it snow, let it snow”, only - you know how to replace… Hey Gerhard, how are you?

Hey, Jerod. I’m good, thank you. Thank you. I had a great holiday. It was a great couple of weeks where I’ve managed to finally disconnect. It’s been, I don’t know, 20 years since I had two weeks completely off…

Even my holidays are only a week. So this was very different, very enjoyable, and I feel so refreshed… So I’m firing on all cylinders.

You unplugged, and now you’re plugged back in.

I stopped it, and I started it, and it’s brand new.

It’s Glade, man. I’m Glade over here, man. Plug it in, plug it in. You know what I’m saying? Smell the scent, the fresh New Year’s scent called 2026…

Some people are going to say this is going to be the best year ever. I’ve heard it said. What do you think, Gerhard?

They keep saying that, and I’m excited about them.

They said that about 2020. [laughter]

2020… We have to admit, it was off to a killer start. I mean, it was really going well.

Right. Pun intended, killer start…

It was COVID. Pun intended? Killer start? That was 2020. 2020 was the year of COVID, and everyone’s “Oh, this is going to be the best year ever”, and then we had three years of misery. So I think –

I just want an easygoing year. You know what I mean? Last year, 2025, 1st of January, we were building shelves. We were redoing studies and whatnot… And the whole year was full on. Like, it was nonstop. Every week there was something significant happening. And this year we would for it to be a bit more chill, maybe a bit more meaningful… So that’s what we’re thinking. But how about you, Adam? How are your holidays?

My holidays were filled with barbecue, and good times.

Wow. Even in winter… So barbecue never stops. It doesn’t know seasons.

It never stops in Texas. Actually, just to shower you all with a few of my picks from my most recent barbecue adventures… If you’re in Zulip, go to the general channel, look for barbecue with three bangs after it, because - why do one bang when you can do three?

Some recent ribs… My gosh, my ribs method is on point, my spatchcock chicken method is on point… No one is disappointed at my barbecue joint.

Very nice. Look at that, we’re going to add some meat on this slide… That’s what happened in real time.

Wow. Real-time meat added. This is – this is intense.

Yeah. And again, just to be clear, it’s Adam’s barbecue. Okay? So no joking aside, we’re talking about barbecue.

I think we have to leave it there… [laughs]

I think we have to leave it there.

I didn’t show a burger, but I do make a mean burger, too. Thank you, Gerhard, for assuming that is something I do rock really good. My smash burgers are on point.

Very nice, very nice. I’m looking forward to that. And so…

My favorite Christmas tree - this is what it looked like.

And for those that are listening, it’s a networking cabinet. There’s lots of blue lights flashing. This is happening in the loft… You have many terabits of network throughput. There’s some switches, there’s UniFi, there’s Mikrotik… This is maybe five years in the works, and every Christmas I take time to improve it little by little. So this year I went really crazy. I redid the whole thing. I redid the whole, for example, DHCP network, VLAN… Man, it’s beautiful.

Your VLANs are beautiful…

I want to be a guest on your network, man. I’m going to get blocked from everything, okay?

Well, well, there’s a big story happening in the background, and it is going to be – I think this will be amazing. This is will be the best network that I have run in my life… But the blue, and the darkness, and it’s like – that was one more Christmas tree in our house, and this was it, where I would just go and tinker for a few hours in between the Christmas dinner and all the Christmas festivities… So it was nice just to spend a bit of time tinkering with hardware. And I’m sure that many of you listening, when it comes Christmas time, when things start quieting down, you get the little projects that you didn’t have time for throughout the year, and then you have some fun. So I’m wondering, did any of you did anything fun this Christmas? …but nerdy fun, that’s what I mean by that.

Nerdy fun… Well, I got upset with something…

[00:07:47.23] …and so I decided to just let it roll. You know what I’m trying to say? I got upset with the amount of RAM usage on my machine… And while I liked the application, I was like “You know what? I’m just kind of tired of having four gigs–” I think it was – no, it was like 1.2 gigs of RAM being used by Clean my Mac… Fancy little utility application, helps you tune, and pay attention, and stuff like that… And I decided to remake it, and that was it. So I remade it. It’s called Mac Tuner. I know there used to be a MacTuner.com, which was, I think, a Mac Magazine, I believe… But Mac Tuner fit. I might change it, who knows… But for now it’s called Mac Tuner. It does all the things, all the things. Analyze, clean up, uninstall… And not just that fake uninstall; the real one, where you get the dirty dirties out. You know what I’m saying, the dirties? All the dirties are out, okay?

My mind is on the dirty burger that you mentioned earlier… [laughter]

Yeah. I mean, that’s about as nerdy as I can get. I mean, I made a little utility that’s for me for now. Soon to be open source, though; soon to be.

I mean, why not, right? Share with the world.

Well, I didn’t create a Mac Tuner, but I’ve found one. I also was thinking, Clean my Mac - how long am I going to run this thing? And the answer is “As long as I ran it, because I’m done now.” I found a tool called Mole, M-O-L-E, which is a command line macOS cleaner that does everything. So maybe you’ve got some competition here, Adam. Maybe you can come out and throw some blows down, like “Here’s why I’m better than Mole.” It’s got a TUI, it’s all command line-based, it does cleaning, optimizing, uninstalling, DaisyDisk, Explorer…

I’m feeling it. I’m feeling intimidated over here, okay?

You’re starting to sweat?

I think he just changed his mind about open-sourcing it.

Here’s your domain name idea, Adam. [unintelligible 00:09:45.15]

That’s good. I could do that.

So I’ve been using that, and I’m very excited, because who doesn’t want to just have all the things right there in their command line? And I didn’t spend any tokens on it. Adam’s got some tokens involved, but his also works the exact way he wants it to.

Yeah. Yeah, absolutely. Mine leverages some Recast stuff as well. It’s kind of cool.

Sweet. Open source that sucker.

Not today. [laughter] Definitely not right now.

But it’s going to be one day…

One day. There’s a bigger launch awaiting, is all I’ll say. There’s a bigger launch awaiting till I’m going to open-source some things.

I’ve been using AppCleaner for many, many years… Now, there’s no TUI, there’s no CLI. It’s just a regular app. It’s a really old one.

Butt you just drag and drop onto it, right?

Pretty much, yeah. And you also have a list of applications… But it’s so old that it’s difficult to find it these days, and it hasn’t updated in a very long time… So I will check Mole out.

Mole’s really cool. Brew-install Mole and you’re done. So you can check it out right here while we’re talking. And I liked AppZapper… And I think AppZapper doesn’t exist anymore, but the cool thing about that was that it would literally make the zap sound, as it – yeah. You drop your app on it and it zapped it. And I just liked that sound.

That’s the only feature that your application needs to have, Adam. If it zaps…

Mole does not zap, so there you have it.

“Make it zap” is our tagline, actually. Make it zap.

There you go. I think that’s a very good debate, actually.

What about you, Gerhard, besides your Christmas tree? Did you…?

I will come back to that. I will come back to the Christmas tree, yeah.

This guy’s got stories, man.

Oh, man. Oh, yes. I have to tease them and be very disciplined, because there’s too much stuff. So I have to be very careful, because it will be an hour and I will not shut up talking about this thing. I mean, it’s just like – anyway. So we will come back to that, I promise.

[00:11:45.18] Last time, when we finished Kaizen #21, this was one of the last thoughts that we shared, which is what’s next. So BAM… Remember BAM? That happened live. OOM crashes, out of memory crashes, and a bunch of other things. The good news is that only one thing happened. OOM crashes…

You’ve only got one thing to talk about… [laughs]

…but this rabbit hole is really, really deep.

Okay. Alright. Take us down the rabbit hole. The OOM, out of memory.

Who remembers this book? Erlang in Anger.

Stuff Goes Bad, by Fred Hebert. Ferd.ca.

Now, I remember “Learn You Some Erlang for Great Good”, but I do not remember this one in particular. So I’m not sure why the other one hit my radar, because he wrote both of them, it seems… But when did this one come out?

So this one, if I look – I just switched to the browser… 2016, 2017, while he was still at Heroku. Remember Heroku? Those were the days.

So about 10 years ago. And Fred – I mean, if you don’t know his blog… It’s just amazing. I’ll just click it very quickly, just to have a look… I think it’s one of the best blogs out there. There’s so much goodness here. So much. But one of my favorites is queues, and queuing, and how queues don’t protect from overload. So queues don’t fix overload. And this is so relevant to today’s conversation as well. But there’s a lot of stuff in the Erlang ecosystem, and there’s many, many things that Fred wrote over the years, that are so relevant to today.

So if I click on Download PDF - by the way, this is a… It’s amazing this book is open source. You can download it, open source, freely available, Creative Commons license… And I’m going to make this a little bit bigger, so we can see what’s happening. And if I search for “Let it crash”, it’s page number one. It’s in the introduction.

Page one. And this idea of “Let it crash” really comes from the Erlang ecosystem. It’s very well renowned there because of how the Erlang VM works, and how all the processes, and the supervision trees just – it was built this way. And we know a thing or two about Erlang, Jerod, right? …because the application, Elixir, the Phoenix framework runs on the same principle.

I know a thing, and you know two, so that’s how we get to a thing or two.

And Adam - I’m sure he knows the big one. But we don’t know whether he’s going to share it. The point is, when you think about “Let it crash”, Jerod, from your development experience with Erlang, with Elixir, Phoenix - is there any situation, any moment where you could experience it and you realized “Huh, that’s nice”?

Well, it’s nice that the [unintelligible 00:14:49.07] seems to handle a lot of the problems with letting it crash. It just goes again, or there’s a supervision tree, and things watching each other, and I don’t have to think about it very much. I can’t think of an instance in development where I was like “This is really useful”, but I’m sure you could come up with one.

Yeah. So you know when you write code, we tend to write code very defensively… Typically try/catch. So you feel like you need to account for every single scenario. And the “Let it crash” philosophy is about not preventing failure; learning from it. What that means is you need to have a context where it’s safe for things to crash, and the overall system will still remain stable.

So how can you build a resilient system - and really, this is about resiliency - where the core of the system will remain running, and the system as a whole will remain running even though parts of it may experience failures? …but those failures will not bring everything down. And that’s really important. So fewer try/catch blocks, don’t code defensively, let it crash, and separate the code that solves the problem from the code that fixes the failures. And the more you can lean into the framework, or the VM, or whatever you have - the system - to deal with failures, the better off you are to focus on the things that are unique to your application.

[00:16:12.15] And Erlang is well renowned for that.

Kind of the opposite philosophy that Go took, as I write some Go code and I write some Elixir code… Where with Go it’s handle every error condition right after you potentially raise one, and make sure that there’s no error. And if you’re not dealing with it, then you’re not writing robust software. And the other philosophy is “Let it crash and deal with it elsewhere.” I think they’re both legitimate, depending on what you’re building.

Agreed. Well, in our case, we had a lot of crashes to deal with… [laughter]

Yeah, we’re taking the Erlang style…

So what we are going to have a look at is all the times that the Pipe Dream has been crashing since our last Kaizen. So since Kaizen 2021, which is October 17th, we had a lot of crashes. And there’s a certain property about the system - and this is Varnish specifically that made these crashes pretty okay. And the property which I’m referring to is when you start the varnishd, the daemon, Varnish itself runs as a thread, and you have many, many threads that do different things. So when we had these out of memory crashes, all that happened - the thread was killed. Which means that the system as a whole didn’t crash, the VM didn’t, the Firecracker VM didn’t crash… The application needed to restart. It was just a thread that was using too much memory, and it restarted within seconds, as in maybe two seconds, and everything was back to normal. Obviously, the cache was cold, but it was good. And that’s why the memory looked a bit interesting, in that it doesn’t release all the memory, the VM doesn’t restart… There’s not many hangs; it restarts and it crashes really, really quickly. So that’s a nice property.

Well, that confuses me. So how does Fly know about it then, if it’s just happening inside of Varnish?

So it’s looking at the process ID, “Which process uses the most memory?” And it’s the same process that’s asking for more memory. So basically, it will just send a signal to that process, and kill that process. But that is just a thread; that maps to a thread. So Varnish itself didn’t crash; it’s just a thread that maps to a process ID that crashed, and then it was restarted by the Varnish daemon.

Okay, so where is Fly involved in that? Because Fly is ,aware because I see all these Fly notices, and I get the Fly emails.

Right. So Fly is aware that there is a process on the machine that is using too much memory, and more memory is being requested. And then it looks like “Okay, which process do I kill?” And in this case, a process with the most memory will get shot, and will get killed.

So Fly as a platform can actually reach in and kill that process without killing the machine, rebooting the VM, or Firecracker, or whatever?

So the Fly platform - it integrates with that functionality, which is a Kernel, it’s a Linux functionality. That’s why an out of memory crash would happen even if you have a single machine; you have too much memory, you don’t have any swap… How do you basically give more memory when there’s no memory left, and when the system is becoming unstable? So then you get just a single process which gets killed. In Fly’s case, they surface that. They surface the fact that there was an out of memory crash, there was an out of memory event, and they send you an email when that happens. It doesn’t mean that the machine had to restart, it doesn’t mean that it stopped serving traffic… It just means there was something that just had to go away, because it was using too much memory.

When I say “too much memory”, obviously it’s a bit more complicated than that, because something was asking for memory, the kernel didn’t have any more memory to allocate, so it just had to look at what needs to be killed, so that I can allocate more memory… Because something is using too much memory. And it just so happens it would be this process, and this thread. So how many crashes do you think that the Pipe Dream had since Kaizen 2021, since October? So we’re talking about three months, maybe a bit more than that…

[00:20:16.01] So Gerhard has presented us a multiple choice quiz. A is 20, B is 40, C is 80, D is 160. Now, I know that I personally receive an email every time this happens, and so I have a little bit of a feeler into this. I delete them, so I can’t go do a quick search. Adam, do you get emails when these Fly things crash?

Not to my knowledge. And if I do, they’re in a box that doesn’t get looked at.

You’ve been saving on some email bandwidth…

I do know, because we send the email… So let’s go back to this one. If I click on this one… Let’s take this one, and you can see everyone that gets an email. I’m just going to make this a little bit bigger, so you can see it services Jerod, Adam and Gerhard.

Yeah. So there must be a filter…

He just doesn’t look at it.

A superhuman saving me. Nice…

That’s okay. So what do we think?

Good thing other people are looking at it…

It’s not an Adam problem. That’s the thing. So that’s a good thing. He’s doing the right thing. He’s just saving his inbox for more important messages.

They ran an LLM on that to the side. So I feel like 160 is too many. I don’t think I’ve gotten 160 emails since October on this particular thread. 20 feels not enough. I’ve certainly got more than 20 emails. So I’m between 40 and 80, and I’m going to think that – gosh, that’s a tough one. I’m going to go with 40. Adam, what do you think?

The price is right. Alright, cool.

43 crashes from October to December; through the end of the year.

Yeah. And then obviously, there were periods when we had quite a few. So if we were to think about what could be happening in Varnish that it’s running out of memory and crashing… So this is us trying to think about the sort of traffic that we serve, trying to think about everything – I mean, now we see every single request that hits Changelog, the CDN as well… And it’s a lot of requests.

So there was something in the system that was using way too much memory, and as a result, the process - or the thread in this case - was crashing.

I mean, I could guess it, but I might even have some insight. So… Should I just say it, or do you want Adam to guess? I mean, my guess based on - also I saw some emails flying through, but… Already I would have suspected that we just have too many large files. These 60 to 80 to 100 megabyte MP3 files loaded into memory, flying every which direction… And you just can’t load up that much memory without some sort of fancy freeing mechanism. And it’s just trying to hold all these MP3s in RAM, I think, and it just can’t do it. So that’s my guess.

Yeah. That was a good guess. And I think – the next question is going to be to the audience, because we know too much.

How are they going to answer it? It’s not real time.

Well, just think about it… We will give some time for people to think.

Okay. We’ll do like a delay here. So if they have a – what’s it called? The feature where you skip silences on… They’re not gonna have any time to think about this.

So quickly, turn that feature off, give yourself some time to think… Go ahead.

Yeah. Or pause. We can also say pause. Now is a good time to pause. And then - what could be the problem? So you’re right - all those large files. We had all the MP3 files; many, many MP3 files. They’re large. All trying to be cached in memory. And that was a problem. So what is many? Well, we have thousands at this point, of MP3 files, across all the podcasts, since the beginning of time. Large - large means anywhere from 30-40 megabytes, to 100+ megabytes. So that’s – I mean, just think, if you had to load a thousand files, that take 100 megabytes… That’s a lot of memory that you need to have available.

[00:24:30.17] And the problem is that once you store these large files, as we discovered, you get memory fragmentation, in that - imagine that you have all the memory available, you keep storing all these files, and at some point there’s no more memory left. So what do you do? Well, you need to see what can you evict from memory, so that you can store the new file. So imagine that you evict a few of those objects, but maybe they aren’t big enough, and you haven’t evicted them fast enough. So then you have this big file that can’t fit anywhere, because the sizes, the holes that you have in memory aren’t big enough for this file to fit. And there’s no defragmentation, or nothing like that that runs in the background… Which means that even though technically you kind of would have space in the memory, for the specific files you may not. And then it can’t be stored in memory. Now, the thing in Varnish is actually called - I kid you not - n_lru_nuked.

So I think the connection to the nuke and to the book, and to “Let it crash” is right there. So lru_nuked, basically - it’s like a forced eviction. So it’s an event where an object has to be evicted from the cache just to make room for a new one, because the storage is full. So you can see how many times this has happened. And that’s like an important metric that if we look at, we can see, “We had too many of these events.” Many objects were being nuked from memory to make room for new objects, but sometimes they wouldn’t fit.

So how badly did it nuke? Because we can measure this, we can look at this. And this is what that looks from a memory perspective. So you can see that the instance was running about maybe four gigs of memory, and then we had a massive spike within minutes, like one or two minutes, to 16 gigabytes. So that’s a lot of data that had to be fit in memory. And you can already see where this is going… Scrapers, and bots, and LLMs… We have so many things happening. And then you can see the memory, it went up. The thread was killed, the child was killed… The Varnish [unintelligible 00:26:40.14] memory came down again, and then it went up again. So the graph that we see here, we can see the first spike, just like maybe a minute apart… The second spike, another crash… It took a little while for it to restore. We’re talking maybe 10 seconds. And then we stabilized around 10 gigabytes. From a CPU perspective, we got like a hundred percent CPU utilization when this happens. Everything is full on, everything – the instance is really struggling to allocate and deallocate and free up memory… And more importantly, we have a lot of traffic flowing through. So how much? 2.29 gigabits, specifically. 2.29 gigabits…

Per second, exactly. And these happen so quickly; you have a huge rush of traffic coming in… And then nothing.

Break: [00:27:40.00]

So why is more traffic coming into the instance than going out? So this is the traffic that the instance is receiving. So we’re receiving 2.29 gigabits, but we’re only sending 145 megabits. Now is a good time to pause and think about why this is happening.

Yeah, don’t skip silence. So when we say the instance, we mean the Varnish instance.

The Varnish instance, yeah.

Which sits between our end user, whatever that is - or users - and our application. Well, actually, and our Cloudflare, not our application.

All our backends. And we have a couple of backends.

Yes, but in the case of MP3 files it’s our R2 origin.

So Varnish is receiving a bunch of data, and sending back an order of magnitude less data. And what’s it receiving - I don’t know, man. I mean, my guess would be we’re uploading MP3s… Now, that’s gonna go straight through the app to R2… Just a DDoS? I mean, what is it? I don’t know.

Yeah… So it is a DDoS, but it’s specifically downloading MP3 files, or starting to download MP3 files, but never finishing.

So you get all these requests for MP3 files, for large files… Varnish is going and fetching them as quickly as it can… So pulling all this data in, so it has it in memory, but the client is never around long enough.

Yeah, exactly. So they basically abort, but Varnish is still pulling in all the data. Now, there is a property… It’s called bresp.doStreamTrue. So what this does - a very weird thing - is it tells Varnish not to buffer the entire backend response if the client is slow. So I’m not going to fetch the entire MP3 file if you only want the first, I don’t know, minute, or two, or a range, or something like that. Now, this is on by default. So by default, that’s how Varnish behaves. So we wouldn’t need to enable this. But if the object is uncacheable, it cannot be stored in cache - do you see where I’m going with this? Memory, you can’t store it in memory… So you keep pulling these files over and over again, and maybe even just fragments of them… So even though the client never receives them, you may be pulling hundreds of files, and the client just goes away. So you’re not pulling the entire file, but you’re still pulling enough, and not able to fit it anywhere, and it just becomes a mess.

This reminds me of the ’90s, when you used to go jean shopping…

[laughs] Tell us… Do tell, Adam.

And you’d go into Abercrombie & Fitch - which I never shopped at, but let’s just imagine I did… I’d go in there and be like “I like all these jeans. Get them all.” I’m trying them all on, and then I just bounce.

Yeah, the person goes to collect them all, they come back and you’re not there.

Here’s a dressing room full of jeans, and Adam’s gone. Bye-bye. See ya.

This really sounds like you’re speaking from experience. Was this a prank [unintelligible 00:32:54.20]

I just made it up just now. I’m just creative like that, you know? On the fly. Creativity.

[laughs] That’s a good one. That’s a good one. On the fly, yes.

It is. On the fly.io. Boom!

Well, what could we do then? What’s going on here?

Exactly. So this was one of the things which I had to deep-dive and understand what on Earth is going on. Where do we store, what’s happening… So there’s a lot, lot more that went into this pull request. It’s pull request 44. I’m calling it “The elephant in the room.” I’m going to switch to the browser, just to have a look at that.

So the title of the pull request is “Storing MP3 files in the file cache.” But that’s the tip, right? The most obvious thing is, “Well, you either have lots and lots of memory to give varnish”, which honestly would be impractical, in the sense that would be way too expensive to store all these files in memory. The next best thing is to have something like a file cache. And by the way, we’re talking about open source Varnish. That’s really important. Anyone can use this, anyone can configure this… You can configure a file cache, which will basically pre-allocate a file on disk, and that’s where these large files will be stored. Pull request 44, the one that we’re looking at, is in the Pipely repository. That’s what this adds.

[00:34:19.08] But there’s significantly more stuff… And if I’m going to – so there’s quite a few files. I highlighted a few, so I’m going to look at this one… So it’s not just that. You also need to tune, for example, thread pools, you need to tune the minimum, the maximum… You need to tune the workspace backend, like “How many memory structures get allocated?” You need to configure the nuke limit… And there’s a couple more things that we had to go through, just to make things stable.

Now, I’m just going to very quickly mention these things. You can go and have a look at the pull request to see what else went into it.. So this was the one file. The other one was the regions. That’s another thing. Not all regions would suffer from this. So you don’t want to allocate too much memory or too much CPU to regions where maybe they don’t get a lot of traffic. And you would think that this thing is easy, but oh man, I have a surprise for you… You can’t mix and match sizes easily in Fly. So you can’t say “Create application groups, and this group will be the small group, and that group will be the big group, and this is just one application…”

It’s not straightforward. So you have to – again, this is how I solved it. Maybe someone listening to this will tell me, “Hey, Gerhard, you’re wrong.” I would love to know that, seriously. So the way I solved it is we deploy in all the regions, because you specify the size once. So you say “My starting size is the large instance type.” It has a certain number of cores, a certain number of memory… And by the way, the disk is the same in all of them, because that’s another problem, so we will sidebar that, or put a pin in that.

So when it comes to the initial deployment, you deploy the one size across all the application instances, and then you go and need to check to see which instances should be scaled down, so that you have the capacity, but the regions that don’t need the capacity can just bring them down. And you do a rolling deploy, in that you replace one for one, you have plenty of capacity to handle the traffic while instances are being rolled… All that good stuff. But we have hot regions, and then we have cold regions. And there’s quite a few things here. Again, if someone knows how to do this better, I would love to hear about that.

And we have the TOML, we have the primary region… There’s a couple of things here… We’ll come back to services and – HTTP services. That’s a fun one. We’ll leave that for a little bit later. Fly Just… We can see how we do the flyctl deploy, we disable HA, because we want only one instance per region… We have 15 regions in total. We specify the CPUs, the memory, all that good stuff, including environment variables… Oh, that’s another thing - we need to adjust the Varnish size based on the memory the instance has. We need to say “Hey, Varnish, you get 70%.” And that’s the other thing that this does. Same thing for the file size. You can’t take up the entire disk. We tell you, based on the disk that we provision, how much space you should use from the disk that gets created. There’s a scaling there, so that’s another good one…

I’m going through the pull request to see if there’s anything else. Oh man, this was a pain… So recreating – like, writing tests for this. Everything is tested, in the sense that which requests would go, or basically which files would get cached in the file store, and which files would be cached in the memory store. So how do you write the tests? Some Varnish logging is included, you have to have anchors… There’s quite a few things. So that’s assetsbackend.vtc. And part of this - it was a huge refactoring.So if you look at the lines of code, I wouldn’t say it’s that many. 1,500 were added, and 1,470 were deleted. So not much changed. I mean, the net is 30 new lines were added. But there was a huge, massive refactoring part of this.

[00:38:21.16] So there’s – again, this was, I think, two-three days of figuring it out, trying things, refactoring things… And if you think that an LLM can help you - well, you try this. [laughs] And it takes longer to go through those iterations than - if you know what you’re looking for, it tends to be easier.

Anyway, it’s very dense, very specific, very difficult to make sure that it’s doing the right thing. But it’s all there. We have the mock backends, we’re reusing things… We split the VCLs – by the way, we finished the splits, so it’s easier to reuse them. So there’s quite a few things there.

Now, this is Kaizen, so we are wondering what improved. After all this work, we rolled it out… What improved? And to answer this question, we need to figure out which region is the busiest one. So out of all the regions that we serve - we have 15 in total - which ones get the most traffic? It’s those hot regions. We’re looking at Fly, the Grafana dashboard for our fly application, the instance of the Pipedream, the current one… And we can see that SJC - San Jose, California - is a nice, big, red circle, which means it has the most traffic… And also NRT, which is Tokyo.

Yeah. And Europe, there’s quite a few… So if I’m going to pull this down a little bit; let’s see… No, I wanted to go here.

What about that new continent? Are we big there?

The new continent? Australia?

No, there’s a new one. There’s a new-new one.

I don’t know. There’s a headline… I thought y’all would get the joke.

Over the holiday there was speculation there was a new continent being announced.

Maybe… It could have been Narnia.

So right now, even this list is basically – if you think about it, it kind of makes sense. It’s US East, US West, Europe… But we have quite a few instances in Europe. We have four. It’s more geographically spread in Europe. And we have Asia. So these are like the big ones. Australia, Africa, and South America - they’re not as busy. They are the least busy regions. Cool… So which instance would you like us to have a look at? So I have a queue right here…

SJC, baby. Let’s go. Let’s go big.

SJC, baby. Alright, let’s see that. So I’m running flyctl, SSH console. I’m using two flags. -s, which is a short one for –select… It’ll prompt me which instance I want to select. And then I have -C. Capital C. It’s different than lowercase c. They do different things. I give it the command to run. And it’s [unintelligible 00:41:11.25] which will give me all the statistics from Varnish at a point in time. So since this instance was running, I will select SJC… There you go. And it will give me all this data, which is all the counters that Varnish is incrementing, is keeping track of different things; of the origins, backends, the memory pool, the disk pool, the lock counters… There’s so much stuff. I’m really, really impressed how many things Varnish has.

So this is what we’re going to do. “We”, because AI.. We’re going to copy all of this, we’re going to ask AI what it thinks of this… How about that? [laughs] There’s just too much data here, so let’s be serious about it. So question to you - which is your favorite AI, Jerod? Which one do you use?

Oh, I don’t like any of them. I would probably start with Claude, and then I would go to Grok, and then I would go to ChatGPT, third.

[00:42:11.24] Okay. So Claude, which one? Which version? Which model?

Opus, man. Give us the Opus.

Opus. Okay. So we’re looking at abacus.ai, something I’ve been using for a long, long time… It allows you – I’m only paying $10 per month for it. Not sponsored, not affiliated in any way… It’s just something that I’ve picked for myself, and I can basically pick any model, and I can just run this. So I have somethin