Featuring

Sponsors

Tiger Data – Postgres for Developers, devices, and agents The data platform trusted by hundreds of thousands from IoT to Web3 to AI and more.

Depot – 10x faster builds? Yes please. Build faster. Waste less time. Accelerate Docker image builds, and GitHub Actions workflows. Easily integrate with your existing CI provider and dev workflows to save hours of build time.

Augment Code – Developer AI that uses deep understanding of your large codebase and how you build software to deliver personalized code suggestions and insights. Augment provides relevant, contextualized code right in your IDE or Slack. It transforms scattered knowledge into code or answers, eliminating time spent searching docs or interrupting teammates.

Framer – Design and publish in one place. Get started free at framer.com/design, code CHANGELOG for a free month of Pro.

Notes & Links

📝 Edit Notes

Chapters

1 00:00 Welcome to The Changelog 01:02
2 01:02 Sponsor: Tiger Data 01:39
3 02:42 Start the show! 01:04
4 03:46 It all starts in Rwanda 03:11
5 06:57 From medical to consumer 02:16
6 09:13 Env concerns 00:54
7 10:07 Drone design 02:26
8 12:33 Space inspiration 01:26
9 13:59 Drones' OS 01:28
10 15:27 Software updates 01:46
11 17:13 Novel problems 02:23
12 19:36 The drone cloud 02:19
13 21:55 Sponsor: Depot 02:50
14 24:45 Regulations 12:57
15 37:42 Observability 02:34
16 40:16 Logging a tornado 01:41
17 41:57 Wind limits 01:40
18 43:37 That wing! 01:42
19 45:19 Range anxiety 02:07
20 47:26 Range vs cost 02:15
21 49:40 Sponsor: Augment Code 02:55
22 52:35 Sponsor: Framer 01:14
23 53:49 Supporting the UX 02:22
24 56:11 Integrations 01:34
25 57:45 Fun features 02:38
26 1:00:23 All day / night 04:11
27 1:04:34 Zip at scale 04:49
28 1:09:22 Heavier packages 02:25
29 1:11:48 Competition 00:59
30 1:12:47 What's next 00:49
31 1:13:36 Next city (Omaha!) 01:11
32 1:14:47 Big tech problems 02:59
33 1:17:47 Manual override 04:48
34 1:22:35 The tech stack 02:38
35 1:25:13 Open orchestration 06:35
36 1:31:48 Wrapping up 00:58
37 1:32:46 Closing thoughts 01:17

Transcript

📝 Edit Transcript

Today, we’re joined by Keenan Wyrobek, founder and CTO of Zipline, a company I’m very, very interested in. Welcome to the show, Keenan.

It’s a pleasure to be here.

Pleasure to have you. Autonomous flying machines, all kinds of deliveries happening… I saw Zipline and I thought, “You know what, this one makes sense.” There’s a lot of people trying to do delivery… Of course, you have autonomous cars upcoming, you have people that are doing tunneling, they’re trying to put tunnels in the ground and ship it… Kind of Futurama style, but underground Futurama… And then we have drones, which - I don’t think Zipline’s the only one playing the drone game, but you guys are, I think, way ahead of the game. Been doing it for a long time. Tell us about the start of this company and where you’re coming from.

Yeah. So we started almost 12 years ago now, and we started in a niche of a niche, of delivering blood to hospitals in Rwanda. That’s where we started.

Yeah… So really, Rwanda earned this for – we knew we wanted to start in health. At the very beginning of this we started exploring this space based on really just stories from family and friends. My wife’s an epidemiologist, and she would tell me these stories about health intervention campaigns that would get stuck on logistics. These would be lik vaccine campaigns where they had the vaccines, they had the doctors, but then the logistics got in the way, and they couldn’t be successful. So yeah, that’s how we started. Went deep on that… I’m personally not an early adopter, I’m very tech skeptic… So I went deep in Central America and Africa at the time expecting to come away with a thousand reasons why we couldn’t possibly make a difference. And the opposite happened. The deeper we went, the more conviction we got that we could make a big difference. And one of those places we were exploring with was Rwanda, and they turned out to be a phenomenal first customer for us, and really a partner figuring this out together.

And blood, of course… Why blood? It’s just this rare commodity everywhere. There’s never enough of it… It’s actually very expensive. We think of it as free, because we donate it, but the cost of the collection, the testing, the transportation is very expensive. Hundreds of dollars a unit; even in places like Rwanda, and way more in places like here. It has a very short shelf life, as little as seven days from the time it’s donated, to the time it expires if you don’t give it to somebody who needs it. So by centralizing blood in one place and then delivering it when you know what blood type is needed for what patient, at what time, it turns out you can virtually eliminate blood waste, saving a ton of money and a ton of lives for the health system. That’s why we started there.

What was the tactical nature of that when you went to Rwanda? Was it discovery first, and then tech later? How did you come to a technical solution? How did you think about it before you built something?

Yeah. I mean, we had built nothing. I literally called a friend of mine who was good at drawing, and he drew pencil sketches, and made a slide deck of the concept… And we were using that to talk to people. In the early days, it was just talking to the operators in these health systems, that actually run the logistics, run this testing, run these labs… But very quickly, we ended up having meetings with the offices of presidents in these countries, and that was a big sign that we were looking at a problem that was a really big deal for these health systems. And yeah, no technology whatsoever. A year or two before we were doing this, Amazon was like “Hey, drone delivery coming next year.” So that drone delivery was an idea that was out there. It wasn’t a real thing yet anywhere… And so we kind of thought of ourselves as like “Hey, if we’re going to do something, let’s find a niche where the value is really high to do this.” Yeah, no technology. And then of course, once the interest was there, we got moving fast.

Was there a point that you and your co-founders looked at each other – you had saved countless lives with delivering blood around Africa, and thought “Burritos. Burritos are next.” Or was that from the start?

It was both there and not there. In a lot of the countries we operate around the world, we start with health systems. That’s how we started in the United States. We started with health deliveries during COVID, and then expanded from there.

In the islands of Southeastern Japan we deliver healthcare supplies and bento boxes. In a lot of places in Africa we operate we started in human health and then expanded into various agricultural use cases, things like genetic diversity for milk cows; huge impact, but super-obscure. We’ve just been layering on other use cases, and slowly worked our way into things like e-commerce and burritos.

[00:07:55.23] What’s compelling – when you think about Zipline, we talk a lot about the impact on health, which is really exciting. We started with the impact of blood. It’s a super-visceral, high-impact, but very rare problem. Most of us don’t need blood transfusions in our life, but when you need it, you really need it. Then we slowly worked our way in health to preventative care, which is actually where the real cost savings for health systems is. Then we’ve worked our way into other areas, like animal health.

One of the things that’s really magical about this, as we step into new things like auto parts delivery - you know, what limits the mechanic’s ability to service cars is usually how fast they can get the parts. The way I think about it is basically for life, in the home or professionally, we’re in the mission of getting you what you need, when you need it, and we do it in a way that’s not just fast - we skip over the traffic; we’re very reliable in that way - but just wildly environmentally friendly. We don’t talk a lot about that, and we can get into that if that’s interesting. We’re talking about a 20x improvement in the overall environmental footprint of the supply chain, and almost every aspect, including things like bird safety. That’s something we don’t talk about a lot. There’s just so many layers to this.

Well, even – I live here in Texas, and we have a lot of wind farms, because here in Texas we have multiple ways… Probably like Nebraska, Jerod - you’ve got multiple ways you’re getting electricity. I’m not so much a fan of them; they’re big and crazy, but I hear people get upset with the bird issue, essentially of that…

Oh, they kill lots of birds.

And maybe to the environmental aspect of it, that there’s still so much of a cost to produce even wind energy that it’s almost not worth it when you compare to how you talk about your efficiencies.

Yeah. I care a lot about birds, so I pay a lot of attention to it… We estimate that when we displace delivery by car, we reduce bird injuries by something like – somewhere between 10x and 100x lower rate. We study that a lot. There’s a lot of aspects to why we do what we do, and why we’re excited about scaling what we do into all these different verticals.

Let’s talk about the design of the drone itself, because we mentioned the tactical approach coming out of the need of delivering these things in Rwanda… When I saw your guys’ design, I was very surprised by it. There’s kind of a more typical quadcopter, larger quadcopter-looking thing; almost like an airplane with blades on it. And then it doesn’t actually do the delivery. It delivers the delivery mechanism, which is a smaller kind of a baby-copter that comes down on a zip… That’s, I presume, where you guys get the name. It lowers it down to the ground, and then raises it back up again. Where did that design come from?

Yeah. That’s sort of a double drone; a drone inside of a drone design. It came from just an obsession of figuring out what would actually work for our customers. Our first platform, that does long range delivery, has a lot of great attributes, but it requires about two parking spots worth of space to deliver in, because it literally just flies over, drops the package with a little paper parachute on it, and the package floats to the ground. But so many of our customers in the health space and otherwise were like “We want home delivery. We want in metro delivery”, where you don’t have that kind of space. And at the time, we’re kind of like “Well, we don’t do that.” Eventually, they asked us enough times si we started thinking about “How would you do this?” And we knew from that platform that one of the things that makes people love our service is you don’t hear our drones. And that’s way harder than it sounds. And we’re like “Well, we don’t know how to do a super-precision delivery without hearing the drone.” And we’ve seen other ways of doing this. It’s usually loud, and noisy, and kicks up a dust storm, and can’t deliver very precisely. So this two-part architecture enables a bunch of things. One thing it enables is quiet. It lets the drone itself stay up at 100 meters, at 300 plus feet up in the air, which is one of the many things we do to make that just silent in even suburban places. Then that little mini drone that comes to the ground - we call it the delivery zip. It comes to the ground, it actually delivers the package for you. It has its own little propellers on it, so in windy days and stuff it can still be super-precise and get into a tight space.

[00:12:14.29] So really, those two things - helping us be really quiet, and be super-precise; we can get into a really tight space with that little drone that comes to the ground, while the big drone stays up high, that does the heavy lifting. It’s literally there to carry the weight, the distance. And yeah, that’s why we have the little drone inside of a drone solution.

It reminds me of how you see things happen in space, when you see a little jettison engine pushing it, nudging it… I’m a sci-fi guy, so I see these films all the time, and I’m like –

It’s kind of like little slight adjustments, but you just have – is it one single large propeller, or is it multiple on the, I guess… What do you call the mini drone? What is that name for that guy?

Yeah, internally we call it the droid delivery zip. We don’t have a great name for it. I like delivery zip, but… Anyway, this little thing that comes to the ground - yeah, it’s actually got three little thrusters, very similar in concept to what you’re saying. It’s got one big thruster in the back, and that thruster’s job is to fight big, heavy winds; and so point it at the wind and it just makes sure it doesn’t get blown off course. And then it’s got a little bow thruster and a stern thruster, if you like boats, to think of it that way, that help it basically stay oriented, and do the side-to-side adjustments, as it’s coming down and staying precise. A lot of our team comes from space, and the control theory behind how you stay precise on that little delivery zip as it comes to the ground is very similar to how you think about reaction control thrusters on a spacecraft.

Because that thing can’t spill your coffee. I mean, sure, it can flip a burrito, no problem, but if you’re lowering a coffee down - it has to stay oriented very well.

I assume - blood, they probably want that stuff spinning, or something, but there are certain fragile items that can’t go… So on the software side then –

“Certain fragile items…” [laughs]

It is. On the software side, are those two divergent operating systems then? Because it seems like they have perhaps different concerns, but they probably have a lot of overlap as well. How did you tackle the software side of these two different drones?

Oh yeah, they’re very different control problems. Yeah, the drone that’s up high - you’re right, it has a wing, so to fly a fixed wing on the wing really efficiently it can hover in place for takeoff/landing, and while it’s doing deliveries. Yeah, it’s a long story, the controls approach, and really it comes down to just massive amounts of testing. Our high volume test sites have done hundreds of thousands of deliveries. We chase weather with a mobile test rig, so we go chasing like hail, and icing conditions, and these kinds of things. A ton of simulation… Literally, half of all the engineering we do at Zipline is test. Just full on half of it. That’s how we can do this, is basically building the test systems, the various software platforms for simulation and things like that, the actual test scenarios, all the ground testing we do… We have an entire building down the street from where I am, that all it is is just hundreds of ground test systems to simulate what we do in the air, on the ground. Yeah, the answer to your question is a long one, but it’s literally half of what we do is all the testing it takes to actually develop the control systems for these things to know that they’re going to be both capable, but also safe.

I also think about updating the software… I mean, you mentioned the test facilities, so I imagine you do a lot of iteration there; you’re constantly deploying potentially over the air updates… I have no idea how you accomplish that mission, but… Divergent operating systems, and also a truly distributed update platform. How do you get new to the individualized? Is it an operating system delivery? Is it an individualized delivery? Can you talk about that at all?

[00:15:55.01] Yeah, absolutely. When we think about software and software deployment, there’s two worlds for us. One we call flight software, and then cloud software. Flight software is all the pieces, a few parts of the cloud, but mostly things on the actual aircraft that we consider flight-critical. That software release process takes us about six weeks. So every six weeks we release software to that system. And just in those six weeks we do tens of thousands of flight tests, we do a bunch of hardware in the loop testing… Think of it as like the matrix, where you take the electronics of the aircraft and basically plug it into a simulation, literally, and fool it. It thinks it’s flying over in Dallas somewhere, but it’s actually over in our basement, plugged into the simulation.

Plus, of course, software in the loop simulation, where you’re doing millions of flight tests. And yeah, that whole journey takes us about six weeks to get to the point where we’re like “Yup, this new software release is good to go.” Then we release it, and then we have an over the air update system that updates all the processes, the software on the aircraft, and deploys. People often ask, “Do we do that in the air or not?” It’s like ““No, no, no. When our aircrafts are docked, they update themselves.” They check themselves, they’re like “Okay, good. Software’s good to go”, and then they’ll fly again.

How many of these problems are novel to you? I know that the defense systems that we have here in the US probably have these problems and maybe have solved a lot of this, but maybe you don’t have access to the DOD’s tech platform, or maybe lack thereof… How much of this is invented here and how much of this is maybe a partner or vendor that you bring in to support over the air updates, for example?

Way more of this is done in-house than I would have ever thought we would have to do. Actually, when we started Zipline, I was like “Great, I’m going to go find somebody who has a drone, buy it from them, modify it a little bit for deliveries and start serving our first customer.” And the best quote I got back then from the companies making drones for defense was a $200,000 drone with a 200-flight warranty if I did not fly it in the rain. And to be clear, in most of these countries it rains pretty much every day. So that wasn’t going to work.

Yeah, we do a lot in-house… And a lot of that’s because we fly in conditions that no one else flies in. Stormy weather, near the ground, or over mountains, stuff like that - no one flies in that. So we’ve had to learn a lot of this stuff ourselves, and build our own datasets and experience and simulation approaches to actually develop these things. But then the other side of what makes us very unique is just the scale. We operate at a very large scale. Over the air updates is a good example, where we need that to be very, very easy and robust, in a way where if you’re operating a smaller scale, a human can do a lot of double checking of that, and things like that, and it doesn’t matter too much. Even our over-the-air update system we’ve built ourselves.

Do you know how many drones are in your fleet roughly, worldwide, to give us an idea of the scale you’re currently at?

Yes, it’s a couple of hundred.

Okay. And you’ve just started in the United States recently. 2021, I think, was the first non-medical flights… This is what I’ve been told by Ian behind the scenes. And now you are in Dallas, and rolling out – I just saw a video from your CEO and co-founder, that there’s a whole bunch of drones getting built right now to just go crazy. But 200 worldwide. And how many of those are over there in Rwanda, doing their thing?

I think it’s close to 400 or so. It’s about 300 outside of our first generation, and well over 100, but I don’t think 200 yet, of our precision delivery, the drone-in-the-drone system.

Gotcha. So how much of your software is orchestration? I assume that’s the cloud side that you referenced, the control of the cloud.

I guess maybe tell us more about the cloud side and what all that entails.

[00:19:48.29] Oh, there’s a lot to it. There’s obviously the pieces for people to place their orders… Every partner we have, we have basically systems to integrate with our partner’s data systems. Then we have our basically dispatch… I literally think of this as like there’s cloud autonomy, or fleet level autonomy, and then aircraft level autonomy. At the fleet level, you’ve got to decide which aircraft is going to go on which mission, at which time, and some of that is based on which aircraft is capable of that mission. Some of that is based on making sure if we told someone “Hey, we’ll deliver to you in this 60-second window”, that we actually dispatch the right aircraft at the right time, get the order ready at the right time so that we’ll hit that window.

And some of it is stuff that – other cloud stuff for us is weather forecasting. We do our own weather forecasting. We can talk about that, it’s a really fun problem. Also, basically, we have to design our own highways. If you think about autonomous vehicles, you take for granted that if you want an autonomous vehicle to drive from here to over the mountains, if you use the road network that’s already been designed, you’ll get there. There’s no such thing for drones, and so we have to do a lot of computation in the cloud to basically give enough prior to our aircraft, so that when they’re flying, they have the equivalent of a highway system. It’s not quite that simple… So that when they’re doing an avoidance maneuver, for example, and planning that live, that they’re doing that with enough prior that they’re not going to get off track and not be able to make their delivery, if that makes sense.

And they generally fly at 300 meters, is that what you said?

They fly at about 100 meters, so 300 feet.

100 meters, 300 feet. Thank you. What kind of stuff do you have to avoid? Obviously, birds, but at that – I mean, cell towers, airstrips, perhaps? What’s the kind of stuff that has to be avoided at that height?

Yeah, for sure. So yeah, you can have very tall cranes, cell towers. You can have inaccurate maps, so you may not be at the height you think you are, so there’s a hill or a mountain there… Yeah, definitely other aircraft. That’s a big part of what we avoid. There’s not a ton of passenger aircraft at that altitude, but there are some, and it’s really important that we avoid them as well. Those are the kinds of things we avoid.

Break: [00:21:57.20]

How much does regulation play into this rollout? I assume every area probably has different rules about how you can go about flying your drones at 300 feet…

Yeah, I wish everybody had rules.

Really? [laughs] Usually, people say they want less rules when they’re trying to scale up.

Yeah… I mean, when you’re a startup and you go to a regulator and you’re like “Cool, I would like to do this thing”, and they’re like “Well, that’s not allowed”, and then basically from there it’s like all by exception. And so you have to kind of work it out, the rules by which you operate at the detail…

And as a startup, that’s a slow process, and startups need to go fast to survive… And yeah, that’s why I said I wish there was some precedent. In some ways, obviously, there’s some luxury in getting to help create the precedent… But yeah, it’s a lot of working with the regulators. Some things about airspaces between countries are very similar, harmonized, but other things are very different. And so yeah, things we have to do here in the US have been very different than Rwanda.

Yeah, it’s a big part of what we do, but it’s also… We’ve gone from that being a big challenge maybe 10 years ago, to now it’s something that we’ve gotten quite good at, and quite good at working with these partners and these regulators as partners, and working with them proactively enough that we can generally work out what needs to be worked out before we need it to serve a customer. But it takes a lot of effort, I will say that.

Yeah, I was thinking of the luxury of not having rules that are already defined, because those can slow you down. But actually, not having rules and not having a clear path towards engagement - that means you have to trailblaze. And to trailblaze, you’ve got to get out a machete and hack away a bunch of stuff, and that can be even slower and more laborious than if somebody else had already cleared the path for you. So that makes sense.

Yeah. There’s a great story – here in the United States, starting about six years ago, we started working on the sensor system on the aircraft that can sense and avoid other aircraft. This is a system that uses machine learning, and machine learning is a class of software that the regulators have no experience with, literally. And it took us about five years of working with them to go from the beginning of this to them approving us to operate. And that was the first time they’ve ever approved AI/ML-based software for safety in the airspace. And so it was near daily conversations with their teams, figuring that out over that five-year journey… And there’s some people at the FAA and at Zipline who I respect a ton for working through the minutiae they had to work through to figure out how what would the requirements be, and how would you validate the system against those requirements. And then actually doing all that work and all that analysis to bring it back to the FAA and be like “Okay, we did what we agreed to. Let’s go through this together.” And yeah, that was a five-year journey there, and it was a hard won battle. When I say battle, I don’t mean like Zipline versus the FAA. It was very – we were very much in it together, because they definitely wanted to figure it out, too. Yeah, it was something else.

[00:27:50.25] This is an example where the word “regulation” - which you haven’t quite said yet - gets really weird. And I don’t want to go into politics, but it seems to be the word that gets thrown around. Regulation, less regulation, more regulation… But as a developer, I really lean back on what you said there was not a presence of, which was specifications, which is really what regulation is. It’s an adherence to a specification that everyone agrees on, for obvious reasons. But you’re in this kind of unique high stakes scenario where maybe regulation could be less pressured, so to speak, and in your case, you’ve got to be a high stakes scenario where if there’s a crash, or if there’s a break a leg situation, which - I’m not sure you like the word crash. I wouldn’t like the word crash if I was in your business, personally, but I just think about the need for that. Can you speak to how that’s played as an ally versus a foe?

Yeah… I think there’s a couple of layers for how we think about this. The first one is because everywhere we’ve been, everywhere we’ve gone, there hasn’t been some, like you said, specification to follow, we’ve taken the mantra that our first job is to convince ourselves. And not in like a handwavy way. It’s like “No, no. This is over our house, over our kids. Are we convinced that we’ve taken a rigorous approach before we suggest that approach to a regulator?” And that’s gone a long way. That mindset is attracting some of the best talent, as figuring some of this stuff out is really complicated. It’s trying to track the talent we need to actually figure this out, and then it’s also forced us to do the work – to convince ourselves, the analysis, all the testing, all that data review to convince ourself is also what the regulators want to see. And so if we’ve done that to ourselves and we’ve really been skeptical internally first, it helps a ton when you’re then having these in-depth conversations with these folks on the regulatory side. They have the completely thankless job of “Okay, we’ve got to figure out the specification for this new thing that no one’s done before.” So if they underdo it and someone gets hurt, they look really bad. And of course, if they overdo the regulation, then everybody in the industry complains of “Oh, they wrote down these regulations that are completely impractical.” So they’re stuck between a rock and a hard place.

And so by doing that legwork upfront and internally proving to ourselves something that we really believe in first, and then bringing it to them - that’s been core from our perspective. And then the only other piece of it that I would say is it’s all relationships. These are all people on the regulatory side, they’ve got a job to do, we’ve got a job to do, and so we spend a lot of time basically working together to make sure we understand what they’re trying to do, they spend a lot of time working to understand what we’re trying to do… Be really open-minded, make sure we’re hearing each other out, like in any relationship, and try to make progress every day, even if it’s not as fast as you might like.

How would you frame – this isn’t an exact one-to-one when I say this, but how would you frame test coverage in terms of a coverage level, confidence level? When you ship something and it’s in a regulation standpoint, or you have a specification you’ve got to adhere to - is there like 95%, 98%? Like, what do you strive towards when it comes to adherence to a spec, or defining the spec? What is confidence to you and your team?

Oh, great question. So there’s a model for thinking about this that was called the Swiss Cheese Model. I did not come up with that name, although I’m known for naming things with really dumb names… It was actually a NASA name, the Swiss Cheese Model. Conceptually, basically what it is - every approach to testing is like one slice of your Swiss cheese. It’s going to have good coverage of some aspects, but if you look at it, it’s going to have holes, for every type of testing you do. You obviously have many different layers of testing that you do, and the intent is if you stack up those layers of Swiss cheese and look at them through the stack, then you’re not going to see through holes, if you’re doing it well.

Examples of our layers of Swiss cheese - you’re going to be familiar with things like unit testing, and static testing, and things like that, that are commonly done in software… Software in the loop testing - so this is where you have full on physics level simulation running of your full system, even your full fleet, and you’re able to then write test scenarios on top of that.

[00:32:10.01] We talked a little bit about the hardware in the loop testing, where you take the hardware itself, the brain of the aircraft, plug it into a simulator and fool it into thinking it’s in the real world… That’s great for testing the actual hardware level components. If you short out a certain bus, does the actual full software stack detect that short and do the right thing? And then a lot of things in flight tests. We do all kinds of different flight tests. We do on the order of 10,000 plus flights a week at this point across our flight testing, and almost every single one of those flights is pushing the system to an extreme in some way. We actually have this fully automated software system we call chaos monkey, where basically what it’s doing as we’re dispatching all of these flights is it is – chaos monkey is literally throwing chaos into the system as those missions are happening, that either will cause the airplane to do something extraordinarily dynamic against the physics limits of the aircraft, or turning off subsystems of the software, killing various rotors, things like that… And making sure that the system, despite any given flight having dozens of weird things happen - nominal events, as we call them - in that flight, does the whole system handle it gracefully and safely?

And this is just on the software side. When we think about hardware, very similar things - many layers of testing of the hardware, to make sure we understand deeply its performance and its reliability. And there are entire teams of folks who look at all that testing, and basically are constantly questioning “Okay, in this release we have a new capability we’re releasing. Did we add enough testing to all the different layers of the Swiss cheese, such that that’s going to be responsible to ship?” And there are times we think we’re ready and we do that review of everything we did, and we’re like “Nope, this isn’t ready. We’re pulling this from this release. It’s got more work to do”, just on the testing side, to get that confidence.

These releases, are they – share as much as you like. You seem to be very sharing, and I like this. I think about releases and I think about “Well, if I released my test level from idea to new capability”, as you said, into production, there’s probably a several layers that are in that middle ground, which is like a true release, but not in a real world scenario, so to speak… How do you plan releases? How do you graduate releases from tests, delivery, observability, test again, potentially… You said you do 10,000 flights a week. Seven days in a week - that’s 1400 flights a day, roughly… Gosh, that’s a lot of flights. But how do you release and do that in a way that’s like “Okay, it’s in production, but it’s not real production.” I’m assuming these things, so… Walk me through that kind of idea.

Okay. So depending on the scale of a given feature… To talk about one of the biggest features we’ve ever released - we just released it recently. So we talked about the drone that flies up high, it has a wing, and can hover. Until about a month ago, in our releases and our operations, we were only hovering. We were not flying on that wing. So we were flying it more like a traditional quadcopter. It’s a pentacopter, but we’re flying it like a traditional quadcopter, just hover, which means the range is very limited. But while we were working on actually being able to fly on the wing - and the software and systems to transition from hover onto fixed wing, as it’s referred to in industry, and then back into hover, including when anything could be going wrong, like any of those rotors could not be working and that kind of thing… There’s a lot to that, and we’ve been working on those features, at some level of development and testing, for two years to enable this. So two or three months before, that release cuts. So when I talk about releasing every six weeks, we cut basically at the beginning of that six weeks.

[00:35:55.22] We will look at all the new features that folks think are ready for a release, and we’ll look at like “How tested are they? How proven are they? How good is the test coverage?” and all these layers of Swiss cheese. And then we’ll decide “Yup, this feature is ready. This feature is ready. That feature is not ready. It’s not going to go in this release”, and we will cut a release. And then once we cut a release, there’s a bit of a stabilization that takes us about a week, to make sure that release is really stable, and then it goes into this very structured release campaign that takes about five weeks on average, which - it depends on the release, but something like 50,000 flights, a whole bunch of hardware in the loop tests and software loop tests and all these other tests we will do again as well, as part of that process. And then the people – all the teams are basically paying a ton of attention to all the data coming out of all that testing. And over the years, we’ve developed all kinds of tooling and dashboards that help all the teams look at that data, and look for anything weird, anything that might we consider a near miss. Obviously, the real miss is if we have a parachute landing during that testing, we’re going to know exactly what goes on there. But we’re looking for anything that’s at all weird. We call it a near miss. And studying it, and seeing if it needs to be addressed, or if it’s fine.

You can kind of think of that flight test as like the last leg. If we find something in flight test, that’s a red flag to us. Like “Hey, the other layers of Swiss cheese aren’t good enough.” So if we find something in the flight test during that campaign, we’re immediately investing in those other levels of Swiss cheese to make them better, and improve their coverage. And then only once the data says we’ve got a system that meets or exceeds the overall target level of safety of our last release, do we then say “This is ready to actually ship.”

Do you have the idea of a black box in your thing? I mean, I know that airplanes do, so I imagine you’ve got some version of a black box, where if it does - I’m sorry - crash, that you’ve got some sort of thing in there… And then I also think about observability, because you’ve got all this testing happening… And I think about like traditional software in the cloud - it’s kind of easy to orchestrate observability. But over the years I’m sure you’ve got some version of WiFi or access to the thing digitally, via a network of sorts, telemetry going… Is there a black box, and how does observability happen in those scenarios?

Yeah, absolutely. So we log a ton of data on the vehicles, at all times. And that’s of course done in all this testing, but also in production. And the philosophy is that anything that could possibly go wrong, or be a near miss, you’ve got to study. And you mentioned crashes - I should mention we have a parachute system for the aircraft, that’s kind of the backup to the redundancy. So there’s redundancy in the aircraft, so it can nominally fly when things don’t work, but the backup to the backup is a parachute. So it’ll float to the ground like a – it’s actually made for us by a skydiving parachute company. So it’s like a skydiver coming to the ground.

So yeah, if we ever have a parachute landing in tests, or operations, or anything that’s even close to a weird data, we go study that, to basically deeply, deeply understand it, because quite often it’s a new insight. We’re so far in the long tail of the problem that many of the problems we see now, they happen one in hundreds of thousands of flights. And so you basically – like, if we didn’t log it, think of it as like “Oh no, now we have to wait for this to happen again, in hundreds of thousands of flights”, which is a total shame.

Total shame. It should happen more often… [laughs]

Exactly. No, that’s not the shame. The shame is not logging it. So we log these things. Now, in addition to the logging, to enable us to like go through those logs and understand everything that happened – we use those logs not just for like root causing a specific problem, we use those logs to understand things in aggregate. So we have tools where you can mine those logs longitudinally, across thousands and thousands of flights, to deeply understand the statistics of a certain thing. That’s really important as well. And then of course, during live operations - yeah, the aircraft are reporting over radios, live, what’s going on. We have a remote operation center that’s monitoring all of that… And so during live operations we have certain things that basically – our pilots, I think of them more like an air traffic controller, where they can be like “Okay, you go back and dock”, and that kind of thing, and understand what’s going on at a very sort of fleet level.

[00:40:16.20] Have you ever logged a tornado?

Hah, we’ve definitely – yeah. So the closest thing we’ve logged – we’ve logged some very small… I don’t think you’d call them a tornado. They’re basically – the things are too small to be considered tornadoes. At some of our testing, we’re actually chasing some tornadoes right now, as we look for hail and more extreme weather events with our mobile test rigs.

The craziest thing we have a lot of log data on is what’s called a microburst. So it’s the beginning of a thunderhead formation, about first 90 seconds of a thunderhead. You have the vertical winds at like 50, 60, 70 miles an hour vertically, in the middle of the microburst; on the sides of the microburst, the same thing goes down. Those are some of the most extreme weather events, that are hard for the drones to handle.

What do they do in those circumstances? Do they turn around and leave, or do they land immediately, or what’s their protocol?

So it really depends how big it is. this is actually one of the first things we developed AI to forecast. It’s because if this is really big, if it’s kilometers across, and really strong, if we’re in it we’re going to end up parachute landing. And we know this because in a lot of our operations around the world when we’re doing emergency deliveries for a patient on the table, delivering blood situations, we will take – we will basically ignore our weather limits and fly in very extreme weather. So we go through these things. But then for non-emergency deliveries, we rely on our forecasting to keep us out of this stuff, in those really extreme events. If they’re small enough - yeah, the control system will fight its way through it. But of course, the aircraft has its physics limits. It can only climb so fast, it can only do so much. And so in certain situations it can’t overcome them, and yeah, that’s when the parachute kicks in.

I assume there’s some sort of wind limit that there’s just no chance for anything to fly through successfully. It’s gotta be around a hundred miles per hour or so, maybe a little bit less… I don’t know, but there’s certain storms where it’s just like, you’d better just parachute out, because there’s no there-there; there’s no success as a possibility, regardless of the size of this thing. 30 seconds at 150 miles per hour is knocking down massive trees here… And so I’m assuming it’s taken a drone off its course quite a bit.

Oh, yeah. There’s a certain mountain pass in Rwanda I’m very familiar with, because we have to fly through this mountain pass to serve about a third of the country there… And in this mountain pass the winds get whipping. The winds can be higher than we fly. We fly that aircraft at 60-70 miles an hour, and the winds get up to a hundred miles an hour… And you’ll see our drones flying backwards as it tries to fight through it, developing the control system and logic to… Because most of those winds are gusty, so if you can stay in the air long enough and then punch through it at the right moment, you can get to mountain pass. And yeah, it was a big breakthrough when we figured out how to make that logic robust. And you see the drone is flying backwards for a while, flying backwards for a while, flying backwards for a while, and then flying through at the right second; it’s estimating the winds dying down online, and then it punches through the mountain pass and gets through.

Do you have any video feeds? Because you’re going to come up with some great drama in those moments if you could capture a video, just when it’s really gnarly, you know?

I’ve got a great video of the logs of that. It’s not [unintelligible 00:43:28.29] You can see the position on the maps, going backwards and things…

Yeah, I don’t think Hollywood’s going to call you for the log file.

Let’s talk more about that wing. So did you guys just put the wing on there because you knew eventually you were going to use it, and so you just had it in manufacturing for a long time, but couldn’t use it?

[00:43:47.05] Yeah, absolutely. We knew we needed the range. You get about 10x more range flying on a wing, for the same energy, than hovering. And our customers need that range. So many of our use cases need that range. And so yeah, we always knew we needed the range of flying on a wing, and we decided to launch without having developed that system yet… Really just because that’s how we’re w