Featuring

Sponsors

Tiger Data – Postgres for Developers, devices, and agents The data platform trusted by hundreds of thousands from IoT to Web3 to AI and more.

Augment Code – Developer AI that uses deep understanding of your large codebase and how you build software to deliver personalized code suggestions and insights. Augment provides relevant, contextualized code right in your IDE or Slack. It transforms scattered knowledge into code or answers, eliminating time spent searching docs or interrupting teammates.

Outshift by Cisco – The open source collective building the Internet of Agents. Backed by Outshift by Cisco, AGNTCY gives developers the tools to build and deploy multi-agent software at scale. Identity, communication protocols, and modular workflows—all in one global collaboration layer. Start building at AGNTCY.org.

Miro – The innovation workspace for the age of AI. Built for modern teams, Miro helps you turn unstructured ideas into structured outcomes—fast. Diagramming, product design, and AI-powered collaboration, all in one shared space. Start building at miro.com

Notes & Links

📝 Edit Notes

Chapters

1 00:00 This week on The Changelog 01:17
2 01:17 Sponsor: Tiger Data 01:38
3 02:54 Start the show! 01:10
4 04:04 Some history 13:17
5 17:21 Research use 01:39
6 19:00 Sponsor: Augment Code 01:33
7 20:33 CLI install patterns 04:40
8 25:13 15k people run the world 01:53
9 27:06 Tracking the funding 03:58
10 31:04 Friends tipping circle 01:55
11 32:59 How he stores everything 03:31
12 36:30 Who's involved 04:18
13 40:48 Footing the bill 09:11
14 49:59 The black sheep 09:34
15 59:33 Sponsor: Outshift by Cisco 01:13
16 1:00:46 Sponsor: Miro 01:27
17 1:02:13 The schema is not simple 04:51
18 1:07:04 Curbing the enthusiasm 03:21
19 1:10:24 Deciding what to work on 05:39
20 1:16:03 Metadata substrate 05:13
21 1:21:16 Designing useful tools 03:38
22 1:24:54 Telemetry via exhaust 02:05
23 1:26:59 Information black holes 02:55
24 1:29:54 Octobox update 01:05
25 1:30:59 Exciting new uses 01:01
26 1:32:00 OSS Taxonomy 07:15
27 1:39:14 What Andrew wants 02:45
28 1:42:00 Wrapping up 00:22
29 1:42:22 Closing thoughts 01:35

Transcript

📝 Edit Transcript

Today, we’re joined by, for us, an old friend, but a long time no talk… Andrew Nesbitt is here with us. And you know, Andrew, I came across Ecosystems, which is ecosyste.ms - nice domain hack; hard to say out loud, but it looks cool in the URL bar…

I came across this and I thought “This is a very cool project. It seems somewhat familiar. I can’t quite put my finger on what it could possibly be… And then I saw it was from you, and I’m like “Oh, it makes total sense.” This is right up your alley. We’ve had you on the show many times back in the day, talking Octobox, talking Libraries.io, talking Ruby ecosystem and dependency management… And it looks like you’re still out there, kind of beating around that same bush. So first of all, welcome back to the show…

Thanks for having me. Yeah, it’s great to be back.

Ecosyste.ms. I mean, okay, we have a lot of context that maybe our listeners don’t share, but take us back to what you’re interested in, which - it seems like you’ve been interested in similar things for a long time. And you built libraries.io around this, and Ecosyste.ms is a very similar thing… I’m wondering if it’s the same old thing, or if it’s a new-new thing… So tell us about your past and like collecting and organizing dependencies, and the information about them, and open source projects, sustainability, and then how that brought you to Ecosyste.ms.

Yeah, okay. So I have been swirling around the world of open source metadata for - it must be coming up to 10 years now. Starting with 24 Pull Requests.

That’s right, 24 Pull Requests, yeah.

And that didn’t kind of start from metadata, but the idea of that project was to encourage people to contribute to open source as part of kind of the run up to Christmas… And after kind of like first getting that off the ground, we quickly ran into “Oh, how do we suggest – where should people go and contribute to?” And a lot of people would try and send a pull request to a project that had no activity, and like the maintainer was gone, or just were struggling to be able to even like work out how to send a pull request to some projects, because they were really not very friendly or easy to contribute to.

And that kind of led me down this path of “Okay, well, how do you define what a good project is?” And then “Can we scale that up, rather than manually having to have people kind of like submit their things and keep those things up to date every year?”, because that project would just kind of come and go every December, and shut down afterwards… So the maintenance there couldn’t be entirely human, because there was thousands of people contributing to that project, and sending pull requests… And it was a lot of data to try and work with.

So I started to build out some basic metrics there to try and go “Does this project look like it has activity that’s happening on it? Does it look like it’s ever received third party contributions?”, and things like that. And that led me to kind of – I got a job at GitHub from there, and then GitHub promptly fell apart internally… Tom Preston-Werner left, it was a horrible time… And so then I left there and started Libraries.io as essentially like a - okay, well, looking at package manager metadata is a different way of kind of getting some measure of what’s an interesting open source project. Like, rather than just using stars, which - stars is a terrible metric, and has very little kind of bearing on a lot of projects, especially as you go down from the massive frameworks, those huge keystone projects. Once you get down to smaller libraries, and also especially the kind of low-level critical projects that are doing a lot of the real work… They don’t get a lot of attention, and a star is basically a measure of attention, how many people are landing on that GitHub repo page.

So package manager metadata was like “Oh, this is really juicy”, because it kind of gives me a hook into saying “These libraries are being used by other people.” But download stats - again, available for most package managers, but not all - is often kind of wildly all over the place for certain projects, especially if they’re used a lot in CI; you’ll just see really inflated download stats. And you also don’t necessarily see those for dev dependencies, the things that people, especially maintainers are installing on their laptops to be able to work on those projects. But they’re not necessarily a runtime dependency of all the applications; there are definitely gems that Ruby and Rails devs use locally, but aren’t shipped with the Rails app, so you would never see those numbers.

[00:08:18.01] And the insight that I kind of accidentally tripped over was if we go mining the dependency information out of open source repositories at a large scale, you actually start to get a really good picture of how people really use open source, and how they don’t use open source. Like, if a project breaks, you probably don’t go and un-star that project, let’s be honest. Not many people are un-starring things. They don’t remember. And also, you don’t un-download a thing. The download counts remains after you downloaded it and was like “Oh, this doesn’t actually work” or “This is not what I wanted”, or it has become unmaintained. Whereas actual “I depend on this thing”, if I remove that thing as a dependency, then numbers go down, and you get a really interesting, strong signal that something is maybe not quite right with that project.

So that kind of led me onto a path of “I should just try and index the dependencies of every open source project ever.” And libraries.io started out as a search engine, designed to be like “I can help you try and find the best package.”

And that was primarily like “This package is well used, so therefore that implies it has good documentation, that it actually works, and other people are using it as kind of a proxy.” And it grew and grew, and became a massive and expensive and difficult project to maintain as a side project, whilst I was doing contracting. And me and Ben, who were working on it, we’re like “Well, what are we gonna do? How can we turn this into a sustainable project that can fund itself?” And at the time, GitHub had just implemented its own dependency graph as well, along with purchasing Dependabot… And that basically – they started giving that away for free. That pulled the rug out of any plans we had to monetize libraries.io directly, as well as a project I was building called Dependency CI… Which never really got off the ground, but was back in the day was like “Oh, this is really cool”, because it could literally like block your pull request to say “You’re trying to add a dependency here that is not good, because it doesn’t have a license”, or it’s got security issues, or other things. And so we ended up selling to Tidelift, to try and find some way of recouping the costs of building out that project… But just before we did, we also made all of the code open source, and all of the data open source. So it was kind of like an airdrop into the community to be like “This is always gonna be here if you wanna use it for purposes.”

Didn’t really work out at Tidelift… There’s a big cultural difference in the founders at Tidelift compared to me and Ben. Me and Ben are very – we really like building and solving problems in the open, and shipping stuff really quickly, and kind of iterating on those things… And Tidelift’s culture was - because they just sold to another company…

Yeah, who bought Tidelift?

Sonar? I can’t remember the name. It’s a security company.

And as a shareholder of Tidelift, I can tell you, I didn’t get anything from the sale.

[00:12:02.19] But Libraries.io was there, and was open source, and after – I took a break for a little while during the pandemic, which - you know, everyone had a kind of a crazy time… I went to do some contracting with Protocol Labs, basically kicking the tires on IPFS and Filecoin, and trying to use it as a real user… It was an interesting time to actually try –

[unintelligible 00:12:31.22] was pretty cool.

Yeah. [laughs] And then at the same time was talking to Schmidt Futures, which is now Schmidt Science… But one of the kind of sub-foundations of the Schmidt Foundation, who were basically saying “We have researchers that were using the data from Libraries.io for research”, but now Libraries.io – when I left Tidelift, they started to remove features of Libraries.io, especially the API access and the data… And Schmidt Futures basically kind of came along and said “Could you stand up another copy of it?” And I was like “We could do that… But what if we rebuilt it from the ground up as infrastructure for research purposes?”, rather than taking the same code, which is like one big search engine, one honking great Rails app, and actually make it into kind of a slightly more – like, take all the lessons learned, but instead of building it as a search engine, instead build it as a base layer of open source metadata, which then can be used to build a Libraries.io on top of it. And that also means we can take some of those lessons that were like “Oh, actually, it turns out contributing to a project that has one absolutely enormous database schema is really difficult.” Like, trying to stand that up yourself is really hard as a contributor. So people will just bounce straight off the project, because they’re like “Well, there’s no way I can possibly comprehend how big the stuff that’s going on here…” And then also, the performance implications of deploying a change, that might be like “Oh, you’re about to touch a table with like a billion rows in it.” That’s gonna be difficult for you to test without me giving you production access… And I really don’t wanna do that to random third party open source contributors.

So Ecosyste.ms is essentially a do-over of Libraries.io. It’s many different Rails apps that are focused on collecting different kinds of open source metadata, and then combining them together in different ways. So there’s a packages service, there’s a repo service that collects the dependency information from repositories, there’s an advisory service, and a commit service, and an issue service… Basically, all the different things that you might be interested in. And each one of them can then be independently worked on and scaled up as different amounts of data [unintelligible 00:15:10.20] and kind of collected in different places.

And that has been going on for nearly - I wanna say three years now… Really kind of like going from – it was a nice kind of year where I just worked on it myself, didn’t really tell anyone about it, just kind of like plugged away… And there are core pieces – because Libraries.io is open source, I was able to reuse the dependency parser and a load of the mappings to the package managers… Actually take that code and kind of reuse that in a way that also allows you to have multiple different package manager registries, where Libraries.io would only support one… Which was really nice when RubyGems had all of its drama recently, and the gem.coop popped up… I was able to go “Oh, I can quickly start indexing gem.coop.” It just fits straight into that new schema. And then since kind of like the past year, it’s just absolutely exploded in usage. The amount of traffic today alone was 50 million requests to the API.

And it has become quite a piece of critical infrastructure to a number of different kind of areas of open source in terms of SBOM enrichment, and also trying to find those critical pieces of open source that need security work or need sustainability efforts to be kind of coordinated around them.

Well, I’m happy to hear that you got to reuse some of your code from Libraries.io, because what I thought was gonna happen when you said “I airdropped it”, I thought you were gonna just catch your own airdrop a few years later and be like “And because I open-sourced it, I just relaunched it under a new…” But obviously, the big rewrite is a very tantalizing thing, especially when you’ve been living with all your mistakes for this time. It’s like “Let’s start over…” But you got to reuse some of your code, which is really awesome. So nice job open-sourcing that when you still had an opportunity to do so.

Yeah, absolutely. You mentioned this is used in research… I guess, research terminology, so to speak. What exactly does that look like? Who are those folks? What kind of research are they doing? Are they developers? Are they developer-adjacent?

I think mostly developer-adjacent, or in the research space I guess you’d call them research engineers… Where lots of computer science researchers are like “We want to study what these behaviors are like across different package managers”, or comparing what are developers doing in this space versus that space, especially around the dependency stuff, to be able to go “Oh, the average number of dependencies in a JavaScript app compared to a Ruby app”, for example, which I think is about 10X… And then looking at kind of “Can you go down those dependency chains and find where the security problems are, or the license problems are?” And also leading into kind of “How can we encourage best practices in this space?” Or work out “How many projects have taken on these various kinds of”, specially – just recently I had a call with someone who’s looking at all the attestations around trusted publishing. Like, can we see the share of usage of packages that have the trusted publishing setup, and are publishing attestations into a SIG store, compared to the overall space? And also then breaking that down across different ecosystems as well.

Break: [00:19:00.27]

This might be silly, but let me ask you this… I’ve been researching some CLIs and I’ve been researching how CLIs install themselves. Sometimes they’ll leverage the actual package manager of the distro, like a Linux distro or something like that… But most, by and large, just give you a URL to curl and pass to Bash, essentially… Which can be problematic if you don’t trust the script.

If I wanted to somehow research CLIs and how they install themselves, and the various ways they install themselves, is that something that this service could do? Is that the level of research it could do?

Yeah. I mean, for one thing, you would be able to quickly find everything that had kind of tagged itself up as a CLI program. I’ve also been indexing every public image on Docker Hub, and basically running an SBOM scanner against each one of those. There would be some juicy insights there, to be able to go “How many–”

“…of these things were installed via a distro package manager, versus “we just have a URL for this”? Which would be recorded in the SBOM, basically to say “Oh, we’ve found this known bit of open source, and it appears to say that it sits in this file system here”, which implies it was installed by apps, or it’s in a random space, like it was probably curled down, along with the Docker file that was used to build that image. And there’s a good kind of million open source Docker images on Docker Hub, or at least individual versions of things. And you also get the interesting aspect there that you can kind of multiply that by the number of downloads that some of these Docker images have… And some of those numbers are crazy. Millions and millions of downloads of a particular image. And of course, those numbers inside that one container are never reflected in the package managers upstream. So just because it was downloaded in Docker doesn’t mean that that actually shows up as being a million downloads in RubyGems, or on Npm. So you start to see some really interesting things, and you start to see those download numbers, or the proxy for a download number of distro packages as well… Which is a really hard number to get a hold of, because every distro package manager is very heavily mirrored, and basically just a file system somewhere exposed over HTTP or Rsync. So no one has good download stats for those things. The only place you really find that is the Debian popularity contest, which is opt in, not opt out.

So you’d be able to go “Oh, okay. Well, I can see – here are the CLI programs that are being manually downloaded inside of Docker images as part of this install process.” It’s not gonna give you everything, but it certainly gives you a good proxy for “Okay, well, I can see where –” Like, relative usage of these things starts to show up, which is where I’ve found the most useful ways of kind of sorting different piles of packages or whole registries, is to go “Okay, well, if I sort this registry by the number of dependent repositories, or the number of dependent packages, which things show up at the top?” And then also, which of those things make up 80% of all of this stuff?

[00:24:28.27] And you actually end up looking, like – I like the 80/20 rule, but it doesn’t actually turn out to be like 20% of packages make up 80% of usage. It’s like 0.01% of packages make up 80% of usage. These tiny amounts. There might be 2000 Node modules total that make up 80% of all of the usage of Npm in terms of downloads, and in terms of discrete dependent repositories…

When you then start to really focus that lens, you see a long tail of stuff that never gets used, and it’s also like all kinds of spam and malware and stuff that floats around. But there’s 10,000-15,000 packages which are the packages that make up most open source usage across all these ecosystems.

It’s kind of amazing how massive that asymmetry is when you pin that down to the individuals…

Yeah, and that’s on average one maintainer per package at that critical level as well… So that’s like 15,000 people maintaining all of open source usage.

That makes the XKCD comic even more poignant. Now you’re the one person in Nebraska - replace Nebraska with wherever they are in the world; probably in different towns…

And how many of them have you had on the Changelog?

That’s a good question. Probably a good percentage of those. Oh, man… So there’s 15,000 people basically running the world for free. [laughs]

Well, I have done a little bit of indexing of how many of those have GitHub Sponsors, or are their projects on Open Collective, or they have some other kind of funding link… And in terms of those top critical packages, it comes out to kind of like - depending on the ecosystem, it’s somewhere between 25% and 50% have some way of “Here’s an automated way you can give me a donation to the project.”

There’s a good chunk of those as well that are massive, corporate-funded projects. Like, all of the AWS RubyGems that make up the AWS CLI are in the top of RubyGems, because they’re just massively used. They don’t need any funding, because Amazon has full-time staff. But there’s a good –

They might need some funding. I hear they’re laying people off again… [laughs]

Hopefully, they didn’t lay off all the Ruby people maintaining the CLI there.

Yeah, that would be awful. So do you track – so you’re tracking those who are able to receive funding in some sort of automated fashion. Do you track funding itself? Like, who’s getting how much money, and how?

Yes. Well, where possible. So I’m tracking – I call it a funding link, and some package managers have funding links support, where you can say “Oh, you can donate to me over here.” Repositories have the funding YAML file, and I go looking for that wherever possible. And you actually see that even on GitLab and Codeberg. I don’t know how well those platforms display it in the UI, but it definitely – because obviously, GitHub Sponsors is not… I don’t think there’s a GitLab Sponsors or a Codeberg Sponsors. Those files do show up all over the place.

[00:27:54.27] And then also being able to go “This repository is owned by a user on GitHub who is part of GitHub Sponsors” is another way of kind of detecting that. Even if they haven’t added their funding YAML file, we can kind of make a hop to say “Oh, here’s one of the maintainers to be able to support that.” I then collect the data from GitHub Sponsors of every – because GitHub Sponsors users are public. You don’t get any financial numbers, but you do get “Here’s the number of active sponsors of things, and here’s the total, like all-time…” It’s quite hard to get time series data out of that API, so instead I basically just kind of snapshot it on a regular basis, to go “Oh, here’s what’s the current state of the world in terms of GitHub Sponsor funding.”

It’s a bit weird, though. A lot of people who have realized that GitHub Sponsors is actually quite a good way to sell digital goods. If you go looking at the top users of GitHub Sponsors who have the most people funding them, they sell things like avatars, and Discord memberships, and eBooks, and things like that. They’re not necessarily kind of selling “Oh, I can maintain this project better for you.” That’s not – like, Open Collective is so much bigger in terms of actually like supporting the projects as a collective, because they’re just set up in a totally different way to GitHub Sponsors.

Yeah. That’s fascinating. So they’re kind of doing sponsorware, insofar as it’s not a donation, or “You’re supporting my work on this project.” It’s like “Actually, there’s a quid pro quo here. We’re going to trade a good or a service for that sponsorship money.” Really, it’s a purchase of some sort of thing.

Yeah, yeah. If you go looking, it’s easy to see GitHub doesn’t make it particularly – like, they don’t have a leaderboard… Which is a good thing, to not – like, putting a leaderboard on things can often produce some very strange behaviors…

There’s also an interesting breakdown of like number of users who sponsor other maintainers, versus companies. Obviously, companies are going to sponsor a lot more in total amount per company, but the distribution is quite surprising in – you’re looking at easily 10 times as many individuals are sponsoring other people on GitHub Sponsors compared to the number of organizations. Like, it’s quite small, really.

And most of that activity is public. So it’s not like there are – you can be anonymous as a GitHub Sponsor, but you can’t really hide the fact that there is a sponsorship happening there. There’s also on Open Collective some massive donations that go to certain projects through company sponsorships, because they’re acting as a fiscal host, rather than just being a platform to collect tips, which is basically how GitHub Sponsors works.

Right. It reminds me of way back in the day, Chad Whitaker’s Gittip, which was later called Gratipay.

And it felt all warm and fuzzy, because people were getting money for their open source, but when you go looking at it very closely, most of that was like the same 50 bucks getting passed around between friends… Not a slush fund, but like a – they just felt good. So I would make 20 bucks a month and I’m using open source, so I would give it to somebody else. And there was really no new – not enough, new money coming in. It was really just money that already existed amongst all of us maintainers kind of patting each other on the back… Which was unfortunate, but just the way it started.

I definitely do that. I sponsor 35 different people on GitHub Sponsors with just a few dollars a month, to just be like “I appreciate your work.” I don’t have a huge amount to support you with, but just as a way of saying “I noticed you and appreciate that you continue to maintain these things that I use.”

[00:31:59.16] Well, I hoped GitHub Sponsors was big enough and mainstream enough to kind of change the shape of that. And maybe it’s done it some, but it sounds like there’s still more indies passing person-to-person kind of sponsorship than there is corporate-to-person.

Yeah, I think the change of interest rate across the world had a massive impact. The nice thing about Open Collective is they are – especially Open Source Collective is very public. You can see the amounts of donations going in and going out… And there was a big drop around the time that – like, post COVID hit and changed all of the finances of these things, and it was like “Oh. Okay, well, open source is no longer one of the–” It’s an easy line item to drop, because “Oh, everything is free, and it just continues to work…” For now, until a security problem comes along and then everyone starts scrambling again.

So you’ve got 12 million packages being tracked, 287 million repositories, 24.5 billion dependencies, 1.9 million maintainers… I’m reading these stats off of your website. There’s a timeline of like public events on GitHub, there’s issues, there’s commits… I mean, there’s just tons of different data points that you’re tracking. How do you store all this stuff? Where do you store it? How big is it all? Because I’m just thinking this is a data management nightmare.

So that 24 billion dependencies is a bit of a headache.

[laughs] I bet. I mean, that’s crazy.

Almost all of this is stored in Postgres.

Individual Postgres instances on dedicated machines in France and Amsterdam, mostly because they’re very affordable. Online.net is a very reliable host, similar to Hetzner or some of these other kind of bare metal machines.

So I do the maintenance of the machine myself, and obviously, scaling up is a little more tricky, because there’s not just a nice Heroku slider anymore… I use Dokku as essentially like the open source Heroku, which is really nice.

Just git push, it builds your Docker image, and then it handles putting NGINX, kind of proxying all of those things. Very nice for like an individual machine. It doesn’t really give you any kind of multi-machine things, but I try to avoid too much complexity when there’s only a very small number of people working on doing the infrastructure. And it’s mostly me, rather than – I calculated, like a back of the napkin thing the other day, I think it would cost me 15 times as much to host on AWS as it does to host it on dedicated machines right now… But these Postgres – each service basically has its own database. So rather than it being one that is enormous, it’s split out… Which at least makes it kind of like I can work on individual ones and be like “Oh, this one is reaching capacity, so it’s time to scale it up”, or “I should make another box of web machines or Sidekiq workers separately. I don’t need to kind of do everything in one big lockstep”, which keeps it fairly easy to do.

And then the whole website is basically read only. Like, you can’t ask – you can’t put data into it as a user. You read from it. And all the data comes in in the background through loading data from package managers, and repositories, and… There’s about 2000 different Git hosts in there that I’m constantly crawling at different rates to go like “Oh, there’s new activity over here.” So I can cache things very aggressively at the kind of HTTP layer. I think the cache hit rate at the moment is about 60% in Cloudflare. At some point I’ve got it all the way up to like 95%, but then you get some AI bots come along and they do some weird stuff, and it’s very hard to cache such a long tail of billions and billions of URLs that might exist on the platform. And Cloudflare on the free plan is not gonna cover an unlimited amount of cache. You just kind of keep rolling over the cache, over and over again.

[00:36:31.09] Is this a solar project again, or is this you and Ben back together…? Who’s the band?

So Ben is working on it part-time. He is also one of the directors at Open Source Collective, which is – you know, that’s a lot of work in itself. And then we have a few people who are doing some part-time work. Martin has done all the design work… Which looks so much better than my efforts of the original – you can see, there’s a couple of older hidden web pages there that are very poorly designed, which is just me making some plain bootstrap pages… And we just had James come on to help with making the project better-documented and easier to onboard as a contributor… Because I was running so fast on standing everything up and scaling it up and collecting all that data that I didn’t really leave a lot of documentation along the way… Which is terrible, but - hopefully, these are pretty basic Rails apps. There’s not a lot of interesting stuff. Like, intentionally trying to make it the most boring tech possible, so that I can focus on the interesting stuff, which is like the parsing or the mapping of the metadata… Which is like each app has that core little nubbin of “Oh, here’s where the real logic sits.” And that’s a nice, well-tested bit of functionality, with a load of Rails scaffolding around it to be like “Okay, write this into Postgres and then serve it up in kind of the quickest way possible.”

Oh, good question. It must be coming up to 20… But some of them are quite small. There’s a load of services that are kind of like stateless. Like, I will just give you a SHA-256 of a tarball that you get from RubyGems or similar, and a lot of those I basically have on the chopping block to try and turn into something a little bit more like – imagine a GitHub Actions, but for analyzing packages… So rather than it happening every time that you commit or every time you open a pull request, instead it’d be like you can define “I wanna run this kind of analysis on this package when a new version comes out.” That might be like copyright and license extraction, or it might be “Do me a capabilities analysis of this go package using the Caps Lock library…” Which will basically go like “Oh, this library just gained network access and it can read environment variables, and it became a crypto miner.” It would be a great way of like being able to highlight some of those changes.

So I wanna pull it down, and make it a little bit kind of like fewer services, but one of those services will be basically the “Which open source analysis do you wanna run against this package?” And then “Here’s a massive fire hose of every activity that is happening”, and you can hook those analyses in, to say “Okay, I wanna run Zizmor every time I see a GitHub Action change”, because Zizmor does the security scan on the YAML config to go like “Oh, you’ve just introduced a foot gun of GitHub Actions here.” And then try and publish all of those analyses back out as a public good, just basically fling that into S3 or something as a way that allows researchers, again, to go and do broad analysis over the whole ecosystem, or multiple ecosystems, without having to spend all their time collecting all of that base data, and normalizing it, and then setting up infrastructure to run all of that across all of those packages. I see that time and time again, where the paper is – like, 50% of the work is “Oh, well, we had to collect all of this data, and we had to make sure that it all fit into the right box.”

[00:40:28.07] And then we could actually start doing the interesting research. So what I hope is we get to a place where it’s like “Oh, you don’t need to do that. You can just use this open dataset”, and that gives you a good starting point to then start to really dig into like “What’s going on in these ecosystems?” That’s the dream anyway.

Well, you’re certainly working your way towards that. So does Schmidt Sciences - do they foot the bill for all of this work?

So they gave a grant initially, to get started. Luckily, they gave it in dollars, and the exchange rate was very positive for a while, so we actually managed to stretch from a one-year grant into a two-year grant… And then Open Collective has been supporting the project as well as a fiscal host, but also as like a customer. So I built a number of tools for them, to help them kind of investigate ways of trying to expand the ability to kind of let companies fund open source, and then also to try and measure the return on investment of giving two projects and try and be able to see “Oh, if I donate money here, or resources, does that turn into actions and changes on the repositories?” And that kept me busy for a good nine months, I think, of building out tools for them whilst they financially supported the project.

And we also have a number of customers who pay for a different license for the data. So the data is CC BY-SA, which is like a copyleft license. You can use it for whatever you like, as long as you also persist the license and you credit where it came from. But if you don’t wanna do that, then you can pay to essentially have a CC0 license. It’s not actually CC-0, because there’s some things there to say “Oh, don’t just completely undercut us and sell that on again.” But we have a number of customers there… And that basically pays for all the hosting costs.

So it’s self-sufficient, it runs itself, as long as – but you don’t get any extra feature development on top of that. So that’s where I’m trying to work on right now, is to get that level of sustainability higher. And we’ve just received a grant from Alpha Omega, to basically make that happen.

Alpha Omega is part of OpenSSF, and their goal is turn money into security. And they have become a big user of Ecosyste.ms for doing analysis of like who are the critical projects in a particular space, who are the ones that are gonna be most likely impacted if there’s a big security vulnerability? Who are the ones who have never had a security vulnerability and maybe don’t know what to do if they get one? …things like that. So they have basically given us a grant to try and help make Ecosyste.ms long-term sustainable. So that’s things like making the project easier for people to onboard onto, and also to be able to kind of charge large companies in different ways. That might be like “Oh, you want an even higher rate limit than the very friendly rate limits that are already on there? Do you wanna go even harder? Well, then you can pay for a super-rate limit, or similar.”

[00:44:07.28] And then also this kind of pipeline of analysis will be another way that – it’ll basically be like “Oh, you wanna run your LLM queries across all these package source code? Well, then you can funnel it through here. We’ll just like tee that up and trigger it every time that we see a new release of a package, or similar.” It will be another way that I think would be – essentially, just like “Oh, you’ve just paying for our CPU to do this analysis”, and then the analysis that comes out the other side, if it is idempotent, I guess… LLM queries are not idempotent. You’re gonna get a different thing every time you do an analysis. But for a lot of those things it will just come out as a public good, and companies will have paid to have it generated, but then it’s shared for everyone to use… Which I think is a nice thing.

I mean, what I’d really like to be able to do then is to actually do revenue share with the people who are maintaining those individual command line tools that do the analysis. Imagine being able to go like “Oh, we can help with supporting Zizmor, and Bullet”, or all of these different things that are command line tools that analyze source code. And rather than you build a whole enterprise company around your command line tool, you can just focus on making that tool really good, and then we can run it at scale for customers, and then just funnel the money back to the maintainers, after whatever infrastructure costs there were to run it, so that you can actually focus on building the open source tools, rather than building the scaffolding around it.

That would be super-cool. So it sounds like there’s a collection of potential income sources, some that are currently working, other ones that you’re working on… The relicensing of the data for a fee seems like a good one. Is that potentially – could you see a world where there’s enough people that want to do that, that that could be enough, or no?

Yeah, I think so. Especially this kind of dependent data, the 25 billion row table is really juicy in terms of the insights that you can get from that. The general package data though is often – like, you can get Claude to generate you an Npm scraper very easily. If you ask it to do it in Ruby, you get code that looks a lot like Libraries.io [unintelligible 00:46:33.17] [laughter]

Do you get a nickel when that happens, or what happens? [laughs]

[unintelligible 00:46:39.08]

Yeah. Well, you know, imitation is the sincerest form of flattery… So just remember that.

Yes. It’s tricky to get that kind of balance of like – we want to give away as much as possible, especially as all of this data comes from open source. Like, it should be open, because it is data about open source. But then how do you continue to pay for that? …whilst companies also can kind of go like “Oh, I could just go fetch it from the source myself.” And trying to get as many different ecosystems support in is a good way of kind of going – like, you really don’t want to try and index the R package manager. Like, you’re not going to have a good time.

So we try and take care of all of the horrible bits… And then also being able to fetch the Linux distro package managers, which is something that I’m trying to add more distro support in… Because each one of those has its own kind of like horrible rabbit holes of weird and wonderful metadata. And trying to work out “How does this fit into the schema?” A lot of it is kind of trying to tie it around the package URL format. Perl - but not Perl word language… Although you can have a [unintelligible 00:47:55.09] That has kind of come out from efforts in the SBOM world, and originally, one of the inspirations was Libraries.io being able to map these things into different ecosystems and kind of say “You have an ecosystem, you have a name of a package, and you have a version. Can we talk about this in a fairly standardized way, as a way of transporting these package bits of metadata between different platforms that are doing analysis of different kinds?” And SBOM is the natural conclusion of that.

[00:48:41.13] Of course, you have two different SBOM standards. There can’t just be one standard for things… But being able to look things up by [unintelligible 00:48:50.15] is something the ecosystem does really well, because you can basically then take an SBOM and just work through it, every single package that’s in there, and say “Can you tell me about this package? Can you tell me what security advisories are affecting the version that I’ve got in my SBOM?” And that is the biggest use right now, is there are lots and lots of people with GitHub Actions that are just enriching their SBOMs with this kind of information.

It’s funny how much more traffic we get on a weekday than on the weekend… And I think it’s just because of the GitHub Action kind of like “Oh, this is happening every time someone commits”, so you see a smash of traffic of them enriching their SBOMs and checking out every package that is in there… And then the weekend comes along, everyone stops working, and the traffic shape completely changes. And also the cache hit rate completely goes through the floor, because suddenly it’s like “Oh, there’s all kinds of other weird and wonderful things happening at the weekend”, especially lots more like researchers and hobbyists using it.

So you’ve mentioned a few of the weird, gnarly things like multiple SBOM specs etc. You have 35 ecosystems on here. Npm, Golang, Docker, to name a few. Crates… NuGet, so you’re in that world… Across 75 registries - so I’m assuming some ecosystems have multiple registries…

Yeah, Maven especially. There’s lots of registries in the Maven world.

And then – oh, even Bower.io. I remember Bower. I don’t know if people are still using that… Anyways.

No one adopts anything, they don’t accept any new packages, but you’ll still find people that use them and download stuff through them, yeah.

So what I’m wondering is, where are the black sheep? Where’s the gnarliest, weirdest – like, let’s not… I don’t wanna create any enemies for you, Andrew, but which of these ecosystems are, in your own heart of hearts, notoriously hard to work with?

Well, the hardest bits are often the change over time, especially when you go back to the really old stuff. The classic one is that you’d think “Oh, Npm - their names are case-insensitive.” But if you go and try and index every name in Npm, you will find about a thousand that are case-sensitive and have clashes with a different, cased version of the name. And those still exist on the registry. They haven’t been removed. And so if you try and make an index against that, you’re gonna have a bad time, because as soon as you actually go to run that, you’re like “Oh, that’s not like that anymore.”

So there’s things like that, that when you go back into the time – going back further and further is like “Oh, there’s weird things here”, especially when the package manager registry has a document database, rather than something that is always enforcing its schema in every record. And Npm used to be CouchDB, which is “Oh, they’ve changed some schemas of the package metadata, so in new packages it looks different than old ones.” Of course, now it’s actually Postgres underneath, and it pretends to be CouchDB, which is interesting, and I imagine a headache in terms of actually maintaining that… But they still have some really old and weird – you just run into like “Ah, this bit of metadata isn’t right for these few packages”, because it was frozen in time. There’s JSON in Postgres now, somewhere… Similarly with Maven, they’ve got lots of different kinds of POM XMLs…

[00:52:40.10] And there’s so many features in the way that Maven can have these nested and parent POMs that is – I don’t really have a background in Java, so I’ve never used Maven as a user, but the amount of different ways that you can describe the data that is stored in a POM XML, and then published out to Maven Central… Of course, once it’s on Maven Central and it’s like frozen in time almost, they don’t then go and update – like, if RubyGems adds a new attribute to their registry, that becomes available in the metadata for every single endpoint, because you know, it’s just a Rails app that’s generating JSON. But for the things that store the files as a historical “We just dumped this file somewhere”, then you’re like “Okay, my code needs to be able to know every different possible version of this, how this worked, and then also be able to recover from it.”

The worst one is the R package manager. It’s not huge, but it is used a lot in the research space… And they don’t have an API. You have to scrape HTML from the thing. They also remove packages quite regularly, which is very strange. So R has this really weird – I think it’s because it’s come from a scientific kind of like non-developer background. It also has one indexed arrays, which not many programming languages have that, right? But their package manager won’t let you pin to an older version of something. It won’t say “I want version one”, even though version 2.0 is out. And the knock-on effect of that is that – so as a user, if I’m gonna say “Install my R packages”, I always get the latest versio