Tackling multiple namespaces in ocgtk

This post is part of a series about my work on ocgtk, an OCaml binding for GTK4. The most recent release is preview1.

My intention with ocgtk was to generate bindings for GTK4 and ignore the nine other libraries it depends on for as long as possible. A look at the interface made that untenable: much of GTK's useful surface uses types from those other libraries.

Skipping methods that touch types from those libraries would leave the OCaml binding without a large portion of the API surface, including several critical methods for building any application.

Accounting for the whole family

I had good reasons for wanting to delay cross-namespace work. GIR1 does a reasonable job of describing C interfaces in a way language bindings can consume, but mapping its type system into a coherent operating model is genuinely hard, especially once you go past primitive types into records, enums, and user-defined types.

C is a permissive language. Its syntax doesn't encode the conventions a library expects you to follow. Memory management is the obvious example: it's up to the library author to decide who frees what, when, and according to what discipline (such as reference counting, manual free, shared ownership, autorelease, zones, etc).

C consumers usually pick those conventions up from documentation and hope they follow them strictly. Code generators need a precise specification per function.

Even when the library narrows the memory model down, several questions remain. GObject uses reference counting, but you still have to know:

• whether the caller or the callee owns each value, and whether ownership transfers between them
• which function manages reference counting, and whether the same one applies to every type
• whether the binding has to use a specific allocator
• whether there's an auto-release mechanism, where values not held onto get reclaimed at the end of some scope (an event loop run, typically)

And GObjects aren't the whole picture. GIR also describes records (separate models for opaque and transparent structs), arrays, and linked lists. The lower-level libraries like Gio and GLib add more variations still, each one another C-interop case the generator has to handle.

Bitfields and enums - the original cross-namespace hack

A significant portion of the GTK API relies on bitfields and enumerations from other libraries, so the original version of ocgtk would rely on a fixed set of pregenerated bindings for these types to get a significant portion of the GTK API "over the line".

However, this was not enough for even basic cases e.g.:

• Widget.get_display → Gdk.Display: Every widget lives on a display. Connecting to multi-monitor APIs, getting screen geometry, creating cursors — all require Gdk.Display. Even Window.set_display is cross-namespace.
• FileDialog.open / save → Gio.File, Gio.Cancellable, Gio.AsyncResult: FileDialog is entirely cross-namespace. Every single method on it involves Gio.File, Gio.Cancellable, or Gio.AsyncResult. Without Gio types, FileDialog generates as an empty shell — you can create one but call nothing on it.
• Image.set_from_pixbuf → GdkPixbuf.Pixbuf: Loading and displaying an image from a file. Needs GdkPixbuf.Pixbuf. Without it: Image.set_from_pixbuf is not available.
• Application.get_menubar / set_menubar → Gio.MenuModel: You want access your menu bar — you need Gio.MenuModel. Every desktop application with a menu hits this immediately.

gir_gen works by loading a GIR file, where each file corresponds to a particular library (i.e. "namespace") and generating the bindings for it. For anything cross-namespace, we had the names of what it was referencing, but we also needed access to the definitions that were referenced in that namespace.

Why? At the C level, we needed to know at least the type of the underlying value (class, enumeration, bitfield, record, etc.) so that we could generate the correct C marshalling code for it. Primitive types just need to be packed correctly into OCaml blocks, strings need to be copied into the OCaml heap, and objects need to be ref counted and put into a custom OCaml block with a custom deallocator (that dereferences the GObject pointer when the block is freed from the OCaml heap by the garbage collector).2

Because gir_gen operates at the library level, we would need to load the GIR files for each library and its transitive dependencies. The obvious fix is to load all nine GIR files at once before generating anything. But the GIR files aren't small:

Instead of loading (and processing) all nine GIR files each time, we preprocess them and create a small "references" s-expression file with just enough detail about each available type, that can be loaded later to assist the generation of libraries that depend upon it.3

Generating cross-namespace type references

The references file only needs to answer the questions that the downstream generator will actually ask about a type it doesn't own. All we need to capture is:

• what kind of type is it (class, interface, record, enumeration, bitfield, or primitive alias)?
• what's its C type name, so we can marshal a pointer or value of it?
• for classes and interfaces, what does it inherit or implement, so we can emit the right polymorphic-variant row in Layer 1?
• can we actually generate it, or has it been excluded? (more on that below)

Anything else is the owning namespace's problem. A generator consuming Gdk.Display from Gtk doesn't need to know how to call gdk_display_get_monitors. It only needs to know that Gdk.Display is a class, its C type is GdkDisplay, and it descends from GObject.

Schematically, a namespace's references file looks like this, with example entries for a class, a record, and an enum:

((cr_namespace_name Gdk)
 (cr_namespace_packages (gdk-4.0))
 (cr_namespace_includes (Gio Pango GdkPixbuf cairo))
 (cr_namespace_c_includes (gdk/gdk.h))
 (cr_entities
  (((cr_name Display)
    (cr_type (Crt_Class (parent (Object))))
    (cr_c_type GdkDisplay))
   ((cr_name RGBA)
    (cr_type (Crt_Record (opaque false)))
    (cr_c_type GdkRGBA))
   ((cr_name MemoryFormat)
    (cr_type Crt_Enum)
    (cr_c_type GdkMemoryFormat)))))

The preprocessing step is what lets the generator scale to large dependency graphs. We don't have to load and process megabytes of XML for every transitive dependency, or hold all nine GIR ASTs in memory at once. Each namespace consumes only the distilled _build/references/<ns>-references.sexp files of its dependencies.

Cross-namespace inheritance

Layer 1 simulates GObject inheritance with polymorphic variants.4 A class's type carries a row of tags for itself and every ancestor:

type t = [`button | `widget | `initially_unowned | `object] Gobject.obj

The point of the row is that you can pass a Button.t to anything expecting a Widget.t or Object.t by coercing it to the parent type with :>. The compiler accepts the coercion because the row on Button.t is a superset of the row on the parent, with no runtime cost.

Inside a single namespace, generating that row is mechanical: walk up the inheritance chain until you hit Object, listing tags as you go.

Cross-namespace breaks the assumption in two ways. First, the chain itself crosses library boundaries. Gtk.Application inherits from Gio.Application, which inherits from GObject. Generating the row for Gtk.Application.t requires knowing that Gio.Application exists in another library, what its ancestors are, and what tags it contributes. Second, the same entity name can appear in more than one namespace. Application lives in both Gio and Gtk; Buffer shows up in several. Naively splicing rows together would leave duplicated tags, which is messy at best.

Build order and the dependency graph

Once the references file exists, the generator run is no longer a single pass over one GIR file. It becomes a walk over a DAG, with one node per namespace and one edge per <include> in the GIR header, producing a references file and a set of bindings at each node in topological order.

For GTK 4, the graph is modest but not trivial:

The order falls out of the GIR metadata. Each file declares what it includes, and a topological sort does the rest. The generator runs in two phases for each namespace:

1. Reference extraction: load the GIR, walk the types we can generate, and emit a <ns>-references.sexp.
2. Code generation: load the namespace's own GIR plus the references files of every transitive dependency (passed via -r flags), and emit the C stubs and OCaml modules.

Splitting the passes this way means the second phase never has to open a dependency's GIR file. By the time we're generating Gtk, Gio's GIR has long since been closed, and all we have in memory is the distilled references for Gio, Gdk, Pango, and the rest.

The namespace relationships are a Directed Acyclic Graph because they don't include each other in cycles.

There's one concession to GIR's shape. GObject and GLib are partially handled inside gir_gen itself rather than treated as ordinary namespaces. GLib's types are threaded through every binding (GError, GList, GVariant), and GObject defines the runtime semantics the whole generator is built around, so pretending they're just two more nodes in the DAG would push too much special-casing into the generic path.5

The overrides layer

Two things pushed us toward a structured override system between preview0 and preview1. Cross-namespace generation enabled many more methods to be emitted, which surfaced GIR issues that had been hiding behind the previous filter. And the opam release pipeline now validates the build across Windows, macOS, and several Linux distributions (Ubuntu, Debian, Fedora, CentOS Stream, OpenSUSE). That exposed problems GIR has no way to express, especially around older library versions and what is or isn't available on each platform.

The new system replaces hardcoded exclusion lists in exclude_list.ml and filtering.ml with a sexp file per namespace, at ocgtk/overrides/<ns>.sexp. The directives there cover three kinds of cases the previous approach couldn't:

• Missing or wrong version annotation. GIR can mark an entity with a version attribute that the generator already turns into a GTK_CHECK_VERSION guard automatically, but the annotation is patchy in practice. (version "4.14") adds or corrects a guard against the host namespace's version. (version (pango "1.50")) does the same against a dependency's version macro, which GIR has no way of expressing at all. The cross-namespace variant is what handles distros like CentOS 9, which ships a recent enough GTK but a Pango too old for PangoTextTransform, a type GTK references in Gtk.TextTag.text-transform.
• OS-only entities. Gio.Subprocess.send_signal, Gio.Credentials.get_unix_user, and the rest of the Unix-isms are described as plain methods in GIR; nothing tells you they're not portable. Without (not_os "windows") they generate uniformly and then fail to link on the wrong platform.
• Build-specific or genuinely-internal types. Gio.SettingsBackend only compiles with G_SETTINGS_ENABLE_BACKEND defined, and Gtk's PrintBackend is internal plumbing not in the public headers. Both are simply (ignore)d.

In the generated output, version and OS guards turn into C wrappers. When the runtime library is too old or the platform doesn't match, the wrapper compiles to a stub that raises an OCaml exception on call rather than failing to link silently. The method exists in the binding either way; whether it actually works depends on the environment at run time.

Overrides are applied during GIR parsing, before the AST that feeds reference extraction and code generation. Ignored entities are gone by the time the references sexp is written, which gives us cross-namespace cascading exclusions for free: when gir_gen is generating Gtk and encounters a method whose parameter is a Gio type that isn't in gio-references.sexp, it drops the method rather than emit a broken stub. FileDialog is the canonical case. Every method on it takes Gio.File, Gio.Cancellable, or Gio.AsyncResult, all of which had to land in Gio's references before any of the FileDialog methods could come through in Gtk.

Why did we focus on this first?

The practical test of any of this is whether the things that used to be empty shells now work. A short before/after:

• FileDialog: previously unusable, because every method referenced Gio.File, Gio.Cancellable, or Gio.AsyncResult.
• Image.set_from_pixbuf: previously missing, because GdkPixbuf.Pixbuf was out of reach.
• Application.set_menubar and get_menubar: previously missing, because Gio.MenuModel was cross-namespace.
• Widget.get_display: previously missing, which had the knock-on effect of hiding most multi-monitor and cursor APIs.

Wrapping up

Cross-namespace support was the biggest piece of work in preview1 because the most-used parts of the GTK API live below it. FileDialog, Image.set_from_pixbuf, and the menu plumbing on Application were unreachable until the generator could consume types it didn't own. Three pieces had to gain a cross-namespace mode: the references file as a scalable way to share types between libraries; the polymorphic-variant scheme as a way to replicate the GObject hierarchy without OCaml classes; and the override layer to paper over the gaps that surfaced once we tried to generate all nine libraries together.

In my next article, I'll go into more detail about some of the problems mentioned that I had to solve, such as how we came to decide on the right approach to emulate inheritance, or how we supported interfaces without explicit language support.

1. GObject Introspection (GIR) is the interface definition language that GTK and its dependencies use to describe their C interfaces. ↩

2. This says nothing of records, which have their own complicated semantics depending on whether we can allocate them (not opaque) or we have to leave this to the record's special allocator or constructors (boxed). ↩

3. By "available" types, we mean the types that can be generated by that library - at this point, there is a number of types (or configurations thereof) we don't have generators for (such as certain types of records), so we exclude them from the references sexp so that higher level libraries that use them also exclude their code that uses them and avoid compilation errors. ↩

4. Layer 1 is the direct OCaml-to-C binding layer, which is necessarily functional. Layer 2 sits on top of it and provides an object-oriented wrapper that mirrors the GObject class hierarchy more familiarly. ↩

5. This is the same shortcut PyGObject and gtk-rs take. It avoids a circularity where the generator needs GObject's types before it can describe anything else. ↩