Go 1.26 made Green Tea the default GC. We observe its cache-friendliness with perf, visualize the heap to see how Go allocates, and poke at a sparse-page problem a non-moving collector like Go's struggles with.
By Phil EatonJuly 19, 2026 Focus
Go 1.25, released last year, introduced a new garbage collector: Green Tea. And in Go 1.26, released a few months ago, Green Tea became the default. That linked article is excellent. We’ll recap it and take a look at a few programs that benefit the most. And we’ll look at some programs that don’t benefit, that trigger Go’s residual garbage collector bugaboo: its non-moving collector cannot reclaim sparse pages.
Taking a step back, Go manages memory by allocating objects of the same size class (an object’s size is rounded up to the nearest size class) within a contiguous chunk (or span in Go terminology) of one or more 8KiB pages. Size-segregated allocation is common in some malloc implementations (like tcmalloc, which Go’s allocator descends from).
Let’s observe this happening in Go, and then compare it with C#. We’ll randomly allocate objects of three different sizes (small, medium and large). Then we’ll check their heap addresses, walk the address space and print out when we hit one of our objects, one character printed out for each 32-bytes we walk.
First install Go and C#.
sudo apt update -y
sudo apt-get install -y dotnet-sdk-10.0
curl -fsSL https://go.dev/dl/go1.26.0.linux-amd64.tar.gz | sudo tar -C /usr/local -xz
export PATH=$PATH:/usr/local/go/bin
Here’s the pseudocode we’re going for.
struct Small { a [32]byte }
struct Medium { a [64]byte }
struct Large { a [128]byte }
constructors = [Small, Medium, Large]
live = [] # stop objects from being collected
for i in range(100):
live.push(new constructors[rand() % len(constructors)])
for pass in [0, 1]:
if pass == 1:
runtime.gc() # trigger the GC
records = []
for obj in live:
records.push((runtime.addressof(obj), runtime.typeof(obj), runtime.sizeof(obj)))
records.sort(key = r -> r.address)
cell = 32
cursor = records[0].address
for (addr, typ, size) in records:
while cursor < addr: # no object of ours here
print("."); cursor += cell
head = typ.name[0]
print(upper(head) + "-" * (size/cell - 1)) # "S" / "M-" / "L---"
cursor += size
Let’s build it in Go.
package main
import (
"bytes"
"cmp"
"fmt"
"math/rand"
"reflect"
"runtime"
"slices"
)
type (
Small struct{ _ [32]byte } // 32 bytes
Medium struct{ _ [64]byte } // 64 bytes
Large struct{ _ [128]byte } // 128 bytes
)
type object struct {
addr uintptr
size int
name byte // 'S' / 'M' / 'L'
}
func main() {
allocs := []func() any{
func() any { return new(Small) },
func() any { return new(Medium) },
func() any { return new(Large) },
}
live := make([]any, 100) // keep refs so GC can't reclaim, and so we know each type
for i := range live {
live[i] = allocs[rand.Intn(len(allocs))]()
}
for pass := 0; pass < 2; pass++ {
if pass == 1 {
runtime.GC() // Go never moves objects: pass 1 is identical to pass 0
}
objs := make([]object, len(live))
for i, o := range live {
t := reflect.TypeOf(o).Elem()
objs[i] = object{reflect.ValueOf(o).Pointer(), int(t.Size()), t.Name()[0]}
}
slices.SortFunc(objs, func(a, b object) int { return cmp.Compare(a.addr, b.addr) })
fmt.Printf("\n=== pass %d (base 0x%x) ===\n", pass, objs[0].addr)
draw(objs)
}
}
func draw(objs []object) {
const cell, width = 32, 60
last := objs[len(objs)-1]
base := objs[0].addr
grid := make([]byte, int((last.addr+uintptr(last.size)-base)/cell))
for i := range grid {
grid[i] = '.'
}
for _, o := range objs {
c0 := int((o.addr - base) / cell)
grid[c0] = o.name
for k := 1; k < o.size/cell; k++ {
grid[c0+k] = '-'
}
}
prev := -1
for off := 0; off < len(grid); off += width {
row := grid[off:min(off+width, len(grid))]
if len(bytes.Trim(row, ".")) == 0 { // row holds none of our objects
continue
}
if prev >= 0 && off != prev+width {
fmt.Println(" ...")
}
fmt.Printf("0x%09x %s\n", base+uintptr(off)*cell, row)
prev = off
}
}
heapwalk.go
Run it and you’ll see something like this.
$ go run heapwalk.go
=== pass 0 (base 0xba4841580c0) ===
0xba4841580c0 M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xba484158840 M-M-M-M-....................................................
...
0xba48415bcc0 ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
...
0xba4841add40 ..........................L---L---L---L---L---L---L---L---L-
0xba4841ae4c0 --L---L---L---L---L---L---L---L---L---L---L---L---L---L---L-
0xba4841aec40 --L---L---L---L---L---L---L---L---L---L---
=== pass 1 (base 0xba4841580c0) ===
0xba4841580c0 M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-M-
0xba484158840 M-M-M-M-....................................................
...
0xba48415bcc0 ............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
...
0xba4841add40 ..........................L---L---L---L---L---L---L---L---L-
0xba4841ae4c0 --L---L---L---L---L---L---L---L---L---L---L---L---L---L---L-
0xba4841aec40 --L---L---L---L---L---L---L---L---L---L---
So even though we allocated randomly among objects of different sizes, we can observe the Go runtime fitting objects of each size next to each other.
And even after we ran the garbage collector it didn’t move anything around.
Now let’s take a look at C#.
using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;
var allocs = new Func<object>[] { () => new Small(), () => new Medium(), () => new Large() };
var live = new object[100]; // keep refs so GC can't reclaim, and so we know each type
var rnd = new Random();
for (int i = 0; i < live.Length; i++) live[i] = allocs[rnd.Next(allocs.Length)]();
// A reference on 64-bit .NET is a plain 8-byte pointer, so reinterpreting one
// with Unsafe.As gives the object's address. Object sizes are measured from
// the heap: allocate several, the smallest gap between consecutive addresses
// is the (aligned) object size, header included.
var size = new Dictionary<Type, int>();
foreach (var make in allocs)
{
var keep = new object[16];
var a = new nint[keep.Length];
for (int i = 0; i < keep.Length; i++) keep[i] = make();
for (int i = 0; i < keep.Length; i++) a[i] = Unsafe.As<object, nint>(ref keep[i]);
Array.Sort(a);
nint best = nint.MaxValue;
for (int i = 1; i < a.Length; i++)
if (a[i] - a[i - 1] > 0 && a[i] - a[i - 1] < best) best = a[i] - a[i - 1];
size[keep[0].GetType()] = (int)best;
}
for (int pass = 0; pass < 2; pass++)
{
if (pass == 1) GC.Collect();
// An address is only valid until the next collection, so pause the GC
// while we take them.
var addrs = new nint[live.Length];
GC.TryStartNoGCRegion(1 << 20);
for (int i = 0; i < live.Length; i++) addrs[i] = Unsafe.As<object, nint>(ref live[i]);
GC.EndNoGCRegion();
var objs = new (nint Addr, int Size, char Name)[live.Length];
for (int i = 0; i < live.Length; i++)
objs[i] = (addrs[i], size[live[i].GetType()], live[i].GetType().Name[0]);
Array.Sort(objs, (x, y) => x.Addr.CompareTo(y.Addr));
Console.WriteLine($"\n=== pass {pass} (base 0x{(long)objs[0].Addr:x}) ===");
Draw(objs);
}
static void Draw((nint Addr, int Size, char Name)[] objs)
{
const int cell = 32, width = 60; // cell = the smallest object's size
var last = objs[^1];
nint b = objs[0].Addr;
var grid = new char[(last.Addr + last.Size - b) / cell];
Array.Fill(grid, '.');
foreach (var o in objs)
{
int c0 = (int)((o.Addr - b) / cell);
grid[c0] = o.Name;
for (int k = 1; k < o.Size / cell; k++) grid[c0 + k] = '-';
}
int prev = -1;
for (int off = 0; off < grid.Length; off += width)
{
var row = new string(grid, off, Math.Min(width, grid.Length - off));
if (row.Trim('.').Length == 0) continue; // row holds none of our objects
if (prev >= 0 && off != prev + width) Console.WriteLine(" ...");
Console.WriteLine($"0x{(long)b + (long)off * cell:x9} {row}");
prev = off;
}
}
class Small { public long a, b; }
class Medium { public long a, b, c, d, e, f; }
class Large { public long a, b, c, d, e, f, g, h, i, j, k, l, m, n; }
HeapWalk.cs
Build and run it.
$ dotnet run HeapWalk.cs
=== pass 0 (base 0x7aea1080a1e0) ===
0x7aea1080a1e0 L---L---L---M-M-SL---L---SM-M-M-M-SL---L---L---L---M-L---SSL
0x7aea1080a960 ---L---M-M-L---M-SL---L---SM-L---M-L---L---L---M-L---L---M-L
0x7aea1080b0e0 ---SL---SSL---L---M-L---L---M-L---L---L---SM-SL---L---SL---L
0x7aea1080b860 ---L---SSSL---L---M-SL---SM-L---SL---M-L---M-L---M-L---SSL--
0x7aea1080bfe0 -L---SSSM-SM-L---M-M-L---M-L---
=== pass 1 (base 0x7aea1080a1e0) ===
0x7aea1080a1e0 L---L---L---M-M-SL---L---SM-M-M-M-SL---L---L---L---M-L---SSL
0x7aea1080a960 ---L---M-M-L---M-SL---L---SM-L---M-L---L---L---M-L---L---M-L
0x7aea1080b0e0 ---SL---SSL---L---M-L---L---M-L---L---L---SM-SL---L---SL---L
0x7aea1080b860 ---L---SSSL---L---M-SL---SM-L---SL---M-L---M-L---M-L---SSL--
0x7aea1080bfe0 -L---SSSM-SM-L---M-M-L---M-L---
And we immediately notice that objects of the same size are not grouped together. (Later on, in a different workload, we'll also notice C# moving objects around in memory.)
The documentation for Go and C# will tell you as much about the behavior of both, but I think it’s nice to also see it demonstrated like this.
Now that we’ve seen how Go memory is allocated, let’s look at how it’s cleaned up.
Mark and sweep#
The garbage collector starts at specific roots (e.g. globals and locals) and, historically in Go, follows each pointer until the GC visits all accessible objects. The mark phase. Then, in a second pass, the GC frees any allocated objects that have not been visited. Since these now-freed objects were not accessible from the root tree in the mark phase, they are by definition dead. The sweep phase.
A challenge arises when you have objects A that point to objects B/C/D of different sizes. Objects of different sizes are allocated in different sections of memory in Go. Or even if you have objects A that point to other objects A that were created at very different times; they’re going to exist in very different parts of memory. In both cases, the GC following pointers now introduces random memory access which is measurably less cache friendly.
In Green Tea, Go now scans a memory span for objects and pointers and queues up future spans for scanning based on pointers it has found, rather than following every pointer roughly as it sees one. And while we can’t show this random access behavior happening without applying patches to Go itself (so that we could observe the mark path as it visits each object), we can observe it happening with perf showing both fewer cache misses (per kilo instruction) and faster overall program runs.
Here’s the pseudocode for our workload.
struct Node {a,b,c,d *Node}
mode = packed | scattered
nodes = new [2_000_000]*Node
for i in 0..nodes.len:
nodes[i] = Node{
a: nodes[(mode == packed ? i + 1 : rand()) % nodes.len],
b: nodes[(mode == packed ? i + 2 : rand()) % nodes.len],
c: nodes[(mode == packed ? i + 3 : rand()) % nodes.len],
d: nodes[(mode == packed ? i + 4 : rand()) % nodes.len]
}
for i in 0..100:
trigger_gc()
keepalive(nodes) # prevent `nodes` from being garbage collected
In order to make the program measurement a little fairer (the scattered version has to do significant work generating random numbers) we’ll separate out the generation of node index offsets:
import array
import random
import sys
n = 2_000_000
order = sys.argv[1] if len(sys.argv) > 1 else ""
if order == "packed":
a = array.array("I", ((i + k) % n for i in range(n) for k in (1, 2, 3, 4)))
elif order == "scattered":
r = random.Random(1)
a = array.array("I", (r.randrange(n) for _ in range(n * 4)))
else:
sys.exit("usage: gen.py packed|scattered")
assert a.itemsize == 4 and sys.byteorder == "little" # matches Go's uint32 cast
with open(order+".idx", "wb") as f:
a.tofile(f)
generate_indexes.py
And the Go workload becomes:
package main
import (
"io"
"os"
"runtime"
"unsafe"
)
type Node struct {
a, b, c, d *Node
}
func main() {
n := 2_000_000
raw, err := io.ReadAll(os.Stdin)
if err != nil {
panic(err)
}
idx := unsafe.Slice((*uint32)(unsafe.Pointer(&raw[0])), n*4)
nodes := make([]*Node, n)
for i := range nodes {
nodes[i] = &Node{}
}
for i, nd := range nodes {
nd.a = nodes[idx[i*4]]
nd.b = nodes[idx[i*4+1]]
nd.c = nodes[idx[i*4+2]]
nd.d = nodes[idx[i*4+3]]
}
for i := 0; i < 100; i++ {
runtime.GC()
}
runtime.KeepAlive(nodes) // keep `nodes` from seeming to fall out of scope
}
readorder.go
Now generate the index files from the Python script. Then build two versions of the Go workload: one with Green Tea and one without.
python3 generate_indexes.py scattered
python3 generate_indexes.py packed
go build -o readorder_greentea readorder.go
GOEXPERIMENT=nogreenteagc go build -o readorder_oldgc readorder.go
Let’s time the two garbage collectors and the two workloads with perf while collecting information on cache misses.
$ for bin in readorder_oldgc readorder_greentea; do
for input in packed.idx scattered.idx; do
echo "=== $bin < $input ==="
perf stat -e cache-references,cache-misses -r 5 \
sh -c "exec ./$bin < $input" > /dev/null
done
done
=== readorder_oldgc < packed.idx ===
Performance counter stats for 'sh -c exec ./readorder_oldgc < packed.idx' (5 runs):
1,130,709,755 cache-references ( +- 0.70% )
290,434,782 cache-misses # 25.69% of all cache refs ( +- 1.02% )
4.230 +- 0.145 seconds time elapsed ( +- 3.44% )
=== readorder_oldgc < scattered.idx ===
Performance counter stats for 'sh -c exec ./readorder_oldgc < scattered.idx' (5 runs):
13,247,268,612 cache-references ( +- 0.38% )
2,325,799,796 cache-misses # 17.56% of all cache refs ( +- 0.15% )
11.052 +- 0.154 seconds time elapsed ( +- 1.39% )
=== readorder_greentea < packed.idx ===
Performance counter stats for 'sh -c exec ./readorder_greentea < packed.idx' (5 runs):
481,414,281 cache-references ( +- 0.27% )
257,894,055 cache-misses # 53.57% of all cache refs ( +- 0.04% )
2.69560 +- 0.00385 seconds time elapsed ( +- 0.14% )
=== readorder_greentea < scattered.idx ===
Performance counter stats for 'sh -c exec ./readorder_greentea < scattered.idx' (5 runs):
3,398,491,016 cache-references ( +- 1.02% )
2,195,902,796 cache-misses # 64.61% of all cache refs ( +- 0.10% )
6.9610 +- 0.0108 seconds time elapsed ( +- 0.16% )
We see very clear improvement for each workload with the new GC. But cache misses seem to have increased with the new GC? That’s not what we expected.
There are two things at play here. First, cache-references and cache-misses in perf usually correspond to L3 cache. So while the percentage of L3 cache misses in the new GC might have gone up, it doesn’t really tell us anything about behavior at the L1 or L2 cache level. And again, we saw the speedup ourselves. So we’re missing something.
Second, the program runtime largely changed between the old GC and the new GC and we haven’t normalized that. perf has an instructions metric we can key on to calculate a standard normalized metric: Misses Per Kilo Instructions (MPKI).
So let’s run perf again and ask for instructions too and calculate MPKI ourselves.
import json, subprocess
for binary in ["readorder_oldgc", "readorder_greentea"]:
for inp in ["packed.idx", "scattered.idx"]:
out = subprocess.run(
["perf", "stat", "-j", "-e", "instructions,cache-misses", "-r", "5",
"sh", "-c", f"exec ./{binary} < {inp}"],
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True).stderr
c = {}
for line in out.splitlines():
line = line.strip().lstrip("[").rstrip("],")
if line.startswith("{"):
r = json.loads(line)
c[r["event"].split(":")[0]] = float(r["counter-value"])
print(f"{binary:<20} {inp:<14} {1000 * c['cache-misses'] / c['instructions']:6.2f} MPKI")
perf.py
Run it.
$ python3 perf.py
readorder_oldgc packed.idx 1.59 MPKI
readorder_oldgc scattered.idx 12.70 MPKI
readorder_greentea packed.idx 1.81 MPKI
readorder_greentea scattered.idx 15.39 MPKI
And we’re still not seeing what we expect! L3 MPKI actually went up with the new GC. But we saw the performance improvements ourselves!
At this point if you want to keep hunting for the cache improvements to show up you’re going to need to switch to a bare metal x86/amd64 Linux machine because most virtual machines do not expose PMU counters needed for L1 events.
I grabbed myself a Vultr bare metal machine and kept going. We will ask perf for L1-dcache-loads and L1-dcache-load-misses to get L1 metrics. Then we’ll calculate MPKI for both the L1 cache and the L3 cache.
import json,
Read the original source
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.