How a single contiguous allocation — and a type system that won't let you feed strings to a scaler — is the real reason datarust fits in 2.3 megabytes.

In the last post I showed you the whole datarust workflow: impute, scale, one-hot, train a logistic regression, evaluate, and save it as JSON — all without a Python runtime in sight. The Docker image shrank from ~900 MB to ~8 MB, and the binary was 2.3 MB.

But I skimmed over something important. I kept saying "the flat memory layout" as if it were a detail. It isn't. It's the whole bet.

Every scaler, every encoder, every model, every metric in datarust runs on top of one data structure. If you understand that structure — why it looks the way it does and what it refuses to let you do — the rest of the library stops being magic.

So let's zoom in. Meet Matrix.

Two containers, on purpose

Real data is mixed. Numbers in one column, strings in the next. In Python, everything flows through one giant numpy.ndarray or a pandas.DataFrame, and the type system just... shrugs. A string column next to a float column gets coerced into object dtype. You'll find out at training time, in the form of an error message three frames deep.

datarust does the opposite. It splits your data into two types at the source:

use datarust::Matrix;
use datarust::matrix::StrMatrix;

let numeric = Matrix::new(vec![
    vec![3.0,  85.0, 24.0],
    vec![12.0, 70.0, 31.0],
    vec![f64::NAN, 95.0, 45.0],
])?;

let categorical = StrMatrix::from_strings(vec![
    vec!["MonthToMonth"],
    vec!["OneYear"],
    vec!["MonthToMonth"],
])?;

Enter fullscreen mode Exit fullscreen mode

Matrix is f64 only. StrMatrix is strings only. They are different types, and the compiler will refuse to compile a program that hands a string column to a scaler. Not at runtime — at compile time. In the last post I called this "putting on glasses for the first time." Let me show you what it actually buys you.

The ColumnTransformer API is built on that split:

ct.add_numeric("scaled", vec![0, 1],
    TransformerKind::StandardScaler(StandardScaler::new()));          // OK
ct.add_categorical("encoded", vec![0],
    CategoricalTransformerKind::OneHotEncoder(OneHotEncoder::new())); // OK
// ct.add_numeric("nope", vec![0],
//     CategoricalTransformerKind::OneHotEncoder(...));               // does not compile

Enter fullscreen mode Exit fullscreen mode

A OneHotEncoder expects strings. The type system makes that a compile error on a numeric column. sklearn can't do this — every column name is just a string, and OneHotEncoder will happily run on floats if you don't read the docs carefully.

Construction is a lie detector

Here's the thing that surprised me most when I actually read the source: Matrix::new doesn't just store your data. It validates it.

let bad = Matrix::new(vec![
    vec![1.0, 2.0, 3.0],
    vec![4.0, 5.0],
]);

Enter fullscreen mode Exit fullscreen mode

This returns an error:

ShapeMismatch { expected: "3 columns", actual: "2 columns at row 1" }

Enter fullscreen mode Exit fullscreen mode

Not a panic. Not NaN silently appearing in column 3. Not a jagged array that flows downstream and poisons your model. A precise, recoverable error, at the moment of construction.

In pandas, a ragged column silently becomes object dtype. In numpy, you get a cryptic error deep in some array-conversion path. In datarust, the constructor is the bouncer, and it's the only place the check needs to happen. From then on, every function that takes a &Matrix can trust the shape without re-checking.

There's a matching from_flat for when you already have a contiguous buffer, and it validates the element count matches the declared shape:

let m = Matrix::from_flat(2, 3, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?;
assert_eq!(m.nrows(), 2);
assert_eq!(m.ncols(), 3);
assert_eq!(m.get(1, 2), 6.0);

Enter fullscreen mode Exit fullscreen mode

Even the allocation is checked: rows.checked_mul(cols) — so you can't ask for a matrix so large that rows * cols overflows usize and wraps around into a buffer too small. I know that sounds paranoid. It's also the difference between a library that panics in production and one that returns Err.

The one allocation

Now for the part I waved my hands at last time. Here's the internal layout:

pub struct Matrix {
    data: Vec<f64>,   // one contiguous buffer, row-major
    rows: usize,
    cols: usize,
}

Enter fullscreen mode Exit fullscreen mode

That's it. Not Vec<Vec<f64>> — no allocation per row, no pointers-to-pointers. Element (i, j) lives at data[i * cols + j]. Your 50,000×200 matrix is one contiguous Vec<f64> of ten million floats, in one heap allocation.

Why does that matter? Three reasons:

  1. Cache locality. When a scaler walks a row, every element is adjacent in memory. With Vec<Vec<f64>>, each row is a separate allocation, and walking rows means jumping between unrelated memory pages.
  2. Auto-vectorization. as_slice() hands the compiler a plain &[f64]. Modern CPUs get to use SIMD, and LLVM gets to prove the loop has no aliasing.
  3. The Python boundary doesn't exist. This is the quiet killer. In a numpy pipeline, every scaler.transform() crosses from Python into C and back, materializing Python float objects for the results. In datarust, transform is a Rust function operating on a Rust buffer. The whole pipeline never leaves native code.

This wasn't always the case. Pre-0.3, Matrix was Vec<Vec<f64>>. When we switched to the flat layout, the same workloads got dramatically faster with no algorithm changes:

Workload (50,000 × 200) Vec<Vec<f64>> (v0.2) flat (v0.3, default)
StandardScaler 115 ms 8.4 ms
MinMaxScaler 81 ms 12.2 ms
RobustScaler 459 ms 137 ms
Pipeline (3 scalers) 662 ms 152 ms
PCA 1056 ms 1008 ms
OneHotEncoder 88 ms 98 ms

RobustScaler got 3.4× faster, Pipeline 4.4×. Same math, same data — only the memory layout changed. Flat beats fancy.

The hot loops are designed around that layout. You almost never call get(i, j) in an inner loop; you grab a row slice and iterate:

for row in m.iter_rows() {      // yields &[f64], zero-copy
    for &v in row {
        // ...
    }
}

Enter fullscreen mode Exit fullscreen mode

row(i) returns a &[f64] slice, and as_slice() exposes the whole buffer for the tightest loops. get exists for the times you need it — and it's bounds-checked in release mode, with a precise panic message. If you want to be defensive, checked_get returns Option<f64> instead.

NaN is a type problem, so we treat it like one

Python has a lovely tradition of letting NaN flow through a pipeline until it silently infects your coefficients. You train, your model returns nan, and you spend an afternoon bisecting which column did it.

datarust's answer is three validators, each with a distinct contract:

m.validate_no_nan()?;       // rejects NaN, allows ±inf — for data that must be complete
m.validate_no_infinite()?;  // rejects ±inf, allows NaN — for imputers
m.validate_finite()?;       // rejects both — for models and metrics

Enter fullscreen mode Exit fullscreen mode

The subtle one is the middle: validate_no_infinite. Imputers like SimpleImputer and KnnImputer are allowed to see NaN — that's the missing-value marker they exist to fix. But an infinity is not a missing value, it's a broken number. So imputers call validate_no_infinite, not validate_no_nan. The distinction is baked into the API, and it means the error message tells you which kind of dirty data you're dealing with.

When the validation fails, the error tells you exactly where:

InvalidInput("NaN value at position (2, 0)")

Enter fullscreen mode Exit fullscreen mode

Row 2, column 0. Not "somewhere." You go straight to the bad cell.

And this is exactly why the preprocessing order matters. The Matrix in the previous post had a missing tenure value in row 2 — a NaN that a scaler would refuse to touch. You can't skip the imputer and hope the scaler tolerates it:

use datarust::imputer::{ImputeStrategy, SimpleImputer};

let imputed = SimpleImputer::new(ImputeStrategy::Mean).fit_transform(&numeric)?;

let table = Table::new(imputed, categorical)?;
let mut ct = ColumnTransformer::new()
    .remainder(Remainder::Drop)
    .add_numeric("scaled", vec![0, 1],
        TransformerKind::StandardScaler(StandardScaler::new()))
    .add_categorical("encoded", vec![0],
        CategoricalTransformerKind::OneHotEncoder(
            OneHotEncoder::new().handle_unknown(HandleUnknown::Ignore),
        ));

let x = ct.fit_transform(&table)?;  // imputed → scaled → encoded, one step

Enter fullscreen mode Exit fullscreen mode

Without the imputer, that last line fails loudly — invalid input: non-finite value NaN at position (2, 0) — instead of silently training a model on broken numbers. The validation isn't there to annoy you. It's there to force the pipeline to be honest about what it handles.

The sparse sibling

One-hot encoding a high-cardinality column produces a lot of zeros. Storing all of them as f64 is waste. So there's a third container: SparseMatrix, a Compressed Sparse Row matrix mirroring scipy.sparse.csr_matrix.

use datarust::matrix::SparseMatrix;

let sp = SparseMatrix::from_triplets(
    3, 4,
    &[(0, 0, 1.0), (1, 2, 3.0), (2, 3, 5.0), (0, 0, 2.0)],
)?;

Enter fullscreen mode Exit fullscreen mode

Notice something: I passed (0, 0) twice. datarust sums duplicate coordinates (1.0 + 2.0 = 3.0), drops entries that sum to zero, and sorts each row by column index — so the invariants a reader expects are always true. And if you hand it a malformed CSR array, it rejects it with a specific message instead of indexing out of bounds later. The constructor validates indptr, column ranges, and per-row ordering. Bad data is rejected here, not at the model.

What it doesn't do (yet)

Honesty section, because the tone of these posts is "here's what I learned," not "buy my perfect library."

  • StrMatrix is not flat. It's still Vec<Vec<String>>, one heap allocation per string. It's fine for categorical columns (which are wide and short), but it's a deliberate asymmetry, and it shows: string-heavy workloads don't get the same cache wins as numeric ones.
  • No views. select_rows and select_columns copy. In numpy you'd get a view with fancy indexing; here, correctness and simplicity win over aliasing.
  • No f32. It's f64 everywhere. An f32 mode would halve memory for big datasets, but it's a sweeping refactor across every estimator. It's on the roadmap, not in the crate.
  • No NumPy interop. The serde feature preserves a nested-JSON wire format for Matrix, so fitted pipelines round-trip through save_json/load_json — but there's no .npy reader yet. That's explicitly listed as under consideration.

Why this is the core feature

Every number in every example in the first article passed through Matrix. The 179–620× speedups against sklearn's ColumnTransformer? Partly the flat buffer, partly never crossing the Python/C boundary. The type-safety that caught a OneHotEncoder on a numeric column before it could run? That's Matrix vs StrMatrix being different types. The JSON model that loads in a WASM module or an ARM embedded target? That's a buffer you can serialize and a shape you can trust.

In other words: the 400× Docker shrink isn't the story. The one Vec<f64> that makes it possible — and the constructor that refuses to let you put garbage in it — is the story.

If you want to see the data structure that everything else is built on, the full source is in crates/datarust/src/matrix.rs — 1,400 lines including tests, and the tests are where the invariants are documented. Or just run the pipeline from the last post and know that every number in it started as a flat buffer that checked its own shape.

datarust is MIT-licensed and available on crates.io, with docs at datarust.dev. The roadmap — including the f32 refactor and the .npy reader — lives at github.com/genc-murat/datarust. If you want to help make Matrix generic over float types, now's a good time.