Since my beloved dearVR was end-of-lifed, I built this in collaboration with GPT-5.6-Sol, with Opus 5 writing the main draft of this article.

I built NekoSpace Binaural, a spatialization VST plugin designed for audio dramas and ASMR, and recently released the first alpha.

This article isn't a celebratory "look what I built" post. Instead, its purpose is to document what worked and what didn't, backed by real numbers.

In particular, my four attempts at handling "elevation" (vertical localization) failed to yield a decisive result. I believe documenting that failure is where the real value lies, so I’ve dedicated most of this post to it.

Why Build It?

Normally, I record using a Neumann KU100 dummy head microphone. While the KU100 captures the actual acoustic space beautifully, there are always cases where you need to place a mono source at an arbitrary position during post-production—such as sound effects, added dialogue, or whisperings right next to the listener's ear. Existing plugins are mostly built for games/VR, and they lack convincing proximity expressions for placing a voice "right by the ear." That was the starting point.

Note: Naturally, you should never run materials already recorded with a KU100 through this plugin, as that would apply head-related transfer functions twice. The target input is strictly dry mono/stereo material.

What Worked

1. Computing ITD from Exact Ray Paths on a Rigid Sphere

Instead of deriving Interaural Time Difference (ITD) from the measurement delays embedded in HRIRs, I designed the system to calculate ITD geometrically on every frame.

Since the ears lie on the surface of a sphere, there are two ray paths from the sound source to the ear: a straight line when the ear is visible, and a tangent + a great-circle arc when the ear falls into the acoustic shadow.

// The ear is "on" the sphere. Line-of-sight exists when the sound source 
// is outside the ear's tangent plane: r * cos(theta) >= a <=> theta <= acos(a/r)
const float thetaVis = std::acos (clampf (a / r, 0.0f, 1.0f));

if (theta <= thetaVis)
    return (sourcePos - earPos).length();           // Straight line

const float tangent = std::sqrt (std::max (r * r - a * a, 0.0f));
return tangent + a * (theta - thetaVis);           // Tangent + arc

Enter fullscreen mode Exit fullscreen mode

The main benefit of this approach is that proximity exaggeration emerges naturally from the geometry, without artificial tuning. At a distance of 12 cm directly to the side, the Interaural Level Difference (ILD) widens up to 24.2 dB. In contrast, standard far-field approximations (like Woodworth's formula) yield only 5.6 dB. The plugin’s NEAR FIELD knob continuously blends between these two models: 0% acts as a standard panner, while 100% applies full per-ear geometry.

A Bug I Ran Into

Initially, I wrote the visibility boundary condition as π/2 + acos(a/r). A code review pointed out the mistake.

The correct boundary is simply acos(a/r). At this boundary, |S - E| == sqrt(r² - a²) (matching the tangent length), making the two conditional branches connect continuously. With the incorrect boundary, the transition into the acoustic shadow occurred at the wrong angle, causing the ITD to jump discontinuously near 90° azimuth. This manifested as audible clicks/pops when panning.

Lesson learned: Whenever geometry can be written in closed form, always write unit tests to verify boundary continuity.

2. Late-Only Voice Ducking

A common requirement in audio drama production is: "I want to add reverb, but I can't let it mud up the vocal clarity." The typical workaround is manually writing reverb send automations, which is tedious and often sounds unnatural.

I implemented a ducking feature that attenuates only the late reverberation while the voice is actively playing. Three specific design decisions made this effective:

(a) Duck the FDN output, NOT the input.

This is key. If you attenuate at the input, the reverb energy never builds up inside the network, causing the reverb to "start growing" the moment the voice stops. By suppressing at the output, the reverb energy accumulated inside naturally emerges at the end of a phrase. The room reflection rises exactly at the end of the spoken line as intended.

(b) Do not touch direct sound or early reflections.

Early reflections carry the spatial "shape" of the room. Furthermore, as described later, they are rendered through HRTFs for each image direction, serving as part of the height cues. Only the late reverb muddies the voice, so that's all we duck.

(c) Make it threshold-less.

env = x > env ? envAtt * (x - env) + env : envRel * (x - env) + env;
ref = x > ref ? x : refRel * (x - ref) + ref;   // Reference level decaying over seconds

// Relative to "recent input level", independent of input gain or speaking style
const float activity = clampf (env / (kActivityFraction * ref), 0.0f, 1.0f);
const float target   = 1.0f - depth * activity;

const float k = target < gain ? duckAtt : duckRel;  // Fast duck, slow recovery
gain += k * (target - gain);

Enter fullscreen mode Exit fullscreen mode

ref is a slowly decaying baseline level. By evaluating activity relative to this baseline, the ducking behaves consistently whether the input is a soft whisper or a loud shout. This eliminated the need for a dB threshold knob on the UI.

Measurement: Isolating the room bus (subtracting the dry rendering), the reverb level dropped to −13.1 dB during speech and returned to −0.07 dB post-speech.

(Note: Initially, I measured the total output and panicked because it only showed a −1.6 dB drop. Since the direct sound dominates the output, that was expected—I had simply measured at the wrong tap point! Always isolate the target bus when measuring DSP effects.)

What Didn't Work: Elevation

Here is the core of the problem. Across four iterations, vertical localization never reached a level where a listener could confidently say, "That sound is coming from above."

Round 1: Rebuilding the Analytical Pinna Model

I modeled pinna notch filters by sweeping the notch frequency from 4.2 kHz to 11.5 kHz based on elevation, adding companion peaks and a high-frequency shelf.

  • Spectral difference (4–14 kHz, top vs. bottom): Expanded from 2.0 dB → 5.8 dB.
  • Perceptual outcome: Tonal timber clearly changed. Vertical positioning remained completely unperceivable.

Round 2: Empirical HRTF (Neumann KU100)

Hypothesizing that the analytical model was too crude, I integrated measured KU100 HRTF data from TH Köln (Bernschütz 2013, Lebedev 2702 grid points, CC BY-SA).

Key steps during dataset conversion:

  1. Removed dataset ITD: Since the engine calculates ITD geometrically, leaving measurement delays in the HRIRs caused double delay. I minimum-phased the response via real-cepsrum unwrapping to strip group delay while preserving magnitude responses. Truncating to 256 taps remained clean because energy was concentrated at the head.
  2. Verified coordinate system & symmetry: SOFA SimpleFreeFieldHRIR uses counter-clockwise positive angles (left is positive), whereas my engine uses right-positive. I verified with tests that an az +90° source produced higher levels in the right ear.
  3. Normalized frontal level: Vital to ensure listening tests compared spectral shape rather than sheer volume differences.
  • Spectral difference (4–14 kHz): 6.4 dB (wider than Analytical Model B's 5.8 dB).
  • Perceptual outcome: Indistinguishable from Analytical B. Vertical position was still imperceptible.

Despite the dataset showing larger numerical spectral differences, perceived spatialization didn't change. This ruled out the hypothesis that "just swapping the dataset will fix it."

Round 3: Pinna-Independent Elevation Cues

If fine spectral pinna notches depend heavily on the listener's own pinna shape, I reasoned that adding pinna-independent cues might help.

(a) Rendering Early Reflections through HRTFs

Previously, early reflections used equal-power panning. This was a flaw: atan2(x, z) yields identical panning gains for sound sources above vs. below. The reflections carried zero directional information. By routing each mirror image reflection through HRTFs based on its actual arrival angle, overhead sounds finally received floor reflections coming from below.

  • Room response difference (top vs. bottom): 5.9 dB.

(b) Modeling Torso / Shoulder Reflections

I added a comb filter notch between 440 Hz–1.2 kHz that moves with elevation. Because it operates in lower frequencies, it is less affected by headphone coloration.

  • Low-frequency difference (0.7–3 kHz): Analytical A = 0.11 dB, Analytical B = 2.15 dB.

(Note: The KU100 dummy head lacks a torso, so it inherently lacks these shoulder reflection cues—likely one reason the empirical dataset didn't outperform analytical models in Round 2.)

  • Perceptual outcome: Still weak.

Round 4: Manual Per-Listener Tuning (Elevation Lab)

I abandoned expanding generic models and shifted toward a listener-tuned manual calibration system.

I created three elevation anchors (Below −60°, Level 0°, Above +60°), each holding 8 parameters: notch frequency, depth, Q, companion peak ratio, HF shelf, and torso delay/gain. Intermediate angles are interpolated; angles beyond ±60° are extrapolated.

Anchors allowed independent tuning of top and bottom responses. In Analytical B, all parameters were derived from sin/cos of elevation, meaning tweaking "Above" invariably distorted "Below".

Verified via unit test: Shifting the Above notch anchor from 9.7 kHz → 14 kHz moved the rendered top notch to 13.4 kHz while the Below notch remained strictly fixed at 4850 Hz.

However, surfacing 24 raw parameters in the UI was completely unusable. It was mathematically sound, but terrible for real-time ears-on tweaking. I collapsed them into 4 macro controls: UP, DOWN, BODY, and FOCUS, hiding raw controls inside an Advanced foldout. At macro value 1.00, it bit-accurately reproduces Analytical B, ensuring "Reset" actually means Reset.

  • Perceptual outcome: "There was a slight sense of vertical motion, but still nowhere near a convincing 'above' localization."

It was better than Round 3—there was a sense of directional movement—but calling it "above" would be dishonest.

Conclusion & Key Takeaways

Metric Analytic A Analytic B KU100 Measured
Upper Spectral Diff (4–14 kHz) 2.0 dB 5.8 dB 6.4 dB
Lower Spectral Diff (0.7–3 kHz) 0.11 dB 2.15 dB —
ILD at az +90° — — 12.7 dB

All four approaches improved the numerical metrics. None achieved convincing vertical perception.

I concluded that this is not an implementation bug, but the known ceiling of static, non-personalized binaural audio. The reasons are clear:

  1. Spectral elevation cues depend heavily on the unique physical shape of the listener's own ears.
  2. Headphones heavily disrupt/re-color that exact 5–12 kHz frequency region.
  3. The most powerful elevation cue—head tracking—is structurally impossible here, as audio dramas are distributed as pre-rendered static audio files.

Consequently, I decided to release the plugin while explicitly stating in the README that elevation cues are weak. The feature remains in the DSP pipeline, but I will not market it as a reliable localization tool. In voice productions, conveying height is far more reliably achieved through script design, footsteps, and spatial reverb—which is how traditional radio dramas have handled it for decades.

I committed all failed experiments and numerical findings directly into the public repository as elevation-findings.md. Negative results are worth preserving.

DSP & Architecture Lessons

1. Choice Lists (Enums) Must Be Frozen Before Release

Deciding this upfront saved the project.

Plugin state management (like JUCE's APVTS) saves choice parameters using denormalized indices, so appending options to the end of a list later won't break existing project files. Adding persistent string keys (e.g., analytic_b) ensures safety across display name changes or localizations.

The real hazard is host automation. VST3/AU automations save values normalized as index / (count - 1). The moment you add an option to an enum, count changes, causing all past host automations to point to the wrong option. This cannot be rescued by plugin state formats alone.

My rule: If you want to add options after release, create a new parameter ID. I explicitly marked frozen IDs in the CHANGELOG. This is why 4 HRTF profiles (including one dev-only profile) were reserved from day one.

(Self-correction note: I previously misspoke on this, asserting APVTS broke saved states. In reality, APVTS saves denormalized indices safely; it's the host automation mapping that breaks. Don't guess format behaviors—read the code!)

2. Keep the DSP Core Independent of JUCE

I enforced an architectural rule: src/dsp must never #include <JuceHeader.h>.

Because of this, 30 acceptance tests run without instantiating a plugin wrapper. Builds and test executions complete in milliseconds, and CI failures can be pinpointed instantly.

3. Don't Trust CI Until You See It Fail (and Pass)

A Git tag triggers GitHub Actions to run configure → build → test → package → publish, generating a draft release. If tests fail, the workflow halts.

However, this failed me once: I added CI but didn't monitor its real run. A runner mismatch (windows-latest dropping VS 2022 components) broke CI on every push without me noticing. Pinning to windows-2022 fixed it. CI implementation isn't done when you commit the file; it's done when you watch it turn green.

Summary

  • Geometric ITD, proximity modeling, and late-only ducking worked as intended.
  • Elevation failed perceptually across 4 iterations despite solid numerical improvements. I accepted this as a fundamental ceiling of static binaural audio and documented it in the README.
  • Freeze irreversible contracts (parameter IDs, choice lists) before v1 release.

As an alpha release, there is still room to polish the sound. It's available as Windows VST3 + Standalone under the AGPLv3 license. If you're interested, feel free to give it a spin—headphones are required!

👉 GitHub: https://github.com/moe-charm/nekospace-audio