📝 Originally published (in Japanese) at forge.workstyle.tech.
When developing a voice conversion app, you often find that the quality of the conversion depends more on the choice of the target voice (anchor) than on the conversion model itself. In our voice conversion app, we used 4 to 12 seconds of recordings from voice actors or narrators as anchors. Initially, we relied on "what sounded good to the ear" to select these segments, but this approach didn’t scale.
Manually listening to and selecting good segments from dozens of recordings was not only tedious but also inconsistent, as judgments varied based on mood. Moreover, the quality of recordings was inconsistent: some had overlapping voices in interviews, excessive emotional intonation, or noise. Continuously filtering out "unsuitable anchor segments" manually was impractical.
This article documents how we replaced the "ear-based selection" with a three-stage filter system. The goal was to mechanically select segments that are "single-speaker, natural, and high-quality." As a result, the minimum PESQ of the anchor set improved from 1.24 to 2.31, and we ultimately compiled over 70 anchors.
Prerequisites: Three Criteria for Anchors
What makes a good anchor? For voice conversion, the target audio must meet the following three conditions simultaneously:
- Single-speaker — If another person’s voice is mixed within the 12-second segment, the speaker embedding becomes an average of two people, resulting in a voice that belongs to no one.
- Natural and calm pronunciation — Shouting, falsetto, singing, or excessive intonation deviate from the speaker’s "natural voice." A straightforward utterance is better as a conversion target.
- High-quality recording — If the target material contains noise or distortion, it will carry over into the conversion result.
The challenge is that these three conditions are distinct properties. Attempting to evaluate them with a single score leads to failure. Therefore, we arranged independent filters in series for each property.
Overall Flow: Applying Filters in Order of Computational Cost
We arranged the filters in order of increasing computational cost and progressively narrowed down the candidates. For each file, we first placed 48 equally spaced candidate windows and then filtered them step by step.
48 candidates (equally spaced)
→ Stage 1: Score naturalness based on prosody → Top 16 candidates
→ Stage 2: Determine single-speaker property using campplus speaker embedding
→ Quality gate: Assess cleanliness using SQUIM PESQ
→ Adopt the most natural window among those meeting single-speaker and quality criteria
Enter fullscreen mode Exit fullscreen mode
Since pyin and RMS analysis are lightweight, we applied them to all candidates. The more resource-intensive speaker embedding and SQUIM were only applied to the top 16 candidates.
Stage 1: Measuring "Naturalness" Through Prosody
The first filter quantifies the "calmness" of pronunciation using pitch (F0) and energy (RMS). We extracted F0 using pyin from librosa and calculated several raw metrics:
- Micro-stability (instab): Median semitone fluctuation between adjacent frames. Smaller values indicate greater stability.
- Intonation range (spread): Difference between p90 and p10 of F0 in semitones. Wider ranges indicate excessive emotion; too narrow suggests monotone speech.
- Voiced certainty (conf): Clarity of periodicity.
- Energy variation coefficient (ecv): Standard deviation of RMS divided by its mean. Higher values indicate greater volume variation, implying emotional speech.
- Longest hold (hold): Maximum duration where F0 remains within ±0.5 semitones. Detects sustained vowels, singing, or elongation.
Interestingly, we evaluated intonation not by "smaller is better" but by deviation from a target value. Both monotone and overly emotional speech are undesirable, so we set a target intonation range (SPREAD_TARGET = 6.0 semitones) based on natural conversation. Deviations from this target were penalized.
def _composite(c: dict, modal_f0: float) -> float:
# Deviation of F0 from the speaker's typical range (detects falsetto/shouting/character voices)
modal_dev = abs(np.log2(max(c["f0"], 1e-6) / max(modal_f0, 1e-6))) * 12.0
pen = (W_SPREAD * abs(c["spread"] - SPREAD_TARGET) # Penalize deviation from target intonation
+ W_ECV * c["ecv"] # Volume variation
+ W_HOLD * max(0.0, c["hold"] - HOLD_FREE_S) # Hold longer than 0.5s
+ W_MODAL * modal_dev) # Deviation from typical F0 range
base = c["conf"] / (1.0 + c["instab"])
return base / (1.0 + pen)
Enter fullscreen mode Exit fullscreen mode
modal_f0 is the median F0 of all candidates in the file, representing the speaker’s typical vocal range. Windows that deviate significantly (falsetto, shouting, character voices) are penalized, prioritizing segments closer to the speaker’s natural voice. Holds are allowed up to 0.5 seconds for natural vowel elongation, with stronger penalties (weight 1.2) beyond that to exclude singing or exaggerated elongation.
Candidates are ranked by this score, and only the top 16 proceed to the next stage.
Stage 2: Verifying "Single-Speaker Property" Using Speaker Embedding
This stage addresses dialogue and interview recordings. Using the same campplus speaker embedding (192 dimensions) as Seed-VC’s anchor embedding, we determine if a 12-second segment contains only one speaker’s voice.
The method is a simple two-speaker test. The 12-second segment is divided into 3-second chunks with 1.5-second overlap. Speaker embeddings are extracted for each chunk. The least similar chunk pair is used as seeds to assign all chunks to two clusters, and the cosine similarity between the two cluster centroids is returned.
E = E / (np.linalg.norm(E, axis=1, keepdims=True) + 1e-9)
S = E @ E.T
# Use the least similar chunk pair as seeds for two clusters
a, b = np.unravel_index(int(np.argmin(S)), S.shape)
assign = (S[a] < S[b]).astype(int) # 0=closer to seed a, 1=closer to seed b
if assign.min() == assign.max(): # Biased assignment (effectively one cluster)
return 1.0
c0 = E[assign == 0].mean(axis=0); c1 = E[assign == 1].mean(axis=0)
# ...L2 normalization...
return float(np.dot(c0, c1)) # Cosine similarity between centroids (higher=single speaker)
Enter fullscreen mode Exit fullscreen mode
If the segment is single-speaker, chunk differences are minimal (e.g., due to phonemes), resulting in high cosine similarity. If speakers alternate, centroids diverge, lowering the value. Candidates with similarity below homo-thresh = 0.55 are not considered single-speaker.
Crucially, the embedding used for selection matches the one used for anchor definition. This ensures consistency between selection criteria and anchor evaluation.
Quality Gate: Ensuring Cleanliness with SQUIM PESQ
The final stage is a quality gate. Using Torchaudio’s SQUIM_OBJECTIVE, which estimates PESQ/STOI/SI-SDR without a reference, we score the cleanliness of the segment. The key advantage is that it works without a reference, allowing quality assessment even for "wild" recordings without original audio.
def _pesq(squim, y16, center_s):
seg = y16[i0:i0 + int(EXT_S * SR16)] # 12s segment
with torch.inference_mode():
_stoi, pesq, _sisdr = squim(torch.tensor(seg).unsqueeze(0).float())
return float(pesq.item())
Enter fullscreen mode Exit fullscreen mode
The threshold is pesq-thresh = 2.3. The final adoption logic selects the most natural window (highest Stage 1 score) among candidates meeting both single-speaker (homog ≥ 0.55) and quality (PESQ ≥ 2.3) criteria. If no candidate meets both, we prioritize single-speaker and compromise on quality incrementally. The hard constraint is "single-speaker," with priority given to "naturalness."
Results: Raising the Minimum Quality
We determined the thresholds by first scoring all anchors using audit_anchor_quality.py and examining the distribution. The audit ranks anchors by quality, making it easy to see how many anchors remain at each threshold.
Before introducing the quality gate, the minimum PESQ of the anchor set was around 1.24, with noisy or distorted segments included. After applying the gate, the distribution of 72 anchors (anchor_quality.json) was as follows:
- Minimum PESQ: 2.31
- p25: 2.72 / Median: 3.08 / Maximum: 4.12
The minimum PESQ increased from 1.24 to 2.31, a tangible improvement. Raising the quality floor of the "worst anchor" enhances the overall app experience. A median PESQ above 3 also indicates consistent material selection.
Pitfalls and Lessons Learned
- Avoid combining properties into a single score. Single-speaker, naturalness, and audio quality are uncorrelated. Weighted sums favor "mediocre in everything" segments. Independent filters in series, with hard constraints (single-speaker) and priorities (naturalness), worked as intended.
- "Smaller isn’t always better." Intonation should not be minimized; monotone speech is also undesirable. Evaluating deviation from a target value ensured natural conversation was selected.
- Use the same metric for selection and anchor definition. Using campplus for single-speaker verification aligned with anchor embedding, preventing inconsistencies.
- Set thresholds based on audits. We determined the 2.3 threshold by examining the distribution, not by guesswork. Deciding thresholds without data analysis is akin to relying on "ear-based selection."
Summary
- Replaced "ear-based selection" with a three-stage quantitative filter:
- Stage 1 (prosody): Score naturalness.
- Stage 2 (campplus embedding): Verify single-speaker using two-speaker clustering.
- Final stage: Quality gate with SQUIM PESQ.
- Intonation evaluated by deviation from a 6-semitone target, not minimization. Falsetto/shouting detected by deviation from typical F0 range.
- Quality gate (PESQ ≥ 2.3) raised minimum PESQ from 1.24 to 2.31. Median: 3.08, with over 70 final anchors.
- Thresholds determined via audit scripts. Single-speaker verification used the same model as anchor embedding for consistency.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.