A 3–5 month field guide for turning a browsing habit into an AI orchestra

Prologue: The Case That Started With a Skip Button

Every great investigation starts with something small and irritating. Yours starts on a Tuesday night. You're three hours deep into a YouTube rabbit hole — lo-fi beats, a Vietnamese city-pop deep cut, a live session you didn't mean to watch twice — and you think, someone should be paying attention to this.

That someone is about to be you. Not as a listener. As a detective.

Here's the case file: somewhere in your YouTube history is a pattern. A shape. You skip certain songs after four seconds. You replay others three times in a row without noticing. You always end up on synthwave at 11pm and never before 6pm. Right now, that pattern is invisible — smeared across a history page that YouTube barely lets you search, let alone reason about. Your job over the next few months is to build the instrument that makes it visible: a system that watches what you actually do, figures out what it means, and hands you back playlists that feel like they were made by someone who knows you. Because they were. That someone is a machine you built, trained on a dataset of one: you.

This won't be a weekend project. It's closer to building a small orchestra — a browser extension as your field agent, a backend as your evidence locker, a couple of ML models as your forensic lab, and a frontend as the stage where the final performance happens. Every piece depends on every other piece. That's what makes it hard, and it's also what makes it worth doing slowly, in public, with your own hands on every layer.

Let's start where every good documentary starts: by following one song from the crime scene to the credits.

Chapter 1: The Case of the Repeating Refrain

Following one song's journey

It's 9:47pm. You click a YouTube video: "Kiss of Life – Midas Touch (Live Performance)." You watch 41 seconds, skip to the chorus, watch another 90 seconds, then close the tab. Here's what needs to happen for that unremarkable moment to eventually shape a playlist called "Monday Chill":

  1. The Witness (Browser Extension). A content script sitting quietly inside the YouTube tab notices the video element load. It reads the video ID, title, channel name, and duration from the page. It watches the <video> element's timeupdate, pause, seeked, and ended events like a stenographer taking notes: play at 0:00, seek to 1:32, pause at 3:04, total watch time 41s + 90s = 131s out of 210s (62%).

  2. The Tip-Off (Event API). Every so often — batched, not on every tick, because nobody wants to be that extension hammering a server — the extension bundles these observations into a small JSON payload and POSTs it to your backend: {video_id, title, channel, watch_segments, timestamp, session_id}.

  3. The Evidence Locker (Database). Your FastAPI backend receives the tip-off, validates it, and writes a raw event row into PostgreSQL. Nothing clever happens yet — this is just chain-of-custody. You never want to lose the raw signal, because every clever thing you do later will be derived from it, and derived things need to be recomputable.

  4. The Lab Work (ML Pipelines, offline). On a schedule — say, once a night — a pipeline wakes up, looks for new raw events, and does the actual detective work:

    • Is this even a music video? (classification)
    • What song is this, really? (metadata resolution via title parsing + external lookups)
    • What does it sound/feel like? (audio embedding + mood tagging)
    • How does this fit with everything else you've watched? (behavioral signal: implicit rating from watch %, skip pattern, replay count, time of day)

  5. The Case Board (Vector store + relational tables). The song's audio embedding gets written to a vector column (via pgvector), its metadata to a normal relational table, and its behavioral score gets updated incrementally — like a detective updating a suspect's file every time new evidence comes in.

  6. The Reveal (Playlist Generator). Periodically, a clustering job looks at the whole case board — all your songs, their embeddings, their mood tags, their time-of-day patterns — and groups them. A cluster of moody, low-energy, evening-heavy tracks with "chill," "lofi," or "acoustic" tags gets surfaced and, because most of its listens happened on Mondays, gets named "Monday Chill."

  7. The Broadcast (Frontend + Player). Your React app queries the backend, gets back a list of playlists with their tracks, and renders them. When you hit play, it doesn't awkwardly embed a YouTube iframe — it either streams audio you've legitimately extracted for personal use, or falls back to a controlled YouTube player, and it logs that playback too, feeding the loop again.

That's the whole organism, end to end. Six weeks from now this will feel obvious. Right now, notice the shape: observe → transmit → store → understand → organize → present, with a feedback loop closing the circle back to step 1. Every phase of this guide builds one of these organs.

The architecture, explained like you're new here 

[Browser: YouTube tab]
        |
   content script  (watches video element)
        |
   background script (batches + sends events)
        |
        v
   Event Ingestion API  (FastAPI, /events endpoint)
        |
        v
   Raw Events Table  (PostgreSQL)
        |
   +----+----------------------------+
   |                                 |
   v                                 v
Metadata & Classification      Audio & Behavior Pipelines
(is it music? what song?)      (embeddings, mood, implicit score)
   |                                 |
   +----------------+----------------+
                    v
        Enriched Track Store
   (PostgreSQL relational + pgvector embeddings)
                    |
                    v
          Clustering / Playlist Engine
        (HDBSCAN, labeling, scheduling)
                    |
                    v
             Playlist API (FastAPI)
                    |
                    v
           React Frontend + Player
     (renders playlists, plays audio, logs playback)
                    |
                    +--> feeds back into Raw Events

Enter fullscreen mode Exit fullscreen mode

Plain English: a browser extension is the only part of this system that can see raw YouTube behavior, so it's your sensor. Everything after that is you progressively distilling raw noise into structured meaning — first "what is this," then "what does it sound like," then "what does it mean that you watched it," then "what group of songs does this belong to." A pipeline is just a name for "a sequence of these distillation steps that runs on a schedule instead of live," which matters because audio analysis and ML inference are too slow to run inline while you're browsing.

Why these particular tools (and not the flashier alternatives)

  • FastAPI, not Django or raw Flask. You'll be exposing a mix of simple CRUD endpoints and ML inference endpoints. FastAPI's automatic request validation via Pydantic means fewer "why did my extension send a malformed payload and silently corrupt my data" nights, and its async support matters once you're streaming audio or calling external APIs (metadata lookups) without blocking.

  • PostgreSQL + pgvector, not a separate vector database. You could stand up Pinecone or Weaviate. For a personal-scale project (thousands, maybe tens of thousands of tracks — not billions), that's solving a problem you don't have yet, at the cost of one more service to run, secure, and back up. pgvector lets you keep relational metadata (title, channel, mood tags) and embeddings in the same database, so a query like "find songs similar to this one, but only ones tagged 'study' and watched after 9pm" is one SQL query with a vector distance operator, not a join across two systems.

  • A CNN (or a pretrained audio model) for audio classification, not a giant end-to-end transformer. Audio classification at the "is this speech, music, or noise" and "what does this sound like" level has been solved well by relatively small convolutional architectures operating on spectrograms (think YAMNet, VGGish). You don't need GPT-scale compute to tell that a track is high-energy electronic versus a mellow acoustic set. This is a deliberate choice to keep your compute budget human-sized — you'll likely run this on a laptop or a modest cloud instance, not a GPU cluster.

  • Transformers for tagging, not for audio itself. Where transformers genuinely earn their complexity is text — parsing messy YouTube titles ("Artist - Song (Official Music Video) [4K]"), zero-shot classifying mood from lyrics or descriptions, and understanding channel/description context. Hugging Face's zero-shot classification pipelines let you throw a candidate label set ("chill," "energetic," "sad," "study," "workout") at a piece of text and get calibrated-ish scores back without training anything — perfect for a first pass, with fine-tuning as a stretch goal.

  • A browser extension, not scraping Google Takeout alone. Takeout gives you history, but it's after the fact, coarse, and doesn't include watch percentage, skips, or replays — exactly the implicit-feedback signal your recommendation logic needs. The extension is how you get behavioral data, not just occurrence data. Takeout is still useful — as bootstrap data and backfill, which Chapter 4 covers.

Notice the pattern in every choice above: pick the tool that matches the actual scale and shape of your problem, not the tool with the best conference talk. You're building a personal system for one very well-instrumented user (you), not a production service for millions. Keep saying that sentence to yourself whenever you're tempted to add Kubernetes.

The living organism

Here's the mental model to carry through the rest of this guide: this system has a nervous system (the extension + event API, always sensing), a subconscious (the nightly pipelines, working while you sleep, turning raw sensation into structured understanding), and a conscious voice (the frontend, presenting a curated, explainable result). If you break the nervous system, nothing gets sensed. If you break the subconscious, sensations pile up unprocessed. If you break the conscious voice, all that understanding has nowhere to be expressed. You'll spend the next several months building each of these organs, and — this is the important part — you'll build them so that each one can be tested and demoed on its own, before they're wired together. That's the difference between a project that has a working prototype at every milestone and a project that only "works" the week before a deadline, if you're lucky.

Chapter 2: The Learning Roadmap — Five Months, Six Phases

You know basic Python and JavaScript. You don't yet know how these pieces talk to each other, or how to think about audio and behavior as data. So the roadmap is designed to stack skills like sediment layers — each phase leaves you with a thin, ugly, but genuinely working slice of the system, and each phase's slice becomes the foundation the next phase builds on. You will never be staring at a blank canvas with six unconnected concepts in your head. You will always be extending something that already runs.

Treat each phase as 2–4 weeks depending on how much time you have per week. If you have 10+ hours/week, aim for the shorter end; if you're doing this around a full-time job, take the longer end. Five months is the "comfortable, learn deeply" pace. Three months is achievable if you already move fast and cut a few of the "stretch" items.

Phase 0: Recon (3–5 days, before Week 1 really starts)

Goal: Set up the skeleton of every future piece, even if each piece does almost nothing.

  • Create the repo structure: extension/, backend/, ml/, frontend/, infra/.
  • Get a FastAPI "hello world" running locally with one route, GET /ping.
  • Get a bare Chrome extension loaded in developer mode that does nothing but log "extension loaded" to the console on any youtube.com page.
  • Spin up a local PostgreSQL instance (Docker is your friend here — docker run postgres).

Learning pit stop:

  • 15 min: FastAPI's official "First Steps" tutorial — just enough to run a server.
  • 20 min: Chrome's Extension "Getting Started" tutorial — focus on the manifest and content scripts, skip the rest for now.

You know you're ready when: you can open a YouTube tab, see "extension loaded" in the console, and hit http://localhost:8000/ping in your browser and get {"status": "ok"}. That's it. Two independent heartbeats.

Phase 1: The Witness — Event Tracking (Weeks 1–3)

Goal: A Chrome extension that logs real YouTube watch events, and a backend endpoint that stores them.

Skills gained: DOM event listeners, Chrome extension messaging (content script <-> background script), REST API design basics, relational schema design for event logs.

What you'll build:

  • A content script that attaches listeners to the YouTube <video> element (play, pause, seeked, ended, timeupdate sampled every few seconds — not every frame, that's overkill and will flood your database).
  • A background script that batches events (e.g., flush every 30 seconds or every 10 events) and sends them via fetch to your backend.
  • A POST /events FastAPI endpoint with a Pydantic model validating the payload, writing rows into a raw_events table.
  • A minimal schema: raw_events(id, video_id, title, channel, event_type, position_seconds, video_duration_seconds, session_id, created_at).

Architecture decision point: Do you send every micro-event (every timeupdate tick) to the server, or do you compute a per-video summary (total watch time, max position reached, number of seeks) in the extension and send one summary per video? Sending raw ticks gives you more analytical flexibility later (you could reconstruct exact skip behavior) at the cost of way more data and server load. Summarizing in-extension is simpler and cheaper but throws away granularity. Recommendation for a learning project: send raw ticks for now — you're optimizing for "what can I learn from this," not production efficiency, and you can always summarize later. You can throttle it (sample every 5 seconds) to keep volume sane.

Learning pit stops:

  • 20 min video: search "Chrome extension message passing background content script" — get comfortable with chrome.runtime.sendMessage.
  • Skim MDN's HTMLMediaElement events page — you need timeupdate, pause, seeked, ended, play.
  • 10 min: Pydantic's "Models" quickstart — you'll lean on this for every endpoint from here forward.

Skeleton to work from (not a finished file):

// content_script.js -- the witness's notebook
function attachListeners(videoEl, videoId) {
  let lastPosition = 0;

  videoEl.addEventListener('timeupdate', () => {
    // TODO: throttle this -- don't fire on every tick
    // TODO: send {event_type: 'progress', position: videoEl.currentTime}
  });

  videoEl.addEventListener('seeked', () => {
    // TODO: compare videoEl.currentTime to lastPosition
    // to detect skip-forward vs rewind
  });

  videoEl.addEventListener('pause', () => {
    // TODO: send pause event with position
  });
}

// TODO: figure out how to get videoId from the page URL
// TODO: figure out how to detect when YouTube's SPA navigation
//       changes video without a full page reload (hint: MutationObserver
//       on the title element, or listen for yt-navigate-finish)

Enter fullscreen mode Exit fullscreen mode 

# backend/main.py -- the evidence intake desk
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class RawEvent(BaseModel):
    video_id: str
    title: str
    channel: str | None = None
    event_type: str  # 'play' | 'pause' | 'seeked' | 'ended' | 'progress'
    position_seconds: float
    video_duration_seconds: float | None = None
    session_id: str

@app.post("/events")
def ingest_event(event: RawEvent):
    # TODO: open a DB connection (consider SQLAlchemy or raw psycopg)
    # TODO: INSERT into raw_events
    # TODO: return a lightweight ack, don't do heavy work here --
    #       this endpoint must stay fast, it's on the hot path
    ...

Enter fullscreen mode Exit fullscreen mode

Milestone / "you know you're ready when": you watch three different YouTube videos for real, then query your database and see a believable, timestamped trail of what you actually did -- including at least one skip you remember making. Take a screenshot. This is the first time the system has seen you.

Phase 2: The Classifier -- Is This Even Music? (Weeks 4-6)

Goal: Given a raw event's title/channel/duration, decide whether it's a music video at all (versus a podcast, tutorial, vlog) -- and if it is, extract a clean artist/song guess.

Skills gained: Text classification basics, regex-as-first-resort vs ML-as-fallback thinking, working with the YouTube Data API, zero-shot classification with Hugging Face.

What you'll build:

  • A nightly (or on-demand) script that reads unprocessed rows from raw_events, groups by video_id, and for each unique video, runs a classification step.
  • First pass -- heuristics, not ML. Channel name contains "Official," "VEVO," "Music"? Title matches patterns like Artist - Title (Official ...)? These heuristics alone will correctly classify a surprising majority of real YouTube music content. Write these first. Resist the urge to reach for a model before you've squeezed out what regex and metadata can do -- this is a core "how to think like an engineer" lesson: cheap and boring beats fancy and unnecessary, every time it's available.
  • Second pass -- zero-shot text classification for the ambiguous remainder, using Hugging Face's zero-shot-classification pipeline with candidate labels like ["music video", "podcast", "tutorial", "vlog", "gaming"].
  • A tracks table that stores the resolved artist/title guess, separate from the raw event log -- this is your first "enriched" table, the first sign of the subconscious turning sensation into understanding.

Architecture decision point: Regex-first-then-ML, or ML-first-then-regex-cleanup? Going regex-first means your model only has to handle genuinely ambiguous cases, which is both faster (you skip inference on 70%+ of videos) and more accurate (you're not asking a general-purpose model to outperform a rule you could write in one line for "channel name ends in VEVO"). This is a recurring theme: use ML for the residual, not the whole problem.

Learning pit stop:

  • 15 min: Hugging Face's zero-shot classification pipeline docs -- run their example locally before writing your own.
  • Skim: the YouTube Data API "Videos: list" reference -- you'll want snippet.description and snippet.tags for extra signal, and eventually contentDetails.duration.

Skeleton:

# ml/classify.py -- the detective's first read of the case file
MUSIC_CHANNEL_HINTS = ["vevo", "official", "records", "music"]

def heuristic_is_music(title: str, channel: str) -> bool | None:
    # Return True/False if confident, None if unsure (defer to ML)
    channel_lower = channel.lower()
    if any(hint in channel_lower for hint in MUSIC_CHANNEL_HINTS):
        return True
    # TODO: add a regex for "Artist - Title" pattern
    # TODO: add a denylist for obvious non-music (podcast, vlog, tutorial keywords)
    return None

def ml_classify(title: str, description: str) -> str:
    # TODO: load huggingface zero-shot pipeline once, reuse across calls
    #       (loading it per-call will make this painfully slow)
    # candidate_labels = ["music video", "podcast", "tutorial", "vlog", "other"]
    ...

def extract_artist_title(title: str) -> tuple[str | None, str | None]:
    # TODO: try a regex like r"^(.*?)\s*-\s*(.*?)(\(|\[|$)"
    # TODO: fall back to None, None if it doesn't match -- don't force a bad guess
    ...

Enter fullscreen mode Exit fullscreen mode

Milestone: run the classifier over a week's worth of real watch history. Print a summary: "Classified 42 videos: 31 music, 11 other." Manually eyeball 10 of them -- if your accuracy feels roughly 80%+, move on; ML classifiers don't need to be perfect, because Phase 5's clustering step will naturally down-weight noise. Perfectionism here is a trap -- resist polishing this past "good enough to build on."

Phase 3: The Forensic Lab -- Audio & Metadata Enrichment (Weeks 7-10)

Goal: For every confirmed music video, extract audio and metadata rich enough to power mood tagging and similarity search.

Skills gained: Working with yt-dlp, basic digital signal processing (spectrograms, mel scales), running a pretrained audio embedding model, external metadata APIs (MusicBrainz or similar), and your first taste of "architecture decision with a real trade-off."

What you'll build:

  • A pipeline step that, given a video_id, downloads (or streams) audio using yt-dlp, for local processing only -- not redistribution. This is the moment to have your privacy/ethics conversation with yourself; more in Chapter 4.
  • A function that converts raw audio into a mel spectrogram -- the standard "image-like" representation that lets you use image-flavored models (CNNs) on sound.
  • Architecture decision point, spelled out for you: you could (a) use a pretrained audio embedding model like YAMNet or OpenL3 to get a general-purpose embedding vector per track with zero training, or (b) train a small CNN yourself on your own labeled data (mood-tagged tracks) to get embeddings tuned to your taste categories. (a) is faster, works immediately, and is what you should ship first. (b) is the better learning experience and gives you a genuinely custom system -- a great Phase 3.5 stretch goal once (a) is working end to end. Don't skip straight to (b); you'll spend three weeks on a training loop for a payoff you could've had in three days.
  • Store the resulting embedding vector in a pgvector column on your tracks table.
  • A metadata enrichment step: look up the resolved artist/title against MusicBrainz's API (free, no key needed for light use) to pull canonical artist name, genre tags if available, and release year.

Learning pit stops:

  • 20 min video: search "mel spectrogram explained simply" -- you need the intuition (frequency content over time, perceptually weighted), not the full DSP math.
  • Read the yt-dlp README's "Usage and Options" section, specifically audio extraction flags (-x --audio-format).
  • Skim: YAMNet's model page on TensorFlow Hub -- its usage example is short and you can adapt it directly.

Skeleton:

# ml/audio_features.py -- the forensic lab bench
import yt_dlp
import librosa
import numpy as np

def download_audio(video_id: str, out_dir: str) -> str:
    # TODO: configure yt_dlp options for audio-only extraction
    # ydl_opts = {'format': 'bestaudio/best', 'outtmpl': f'{out_dir}/%(id)s.%(ext)s', ...}
    # NOTE: this is for local, personal analysis only -- see Chapter 4
    ...

def compute_mel_spectrogram(audio_path: str) -> np.ndarray:
    # TODO: y, sr = librosa.load(audio_path)
    # TODO: librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128)
    # TODO: convert to log scale (librosa.power_to_db) -- raw mel values
    #       span orders of magnitude, log scale is what models expect
    ...

def get_embedding(audio_path: str) -> np.ndarray:
    # TODO: load a pretrained model (YAMNet via tensorflow_hub, or
    #       openl3.get_audio_embedding) and run inference
    # TODO: pool frame-level embeddings into a single fixed-length vector
    #       (mean pooling is a fine first choice)
    ...

Enter fullscreen mode Exit fullscreen mode

Architecture decision point (data storage): store raw audio permanently, or delete it after extracting the embedding? Storing it burns disk and raises copyright/privacy questions for no real benefit -- you only need the embedding, not the waveform, for everything downstream. Recommendation: extract-then-delete, keep only the derived vector. Write this down as a design principle now; it'll save you a difficult conversation with yourself in Month 4 when your disk is full of MP3s you don't need.

Milestone: pick five songs you know well, run them through the pipeline, and eyeball their embeddings' pairwise cosine similarity. Two songs by the same artist in the same genre should be closer to each other than to a wildly different genre. If your embeddings pass this smell test, you have a forensic lab that actually works. Screenshot the similarity matrix -- this is the moment the system starts to have taste.

Phase 4: The Behavioral Profiler -- Learning What You Actually Like (Weeks 11-14)

Goal: Turn raw watch behavior into an implicit "how much did you like this" score, and build your first learned model of taste.

Skills gained: Implicit feedback theory, feature engineering from event logs, neural collaborative filtering, thinking in terms of "signal vs noise" in behavioral data.

What you'll build:

  • A feature engineering step that, per track, computes: percent watched, replay count within a session, skip-within-first-10-seconds flag, time-of-day distribution, day-of-week distribution.
  • An implicit score -- not a 1-5 star rating (you never gave one), but a derived number. A simple, honest starting formula: 
implicit_score = w1 * completion_ratio
                + w2 * log(1 + replay_count)
                - w3 * early_skip_penalty

Enter fullscreen mode Exit fullscreen mode

where completion_ratio = watched_seconds / video_duration, capped at 1.0, early_skip_penalty is 1 if you skipped within the first 10 seconds and never returned, else 0, and w1, w2, w3 are weights you pick by hand at first (try w1=1.0, w2=0.5, w3=0.8) and refine once you have enough data to eyeball whether the resulting rankings feel right. This is implicit feedback in a nutshell: you're inferring preference from behavior, not asking for it directly, and every implicit signal is a proxy, not ground truth -- a song you love but skip because you're not in the mood right now looks identical, in raw data, to a song you dislike. Time-of-day and mood context (Phase 5) is what starts to disambiguate the two.

  • The math of implicit feedback, briefly: classic explicit recommender systems (star ratings) treat missing data as "unknown." Implicit feedback systems (like this one) treat everything as a signal -- even the absence of a play is information (you had the chance to watch it and didn't). The seminal framing here, worth reading about, is "confidence-weighted preference": you're not just modeling whether you like something, you're modeling how confident the system should be in that preference, where confidence scales with how many times you've interacted with a track.

  • A first neural collaborative filtering sketch, once you have a few hundred (track, implicit_score, context) rows. Neural CF replaces the classic matrix-factorization approach (learn a vector per user, a vector per item, dot-product them to predict preference) with small embedding layers you can extend to include context features (time of day, mood tag) alongside the learned latent vectors.

Learning pit stops:

  • 25 min: search "implicit feedback recommender systems explained" -- get comfortable with the idea that "no interaction" is still data.
  • Skim: Keras's Embedding layer docs -- this is the core building block of neural CF.
  • Optional deeper dive (30 min): the original "Collaborative Filtering for Implicit Feedback Datasets" paper's abstract and intro -- you don't need the full math, just the framing.

Skeleton (Keras):

# ml/behavior_model.py -- teaching the system your taste, one skip at a time
import tensorflow as tf
from tensorflow.keras import layers, Model

def build_neural_cf(n_tracks: int, embedding_dim: int = 32) -> Model:
    track_input = layers.Input(shape=(1,), name="track_id")
    context_input = layers.Input(shape=(4,), name="context_features")  # e.g. hour_of_day, day_of_week, recent_genre_onehot...

    track_embedding = layers.Embedding(n_tracks, embedding_dim)(track_input)
    track_vec = layers.Flatten()(track_embedding)

    # TODO: concatenate track_vec with context_input
    # TODO: pass through 1-2 Dense layers with ReLU
    # TODO: final Dense(1, activation='sigmoid') to predict a
    #       normalized implicit_score in [0, 1]

    ...
    return Model(inputs=[track_input, context_input], outputs=..., name="neural_cf")

def compute_implicit_score(row) -> float:
    completion_ratio = min(row["watched_seconds"] / row["duration_seconds"], 1.0)
    replay_bonus = 0.5 * (row["replay_count"] ** 0.5)  # diminishing returns
    early_skip_penalty = 0.8 if row["early_skip"] else 0.0
    return max(0.0, completion_ratio + replay_bonus - early_skip_penalty)

Enter fullscreen mode Exit fullscreen mode

Architecture decision point: do you train one global model, or treat this as pure feature engineering with no trained model at all (just weighted heuristics)? With a personal dataset of maybe a few thousand tracks and a few weeks of history, you may not have enough data for a neural model to meaningfully outperform a well-tuned heuristic score. Recommendation: ship the heuristic first (it's basically free), and treat the neural CF model as a learning exercise you compare against the heuristic -- a great "does my fancy model actually beat my simple baseline?" experiment, which is one of the most important habits in applied ML.

Milestone: for a handful of tracks you have strong feelings about, check whether your implicit score ranks them the way you'd expect -- your most-replayed song should sit near the top, something you skipped every time near the bottom. If the ranking basically matches your gut, you've built a taste profile that didn't exist an hour ago in any explicit form.

Phase 5: The Case Board -- Clustering & Playlist Generation (Weeks 15-18)

Goal: Group tracks into meaningful clusters and turn each cluster into a named, explainable playlist.

Skills gained: Unsupervised clustering (HDBSCAN, K-Means), cluster labeling strategies, combining embeddings from different modalities (audio + behavior + text mood tags), scheduling recurring jobs.

What you'll build:

  • A combined feature vector per track: concatenate (or weighted-sum) the audio embedding, a mood-tag one-hot/multi-hot vector (from zero-shot classification against labels like "chill," "energetic," "sad," "focus," "romantic," "aggressive"), and behavioral context features (average time-of-day, day-of-week distribution).
  • A clustering pass using HDBSCAN over that combined vector space. Why HDBSCAN over plain K-Means: K-Means forces every point into a cluster and requires you to pre-specify the number of clusters -- a bad fit here, because you genuinely don't know how many "moods" exist in your listening history, and forcing an odd one-off track into a cluster it doesn't belong in produces bad playlists. HDBSCAN discovers the number of clusters itself and explicitly labels outliers as noise (cluster = -1) rather than cramming them somewhere.
  • A cluster-labeling step: for each cluster, look at the most common mood tags among its members, the most common time-of-day, and generate a human name via simple rules ("if dominant mood is 'chill' and dominant time is evening -> name candidates: 'Evening Chill', 'Wind Down'"). This is a fun place to eventually swap in an LLM call ("given these five representative songs and these mood tags, suggest a playlist name") as a stretch goal.
  • Specialized playlist types layered on top of the general clustering:
    • Most-played: pure sort by implicit_score / play_count, no clustering needed.
    • Mood playlists: direct output of the HDBSCAN clusters.
    • Study / focus: filter tracks where the zero-shot classifier's "instrumental" or "low-vocal-density" signal is high and mood tag includes "calm" or "focus" -- this is a good spot to add a simple vocal-presence heuristic (spectral features like zero-crossing rate can proxy for vocal presence without a dedicated model).
    • Travel: filter by tracks played during sessions your extension can tag as "away from home" if you choose to add coarse geolocation (optional, privacy-sensitive -- see Chapter 4) -- or more simply, tracks played during longer, uninterrupted sessions (a proxy for commutes or trips).
    • Language: filter using a language-detection pass over the (translated or original) lyrics/title text, using a lightweight library like langdetect or a Hugging Face language-ID model.

Learning pit stops:

  • 20 min: read HDBSCAN's own documentation "How HDBSCAN Works" page -- it's written for exactly this level of newcomer and has excellent visuals.
  • Skim: scikit-learn's clustering comparison page to see K-Means vs DBSCAN vs HDBSCAN side by side on toy datasets -- seeing the failure modes visually will save you hours of confusion later.

Skeleton:

# ml/cluster.py -- the case board, where scattered evidence becomes a story
import hdbscan
import numpy as np

def build_combined_vector(audio_emb, mood_vec, behavior_features) -> np.ndarray:
    # TODO: consider normalizing each component before concatenating --
    #       otherwise a component with larger raw magnitude (e.g. audio_emb)
    #       will dominate the distance metric HDBSCAN uses
    ...

def cluster_tracks(vectors: np.ndarray, min_cluster_size: int = 5):
    clusterer = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, metric="euclidean")
    labels = clusterer.fit_predict(vectors)
    # labels == -1 means "noise" -- HDBSCAN is telling you this track
    # doesn't fit cleanly anywhere yet. That's useful information, not a bug.
    return labels

def name_cluster(track_rows_in_cluster) -> str:
    # TODO: find most common mood tag, most common time-of-day bucket
    # TODO: map (mood, time_bucket) -> candidate name via a lookup table
    # TODO: fall back to "Mix #<id>" if nothing matches cleanly
    ...

Enter fullscreen mode Exit fullscreen mode

Architecture decision point: re-cluster from scratch every time, or incrementally assign new tracks to existing clusters? Full re-clustering is simpler to reason about and is what you should build first -- run it nightly over your whole track history. Incremental assignment (using each cluster's centroid or a trained classifier) is faster and preserves playlist identity over time (your "Monday Chill" playlist doesn't get renamed every week), and is a great optimization once the naive approach is working and you notice it's slow or that playlist churn is annoying you.

Milestone: run the full pipeline end to end and get back a JSON structure of named playlists with real tracks in them. Take a screenshot of the raw output -- you now have an AI DJ's set list, even before there's a frontend to look at it.

Phase 6: The Broadcast -- Web Player & Full Integration (Weeks 19-22)

Goal: A React frontend that displays your playlists and actually plays music, wired end-to-end to everything you've built.

Skills gained: React component architecture, working with an audio player (HTML5 <audio> or the YouTube IFrame API), API integration patterns, and the specific discipline of wiring together five already-working pieces without breaking any of them.

What you'll build:

  • A GET /playlists endpoint that returns your generated playlists with track metadata.
  • A React app with a playlist grid, a track list per playlist, and a persistent bottom player bar (the "Spotify-like" feel).
  • Playback, decision point spelled out: you have two real options. (a) Embed the YouTube IFrame Player API and drive it programmatically -- simplest, fully compliant with YouTube's terms, but visually and functionally you're still "a YouTube player with a nicer wrapper," and you can't easily do gapless playback or true background audio. (b) Serve audio you've extracted yourself, from your own backend, for genuinely personal, non-redistributed use -- gives you a true from-scratch audio player experience (waveforms, gapless transitions, real <audio> element control) but carries real copyright and terms-of-service responsibilities you must take seriously and keep strictly personal/local, never exposed publicly. Recommendation: build the IFrame API version first, since it's unambiguously fine to build and ship as a personal tool, and treat the self-hosted audio player as an explicitly-labeled "local-only, for-my-eyes-only experiment" if you choose to explore it, never something you deploy publicly.
  • A "similar tracks" widget: given the currently playing track's embedding, query pgvector for nearest neighbors (ORDER BY embedding <-> :current_embedding LIMIT 5) and render them as suggestions.
  • Wire the frontend's play events back into the extension's event pipeline (Chapter 1's feedback loop closes here) -- plays that originate in your own app should count as behavioral signal too.

Learning pit stops:

  • 20 min: the YouTube IFrame Player API "Getting Started" guide -- focus on loadVideoById, playVideo, and the onStateChange event.
  • Skim: any short React hooks refresher if it's been a while (useState, useEffect, useContext for a simple global "currently playing" state).

Skeleton:

// frontend/src/components/Player.jsx -- the stage where the case gets solved
import { useState, useEffect, useRef } from "react";

export default function Player({ currentTrack }) {
  const playerRef = useRef(null);

  useEffect(() => {
    if (!currentTrack) return;
    // TODO: if using YouTube IFrame API:
    //   playerRef.current.loadVideoById(currentTrack.video_id);
    // TODO: log a 'play' event back to your /events endpoint here,
    //       reusing the same schema Phase 1 established
  }, [currentTrack]);

  return (
    <div className="player-bar">
      {/* TODO: track title, artist, progress bar */}
      {/* TODO: mount the YouTube IFrame player (hidden or small) here */}
    </div>
  );
}

Enter fullscreen mode Exit fullscreen mode 

# backend/main.py (addition) -- serving the case board to the courtroom
@app.get("/playlists")
def get_playlists():
    # TODO: query your clusters/playlists table
    # TODO: for each playlist, include a handful of representative tracks
    #       and the playlist's dominant mood tags for display
    ...

@app.get("/tracks/{track_id}/similar")
def similar_tracks(track_id: str, limit: int = 5):
    # TODO: SELECT * FROM tracks ORDER BY embedding <-> (
    #         SELECT embedding FROM tracks WHERE id = :track_id
    #       ) LIMIT :limit
    ...

Enter fullscreen mode Exit fullscreen mode

Milestone -- the big one: open your app, see real named playlists built from your real YouTube behavior, click play, and hear music. Take a screenshot of "Monday Chill" (or whatever your system named it) with real tracks in it. This is the moment described in the opening story finally closes the loop -- you are, quite literally, looking at the output of a detective story you wrote about yourself.

Chapter 3: The Brain of the System — Meet Your Neural Concierge

Phases give you a build order. This chapter zooms into the thinking tools you'll reach for repeatedly — the habits of mind that separate "I copied a tutorial" from "I understand what I built."

How to search, read docs, and debug like you've done this before

You haven't done this before, and that's fine — nobody starts out knowing how to google effectively for a niche integration. Here's the actual process, made explicit:

  1. Name the smallest unit of your confusion. Not "how do I do audio classification" — that's a whole field. Instead: "how do I get a fixed-length vector out of a variable-length audio file using a pretrained model." That sentence is close to a good search query.
  2. Search for the library name plus the verb, not the concept plus the verb. "yt-dlp extract audio python" beats "how to download youtube audio." Library-specific queries land you on docs and GitHub issues where someone already hit your exact wall.
  3. Read the signature before the prose. When you land on a function's docs, look at its parameters and return type first. Half the time that alone answers your question faster than the paragraph above it.
  4. When you're stuck for more than 20-30 minutes, write down what you expected to happen and what actually happened, in one sentence each. This is the single highest-leverage debugging habit there is — most of the time, writing that sentence reveals the bug before you've even searched anything, because it forces you to notice the gap between your mental model and reality.
  5. Reproduce in isolation. If your pipeline breaks on the audio-embedding step, don't debug it inside the full nightly job — copy the failing call into a throwaway script or a notebook cell with one hardcoded input. You want the shortest possible loop between "change something" and "see the result."

You'll use this loop dozens of times across the next five months. It's worth more than any specific fact in this guide.

Trade-off thinking, one more time with feeling

Every phase above included at least one "architecture decision point," and that's not decoration — it's the actual skill this project is teaching you. Real systems are built from a sequence of decisions where there's no objectively correct answer, only a right answer for your constraints. The questions to ask yourself every time are the same four:

  • What does each option cost me right now (time, complexity, new concepts to learn)?
  • What does each option cost me later (technical debt, rework, scaling pain)?
  • Do I actually have the problem this option solves, or am I solving a problem I imagine having?
  • Can I try the cheap option first and upgrade later, or is this a decision that's expensive to reverse?

Notice that almost every recommendation in this guide follows the same shape: start with the boring, cheap, well-understood option; treat the fancy option as a deliberate, isolated upgrade once you've felt the limits of the simple one. That's not a lack of ambition — it's how you make sure the ambition is spent on the parts of the project that are genuinely novel (your specific taste model, your specific clustering logic) instead of getting burned on generic infrastructure choices that a thousand tutorials have already solved for you.

NLP for mood: zero-shot first, fine-tuning as a deliberate upgrade

You'll use Hugging Face's zero-shot-classification pipeline in two places: initial content classification (Phase 2) and mood tagging (Phase 5). It's worth understanding why zero-shot works at all — the underlying model (typically an NLI, natural-language-inference, model like BART-MNLI) was trained to judge whether one sentence entails another. Zero-shot classification is a clever repurposing: it turns "does this song feel energetic?" into "does the premise 'this text describes energetic music' entail the hypothesis built from your candidate label," and ranks labels by entailment confidence. You get calibrated-ish multi-label scores over any label set you dream up, with zero training.

The catch: zero-shot models are working from text — titles, descriptions, maybe lyrics if you fetch them — not from the audio itself. A song titled "Sunny Day Vibes" will zero-shot-classify as upbeat even if the actual track is a somber ballad using the title ironically. This is exactly why Phase 5's combined vector blends the zero-shot mood tags with the audio embedding — text tells you what the creator labeled the song as, audio tells you what it actually sounds like, and behavior tells you how you actually responded to it. None of the three signals is trustworthy alone; together they triangulate something closer to the truth.

When to fine-tune instead of relying on zero-shot: once you've hand-corrected mood tags on maybe 200-300 tracks (a natural byproduct of using your own app and noticing "this is tagged wrong"), you have enough labeled data to fine-tune a small classifier — even a lightweight one trained on top of frozen text embeddings — specifically on your vocabulary of moods and your taste in how songs get labeled. This is a genuinely great Month 4-5 stretch goal: it's the moment your mood tagger stops being "a generic pretrained model" and starts being "a model that knows your ears."

The math of implicit feedback, one level deeper

Section from Phase 4 gave you a working formula. Here's the conceptual core worth sitting with: in explicit feedback (star ratings), the model's job is to predict a rating, and the loss function punishes wrong ratings directly. In implicit feedback, there's no rating to predict — only binary or continuous signals of engagement. The standard reframe (going back to Hu, Koren & Volinsky's foundational work) is to treat every (user, item) pair as having a binary preference (di