Most people use OpenTelemetry and SigNoz to watch their CPU usage, find memory leaks, or figure out why their API is taking 400ms instead of 200ms. But for the Agents of SigNoz Hackathon (Track 3: Observe Anything Weird), I wanted to do something completely different.
I didn't want to watch hardware. I wanted to watch hackers.
That is how Triage was born. It is an OpenTelemetry-powered Blue Team SOC (Security Operations Center) that tracks active cyber attacks instead of just generic application performance.
The Original Vision vs. The Reality
Building this sounded straightforward on paper: catch bad traffic, wrap it in an OpenTelemetry span, send it to SigNoz, and show it on a custom dashboard. But actually deploying this beast before the deadline was a completely different story.
If you have ever tried to deploy a full-stack Next.js app, a Python honeypot, and an OTel pipeline while the clock is ticking, you know exactly what kind of panic I am talking about. Here is what actually happened behind the scenes.
1. The Azure VM & Docker Crash
I started by spinning up an Azure virtual machine to self-host the SigNoz backend. I pulled the repo, ran docker compose up -d, and immediately watched my server completely freeze. Turns out, my free tier Azure instance (Standard_B2ats_v2) only had 1 GiB of RAM. You simply cannot run a massive ClickHouse database and a full OTel collector on 1GB of memory without it crashing instantly. I had to pivot fast and rely on the cloud endpoints.
2. The Vercel vs Localhost Trap
Once I got the backend running, I hit my next wall. My Vercel Next.js dashboard kept throwing 500 Internal Server Error on the threat simulation API, and the SigNoz connection kept reading OFFLINE (fetch failed). I was staring at my logs losing my mind until it clicked. My Vercel environment variables had SIGNOZ_API_URL set to http://localhost:8080 and my Python script was looking for OTel on localhost:4318.
Note to self (and everyone else): Vercel is a cloud server. It has no idea what your local Kali Linux machine or Azure VM is doing on localhost. I had to either use Ngrok to tunnel my local endpoints to the public internet or point everything strictly to public cloud URLs so Vercel could actually talk to the backend.
3. The Supabase Keys Nightmare
To persist the threat logs for the dashboard, I was connecting to Supabase. But nothing was writing to the database and the UI was stuck on "UNCONFIGURED".
It turns out I had mistakenly treated my master database key like a public variable. In Next.js, if you prefix a secret key with NEXT_PUBLIC_, you expose it to the browser. But my API routes were strictly looking for the exact string SUPABASE_SERVICE_ROLE_KEY to securely insert data. A single prefix mismatch broke the entire flow. Once I stripped the prefix, passed the raw service key to the Vercel settings, and hit redeploy, the database finally connected and the traces started flowing.
The Features We Built
Despite the deployment chaos, the final system works beautifully. Here is what Triage actually does:
AI-Powered Threat Analysis (Groq & Llama-3.1)
When a normal APM sees a 403 error, it just logs a generic spike on a graph. When Triage intercepts an SQL injection or a leaked API key, it captures the raw payload and immediately sends it to Groq. The LLM generates a clean, 3-sentence summary of the attack vector, which we attach directly to the OpenTelemetry trace.
Python TCP Honeypot Trap
I didn't just want to watch web traffic. I wrote an asyncio Python script that acts as a honeypot listening on port 2222 (mocking SSH) and 63790 (mocking Redis). When automated bots or tools like Nmap run a port scan against my server, the honeypot catches the IP, generates a custom OTel trace, and ships it to the collector via OTLP HTTP on port 4318.
SRE Sidekick (Auto-Ban)
We built an active defense mechanism. If the system tracks 5 attacks from the same IP within 60 seconds, it triggers a self-healing protocol. It drops an sre.auto_ban.triggered span and instantly enforces a firewall block on that attacker.
Cyberpunk SOC Command Center
I built the entire frontend in Next.js 16. It pulls live telemetry traces, shows real-time threat stats, and features a "Red Team Simulator". Judges and users can safely simulate SQL injections or data leaks directly in the browser with one click to see the telemetry populate live.
How I Actually Used SigNoz
This is where it gets fun. I didn't use any of the standard HTTP auto-instrumentation for the security layer. I built custom OpenTelemetry spans tailored specifically for a Blue Team.
Instead of tracking standard routes, I mapped out an adversary's footprint. Here is exactly what I sent to the SigNoz collector:
- Custom Spans: I created custom span names like security.trap when a web exploit was caught, and honeypot.port_scan for TCP probes.
- Security Attributes: Inside these spans, I injected attributes like security.severity = 'CRITICAL', security.threat_type = 'SQL_INJECTION', and security.source_ip.
- AI Metrics: For every payload analyzed by Groq, the trace included llm.latency, llm.model, and threat.goal.
By doing this, I could open the SigNoz Query Builder and run a simple query to see every critical SQL injection attempt across my entire infrastructure, mapped out in real-time.
Final Thoughts
This hackathon pushed me to my absolute limits. Between fighting cloud networking rules, debugging Next.js server crashes, and forcing OpenTelemetry to act like a SIEM, I learned more in a few days than I have in months.
SigNoz proved to be incredibly flexible. It isn't just for tracking uptime. If you structure your telemetry right, you can observe absolutely anything. Even the weird stuff.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.