30 to 70 PRs a Day: How We Managed to Not Wreck Our Systems
The Honeycomb engineering team set out to double our productivity in a year. This is how we did it, what we did to keep things stable, what it cost us, and what we’re still figuring out.
Honeycomb Blog
@honeycomb-blog
Publisher
57
Posts
Posts by Honeycomb Blog
AI Amplifies Your Existing Practices: Lessons from Our Shift to an AI-First Strategy
In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2. 5x-ed our throughput using AI without breaking everything or lowering our standards for quality.
Transforming How We Run Kafka at Honeycomb
How we pulled off a large-scale Kafka migration project by learning from past mistakes, prioritizing rollback safety, and building shared knowledge.
Shipping Is Your Company's Heartbeat: A Letter from a CTO
The world is especially hard right now. The future of the software engineering profession looks more uncertain than ever.
Lattice Watch: Smarter Guardrails for Design System Observability
In this blog, learn why Juliana built Lattice Watch—an AI-assisted code review tool—to improve design system adherence at Honeycomb.
Multi-Agent Collaboration on a Shared Canvas
This post was co-written with Staff Software Engineer Martin Holman . Honeycomb Canvas is a collaborative investigation environment.
From Prototype to Production With AWS AgentCore
This post was co-written with Staff Software Engineer Martin Holman . "Hello world, this is your agent speaking!
Could vs. Should: The First Year Managing an SRE Team
As of today, I’ve drafted this post upwards of 10 times—it’s old enough that the version I first started working on was called “Reflections on 1 Year of SRE Management” (I’m currently at 2. 5 years).
What Customers Are Doing With AI and Honeycomb
At O11yCon , we talked to engineering teams across the industry, and the numbers are starting to get genuinely wild: Mixpanel DevOps Engineer Eddie Bracho told us their engineering team is generating 50% more PRs than before AI came into the mix (sorry). That kind of velocity is exciting, but i...
Instrumenting AI Agents for the Agent Timeline: A Practical OpenTelemetry Guide
AI agents are nondeterministic, multi-step, and opaque. When one fails in production, "the model said something weird" is the cheapest, most useless line in your incident postmortem.
Honeycomb Incident Report: Kafka Maintenance on May 4 and 7, 2026
On May 4th in the EU instance, and May 7th in the US instance, Honeycomb ran its only scheduled maintenance session with major planned downtime in the last five years. During the maintenance window, Honeycomb kept accepting your events, but recent data was not queryable while the work was underw...
Agent Timeline Is Now Generally Available
Honeycomb’s Agent Timeline gives you a unified view of LLM behavior and multi-agent workflows. Now generally available!
The Second Edition of Observability Engineering Is Here
IT’S HERE it’s here it’s here it’s here!!!! The second edition of Observability Engineering is available for download, and since Honeycomb is the sponsor, you can now download it from our website (the dead tree version will take another month).
Observability: Are You Measuring What Actually Matters?
Old observability metrics like uptime and MTTR aren't enough anymore. Teams must connect technical signals to business outcomes, like savings and speed.
Graviton5 in Production at Honeycomb: Per-service Results From the m8g to m9g Migration
This is the fourth installment in the Graviton retrospective series we've been writing since 2021. The methodology is the same one I always reach for: hold the workload constant, run both generations on the same Kubernetes namespace concurrently, and let the per-pod numbers speak.
Running the OpenTelemetry Collector as a Lambda
The OpenTelemetry Collector is usually deployed as a long-running process: a sidecar, a DaemonSet, an EC2 instance, a docker container on my computer. It sits there listening for telemetry.
It Can Only Goodhart Happen
You’ve probably read this quote in relation to any number of things over the years. People complaining about arbitrary metrics like PRs merged, lines of code produced, and now, token usage.
How Support Uses Honeycomb to Debug Honeycomb
You'd think that working at an observability company means everyone knows exactly where to find everything in the data. It doesn't.
Top 7 Dynatrace Alternatives in 2026
Dynatrace is a full-stack observability and application performance monitoring (APM) platform designed for large-scale enterprise environments. It became popular because it combined infrastructure monitoring, APM, logs, digital experience monitoring, and AI-assisted root-cause analysis into a si...
Everything We Talked About at O11yCon 2026
We just wrapped O11yCon 2026, and this year's conversations hit differently. Agent-based software development is here, now .
Honeycomb Canvas: The Multiplayer Workspace for the Agentic Era
Last week , we launched a major update to Canvas, our investigation workspace. The new Canvas has evolved from an AI co-pilot you chat with to a place where your whole team, human and agent, can work the same problem on the same surface.
Agent Timeline: The Flight Recorder for Your AI Agents
Last week , we introduced Agent Timeline, a powerful new observability experience purpose-built for debugging AI agent workflows in production. Agent Timeline uniquely connects AI-layer visibility to full-stack observability by organizing telemetry around an agentic conversation.
How Honeycomb Is Embracing the Challenges of End-to-End Observability with Embrace
From Honeycomb’s perspective Customers regularly come to us looking to solve their observability problem by connecting the dots from frontend to backend. It sounds straightforward in theory, but in practice it's one of the hardest problems in modern application monitoring.
Honeycomb Achieves the AWS Financial Services Competency
Honeycomb is proud to share that we have achieved the Amazon Web Services (AWS) Financial Services Competency. This recognition validates our technical expertise and proven customer success in assisting financial services organizations with building, running, and understanding their production s...
Innovation Week Day 2: Observability for AI, and Observability With AI
AI is reshaping the SDLC in two directions at once. AI-generated code is shipping faster and with less human supervision than ever before, while agents and LLMs are running directly in production, where they behave very differently from traditional software: non-deterministic, with a wider blast...
Innovation Week Day 1: The SDLC Is Collapsing, and Observability Has Never Mattered More
The software development lifecycle is collapsing. The multi-stage pipeline that defined how software got built and shipped for decades is compressing into rapid loops of intent and validation, with agents now part of the teams building and running it.
Making Semantic Conventions Work for You With OpenTelemetry Weaver
Your dataset has hundreds of attributes. Some are self-explanatory: http.
Span or Attribute in OpenTelemetry Custom Instrumentation
More attributes on one span means more correlation power and cheaper telemetry. Learn when to reach for spans or logs instead in OTel custom instrumentation.
Top 7 Grafana Alternatives in 2026
Grafana is one of the most widely adopted visualization and monitoring tools in the observability space. Its open-source roots, flexible plugin system, and native support for Prometheus, Loki, and Tempo have made it a default starting point for many engineering teams.
Taming Log Noise With the OpenTelemetry Collector's Drain Processor
Do you receive 50 million log lines per day and struggle to see what actually matters? Health checks, heartbeat pings, connection pool messages—they all drown out the errors and anomalies you're trying to find.
New in the Honeycomb Academy: Learn to Use the Honeycomb MCP
The Honeycomb MCP course in the Honeycomb Academy gives you a starting point when you're not sure what to ask and teaches you how to direct an investigation.
Approaching the Parhelion
A parhelion is created when light refracts through hexagonal ice crystals in the atmosphere, forming bright spots that appear on the horizon, connected by a faint halo. You don’t have to squint very hard to appreciate how relevant this is to our current AI moment, where we’re surrounded by token...
Managing OpenTelemetry Semantic Convention Migrations With the Collector
Real production data tells the story better than I can. Juraci Paixão Kröhling , a friend and fellow observability practitioner at OllyGarden , recently shared an example from an anonymized production environment: 1,830 occurrences of http.
Fast AI Feedback Loops with Honeycomb and OpenTelemetry
Are you writing agentic applications, but aren’t sure what the agents are doing? Finding out too late that you've blown the budget with super expensive models?
You Don't Need Three Pillars, You Need Single Threads
Last week was a great reminder for me about the challenges of the traditional model of observability defined by the “three pillars” of metrics, logs, and traces. One of the customers I’m currently working with is a large financial institution that has a robust three pillar implementation.
ICYMI: Is This Code Worth Running? Here’s How to Know
We’ve been exploring what about software development and observability changes with AI, and what doesn’t. Our conclusion: these 5 principles will remain true.
Optimizing the OpenTelemetry Python SDK for LLM Workloads
Agentic workloads thrive with precision tooling. Just like developers, they need the rich context, high cardinality, and fast feedback loops that allow them to ask exploratory open-ended questions of their code.
Your Questions About AI Agents and Production Feedback Answered
On April 1st, I joined Akshay Utture from Augment Code for a webinar on how AI agents use production feedback to improve code . We covered a lot of ground—the DORA report's findings on AI driving both throughput and instability, why observability is shifting left into the development process,...
Uncertainty and Change Are Everywhere in Software Development
This article introduces a set of theory and practice for adapting to what is one of the most rapid changes to how work is done in arguably any career: AI.
The Fundamentals: Fast, Deep, and Ready for What Comes Next - Part 3
The previous two posts in this series have looked at some of the use cases Honeycomb customers are implementing to observe LLMs in production and power agentic observability workflows. In this third and final post, we’ll take it back to basics and look at how the fundamental capabilities and inf...
AI Working for You: MCP, Canvas, and Agentic Workflows - Part 2
In our previous post in our series on observability for the agent era, we looked at how Honeycomb provides unique visibility into LLMs operating in your production environment. Now, let’s flip it around and explore how Honeycomb provides observability insights uniquely suited to helping your AI...
Honeycomb Is Built for the Agent Era. Here's the Proof - Part 1
The agent era is here. Engineering teams are shipping AI-powered products, deploying multi-agent systems, and trying to figure out what observability even means for non-deterministic systems.
KubeCon + CloudNativeCon EU 2026: What We Learned About AI, Observability, and Fast Feedback Loops
Honeycomb was excited to attend KubeCon + CloudNativeCon Europe, where one theme stood out across sessions: as AI reshapes how software is built and run, teams are being pushed to rethink how they understand their systems. Without strong observability and feedback loops, AI can accelerate confus...
From Honeycomb Customer to Bee: An Observability Champion's Journey
How Josh Parsons championed observability at Amperity, ran a winning tooling evaluation, and ultimately joined Honeycomb.
Accelerate Your OpenTelemetry Migrations With Honeycomb's Agent Skills
Since releasing our hosted MCP server last year, we've been thrilled to see customers not just adopt it but build Honeycomb deeply into their agentic development and observability workflows. Users have embraced it, leveraging Honeycomb to stay in conversation with their code and understand how...
Scary Things Happen in Production. Context Helps You Find Them.
Production is a rowdy place of chaos, especially at scale. When you have millions of requests per second flowing through your system, weird things are always happening.
Leveraging Cognitive Diversity to Tackle System Complexity
Most engineering leaders today understand that diversity matters. They've built teams that reflect a range of backgrounds, functions, and experience levels.
Production Is Where the Rigor Goes
In early February, Martin Fowler and the good folks at Thoughtworks sponsored a small, invite-only unconference in Deer Valley, Utah—birthplace of the Agile Manifesto—to talk about how software engineering is changing in the AI-native era. They recently published a summary of key insights a...
Shifting Metrics Right
In the shift left era where it feels like we’re pushing everything as far to the start of the SDLC as we can, it may seem counterintuitive to shift anything right. That is, however, exactly what I suggest when it comes to generating metrics.
Evaluating Observability Tools for the AI Era
Every observability vendor has an AI story right now. Most have an MCP .
Observability Where You Work: Introducing the Honeycomb Slackbot in Beta
Engineers are constantly context switching between tools, adding cognitive overhead on top of already complex work. You're deep in an investigation, you need to analyze some data, pull up a runbook somewhere else, and share findings back in Slack.
Create a Custom Service Health Board With the Honeycomb MCP
Your software is sending data to Honeycomb. Now where is the dashboard you want?
Your Questions About AI-Assisted Development Answered
We recently hosted a webinar on AI-assisted development with DORA , and the audience had a lot of questions—far more than we could get to in an hour. I picked out six that get at the stuff people are wrestling with day to day.
Incident Report: Exercises, Cleanups, and Evacuations
Every year, Honeycomb runs disaster recovery scenarios in multiple environments, including in production. Although each of our instances runs in a single region, on at least three Availability Zones (AZs), we have multiple plans for partial regional failures, and particularly, zonal failures.
Using Core Web Vitals in Honeycomb Frontend Telemetry
Google's Core Web Vitals (CWVs) measurements have been used by web administrators and SREs to review frontend application performance metrics, and have been factored into Google's page rankings since 2021. They are also used in Google Analytics, which crawls websites and evaluates performance me...
The Next Era of Observability: Founders’ Reflections - Additional Q&A
What happens when the people who helped define observability take a hard look at AI? That’s what Honeycomb co-founders Christine Yen (CEO) and Charity Majors (CTO) dug into during this webinar, starting with the early days of observability (back when it wasn’t even a category yet).