How Honeycomb improved observability with LLMs

Phillip Carter, Principal Product Manager at honeycomb.io, joins us to discuss developments in OpenTelemetry, how Honeycomb has used LLMs to improve observability, best practices for building with LLMs, and the future of LLMs and machine learning as a whole.

The history of observability

About five years ago, most observability and APM tools all had custom instrumentation, custom agents, and SDKs. By signing a contract to use one, it became extremely hard to leave it when prices changed, the tool stopped gaining traction, or other reasons. 

Apart from migration training, at the most fundamental level, data generated to power the tool was locked into a particular vendor. 

Initiatives, such as OpenCensus and OpenTracing, pushed for a vendor-neutral standard to combat this dilemma. However, they did not dig deep enough to gain the amount of traction required to change anything. This led to a horrendous situation — multiple competing and incompatible standards, all with the dominant marker forces behind them.

In 2019, several prominent leaders in the space come together and concluded that the environment was chaos from a vendor standpoint and non-friendly from a user’s perspective. It was time to make a massive change.

With this shift moving to OpenTelemetry no longer constitutes a sacrifice. With better benefits and features than proprietary options, as well as the best observability and the best generated data offerings, OpenTelemetry seems like the obvious choice. 

How HoneyComb leaned into LLMs

Last year, Phillip wrote a mini-manifesto explaining why honeycomb should include machine learning in their product.

“All of a sudden, this idea of being able to just say what I want to be a query in plain English because I have my mental model of what I want to be a query,” Phillip says.

After about a month, Phillip had a prototype developed.

“I worked with a developer, and he spent about two days putting something janky in the UI that used our API to show that query. It had the end-to-end working of a textbox, you could hit enter, just make sure you did not do a whole bunch of different things in this UI, otherwise, it would crash. Then you would get a query that was executed,” Phillip says. “Everyone’s minds were blown. Now this is possible.”

Honeycomb’s mindset shifted instantly. With an agreement with the leadership of a one-month deadline, Phillip and his partner were off to excitedly rush to the end result, ruthlessly prioritizing the entire time.

When the project was complete and the press release was shared, it went wild. Through all of the problems, the tool collected user feedback that, in turn, was used to iterate and grow quickly. 

“We used an observability mindset to iterate with something in production, pushing code every single day to fix one thing, and then take a look at the next 24 hours of data and ask if this materially improved what we saw improved,” Phillip says. 

By focusing on reoccurring questions and problems, the team had a constant improvement mindset. The most important aspect of Honeycomb’s push is user interaction. 

“By and large, we want you interacting with it with this thing. Once you have a query in place, it's really, really easy to click in a box and group by another field or add another visualization or something like that. And if you know you can do that, we got you. And I mean that in a positive sense. People come to us because they want to query their data.” Phillip says. 

‍

Interested in learning more? Listen to our full conversation, where Phillip explains the history of LLMs, what is changing, what the future can bring, and more. Listen on Apple Podcasts, Spotify, or your favorite podcast player.