How Duolingo automates service creation and maintenance to tackle more impactful infra work

Duolingo is a popular language-learning app that launched in 2012, with the aim of offering fun, free, and effective education. Throughout the years, it’s expanded to offer ever more languages and the Duolingo English Test, amongst other education products. In his nine year tenure, Principal Software Engineer, Art Chaidarun, has been behind the scenes helping guide and scale with this success. In that time, he’s touched dozens of initiatives and projects across Android, iOS, and web development. Now, he’s extra focused on developer productivity, which led him down the path to OpsLevel.

Challenge 

Scattered microservice information

Back in 2015, Duolingo began the switch from a monolith to microservices. “At that point, it was like the wild west with no central management or standardization around what language versions or libraries our services were using,” Art says. “Engineering had problems around visibility into microservices and their tooling, which also made enforcing company-wide changes nearly impossible.”

To solve the problem, Duolingo created several home-grown microservice portals over the years—but they never quite reached their potential. “The attempts weren’t  comprehensive, focusing only on a few aspects of each microservice rather than the full picture. Sometimes we would make it an intern project, but it would be harder to prioritize ongoing development work after that person went back to school,” Art explains. “All this custom code underlying everything made it hard to change or update, especially for platform engineers without much frontend experience.” As Duolingo continued its explosive growth, the in-house solution was becoming an all-out problem, leading Art to explore a better solution.

Solution

An automated single source of truth

In searching for something with the right balance of flexibility and out-of-the-box functionality to help Duolingo finally have an automated, single source of truth for their catalog, OpsLevel stood out as the right solution.

Getting to work right away, Art set up Duolingo’s instance in OpsLevel to get services, documentation, and tooling logged. Once he set it up, he created a script to run every hour using OpsLevel’s API to pull from sources like GitHub and Rollbar to ensure that the data was always up to date, automatically. “Once we had the script, and the data was reliable and hands off, that was really appealing to people.”

Another selling point was OpsLevel’s rubric—not just because it replaced Duolingo’s outdated, manual one, but because of its gamification. “I don’t have to force teams to get in there and get their services to higher levels of maturity. They’ve simply been completing silver and gold level checks because it’s gamified,” Art says. 

Use cases

Service creation templates and quarterly campaigns

The real game changer has been service creation templates, which not only save developers time when they’re adding a service, but also give them the peace of mind knowing that they did it correctly—with all services standardized this way. “It’s a lot more self-serve now. You don’t have to ask a bunch of teammates how it’s done or wait for approvals. I’ve also seen an uptick in contributions to make the OpsLevel templates better, which was harder in the old system since people found its logic less straightforward or were just not engaged enough to propose feedback.” Duolingo's Service Creation templates include a React frontend template, Python and Java backend templates, and a minimal template for all other use cases, such as a repository of CI/CD scripts.

Duolingo also uses quarterly campaigns to prioritize platform-level infrastructure tasks that were previously overlooked or put off, such as updating libraries and fixing security holes.

"The CTO approves the checks that go into a campaign every quarter, like upgrading Python services, which are a lot more challenging to do without this visibility. The quarterly campaigns are consistently achieving completion rates of 99–100%.”

Art Chaidarun
Principal Software Engineer at Duolingo 

Results

Tackling new work that wasn’t possible before

Having implemented OpsLevel, Art notices that, “we now do things that we just didn’t before. I also see people are a lot more engaged in getting their services up to speed and going for the gold level.”

Duolingo’s initial problems around microservice visibility and enforcing company-wide changes have been completely solved in that, “all our services are in the catalog, they get updated every hour with an automated script, and our quarterly campaigns are respected and get completed.” 

“By taking care of this day-to-day maintenance," Art adds, "OpsLevel frees up more time for our platform teams to actually support our product teams to build bigger and better Duolingo features.”