Sleuth helps Jackpocket increase releases by 200 percent

Quick facts:

• Engineering leaders at Jackpocket wanted to engrain industry best practices as they scaled their team
• The “Accelerate” book, DORA metrics, and Sleuth helped them measure and improve on engineering efficiency
• Since implementing Sleuth, the Jackpocket team increased releases on their largest repo by about 200%, going from 1 per week to 3.2 per week (other projects are released multiple times per day)
• Jackpocket operationalizes engineering efficiency best practices using Sleuth and DORA metrics

What does it take to establish a DevOps culture within an existing engineering team with set processes and procedures? How do you convince an engineering team to shift from outdated but comfortable practices to modern and efficient ways of working?

To prepare for scaling their team, Josh Nykamp, VP of engineering, and Todd Resudek, principal engineer at Jackpocket, knew they needed to adopt mature processes from the get-go and avoid mistakes they had seen while working at other companies.

“We made some very deliberate choices about who we're going to hire, where we're going to scale and what we're going to do along the way,” Josh says.

To get their team on the same page, they had to build trust in what they wanted to do and show that their ideas were common best practices for a modern SDLC. Keep reading to learn how they did it, or watch the video below.

Introducing DORA metrics and Sleuth

They started by reading “Accelerate” by Nicole Forsgren, Jez Humble and Gene Kim, as a team.

It provided a lexicon and framework for what to measure – the DORA metrics. Next, they wanted a tool to help them track the metrics, relay that information to the team, and help engrain the best practices the book promotes.

“It was a pretty easy solution for us to evaluate Sleuth,” Todd says. “We needed to measure DORA metrics, track how we're doing on a team-by-team basis, and see where we’re making progress. Sleuth checked all the boxes and made the knowledge that we got from ‘Accelerate’ into something measurable and actionable.”

Solving a key problem: Infrequent releases

Prior to using Sleuth and measuring the DORA metrics, the Jackpocket team struggled with infrequent releases, which created large batch sizes and frequent rollbacks.

They focused first on reducing batch size, using the deployment frequency DORA metric as a proxy for measurement.

“Batch size is really our linch pin on everything,” Todd says. “If we get our batch size down, it cures a lot of other ills. The bugs can't be too big, the releases in general are pretty stable because you're not releasing that much code, and the mean time to recovery after a rollback is minimal.”

Focusing on deploying more frequently as a team inherently meant releasing in smaller batches.

“And because the batch sizes are smaller, we're seeing less frequent rollbacks and incidents,” Todd says. “We have different teams working on different projects all the time, and Sleuth gives us one place we can go to aggregate all the data, just say, this one is doing this, this one's doing that, and disseminate the information to the appropriate teams in a digestible way.”

Within the past year, the team increased their deploy frequency 200%, going from .2 releases a day to .64 a day.

Operationalizing DORA metrics

Jackpocket’s engineering manager runs biweekly meetings to review the data and projects, note what needs to be improved, and add and remove projects.

Sleuth provides project and team views; Jackpocket looks first at the team level, reviewing all the PRs that went out from the team members and the aggregation of that data. If there's a drop in one of the metrics, they drill down into the logs on the Sleuth dashboard to see where and why the incident occurred.

Josh also uses Sleuth to enforce team practices. For example, reviewing recovery time in Sleuth showed that the team had open incidents. That led to realizing they were handling incidents inconsistently. Using Sleuth has helped him get the team on the same page.

“Normally, I’d have to look through all the services to figure out what’s going on,” he says. “I keep Sleuth pinned on a tab in my browser, so I look at it every day. It gives me a single pane of glass to see everything.”