Puma uses Sleuth to show value of key engineering initiative

Visibility is a key component of any digital transformation. You know how it goes – your stakeholders ask for proof of how it’s going and what impact their investment is having. Inquiring minds want to know what’s shipping, when, and how often.

That’s exactly what retailer Puma needed as they built a new headless commerce solution and started to release more frequently. Read or watch the full story below.

The pain

Previously, on their monolithic packaged software system, Puma’s once-a-week releases were easy to track. But as they began building a new headless front end and a new API layer, developers were releasing much more often.

“It was proving difficult to keep track of what had been released into each environment,” says Michael Gaskin, Senior DevOps Manager, Global e-Commerce at Puma. “The desire to have visibility into that, and to show how much more frequently we were releasing in the new system compared to the old one, as well as change lead time metrics, led us to Sleuth.”

The process

After setting up webhook-based tracking in Sleuth, and eventually Github Actions-based tracking, Michael’s team started to see the deployments rolling into Slack. For each code deployment in Sleuth, they can see the most recent releases and the Jira issues included with them.

“This was the kind of visibility we were looking for. It really helped us to keep track of what was going where,” Michael said.

And with Sleuth’s automations and notifications, the team no longer had to follow its old process of collecting in a Confluence page the tickets that are being released, and manually announcing the releases in all of their Slack channels.

Finally, Michael used Sleuth’s issue-based change lead time metrics to dig into the details and see how long things take from any point in Puma’s deployment process.

The gains

With this visibility in place and the DORA metrics collected in Sleuth, Michael was able to show that the team completed about 400% more releases in one year on the headless system compared to the old system.

“Sleuth is the means by which we measure that,” he says. “It’s really important when communicating with upper management. We’ve been able to show them that yes, in fact, among other things, we have achieved the ability to do these releases more often.”

Of all the DORA metrics, Puma’s deployment frequency has improved the most since using Sleuth.

“You can really see that when it comes to the number of preview environments that are being deployed to each day,” Michael says. “Each time somebody opens a pull request and proposes to merge changes in domain, we're spinning up a preview environment on demand, and it's typically 40+ deployments per day into those previews.”

The team can see a lot of coding being done, pull requests being opened, and potential new features waiting in the wings. “And now we need to clear the path for them from those preview environments into production, and that's what I'm really looking forward to doing in the future,” Michael says.

And, the change lead time metrics uncovered interesting discoveries about how the team operates.

For example, a team that was creating a new website for one region prefers a classic approach to the work and creates many issues at the beginning of the process that might not be delivered for a very long time afterward.

“We noticed that some of the issues were created very far in advance,” Michael says. “It's eye opening to see just how long some of them actually take before they come to fruition. And that kind of very long lead time is probably not a good thing, because requirements decay the longer they sit out there. Sleuth’s issue-based lead time tracking brings visibility to that.”

Bonus value

Puma also saw unexpected benefits from Sleuth Actions, currently in beta, which they used to get URLs from Github Actions that were doing deployments, and then send the links into a Jira field to enhance usability internally.

“It wasn't clear enough for our users just to say that something was deployed to staging or to development,” Michael explains. “They needed a link to click on that would take them to the place where this functionality had been deployed.”

The Sleuth team worked with Michael and delivered a way to create the URLs dynamically and assemble them, so the links could then go into a Jira field.

“Sleuth became a kind of Swiss army knife to help orchestrate things throughout our whole deployment pipeline,” Michael says. “A lot of tools are getting into DORA metrics tracking, but with Sleuth, we gain additional abilities that we don't have elsewhere.”

A word on support

Michael makes a special note that Sleuth’s support experience is a differentiator. He appreciates that when his team has needs, they can be close to the product team, which makes enhancements quickly.

“I had the greatest support experience of all time with Sleuth, which was when we were trying to set up SSO with Puma IT and we were getting a 500 error. When I went to the Sleuth Slack channel to report it, they were already waiting there for me, saying, ‘Hey, I see you're getting a 500 error when you're trying to set up SSO, so we'll fix that for you.’ No other company has ever done that in my whole experience with any tool. Nobody has managed to report an issue like that before I reported it to them.”