Sleuth vs. Swarmia vs. Jellyfish

-In this article, we will be comparing three of the best Engineering Intelligence tools (also known as Software Development Analytics tools): Sleuth, Jellyfish, and Swarmia.

Whether you’re looking to improve developer productivity or experience, or simply wanting to get data on engineering work for executive-level updates and business alignment purposes, this comparison article should help.

What is Engineering Intelligence?

Engineering Intelligence (EI), or formally Software Engineering Intelligence, is a fancy word to describe Business Intelligence (BI) but for software engineering data. You can thank Gartner for using that term.

EI tools provide visibility specifically into the work of a software Engineering organization - from delivery and operational progress and status, to efficiency, morale, and resource allocation.

EI tools do so by gathering data and metadata from developer tools, such as issue trackers like Jira, version control systems like GitHub, incident management tools like PagerDuty, and the rest of the toolchain.

Evaluating EI tools

If you're in charge of kickstarting an engineering metrics program, or if improving developer productivity is a top objective, this comparison article should help.

We begin by defining the comparison criteria, which we've divided into four categories:

Delivery Progress

Delivery Progress is all about the status and progress of currently ongoing Engineering work. This data is primarily captured in issue trackers and source control tools.

We grade EI tools on Delivery Progress based on how well they can use such data to answer questions like:

• "How are we doing overall?"
• "Are our in-flight projects on track?"
• "Are we shipping with operational excellence?"

Execution Quality

While Delivery Progress tells us how work is going at present, Execution Quality tells us how well we're doing the work - i.e. how efficient the team has been in the software development process.

We grade EI tools on Execution Quality based on how well they can surface insights that lead to identification of improvement opportunities, both from productivity and DevEx perspectives.

Examples: how long PRs stay open and get reviewed, how big or mature the PRs are, team morale, and more.

Alignment

Alignment is about seeing the forest from the trees to answer questions like "are we working on the right things?" and "are we making the most out of resources?".

We grade EI tools on Alignment based on how well they track things like the type of work being done, the resources being allocated to the work, and the impact of the work in terms of goals and outcomes.

Ease of Adoption

Ease of Adoption goes way beyond ease of implementation. In this day and age, any SaaS tool worth its salt shouldn’t require days or weeks in man-hours to install anyway.

When we say Ease of Adoption, we mean capabilities that make it much easier for everyone to trust the data and start building a data-driven culture.

For example, they include features that improve data hygiene, and habit-forming features that get everyone used to using data for decision making.

Comparing Sleuth, Jellyfish, and Swarmia

Let's compare the top three EI tools in the market using the categories described above:

Sleuth vs. Jellyfish vs. Swarmia on Delivery Progress

Sleuth, Swarmia, and Jellyfish are pretty similarly matched in terms of the ability to report on Delivery Progress, the bread-and-butter of tools in this space.

Because information about planned and current Engineering work is captured in the form of issue tickets and pull requests, all three tools have extensive features to slice and dice such data by status, project, severity, and the like.

Note that you’d need to be tracking incidents, bugs, and escalations in an issue tracker for Swarmia to be able to report on those.

Sleuth vs. Jellyfish vs. Swarmia on Execution Quality

Execution Quality rates tools on their ability to provide deeper visibility into the development work, which practically revolves around pull requests.

For example, PR Review Lag Time is the amount of time it takes for a PR to get reviewed. If the value of this metric increases, chances are there may be PRs stuck in the review cycle.

All three tools are similar in their ability to provide insights into Execution Quality - plentiful for leaders and teams looking to identify bottlenecks and opportunities for improvement.

However, it is worth noting that with Jellyfish you’d need to be tracking PR progress using an issue tracker, which isn’t going to be the case for all Engineering organizations.

Sleuth vs. Jellyfish vs. Swarmia on Alignment

Alignment is a management concern, so invariably metrics in the category are focused on resources (people), how they are allocated, how they translate to dollars, and their morale and impact on the business.

Jellyfish leads this category, with Sleuth and Swarmia following slightly behind. Unless you need a very specific feature like ability to project investment efforts based on historical data, you can accomplish a lot with either Sleuth or Swarmia.

Sleuth vs. Jellyfish vs. Swarmia on Ease of Adoption

One important driver of Ease of Adoption is data trustworthiness. Features such as “view source” to peek at the data behind a metric or chart, and those that improve data hygiene - such as the ability to auto-notify codeowners of PRs without issue keys - help build trust.

Another important driver is manual burden. Time spent doing monitoring and reporting is time not spent on development or alignment. Features that automate manual things like writing report summary, generating scores, and surfacing outliers in the data can help here.

Finally, habit-forming features are critical for building a data-driven engineering culture. Features such as auto-creation of pre-populated templates for review meetings, such as Monthly CTO Reviews or Weekly Sprint Planning,

Sleuth leads this category given its coverage of features that drive Ease of Adoption.

Conclusion

We’ve just evaluated the top three tools for Engineering Intelligence through four categories: Delivery Progress, Execution Quality, Alignment, and Ease of Adoption. Here's where we end up:

We’ve just evaluated the top three tools for Engineering Intelligence through four categories: Delivery Progress, Execution Quality, Alignment, and Ease of Adoption.

Sleuth and Jellyfish are pretty much even on Delivery Progress, with Swarmia following right behind.

Sleuth and Swarmia have a slight lead over Jellyfish in the Execution Quality category because of Jellyfish’s inability to surface certain metrics through git data instead of issue tickets.

Jellyfish has invested the most in the Alignment category but all three tools are similarly capable.

Sleuth on the other hand has invested the most in Ease of Adoption. Among all four categories, Ease of Adoption is arguably the most important - the success of your productivity initiative or metrics program ultimately depends on how much value the teams are getting from using the tool.

Hopefully this article has given you the head start on your research for the best tools for the job, and the confidence to engage with vendors in this space.

---
If you’d like to learn more about Sleuth, you can request a demo here.