Sauce Labs Announces Results of Inaugural Continuous Testing Benchmark Report

• Industry-first report identifies benchmarks necessary to achieve continuous testing excellence
• Leverages anonymized data from millions of real-world tests conducted on company’s continuous testing platform to track organizations’ performance against defined benchmarks
• Enables organizations to measure how their continuous testing efforts compare with peers
• Reveals need for increased focus on improving test quality and shortening test run times

AUSTIN, Texas, April 25, 2019 (GLOBE NEWSWIRE) -- Sauce Labs Inc., provider of the world’s most comprehensive and trusted continuous testing cloud, today at SauceCon announced the results of its inaugural Continuous Testing Benchmark, a new report that for the first time enables organizations to measure how their continuous testing efforts stack up with those of their peers. Leveraging anonymized data from millions of actual user tests performed on the company’s continuous testing platform, the Sauce Labs Continuous Testing Benchmark identifies the four best practices necessary to achieve continuous testing excellence, and tracks organizations’ collective performance against those benchmarks. The report reveals that room for improvement exists across the entire spectrum of continuous testing excellence, most notably in the areas of test quality and test run time. 

A first-of-its-kind for the continuous testing industry, the report clearly defines the four key pillars of continuous testing, along with the performance benchmark necessary to achieve excellence in each category, including:

• Test Quality
  • Benchmark for Excellence: Pass at least 90% of all tests run 
• Test Run Time
  • Benchmark for Excellence: Test run times averaging 2 minutes or less
• Test Platform Coverage
  • Benchmark for Excellence: Test against at least 5 platforms on average
• Test Concurrency
  • Benchmark for Excellence: Leverage at least 75% of available test capacity during peak testing periods

Real-world data shows need for increased focus on key best practices

The report also reveals how organizations performed against those benchmarks across millions of tests run on the Sauce Labs Continuous Testing Cloud. While the majority of organizations excel in the areas of test platform coverage and test concurrency, as continuous testing matures, more work is needed to ensure excellence in test quality and test run time. Of note:

• Test Quality: Just 18.75% of organizations pass at least 90% of the tests they run
   
• Test Run Time: Only 35.94% of organizations complete their tests in an average of 2 minutes or less
   
• Test Platform Coverage: Nearly two-thirds (62.53%) of organizations test across 5 or more platforms on average
   
• Test Concurrency: A full 70.88% of organizations utilize at least 75% of their available testing capacity during peak testing periods

Collectively, just 6.23% of organizations achieved the benchmark for excellence across all four categories, underscoring the relatively nascent stage of continuous testing as a strategic imperative in the DevOps era.

“Our aim with this report is to provide developers and quality assurance teams with something they’ve never had: a clearly defined set of metrics they can use to measure how they’re performing in their pursuit of continuous testing excellence,” said Charles Ramsey, CEO, Sauce Labs. “As organizations continue to prioritize continuous testing as the foundation of their agile development efforts, we are excited to see how their performance against these benchmarks improves over time, and we look forward to doing our part to help them reach their goals.”

Test implementations at a glance

In addition to providing organizations with a benchmark for achieving continuous testing excellence, the report also takes a closer look at current testing implementations among Sauce Labs users to get a better understanding of the browsers and devices against which organizations are most commonly testing. Top-line results include:

• Nearly two-thirds (65.5%) of all desktop browser tests in the Sauce Labs cloud are run on Google Chrome. Internet Explorer was a distant second at 12.4%.
   
• Among desktop browser tests, organizations in the technology and banking sectors account for 43.1% and 33.9% of all tests, respectively. Retail, media and healthcare round out the top five, in that order, but significantly lag the technology and banking sectors.
   
• Among mobile tests, 77.9% are run on mobile web browsers, compared to just 22.1% on native mobile applications. Whether web or native, more than three-quarters (78.7%) of all mobile tests were run by organizations in the technology sector.

Data and Methodology

The data used in this report comes from the Sauce Labs Continuous Testing Cloud, the largest and most comprehensive testing cloud in the world, with more than 2 billion tests completed to date. This inaugural benchmark report analyses anonymized user data from June 2018 through December 2018.

Download the full report

To download a copy of the full Sauce Labs Continuous Testing Benchmark Report, go here.

About Sauce Labs
Sauce Labs ensures the world’s leading apps and websites work flawlessly on every browser, OS and device. Its award-winning Continuous Testing Cloud provides development and quality teams with instant access to the test coverage, scalability, and analytics they need to rapidly deliver a flawless digital experience. Sauce Labs is a privately held company funded by Toba Capital, Salesforce Ventures, Centerview Capital Technology, IVP, Adams Street Partners and Riverwood Capital. For more information, please visit https://saucelabs.com.

Media Contact
Scott Desiere
scott.desiere@saucelabs.com
(619) 952-3758
Twitter: @ScottDesiere