Community post by Saqib Jan
As technologies become more advanced year on year, the complexity of software testing increases, too.
When building a testing strategy, companies typically map their operations into three segments: the people, the process, and the technology involved. And the hardest challenges companies face are often around the technology itself, like dealing with complex test environments that have various frameworks and APIs at different levels.
It’s exceedingly important to create a realistic test environment with the right APIs to test how everything works together from start to finish. But managing a large testing infrastructure is a major hurdle for all companies – startups and enterprises alike.
This is because every month dozens of mobile devices are launched by various vendors, and there’s also the launch of operating systems. And so, it means that your potential customers could land on your website or app from anywhere, and buying all the devices for testing with all the different operating systems is not economical which makes testing inordinately hard.
But engineering leaders see in TestOps an opportunity to deliver high-quality software at an accelerated pace. “By taking a TestOps approach to software testing, organizations can determine the most cost-effective strategies in the long term—removing blockers, empowering development teams to deliver better features and apps, and enabling cross-functional conversations about high availability testing practices,” says Mayank Bhola CTO of a software testing platform LambdaTest.
Tackling Complexity With TestOps
You want a testing environment that is not limited by the devices, operating system, or programming language, and can quickly allow you to create real-time machines with the exact testing environment you need. But building and maintaining this kind of infrastructure requires significant resources and bandwidth.
You could build your own in-house device lab, but the reality is that if you buy ten devices today, a few months later, ten more will hit the market. You’ll then need to buy and maintain these new devices, along with your older ones. And even then your ambitious IT requirements will inevitably have to grapple with scaling problems, let alone changes to tooling and processes.
Shay Elmualem, Principal Tech Lead at Legit Security, shares, “To prioritize test coverage, we first focus on the most commonly used browsers among our users—typically Chrome and Firefox. We use a matrix setup in GitHub Actions to run our tests across these different browsers simultaneously, which helps us cover multiple platforms in parallel without sacrificing speed.” This systematic approach helps ensure consistent cross-browser compatibility and is by practice crucial for user experience.
Because every company with a growing product portfolio might find it increasingly difficult to keep up with the latest devices and operating systems while ensuring consistent testing across different teams. It’s very exhaustive and when you try to automate at scale, things usually fail.
Bhola, in our email conversation, discussed how LambdaTest empowers software testing with highly available instant infrastructure by dynamically updating testing capabilities based on real-time metrics and usage patterns. “We consistently monitor device, platform, browser, and version usage in production through a monitoring tool over a rolling window of 30 days. And based on this data, we dynamically update our testing capabilities to align with customer needs. This involves prioritizing test cases that target the most widely used configurations and eliminating outdated device configurations that are no longer in demand.”
This ensures that testers have access to the most relevant testing environments. Bhola further enthused, “We also analyze the usage patterns of commands to identify the most frequently executed commands by customers and ensure that our test suite includes relevant test cases to cover these scenarios comprehensively.”
You Don’t Need To Test Everything!
To ensure testing produces high availability and high user satisfaction, targeted testing and leveraging customer telemetry are growing trends for achieving sustainable app performance. It’s not about testing everything, but instead focusing on the areas that often have the most significant impact according to Jeff Friedman, Director of Engineering at RAD Security.
By focusing on the most critical areas of the application, teams can optimize their testing efforts and ensure that the highest-value features are delivered efficiently. “We can maintain high velocity in the dev environment with targeted tests, to ensure feature functionality, while relying on more stable, higher-level tests in the CI and pre-production environments to ensure that system-level functionality isn’t impacted by code changes. We use customer telemetry and the Pareto Principle (80-20 rule) to focus development and testing efforts on the highest-payoff configurations, using CI compatibility matrix testing to guide fast-follow compatibility fixes,” Friedman elaborated in an email interview.
“Our CI environment can test across multiple cloud and Kubernetes configurations for backend services, and different browser types and form factors for the frontend,” Friedman explained. “We standardize our testing process with uniform open source telemetry tooling and testing platforms to manage a standard set of CI checks as well.” This helps ensure a high quality level across the board, as well as a high level of consistency across applications.
Prioritizing environments (devices, platforms, browsers) to focus on has remained unchanged for decades. “This is an area where I see a missed opportunity,” says Kohsuke Kawaguchi, famous for creating Jenkins, the open-source automation server. “The abundance of data,” he exposits, “combined with machine learning, should enable us to make much more intelligent, fine-grained decisions about which tests on which environments are worthwhile.”
“Infrastructure as code, device clouds, and containers are great—until they’re not. These approaches often scale testing without improving efficiency. In other words, you’re just burning money,” Kawaguchi pointed out in an email interview. “To truly enhance efficiency, teams need to do more with the same budget. Focus on selecting tests that matter and use AI to screen test failures. You can spend 10 times the money to run 10 times more tests, but if the people processing the results can’t handle 10 times the input, it’s not effective.”
TestOps is the Way Forward to Enhance Quality with Speed
Everyone wants a highly available testing environment. Speed is critical, not just for rapid development and efficient deployment, but also for testing. Realistically, speed can be both the solution and the problem when developing higher-quality software. So, you need to make sure you’re not chasing speed and exhausting resources to the point where you unintentionally compromise on quality.
Matthew Jones, a distinguished engineer and Chief Ansible Architect at Red Hat, shares his experience in managing software delivery. “Because of the nature of how we deliver software, we work really hard to control our requirements and dependencies so that we can limit our test matrix. We specifically normalize all of our deployment types so that they look and act the same no matter where we’re deploying them.”
By emphasizing the need to control requirements, dependencies, and deployment types, Jones underscores how a streamlined approach can significantly reduce the complexity of the testing process. “This simplification,” he affirms, “allows organizations to trust their layered testing with fewer artifacts, leading to faster development and deployment cycles.”
TestOps is the most economical and best way to effectively productionize testing. “It is emerging as a critical requirement in cross-functional teams because it offers businesses an efficient way to manage testing and operations throughout their software development life cycle,” Bhola remarks. “You can solve for managing infrastructure complexities while keeping up with evolving technologies to better meet performance expectations and offer more value to meet complex user needs.”
Most interestingly, to this end, more businesses today are adopting modern TestOps methodologies and leveraging the benefits of cloud-based testing platforms that provide instant high-performance infrastructure to ensure consistency across multiple test environments—staging, QA, and production—and accelerate the delivery of high-quality, reliable applications with more efficient release cycles.
Author: Saqib Jan
Email: sakimjan8@gmail.com
LinkedIn: https://linkedin.com/in/s-jan
BIO: Saqib Jan is a technology analyst with experience in application development, FinOps, and cloud technologies.