Test Like Google: Best Practices from Industry Giants

We can learn a lot from industry giants by observing their methods and approaches in workflows.

How do they manage to maintain the software testing process within millions of code lines?

How do they organize workflow with thousands of QA engineers?

How do they cope with the project expansion?

Experience of companies like Google or Spotify can help improve testing processes in small companies and teams.

Let’s take a closer look at how giants test.

Google: the importance of code coverage

Still, many engineers argue about the importance of such a measure of software testing efficiency as code coverage. Specialists from Google insist that code coverage data can be valuable information for evaluating risks and bottlenecks in a testing activity. Carlos Arguelles, Marko Ivanković‎, and Adam Bender share best practices for code coverage:

• Code coverage can help to reduce bugs and failures. Experience of QA-engineers from Google showed that increasing code coverage leads to changes in approaches and attitudes to testing. Teams with code coverage as primary goals tend to make better testability of their products. They write more efficient code for testing to achieve testing goals easier and less time-consuming.
• Use mutation testing to ensure high test coverage. Complete code coverage can be wasteful and doesn’t guarantee high-quality test coverage. A high percentage of code coverage doesn’t mean that all functions were tested correctly. It means that the code was overall tested. To ensure high quality of test coverage, Google experts recommend using mutation testing. This method includes implementing small code changes and checking how the test sets identify them.
• The percentage of code coverage depends on many factors. We shouldn’t aim for high code coverage, but low coverage leads to a large number of failures as well. And the question is, what is specific code coverage for a particular system? To answer this question, we should consider such things as criticality, complexity, and frequency of changing code. Code coverage is a business decision, and product owners should define it.
• Analyze what to cover. We can’t get 100% code coverage, so QA-engineers should observe that the most valuable parts of code are covered. A developer team should discuss and think not about how many code lines are covered, but what exactly is covered.

Spotify: reliable strategy

Spotify is a fast-growing company. It was established in 2008 with 150 employees. In 2019 the number of employees grew up to 4 405. Today Spotify serves about 300 million users worldwide, making it the most popular streaming service in the world.

We can learn lessons about how Spotify scales its testing and development processes.

• Set product goals for QA-team. Spotify has cross-functional teams. Every team has goals and a set of particular skills to approach these goals. The team structure depends on goals. Hence, some teams consist only of developers and some have developers and testers. Thus, testers cooperate closely with the development team and focus on the primary goals of the product. Such an approach allows Spotify to scale the development process efficiently.
• Automation testing is a tool, but not a panacea. Test and Development Manager in Spotify Kristian Karl claims that software testers can’t be replaced by automation. Testing requires human experience and knowledge. Automation testing is a robust tool to speed up the process, but humans make decisions and analyze automated reports. Spotify uses automation as one of the scale tools. It allows testers to leave routine for algorithms and focus on product goals.

SpaceX: continuous testing is a must

SpaceX is an innovative company known for commercial space transportation, a reusable launch system, and high efficiency.

Nicholas Chaillan, chief software officer for the Air Force, said that the SpaceX development department system is five times more efficient than companies with classic workflow. How does SpaceX cope with the testing process with such performance?

DevOps and the Agile approaches allow SpaceX QA-engineers to provide mass automation testing. Through the development cycle, testers provide continuous testing to get instant feedback and eliminate existing risks. Thereby they are working proactively, testing often and early.

The company developed a strategy on how to implement continuous testing in the development process.

1. Prioritize value. Like Google, SpaceX recommends using code coverage metrics to understand what you should automate and what shouldn’t. It helps to optimize continuous testing and enhance activities that are already taken.
2. The key is automated end-to-end testing. Make an impact analysis part of continuous integration. It allows companies to analyze how adding new functions or changing code can affect an overall system or some parts of its system.
3. The team should have a stable and easily replicated test environment. A tool like Virtual Machine Snapshot can help save data state, return to testing, or resume work.
4. Use Artificial Intelligence and Machine learning to analyze testing reports. AI-based tools speed up deployment and optimize the testing process.
5. Build a robust continuous integration architecture. The main advantage of the CI system is short periods between build and tests of the code. Continuous testing must be involved in the development process and involve necessary types of testing.

Sum Up

Every company has its testing strategy, process, and approaches. Each team selects and adjusts the testing process based on the ultimate goals and product capabilities. The main task of any specialist is to think about the client and the end-user, adapt to new requirements, and adopt the experience of the best in the industry.