There are large differences in how companies test and validate their products, and with increasingly better connectivity and computing power and abilities, this disparity may even grow.
Testing and validation processes are a vital part of all steps of the system engineering V-model (a framework for testing and validation, illustrated in Exhibit 1). Typically they account for 20 to 30 percent of development costs. In the preconcept phase, great companies test customer perception of the planned product in order to optimally design the product to customer needs and to improve probability of market success. In the early development phases, virtual tools are used to ensure that the different components fit together, that the product can be built, and that all functional requirements are met. In later stages, all kinds of tests are required, for example, to ensure safety and environmental requirements are met, durability is proved, and so on. Finally, close to production, testing reveals whether the product can be built, quality is demonstrated, and final checks before the start of production are made.
With so much at stake, how does a great company do testing and validation? Which trends do we see, and what are typical pitfalls for less advanced companies?
Three levels of sophistication for testing and validation
In our work with clients, we have seen the following levels of sophistication in validation and testing:
- Function focused. Companies using this approach do not coordinate planning for testing and validation within or across functions. The absence of coordination results in a high number of tests, many of which are redundant. Ad hoc testing and validation generally occur at the end of the development process due to previously limited integration and parallelization. Companies typically conduct physical tests of components or the overall product and make limited use of virtual simulations. Test results often represent an old development stage due to engineering progress during the build-up time of weeks or months, and therefore results have only limited validity.
- Early-stage integrator. Recognizing the inadequacies of the function-focused approach, some companies have improved coordination and planning within and across functional departments and formally integrated testing and validation at each quality gate or milestone. They have established separate testing processes for parts, components, and systems. Integrating test scenarios at gates and milestones has allowed them to reduce the number of physical tests. They have also substituted virtual tests for some physical tests, using a mix of both types. They emphasize quality by testing for the achievement of performance requirements. Although these early-stage integrators test at the level of components and systems, they only integrate at certain milestones in the gate process and only integrate customer feedback at the later development stages. They also take a traditionally narrow perspective by focusing on internal projects and products rather than considering and integrating the testing processes and responsibilities of value chain partners.
- Advanced integrator. Leading companies have moved beyond the traditional perspective on testing and validation. They follow a vision of hardware-free product development that some companies, especially in low-volume industries, have implemented a long while ago due to the high cost of complete product tests. These advanced integrators have a well-defined approach:
- They plan testing and validation scenarios across functions and value chain steps and integrate customer-testing and validation procedures early in the development process.
- They establish a clear split between testing for parts, components, and systems; conduct only the minimum number of physical tests; and take a step-by-step approach to testing components, systems, modules, and final assembly.
- They extensively leverage virtual simulation and make virtual testing a prerequisite for any physical tests.
- They change their product development processes (PDPs) by integrating virtual simulation loops, and they institute strong feedback and improvement loops at each stage of testing.
- Using a modularization approach, they are able to minimize the number of virtual tests and avoid redundant tests.
- They often install centralized responsibility for testing in order to leverage previous results and to optimally manage the number and the utilization of the remaining physical prototypes as well as the testing equipment.
Levers successful companies pull
There are many levers successful companies pull to optimize testing and validation (Exhibit 2). We have summarized them under two big themes: push toward much more virtual testing and validation, and optimized strategy for and use of remaining physical assets, test equipment, and prototypes.
Push toward much more virtual testing and validation
Best-practice examples from different industries show that a significant share of product validation already can be done in an entirely virtual way today. This can be witnessed in the aerospace industry in particular. Dassault Aviation1 launched a business jet, the Falcon 7X, without using physical prototypes. Developers worked on the virtual platform as if it were a physical one and shared a common database with 27 development partners around the world on a nearly real-time basis. The use of a virtual platform allowed the manufacturer to reduce assembly time by 50 percent and tooling costs by 66 percent. It also eliminated assembly problems and the need for physical aircraft prototypes.
To facilitate a prototype-free development, a growing number of virtual tools are available for use in all phases of product development. In the early design phase, software tools like Autodesk can be used to build and visualize virtual 3-D design models. They eventually allow design decisions to be based on them instead of physical clay models. Automotive OEMs further support the decision process by using virtual- and augmented-reality technology. Exterior design sketches can be projected on generic clay models to allow for an appearance close to reality. In the area of interior design, test persons can be equipped with virtual-reality glasses showing a virtual mock-up of the future instrument board and surroundings, thereby enabling, for example, early action on customer feedback on concept alternatives.
In the later product development phase, a range of computer-aided engineering (CAE) tools allow for validating different product characteristics using virtual simulation instead of physical tests. A common use case for CAE tools in automobile development is the simulation of linear resistance, rigidity, and dynamics of different vehicle components (such as body, chassis, and steering system), which is done using finite-element analysis (FEA) software. Additional common applications include the simulation of vehicle kinematics or nonlinear deformation as well as aerodynamic simulation of the exterior surface or simulation of airflow in the engine compartment. Crash and acoustic simulations are two examples of more demanding virtual test cases, which have advanced significantly over the past years.
Apart from hardware components, virtual testing is also used for validation of embedded software using virtual control units that simulate real input and output signals. Finally, the product's feasibility for production and assembly can also be assessed virtually. The AutoForm-StampingAdviser software, for example, allows for the assessment of the construction and process feasibility of sheet-metal parts. Also, production planning can be done virtually based on visualization and simulation of production work flows.
Although the offering of virtual tools in product development has come far, entirely prototype-free development still is a distant vision for most companies in automotive and other industries. On their path toward that vision, they first need to overcome different challenges. Among these are the necessary advancement of existing virtual tools in order to increase their range of functions and, even more, their reliability. Examples of deficits in today's tool landscape include the integral assessment of small signals. In automobile development, a driver's overall impression of a car is based on a holistic assessment of a combination of small signals. However, for example, in acoustics simulation, tools still struggle with facilitating a holistic assessment that matches human perception. The same is true for chassis fine-tuning, which still has to be done manually. As another example, today's computer-aided-design (CAD) programs are rarely capable of correctly calculating cable length, given that cables are woven together and laid around curves in a highly complex pattern in cars, airplanes, and other products. Beyond that, simulation tools today often lack user friendliness as well as the possibility of an integration with other tools or existing IT infrastructure. As a result, there often is no transparency, consistency, and uniform quality of data in the simulation.
We expect that capabilities of virtual tools for simulation will continue to increase in the next years. It is critical for companies to leverage this development in order to succeed on the path toward prototype-free development and especially on the path toward a more feature-oriented development world. With this in mind, companies need to expand current simulation skills and competencies. Moreover, they need to improve the user experience and standardize software tools to increase transparency and consistency in virtual simulation. Beyond that, building trust in virtual methods is key. This is best achieved by using or even developing tools in house and continuously validating simulation results in learning processes.
Apart from technical challenges, there are a range of organizational challenges to be handled to reach prototype-free development. For many companies, the organizational responsibility for virtual simulation is not clearly defined. Virtual testing is rarely anchored rigorously enough in the PDP (Exhibit 3). This requires defining virtual milestones with specified data freezes and consequently linking them to physical product validation to make sure that no prototype is ordered without prior simulation. Laying out a consistent virtual path in the PDP is essential for increasing the importance of and commitment to simulation within the organization and fostering its recognition as key for efficient development work. It also helps to overcome cultural hurdles due to long-established hardware-oriented development practices. Cost pressure can act as another catalyst for virtualization of product development. In this regard, introducing and limiting a shared budget for virtual simulation and physical tests has proven constructive to shift focus on virtual tests. Similar effects have been successfully achieved by introducing strict control mechanisms for approving physical prototypes and KPI systems to monitor the use of virtual methods versus physical prototypes.
Pursuing the vision of prototype-free development using the described levers enables faster testing while also reducing costs related to quality issues. Virtualization enables greater flexibility, the reuse of simulations, and the earlier testing of a large number of concepts. However, since simulations do not fully capture the complexity of the physical world, there is risk that in some simulation areas some errors will not be identified. Therefore, companies must stringently assess the need for simulation and the risk profile of each potential application (Exhibit 4). In cases where simulations are highly reliable, companies can capture significant savings by eliminating physical tests.
Optimized strategy for remaining physical assets
In addition to the push for virtualization, we synthesized the best practices we have seen in different advanced integrators into a set of levers that companies should pull in order to improve their testing procedures (Exhibit 5). We will describe two levers in more detail.
Central management of testing. From our experience, central management of the testing equipment as well as the different stages of hardware prototypes is very important. Historically, most departments in the engineering organization were allowed to order their own hardware for their testing needs, with the effect that the equipment utilization of these prototypes and testing facilities was very low. Central management of these assets enables much better utilization and planning for assets and therefore much lower costs.
However, we are also convinced that test assets should become bottlenecks on the critical path of product development. A smart project planning and—if needed—leverage of third-party test facilities can efficiently deliver on-time test results.
Testing/prototyping strategy
In a world of growing complexity as well as increasing modularity, on a path to more feature-based development, it is of growing importance to define what needs to be tested when and by whom in order to ensure optimal and cost-effective product validation.
Often, testing at the lowest level of products (modules and components) is more cost efficient than testing at the highest levels (systems and assembled products) and allows for an early assessment of maturity. As much testing as possible should be conducted at the level of modules and components. We expect that the technical means for lowest-level testing will continue to improve and that the advancement of big data analytics in product development as well as quality will allow for a much more robust design at early stages in the PDP.
Since most of the components and modules are delivered through suppliers, manufacturers need to build strong networks and value chains with suppliers that are continuously improving their testing and validation capabilities. In the automotive industry, for example, some Tier 2 and Tier 3 suppliers now serve as developers, Tier 1 suppliers are module integrators, and OEMs are vehicle integrators. To offer the required capabilities, suppliers need to improve their own testing and validation processes, increase their knowledge, and build networks that enable them to provide required services.
What is at stake?
By adopting a set of practices to improve testing and validation processes, manufacturers can capture a wide variety of benefits:
- Testing and validation cost. Most obvious is the effect on the direct cost of testing and validation. Since these costs sum up to 20 to 30 percent of development cost, there is a significant impact through improving these processes.
- Product quality and costs. More important, improved testing and validation is able to reduce late changes caused by quality or performance issues or the need to integrate additional features. Addressing such issues late in the development process is especially expensive (Exhibit 6). Identifying and remedying quality issues before launch also allows companies to reduce costs related to poor quality during a product's lifetime.
- Market success. By applying insights from testing and validation, companies can reduce rework and improve production ramp-up and thereby accelerate time to market. They can also enhance their abilities to integrate features that target specific customer needs early in the design process and improve customers' perception of quality. The result is greater customer satisfaction and higher market share (Exhibit 7). High quality is directly linked to increased revenue from repurchases and upgrades, new business gained by word of mouth, and lower costs to serve customers.
Companies seeking to become advanced integrators need to assess their opportunities to broaden their perspective on testing and validation and promote standardization, virtualization, and cross-functional planning. Questions to consider include:
- How can we gradually increase our share of virtual testing, ultimately leading to prototype-free development?
- How can we best anchor virtual testing in our organization and our formal product development process?
- How can we integrate our customers and value chain partners into the testing and validation process?
- What are the opportunities for standardizing and improving our procedures internally as well as with our suppliers?
- What steps can we take to break our functional silos?
- Can we parallelize and shorten our testing and validation cycles?
Companies that succeed in reaching the highest maturity stage of testing and validation will be rewarded with successful products that are developed and produced at optimal cost.
About the authors: Tobias Geisbüsch is a senior expert in McKinsey’s Munich office, Carolin Heck is a consultant in the Cologne office, Ruth Heuss is a partner in the Berlin office, Jonas Reinemann is a consultant, and Thorsten Schleyer is an associate partner in the Munich office.