Conclusion: The 2018 Wilson Research Group Functional Verification Study

Deeper Dive into Non-Trivial Bug Escapes and Safety Critical Designs

This blog is a continuation of a series of blogs related to the 2018 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I presented verification results findings in terms of schedules, number of required spins, and classification of functional bugs. In this blog, I conclude the series by having a little fun with some of the findings from our recent study.

You might recall from our 2014 study we did a deeper dive into the findings made a non-intuitive observation related to design size and first silicon success. That is, the smaller the design the less likelihood of achieving first silicon success (see 2014 conclusion blog for details). This observation still held true in 2016 as well as this year’s study. This is likely due to projects working on smaller designs typically are less mature in their verification processes, as well as a higher percentage of designs with analog.

For this year’s study, we decided to do a deeper dive related to the following:

1. Verification maturity and non-trivial bug escapes into production, and
2. Safety critical designs and silicon success.

Verification Maturity and Silicon Success

The study results show that the IC/ASIC market has matured its verification processes overtime, and we see the FPGA market also starting to mature its processes. This maturity is likely due to the growing complexity of designs and related efforts to control cost and engineering headcount through the adoption of FPGA design and verification solutions that increase productivity.

Perhaps the most concerning finding from this year’s study relates to the number of FPGA projects with non-trivial bug escapes into production. However, we did find an interesting correlation between the improvement of reduced functional flaws contributing to non-trivial bug escapes and the maturing of FPGA projects’ functional verification processes. The data suggest that projects that are more mature in their functional verification processes will likely experience fewer bug escapes.

To test this claim, we partitioned the study participants into two separate groups: FPGA projects with no bug escapes and FPGA projects that experienced a bug escape. We then examined the percentage adoption of various verification techniques. The results are shown in Fig. Conclusion-1. You might note that the  various verification techniques related to bug escapes and no bug escapes does not sum to 100 percent.  The reason for this is that we performed our analysis in terms of technique adoption independently on the separate groups: bug escapes and no bug escapes.

Figure Conclusion-1. Process Maturity and Non-Trivial Bug Escapes

These findings are statistically significant in that the group with no bug escapes tended to have higher adoption of various verification techniques, which suggest they are more mature in their verification process. However, what we are unable to measure from our study is how effective a project was in adopting any of these processes. For example, a project that experienced a bug escape could claim that they have adopted functional coverage, yet the fidelity of their functional coverage model might be poor due to their inexperience. From our study data, we are unable to assess successful or effective adoption for any particular verification technique.

Safety Critical Designs and Silicon Success

The second aspect of our 2018 study that we decided to examine a little deeper relates to safety critical designs and their silicon success. Intuitively, one might think that the rigid and structured process required to adhere to one of the safety critical development processes (such as, DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601 for medical, etc.) would yield higher quality in terms of preventing bugs and achieving silicon success.

Figure Conclusion-2 shows the percentage of FPGA projects that claimed to be working on a safety critical design.

Figure Conclusion-2. Percentage of FPGA projects working on safety critical designs

We partitioned the findings into two groups: projects working on non-safety critical and safety critical designs. Then in Figure Conclusion-3 we examined the number percent of bug escapes and no bug escapes for projects working on safety critical designs.

Figure Conclusion-3. Non-trivial bug escapes for safety critical vs. non-safety critical FPGA designs

Clearly, the data suggest that a development process adopted to ensure safety does not necessarily ensure quality. Perhaps this is non-intuitive. However, to be fair, many of the safety critical features implemented in today’s designs are quite complex and increase the verification burden.

This concludes the findings from the 2018 Wilson Research Group Study.

Quick links to the 2018 Wilson Research Group Study results

• Prologue: The 2018 Wilson Research Group Functional Verification Study
• Understanding and Minimizing Study Bias (2018 Study)
• Part 1 – FPGA Design Trends
• Part 2 – FPGA Verification Effectiveness Trends
• Part 3 – FPGA Verification Effort Trends
• Part 4 – FPGA Verification Effort Trends (Continued)
• Part 5 – FPGA Verification Technology Adoption Trends
• Part 6 – FPGA Verification Language and Library Adoption Trends
• Part 7 – IC/ASIC Design Trends
• Part 8 – IC/ASIC Resource Trends
• Part 9 – IC/ASIC Verification Technology Adoption Trends
• Part 10 – IC/ASIC Language and Library Adoption Trends
• Part 11 – IC/ASIC Power Management Trends
• Part 12 – IC/ASIC Verification Results Trends
• Conclusion: The 2018 Wilson Research Group Functional