Thought Leadership

Part 2: The 2020 Wilson Research Group Functional Verification Study

November 10, 2020 • 3 MIN READ
By Harry Foster

In my previous blog, I introduced the 2020 Wilson Research Group Functional Verification Study (click here). The objective of my previous blog was to provide an overview on our large, worldwide industry study. The key findings from this study will be presented in a set of upcoming blogs.

In this blog, I present the findings from our new study related how successful FPGA projects are in terms of verification effectiveness.

FPGA Verification Effectiveness

Non-Trivial Bug Escapes

IC/ASIC projects have often used the metric “number of required spins before production” as a benchmark to assess a project’s verification effectiveness. Historically, about 30 percent of IC/ASIC projects are able to achieve first silicon success, and most successful designs are productized on the second silicon spin. Unfortunately, FPGA projects have no equivalent metric. As an alternative to IC/ASIC spins, our study asked the FPGA participants “how many non-trivial bugs escaped into production?” The results shown in fig. 2-1 are somewhat disturbing. In 2020, only 17 percent of all FPGA projects were able to achieve no bug escapes into production, which is worse than IC/ASIC in terms of first silicon success, and for some market segments, the cost of field repair can be significant. For example, in the mil-aero market, once a cover has been removed on a system to upgrade the FPGA, the entire system needs to be revalidated.

Fig. 2-1. Non-trivial FPGA bug escapes into production.

Types of Flaws Resulting in Non-Trivial Bug Escapes

Fig. 2-2 shows various categories of design flaws contributing to FPGA non-trivial bug escapes. The percentage of “logic or functional flaws” remains the leading cause of bugs. New flaws being tracked in the 2020 study are associated with safety (8 percent) and security (6 percent) features. Obviously multiple flaws can contribute to bug escapes, which is the reason the total percentage of flaws sums to more than 100 percent.

Fig. 2-2. Types of flaws resulting in FPGA bug escapes.

Fig. 2-3 demonstrates the root cause of logical or functional flaws by various categories. The data suggest design errors are the leading cause of functional flaws, and the situation is worsening. In addition, problems associated with changing, incorrect, and incomplete specifications are a common theme often voiced by many verification engineers and project managers.

Fig. 2-3. Root cause of FPGA functional flaws.

Design Completion Compared to Original Schedule

In addition to bug escape metrics that we used to determine an FPGA project’s effectiveness, another metric we tracked was project completion compared to the original schedule, as shown in fig. 2-4. Here we found that 68 percent of FPGA projects were behind schedule. One indication of growing design and verification complexity is reflected in the increasing number of FPGA projects missing schedule by more than 50 percent during the period 2012 through 2020.

Fig. 2-4. Actual FPGA project completion compared to original schedule.

In my next blog, I’ll focus on verification effort trends related to FPGA designs.

Quick links to the 2020 Wilson Research Group Study results

What to read next:

Comments

One thought about “Part 2: The 2020 Wilson Research Group Functional Verification Study”

Leave a Reply

This article first appeared on the Siemens Digital Industries Software blog at https://blogs.sw.siemens.com/verificationhorizons/2020/11/10/part-2-the-2020-wilson-research-group-functional-verification-study/