{"id":12393,"date":"2016-09-07T08:07:03","date_gmt":"2016-09-07T15:07:03","guid":{"rendered":"https:\/\/blogs.mentor.com\/verificationhorizons\/?p=12393"},"modified":"2026-03-27T08:37:41","modified_gmt":"2026-03-27T12:37:41","slug":"part-4-the-2016-wilson-research-group-functional-verification-study","status":"publish","type":"post","link":"https:\/\/blogs.sw.siemens.com\/verificationhorizons\/2016\/09\/07\/part-4-the-2016-wilson-research-group-functional-verification-study\/","title":{"rendered":"Part 4: The 2016 Wilson Research Group Functional Verification Study"},"content":{"rendered":"

FPGA Verification Effectiveness Trends<\/strong><\/h2>\n

This blog is a continuation of a series of blogs related to the 2016 Wilson Research Group Functional Verification Study (click here<\/a>). \u00a0In my previous blog (click here<\/a>), I focused on the amount of effort spent in FPGA verification. We have seen in previous blogs that an increasing amount of effort is being applied to FPGA functional verification. In this blog I focus on the effectiveness of verification in terms of FPGA project schedule and bug escapes.<\/p>\n

FPGA Schedules<\/strong><\/h3>\n

Figure 1 presents the design completion time compared to the project\u2019s original schedule. What was a surprise in the 2014 findings is that we saw an improvement in the number of FPGA projects meeting schedule\u2014compared to 2012 and 2016. This could have been an anomaly or sampling error associated with this question in 2014, as I discussed when describing confidence intervals in the prologue<\/a>. However, we saw similar results when looking only at ASIC\/IC designs in 2016 (which will be published in a future blog). At any rate, it does raise an interesting question that might be worth investigating.<\/p>\n

\"BLOG-2016-WRG-figure-4-1\"<\/p>\n

Figure 1. FPGA design completion time compared to the project\u2019s original schedule<\/strong><\/p>\n

FPGA Lab Iterations<\/strong><\/h3>\n

ASIC\/IC projects track the number of required spins that occur prior to market production.\u00a0 In fact, this can be a useful metric for determining the overall verification effectiveness of an ASIC\/IC project.\u00a0 Unfortunately, we lack such a metric for FPGA projects.\u00a0 For the 2014 study, we decided to ask the question related to the average number of lab iterations required before the design went into production, and asked the same question again in 2016. The results are shown in Figure 2.<\/p>\n

\"BLOG-2016-WRG-figure-4-2\"<\/p>\n

Figure 2. Number of FPGA iterations in the lab<\/strong><\/p>\n

FPGA Non-Trivial Bug Escapes to Production<\/strong><\/h3>\n

I have never been convinced that lab iterations is analogous to ASIC\/IC respin as a verification effectiveness metric.\u00a0 So for our 2016 study we decided to also ask the question concerning the number of non-trivial bugs that escape into production and are found in the field. The results were surprising and are presented in Figure 3. Only 22 percent of today\u2019s FPGA design projects are able to produce designs without a non-trivial bug escaping into the final product. For many market segments (such as safety critical designs) the cost of upgrading the FPGA in the field can be huge since this requires a complete revalidation of the system.<\/p>\n

\"BLOG-2016-WRG-figure-4-3\"<\/p>\n

Figure 3. Number of FPGA iterations in the lab (no trend data available)<\/strong><\/p>\n

FPGA Bug classification<\/strong><\/h3>\n

For the 2016 study, we asked the FPGA project participants to identify the type of flaws that were contributing to rework. In Figure 4, I show the two leading causes of rework, which are logical and functional bugs, as well as clocking bugs. The data seems to suggest that these issues are growing. Perhaps due to the design of larger and more complex FPGAs.<\/p>\n

\"BLOG-2016-WRG-figure-4-4\"<\/p>\n

Figure 4. Types of Flaws Resulting in FPGA Rework<\/strong><\/p>\n

In Figure 5, I show trends in terms of main contributing factors leading to logic\u00a0and functional flaws\u2014and you can see that design errors are the main cause of functional flaws.\u00a0 But note that a significant amount of flaws are related to some aspect of the specification\u2014such as changes in the specification\u2014or incorrect or incomplete specifications. Problems associated with the specification process are a common theme I often hear when visiting FPGA customers.<\/p>\n

\"BLOG-2016-WRG-figure-4-5\"<\/p>\n

Figure 5. Root cause of FPGA functional flaws<\/strong><\/p>\n

In my next blog (click here<\/a>), I plan on presenting the findings from our study for FPGA verification technology adoption trends.<\/p>\n

Quick links to the 2016 Wilson Research Group Study results<\/strong><\/p>\n