Debugging requires a multifaceted solution

PROLOGUE: Over the weekend, I was thinking about a recent visit I had with an advanced ASIC team manager who told me that they had optimized most aspects of their verification flow to such an extent that most of their remaining effort was spent in debugging. So, I decided to work up a draft blog on debugging. However, this morning, when I was preparing to post my blog, I noticed that Richard Goering had beat me to the punch and had posted a blog on debugging about two weeks ago. Having reviewed his blog, I think we are both in agreement—debugging is a huge bottleneck in the flow. I think that debugging must be looked at as a solution, and not a tool feature. However, there are many aspects of debugging beyond traditional simulation triage of design models and testbench components—ranging from embedded software, to power and performance analysis, to code and functional coverage closure, etc. There really isn’t a unified solution—debugging must be considered an integral part of each aspect of design and verification.

ACT 1:  My original blog from this weekend….

“Bloody instructions, which, being taught, return to plague the inventor….”

William Shakespeare, Macbeth, act 1, scene 7

All right, even Shakespeare had issues with debugging. But before I get into all of that, let me set the stage with a little background info…

First, let me say that I love my job. My role at Mentor Graphics consists of a diverse set of tasks. Yet, probably my most rewarding work involves studying and assessing today’s electronics industry. The objective of this work is to help Mentor identify discontinuities in today’s EDA solutions, as well as understand emerging verification challenges. But what I like most about my work is that it allows me to participate in detailed discussions with various project teams and multiple industry thought leaders across multiple market segments.

A couple of years ago, I was performing a detailed verification assessment for an ASIC project team. As I usually do when I conduct these kinds of assessments, I asked the team what was the biggest bottleneck in their flow. This one enthusiastic, young engineer started waving his hand vigorously at me and said: “I know, I know…..it’s layoffs!” Okay, so after the group recovered itself from an outburst of nervous chuckles, I pressed forward with my question. It turned out that the group unanimously agreed that debugging was generally a significant, yet often underestimated, effort associated with their flow. Perhaps this shouldn’t surprise anyone, when you consider that the Collett International 2003 IC/ASIC Design Closure study found that 42 percent of the verification effort was consumed in writing test and creating testbenches, while 58 percent was consumed in debugging. More recently, a 2007 Farwest Research study, chartered by Mentor Graphics, found that 52 percent of a dedicated verification engineers effort was consumed in debugging.

The problem with debugging is that the effort is not always obvious since it applies to all aspects of the design and verification flow and often involves many different stakeholders. For example, architectural modeling, RTL coding, testbench implementation, transaction modeling, embedded software, coverage modeling and closure, and on and on and on. What makes it particularly insidious is that it is extremely difficult to predict or schedule. In fact, what you will find is that a mature organization relies on historical data extracted from their previous project’s debugging effort metrics in order to estimate their future project effort. However, due to the unpredictable nature of debugging, history doesn’t always repeat itself. And unfortunately, there is no silver bullet in terms of a single debugging tool or strategy. Multiple solutions, ranging from RTL implementation debugging, to OVM object-oriented testbench component debugging, to embedded software debugging capabilities, to coverage closure are required. Fortunately, multiple good solutions have emerged, ranging from assertions for reducing RTL debugging effort, to SystemVerilog dynamic structures analysis and debugging, to processor-driven verification debugging solutions for embedded software verification, to the intelligent testbench for automating coverage closure.

EPILOGUE: I opened this blog humorously with a quote from Shakespeare. Yet, today’s debugging effort is no laughing matter, and it contributes significantly to a project’s overall design and verification effort. I’ll conclude this blog with a sobering quote from Brian Kernighan (the K in the K&R C language) who once pointed out:

Debugging is twice as hard as writing the code in the first place.

I’m curious about your thoughts. Does debugging consume a significant amount of effort in your flow? If not, what is the biggest bottleneck in your flow?