Experiences with Test Driven Development (TDD) for logic (chip) design in Verilog or VHDL

Question

I have looked on the web and the discussions/examples appear to be for traditional software development. Since Verilog and VHDL (used for chip design, e.g. FPGAs and ASICs) are similar to software development C and C++ it would appear to make sense. However they have some differences being fundamentally parallel and requiring hardware to fully tests.

What experiences, good and bad, have you had? Any links you can suggest on this specific application?

Edits/clarifications: 10/28/09: I'm particularly asking about TDD. I'm familiar with doing test benches, including self-checking ones. I'm also aware that SystemVerilog has some particular features for test benches.

10/28/09: The questions implied include 1) writing a test for any functionality, never using waveforms for simulation and 2) writing test/testbenches first.

11/29/09: In Empirical Studies Show Test Driven Development Improves Quality they report for (software) TDD "The pre-release defect density of the four products, measured as defects per thousand lines of code, decreased between 40% and 90% relative to the projects that did not use TDD. The teams' management reported subjectively a 15–35% increase in initial development time for the teams using TDD, though the teams agreed that this was offset by reduced maintenance costs." The reduced bugs reduces risk for tape-out, at the expense of moderate schedule impact. This also has some data.

11/29/09: I'm mainly doing control and datapath code, not DSP code. For DSP, the typical solution involves a Matlab bit-accurate simulation.

03/02/10: The advantage of TDD is you make sure the test fails first. I suppose this could be done with assertions too.

Answer 1

I write code for FPGAs, not ASICS... but TDD is my still my preferred approach. I like to have a full suite of tests for all the functional code I write, and I try (not always successfully) to write testcode first. Staring at waveforms always happens at some point when you're debugging, but it's not a good way of validating your code (IMHO).

Given the difficulty of performing proper tests in the real hardware (stimulating corner cases is particularly hard) and the fact that a VHDL-compile takes seconds (vs a "to hardware" compile that takes many minutes (or even hours)), I don't see how anyone can operate any other way!

I also build assertions into the RTL as I write it to catch things I know shouldn't ever happen. Apparantly this is seen as a bit "weird", as there's a perception that verification engineers write assertions and RTL designers don't. But mostly I'm my own verification engineer, so maybe that's why!

Answer 2

I use VUnit for test driven development with VHDL.

VUnit is a Python library that invokes the VHDL compiler and simulator and reads the results of the simulation. It also provides several nice VHDL libraries that makes it a lot easier to write better test benches, such as a communication library, logging library and a checking library.

There are many possibilities since it is invoked from Python. It is possible to both generate test data, as well as check the output data from the test in Python. I saw this example the other day where they used Octave - a Matlab copy - for plotting test results.

VUnit seems very active and I have several times been able to actually ask questions directly to the developers and gotten help quite quickly.

A downside is that it is harder to debug compilation errors since there are so many function/procedure variations with the same name in the libraries. Also, some stuff is done behind the scene by preprocessing the code, which means that some errors might show up in unexpected places.

Answer 3

I don't know a lot about hardware/chip design, but I am deeply into TDD, so I can at least discuss suitability of the process with you.

The question I'd call most pertinent is: How quickly can your tests give you feedback on a design? Related to that: How quickly can you add new tests? And how well do your tools support refactoring (changing structure without changing behavior) of your design?

The TDD process depends a great deal on the "softness" of software - good automated unit tests run in seconds (minutes at the outside), and guide short bursts of focused construction and refactoring. Do your tools support this kind of workflow - rapidly cycling between writing and running tests and building the system under test in short iterations?

Answer 4

The SystemVerilog extensions to the IEEE Verilog Standard include a variety of constructs which facilitate creating thorough test suites for verifying complex digital logic designs. SystemVerilog is one of the Hardware Verification Languages (HVL) which is used to verify ASIC chip designs via simulation (as opposed to emulation or using FPGA's).

Significant benefits over a traditional Hardware Design Language (Verilog) are:

• constrained randomization
• assertions
• automatic collection of functional and assertion coverage data

The key is to have access to simulation software which supports this recent (2005) standard. Not all simulators fully support the more advanced features.

In addition to the IEEE standard, there is an open-source SystemVerilog library of verification components available from VMM Central (http://www.vmmcentral.com). It provides a reasonable framework for creating a test environment.

SystemVerilog is not the only HVL,and VMM is not the only library. But, I would recommend both, if you have access to the appropriate tools. I have found this to be an effective methodology in finding design bugs before becoming silicon.

Answer 5

With regard to refactoring tools for hardware languages, I'd like to point you to our tool Sigasi HDT. Sigasi provides an IDE with built-in VHDL analyzer and VHDL refactorings.

Philippe Faes, Sigasi

Answer 6

ANVIL– ANother Verilog Interaction Layer talks about this some. I haven't tried it.

Answer 7

I never actively tried TDD on an RTL design, but I had my thoughts on this.

What I think would be interesting is to try out this approach in connection with assertions. You would basically first write down in form of assertions what you assume/expect from your module, write your RTL and later you can verify these assertions using formal tools and/or simulation. In contrast to "normal" testcases (where you probably would need to write directed ones) you should have much better coverage and the assertions/assumptions may be of use later (e.g. on system level) as well.

However I wouldn't fully rely on assertions, this can become very hairy.

Maybe you can express your thoughts on this as well, as you are asking for it I guess you carry some ideas in your head?

Answer 8

What is TDD for you? Do you mean having all your code exercised by automatic tests at all times, or do you go further to mean that tests are written before the code and no new code is written unless tests fail?

Whichever approach you prefer, HDL code testing isn't very different from software testing. It has its pluses (much better coverage and depth of testing) and minuses (difficult to set up and cumbersome relatively to software).

I've had very good experience with employing Python and generic HDL transactors for implementing comprehensive and automatic tests for synthesizable HDL modules. The idea is somewhat similar to what Janick Bergeron presents in his books, but instead of SystemVerilog, Python is used to (1) generate VHDL code from test scenarios written in Python and (2) verification of results written by the monitoring transactors that accept waveforms from the design during simulation.

There's much more to be written about this technique, but I'm not sure what you want to focus on.