SystemVerilog Guru Cliff Cummings is back in town and he will be holding another seminar on April 19th at the MCCIA auditorium on Senapati Bapat Road from 4:00pm to 7:30pm. Most recently, Cliff was here in November 2009 and this seminar gives a great opportunity for engineers to  re-engage with him. This event is completely free, but registration is required. Please visit this link to register and view the agenda.

 This event is co-sponsored by Qlogic and Cadence who I must thank profusely on behalf of the PuneChips community. It is not very often that internationally renowned experts visit our city and hold free seminars, but QLogic and Cadence have made it possible. So, I encourage everyone who has any interest in SystemVerilog to attend and participate.

photo credit: oskay

This is the second in the blog series titled Field Manual for Verification Planning written for PuneChips by Suhas Belgal . The blogs deal with functional verification of digital ICs and cover mostly the pre-silicon verification phase.  

Welcome to the second article in the Chip Design Verification blog series. In this article, we will look at the Test-plan development part of the verification program. We are going to explore the method to the madness of developing effective test-plans.  

Some of the questions that come to the mind are: how do we know if the test-plan is complete? How do we map the test-plan to the ‘tests’? How do we ensure coherency between the test-plan and the test data base throughout the project (and beyond)? What’s a good test-plan template? How should the cases be organized? What additional data or information needs to be in the test-plan? Throughout this article, we will address these questions. What one should expect here is not a ready-made solution, but the underlying philosophy, various options available for implementation and key considerations. As mentioned in the introductory article, there is an ‘intellectual part’ which requires the best and the brightest engineering mind and cannot be substituted by any tool or practice. This will be clearly identified wherever applicable.  

Example

Let’s revisit our example. The DUT is a simple SOC with some standard SOC components – a host processor, a co-processor (such as a DSP or some such computational element), internal buses for both control and high speed data transfers, memory sub-system (DDRs, SRAMs), peripheral IO interfaces such as USB, UART, and internal SOC control elements such as IO muxes, clock/power management unit, interrupt management unit.  

The following block diagram illustrates our example. 

Example DUT

What Rather Than how

The first step in any verification program is to review the Design/Architecture Specification documents along with any other relevant supporting documents such as ‘Standards Specifications’. This becomes the basis of what needs to be tested. At this stage, don’t worry about the how this block or chip needs to be tested or any other logistical issues such as schedule, simulation speed, resources etc., as it would cause unnecessary distraction, and might cause you to overlook some of the test-cases. Any cracks at this stage are the most expensive. Achieving a high quality list of ‘what’ needs testing is indeed an intellectual process – this list forms the ‘denominator’ in the coverage ratio – regardless of tool or method used for measuring coverage.  

Given the importance, this step needs undivided attention. Block off time on your calendar, hide in conference rooms, work from home, or do whatever it takes to focus. In addition, indulge in lots of formal and informal brainstorming sessions with various members of the team such as the architects, principle designers, other senior verification engineers, software/firmware engineers, and even marketing personnel. During these discussions, don’t let the other person drag you into the ‘how’ or any other logistical issue like schedules or resources. Also note that everyone will be providing you their perspective based on their roles/background. I call these Swiss cheese slices; all will have holes, but stacked on top of each other will give you a solid list of cases.  

Lastly, start organizing this list hierarchically and in sections. Typically, there will be following sections:  

• Architectural or black box cases
  • eg, Read a sector from SATA interface with the interrupt enabled. In the Interrupt Service Routine (ISR), examine the contents of the sector read and clear the interrupt.
• Software or use cases
  • eg, The boot sequence; Bus enumeration sequence for the USB port
• Micro-architectural or Design cases (aka white box)
  • eg, state machine interactions; buffer full/empty conditions
• Block/sub-block level
  • eg, USB Link block level: Rest of the chip can be substituted by a Bus Functional Model (BFM)
• Cluster or System level interactions
  • eg, System Memory coherency and interactions with multiple requestors
• Compliance
  • eg, SATA, USB standards Compliance for interoperability;
• Error cases
  • eg, SATA Device sends erroneous packets
• Performance
  • eg, Memory Bandwidth  

Example of a hierarchy:  

• Major Feature: eg. USB packet Transfer types
  • Minor Feature: eg. Bulk Transfer
    • Test Scenario or a Test Matrix: eg. Minimum and Maximum size Bulk OUT transfers

In addition to writing down the test scenario, it is extremely important to note down any assumptions or questions one might have.  

A Generic Block

Let’s create a generic block to illustrate the process of identifying thorough top-down test scenarios:  

Generic Test Block

This block has several input and output data ports. There is a separate interface to access control/status registers. There are two clock domains and internal memory for local storage. In addition, there are some side band signals, along with several asynchronous events coming into the block such as reset, clock disable, mode control signal and so on. As an exercise, try mapping any blocks or designs you have worked in the past into this – you will be amazed! Make it even more interesting – map a microprocessor into this block!  

First order Test Scenarios for this generic block:  

• Access to all control/status register
• Access to all memory elements (both via standard datapath, and any special backdoor access)
• Complete protocol testing of all input and output interfaces (control and datapath)
• Exhaustive/interesting testing of all control logic (first order and ‘interesting’ register coverage)
• Exhaustive/interesting testing of any data computation performed in the block
• All possible/useful combinations of the two clocks
• Side band signal functionality
• All asynchronous events crossed with each other and skewed against each other
• All asynchronous events during ‘important or interesting’ states  of the block
• Memory element access during operations – corner cases (buffer full, empty)
• Stalling
• Hard and soft reset behavior
• Power management cases
• Performance

We just saw the ‘science’ portion of test-planning! Generating a robust first order list of scenarios for any block should be possible by going through the above exercise. This starts becoming an ‘art’ (or the intellectual process), once we start creating second order or combination tests; in short, the optimization process.  

Priorities

Now that we have a list of ‘all’ cases that need to be tested or covered, next thing to do is to prioritize them according to the importance. This can be used throughout the project to make tough schedule related calls. The priority should also be used to generate weighted coverage numbers.  

What is the basis of the priorities or the importance? The following set will serve as a useful guideline:  

• Atomic hardware functions that cannot be worked around using software. For Instance, basic addition instruction in a microprocessor
• Advertised features or normal operation of the machine are more important than others
• Any bug that can cause a catastrophic failure in the normal operation of the machine.

Another pragmatic view point is ‘assuming worst case scenarios’ – let’s say a bug that slips affects the reset or the boot sequence – this chip will be DOA (Dead on Arrival) – no bring-up or characterization can be done on this chip. Instead, say, the access to certain memory locations don’t work – this is definitely not something to be proud of, but, on the positive side, at least the operations of the chip using the lower memory locations can be tested out (including the development of the software). Thus, one would put the boot sequence test at a higher priority compared to the access to the entire memory range. Again, note, this example was just to illustrate relative priorities. Any verification plan that does not cover access to the entire memory map is indeed a very poor one!  

Note that this was just to illustrate how one can go about the prioritizing. There are a lot more factors that need to be considered for prioritizing that depends on your project goals.  

Reviews

We have identified, documented and prioritized all the test-cases (the ‘what’ portion). It’s time for a formal review. Very important to note – DO NOT WAIT to complete identifying and documenting all the cases before calling a review. As we all know, verification is an NP complete problem, and thus, one can never say that their plan is theoretically complete! Use judgment, and call the review once the plan is at, say, 90% mark. Some useful guidelines for the review:  

• Circulate the review material well in advance so that the audience has a chance to study it.
• No lengthy text or narration.
• Walk through the cases hierarchically (breadth first)
• Use appropriate visual forms such as tables, lists, pictures (remember, a picture is worth a 1000 words)
• Start with a block diagram and a description of the DUT ‘in your own words’
• Required Audience: Design counterparts, architects, and senior design/verification members, owners of adjacent blocks, owners of central blocks, software/firmware engineers, System/board designers and Managers.
• Have your manager or colleague collect action items.
• Don’t let anyone hijack the meeting. Keep it under your control – it’s your meeting.
• Call extension meetings if all the material cannot be covered in one session.
• Solicit feedback on the ‘priority settings’.
• Follow up on all action items and send the updated plan once all the action items are completed.
• Don’t get into the ‘how portion’ (or don’t let any drag you into the implementation) – Cover that topic in a separate review.  

The ‘how’ Portion

Now that all the cases that need to be covered are completely documented and reviewed, let’s look at the ‘how’ part. This part will determine or form the specification for the test-bench.  

The first order of classification will be based on whether something is tested using simulation, formal method or emulation.  Simulation provides more controllability and observability. This makes it easier and more practical to hit white-box cases. Also, debugging is much harder on the emulator. You don’t want to be exposed to first order bugs in the basic operation during the emulation. In fact, simulation based testing needs to be used as a screen before starting the emulation.  

Within simulation the scope of the DUT is the other decision point. Most of the cases intrinsic to a block should be tested at a block level. This makes simulations faster, debugging quicker and test setups easier. Also, during earlier phases of the project, all the adjacent blocks may not be developed or stable for cluster or system level testing.  

Formal method or tools are still limited in terms of ability. These are best suited for smaller blocks that are well specified.  

Cases that need a large number of cycles are best suited for emulation. Other types of cases suited for emulation or prototyping are the ones that test interoperability with real-life interfaces or devices such as SATA or USB, in our example.  

To summarize, here are some of the key factors influencing the testing method:  

• Debug-ability: Areas most likely to have lots of bugs. This is true for normal machine operation during initial phases of verification (fresh RTL code).
• Cases hard to control: error cases, multiple events happening at precise points
• Cycles needed to setup and exercise the case
• Requirement of real life devices to provide the stimulus/response(interoperability)
• Testing speed (some cases need at-speed testing)
• Number of theoretical cases. Some scenarios can explode – and there may be an opportunity to use formal methods to cover such scenarios  

Another practical tip here is to put greater emphasis on debugging ease and simulation times/turn times during the high bug phase, for instance, when there is fresh RTL code.  There is no need to worry about phases or cases where the probability or likelihood of hitting a bug is very low.  

Notes:  

• The test-bench and the test cases should be design in such a way that most or majority of the tests at a lower scope can be reused at a higher level.  Block level cases should be reusable at cluster level, and system level simulation cases should be reusable on emulators. This provides two benefits: the lower level or scope tests can be used as a screen to start testing at the higher level, thereby eliminating any build or database coherency issues. Secondly, the test setup knowledge at lower levels can be used at higher levels. For instance, for system level test cases, one should not be required to understand detailed setup up procedures of a SATA transfer in the context of the SATA Link when it has already been put in place at a lower level test-bench.
• Lastly, apply the 80-20 rule for test-bench designs. That is 80% (read as majority) of cases should be supported by the mainstream test-bench. For the remaining 20% (read as minority), a special ability or a hook needs to be added to the test-bench. Again, apply the 80-20 rule for this remaining 20% and keep going till all cases are covered. This will be reviewed again, and in greater detail, with examples in a future article covering test-bench designs.  

Coverage Measurement/Key Indicators/Metrics

Once the test-plan has been filled with all the test cases (or coverage scenarios) along with the priorities and testing methods, one can start creating various indicators and metrics.  

A single number providing the state of the verification program is always desirable. However, it is important to build this in a hierarchical fashion. This way, one gets to look at the coverage at various granularities such as block based, feature based, scope based and so on. This helps to make tactical project decisions.  In the final section we will look at how all of this data can be organized and consumed.  

The most popular methods of measuring coverage are:  

• Code coverage – toggle, block, condition, state machine, expression.
  This is the easiest way to generate coverage information. It is built into most simulators these days and can be turned ON or OFF with the flick of a switch.
  The advantages of code coverage are the ease of use, and detection of any first order hole. On the down side, it doesn’t quite tell us if we are done. Any ‘missing’ RTL code cannot be detected. Also, the coverage information is mostly combinatorial in nature. Sequential cases don’t get measured. Lastly, dead logic or architecturally irrelevant cases provide false negatives. 
  It is a necessary but not a sufficient condition. Lastly, code coverage monitoring in the mid-phase of the project is a good way to track project progress.
• Functional coverage:
  One of the things that code coverage doesn’t provide is a coverage view abstracted at a higher level. For instance, one will get information about whether all the bits toggled on the address bus, but it will not tell us if all ‘regions’ of the memory were accessed by a particular memory master. This kind of abstracted coverage information starts tying closely with the desired functionality of a particular block. In most of the modern HVLs (Hardware Verification Languages), one can easily instrument these ‘coverage points’ or ‘coverage buckets’ to provide an abstracted view of the coverage. This is the most effective way to track coverage, provided test-benches are developed using HVLs.
• Assertion Based Coverage:
  This is a form of functional coverage.  There are several assertion languages and libraries that one can choose from. Interesting cases can be coded as assertions and the simulator can then ‘watch’ for these assertions during simulations. Note, one can construct very ‘smart’ sequential or ‘temporal’ assertion, and tie these closely to the coverage-plan/test-plan.
• Register Coverage:
  This is a special type of functional coverage. Covering all the bits or knobs that control a particular block’s behavior can provide very useful first order coverage. One can then create combinations of various fields to cover interesting cases.  

Knowing or deciding ‘what’ needs to be measured and setting goals is more of an art than science. This is the intellectual exercise. Let’s take a block with 20 control bits or knobs. If you let a coverage tool measure the coverage on this without any constraints, it will look for all permutations and combinations of these bits or fields – that’s more than a million cases and most of the cases may be useless or irrelevant. Identifying or selecting ‘interesting’ cases is indeed an intellectual exercise. How good someone is in doing this will determine the efficiency and robustness of the verification project.  

Another way to reduce the coverage set or optimize it without risk is to look at orthogonal cases, based on the design and usage. For instance, one may never use two features or blocks of the design simultaneously – say, SATA and the Memory Card interface will never be in a product together. This can be used to drastically reduce the number of test cases. Note, that this can be dangerous if used without proper care. First of all, this has to be documented very clearly. Even better, make it a requirement that such cases need to be officially accepted by the program management.  

Test-Plan Management System

Last but not the least is how do we manage all this data? There is a lot of important data that needs to be created, stored and accessed with different views. Traditional ‘Word’ or even Excel based test-plans are not enough. These are not ‘executable’, hard to keep coherent with the test-base. Oftentimes, the testplan document never gets updated once it is reviewed, and by the end of the project it is almost obsolete!  

The real solution is to create a database for all the information, similar to bug databases. There are several solutions available in the market. Cadence’s VManager™ or Mentor’s ReqTracer™ are some of the examples. Jasper DA offers a freeware named Gameplan ™. Or, one can develop an in-house tool to manage the test-plans. Let’s look at how we might want to organize and access this data.  

Key factors to keep in mind:  

• Ease of use.  Anything complex becomes a deterrent.
•  Test-plan is a live document – keeps getting updated whenever new ideas prop up, and everyone ought to be viewing the most recent version
• Ability for different views
• Marrying the scenarios with simulation or implementation data
• Ability to tightly couple with the testbench collateral
• Track specification/design changes seamlessly
• Removing any room for ‘oops’ through automation
• Query based access
• Using test-plan to manage status information including coverage data.

Records

The atomic record in this database is a test case (or a test scenario). Some of the important fields are:  

• Summary and description field,
•  Module, feature, sub-feature
• Owner
• Test-case submitter (there may be a situation that someone other than the block owner thinks of a case)
• Testing Method(s) used: simulation/emulation/formal
• Scope(s): Block/Cluster/System
• Priority/Weight
• Test(s) that will cover this scenario
• Assumptions/Questions associated
• Coverage measurement method
• Status (based on back-annotated simulation data)
• Tags:  This can be used for queries to build different types of ‘test-lists’ or ‘regression lists’ based on the need.
• Simulation directives

Of course, once you start thinking down this route, there may be other attributes that you can use to make the system even more efficient for your environment.  

Access

Various access points are desired for proper use of this data. Some of these are:  

• Easy to use GUI based system to enter test cases, one at a time
• Importing (from, say, an excel spreadsheet)
• Backdoor access for simulation scripts (query based)
• Exporting into standard formats such as excel (query based)
• Back annotation of results and other status information
• Reporting – html or other forms based on queries (for reviews)
• Metric reporting in a tabular or graphical form (query based) 

So, we have seen the art and science behind creation of test-plans/coverage plans. Test/coverage plan development is a very creative process. To begin with, one needs to have an in-depth knowledge of the DUT being implemented. The ‘hunch’ is a cumulative knowledge of the protocols involved, usage perspective including the use-cases, intent of  the features, design methods used, historical perspective (knowing USB1.0,2.0 while testing 3.0), knowing where the bugs lie. The ‘hunch’ then allows one to prioritize, shortlist ‘interesting cases’. This allows crafting of the next ‘killer’ test case. However, this alone is not enough. Verification is as much about discipline. One might catch all the killer corner cases in a DUT, but completely overlook an entire section! Cases like these are not uncommon. Having a disciplined systematic approach in combing through all the possible test scenarios is a must. This is the ‘science’ behind verification test planning.  

In the next article, we will focus on test-bench design – on how to build robust and reusable test-benches. You might have the best or most exhaustive test-plan, but a poor test-bench can be a project killer.

About the Author:  Suhas Belgal has 17 plus years of experience in Chip Design, Emulation, Modeling and Verification, including 9 years as a Verification Manager. During these years, Suhas has worked for several multi-billion dollar companies such as Intel , LSI, Mentor Graphics, and various start ups, and co-founded a Verification Services company. Over the years, Suhas has played key roles several high profile design teams such as Pentium II, and successfully led several SoC chips to production.  He has experience in a wide range of Verification Methods and tools, and has been a presenter and panel member at various conferences, including the DAC. He has a master’s degree in EE from University of Texas at Austin and a bachelor’s from VJTI, Mumbai.

This is the first in a series of blogs written for PuneChips by Suhas Belgal titled Field Manual for Verification Planning. The blogs deal with functional verification of digital ICs and cover mostly the pre-silicon verification phase.     

The objective of this series is to provide a view of the ‘art’ of design verification. Everyone has heard the quote “Verification is an NP complete problem – it can never be done”.  If so, how should one schedule the verification program? When is the chip really verification clear for tape-out or production? If it is art, how does one measure the quality? Or, how does one turn it into ‘science’ and bring predictability into the equation?  Recently, while reading Robert Pirsig’s famous book “Zen and the art of Motorcycle Maintenance”, the questions of ‘art’, ‘science’ and ‘quality’ of the chip being designed crossed my mind…This series will provide a practical insight into the various aspects of verification, different tools, methodologies, best known practices, key indicators, tracking and management while trying to reflect on the fundamental question of whether verification can be completed. All of these will help bringing in the much needed predictability into the verification program. However, all of the best know practices and automation tools in the world can still not replace the need for engineering intelligence. You still need the best and the brightest minds to tackle the challenges. On the flip side, this is what makes verification challenging and interesting, and will attract the best minds out there. So, sprinkled throughout this series, you will find the term ‘intellectual process’ or ‘intellectual exercise’; the part of the process which still needs human intelligence or an engineering discretion will be identified as the intellectual process or the intellectual exercise.The basic goal of any chip design verification project is to find ‘all’ the bugs before tape-out! One way to bring pragmatism is to clearly identify the context of the statement of bug free design. That is, the design should be bug free for a crisply identified goal such as ‘customer demo/samples’ or ‘to enable software development’.  Even with such constraints, it still remains to be an NP complete problem. This is where statistics, probability can come in, and various indicators can be used to define the verification quality.          

In addition to the fundamental goal, there are other objectives such as productivity, efficiency, resource usage, ‘finding critical bugs earlier’ and so on. These are as important, as a matter of fact, even more important oftentimes than the basic question of ‘have we caught all the bugs’.     

Basic Dimensions   

The three basic dimensions of verification are ‘Coverage’, ‘Stimulus’ and ‘Checker’. Regardless of the tools or methodology, any verification environment consists of these three parts.       

Coverage addresses the fundamental question of how complete the verification is. This being an NP complete problem can never be complete, theoretically. However, in reality, setting the ‘denominator’ of the ratio – ‘covered vs planned’, becomes an intellectual exercise. Certain practices and pitfalls will be covered in detail in the ‘coverage/testplan’ topic.      

Stimulus can be deterministic, but can explode very quickly. Say, a 2 input functional block can be covered exhaustively in 4 cases. But, a 256 input block will require  1.1579….e+77 combinations – impractical! Even worse, sequential elements add a time dimension. Finding proper methods to select or prioritize ‘important’ or ‘high leverage’ stimulus is an intellectual exercise.       

Checker or rather, not having a thorough checker will make the two other dimensions useless. One can have a ‘complete’ checker only if the definition of that the expected behavior is complete. Usually, this is covered by the Design or the Architecture Specification of the chip or the product. Proper interpretation and checking for completeness is an intellectual process.     

Flow/Process  

A typical flow involves the following steps/phases, interspersed with reviews (every phase should begin and end with reviews).       

1. Design/Specification study
2. Coverage planning (traditionally known as test-plan development)
3. Testbench design planning
4. Setting up the Verification Environment – databases, bug tracker, templates, regression/simulation environment, debug process, indicator tracking
5. Testbench implementation
6. Coverage plan implementation
7. Bringup (fresh RTL tested against fresh testbench)
8. Feature coverage
9. Coverage exploration
10. Final Checklist

Verification Methods  

Several methods can be employed to carry out pre-silicon verification. The primary being Simulation based, Emulation based and the Formal method. All have pros and cons and can be used to complement each other.       

Simulation method utilizes dynamic simulation techniques used at different scopes such as ‘module or block’ level, ‘cluster’ level or the ‘full chip’ level.  Lower granularity helps speed up simulations, catch basic bugs quickly.  However, cluster or full chip environments help check the interactions between the blocks, which are a common source of bugs. However, at higher levels, simulation speed slows down, and one starts encountering ‘controllability/observability ’ problems.       

Emulation, hardware acceleration, proto-typing allows testing at much higher speed, and possibly at-speed.  Hardware/software co-verification, using real life devices can be accomplished using emulators.  Primary advantage is to get large number of cycles needed reach certain states of the design, say testing thousands of HD video frames. Also, one can actually bring-up peripheral devices such as SATA hard drives, thus taking out any risk in the implementation. Downsides are the cost and debug ability.       

Formal methods prove behavior of a certain section of the design to match a certain set of properties mathematically. Thus, it’s exhaustive and complete! However, there are logistical limitations to the current generation of tools such as design size, speed, and even then, coverage is still only as good as the property set. Defining the property set is an intellectual process.     

Productivity  

Productivity or the efficiency pertains to the processes, environment, tools, methods which can improve the verification cycle. The major costs are the ‘direct’ $ cost, human power cost and the cost in terms of total calendar time.  Simulators, emulators, server farms, other tools contribute to the direct $ cost. Calendar time is the critical path, and accounts for the processes that cannot be temporally scaled.       

Buy vs. brew is always an important decision, and comes across multiple times during a project. The maintenance cost of developing a tool in-house should not be ignored while making this decision.       

The non-deterministic, open-ended nature of verification complicates resource planning too. Predicting the number of simulation licenses needed, server farm size/capacity and estimating the total cycles needed to flush out ‘all the bugs’ is an intellectual process.       

Simulation license usage, cycles, build times, run-times need to be tracked before they can be improved. Scripts/tools to track these indicators should be planned for. In addition, the bug tracking (rate of opening/closing bugs, turnaround times etc), rate of test development, coverage improvement needs to be constantly measured for improvement.     

Environment   

Verification Environment is a key part of the verification project. This includes organizing the verification collateral, selecting an efficient and robust revision control system, work flow, automation to avoid manual mistakes and improve efficiency, integration of various pieces of verification and choosing an efficient platform for the entire team, including designers, to develop complex projects. And, in today’s global development community, an efficient environment is the key to success. Groups can be spread all over the world, but their development environment should be identical or seamless.       

Discipline and attention to detail are extremely important. How many times have we heard or experienced cases where bugs have slipped through the cracks in spite of having a ‘test’ that should have caught them – just because the test was not run on the final version of the netlist, or the test was not a part of a certain regression list. These mistakes are expensive, to say the least. Adding mere stress and pressure on engineers doesn’t help either. A fool-proof process and a set of tools/scripts can mitigate these circumstances.     

Verification Language    

The biggest wars in the verification world are on this topic. The modeling language for creating test-bench and the tests is central to any verification strategy. Starting with a bit of a historical perspective, Verilog or VHDL have been traditionally used for verification along with design. The ‘behavioral’ constructs in these languages aid verification tasks. Even today, several companies/projects rely on verification strategies based entirely on Verilog or VHDL. Using Verilog or VHDL has its advantages; the language knowledge is universal, no special simulators are needed, and most of the EDA tools understand and support these languages. On the downside, these languages were primarily designed to describe digital circuits/logic. They don’t have powerful data structure constructs. Randomization support is very limited.       

To circumvent these limitations, common powerful languages/scripts such as C and Perl have been introduced. C or Perl provide the language or programming power. However, they are still not specifically ‘verification languages’, and every project/company tend to have their own implementation of the methodology.       

Over 10 years ago, a new breed of specialized languages known as ‘Hardware Verification Languages (HVL)’ came into existence, starting with Vera and Specman. Vera was born inside Sun Microsystems as an internal verification language/tool. This was spun off and eventually became part of Synopsys. Most recently, the industry trend has been towards SystemVerilog, in an effort to standardize on the language.  HVLs provide strong Object Oriented Structure and advanced features for randomization and constraint solving. In addition, they have become platforms for myriad of verification functions such as coverage monitoring, assertions and so on.      

Finally, the most recent development has been that of a methodology layer via libraries. VMM, OVM are the two main methodologies in the market today for System Verilog.      

One could argue that everything that is provided by these higher level HVLs or the libraries can be implemented in Verilog or VHDL. True! But, firstly you get a tremendous boost in productivity as these libraries or languages provide a high number of pre-defined functions, constructs. Secondly, these libraries provide a framework that inherently makes the collateral reusable and efficient.        

Random, Directed, Emulation?     

Another contentious area for design and verification teams is to decide between random, constrained random or directed testing. Another dimension of this debate is the simulation vs. emulation decision. The pros and cons will be discussed in detail in a future article of this series.        

Sometimes, it helps using analogues. For instance, how would one test out a car – would one use a test-track with ‘simulated’ skids, obstacles and other external environmental factors, or just drive 100K miles on an expressway. One could use this analogy further and even look at orthogonals – say, if the blinkers have been tested in the garage, is it necessary to test them while driving through winding hilly roads, or say, while driving through winding hilly roads while the outside temperature is sub zero and at night. This helps reduce the set of interesting test cases.      

Reviews/Checklist    

As mentioned earlier, discipline is very important in Verification. Verification is the final safety net prior to tape-out. Any hole can potentially cost millions. In addition to inserting checks and balances into the tools/scripts/processes, reviews and checklists have a significant role in any verification project.        

For reviews, key things to understand are:      

• Primary intent should be to solicit feedback from other team members. Thus, every attempt should be made to communicate the content clearly to the attendees. A good idea or a clarification or detection of an error can save lots of time, frustration downstream.
• Put as many figures, tables as possible. Avoid textual paragraphs. A picture is worth 1000 words!
• Example code review is highly recommended.
• Time should be used efficiently but one should not limit wall-clock times     

Some of the useful review/checklists, other than the usual, are:      

• Tape-out checklist. Some of the items are
  • Waived coverage points or tests along with justification
  • Waived bugs along with justification
  • Uncovered planned items, if any, with justification/risk assessment
  • ‘What-else’ checklist: Once all the planned verification activities are completed along with a satisfactory bug curve, a series of what else reviews are recommended. This is a free flowing brainstorming of what else can possibly be done to find the bugs. As we all know, all bugs can never be found – which also means there are always more bugs in the design to be found and these reviews can potentially lead to them.
  • ‘Last set of bugs’ – (This category or the review needs a better name!): Towards the end, around the time the ‘what-else’ reviews are held, the last set of bugs uncovered should be reviewed or analyzed for the following:
    • What found the problem – was it accidental?
    • What caused the bug – did it exist all along or a recent event caused it?
    • Could it have been caught earlier?
    • What if it had slipped? Is there a workaround? This tells the severity of the bug    

These questions and discussions usually give rise to a few more clues or ideas about how to look for the remaining residual bugs.      

Modeling     

Modeling is a very generic term. In the context of verification, this is used for a ‘model’ that describes the correct or golden behavior. Often times, this arises out of architectural exploration efforts.        

Models can be developed in C (most common), MatLab, SystemC, SystemVerilog, and Verilog or for that matter, any language.        

When used as a golden model for verification purposes, it is very important to consider the verification requirements as this can have a very high impact on the productivity or efficiency of the project.  Most of the time, there are surprises, as the models are developed much before verification starts and/or by different groups. Avoid them by planning and collaborating ahead of time with the modeling team. Verifying these models independently is always an interesting and important problem.     

Performance Verification     

This being very important, requires explicit attention. Performance, as opposed to functional verification, can be tricky due to two things. Firstly, setting up cases or ‘checking mechanism’ is not covered by the traditional functional verification collateral – thus, it is more work and often comes as a surprise. Secondly, identifying, defining and quantifying performance metrics is non-trivial. For instance, if the specification identifies the startup time of, say, a product like an iPod to be less than 2 seconds, then one needs to identify functionality in the hardware that contributes to this delay and then use that number to check for correctness. Sometimes, ‘quality’ aspects are not clearly quantified – especially, the acceptable numbers. Video quality is a good example of this.      

Other Verification Areas     

While planning for verification, following areas or special cases need to be considered   

• Handling clock domain crossings
• Simulation artifacts, for instance, code that could mask propagation of Xs
• Design rule violations that will not be caught during traditional simulation/emulation
• Check for  potential errors introduced by the RTL -> GDS process
• Analog components, and their interface with the digital logic
• Process variations and the logic implemented to compensate, for instance, DLLs.
• Power simulations are going to be commonplace going forward

Example DUT     

Let’s take an example DUT – a simple SOC. We will throw in some standard SOC components – a host processor, a co-processor (such as a DSP or some such computational element), internal buses for both control and high speed data transfers, a memory sub-system (DDRs, SRAMs), peripheral IO interfaces such as USB, PCIe, UART, and internal SOC control elements such as IO muxes, clock/power management unit, interrupt management unit.       

The following block diagram illustrates our example. All the future topics in this series will be discussed in the context of this example. 

Example of a SoC DUT

Summary

Throughout this series, we will make an effort to identify the ‘art’ and ‘science’ involved in Chip verification, with practical tips or some of the best known practices. Through use of advanced techniques, tools, methods, discipline, best practices one can mitigate the non-deterministic nature of verification. These can be used for accurate scheduling, planning and a high quality execution of verification projects leading to successful first pass silicon. However, there is an ‘intellectual’ part of the process that still requires the best and the brightest minds.  And, that’s where the satisfaction or the intellectual rewards lie. Verification accounts for 70% of the pre-silicon development efforts, according to some estimates – let’s not leave it to chance by quoting the famous ‘verification is an NP complete problem’ – it can be harnessed and this has been already demonstrated by several successful groups, companies.       

In the next session, we will cover ‘test-plan/coverage plan’ topic in detail.       

About the Author:  Suhas Belgal has 17 plus years of experience in Chip Design, Emulation, Modeling and Verification, including 9 years as a Verification Manager. During these years, Suhas has worked for several multi-billion dollar companies such as Intel and LSI, various start ups, and co-founded a Verification Services company. Over the years, Suhas has played key roles several high profile design teams such as Pentium II, and successfully led several SoC chips to production.  He has experience in a wide range of Verification Methods and tools, and has been a presenter and panel member at various conferences, including the DAC. He has a master’s degree in EE from University of Texas at Austin and a bachelor’s from VJTI, Mumbai.      

This content has been licensed to PuneChips under a Creative Commons Attribution-Noncommercial 2.5 India License. Contact Suhas Belgal for details of how to attribute and re-use for non-commercial as well as commercial distribution.    

As the complexity of Integrated Circuits (specifically ASIC and SoC) increases, and as their sizes keep reducing, the task of testing the chip gets more and more challenging. Engineers need to come up with better and different methodologies to ensure what goes to the factory for manufacturing is actually what they intended to deliver. Verification occurs at various stages in the ASIC development cycle. How much is enough at each stage is a problem that needs to be addressed on a case to case basis. A sound knowledge of various techniques and awareness of capabilities and limitations of each technique goes a long way in making decisions about when, where and what.

ESL or Electronic System Level testing is the newest trend. Supporters of ESL claim that it is a highly powerful system level modeling tool. It enables fast software bring-up if combined with an emulation/FPGA prototyping platform. ESL has been used successfully to validate systems for mobile applications where only one peripheral/application is active on the processor bus. ESL does not seem suitable for systems where multiple processes and interfaces are active simultaneously, like for example in a networking system.

Formal verification, a static verification technique which is mainly assertion based, is useful to check control paths. It cannot be used to verify datapaths. Dynamic simulation is a very effective way of verifying functionality of every block in the ASIC including the datapath. Gate level simulations performed after the back annotated placement and routing data is available are used to identify timing related issues or omissions/errors in stating multi-cycle paths.

The need to find hardware bugs as early as possible in the ASIC lifecycle drives the emulation and/or FPGA prototyping effort. Both these techniques enable the testing of scenarios which are generally not possible to test in dynamic functional verification, well before the actual silicon comes back from the fab. Emulation or prototyping also accelerate fast software ramp up and the software team can get a development platform ready well before the actual chip is available. Emulation involves running test cases on hardware accelerated platforms like Palladium from Cadence and Veloce from Mentor. For FPGA prototyping, Single or multiple FPGAs are used to build a PCB system targeted for the testing of the ASIC/SoC. The ASIC code is then fully or partially programmed on the FPGA/s and functionality can thus be tested.

Scenarios with much longer simulation times than what normal functional simulation allows can be run on the emulation platforms. All the internal signals are available for viewing and debug, just like in functional simulation. The FPGA prototype platform does enable longer test time, but the debugging available is limited. The hardware accelerators are costly, and investing in them makes sense if a company has lot of ASIC programs running simultaneously. For companies which have similar chips planned back to back, investing in a home grown FPGA based emulation/prototyping platform makes sense. Another advantage FPGA prototyping is that the RTL goes through a complete synthesis and place and route cycle and testing is done on a circuit which is as close to the real ASIC as possible.

To ensure that a bug free product reaches the customer is a complex activity and poses multiple challenges. Coverage, legacy code, repeatability are issues that need to be tackled. Ensuring that the coverage is at an acceptable level is important. Code coverage is run to find out if all the possibilities of a written code are exercised in a test suite. Simulators from cadence (ius), synopsys(vcs) and mentor (modelsim) have their own code coverage analyzers. Functional coverage means to find out if each feature listed in the specification for an ASIC/SoC is verified. It is essential that the functional specification document has an individual numbered paragraph for each feature so that traceability is easier. Functional coverage is an activity that needs planning, reviews and careful test case designing. Methodologies like eRM (e reuse methodology – Specman based) and OVM (open verification methodology – System verilog based) do assist checking functional coverage, but the inputs provided need careful specification and reviews.

Reviews, not just for coverage, but at every stage in the ASIC cycle are extremely important. One of the challenges encountered while designing an ASIC is that the hardware team interprets a certain behavior from software and the software expects that certain things are taken care of in hardware. It is very important to involve members from design team, verification team, architecture team, software & firmware team for verification review.

It takes a good amount of effort to come up with a verification environment, and it is very common for a team to use what has worked before when schedules are demanding. Legacy environment saves lot of time, but it also handicaps the team. Talking about saving time, efficiency goes a long way in shrinking the schedules. The initial time and effort investment in automation of repetitive tasks save lot of time in future. Use of re-usable methodologies will definitely save time and effort.

Finally, while choosing the verification flow for a certain ASIC, team needs to look at what is available in terms of resources as well as time, understand the end user requirement, and make a decision on which technique to employ at what stage.

About the Author

Arati has close to 9 years experience in ASIC front end design and verification. Post silicon validation and FPGA prototyping is her recent area of interest and expertise. Arati has worked with Wipro Technologies and Conexant Systems. Arati did her B.E. from University of Pune and M.Tech from CEDT, Indian Institute of Science, Bangalore. See her linked-in profile for more details.