Abstract
The testing and verification of a complex hardware or software system, such as modern integrated circuits found in everything from smartphones to servers, can be a difficult process. One of the most difficult and time-consuming tasks a verification team faces is reaching coverage closure, or hitting all events in the coverage space. Coverage-directed-generation (CDG), or the automatic generation of tests that can hit hard-to-hit coverage events, and thus provide coverage closure, holds the potential to save verification teams significant simulation resources and time. In this paper, we propose a new approach to the CDG problem by formulating the CDG problem as a noisy derivative free optimization problem. However, this formulation is complicated by the fact that derivatives of the objective function are unavailable, and the objective function evaluations are corrupted by noise. We solve this noisy optimization problem by utilizing techniques from direct optimization coupled with a robust noise estimator, and by leveraging techniques from inverse problems to estimate the gradient of the noisy objective function. We demonstrate the efficiency and reliability of this new approach through numerical experiments with a noised quadratic function and an abstract model of part of IBM’s NorthStar processor, a superscalar in-order processor designed for servers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
There are many other templates with different values of SW and TW that achieve the same probability.
(24 Iterations) \(\times \left( 25 \frac{{\text {Points}}}{{\text {Iteration}}}\right) \times \left( 25 \frac{{\text {Simulations}}}{{\text {Point}}}\right) = 15{,}000\) Simulations.
(35 Total Iterations) \(\times (25 \frac{{\text {Points}}}{{\text {Iteration}}}) \times (25 \frac{{\text {Simulations}}}{{\text {Point}}}) = 21{,}875\) Simulations.
References
Berahas AS, Byrd RH, Nocedal J (2019) Derivative-free optimization of noisy functions via quasi-newton methods. SIAM J Optim 29:965–993
Borkenhagen J, Sorino S (1999) 4th generation 64-bit PowerPC-compatible commercial processor design. IBM Server Group White Paper
Burden RL, Faires JD (2010) Numerical analysis. Cengage learning
Conn A, Scheinberg K, Vicente L (2009) Introduction to derivative-free optimization. SIAM, Philadelphia
Fine S, Ziv A (2003) Coverage directed test generation for functional verification using Bayesian networks. In: Design automation conference
Golub GH, von Matt U (1997) Generalized cross-validation for large-scale problems. J Comput Graph Stat 1:1–34
Golub GH, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21:215–223
Huyer W, Neumaier A (1999) Global optimization by multilevel coordinate search. J Glob Optim 14:331–355
Kelley C (2011) Implicit filtering. SIAM, Philadelphia
Mishra P, Dutt N (2002) Automatic functional test program generation for pipelined processors using model checking. In: 7th annual IEEE international workshop on high-level design validation and test, pp 99–103
Moré JJ, Wild SM (2011) Estimating computational noise. SIAM J Sci Comput 33:1292–1314
Nativ G, Mittermaier S, Ur S, Ziv A (2001) Cost evaluation of coverage directed test generation for the IBM mainframe. In: Proceedings of the 2001 international test conference, pp 793–802
Nocedal J, Wright S (1999) Numerical optimization. Springer, New York
Pétard H (1938) A contribution to the mathematical theory of big game hunting. Am Math Monthly 45:446–447
Pintér JD (1996) Global optimization in action. Springer, New York
Piziali A (2004) Functional verification coverage measurement and analysis. Springer, New York
Powell M (2006) The NEWUOA software for unconstrained optimization without derivatives. In: Pillo GD, Roma M (eds) Large-scale nonlinear optimization. Springer, Boston, pp 255–297
Prashanth LA, Bhatnagar S, Fu M, Marcus S (2017) Adaptive system optimization using random directions stochastic approximation. IEEE Trans Autom Control 62(5):2223–2238
Rios LM, Sahinidis NV (2013) Derivative-free optimization: a review of algorithms and comparison of software implementations. J Glob Optim 56:1247–1293
Spall JC (1987) A stochastic approximation technique for generating maximum likelihood parameter estimates. In: 1987 American control conference, pp 1161–1167
Spall JC (1992) Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans Autom Control 37(3):332–341. https://doi.org/10.1109/9.119632
Spall JC (1998) An overview of the simultaneous perturbation method for efficient optimization. Johns Hopkins APL Tech Dig 19(4):482–492
Tenorio L (2017) An introduction to data analysis and uncertainty quantification for inverse problems. SIAM, New York
Acknowledgements
EH and BI’s work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). RG, BS, and AZ’s work is supported by IBM.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Gal, R., Haber, E., Irwin, B. et al. How to catch a lion in the desert: on the solution of the coverage directed generation (CDG) problem. Optim Eng 22, 217–245 (2021). https://doi.org/10.1007/s11081-020-09507-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11081-020-09507-w