Change Based Validation and Yield Based Optimization for Recommending Test Cases

Abstract

The validation team needs to validate any new feature that is added to a product or any change that is made. As result of this, test case gets added to the test suite. This test suite contains all list of test cases that need to be performed. Overtime, the number of test cases has increased. Executing all of these test cases for every validation cycle is time consuming task. Change Based Validation is an optimization technique which assumes that all bug fixes are a direct or indirect consequence of a change in source. In UEFI (Unified Extensible Firmware Interface) firmware, the changes are bug fixes or implementation of a requirement. The validation teams run their test suite to cover any possible issues due to changes in new release but it requires expertise and time to map any incoming changes to corresponding tests. The project aims to extract this knowledge from changed descriptions and identify the impacted domains as well as features within domains to test. The project follows the usage of heuristic models in the beginning and deploys data augmentation/ weak supervision techniques to utilize other state-of-the-art NLP techniques in the latter phase. Yield based optimization is technique which tries to achieve the same objective, that is, reduced set of test cases that should be included in the test plan for validation. This method relies on the execution history of test cases and extracts those test cases which have failed in the past few runs. This reduced test suite helps in reducing the overall execution time and cost, and eliminates need of manual selection of test plan.