Monday, May 17, 2004

Kaner on Testing

by Asim Jalis

I have recently been reading Kaner's notes (and books) on testing. He has a list of 11 testing techniques and claims that each technique used alone eliminates only certain types of errors. To create highly reliable software multiple techniques must be used together in concert. For more details look at: Here is a list of the 11 techniques together with my notes on what each one is. The URL above has notes on all the techniques. The 11 dominating techniques or paradigms of black-box testing: - Domain testing: In domain testing you are looking the individual fields or variables, partitioning the possible values into equivalence classes (classes of values for which you expect every value in the class to yield the same result in a test), and looking at the boundary conditions. - Risk-based testing: 1. Make a prioritized list of risks. 2. Perform testing that explores each risk. 3. As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop. Imagine a problem and then look for it. Harsh testing for vulnerable areas of the program. - Scenario testing: In scenario testing, you devise stories of how a particular user might use the application and the execute the story. Challenging cases that reflect real use. Real, motivating, credible, complex use, results are easy to evaluate. - Function testing: Define all the functions or features that a program provides. Determine how you would know if a function worked. Test each function. Assesses capability rather than reliability. Does not cover interaction. - Specification-based testing: Check the product.s conformance with every statement in every spec, requirements document, etc. Could also be User documentation testing. - Regression testing: In regression testing, you're focusing on previously identified (and fixed) bugs to ensure that they're still fixed. - Stress testing: Push system's resource requirements. Increase load. Increase rate of hitting the system. - User testing: Determine different roles that users will play. Use the system as these different users will use it. - State-model based testing: Create a finite state machine that represents the system. Create test-cases by exploring all the paths that the state transitions can take. And then apply these test cases to the system. - High volume automated testing: Generate large number of test cases using automated techniques and either hit the system with all of them, or use some kind of random sampling, or choose some intelligently chosen representative test cases from all the sets. - Exploratory testing: Exploratory testing is defined as "any testing in which the tester dynamically changes what they're doing for test execution, based on information they learn as they're executing their tests." To me, exploratory testing is a "meta-type". While it does require its own mind set (and thus qualifies as a separate item on the list), any of the other types can also be done in an exploratory manner. Next, to flesh these out we apply them printf. - Domain testing: What are the equivalent classes of inputs and what are their boundaries? Numbers, strings, floating points. Empty string, one character string, negative numbers, positive numbers, MAXINT, -MAXINT, MAXLONG, -MAXLONG. -1, tiny floating point numbers, large floating point numbers. Vary n, m in n.m, 0 through large values, make n < m, m < n, n = m. Zero args, one arg, many args. Use null. Pass in incorrect arguments. - Risk-based testing: The highest risk might be the string printf, since that risks buffer overruns. Test that first. The obscure classes might be lower risk. Floats and ints would be important too. Look for places where buffer overrun might occur. Look for memory exhaustion. Look for really slow algorithm. - Scenario testing: Imagine a physicist uses this to generate a table, or a report. Use it to generate a report, a table of data. - Function testing: Render number, string, floats, etc. - Specification-based testing: Write a test for each statement in the man page. - Regression testing: Automate the tests and run them repeatedly. - Stress testing: Use very large string. Use really large format string. Use very large number of arguments. See when failure occurs. Use null. Pass in incorrect arguments. Pass arguments in wrong order. How does it indicate failure. Test the return values. - User testing: Define user roles. How would people use this function? A programmer is one user. He might use it to generate hello world type messages. Or for debugging messages. How fast does it run. Does it require a flush to print out the debugging output? What if a program crashes right after it prints. Another user could be someone developing a website. - State-model based testing: Single state function. - High volume automated testing: Run it repeatedly. Run it for all the numbers. Run it for string from length 0 to length N. Run it for all permutations of 3 or 4 or 5 or 6 arguments. - Exploratory testing: Try different things. Impact on memory. Measure malloc calls. Measure free calls. Count these calls. Count how much is allocated. Impact of parameter length and argument count on performance.