Testcover.com - Performance for Comparison

	Testcover.com
		Performance for Comparison

Home
Existing User Login
Brochure
Sign up for Risk-Free Trial
About Testcover.com
Frequently Asked Questions
Tutorial
Performance
Background
Partners
Subscription Prices
Promotion Activations
Student Subscriptions
Privacy Policy
Terms & Conditions
Contact Information

How does your test case generator compare?

Test Model	Test cases		Response time
(values^factors)^a	(number)^b	(key)^c	(seconds)^d
2¹⁰	6	•	2
2³⁰	8	•	2
2¹²⁰	10	•	2
3⁵	11	•	2
3⁶2²⁴	15		2
3²⁰	15	=	2
3³⁰	18	=	2
3⁹⁰	23	=	2
4¹3³2⁶	12	•	2
4⁵	16	•	2
4¹3³⁹2³⁵	21	=	2
4⁵2²⁰	22		2
4⁸	23		2
4²⁴	28	=	2
4⁵3²⁰	29		2
4¹⁵3¹⁷2²⁹	29	=	2
4⁸⁰	40	=	2
5¹4⁴2¹²	20	•	2
5²3⁹2²⁰	25	•	2
5²4³3³	25	•	2
5³3²⁷	27		2
5⁸	33	=	2
5¹¹	38	=	2
5³⁰	45	=	2
6¹5⁵2²⁴	30	•	2
6²5¹3¹²	36	•	2
6⁹	46	=	2
6¹²	57		2
6²⁴	72	=	2
7¹6²3¹²	42	•	2
7³4²⁰	49	•	2
7⁴4¹²3¹⁵	49	•	2
7¹⁰	61	=	2
7¹⁴	81	=	2
7³⁰	91	=	4
8¹7²6¹4⁵3⁴2⁸ ^e	61		3
8³5²⁸	64	•	3
8⁶5¹⁰4¹⁵	64	•	3
8¹¹	78	=	2
8³⁰	120	=	5
9⁵5²⁵	81	•	3
9¹⁰	81	•	2
9²6¹⁶	84		2
9³⁰	153	=	9
10³4²⁸	100	•	3
10²⁰	181		7
11⁹7²¹	121	•	6
11¹⁴	167		4
12³7²⁸	144	•	8
12¹⁹	210	=	10
13¹¹8²⁰	169	•	12
13²⁷	325	=	55
14²7²⁹	196	•	14
15²9²⁷	225	•	22
16⁴10²⁶	256	•	36
22¹15¹10¹²	330	•	15

Notes:

Product notation describing test model. Each exponent gives the number of test factors having the number of test values given by the corresponding base. For example, model 3⁶2²⁴ has 6 test factors with 3 values each and 24 factors with 2 values each. The indicated product gives the total number of test cases: 3⁶2²⁴ = 12,230,590,464.

Number of test cases generated to cover all pairs of test factor values for this model. The number can vary if there are constraints excluding some test cases. No constraints were applied, except for the model described in Note e below.

Key to bounds on covering array size.

	•	Array size is at lower bound (optimal)
	=	Array size equals best published upper bound^f

Elapsed time in seconds from request submission until results display. Response time can vary with service load, access speed and other factors. These response times were measured with a light service load and broadband internet access.

Test model for the Advanced Web Search Example, demonstrated at the STAREAST 2005 EXPO. The 21 test factors led to 2715 factor value pairs; all but 6 were covered to comply with required constraints. See the EXPO Technical Presentation on our Background page for more detail. These results (61 test cases currently vs. 63 at STAREAST) illustrate an improvement in the service since the conference.

References for bounds on covering array size.
On the state of strength-three covering arrays. M. Chateauneuf, D. L. Kreher. Journal of Combinatorial Designs, 10(4):217-238, 2002.
Products of mixed covering arrays of strength two. Charles J. Colbourn, Sosina S. Martirosyan, Gary L. Mullen, Dennis Shasha, George B. Sherwood, Joseph L. Yucas. Journal of Combinatorial Designs, 14(2):124-138, 2006.
Covering arrays on graphs: connections to design theory, extremal combinatorics and algebraic combinatorics. Karen Meagher. Presentation for LaCIM, February 2004.
Group construction of covering arrays. Karen Meagher, Brett Stevens. Journal of Combinatorial Designs, 13(1):70-77, 2005.
Group construction of covering arrays - part 2. Karen Meagher. Technical Report, January 2005.
Upper bounds for covering arrays by tabu search. Kari J. Nurmela. Discrete Applied Mathematics, 138(1-2):143-152, 2004.
Strength two covering arrays: existence tables and projection. Charles J. Colbourn. Discrete Mathematics, 308(5-6):772-786, 2008.
See also Comparison of Efficiency at pairwise.org tools page.

These results were updated May 1, 2008. They are subject to change, particularly improvement, without notice.