Test.Tasty.QuickCheck

Create a TestTree for a QuickCheck Testable property

Create a test from a list of QuickCheck properties. To be used with Test.QuickCheck.allProperties. E.g.

tests :: TestTree
tests = testProperties "Foo" $allProperties

Number of test cases for QuickCheck to generate

Replay seed

If a test case fails unexpectedly, show the replay token

Size of the biggest test cases

Maximum number of of discarded tests per successful test before giving up.

Show the test cases that QuickCheck generates

Number of shrinks allowed before QuickCheck will fail a test.

Timeout for individual tests within a property.

Attaches a label to a test case. This is used for reporting test case distribution.

For example:

prop_reverse_reverse :: [Int] -> Property
prop_reverse_reverse xs =
  label ("length of input is " ++ show (length xs)) $
    reverse (reverse xs) === xs

>>> quickCheck prop_reverse_reverse+++ OK, passed 100 tests:7% length of input is 76% length of input is 35% length of input is 44% length of input is 6...

Each use of label in your property results in a separate table of test case distribution in the output. If this is not what you want, use tabulate.

The type of properties.

A successful test run

A failed test run

Given up

A property that should have failed did not

The number of test cases having each class (see classify)

The test case's classes (see classify)

The test case's labels (see label)

The test case which provoked the failure

The number of test cases having each combination of labels (see label)

Number of tests skipped

Number of unsuccessful shrinking steps performed since last successful shrink

Number of unsuccessful shrinking steps performed

Number of successful shrinking steps performed

Number of tests performed

Printed output

Why did the property fail

Data collected by tabulate

The exception the property threw, if any

What seed was used

What was the test size

The existentially quantified witnesses provided by witness

Implication for properties: The resulting property holds if the first argument is False (in which case the test case is discarded), or if the given property holds. Note that using implication carelessly can severely skew test case distribution: consider using cover to make sure that your test data is still good quality.

NonEmpty xs: guarantees that xs is non-empty.

Monomorphise an arbitrary property by defaulting all type variables to Integer.

For example, if f has type Ord a => [a] -> [a] then $(monomorphic 'f) has type [Integer] -> [Integer].

If you want to use monomorphic in the same file where you defined the property, the same scoping problems pop up as in quickCheckAll: see the note there about return [].

Test a polymorphic property, defaulting all type variables to Integer.

Invoke as $(polyQuickCheck 'prop), where prop is a property. Note that just evaluating quickCheck prop in GHCi will seem to work, but will silently default all type variables to ()!

$(polyQuickCheck 'prop) means the same as quickCheck $(monomorphic 'prop). If you want to supply custom arguments to polyQuickCheck, you will have to combine quickCheckWith and monomorphic yourself.

If you want to use polyQuickCheck in the same file where you defined the property, the same scoping problems pop up as in quickCheckAll: see the note there about return [].

Test a polymorphic property, defaulting all type variables to Integer. This is just a convenience function that combines verboseCheck and monomorphic.

If you want to use polyVerboseCheck in the same file where you defined the property, the same scoping problems pop up as in quickCheckAll: see the note there about return [].

Combine two generator perturbing functions, for example the results of calls to variant or coarbitrary.

Random generation and shrinking of values.

QuickCheck provides Arbitrary instances for most types in base, except those which incur extra dependencies. For a wider range of Arbitrary instances see the quickcheck-instances package.

Lifting of the Arbitrary class to unary type constructors.

Lifting of the Arbitrary class to binary type constructors.

Used for random generation of functions. You should consider using Test.QuickCheck.Fun instead, which can show the generated functions as strings.

If you are using a recent GHC, there is a default definition of coarbitrary using genericCoarbitrary, so if your type has a Generic instance it's enough to say

instance CoArbitrary MyType

You should only use genericCoarbitrary for data types where equality is structural, i.e. if you can't have two different representations of the same value. An example where it's not safe is sets implemented using binary search trees: the same set can be represented as several different trees. Here you would have to explicitly define coarbitrary s = coarbitrary (toList s).

Apply a binary function to random arguments.

Apply a ternary function to random arguments.

Apply a function of arity 4 to random arguments.

Generates a random ASCII character (0-127).

Generates an element of a bounded enumeration.

Generates an integral number. The number is chosen uniformly from the entire range of the type. You may want to use arbitrarySizedBoundedIntegral instead.

Generates an element of a bounded type. The element is chosen from the entire range of the type.

Generates a printable Unicode character.

Generates an integral number from a bounded domain. The number is chosen from the entire range of the type, but small numbers are generated more often than big numbers. Inspired by demands from Phil Wadler.

Uniformly generates a fractional number. The number can be positive or negative and its maximum absolute value depends on the size parameter.

Generates an integral number. The number can be positive or negative and its maximum absolute value depends on the size parameter.

Generates a natural number. The number's maximum value depends on the size parameter.

Generates any Unicode character (but not a surrogate)

A coarbitrary implementation for enums.

A coarbitrary implementation for integral numbers.

A coarbitrary implementation for real numbers.

coarbitrary helper for lazy people :-).

Generic CoArbitrary implementation.

Shrink a term to any of its immediate subterms, and also recursively shrink all subterms.

Generates an infinite list.

Generates an ordered list.

Recursively shrink all immediate subterms.

Shrink an element of a bounded enumeration.

Example

data MyEnum = E0 | E1 | E2 | E3 | E4 | E5 | E6 | E7 | E8 | E9
   deriving (Bounded, Enum, Eq, Ord, Show)

>>> shrinkBoundedEnum E9[E0,E5,E7,E8]

>>> shrinkBoundedEnum E5[E0,E3,E4]

>>> shrinkBoundedEnum E0[]

Shrink a real number, preferring numbers with shorter decimal representations. See also shrinkRealFrac.

Shrink an integral number.

Shrink a list of values given a shrinking function for individual values.

Map a shrink function to another domain. This is handy if your data type has special invariants, but is almost isomorphic to some other type.

shrinkOrderedList :: (Ord a, Arbitrary a) => [a] -> [[a]]
shrinkOrderedList = shrinkMap sort id

shrinkSet :: (Ord a, Arbitrary a) => Set a -> [Set a]
shrinkSet = shrinkMap fromList toList

Non-overloaded version of shrinkMap.

Returns no shrinking alternatives.

Shrink a fraction, preferring numbers with smaller numerators or denominators. See also shrinkDecimal.

All immediate subterms of a term.

Generates a list of a given length.

A special error value. If a property evaluates discard, it causes QuickCheck to discard the current test case. This can be useful if you want to discard the current test case, but are somewhere you can't use ==>, such as inside a generator.

Given a property, which must use label, collect, classify or cover to associate labels with test cases, find an example test case for each possible label. The example test cases are minimised using shrinking.

For example, suppose we test delete x xs and record the number of times that x occurs in xs:

prop_delete :: Int -> [Int] -> Property
prop_delete x xs =
  classify (count x xs == 0) "count x xs == 0" $
  classify (count x xs == 1) "count x xs == 1" $
  classify (count x xs >= 2) "count x xs >= 2" $
  counterexample (show (delete x xs)) $
  count x (delete x xs) == max 0 (count x xs-1)
  where count x xs = length (filter (== x) xs)

labelledExamples generates three example test cases, one for each label:

>>> labelledExamples prop_delete*** Found example of count x xs == 00[][]*** Found example of count x xs == 10[0][]*** Found example of count x xs >= 25[5,5][5]+++ OK, passed 100 tests:78% count x xs == 021% count x xs == 1 1% count x xs >= 2

A variant of labelledExamples that returns a result.

A variant of labelledExamples that takes test arguments.

A variant of labelledExamples that takes test arguments and returns a result.

A modifier for testing functions.

prop :: Fun String Integer -> Bool
prop (Fn f) = f "banana" == f "monkey"
           || f "banana" == f "elephant"

A modifier for testing binary functions.

prop_zipWith :: Fun (Int, Bool) Char -> [Int] -> [Bool] -> Bool
prop_zipWith (Fn2 f) xs ys = zipWith f xs ys == [ f x y | (x, y) <- zip xs ys]

A modifier for testing ternary functions.

Generation of random shrinkable, showable functions.

To generate random values of type Fun a b, you must have an instance Function a.

See also applyFun, and Fn with GHC >= 7.8.

The class Function a is used for random generation of showable functions of type a -> b.

There is a default implementation for function, which you can use if your type has structural equality. Otherwise, you can normally use functionMap or functionShow.

Extracts the value of a function.

Fn is the pattern equivalent of this function.

prop :: Fun String Integer -> Bool
prop f = applyFun f "banana" == applyFun f "monkey"
      || applyFun f "banana" == applyFun f "elephant"

Extracts the value of a binary function.

Fn2 is the pattern equivalent of this function.

prop_zipWith :: Fun (Int, Bool) Char -> [Int] -> [Bool] -> Bool
prop_zipWith f xs ys = zipWith (applyFun2 f) xs ys == [ applyFun2 f x y | (x, y) <- zip xs ys]

Extracts the value of a ternary function. Fn3 is the pattern equivalent of this function.

Provides a Function instance for types with Bounded and Enum. Use only for small types (i.e. not integers): creates the list [minBound..maxBound]!

Provides a Function instance for types with Integral.

The basic building block for Function instances. Provides a Function instance by mapping to and from a type that already has a Function instance.

Provides a Function instance for types with RealFrac.

Provides a Function instance for types with Show and Read.

Provides a Function instance for types isomorphic to Void.

An actual Function Void instance is defined in quickcheck-instances.

A generator for values of type a.

The third-party packages QuickCheck-GenT and quickcheck-transformer provide monad transformer versions of Gen.

Generates a random element in the given inclusive range. For integral and enumerated types, the specialised variants of choose below run much quicker.

Generates a random element over the natural range of a.

A fast implementation of choose for bounded integral types.

A fast implementation of choose for enumerated types.

A fast implementation of choose for Int.

A fast implementation of choose for Integer.

Generates one of the given values. The input list must be non-empty.

Chooses one of the given generators, with a weighted random distribution. The input list must be non-empty.

Run a generator. The size passed to the generator is always 30; if you want another size then you should explicitly use resize.

Returns the size parameter. Used to construct generators that depend on the size parameter.

For example, listOf, which uses the size parameter as an upper bound on length of lists it generates, can be defined like this:

listOf :: Gen a -> Gen [a]
listOf gen = do
  n <- getSize
  k <- choose (0,n)
  vectorOf k gen

You can also do this using sized.

Takes a list of elements of increasing size, and chooses among an initial segment of the list. The size of this initial segment increases with the size parameter. The input list must be non-empty.

Generates an infinite list.

Generates a list of random length. The maximum length depends on the size parameter.

Generates a non-empty list of random length. The maximum length depends on the size parameter.

Randomly uses one of the given generators. The input list must be non-empty.

Overrides the size parameter. Returns a generator which uses the given size instead of the runtime-size parameter.

Generates some example values and prints them to stdout.

Generates some example values.

Adjust the size parameter, by transforming it with the given function.

Generates a random permutation of the given list.

Used to construct generators that depend on the size parameter.

For example, listOf, which uses the size parameter as an upper bound on length of lists it generates, can be defined like this:

listOf :: Gen a -> Gen [a]
listOf gen = sized $ \n ->
  do k <- choose (0,n)
     vectorOf k gen

You can also do this using getSize.

Generates a random subsequence of the given list.

Generates a value that satisfies a predicate.

Generates a value for which the given function returns a Just, and then applies the function.

Tries to generate a value that satisfies a predicate. If it fails to do so after enough attempts, returns Nothing.

Modifies a generator using an integer seed.

Generates a list of the given length.

ASCIIString: generates an ASCII string.

Blind x: as x, but x does not have to be in the Show class.

Fixed x: as x, but will not be shrunk.

InfiniteList xs _: guarantees that xs is an infinite list. When a counterexample is found, only prints the prefix of xs that was used by the program.

Here is a contrived example property:

prop_take_10 :: InfiniteList Char -> Bool
prop_take_10 (InfiniteList xs _) =
  or [ x == 'a' | x <- take 10 xs ]

In the following counterexample, the list must start with "bbbbbbbbbb" but the remaining (infinite) part can contain anything:

>>> quickCheck prop_take_10*** Failed! Falsified (after 1 test and 14 shrinks):"bbbbbbbbbb" ++ ...

Large x: by default, QuickCheck generates Ints drawn from a small range. Large Int gives you values drawn from the entire range instead.

Negative x: guarantees that x < 0.

NonNegative x: guarantees that x >= 0.

NonPositive x: guarantees that x <= 0.

NonZero x: guarantees that x /= 0.

Ordered xs: guarantees that xs is ordered.

Positive x: guarantees that x > 0.

PrintableString: generates a printable unicode String. The string will not contain surrogate pairs.

Shrink2 x: allows 2 shrinking steps at the same time when shrinking x

Shrinking _ x: allows for maintaining a state during shrinking.

Small x: generates values of x drawn from a small range. The opposite of Large.

Smart _ x: tries a different order when shrinking.

Sorted xs: guarantees that xs is sorted.

UnicodeString: generates a unicode String. The string will not contain surrogate pairs.

Conjunction: p1 .&&. p2 passes if both p1 and p2 pass.

Nondeterministic choice: p1 .&. p2 picks randomly one of p1 and p2 to test. If you test the property 100 times it makes 100 random choices.

Disjunction: p1 .||. p2 passes unless p1 and p2 simultaneously fail.

Like /=, but prints a counterexample when it fails.

Like ==, but prints a counterexample when it fails.

If a property returns Discard, the current test case is discarded, the same as if a precondition was false.

An example is the definition of ==>:

(==>) :: Testable prop => Bool -> prop -> Property
False ==> _ = property Discard
True  ==> p = property p

The class of properties, i.e., types which QuickCheck knows how to test. Typically a property will be a function returning Bool or Property.

Modifies a property so that it will be tested repeatedly. Opposite of once.

Check that all coverage requirements defined by cover and coverTable are met, using a statistically sound test, and fail if they are not met.

Ordinarily, a failed coverage check does not cause the property to fail. This is because the coverage requirement is not tested in a statistically sound way. If you use cover to express that a certain value must appear 20% of the time, QuickCheck will warn you if the value only appears in 19 out of 100 test cases - but since the coverage varies randomly, you may have just been unlucky, and there may not be any real problem with your test generation.

When you use checkCoverage, QuickCheck uses a statistical test to account for the role of luck in coverage failures. It will run as many tests as needed until it is sure about whether the coverage requirements are met. If a coverage requirement is not met, the property fails.

Example:

quickCheck (checkCoverage prop_foo)

Check coverage requirements using a custom confidence level. See stdConfidence.

An example of making the statistical test less stringent in order to improve performance:

quickCheck (checkCoverageWith stdConfidence{certainty = 10^6} prop_foo)

Reports how many test cases satisfy a given condition.

For example:

prop_sorted_sort :: [Int] -> Property
prop_sorted_sort xs =
  sorted xs ==>
  classify (length xs > 1) "non-trivial" $
  sort xs === xs

>>> quickCheck prop_sorted_sort+++ OK, passed 100 tests (22% non-trivial).

Attaches a label to a test case. This is used for reporting test case distribution.

collect x = label (show x)

For example:

prop_reverse_reverse :: [Int] -> Property
prop_reverse_reverse xs =
  collect (length xs) $
    reverse (reverse xs) === xs

>>> quickCheck prop_reverse_reverse+++ OK, passed 100 tests:7% 76% 35% 44% 6...

Each use of collect in your property results in a separate table of test case distribution in the output. If this is not what you want, use tabulate.

Take the conjunction of several properties.

Adds the given string to the counterexample if the property fails.

Checks that at least the given proportion of successful test cases belong to the given class. Discarded tests (i.e. ones with a false precondition) do not affect coverage.

Note: If the coverage check fails, QuickCheck prints out a warning, but the property does not fail. To make the property fail, use checkCoverage.

For example:

prop_sorted_sort :: [Int] -> Property
prop_sorted_sort xs =
  sorted xs ==>
  cover 50 (length xs > 1) "non-trivial" $
  sort xs === xs

>>> quickCheck prop_sorted_sort+++ OK, passed 100 tests; 135 discarded (26% non-trivial).Only 26% non-trivial, but expected 50%

Checks that the values in a given table appear a certain proportion of the time. A call to coverTable table [(x1, p1), ..., (xn, pn)] asserts that of the values in table, x1 should appear at least p1 percent of the time that table appears, x2 at least p2 percent of the time that table appears, and so on.

Note: If the coverage check fails, QuickCheck prints out a warning, but the property does not fail. To make the property fail, use checkCoverage.

Continuing the example from the tabular combinator...

data Command = LogIn | LogOut | SendMessage String deriving (Data, Show)
prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  wellFormed cmds LoggedOut ==>
  'tabulate' "Commands" (map (show . 'Data.Data.toConstr') cmds) $
    ...

...we can add a coverage requirement as follows, which checks that LogIn, LogOut and SendMessage each occur at least 25% of the time:

prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  wellFormed cmds LoggedOut ==>
  coverTable "Commands" [("LogIn", 25), ("LogOut", 25), ("SendMessage", 25)] $
  'tabulate' "Commands" (map (show . 'Data.Data.toConstr') cmds) $
    ... property goes here ...

>>> quickCheck prop_chatroom+++ OK, passed 100 tests; 2909 discarded:56% 017% 110% 2 6% 3 5% 4 3% 5 3% 7Commands (111 in total):51.4% LogIn30.6% SendMessage18.0% LogOutTable 'Commands' had only 18.0% LogOut, but expected 25.0%

Discards the test case if it does not complete within the given number of microseconds. This can be useful when testing algorithms that have pathological cases where they run extremely slowly.

Take the disjunction of several properties.

Indicates that a property is supposed to fail. QuickCheck will report an error if it does not fail.

Explicit universal quantification: uses an explicitly given test case generator.

Like forAll, but without printing the generated value.

Like forAll, but with an explicitly given show function.

Like forAll, but tries to shrink the argument for failing test cases.

Like forAllShrink, but without printing the generated value.