Test.QuickCheck.Property

Execute the IORose bits of a rose tree, returning a tree constructed by MkRose.

Apply a function to the outermost MkRose constructor of a rose tree. The function must be total!

Wrap a rose tree in an exception handler.

Wrap the top level of a Prop in an exception handler.

Wrap all the Results in a rose tree in exception handlers.

Different kinds of callbacks

The result of a single test.

Adjust the test case size for a property, by transforming it with the given function.

Shrinks the argument to a property if it fails. Shrinking is done automatically for most types. This function is only needed when you want to override the default behavior.

Disables shrinking for a property altogether. Only quantification inside the call to noShrinking is affected.

Adds a callback

Adds the given string to the counterexample if the property fails.

Adds the given string to the counterexample if the property fails.

Performs an IO action after the last failure of a property.

Performs an IO action every time a property fails. Thus, if shrinking is done, this can be used to keep track of the failures along the way.

Prints out the generated test case every time the property is tested. Only variables quantified over inside the verbose are printed.

Note: for technical reasons, the test case is printed out after the property is tested. To debug a property that goes into an infinite loop, use within to add a timeout instead.

Prints out the generated test case every time the property fails, including during shrinking. Only variables quantified over inside the verboseShrinking are printed.

Note: for technical reasons, the test case is printed out after the property is tested. To debug a property that goes into an infinite loop, use within to add a timeout instead.

Indicates that a property is supposed to fail. QuickCheck will report an error if it does not fail.

Modifies a property so that it only will be tested once. Opposite of again.

Modifies a property so that it will be tested repeatedly. Opposite of once.

Configures how many times a property will be tested.

For example,

quickCheck (withMaxSuccess 1000 p)

will test p up to 1000 times.

Configures how many times a property is allowed to be discarded before failing.

For example,

quickCheck (withDiscardRatio 10 p)

will allow p to fail up to 10 times per successful test.

Configure the maximum number of times a property will be shrunk.

For example,

quickCheck (withMaxShrinks 100 p)

will cause p to only attempt 100 shrinks on failure.

Configure the maximum size a property will be tested at.

Return a value in the witnesses field of the Result returned by quickCheckResult. Witnesses are returned outer-most first.

In ghci, for example:

>>> [Wit x] <- fmap witnesses . quickCheckResult $ \ x -> witness x $ x == (0 :: Int)*** Failed! Falsified (after 2 tests):1>>> x1>>> :t xx :: Int

Check that all coverage requirements defined by cover and coverTable are met, using a statistically sound test, and fail if they are not met.

Ordinarily, a failed coverage check does not cause the property to fail. This is because the coverage requirement is not tested in a statistically sound way. If you use cover to express that a certain value must appear 20% of the time, QuickCheck will warn you if the value only appears in 19 out of 100 test cases - but since the coverage varies randomly, you may have just been unlucky, and there may not be any real problem with your test generation.

When you use checkCoverage, QuickCheck uses a statistical test to account for the role of luck in coverage failures. It will run as many tests as needed until it is sure about whether the coverage requirements are met. If a coverage requirement is not met, the property fails.

Example:

quickCheck (checkCoverage prop_foo)

Check coverage requirements using a custom confidence level. See stdConfidence.

An example of making the statistical test less stringent in order to improve performance:

quickCheck (checkCoverageWith stdConfidence{certainty = 10^6} prop_foo)

The standard parameters used by checkCoverage: certainty = 10^9, tolerance = 0.9. See Confidence for the meaning of the parameters.

Attaches a label to a test case. This is used for reporting test case distribution.

For example:

prop_reverse_reverse :: [Int] -> Property
prop_reverse_reverse xs =
  label ("length of input is " ++ show (length xs)) $
    reverse (reverse xs) === xs

>>> quickCheck prop_reverse_reverse+++ OK, passed 100 tests:7% length of input is 76% length of input is 35% length of input is 44% length of input is 6...

Each use of label in your property results in a separate table of test case distribution in the output. If this is not what you want, use tabulate.

Attaches a label to a test case. This is used for reporting test case distribution.

collect x = label (show x)

For example:

prop_reverse_reverse :: [Int] -> Property
prop_reverse_reverse xs =
  collect (length xs) $
    reverse (reverse xs) === xs

>>> quickCheck prop_reverse_reverse+++ OK, passed 100 tests:7% 76% 35% 44% 6...

Each use of collect in your property results in a separate table of test case distribution in the output. If this is not what you want, use tabulate.

Reports how many test cases satisfy a given condition.

For example:

prop_sorted_sort :: [Int] -> Property
prop_sorted_sort xs =
  sorted xs ==>
  classify (length xs > 1) "non-trivial" $
  sort xs === xs

>>> quickCheck prop_sorted_sort+++ OK, passed 100 tests (22% non-trivial).

Checks that at least the given proportion of successful test cases belong to the given class. Discarded tests (i.e. ones with a false precondition) do not affect coverage.

Note: If the coverage check fails, QuickCheck prints out a warning, but the property does not fail. To make the property fail, use checkCoverage.

For example:

prop_sorted_sort :: [Int] -> Property
prop_sorted_sort xs =
  sorted xs ==>
  cover 50 (length xs > 1) "non-trivial" $
  sort xs === xs

>>> quickCheck prop_sorted_sort+++ OK, passed 100 tests; 135 discarded (26% non-trivial).Only 26% non-trivial, but expected 50%

Collects information about test case distribution into a table. The arguments to tabulate are the table's name and a list of values associated with the current test case. After testing, QuickCheck prints the frequency of all collected values. The frequencies are expressed as a percentage of the total number of values collected.

You should prefer tabulate to label when each test case is associated with a varying number of values. Here is a (not terribly useful) example, where the test data is a list of integers and we record all values that occur in the list:

prop_sorted_sort :: [Int] -> Property
prop_sorted_sort xs =
  sorted xs ==>
  tabulate "List elements" (map show xs) $
  sort xs === xs

>>> quickCheck prop_sorted_sort+++ OK, passed 100 tests; 1684 discarded.List elements (109 in total): 3.7% 0 3.7% 17 3.7% 2 3.7% 6 2.8% -6 2.8% -7

Here is a more useful example. We are testing a chatroom, where the user can log in, log out, or send a message:

data Command = LogIn | LogOut | SendMessage String deriving (Data, Show)
instance Arbitrary Command where ...

There are some restrictions on command sequences; for example, the user must log in before doing anything else. The function valid :: [Command] -> Bool checks that a command sequence is allowed. Our property then has the form:

prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  valid cmds ==>
    ...

The use of ==> may skew test case distribution. We use collect to see the length of the command sequences, and tabulate to get the frequencies of the individual commands:

prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  wellFormed cmds LoggedOut ==>
  'collect' (length cmds) $
  'tabulate' "Commands" (map (show . 'Data.Data.toConstr') cmds) $
    ...

>>> quickCheckWith stdArgs{maxDiscardRatio = 1000} prop_chatroom+++ OK, passed 100 tests; 2775 discarded:60% 020% 115% 2 3% 3 1% 4 1% 5Commands (68 in total):62% LogIn22% SendMessage16% LogOut

Checks that the values in a given table appear a certain proportion of the time. A call to coverTable table [(x1, p1), ..., (xn, pn)] asserts that of the values in table, x1 should appear at least p1 percent of the time that table appears, x2 at least p2 percent of the time that table appears, and so on.

Note: If the coverage check fails, QuickCheck prints out a warning, but the property does not fail. To make the property fail, use checkCoverage.

Continuing the example from the tabular combinator...

data Command = LogIn | LogOut | SendMessage String deriving (Data, Show)
prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  wellFormed cmds LoggedOut ==>
  'tabulate' "Commands" (map (show . 'Data.Data.toConstr') cmds) $
    ...

...we can add a coverage requirement as follows, which checks that LogIn, LogOut and SendMessage each occur at least 25% of the time:

prop_chatroom :: [Command] -> Property
prop_chatroom cmds =
  wellFormed cmds LoggedOut ==>
  coverTable "Commands" [("LogIn", 25), ("LogOut", 25), ("SendMessage", 25)] $
  'tabulate' "Commands" (map (show . 'Data.Data.toConstr') cmds) $
    ... property goes here ...

>>> quickCheck prop_chatroom+++ OK, passed 100 tests; 2909 discarded:56% 017% 110% 2 6% 3 5% 4 3% 5 3% 7Commands (111 in total):51.4% LogIn30.6% SendMessage18.0% LogOutTable 'Commands' had only 18.0% LogOut, but expected 25.0%

Implication for properties: The resulting property holds if the first argument is False (in which case the test case is discarded), or if the given property holds. Note that using implication carelessly can severely skew test case distribution: consider using cover to make sure that your test data is still good quality.

Considers a property failed if it does not complete within the given number of microseconds.

Note: if the property times out, variables quantified inside the within will not be printed. Therefore, you should use within only in the body of your property.

Good: prop_foo a b c = within 1000000 ...

Bad: prop_foo = within 1000000 $ \a b c -> ...

Bad: prop_foo a b c = ...; main = quickCheck (within 1000000 prop_foo)

Discards the test case if it does not complete within the given number of microseconds. This can be useful when testing algorithms that have pathological cases where they run extremely slowly.

Explicit universal quantification: uses an explicitly given test case generator.

Like forAll, but with an explicitly given show function.

Like forAll, but without printing the generated value.

Like forAll, but tries to shrink the argument for failing test cases.

Like forAllShrink, but with an explicitly given show function.

Like forAllShrink, but without printing the generated value.

Nondeterministic choice: p1 .&. p2 picks randomly one of p1 and p2 to test. If you test the property 100 times it makes 100 random choices.

Conjunction: p1 .&&. p2 passes if both p1 and p2 pass.

Take the conjunction of several properties.

Disjunction: p1 .||. p2 passes unless p1 and p2 simultaneously fail.

Take the disjunction of several properties.

Like ==, but prints a counterexample when it fails.

Like /=, but prints a counterexample when it fails.

Checks that a value is total, i.e., doesn't crash when evaluated.