Criterion.Types

Top-level benchmarking configuration.

Control the amount of information displayed.

A pure function or impure action that can be benchmarked. The Int64 parameter indicates the number of times to run the given function or action.

Specification of a collection of benchmarks and environments. A benchmark may consist of:

• An environment that creates input data for benchmarks, created with env.

• A single Benchmarkable item with a name, created with bench.

• A (possibly nested) group of Benchmarks, created with bgroup.

A collection of measurements made while benchmarking.

Measurements related to garbage collection are tagged with GC. They will only be available if a benchmark is run with "+RTS -T".

Packed storage. When GC statistics cannot be collected, GC values will be set to huge negative values. If a field is labeled with "GC" below, use fromInt and fromDouble to safely convert to "real" values.

Convert a (possibly unavailable) GC measurement to a true value. If the measurement is a huge negative number that corresponds to "no data", this will return Nothing.

Convert from a true value back to the packed representation used for GC measurements.

Convert a (possibly unavailable) GC measurement to a true value. If the measurement is a huge negative number that corresponds to "no data", this will return Nothing.

Convert from a true value back to the packed representation used for GC measurements.

Field names and accessors for a Measured record.

Field names in a Measured record, in the order in which they appear.

Normalise every measurement as if measIters was 1.

(measIters itself is left unaffected.)

Run a benchmark (or collection of benchmarks) in the given environment. The purpose of an environment is to lazily create input data to pass to the functions that will be benchmarked.

A common example of environment data is input that is read from a file. Another is a large data structure constructed in-place.

Motivation. In earlier versions of criterion, all benchmark inputs were always created when a program started running. By deferring the creation of an environment when its associated benchmarks need the its, we avoid two problems that this strategy caused:

• Memory pressure distorted the results of unrelated benchmarks. If one benchmark needed e.g. a gigabyte-sized input, it would force the garbage collector to do extra work when running some other benchmark that had no use for that input. Since the data created by an environment is only available when it is in scope, it should be garbage collected before other benchmarks are run.

• The time cost of generating all needed inputs could be significant in cases where no inputs (or just a few) were really needed. This occurred often, for instance when just one out of a large suite of benchmarks was run, or when a user would list the collection of benchmarks without running any.

Creation. An environment is created right before its related benchmarks are run. The IO action that creates the environment is run, then the newly created environment is evaluated to normal form (hence the NFData constraint) before being passed to the function that receives the environment.

Complex environments. If you need to create an environment that contains multiple values, simply pack the values into a tuple.

Lazy pattern matching. In situations where a "real" environment is not needed, e.g. if a list of benchmark names is being generated, a value which throws an exception will be passed to the function that receives the environment. This avoids the overhead of generating an environment that will not actually be used.

The function that receives the environment must use lazy pattern matching to deconstruct the tuple (e.g., ~(x, y), not (x, y)), as use of strict pattern matching will cause a crash if an exception-throwing value is passed in.

Example. This program runs benchmarks in an environment that contains two values. The first value is the contents of a text file; the second is a string. Pay attention to the use of a lazy pattern to deconstruct the tuple in the function that returns the benchmarks to be run.

setupEnv = do
  let small = replicate 1000 (1 :: Int)
  big <- map length . words <$> readFile "/usr/dict/words"
  return (small, big)

main = defaultMain [
   -- notice the lazy pattern match here!
   env setupEnv $ \ ~(small,big) -> bgroup "main" [
   bgroup "small" [
     bench "length" $ whnf length small
   , bench "length . filter" $ whnf (length . filter (==1)) small
   ]
 ,  bgroup "big" [
     bench "length" $ whnf length big
   , bench "length . filter" $ whnf (length . filter (==1)) big
   ]
 ] ]

Discussion. The environment created in the example above is intentionally not ideal. As Haskell's scoping rules suggest, the variable big is in scope for the benchmarks that use only small. It would be better to create a separate environment for big, so that it will not be kept alive while the unrelated benchmarks are being run.

Same as env, but but allows for an additional callback to clean up the environment. Resource clean up is exception safe, that is, it runs even if the Benchmark throws an exception.

Create a Benchmarkable where a fresh environment is allocated for every batch of runs of the benchmarkable.

The environment is evaluated to normal form before the benchmark is run.

When using whnf, whnfIO, etc. Criterion creates a Benchmarkable whichs runs a batch of N repeat runs of that expressions. Criterion may run any number of these batches to get accurate measurements. Environments created by env and envWithCleanup, are shared across all these batches of runs.

This is fine for simple benchmarks on static input, but when benchmarking IO operations where these operations can modify (and especially grow) the environment this means that later batches might have their accuracy effected due to longer, for example, longer garbage collection pauses.

An example: Suppose we want to benchmark writing to a Chan, if we allocate the Chan using environment and our benchmark consists of writeChan env (), the contents and thus size of the Chan will grow with every repeat. If Criterion runs a 1,000 batches of 1,000 repeats, the result is that the channel will have 999,000 items in it by the time the last batch is run. Since GHC GC has to copy the live set for every major GC this means our last set of writes will suffer a lot of noise of the previous repeats.

By allocating a fresh environment for every batch of runs this function should eliminate this effect.

Same as perBatchEnv, but but allows for an additional callback to clean up the environment. Resource clean up is exception safe, that is, it runs even if the Benchmark throws an exception.

Create a Benchmarkable where a fresh environment is allocated for every run of the operation to benchmark. This is useful for benchmarking mutable operations that need a fresh environment, such as sorting a mutable Vector.

As with env and perBatchEnv the environment is evaluated to normal form before the benchmark is run.

This introduces extra noise and result in reduce accuracy compared to other Criterion benchmarks. But allows easier benchmarking for mutable operations than was previously possible.

Same as perRunEnv, but but allows for an additional callback to clean up the environment. Resource clean up is exception safe, that is, it runs even if the Benchmark throws an exception.

Construct a Benchmarkable value from an impure action, where the Int64 parameter indicates the number of times to run the action.

Create a single benchmark.

Group several benchmarks together under a common name.

Add the given prefix to a name. If the prefix is empty, the name is returned unmodified. Otherwise, the prefix and name are separated by a '/' character.

Retrieve the names of all benchmarks. Grouped benchmarks are prefixed with the name of the group they're in.

Outliers from sample data, calculated using the boxplot technique.

A description of the extent to which outliers in the sample data affect the sample mean and standard deviation.

Analysis of the extent to which outliers in a sample affect its standard deviation (and to some extent, its mean).

Results of a linear regression.

Data for a KDE chart of performance.

Report of a sample analysis.

Result of a bootstrap analysis of a non-parametric sample.