package ppx_quick_test

You can search for identifiers within the package.

in-package search v0.2.0

Spiritual equivalent of let%expect_test, but for property based tests as an ergonomic wrapper to write quickcheck tests.

Install

Dune Dependency

github.com Readme MIT License Edit opam file Versions (1)

Authors

J

Jane Street Group, LLC

Maintainers

J

Jane Street developers

Sources

v0.17.0.tar.gz

sha256=d9556f991f7a75fb534a4a808fed3a18d0fd7ed55ecaa9a9bfefe9867d73b0d8

Description

Part of the Jane Street's PPX rewriters collection.

Published: 26 May 2024

README

README.mdx

ppx_quick_test
==============

let%quick_test
==============

The `let%expect_test` equivalent for quickcheck - a syntax extension for writing
quickcheck tests. For example:

```ocaml skip
open! Core

let%quick_test "int comparison is transitive" =
  fun (a : int) (b : int) (c : int) -> assert (if a > b && b > c then a > c else true)
;;
```

## New syntactic constructs

The following construct is now a valid structure item:

```ocaml skip
let%quick_test "name" = fun (<param> : <type>) ... (<param> : <type>) -> <body>
```

- We may write `_` instead of `"name"` for anonymous tests
- There must be at least one provided parameter
- Types must be provided for all parameters
- Types must implement `sexp_of_t`
- The `<body>` tests some property of the supplied parameters.
- The `<body>` returns a `unit` on a success, and raises on a failure. If 
  `ppx_quick_test_async` is opened, it returns a `unit Deferred.t`.


## How do I get started?

Add `ppx_quick_test` as a preprocessor to your build file. For example

```lisp skip
(library ((name your_lib) (preprocess (pps (ppx_quick_test)))))
```

NOTE: If you are using dune outside of Jane Street, the syntax is:

```lisp skip
(library (name your_lib) (preprocess (pps ppx_quick_test)))
```

## Automatic Regression Tests


If quick test finds a failing input, ppx_quick_test can "remember" failing inputs with
`[@remember_failures]`. Using this requires your type to additionally have `t_of_sexp`.

This feature is opt-in as it might not make sense to remember every test case that has
failed, specially the failures that arise as you are writing your generator. After you've
finished writing your generator, and would like to remember future failures, you can add
`[@remember_failures]`.

After the attribute is added, every time a failures occurs, a .corrected file will be
produced that adds the failing input next to the `[@remember_failures]` attribute. The
regression test is added _after_ you accept the corrected test output. Details are
provided below, in the Attributes section.

## Attributes

### Test Scoped Attributes

Test scoped attributes are used to configure a quick test. They are attached to the
test name. For example:

```ocaml skip
let%quick_test "name" [@first] [@second] =
    fun  (param1 : type1) (param2 : type2) -> <body>
let%quick_test _ [@first] [@second] =
    fun  (param1 : type1) (param2 : type2) -> <body>
```

Available test scoped attributes are:

`[@config <EXPR>]`

- Sets the quickcheck configuration to use in the test
- EXPR Type    = `Base_quickcheck.Test.Config.t`
- EXPR Default = `Base_quickcheck.Test.default_config`

<!-- $MDX file=examples/examples.ml,part=config -->
```ocaml
let%quick_test (_ [@config
                  { Base_quickcheck.Test.default_config with test_count = 10_000 }])
  =
  fun (a : int) (b : int) -> assert (a + b = b + a)
;;
```

`[@cr <EXPR>]`

* Sets the CR level to print when a test is failing
* EXPR Type    = `Expect_test_helpers_base.CR.t`
* EXPR Default = `Expect_test_helpers_base.CR.CR`


`[@examples <EXPR>]`

* User provided example inputs to test your property
* EXPR Type    = `(t1 * ...* tn) list`
* EXPR Default = `[]`

<!-- $MDX file=examples/examples.ml,part=examples -->
```ocaml
let%quick_test (_ [@examples [ 1, 0; 0, 0 ]]) =
  fun (a : int) (b : int) -> assert (a + b = b + a)
;;
```


`[@rembember_failures <SEXP 1> ... <SEXP N>]`

* Machine provided example inputs to test your property
* Requires `t_of_sexp` to convert the examples to the concrete types
* Sexp examples are autogenerated when a test fails from a non-example input
* Corrected files are produced containing the generated sexp example
* SEXP Type = `string` literal representing a `Sexp.t` representing a `(t1 * ... * tn)`

<!-- MDX file=examples/examples.ml,part=remember_failures -->
```ocaml skip
let%quick_test _ [@remember_failures {|(1 0)|} {|(0 0)|}] =
  fun (a : int) (b : int) -> assert (a + b = b + a)
;;
```

### Type Scoped Attributes

Type scoped attributes used to configure a property of a specific input type.
They are attached to the parameter's type. For example:

```ocaml skip
let%quick_test _ =
    fun  (param1 : type1 [@first] [@second])
         (param2 : type2 [@third])
         -> <body>
```

Available type scoped attributes are:

`[@generator <EXPR>]`

* Use when you want an alternative/custom quickcheck generator or
  when you have no quickcheck generator avilable on your types
* EXPR Type    = `your_type Base_quickcheck.Generator.t`
* EXPR Default = `[%quickcheck.generator: your_type]`

```ocaml skip
let%quick_test _ = fun (a : int [@generator Int.gen_incl 0 10])
                       (b : int)
                       -> assert (a + b = b + a)
```


`[@shrinker <EXPR>]`

* Use when you want to use an alternative/custom quickcheck shrinker
* EXPR Type    = `your_type Base_quickcheck.Shrinker.t`
* EXPR Default = `[%quickcheck.shrinker: your_type]`
* NOTE: If you do not want a shrinker, you can default to `Base_quickcheck.Shrinker.atomic`

## Examples

### Hello World

<!-- $MDX file=examples/examples.ml,part=basic -->
```ocaml
let%quick_test _ = fun (a : int) (b : int) -> assert (a + b = b + a)
```

### Kitchen Sink

The following examples shows the usage of all available attributes set to their
default values.

<!-- MDX file=examples/examples.ml,part=all_attributes -->
```ocaml skip
let%quick_test (_ [@config Base_quickcheck.Test.default_config]
                [@cr Expect_test_helpers_base.CR.CR]
                [@hide_positions false]
                [@generator [%quickcheck.generator: int * int * int]]
                [@shrinker [%quickcheck.shrinker: int * int * int]]
                [@examples []]
                (* [@remember_failures] *))
  =
  fun (a : int) (b : int) (c : int) -> assert (if a < b && b < c then a < c else true)
;;
```

### Using Async

If you'd like to write a quick_test that uses Async, you can use the library
`ppx_quick_test_async` by adding it to your jbuild's `libraries` field and opening it at
the top of your test file. This will make your `let%quick_test` need to return a `unit
Deferred.t` instead of a `unit`.

## FAQ

Q: Why do the input types appear as a tuple in some contexts (e.g. quickcheck generation
and example passing) even though we provide a function that takes in multiple parameters
(instead of a single parameter of the tuple-typed)?

A: We translate the provided multi-parameter function to a single parameter function that takes
in the tuple of input types. We perform this translation to make the function syntax easier to
remember and read (no commas required between function arguments). Under the hood, we are always
dealing with the tupled version of the input arguments.


Q: How do I resolve error: `Error: unbound value <your_type>.t_of_sexp`?

A: This is caused by having a (likely autogenerated) sexp example in `[@remember_failures <SEXP>]`
   that can't be parsed because `<your_type>` doesn't have an `t_of_sexp` function.

   Options:
   - Add `t_of_sexp` to `<your_type>` by using `[@@deriving sexp]` or specifying it manually
   - Move the failing example to `[@examples]` by replacing `[@remember_failures <SEXP>]` with
   `[@examples [<VALUE REPRESENTED BY SEXP>]]`


Q : How do I resolve error: `Error: unbound value <your_type>.quickcheck_generator`?

A: This is caused by a failure to generate a quickcheck generator on `your_type`.

   Options:
   - Add `quickcheck_generator` to `<your_type>` by using `[@@deriving quickcheck ~generator]` or specifying it manually
     (`[%quickcheck.generator : <type>]` could be helpful)
   - Provide a generator to use using the `[@generator]` attribute (see the Attributes section)


Q : How do I resolve error: `Error: unbound value <your_type>.quickcheck_shrinker`?


A: This is caused by a failure to generate a quickcheck shrinker on `your_type`.
   Options:
   - Add `quickcheck_shrinker` to `<your_type>` by using `[@@deriving quickcheck ~shrinker]` or specifying it manually
     (`[%quickcheck.shrinker : <type>]` could be helpful)
   - Provide shrinker to use using the `[@shrinker]` attribute (see the Attributes section)
   - Disable shrinking by providing the atomic shrinker: `[@shrinker Base_quickcheck.Shrinker.atomic]`


Q : Why don't you support syntax `let%quick_test "name" (a : int) (b : int) = ...` instead of
`let%quick_test "name" = fun (a : int) (b : int) -> ...`?

A: This is caused by a limitation in the underlying parser - the former syntax is unable to be parsed, even
though it is a valid AST representation.


## Expansion

The ppx below:

```ocaml skip
let%quick_test "relation is transitive" (a : int) (b : int) (c : int) =
assert (if relation a b && relation b c then relation a c else true)
```

will expand into something like:

```ocaml skip
let%expect_test "relation is transitive" =
  Expect_test_helpers_base.quickcheck_m
    [%here]
    (module struct
      type t = int * int * int [@@deriving quickcheck, sexp_of]
    end)
    (fun (a, b, c) ->
      assert (if relation a b && relation b c then relation a c else true))
```

Note: Rather than using `quickcheck_m` we use the a custom method in the runtime library with
similar functionality.


## Improvement Ideas

* Find all instances of `let%quick_test`s in the tree.  Extract them and spend time running
  them outside the context of a build step using additional randomized inputs (maybe integration
  AFL)?

Dependencies (15)

ppxlib >= "0.28.0"
dune >= "3.11.0"
ppx_sexp_message >= "v0.17" & < "v0.18"
ppx_sexp_conv >= "v0.17" & < "v0.18"
ppx_jane >= "v0.17" & < "v0.18"
ppx_here >= "v0.17" & < "v0.18"
ppx_expect >= "v0.17" & < "v0.18"
expect_test_helpers_core >= "v0.17" & < "v0.18"
core_kernel >= "v0.17" & < "v0.18"
core >= "v0.17" & < "v0.18"
base_quickcheck >= "v0.17" & < "v0.18"
base >= "v0.17" & < "v0.18"
async_kernel >= "v0.17" & < "v0.18"
async >= "v0.17" & < "v0.18"
ocaml >= "5.1.0"

Dev Dependencies

None

Used by (1)

bonsai >= "v0.17.0"

Conflicts

None

On This Page

Description
Readme
Dependencies (15)
Dev Dependencies
Used by (1)
Conflicts