Learn · Intermediate

Property-based testing: test the rule, not just the examples

Property-based testing generates many inputs automatically and checks rules that should always be true, rather than writing only a few example-by-example assertions. It matters because serious bugs live in edge cases between the examples developers thought to write. A good property turns a vague expectation—“this parser works”—into a rule a computer can try hard to break.

A conventional unit test might say that sorting [3, 1, 2] produces [1, 2, 3]. That is useful, but tiny. A property-based test says: for every list the generator produces, sorting it must return a list in nondecreasing order, contain exactly the same multiset of elements, and return the same result if sorted again. Now the test framework can try empty lists, duplicate-heavy lists, negative numbers, extreme integers, and weird lengths without a developer inventing each one.

John Hughes and Koen Claessen popularized the approach through QuickCheck, a lightweight testing tool for Haskell. Their insight was that programmers often know broad truths about a function even when they cannot enumerate every expected output. A serializer and parser should round trip: decode(encode(x)) should equal x. Adding zero should change nothing. Combining a value with an identity should return the original. Converting temperature from Celsius to Fahrenheit and back should recover the original within a tolerance. These are properties.

The best analogy is quality control for a factory. Example-based tests inspect a few familiar products at the end of the line: this red cup, that blue cup, a large cup. Property-based testing describes what every cup must satisfy—does not leak, has the stated volume within tolerance, and fits the lid—and then asks the factory to produce many variants to inspect. It catches a flaw that only appears with the smallest size or an unusual material mix.

Input generation is the craft. A naive random generator creates mostly nonsense, which can be useful for robustness but may miss the valid structured cases where business logic fails. A good property-based generator knows the shape of a JSON object, a date, a bank transfer, or an HTTP request. It can deliberately vary optional fields, boundary sizes, unicode text, time zones, repeated items, and invalid-but-nearly-valid representations. The goal is not randomness for its own sake. It is broad, structured pressure against a claim.

When a test fails, shrinking makes the method practical. Suppose a complicated generated document causes a parser mismatch. Rather than hand the developer a thousand-line input, the framework repeatedly removes fields, shortens strings, and simplifies nested structures while preserving the failure. It may reduce the case to a three-character string and a single option flag. That smallest counterexample often teaches more than the original full failure. Shrinking is why property-based testing is not just “throw random data at it.”

Properties come in a few reusable families. Round-trip properties check that encode/decode, serialize/deserialize, encrypt/decrypt under the right keys, or compile/run preserve intended values. Invariants state something that remains true, such as a balance never becoming negative or a tree remaining ordered after insertion. Metamorphic properties compare results after a harmless transformation: a search result should not change when irrelevant whitespace is added, for example. Reference-model properties compare a fast implementation with a simple slow one on small inputs. Idempotence checks that doing something twice is the same as doing it once, such as normalizing a URL.

The technique has a direct security role. Many vulnerabilities occur at boundaries: an unexpected tab in a cookie attribute, an unusual certificate configuration, a nonstandard encoding, or a malformed header that passes one layer and confuses another. The day's six curl CVEs are a reminder that mature software can fail in narrow states and option combinations. Property tests will not discover every security vulnerability, but they are excellent at expressing parser, state-machine, and API invariants that should never be violated.

Property-based testing overlaps with fuzzing but is not identical. Fuzzing often mutates inputs broadly and watches for a crash, timeout, sanitizer warning, or other failure signal. Property-based testing starts with an explicit behavioral oracle: a rule the output must satisfy. The techniques complement each other. A fuzzer can explore strange bytes at scale; a property test can expose a quiet logical wrong answer even when nothing crashes. The fuzzing survey by Manès and colleagues maps the broader automated-testing landscape.

AI changes the economics but not the principle. A coding model can suggest properties, create generators, and translate a bug report into a regression test. It can also confidently invent a property that is false or incomplete. The defensible workflow is to have humans review the claimed invariant, let automation generate cases and shrink failures, then keep every discovered bug as a permanent example-based regression test. That is a healthy division of labor: people choose what must be true; machines become tireless adversaries.

The honest caveat is that a property can be wrong, weak, or expensive to check. “The output equals the correct answer” is a perfect property only if you already have an oracle. Random generation may also miss a rare structured pattern unless the generator knows to produce it. Property tests complement example tests, integration tests, code review, fuzzing, and—where needed—formal proof assistants.

The enduring habit is simple: whenever you write a test case, ask what general law made that case worth testing. If you can state the law, encode it. The next unexpected input may then find the bug before an attacker, user, or production incident does.

Ground Truth. (2026, September 4). Property-based testing: test the rule, not just the examples. Ground Truth. https://groundtruth.day/learn/property-based-testing.html

@misc{groundtruth:property-based-testing,
  title  = {Property-based testing: test the rule, not just the examples},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/property-based-testing.html}
}

Topics: testing · software-engineering · verification · security · fundamentals