## Definition **Continuous validation** is the practice of running all executable specifications automatically on every change, using a dedicated CI environment, so that functional regression is detected within minutes rather than at the end of a release cycle. Gojko Adzic treats it in [[Specification by Example - Gojko Adzic]] as the operational discipline that gives [[Living Documentation]] its guarantee: the documentation cannot lie because the system runs it. ## Why it is harder than unit-test CI Executable specifications are functional acceptance tests. Unlike unit tests they often require a deployed application, a real or realistic database, and potentially external services. This creates three failure categories that unit CI does not face: - **Environmental instability** — Tests pass locally but fail in CI because of network issues, timing, or shared external services. - **Slow feedback** — A full acceptance suite can run for hours; associating a failure with a specific commit becomes hard. - **Managing known failures** — At scale (tens of thousands of checks) some failures cannot be fixed immediately and must be tracked without polluting the main suite. ## Making validation reliable The primary reliability interventions from Adzic's research: - **Dedicated environment** — Never share the validation environment with demos or manual testing. Unplanned data changes are the second most common source of instability. - **Fully automated deployment** — If a human must intervene at any step to deploy the system to the validation environment, the validation is not continuous. - **Test doubles for external systems** — Replace unreliable or unavailable external services with controlled fakes that read from version-controlled configuration files. Periodically validate that the double still matches the real service contract. - **Business time** — Introduce a configurable clock abstraction into the production system. Tests that depend on end-of-day jobs, expiring contracts, or cache headers can then control the system's perception of time without waiting for the real calendar. - **Wait for events, not elapsed time** — A `Wait 10 seconds` step is a slow and brittle workaround. Poll a queue, a database flag, or a service endpoint instead; the test finishes as soon as the system does. ## Getting fast feedback at scale When a suite grows beyond what can run in a single pass, partition it: - **Modular packs by functional area** — Separate accounting tests from payment tests so that a failure in one area does not block development in another. - **Current iteration pack** — Group the specifications for features in active development. This pack may include failing tests for unimplemented features and should run after every change in the iteration. - **Quick / slow split** — A subset of fast tests runs on every commit; the full suite runs overnight or on demand. - **Parallel execution** — Multiple environments running independent packs can reduce a six-hour suite to twenty minutes. ## Managing failing tests Adzic's nuanced position: at scale, some failures cannot be fixed immediately. The response is not to disable them quietly (which lets them be forgotten) but to move them to a *known regression failures pack* that runs separately. This pack is monitored: if it grows, it triggers a process decision. Each failing test in the pack should have an associated ticket. Automated checks that list which tests are disabled and link them to tickets (Adzic cites Iowa Student Loan's FitNesse approach) prevent the pack from becoming a dumping ground. ## Relationship to Living Documentation and SDD Continuous validation is the mechanism that keeps [[Living Documentation]] honest. In [[Spec-Driven Development]] it is also the feedback loop for generated code: once an agent produces an implementation, the continuous validation suite determines whether the generated behaviour matches the specification. A red suite is the signal to iterate; a green suite is the definition of done. ## Related - [[Living Documentation]] - [[Executable Acceptance Criterion]] - [[Gherkin Scenario]] - [[Specification by Example]] - [[Spec-Driven Development]] ## Sources - [[Specification by Example - Gojko Adzic]]