What?

What to check once the code is running: whether it works where it’s deployed, how it fails, and whether you find out. Third of three, after for Readers (the code) and for Builders (the build).

Category	Tool	Owns
Smoke testing	Actuator health + a few real requests	Did the release start and answer?
Canary release	Argo Rollouts / Flagger	A small share of traffic first, rolled back on data
Toggles	OpenFeature	Releasing a change separately from deploying it
Observability	Micrometer + Prometheus	The golden signals, measured not guessed
Reliability	SLOs and error budgets	How fast you’re allowed to change things
Chaos testing	Toxiproxy, Chaos Monkey for Spring Boot	That the fallbacks actually fall back
Security testing	OWASP ZAP, human pen testing	The running system, attacked on purpose

Why?

• The build proves the artifact, not the deployment. Config, network, credentials and real traffic exist only in production.
• Generated code fails in plausible ways. It handles the happy path and invents the error path. Only real failures show which one you got.
• Fast change needs fast rollback. If agents ship more often, every release has to be cheap to undo.

How?

Smoke testing: Ask the release one question before anyone else does

build.gradle.kts

implementation("org.springframework.boot:spring-boot-starter-actuator")

.github/workflows/deploy.yml

- run: curl --fail --retry 10 --retry-connrefused https://staging.example.com/actuator/health

• Smoke, not regression. Run a handful of checks in seconds: it started, it’s healthy, and one real request returns a real answer.
• Health is necessary but not enough. UP means the process and its dependencies answered. Add one end-to-end request that exercises the main path.
• Fail the deploy, not a dashboard. A failed smoke test stops promotion and rolls back automatically.

Canary release: Let a few users find it first

rollout.yaml

strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }

• Compare, don’t eyeball. Promote only if the canary’s error rate and latency match the stable version’s. Argo Rollouts’ AnalysisTemplate and Flagger both query Prometheus for this.
• Roll back on the data. The rollback should be automatic and decided by the metrics. A human watching a graph is the slow path.
• Small is the point. At 5%, a bad release costs 5% of requests for 10 minutes.

Toggles: Ship dark

src/main/java/…/Menu.java

var flags = OpenFeatureAPI.getInstance().getClient();
if (flags.getBooleanValue("croissants", false)) { // off until someone means it
  return todaysBakes.with(croissants());
}

• Deploy and release are separate acts. A change to a working system ships off by default and reaches users only when someone flips the flag deliberately, and the flip can be undone.
• OpenFeature is the vendor-neutral API (dev.openfeature:sdk). The flag provider can change without changing the code.
• Flags are debt. Give each flag an owner and a removal date. A flag that’s been on everywhere for a month is dead code with a switch.

Observability: Measure the four golden signals

build.gradle.kts

runtimeOnly("io.micrometer:micrometer-registry-prometheus")

• Latency, traffic, errors, saturation. Every service exposes them. Spring Boot’s Actuator serves them at /actuator/prometheus, with HTTP server metrics built in.
• Measure latency as percentiles, not averages. p99 is what the slowest users feel.
• Measure what the agent can’t see. Tag the model calls: tokens, latency and failures per model. Spring AI’s observations do this.

Reliability: Let the error budget set the pace

• SLI, SLO, budget. The SLI is what you measure (the share of requests under 500 ms that succeed). The SLO is the target (99.5% over 28 days). The budget is what’s left: 0.5%, about 3 hours 22 minutes.
• Budget left: ship. Budget spent: stabilise. The rule is agreed in advance, not argued during an incident.
• Alert on how fast the budget is burning, not on single errors.

Chaos testing: Break it on purpose

src/test/java/…/SlowOvenTest.java

@Container
static ToxiproxyContainer toxiproxy = new ToxiproxyContainer("ghcr.io/shopify/toxiproxy:2.12.0");
 
@Test
void aSlowOvenStillSellsYesterdaysBread() throws IOException {
  var toxics = new ToxiproxyClient(toxiproxy.getHost(), toxiproxy.getControlPort());
  var oven = toxics.createProxy("oven", "0.0.0.0:8666", "oven:8080");
  oven.toxics().latency("slow-oven", ToxicDirection.DOWNSTREAM, 5_000);
  // assert the counter serves yesterday's loaf within 2 seconds
}

• Start in tests. Toxiproxy (org.testcontainers:testcontainers-toxiproxy) adds latency or cuts connections between the service and a real dependency. Every timeout and fallback gets a test that triggers it.
• Then in staging, then game days. Chaos Monkey for Spring Boot injects latency and exceptions into running beans. Game days do the same with people watching and a rollback ready.
• Have a hypothesis. “When the oven is slow, the counter sells yesterday’s bread within 2 seconds.” A chaos experiment without one is just an outage.

Security testing: Attack the running system

.github/workflows/nightly.yml

- run: docker run --rm ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://staging.example.com

• Run DAST every night against staging. The ZAP baseline scan spiders the site and reports passive findings, such as missing headers, leaky errors and insecure cookies, without attacking.
• Pen testing is people. A scoped human test before a major release and on a fixed schedule finds what scanners can’t: broken authorisation, business-logic abuse, chained flaws.
• Agents are an attack surface too. If the service calls a model with user input, test for prompt injection and data exfiltration like any other injection.

The run book

• Every alert links to a page saying what to do. An alert nobody knows how to act on is noise.
• Practise rollback before you need it. A rollback path that has never been run doesn’t work.