Bytes, Rust, Code That Sparks Joy for Operators

What?

What to check once a Rust service is running: whether it works where it’s deployed, how it fails, and whether you find out. Third of three, after for Readers (the code) and for Builders (the build).

Category	Tool	Owns
Smoke testing	Your own `/health` routes + a few real requests	Did the release start and answer?
Shutdown	`with_graceful_shutdown` + SIGTERM	Draining before the rollout kills it
Panics	`CatchPanicLayer`, a panic hook	A crash you hear about
Canary release	Argo Rollouts / Flagger	A small share of traffic first, rolled back on data
Toggles	OpenFeature (`open-feature`)	Releasing a change separately from deploying it
Observability	OpenTelemetry, with `tracing`; tokio-metrics	The golden signals, and a starved runtime
Reliability	SLOs and error budgets	How fast you’re allowed to change things
Chaos testing	Toxiproxy, `fail` fail points	That the fallbacks actually fall back
Security testing	ZAP, cargo-fuzz, human pen testing	The running system and its parsers, attacked on purpose

Why?

The build proves the artifact, not the deployment. Config, network, credentials and real traffic exist only in production.
Generated code fails in plausible ways. It handles the happy path and invents the error path. Only real failures show which one you got.
Fast change needs fast rollback. If agents ship more often, every release has to be cheap to undo.
No framework does this for you. Health checks, graceful shutdown, telemetry and panic handling are each code you write in a Rust service. Each is code an agent can leave out, and the service still compiles and passes its tests.

How?

Smoke testing: Ask the release one question before anyone else does

src/main.rs

let app = Router::new()
    .route("/health/live", get(|| async { StatusCode::OK })) // the process answers
    .route("/health/ready", get(ready)); // and so does everything it needs

.github/workflows/deploy.yml

- run: curl --fail --retry 10 --retry-connrefused https://staging.example.com/health/ready

Smoke, not regression. Run a handful of checks in seconds: it started, it’s healthy, and one real request returns a real answer.
Write two health routes, and keep them apart. live says the process answers, and nothing more. ready checks the dependencies: the database, the model, the queue. If live checks the database, the orchestrator restarts every pod when the database blips, which turns a dependency outage into your outage.
Health is necessary but not enough. Add one end-to-end request through the main path.
Fail the deploy, not a dashboard. A failed smoke test stops promotion and rolls back automatically.

Shutdown: Drain before you die

src/main.rs

axum::serve(listener, app)
    .with_graceful_shutdown(shutdown_signal()) // stop accepting, finish what's in flight
    .await
    .expect("the server runs");

src/main.rs

async fn shutdown_signal() {
    let ctrl_c = async { signal::ctrl_c().await.expect("the Ctrl-C handler installs") };
    let terminate = async {
        signal::unix::signal(signal::unix::SignalKind::terminate())
            .expect("the SIGTERM handler installs")
            .recv()
            .await;
    };
    tokio::select! { () = ctrl_c => {}, () = terminate => {} }
}

Every rollout sends SIGTERM to the old pods. Without a handler, the process dies at once and drops every request in flight. Kubernetes waits 30 seconds by default before SIGKILL, so there’s time to drain, but only if you wire it up.
A missing handler makes every deploy look bad. The error spike lands during the canary step, so the analysis blames the new version for what the old one did on the way out.
Bound the drain. with_graceful_shutdown waits for every connection to finish, however long that takes. Wrap it in a tokio::time::timeout shorter than the grace period, so a slow request doesn’t turn into a SIGKILL.
Flush the telemetry last. OpenTelemetry exports in batches, so the last spans and metrics before a shutdown are lost unless you call the providers’ shutdown() after the server stops.

Panics: Make a crash something you hear about

src/main.rs

std::panic::set_hook(Box::new(|info| tracing::error!(%info, "panic"))); // into the logs, not just stderr
 
let app = Router::new()
    // …
    .layer(CatchPanicLayer::new()); // a panicking handler answers 500, which the metrics count

A panic in a handler is quiet by default. The connection’s task dies, the client sees a dropped connection rather than a 500, the server carries on, and the message goes to stderr, outside your structured logs. CatchPanicLayer from tower-http turns it into a 500 your error rate counts. The hook puts it in your logs and traces.
A panic in a spawned task waits in its JoinHandle. Drop the handle and nobody ever finds out. Await every handle, or use a JoinSet, and log the JoinError.
A panic while holding a std::sync::Mutex poisons it. Every later .lock().unwrap() panics too, so one bug turns into a stream of panics. Handle the poisoned lock, or use a mutex that doesn’t poison, such as parking_lot’s.
panic = "abort" is a choice, not a fix. It makes every panic end the process, so the orchestrator restarts it, which is loud. But it takes every in-flight request with it, and CatchPanicLayer stops working, because there’s no unwinding left to catch. Services usually keep the default, unwinding, and catch and count panics instead.

Canary release: Let a few users find it first

rollout.yaml

strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }

Compare, don’t eyeball. Promote only if the canary’s error rate and latency match the stable version’s. Argo Rollouts’ AnalysisTemplate and Flagger both query the metrics for this.
Roll back on the data. The rollback should be automatic and decided by the metrics. A human watching a graph is the slow path.
Small is the point. At 5%, a bad release costs 5% of requests for 10 minutes.
The comparison is only as good as what’s counted. Without a shutdown handler and counted panics, it’s measuring the wrong thing.

Toggles: Ship dark

src/menu.rs

let flags = OpenFeature::singleton().await.create_client();
let croissants = flags
    .get_bool_value("croissants", None, None)
    .await
    .unwrap_or(false); // off until someone means it, and off if the provider is down

Deploy and release are separate acts. A change to a working system ships off by default and reaches users only when someone turns the flag on deliberately, and turning it on can be undone.
OpenFeature is the vendor-neutral API. The flag provider can change without changing the code. The Rust SDK (open-feature 0.3) is pre-1.0, so expect its API to change.
The default is the safe answer. unwrap_or(false) means a flag-service outage switches features off, not on.
Flags are debt. Give each flag an owner and a removal date. A flag that’s been on everywhere for a month is dead code with a switch.

Observability: Measure the four golden signals, and the runtime

src/telemetry.rs

let resource = Resource::builder().with_service_name("bakery").build();
let tracer_provider = SdkTracerProvider::builder()
    .with_batch_exporter(SpanExporter::builder().with_tonic().build()?)
    .with_resource(resource.clone())
    .build();
let meter_provider = SdkMeterProvider::builder()
    .with_periodic_exporter(MetricExporter::builder().with_tonic().build()?)
    .with_resource(resource)
    .build();
global::set_meter_provider(meter_provider.clone());
 
tracing_subscriber::registry()
    .with(tracing_subscriber::fmt::layer())
    .with(tracing_opentelemetry::layer().with_tracer(tracer_provider.tracer("bakery")))
    .init();

src/telemetry.rs

let duration = global::meter("bakery")
    .f64_histogram("http.server.request.duration")
    .with_unit("s")
    .with_boundaries(vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0])
    .build();

OpenTelemetry is the standard. One pipeline for traces, metrics and logs, sent over OTLP to a collector, and from there to whatever backend you use. Prometheus can take OTLP directly, or scrape the collector.
Use tracing for the code, and OpenTelemetry for the export. tracing is how Rust code is instrumented: spans and events, which libraries such as tower-http, sqlx and reqwest already emit. tracing-opentelemetry sends them on as OpenTelemetry spans. Instrument with tracing, not the OpenTelemetry API, so the instrumentation outlives a change of exporter.
Latency, traffic, errors, saturation. Every service exposes them. tower-http’s TraceLayer gives a span per request. Record the request duration as a histogram, named by the semantic conventions so dashboards and alerts work unchanged.
Set the histogram boundaries. OpenTelemetry’s default boundaries go from 0 to 10,000 and suit milliseconds. The conventions record seconds, so with the defaults nearly every request lands in the first bucket and the p99 means nothing.
Measure latency as percentiles, not averages. p99 is what the slowest users feel.
In async Rust, saturation includes a starved runtime. A blocking call inside an async fn (std::fs, a sync HTTP client, a long CPU loop) holds a worker thread, and latency rises for every request on it, not just the slow one. Generated code does this often, and nothing in the build catches it. OpenTelemetry can’t see inside the runtime. tokio-metrics can: how busy the workers are and how long tasks take to poll. Record its numbers as OpenTelemetry gauges, so they arrive with everything else. tokio-console shows which task is the culprit. The fix is spawn_blocking.
Measure what the agent can’t see. Tag the model calls: tokens, latency and failures per model.

Reliability: Let the error budget set the pace

SLI, SLO, budget. The SLI is what you measure (the share of requests under 500 ms that succeed). The SLO is the target (99.5% over 28 days). The budget is what’s left: 0.5%, about 3 hours 22 minutes.
Budget left: ship. Budget spent: stabilise. The rule is agreed in advance, not argued during an incident.
Alert on how fast the budget is burning, not on single errors.
Count caught panics as errors. A 500 from CatchPanicLayer spends the budget like any other.

Chaos testing: Break it on purpose

src/oven.rs

fail::fail_point!("oven::bake", |_| Err(OvenError::Cold)); // compiled out unless the failpoints feature is on

tests/cold_oven.rs

#[tokio::test]
async fn a_cold_oven_still_sells_yesterdays_bread() {
    let scenario = fail::FailScenario::setup();
    fail::cfg("oven::bake", "return").unwrap();
    // assert the counter serves yesterday's loaf
    scenario.teardown();
}

Start in tests, between processes, with Toxiproxy. It runs as a container in front of a real dependency, and adds latency or cuts connections. Drive it from Rust with toxiproxy_rust or its HTTP API. Every timeout and fallback gets a test that triggers it.
Inside the process, use fail points. With the fail crate, fail_point! marks a place that can be made to fail, sleep or panic, and costs nothing unless the failpoints feature is on. TiKV uses it to test its error paths.
Fail points are global to the process. Two tests setting them at once interfere, so run them in their own test binary with --test-threads=1, or under cargo-nextest, which runs each test in its own process.
Then in staging, then game days. Turn on fail points in a staging build, or put Toxiproxy in front of a staging dependency. Game days do the same with people watching and a rollback ready.
Have a hypothesis. “When the oven is cold, the counter sells yesterday’s bread within 2 seconds.” A chaos experiment without one is just an outage.

Security testing: Attack the running system, and fuzz what parses

.github/workflows/nightly.yml

- run: docker run --rm ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://staging.example.com

fuzz/fuzz_targets/parse_order.rs

#![no_main]
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
    let _ = bakery::Order::parse(data); // an Err is fine; a panic or a hang is a finding
});

Run DAST every night against staging. The ZAP baseline scan spiders the site and reports passive findings, such as missing headers, leaky errors and insecure cookies, without attacking.
Fuzz everything that parses untrusted input. Safe Rust rules out memory corruption, not panics, and a panic on a crafted request is a denial of service. cargo-fuzz (cargo +nightly fuzz run parse_order) feeds a parser millions of generated inputs and keeps the ones that crash it. Fuzzing is common practice in Rust for parsers, decoders and anything at a trust boundary.
Keep the crashes as tests. Each input that crashed the parser goes into a regression test, so the bug stays fixed.
Pen testing is people. A scoped human test before a major release and on a fixed schedule finds what scanners can’t: broken authorisation, business-logic abuse, chained flaws.
Agents are an attack surface too. If the service sends user input to a model, test for prompt injection and data exfiltration like any other injection.

The run book

Every alert links to a page saying what to do. An alert nobody knows how to act on is noise.
Say where the panics go. The page says where the panic logs and traces land, and how to start a tokio-console session against a live process.
Practise rollback before you need it. A rollback path that has never been run doesn’t work.

Sources

axum’s graceful shutdown example and tower-http’s CatchPanicLayer.
The Rust Book on panics, including panic = "abort", and std::sync::Mutex poisoning.
OpenTelemetry for Rust, tracing-opentelemetry, and the HTTP metrics semantic conventions, with their recommended boundaries in seconds.
tokio-metrics, tokio-console and OpenFeature’s Rust SDK.
fail-rs, Toxiproxy, the Rust Fuzz Book and the ZAP baseline scan.