Distributed systems and ML infrastructure

InferSched Architecture

A visual walkthrough of how InferSched accepts inference requests, distributes them across Routers and Engines, schedules continuous batches, and safely recovers from failures.

By default InferSched simulates model execution with a deterministic cost model, so most measurements describe scheduler behavior and distributed correctness. Stage 10 adds a llama.cpp backend that runs a real model on the same batches; its rows below are measured on a GPU.

System overview

InferSched end-to-end request and result path

Who owns what?

FastAPI Gateway

Validates HTTP requests, assigns correlation IDs, applies bounded admission, and returns completed results.

Kafka

Retains request events and divides partitions among Routers through a consumer group.

Router Fleet

Owns coordination, durable attempts, load- and prefix-aware dispatch across Engines, retry, cancellation, rebalance, and recovery.

Engine Fleet

Owns continuous batching across RPCs, active sequences, paged KV memory, caches, preemption, and fenced cancellation.

PostgreSQL

Owns authoritative lifecycle state, ownership epochs, fenced finalization, and the transactional outbox.

Redis

Holds disposable prefix-affinity hints that steer requests to an Engine likely to hold their KV prefix. Losing Redis costs cache locality, never correctness.

How one request moves through the system

  1. Accept: FastAPI validates the request, assigns request and trace IDs, and publishes protobuf to Kafka.
  2. Assign: Kafka gives the request partition to exactly one active Router in the consumer group.
  3. Fence: The Router records the request in PostgreSQL and obtains a partition epoch and attempt ID.
  4. Execute: The Router picks the least-loaded Engine, or the one a Redis hint says holds the prompt prefix, and dispatches over gRPC with many attempts in flight. The Engine admits the request into its running batch and manages its KV pages; failed attempts are retried on another Engine.
  5. Finalize: PostgreSQL accepts the completion only when the partition, epoch, Engine, attempt, and state are current.
  6. Publish: The result and outbox row commit together. The relay publishes to Kafka after the database transaction.
  7. Return: The gateway correlates the Kafka result with the waiting HTTP request and returns it to the client.

Guided implementation breakdown

Follow these stages in order when learning, presenting, or extending the project. Each stage depends on the correctness boundary established by the stage before it. Expand a stage to see its objective, the implementation to inspect, and the evidence that proves it works.

0 Foundation and contracts Define the build, request schema, service boundaries, and test configurations. Implemented

Understand

The protobuf contract separates Router coordination from Engine execution. CMake provides default, ASan, and TSan configurations.

Read

proto/infersched.proto, CMakePresets.json, and cpp/CMakeLists.txt.

Evidence

The same request, fence, dispatch, progress, cancellation, registration, and heartbeat messages are used by integration tests and real processes.

1 Deterministic Engine core Build correct lifecycle, batching, cache, KV-allocation, and simulated-time primitives. Implemented

Understand

Requests move through explicit states. The deterministic runner advances a fake clock using a reproducible prefill and decode cost model.

Read

execution_state.cpp, deterministic_engine.cpp, cost_model.cpp, and paged_kv_cache.cpp.

Evidence

Transactional allocation, reference counts, state transitions, cache keys, impossible-request rejection, and zero-leak invariants have focused tests.

2 Continuous batching and KV pressure Admit new work between decode iterations and remain correct when memory fills. Implemented

Understand

The scheduler admits ready requests, reuses prefixes, grows decode pages, evicts cache entries, and preempts lower-value work when necessary.

Read

continuous_engine.cpp, request_queue.cpp, radix_prefix_cache.cpp, and result_cache.cpp.

Evidence

Seeded workloads verify mid-decode admission, true shared prefixes across different prompts, completion despite preemption, and zero KV leakage.

3 Concurrency and scheduler benchmarks Remove scale bottlenecks and measure scheduler overhead without mislabeling it as GPU throughput. Implemented

Understand

A heap-backed request queue and sharded registry replace avoidable global or quadratic paths. Candidate optimizations are kept only when measurements justify them.

Read

request_queue.hpp, sharded_registry.hpp, mpsc_ring.hpp, and scheduler_bench.cpp.

Evidence

The 100,000-request CPU path completes with zero leaks. TSan verifies concurrency, and the slower lock-free queue is documented as a rejected optimization.

4 Real process distribution Run Routers and Engines as separate processes with a stable network protocol. Implemented

Understand

Routers host registration and heartbeat RPCs. Engines host dispatch, cancel, and drain RPCs. Dispatch streams acknowledgement and progress events.

Read

router_service.cpp, engine_service.cpp, router_main.cpp, and engine_main.cpp.

Evidence

Process tests cover concurrent dispatch, stale epoch and attempt rejection, same-Engine overlap rejection, SIGKILL, replacement, and drain.

5 Durable at-least-once pipeline Make PostgreSQL authoritative and publish accepted results without an unsafe dual write. Implemented

Understand

Idempotent ingestion tolerates Kafka redelivery. Conditional finalization accepts only the current fence. A transactional outbox connects database state to Kafka publication.

Read

durable_store.cpp, request_ingestor.cpp, outbox_relay.cpp, offset_tracker.cpp, and durable_router_main.cpp.

Evidence

Live Kafka and PostgreSQL tests prove duplicate handling, poison-to-DLQ publication, one accepted result, restart recovery, broker acknowledgement, and safe contiguous offsets.

6 Multi-Router rebalance and recovery Scale coordination by Kafka partitions while fencing late work after ownership changes. Implemented

Understand

Kafka decides assignment, while PostgreSQL decides authority. New owners increment partition epochs, recover unresolved requests, and create new attempts.

Read

kafka_poller.cpp, durable_router_main.cpp, and docs/p6-multi-router-rebalancing.md.

Evidence

Chaos tests kill a Router and Engine, trigger consumer-group rebalance, flush Redis, and still require all authoritative results to complete exactly once in PostgreSQL.

7 Control plane, observability, and evidence Expose the system through HTTP and measure overload, latency, utilization, caches, and the full distributed path. Implemented

Understand

The gateway bounds pending work, acknowledges only broker-persisted requests, long-polls results, expires lost ones, exports Prometheus metrics, and emits trace spans. Open-loop load arrives independently of completions and is spread across client processes so the client never measures itself.

Read

python/infersched_api/app.py, python/loadgen.py, histogram.cpp, and docs/benchmarks.md.

Evidence

Tests cover HTTP 429 and the live FastAPI path. Reproducible baselines report bucketed latency, utilization, cache hit rates, completion, and zero KV leaks.

8 Engine fleet routing and coordination hints Run many fenced attempts per Router across an Engine fleet, with failover, cancellation, and Redis prefix affinity. Implemented

Understand

Dispatch workers share a small Postgres pool and lease it only for fenced writes. Each attempt goes to the least-loaded Engine unless an incarnation-checked Redis hint points at a warm prefix cache. Failures retry elsewhere; unrecoverable requests become owner-fenced terminal failures.

Read

durable_router_main.cpp, engine_selector.cpp, redis_hints.cpp, engine_service.cpp, and docs/p8-multi-engine-routing.md.

Evidence

A slow Engine is SIGKILLed mid-flight and all 60 requests still complete exactly once; prefix hints are followed and then flushed without losing a request; an in-process gRPC test shows a late short request finishing inside a long running batch.

9 Compute efficiency Show that scheduling decisions spend less modeled accelerator time per delivered token, not just raise throughput. Implemented

Understand

The Engine accounts batch occupancy, useful versus wasted tokens, and prefill computed versus reused. Each experiment changes one decision: admission policy, routing policy, prefix hints, or cancellation.

Read

continuous_engine.cpp, engine_service.cpp, scripts/exp_batching_policy.sh, scripts/exp_fleet_efficiency.py, and docs/p9-compute-efficiency.md.

Evidence

A deterministic CTest freezes the batching result; fleet runs repeat three times with compute deltas stable across host load. Findings that cut the other way (prefill interference, consolidation using less compute than spreading) are reported too.

10 Real-model serving Run a real model on the scheduler's batches and measure serving throughput, latency, and GPU utilization. Implemented

Understand

A StepExecutor seam separates scheduling decisions from execution. The llama.cpp executor turns each iteration into one fused prefill+decode call, backs the paged KV cache with real cells, and shares cached prefixes through sequence tags instead of copies.

Read

step_executor.hpp, llama_executor.cpp, real_serving_bench.cpp, scripts/exp_real_serving.py, and docs/p10-real-model-serving.md.

Evidence

On an H200 with Qwen3-8B, continuous batching delivers 1.35× the tokens per GPU-second of request-level batching and 9.4× that of serial serving, for 30% less energy per token than request-level batching; it matches vLLM's request throughput within 1–4% on identical requests, and prefix caching cuts prompt compute by 85%. Real-model tests check that reused prefix KV gives the same greedy output as recomputing it, and that no KV cell leaks.

Measured results

Experiment Workload Observed result
Scheduler CPU microbenchmark 100,000 requests, simulated compute disabled 732,310 scheduler req/s
Open-loop cost simulation 10,000 requests at 750 offered QPS 750.9 QPS and 78.0% utilization
Open-loop latency Same 750-QPS workload Approximately 65 ms bucketed E2E p99
Large correctness run 100,000 requests at 750 QPS 100% complete and zero KV leaks
Controlled cache comparison 250 offered QPS E2E p99 reduced from 8.2 ms to 2.0 ms
Router coordination path (Linux host) Request burst, zero simulated compute 68 req/s serial Router → ~1,450 req/s concurrent Router
Full HTTP distributed path (Linux host) 6,000 open-loop requests at 600 offered QPS, 2 Engines 558 achieved QPS, 58 ms p50, 98 ms p99, 0 errors
Continuous vs request-level batching (deterministic) Mixed output lengths, memory-bound cost model Sustains 400 vs ~188 QPS; 44% less decode compute per token at 300 QPS
Redis prefix affinity (Linux host) 4 Engines, 64 shared 512-token prefixes Prefill computed −84%, fleet hardware time −29% to −33%
Cancellation of abandoned attempts (Linux host) 10% of requests miss a 60 ms deadline Hardware time −34% to −42% for the same completions
Real model, batching policy (H200, Qwen3-8B FP16) 256-request burst, llama.cpp executing the Engine's batches 1,683 vs 1,251 vs 179 output tok/s (continuous, request-level, serial); 0.27 vs 0.38 vs 3.56 J per token
Real model vs llama-server and vLLM (H200) Identical token-id requests, open loop up to 17.4 QPS 1.96× llama-server's throughput on the same kernels; within 1–4% of vLLM up to 15.1 QPS, at 1.6–3.4× vLLM's time per token
Cost model calibrated to measured steps (H200) 4,298 steps fitted; simulator rerun on the same workload Predicts measured capacity within 3–11%; real batching gain 1.35× vs the calibrated model's 1.43×

In-process percentiles use power-of-two histogram bucket upper bounds; distributed percentiles are exact and client-observed on a shared Linux host. The distributed measurements are single-Router, single-gateway results, not a horizontal scaling curve. See the benchmark report for methodology and claim boundaries.