Client
- Application
- Open-loop load generator
Distributed systems and ML infrastructure
A visual walkthrough of how InferSched accepts inference requests, distributes them across Routers and Engines, schedules continuous batches, and safely recovers from failures.
Partition ownership epochs, request attempts, lifecycle state, conditional finalization, poison messages, and transactional outbox rows.
Publishes results and DLQ records to Kafka, then marks rows published only after broker acknowledgement.
Disposable prefix-affinity hints (Engine incarnation per prompt prefix). Flushing Redis cannot change authoritative request state.
Validates HTTP requests, assigns correlation IDs, applies bounded admission, and returns completed results.
Retains request events and divides partitions among Routers through a consumer group.
Owns coordination, durable attempts, load- and prefix-aware dispatch across Engines, retry, cancellation, rebalance, and recovery.
Owns continuous batching across RPCs, active sequences, paged KV memory, caches, preemption, and fenced cancellation.
Owns authoritative lifecycle state, ownership epochs, fenced finalization, and the transactional outbox.
Holds disposable prefix-affinity hints that steer requests to an Engine likely to hold their KV prefix. Losing Redis costs cache locality, never correctness.
Follow these stages in order when learning, presenting, or extending the project. Each stage depends on the correctness boundary established by the stage before it. Expand a stage to see its objective, the implementation to inspect, and the evidence that proves it works.
The protobuf contract separates Router coordination from Engine execution. CMake provides default, ASan, and TSan configurations.
proto/infersched.proto, CMakePresets.json, and cpp/CMakeLists.txt.
The same request, fence, dispatch, progress, cancellation, registration, and heartbeat messages are used by integration tests and real processes.
Requests move through explicit states. The deterministic runner advances a fake clock using a reproducible prefill and decode cost model.
execution_state.cpp, deterministic_engine.cpp, cost_model.cpp, and paged_kv_cache.cpp.
Transactional allocation, reference counts, state transitions, cache keys, impossible-request rejection, and zero-leak invariants have focused tests.
The scheduler admits ready requests, reuses prefixes, grows decode pages, evicts cache entries, and preempts lower-value work when necessary.
continuous_engine.cpp, request_queue.cpp, radix_prefix_cache.cpp, and result_cache.cpp.
Seeded workloads verify mid-decode admission, true shared prefixes across different prompts, completion despite preemption, and zero KV leakage.
A heap-backed request queue and sharded registry replace avoidable global or quadratic paths. Candidate optimizations are kept only when measurements justify them.
request_queue.hpp, sharded_registry.hpp, mpsc_ring.hpp, and scheduler_bench.cpp.
The 100,000-request CPU path completes with zero leaks. TSan verifies concurrency, and the slower lock-free queue is documented as a rejected optimization.
Routers host registration and heartbeat RPCs. Engines host dispatch, cancel, and drain RPCs. Dispatch streams acknowledgement and progress events.
router_service.cpp, engine_service.cpp, router_main.cpp, and engine_main.cpp.
Process tests cover concurrent dispatch, stale epoch and attempt rejection, same-Engine overlap rejection, SIGKILL, replacement, and drain.
Idempotent ingestion tolerates Kafka redelivery. Conditional finalization accepts only the current fence. A transactional outbox connects database state to Kafka publication.
durable_store.cpp, request_ingestor.cpp, outbox_relay.cpp, offset_tracker.cpp, and durable_router_main.cpp.
Live Kafka and PostgreSQL tests prove duplicate handling, poison-to-DLQ publication, one accepted result, restart recovery, broker acknowledgement, and safe contiguous offsets.
Kafka decides assignment, while PostgreSQL decides authority. New owners increment partition epochs, recover unresolved requests, and create new attempts.
kafka_poller.cpp, durable_router_main.cpp, and docs/p6-multi-router-rebalancing.md.
Chaos tests kill a Router and Engine, trigger consumer-group rebalance, flush Redis, and still require all authoritative results to complete exactly once in PostgreSQL.
The gateway bounds pending work, acknowledges only broker-persisted requests, long-polls results, expires lost ones, exports Prometheus metrics, and emits trace spans. Open-loop load arrives independently of completions and is spread across client processes so the client never measures itself.
python/infersched_api/app.py, python/loadgen.py, histogram.cpp, and docs/benchmarks.md.
Tests cover HTTP 429 and the live FastAPI path. Reproducible baselines report bucketed latency, utilization, cache hit rates, completion, and zero KV leaks.
Dispatch workers share a small Postgres pool and lease it only for fenced writes. Each attempt goes to the least-loaded Engine unless an incarnation-checked Redis hint points at a warm prefix cache. Failures retry elsewhere; unrecoverable requests become owner-fenced terminal failures.
durable_router_main.cpp, engine_selector.cpp, redis_hints.cpp, engine_service.cpp, and docs/p8-multi-engine-routing.md.
A slow Engine is SIGKILLed mid-flight and all 60 requests still complete exactly once; prefix hints are followed and then flushed without losing a request; an in-process gRPC test shows a late short request finishing inside a long running batch.
The Engine accounts batch occupancy, useful versus wasted tokens, and prefill computed versus reused. Each experiment changes one decision: admission policy, routing policy, prefix hints, or cancellation.
continuous_engine.cpp, engine_service.cpp, scripts/exp_batching_policy.sh, scripts/exp_fleet_efficiency.py, and docs/p9-compute-efficiency.md.
A deterministic CTest freezes the batching result; fleet runs repeat three times with compute deltas stable across host load. Findings that cut the other way (prefill interference, consolidation using less compute than spreading) are reported too.
A StepExecutor seam separates scheduling decisions from execution. The llama.cpp executor turns each iteration into one fused prefill+decode call, backs the paged KV cache with real cells, and shares cached prefixes through sequence tags instead of copies.
step_executor.hpp, llama_executor.cpp, real_serving_bench.cpp, scripts/exp_real_serving.py, and docs/p10-real-model-serving.md.
On an H200 with Qwen3-8B, continuous batching delivers 1.35× the tokens per GPU-second of request-level batching and 9.4× that of serial serving, for 30% less energy per token than request-level batching; it matches vLLM's request throughput within 1–4% on identical requests, and prefix caching cuts prompt compute by 85%. Real-model tests check that reused prefix KV gives the same greedy output as recomputing it, and that no KV cell leaks.
| Experiment | Workload | Observed result |
|---|---|---|
| Scheduler CPU microbenchmark | 100,000 requests, simulated compute disabled | 732,310 scheduler req/s |
| Open-loop cost simulation | 10,000 requests at 750 offered QPS | 750.9 QPS and 78.0% utilization |
| Open-loop latency | Same 750-QPS workload | Approximately 65 ms bucketed E2E p99 |
| Large correctness run | 100,000 requests at 750 QPS | 100% complete and zero KV leaks |
| Controlled cache comparison | 250 offered QPS | E2E p99 reduced from 8.2 ms to 2.0 ms |
| Router coordination path (Linux host) | Request burst, zero simulated compute | 68 req/s serial Router → ~1,450 req/s concurrent Router |
| Full HTTP distributed path (Linux host) | 6,000 open-loop requests at 600 offered QPS, 2 Engines | 558 achieved QPS, 58 ms p50, 98 ms p99, 0 errors |
| Continuous vs request-level batching (deterministic) | Mixed output lengths, memory-bound cost model | Sustains 400 vs ~188 QPS; 44% less decode compute per token at 300 QPS |
| Redis prefix affinity (Linux host) | 4 Engines, 64 shared 512-token prefixes | Prefill computed −84%, fleet hardware time −29% to −33% |
| Cancellation of abandoned attempts (Linux host) | 10% of requests miss a 60 ms deadline | Hardware time −34% to −42% for the same completions |
| Real model, batching policy (H200, Qwen3-8B FP16) | 256-request burst, llama.cpp executing the Engine's batches | 1,683 vs 1,251 vs 179 output tok/s (continuous, request-level, serial); 0.27 vs 0.38 vs 3.56 J per token |
| Real model vs llama-server and vLLM (H200) | Identical token-id requests, open loop up to 17.4 QPS | 1.96× llama-server's throughput on the same kernels; within 1–4% of vLLM up to 15.1 QPS, at 1.6–3.4× vLLM's time per token |
| Cost model calibrated to measured steps (H200) | 4,298 steps fitted; simulator rerun on the same workload | Predicts measured capacity within 3–11%; real batching gain 1.35× vs the calibrated model's 1.43× |
In-process percentiles use power-of-two histogram bucket upper bounds; distributed percentiles are exact and client-observed on a shared Linux host. The distributed measurements are single-Router, single-gateway results, not a horizontal scaling curve. See the benchmark report for methodology and claim boundaries.