NarsilRust-native inference serving engine

One Rust serving shell — Burn-native kernels and native torch (libtorch + FBGEMM) underneath, many transports on top.

Backend-agnostic core

A small InferenceBackend trait sits behind tonic gRPC, a worker pool, and a continuous-batching collector. Backends plug in without touching the serving shell.

Native torch inference (route A)

The torch-dlrm backend loads a TorchRec TorchScript artifact through tch-rs and runs the real FBGEMM table-batched embedding + interaction kernels — matching the C++ baseline's latency.

Burn-native path

A pure-Rust Burn CUDA DLRM backend for the no-libtorch story, plus an opt-in cuTile fused-interaction kernel for hardware that supports it.

Continuous batching

The batch collector coalesces concurrent requests into one fused forward — on the DLRM workload this beats the per-request C++ baseline on throughput.

What is Narsil?

Narsil is a Rust inference serving engine. Its thesis is to be the best-in-class serving shell — owning the transport, scheduling, and batching in safe Rust — while staying pluggable about the compute underneath:

Internally: native torch inference (tch-rs → libtorch + FBGEMM) and custom Rust/cuTile kernels (Burn).
Externally: many transports — gRPC today, with HTTP, QUIC, NCCL, and UCCL on the roadmap.

The current reference workload is DLRM (Deep Learning Recommendation Model), benchmarked against the official TorchRec C++ gRPC server.

The headline result

Serving the same INT8 DLRM artifact on an NVIDIA L4:

Path	conc 1 — p50	conc 16 — samples/s
Narsil route A (libtorch/FBGEMM)	1.02 ms	~438,000
TorchRec C++ gRPC (documented baseline)	1.033 ms	~215,000
Narsil Burn CUDA DLRM (FP32)	13.04 ms	~7,900

The conc-1 p50 is measured in worker mode; the conc-16 throughput in batch mode (see Benchmarks for the per-mode tables). Route A matches the C++ baseline's single-request latency and exceeds its throughput via continuous batching. See the full tables and caveats there, and the Decision Log for how we got here.