Rust Course Design Proposal

The Rust course builds vector indexes in a standalone crate, then connects them to SQL through DataFusion in the final required chapter. The collection and search interfaces remain independently testable without Arrow or a query engine. At the boundary, a small adapter makes the relationship between SQL semantics and index execution visible.

SQL Integration Boundary

DataFusion provides SQL parsing, planning, Arrow execution, and the extension points used by the course. It supports array_distance, cosine_distance, and inner-product functions. A custom TableProvider exposes the collection. The design pins DataFusion 54.1.0 and uses ExecutionPlan::try_pushdown_sort and with_fetch: the physical optimizer offers the scan a requested ordering, removes the generic sort after the scan accepts it, and passes the literal limit to the index.

All DataFusion-specific code lives in the adapter crate. Students use the pinned public APIs and do not edit DataFusion itself.

The course begins and ends with the same query:

SELECT id, payload
FROM points
ORDER BY cosine_distance(embedding, [0.1, 0.2, 0.3])
LIMIT 10;

In the opening chapter, DataFusion evaluates the distance for every row and performs a generic top-k sort. In the final chapter, the adapter recognizes the compatible metric, constant query vector, ordering direction, and literal limit, then produces the course-defined VectorIndexScanExec. EXPLAIN makes the change visible.

Queries outside the supported pattern retain the exact plan. In particular, the course does not claim that arbitrary WHERE predicates can be applied after an ANN top-k without changing the result. Refusing an unsafe rewrite is part of the SQL contract.

For example, suppose the nearest point belongs to tenant B and the second-nearest point belongs to tenant A. For WHERE tenant = 'A' ORDER BY distance LIMIT 1, taking one ANN result and then applying the filter returns no rows. Applying the filter first returns tenant A’s point. The adapter must keep the exact plan for this query.

Architecture

The dependency direction is:

DataFusion SQL adapter --> collection API --> exact / IVF / graph index
                               ^
                               |
                    tests, datasets, benchmark

The adapter owns Arrow conversion, SQL-pattern recognition, plan properties, and result batches. The collection owns IDs, dimensions, metrics, and index selection. The indexes know nothing about SQL, Arrow, or asynchronous execution.

This separation keeps every algorithm testable with ordinary Rust values. It also makes the final integration diff small enough for students to explain line by line.

System Contracts

The course establishes these contracts before students implement an ANN index.

1. A collection has one fixed dimension and one distance metric. A query or bulk-loaded point with another dimension is rejected.
2. Stored vectors use f32, while distance accumulation uses f64.
3. Exact search defines the ground truth. ANN benchmarks always report recall together with latency.
4. Results use deterministic tie-breaking so tests do not depend on heap or hash-map iteration order.
5. The required lifecycle is bulk load, build, freeze, and query. Persistence and online mutation after a build are out of scope.
6. SQL uses an ANN scan only when the adapter can prove that the query matches the index contract. All other queries use DataFusion’s exact plan.

These rules are visible in public types, tests, and EXPLAIN output rather than scattered across chapter prose.

Course Progression

The required path is sized for roughly one focused week, but it is not divided into artificial days. Learners may spread it across more sessions. The chapters follow conceptual density: IVFFlat and the graph indexes are longer than the baseline, evaluation, and adapter chapters.

The ordering is intentional. Exact search becomes the first working implementation and the oracle for every approximate index. The benchmark comes before ANN so probes, beam width, and ef_search are evaluated rather than guessed. IVFFlat introduces the recall/latency tradeoff with a simple candidate-generation model. HNSW follows NSW so hierarchy is the only new graph idea in that chapter. SQL integration comes last so it wraps an already tested search contract.

SQL appears in the opening and final chapters rather than consuming the middle of the course. Students know the target plan from the beginning, but the ANN algorithms remain ordinary library code. The final chapter demonstrates how little adapter code is needed and how much semantic care that small amount of code still requires.

Before finishing the course, students should be able to explain:

• why deterministic top-k ordering matters;
• why every ANN performance number needs a recall number from the same query set;
• how probes and ef_search change the candidate budget in different index families;
• which SQL expression shapes are safe to lower to an ANN scan; and
• why a filtered or differently ordered query must fall back to exact execution.

What We Intentionally Leave Out

The required implementation does not include:

• online upserts and deletes after an index is built;
• persistent point or index formats, write-ahead logging, crash recovery, background rebuilds, or compaction;
• concurrent readers and writers or distributed execution;
• filtered ANN search, hybrid lexical search, or joins pushed into the vector index;
• quantization, GPU execution, or memory-mapped index layouts; or
• an HTTP or production-compatible database protocol.

These are follow-up projects, not hidden requirements. The final collection is useful for bulk-load-and-query workloads and for demonstrating SQL integration, but it is not a production database.

Your feedback is greatly appreciated. Join our Discord community.
Found an issue? Open an issue or pull request at github.com/skyzh/write-you-a-vector-db.
write-you-a-vector-db © 2024-2026 by Alex Chi Z. All Rights Reserved.