In this post, we'll explore the overflow mechanism in SQLite and implement it in our own toy database, allowing us to read large texts and blobs.

As usual, the source code for this post is available on Github.

Overview

Overflow pages structure
------------------------ 

Page 3                                Page 4
┌─────────────────────────────────┐   ┌─────────────────────────────────┐
│ Next: Page 4 │   Data...        │┌─>│ Next: NULL   │   Data...        │
└─────────────────────────────────┘│  └─────────────────────────────────┘
       │                           │
       └───────────────────────────┘

When a field's length exceeds the maximum usable page size, our database has two questions to answer:

1. where and in what format to store data that doesn't fit in a page?

2. what to write in the B-tree cell?

The answer to the first question is quite elegant. The data that doesn't fit is split into multiple overflow pages, using the following structure: the first four bytes of every overflow page contain the page number of the next overflow page (or 0 if there are no more overflow pages), and the rest of the page contains the overflow data. Therefore, overflow pages form a linked list.

As for the second question, the B-tree cell will contain the firstN bytes of the complete payload (where N is calculated using a formula we'll explore in the following sections), followed by the page number of the first overflow page.

With this quick overview, let's dive in the implementation!

Rust edition

Some code in this post uses the new Let chains feature from Rust 1.88.0. If you want to follow along, make sure to update your Cargo.toml to use the 2024 edition:

# Cargo.toml
[package]
# [...]
edition = "2024"

Erratum

If you're one of the early readers of the first post, and you coded along, a small mistake might have slipped into your code: the read_varint_at function - which decodes a variable-length integer - treats varints as little-endian, while SQLite uses big-endian encoding. Here is the corrected version:

pub fn read_varint_at(buffer: &[u8], mut offset: usize) -> (u8, i64) {
    let mut size = 0;
    let mut result = 0;

    while size < 9 {
        let current_byte = buffer[offset] as i64;
        if size == 8 {
            result = (result << 8) | current_byte;
        } else {
            result = (result << 7) | (current_byte & 0b0111_1111);
        }

        offset += 1;
        size += 1;

        if current_byte & 0b1000_0000 == 0 {
            break;
        }
    }

    (size, result)
}

You'll notice that we now shift the result accumulator at each iteration instead of shifting the current_byte. The ninth byte special case is also handled differently, as it is no longer the most significant byte.

Building the test database

To test our implementation, we'll need a test database that contains rows with large enough fields to trigger the overflow mechanism. To create such a database, you can use the following commands:

sqlite3 test.db
sqlite> create table t1(id integer, value string);
sqlite> insert into t1(id, value) values (42, printf('%.*c', 10000, 'a'));

printf('%.*c', 10000, 'a') returns a string composed of 10000 as.

Computing the local payload size

As we mentioned, when reading a record that's too big to fit in a single page, our first task is to compute the local payload size. This is the number of bytes that will be stored directly in the B-tree leaf cell. To compute this size, we first need to define a few variables:

• P, the payload size

• U, the usable page size (which is the page size minus the number of reserved bytes)

• X = U - 35, which is the overflow threshold: if the payload size is less than or equal to X it will be stored entirely in a B-tree leaf cell, without overflow

• M = ((U-12)*32/255)-23, the minimum local payload size

• K = M+((P-M)%(U-4)), the maximum local payload size

The local payload size is computed according to the following rules:

1. If P <= X, the local payload size is P and there is no overflow.

2. Otherwise, there is an overflow, and:

   • If P <= K, the local payload size is K.

   • Otherwise, the local payload size is M.

We'll start by implementing the computation of U, the usable page size. It differs from the page size since SQLite reserves a few bytes at the end of each page for use by extensions. The exact number of reserved bytes is defined by a two-byte integer in the database header, at offset 20. First, we'll extend our DbHeader struct to include the number of reserved bytes:

// src/page.rs

#[derive(Debug, Copy, Clone)]
pub struct DbHeader {
    pub page_size: u32,
+   pub page_reserved_size: u8,
}

+impl DbHeader {
+    pub fn usable_page_size(&self) -> usize {
+        self.page_size as usize - (self.page_reserved_size as usize)
+    }
+}

We also added a utility method that returns the value of U, computed from the page size and reserved bytes count.

Our db header parsing function must also be updated to populate the new field:

// src/pager.rs
+const HEADER_PAGE_RESERVED_SIZE_OFFSET: usize = 20;

// [...]

pub fn parse_header(buffer: &[u8]) -> anyhow::Result<page::DbHeader> {
    if !buffer.starts_with(HEADER_PREFIX) {
        let prefix = String::from_utf8_lossy(&buffer[..HEADER_PREFIX.len()]);
        anyhow::bail!("invalid header prefix: {prefix}");
    }

    let page_size_raw = read_be_word_at(buffer, HEADER_PAGE_SIZE_OFFSET);
    let page_size = match page_size_raw {
        1 => PAGE_MAX_SIZE,
        n if n.is_power_of_two() => n as u32,
        _ => anyhow::bail!("page size is not a power of 2: {}", page_size_raw),
    };

+   let page_reserved_size = buffer[HEADER_PAGE_RESERVED_SIZE_OFFSET];

-   Ok(page::DbHeader { page_size })
+   Ok(page::DbHeader {
+       page_size,
+       page_reserved_size,
+   })
}

With this in place, we can implement the local_payload_size method according to the formulas defined above:

// src/page.rs

// [...]

impl PageHeader {
    fn local_payload_size(
        &self,
        db_header: &DbHeader,
        payload_size: usize,
    ) -> anyhow::Result<usize> {
        match self.page_type {
            PageType::TableInterior => bail!("no payload size for interior pages"),
            PageType::TableLeaf => {
                let usable = db_header.usable_page_size();
                let max_size = usable - 35;
                if payload_size <= max_size {
                    return Ok(payload_size);
                }
                let min_size = ((usable - 12) * 32 / 255) - 23;
                let k = min_size + ((payload_size - min_size) % (usable - 4));
                let size = if k <= max_size { k } else { min_size };
                Ok(size)
            }
        }
    }

    pub fn local_and_overflow_size(
        &self,
        db_header: &DbHeader,
        payload_size: usize,
    ) -> anyhow::Result<(usize, Option<usize>)> {
        let local = self.local_payload_size(db_header, payload_size)?;
        if local == payload_size {
            Ok((local, None))
        } else {
            Ok((local, Some(payload_size.saturating_sub(local))))
        }
    }
}

For table-leaf pages, the implementation of local_payload_size is a straightforward translation of the formulas. For interior pages, we simply return an error, as they do not contain an actual payload, so the concept of a local payload size does not apply. In future posts, we'll discover index pages, which will require us to implement a slightly altered version of the formulas.

We also defined a convenient method that computes the local payload size and the overflow size, if any.

How are we going to use this information? When we parse a B-tree leaf cell, we'll compute the local and overflow sizes. If the overflow size is not None, we know that a pointer to the first overflow page is stored right after the local payload, so we'll read this pointer and record it in the resulting Cell struct. We don't want to read the content of the overflow pages just yet, as we have no way to know if the query will actually need it.

Let's modify our Cell struct and the corresponding parsing function:

// src/page.rs

// [...]

#[derive(Debug, Clone)]
pub struct TableLeafCell {
pub size: i64,
    pub size: i64,
    pub row_id: i64,
    pub payload: Vec<u8>,
+   pub first_overflow: Option<usize>,
}

// src/pager.rs

// [...] 

-fn parse_table_leaf_cell(mut buffer: &[u8]) -> anyhow::Result<page::Cell> {
+fn parse_table_leaf_cell(
+   db_header: &DbHeader,
+   header: &PageHeader,
+   mut buffer: &[u8],
+) -> anyhow::Result<page::Cell> {
    let (n, size) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

    let (n, row_id) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

+   let (local_size, overflow_size) = header.local_and_overflow_size(db_header, size as usize)?;
+   let first_overflow = overflow_size.map(|_| read_be_double_at(buffer, local_size) as usize);

-   let payload = buffer[..size as usize].to_vec();
+   let payload = buffer[..local_size].to_vec();

    Ok(page::TableLeafCell {
        size,
        row_id,
        payload,
        first_overflow,
    }
    .into())
}

The modifications to parse_table_leaf_cell are straightforward: we leverage our utility method to compute the local and overflow sizes, as well as the first overflow page pointer, if any. Note that we modified the function's signature to accept a reference to the database header, so you'll need to propagate this change to the caller, and adapt the signature of parse_table_interior_cell as well.

We're not far from having a complete implementation of the overflow mechanism. What's left is to implement a way to read and parse the overflow pages, and exercise it when reading fields from a cursor.

Reading overflow pages

Before we implement the parsing, we'll create a type to represent an overflow page:

// src/page.rs

// [...]

#[derive(Debug, Clone)]
pub struct OverflowPage {
    pub next: Option<usize>,
    pub payload: Vec<u8>,
}

Parsing an overflow page is quite simple: the first four bytes contain the next overflow page pointer (or 0 if there are no more overflow pages), and the rest of the page contains the overflow payload,

// src/pager.rs

// [...]

fn parse_overflow_page(buffer: &[u8]) -> page::OverflowPage {
    let next = read_be_double_at(buffer, 0);
    page::OverflowPage {
        payload: buffer[4..].to_vec(),
        next: if next != 0 { Some(next as usize) } else { None },
    }
}

Since our Pager's cache expects to only store Page structs, we need to adapt it so it can read and cache either a Page or an OverflowPage. First we'll define a new enum to represent either a Page or an OverflowPage:

// src/pager.rs

// [...]

#[derive(Debug, Clone)]
enum CachedPage {
    Page(Arc<page::Page>),
    Overflow(Arc<page::OverflowPage>),
}

impl From<Arc<page::Page>> for CachedPage {
    fn from(value: Arc<page::Page>) -> Self {
        CachedPage::Page(value)
    }
}

impl TryFrom<CachedPage> for Arc<page::Page> {
    type Error = anyhow::Error;

    fn try_from(value: CachedPage) -> Result<Self, Self::Error> {
        if let CachedPage::Page(p) = value {
            Ok(p.clone())
        } else {
            bail!("expected a regular page")
        }
    }
}

impl From<Arc<page::OverflowPage>> for CachedPage {
    fn from(value: Arc<page::OverflowPage>) -> Self {
        CachedPage::Overflow(value)
    }
}

impl TryFrom<CachedPage> for Arc<page::OverflowPage> {
    type Error = anyhow::Error;

    fn try_from(value: CachedPage) -> Result<Self, Self::Error> {
        if let CachedPage::Overflow(o) = value {
            Ok(o.clone())
        } else {
            bail!("expected an overflow page")
        }
    }
}

It's a simple enum with two variants and a few conversion traits to allow our Pager to handle both types seamlessly.

Then, we need to adapt our Pager to support both types. We'll extract the bulk of the logic into a new genericload method that takes a page number and a parsing function and parses uncached pages with the provided function. Here is the updated Pager implementation:

// src/pager.rs

// [...]

#[derive(Debug)]
pub struct Pager<I: Read + Seek = std::fs::File> {
    input: Arc<Mutex<I>>,
    pages: Arc<RwLock<HashMap<usize, CachedPage>>>,
    header: DbHeader,
}

impl<I: Read + Seek> Pager<I> {
    pub fn new(header: DbHeader, input: I) -> Self {
        Self {
            input: Arc::new(Mutex::new(input)),
            pages: Arc::default(),
            header,
        }
    }

    pub fn read_overflow(&self, n: usize) -> anyhow::Result<Arc<page::OverflowPage>> {
        self.load(n, |buffer| Ok(parse_overflow_page(buffer)))
    }

    pub fn read_page(&self, n: usize) -> anyhow::Result<Arc<page::Page>> {
        self.load(n, |buffer| parse_page(&self.header, buffer, n))
    }

    fn load<T>(&self, n: usize, f: impl Fn(&[u8]) -> anyhow::Result<T>) -> anyhow::Result<Arc<T>>
    where
        Arc<T>: Into<CachedPage>,
        CachedPage: TryInto<Arc<T>, Error=anyhow::Error>,
    {
        {
            let read_pages = self
                .pages
                .read()
                .map_err(|_| anyhow!("poisoned page cache lock"))?;

            if let Some(page) = read_pages.get(&n).cloned() {
                return page.try_into();
            }
        }

        let mut write_pages = self
            .pages
            .write()
            .map_err(|_| anyhow!("failed to acquire pager write lock"))?;

        if let Some(page) = write_pages.get(&n).cloned() {
            return page.try_into();
        }

        let buffer = self.load_raw(n)?;
        let parsed = f(&buffer[0..self.header.usable_page_size()])?;
        let ptr = Arc::new(parsed);

        write_pages.insert(n, ptr.clone().into());

        Ok(ptr)
    }

    fn load_raw(&self, n: usize) -> anyhow::Result<Vec<u8>> {
        let offset = n.saturating_sub(1) * self.header.page_size as usize;

        let mut input_guard = self
            .input
            .lock()
            .map_err(|_| anyhow!("poisoned pager mutex"))?;

        input_guard
            .seek(SeekFrom::Start(offset as u64))
            .context("seek to page start")?;

        let mut buffer = vec![0; self.header.page_size as usize];
        input_guard.read_exact(&mut buffer).context("read page")?;

        Ok(buffer)
    }
}

Putting it all together

The main building blocks of our implementation are in place: we now how to detect when a row is too large to fit in a single page, and we have a way to load overflow pages through our Pager. The last step is to lazily read the overflow data when accessing a field that requires it.

To do this, we'll implement an OverflowScanner with a read method that takes as input the index of the first overflow page and the minimum amount of overflow data to read. The scanner will follow the linked list until the required amount of data is read of there are no more overflow pages.

// src/cursor.rs

// [...]

#[derive(Debug)]
struct OverflowScanner {
    pager: Pager,
}

impl OverflowScanner {
    pub fn new(pager: Pager) -> Self {
        Self { pager }
    }

    pub fn read(&self, first_page: usize, size: usize) -> anyhow::Result<(Option<usize>, Vec<u8>)> {
        let mut next_page = Some(first_page);
        let mut buffer = Vec::with_capacity(size);

        while buffer.len() < size
            && let Some(next) = next_page
        {
            let overflow = self.pager.read_overflow(next)?;
            next_page = overflow.next;
            buffer.extend_from_slice(&overflow.payload);
        }

        Ok((next_page, buffer))
    }
}

Our Cursor will use this new scanner in the following way: When reading a field, we'll compute the end offset of the field (based on the field's offset and size). If that end offset exceeds the size of the currently loaded payload, we'll read (end_offset - payload.len()) bytes through the overflow scanner, and append the result to the payload. If we wanted to read fields that are so large that they can't fit in RAM, we should implement a more sophisticated streaming mechanism, but for our purposes, reading the overflow data into memory is enough.

We'll start by implementing a utility method to compute the end offset of a field:

// src/cursor.rs

// [...]

impl RecordField {
    pub fn end_offset(&self) -> usize {
        let size = match self.field_type {
            RecordFieldType::Null => 0,
            RecordFieldType::I8 => 1,
            RecordFieldType::I16 => 2,
            RecordFieldType::I24 => 3,
            RecordFieldType::I32 => 4,
            RecordFieldType::I48 => 5,
            RecordFieldType::I64 => 8,
            RecordFieldType::Float => 8,
            RecordFieldType::Zero => 0,
            RecordFieldType::One => 0,
            RecordFieldType::String(size) | RecordFieldType::Blob(size) => size,
        };

        self.offset + size
    }
}

Next, we'll implement the overflow reading logic in the Cursor:

// src/cursor.rs

// [...]

#[derive(Debug)]
pub struct Cursor {
    header: RecordHeader,
    payload: Vec<u8>,
+   pager: Pager,
+   next_overflow_page: Option<usize>,
}

impl Cursor {
-   pub fn owned_field(&self, n: usize) -> Option<OwnedValue> {
-       self.field(n).map(Into::into)
-   }
+   pub fn owned_field(&mut self, n: usize) -> anyhow::Result<Option<OwnedValue>> {
+       Ok(self.field(n)?.map(Into::into))
+   }

-   pub fn field(&self, n: usize) -> Option<Value> {
+   pub fn field(&mut self, n: usize) -> anyhow::Result<Option<Value>> {
-       let record_field = self.header.fields.get(n)?;
+       let Some(record_field) = self.header.fields.get(n) else {
+           return Ok(None);
+       };

+       let end_offset = record_field.end_offset();

+       if end_offset > (self.payload.len() - 1)
+           && let Some(overflow_page) = self.next_overflow_page
+       {
+           let overflow_size = end_offset.saturating_sub(self.payload.len());
+           let (next_overflow, overflow_data) = OverflowScanner::new(self.pager.clone())
+               .read(overflow_page, overflow_size)
+               .context("read overflow page")?;
+           self.next_overflow_page = next_overflow;
+           self.payload.extend_from_slice(&overflow_data);
+       }

-       match record_field.field_type {
+       let value = match record_field.field_type {
            RecordFieldType::Null => Some(Value::Null),
            RecordFieldType::I8 => Some(Value::Int(read_i8_at(&self.payload, record_field.offset))),
            RecordFieldType::I16 => {
                Some(Value::Int(read_i16_at(&self.payload, record_field.offset)))
            }
            RecordFieldType::I24 => {
                Some(Value::Int(read_i24_at(&self.payload, record_field.offset)))
            }
            RecordFieldType::I32 => {
                Some(Value::Int(read_i32_at(&self.payload, record_field.offset)))
            }
            RecordFieldType::I48 => {
                Some(Value::Int(read_i48_at(&self.payload, record_field.offset)))
            }
            RecordFieldType::I64 => {
                Some(Value::Int(read_i64_at(&self.payload, record_field.offset)))
            }
            RecordFieldType::Float => Some(Value::Float(read_f64_at(
                &self.payload,
                record_field.offset,
            ))),
            RecordFieldType::String(length) => {
                let value = std::str::from_utf8(
                    &self.payload[record_field.offset..record_field.offset + length],
                )
                .expect("invalid utf8");
                Some(Value::String(Cow::Borrowed(value)))
            }
            RecordFieldType::Blob(length) => {
                let value = &self.payload[record_field.offset..record_field.offset + length];
                Some(Value::Blob(Cow::Borrowed(value)))
            }
            RecordFieldType::One => Some(Value::Int(1)),
            RecordFieldType::Zero => Some(Value::Int(0)),
-       }
_       };

+       Ok(value)
    }
}

Note that we added fields to the Cursor and slightly modified its methods signature. These changes will need to be propagated to the consumers of the Cursor struct.

Conclusion

This concludes our implementation of the overflow mechanism. Our little database can now read large TEXT and BLOB fields that are split across multiple pages. In the next post, we'll get back to the query engine and implement simple WHERE clauses, allowing us to filter rows based on their content.

• the structure of the AST is dictated by the syntax of the source language, and does not lend itself especially well to the tasks we need to perform before generating the assembly code, such as control-flow analysis, optimization, and generally breaking-up high-level operations into sequences of assembly instructions
• if we want to extend our compiler to support more CPU architectures, we would need to duplicate the entire code generation logic for each architecture, thus hurting the maintainability of the codebase

For these reasons, we'll split the process of generating assembly code into multiple steps. First, we'll translate the AST into an intermediate representation (IR) that is more suitable for optimization and code generation. Then, we'll perform optimization passes on the IR, and finally we'll generate assembly code from the optimized IR. The process of translating the AST into the IR is often referred to as "lowering" the AST into the IR.

What will our IR look like? On one end of the spectrum, we could design a very high-level IR that closely resembles the source language, and on the other we could design a low-level linear IR that mimics the assembly code. We'll go for a hybrid graph-based approach, where branchless sections of the code are represented as linear sequences of low-level instructions -which we'll call "basic blocks"- and conditional jumps are represented as edges between the basic blocks. This structure is called a control-flow graph (CFG).

As an example, let's take the following Pylite program, which computes nth value of the Fibonacci sequence:

def fib(n: int) -> int:
    a = 1
    b = 1

    while n > 0:
        c = b
        b = a + b
        a = c
        n = n - 1

    return a

The corresponding CFG is shown below:

          ┌───────┐
          │ start │
          └───┬───┘
              │
              ▼
          ┌───────┐
          │ a = 1 │
          │ b = 1 │
          └───┬───┘
              │
              ▼
          ┌───────┐  False  ┌──────────┐
  ┌──────▶│ n > 0 │────────▶│ return a │
  |       └───┬───┘         └──────────┘
  |           │ True                  
  |           ▼                     
  |    ┌───────────────┐           
  |    │ c = b         │          
  |    │ b = a + b     │         
  |    │ a = c         │        
  |    │ n = n - 1     │       
  |    └────┬──────────┘      
  |         │             
  └─────────┘

You'll notice that control-flow constructs -such as the while in our function definition- are absent from the basic blocks. This is because they are represented as edges between the basic blocks.

For now our CFGs will be much simpler, as our function bodies will only contain a single return statement.

Let's start by defining the IR nodes:

//! compiler/src/ir/nodes.rs

#[derive(Debug, Clone)]
pub enum Decl {
    Function(Function),
}

#[derive(Debug, Clone)]
pub struct Function {
    pub name: String,
    pub cfg: Cfg,
}

#[derive(Debug, Clone)]
pub struct Cfg {
    pub blocks: Vec<BasicBlock>,
}

#[derive(Debug, Clone)]
pub struct BasicBlock {
    pub instructions: Vec<Instruction>,
}

#[derive(Debug, Clone)]
pub enum Instruction {
    Return(Operand),
}

#[derive(Debug, Clone)]
pub enum Operand {
    Immediate(i64),
}

Here is the IR representation of our simple Pylite main function:

Function(
    Function {
        name: "main",
        cfg: Cfg {
            blocks: [
                BasicBlock {
                    instructions: [
                        Return(
                            Immediate(
                                1,
                            ),
                        ),
                    ],
                },
            ],
        },
    },
),

At this stage, there is a direct corespondance between the AST nodes and their IR counterparts. Let's build a Generator struct to perform the mapping between the two representations. It's only public method generate will translate a single AST Decl into the matching IR construct.

use crate::ast;

use super::nodes::{BasicBlock, Cfg, Decl, Function, Instruction, Operand};

pub struct Generator {
    instructions: Vec<Instruction>,
}

impl Generator {
    pub fn new() -> Self {
        Self {
            instructions: Vec::new(),
        }
    }

    pub fn generate(mut self, decl: &ast::Decl) -> Decl {
        match &decl.kind {
            ast::DeclKind::Function(f) => {
                self.generate_function(f);
                Decl::Function(Function {
                    name: f.name.name.clone(),
                    cfg: Cfg {
                        blocks: vec![BasicBlock {
                            instructions: self.instructions,
                        }],
                    },
                })
            }
        }
    }

    fn generate_function(&mut self, function: &ast::FunDecl) {
        for stmt in &function.body {
            self.generate_stmt(stmt);
        }
    }

    fn generate_stmt(&mut self, stmt: &ast::BlockStatement) {
        match &stmt.kind {
            ast::BlockStatementKind::Return { value } => {
                let operand = self.generate_expr(value);
                self.instructions.push(Instruction::Return(operand))
            }
        }
    }

    fn generate_expr(&mut self, stmt: &ast::Expr) -> Operand {
        match &stmt.kind {
            ast::ExprKind::IntLit { value } => Operand::Immediate(*value),
        }
    }
}

Assembly language 101

At the end of the compilation pipeline, our compiler will output assembly code. Assembly code is a textual representation of the machine code that's executed by the CPU. Therefore, assembly code is specific to a given CPU architexture, and the assembly code generated by our compiler will only run natively on x86 CPUs (which is a prevalent architecture in both personal computers and servers).

By the end of this section, you'll know enough about assembly programming to understand each line of the assembly equivalent of our Pylite main function.

Your first assembly program

.globl main     ; on macOs replace with .globl _main
main:           ; on macOs replace with _main:
    mov rax, 1
    ret

This is the assembly equivalent of our Pylite main function. In assembly, each line represents a single instruction or assembler directive, and instructions are executed sequentially, from top to bottom. Let's break this down line by line:

    .globl main

This line is an assembler directive that declares the main label as a global symbol. By default, symbols are local to the source file they are defined in, so referencing main from another source file would result in an error.

In this case, there is only one source file, so why do we need to declare main as a global symbol? Well,main is the actual entry point of our program: the entry point is the _start symbol, defined in an object file named crt0.o (C runtime zero). crt0 is part of the C standard library, and it is responsible for setting up the execution environment of our program before invoking our main function.

main:

This is the definition of the main label. Unlike mainstream programming languages, assembly does not have functions or procedures, but instead uses labels to mark locations in the code that can be used as targets for jumps and calls. We'll make extensive use of labels when implementing control flow structures and function calls in our compiler.

On macOS, the label must be prefixed with an underscore, so it would be _main: instead of main:.

    mov rax, 1

This instruction moves the immediate value 1 into the rax register. Registers are small, fast and fixed-size storage locations within the CPU that are used to hold values that are being processed. There is a fixed set of registers, some of which are "general-purpose" registers, meaning that they can be used for almost any operation, while others are specialized and have very specific behaviors.

rax is one of the general-purpose registers, and like most registers in a 64-bit CPU architecture, it can hold 64 bits of data. There are 15 other general-purpose registers, named rbx, rcx, rdx, rsi, rdi, rsp, rbp, and r8 to r15. Don't feel compelled to memorize the full list right now, as we'll discuss a lot more about registers in the next chapters.

    ret

Through a mechanism that we'll explore in greater depth in the next chapters, ret orders the CPU to resume execution from the point where we jumped to the main label. In this case, it will return to the code in crt0, which calls the exit system call with the value in rax as the exit code of the program.

Building the executable

We'll use gcc to assemble and link our assembly code. The following sections will show you how to set up your environment to assemble and run the assembly code generated by our compiler.

macOS setup

On macOS, we'll use the homebrew package manager to install gcc. To install, homebrew, follow the instructions on the official website. You can check your installation by running the following command in your terminal:

$ brew --version

Once homebrew is installed, you can install gcc by running:

$ brew install gcc

If your computer is running an x86 CPU, you're all set! But if you have an Apple Silicon CPU, there is one last thing to be aware of: while it is perfectly fine to run the compiler on an Apple Silicon CPU, the assembly code generated by our compiler will only run on x86 CPUs. To run the generated assembly code, you'll need to use Rosetta 2, which is a translation layer that allows x86 code to run on Apple Silicon CPUs. You can install Rosetta by running the following command in your terminal:

$ softwareupdate --install-rosetta

Once Rosetta 2 is installed, you can run gcc as an x86 binary by first opening an x86 shell and running gcc from there:

$ arch -x86_64 zsh
$ <your_gcc_invocation>

Linux setup

On Linux, you can install gcc using your distribution's package manager. For example, on Ubuntu, you can run the following command:

$ sudo apt install gcc

You can check your installation by running the following command:

$ gcc --version

Windows setup

Our compiler will not run natively on Windows, but you can use the Windows Subsystem for Linux (WSL) to run it. WSL allows you to run a Linux distribution on Windows, so you can use the same instructions as for Linux to install gcc and run the compiler. To install WSL, follow the instructions on the official website.

Assembling and linking the assembly code

For this section, we'll assume that our assembly code is saved in a file named return_code.s. We can assemble and link the assembly code with a single gcc command:

$ gcc -masm=intel return_code.s -o return_code

It instructs gcc to assemble and link the code in return_code.s, and produces an executable named return_code. The -masm=intel flag tells gcc to use the Intel syntax for the assembly code, which is the syntax used in our compiler. By default, gcc uses the AT&T syntax, which is slightly different.

We finally have a working executable! You can run it by executing the following command:

$ ./return_code

And predictably... nothing happens. Our program runs and exits, but is does not print anything to the console. To verify that it worked properly, we can inspect the exit code of the previous command by running:

$ echo $?
1

It prints 1, which is the value we returned from the main function. You can try to substitute 1 with any other small integer in the assembly code, rerun gcc, and run the executable again to see the exit code change accordingly.

Code generation

We're ready to build the final piece of our compiler: the code generator. It will take the IR representation of our program as input, and generate the corresponding x86 assembly code. This will be a two-step process: first, we'll translate the IR data structure into another data structure representing the linear sequence of assembly instructions, then we'll emit the actual textual assembly code.

As we'll discover in the next chapters, splitting this process into two steps will allow us to perform some transformations at the assembly level before emitting the final assembly code.

The final assembly code for our Pylite main function will look like this:

.globl main
main:
    mov rax, 1
    ret

We'll define a few rust types to represent assembly instructions and operands.

//! compiler/src/codegen/asm.rs

#[derive(Debug, Clone)]
pub struct Function {
    pub label: String,
    pub instructions: Vec<Instruction>,
}

#[derive(Debug, Clone)]
pub enum Instruction {
    Mov { src: Operand, dst: Operand },
    Ret,
}

#[derive(Debug, Clone)]
pub enum Operand {
    Register(Register),
    Immediate(i64),
}

#[derive(Debug, Copy, Clone)]
pub enum Register {
    Rax,
}

impl std::fmt::Display for Register {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            Register::Rax => write!(f, "rax"),
        }
    }
}

To translate the IR into assembly instructions, we need to iterate over each basic block of a function's CFG and generate the corresponding assembly Instruction. To that end, we'll define a FnGenerator struct that wraps a function's IR representation and a mutable assembly instructions buffer and iterates over the basic block's instructions, pushing the corresponding assembly instructions into the buffer. We'll also create a utility function that instantiates a FnGenerator for every function in the IR program and returns a Vec of assemblyFunction structs, each containing the function's label and the corresponding assembly instructions.

use crate::ir;

use super::asm::{Function, Instruction, Operand, Register};


struct FnGenerator<'f> {
    function: &'f ir::Function,
    instructions: Vec<Instruction>,
}

impl<'f> FnGenerator<'f> {
    fn new(function: &'f ir::Function) -> Self {
        Self {
            function,
            instructions: Vec::new(),
        }
    }

    fn generate(mut self) -> Vec<Instruction> {
        for block in &self.function.cfg.blocks {
            self.generate_block(block);
        }
        self.instructions
    }

    fn generate_block(&mut self, block: &ir::BasicBlock) {
        for instruction in &block.instructions {
            match &instruction {
                ir::Instruction::Return(op) => {
                    let operand = self.generate_operand(op);
                    self.push(Instruction::Mov {
                        src: operand,
                        dst: Operand::Register(Register::Rax),
                    });
                    self.push(Instruction::Ret)
                }
            }
        }
    }

    fn generate_operand(&mut self, operand: &ir::Operand) -> Operand {
        match operand {
            ir::Operand::Immediate(value) => Operand::Immediate(*value),
        }
    }

    fn push(&mut self, instruction: Instruction) {
        self.instructions.push(instruction);
    }
}

pub fn generate(program: &[ir::Decl]) -> Vec<Function> {
    program
        .iter()
        .map(|decl| {
            let ir::Decl::Function(f) = decl;
            Function {
                label: f.name.clone(),
                instructions: FnGenerator::new(f).generate(),
            }
        })
        .collect()
}

The final step before we can assemble and run our program is to render the assembly code into its textual representation.

One thing to note is that the format of labels is inconsistent across platforms: on macOS function labels must start with an underscore, which is not the case on Linux. To make our code portable, we'll use conditional compilation, and define two different implementations of the function that renders function labels: one that prepends an underscore to the label on macOS, and an other that returns the label as is on Linux.

//! compiler/src/codegen/render.rs

#[cfg(target_os = "macos")]
fn format_fun_label(label: &str) -> String {
    format!("_{}", label)
}

#[cfg(target_os = "linux")]
fn format_fun_label(label: &str) -> String {
    label.to_string()
}

With this in place, we can implement our render_program function, which takes a slice of assembly Function structs and returns the textual representation of the assembly code.

//! compiler/src/codegen/render.rs

use super::asm::{Function, Instruction, Operand};

pub fn render_program(program: &[Function]) -> String {
    let mut code = String::new();
    for function in program {
        code.push_str(&render_function(function));
    }
    code
}

fn render_function(function: &Function) -> String {
    let label = format_fun_label(&function.label);
    let mut code = format!("\t.globl {label}\n{label}:\n");

    for instruction in &function.instructions {
        let rendered = render_instruction(instruction);
        code.push_str(&format!("\t{rendered}\n"));
    }

    code
}

fn render_instruction(instruction: &Instruction) -> String {
    match instruction {
        Instruction::Mov { src, dst } => {
            format!("mov {}, {}", render_operand(dst), render_operand(src))
        }
        Instruction::Ret => "ret".to_string(),
    }
}

fn render_operand(operand: &Operand) -> String {
    match operand {
        Operand::Register(register) => register.to_string(),
        Operand::Immediate(i) => i.to_string(),
    }
}

// [...]

Putting it all together

At this stage, we have all the components of a complete compiler. Let's put it to the test by writing a straightforward main function that binds all of these components to turn our Pylite code into an assembly program:

//! compiler/src/main.rs

mod ast;
mod codegen;
mod ctx;
mod db;
mod error;
mod id;
mod ir;
mod parse;

use id::UniqueIdGenerator;

fn main() {
    // Read the input source code
    let input_file = std::env::args().nth(1).expect("missing input file");
    let source_code = std::fs::read_to_string(&input_file).expect("failed to read input file");

    let id_gen = UniqueIdGenerator::default();

    // Parse the code into an Abstract Syntax Tree
    let ast_statements = parse::parser::Parser::new(id_gen.clone(), &source_code)
        .parse_module()
        .expect("failed to parse module");

    // Lower the AST to the Intermediate Representation
    let ir_statements = ast_statements
        .iter()
        .map(|stmt| {
            let ast::StatementKind::Decl(decl) = &stmt.kind;
            ir::gen::Generator::new().generate(decl)
        })
        .collect::<Vec<_>>();

    // Generate and print the assembly code
    let asm = codegen::gen::generate(&ir_statements);
    println!("{}", codegen::render::render_program(&asm));
}

With our input program in res/samples/return_const/main.py, we can run our compiler and inspect it's output using the following commands:

$ cargo run -- res/samples/return_const/main.py > out.s
$ cat out.s
        .globl _main
_main:
        mov rax, 1
        ret

It seems that the compilation was successful! We can execute the program and observe its output code by running the following commands:

$ gcc -masm=intel out.s -o main
$ ./main
$ echo $?
1

If you are using a mac with an Apple Silicon processor, remember to run arch -x86_64 zsh before typing these commands.

We just ran our first Pylite program! In the next section, we'll start laying the foundations for a more robust architecture.

So we'll start with something simpler. How much simpler? Well, here is the first Pylite program that our compiler will translate to machine code:

def main():
  return 1

It just sets a return code and exits. This is the simplest program that we can write while having an easily observable effect. It will be our "Hello World" program for now.

Throughout this chapter, we will build the basic building blocks of our compiler, including the parser, and translation passes to the intermediate representation and machine code. Each of these blocks will only support the minimum features needed to compile our "Hello World" program, and will be expanded in later chapters to support more complex constructs.

Setting up the project

We will start by creating a new cargo workspace to host our compiler.

$ mkdir pylite
$ cd pylite

For now, this workspace will contain a single compiler crate:

# Cargo.toml
[workspace]
resolver = "2"
members = ["compiler"]

We can now create the compiler crate:

$ cargo new compiler --bin
$ cargo run 2>/dev/null
Hello, world!

Our top-level pylite folder should have the following structure:

.
├── Cargo.toml
└── compiler
    ├── Cargo.toml
    └── src
        └── main.rs

Parsing Pylite

The textual representation of a program does not lend itself well to the kind of analysis and transformation that we need to perform. In order to resolve types, associate identifiers with their definitions and generate our low-level intermediate representation, we need to convert the unstructured sequence of characters that make up the source code into a tree-like structure that represents the program's syntax. This process if called parsing, and the data structure it produces is called an Abstract Syntax Tree (AST). The parsing process is typically split into two parts: lexical analysis (or tokenization) and syntactic analysis (or parsing).

Source code
===========

def main():
    return 1


Token sequence
==============

┌───┐ ┌────┐ ┌─┐ ┌─┐ ┌─┐ ┌──────┐ ┌──────┐ ┌─┐ ┌──────┐
│def│ │main│ │(│ │)│ │:│ │INDENT│ │return│ │1│ │DEDENT│
└───┘ └────┘ └─┘ └─┘ └─┘ └──────┘ └──────┘ └─┘ └──────┘


Abstract Syntax Tree
====================

┌─────────────┐
│ FunctionDef │
│   (main)    │
└─────────────┘
        │
        └── ┌─────────────┐
            │ Return Stmt │
            └─────────────┘
                    │
                    └── ┌─────────┐
                        │ Literal │
                        │    1    │
                        └─────────┘

During lexical analysis, we group individual characters into tokens, which are the smallest meaningful units of the language. For example, the characters d, e and f will be grouped into a single token representing the def keyword. Irrelevant characters, such as non-significant whitespace, are discarded during this process. Since Pylite - like Python - is indentation-sensitive, we need to keep track of the current indentation level. This will be done by inserting INDENT and DEDENT tokens into the token sequence whenever the indentation level changes.

The syntax analysis phase will then take this sequence of tokens and match it against our language's syntax rules (or grammar) to build a tree-like structure called an Abstract Syntax Tree (AST).

Writing the lexer

The first step in writing our lexer is to choose a representation for our tokens. Rust's enums are a great fit for this purpose. We'll also pair each token with a Span struct that will hold the token's start and end byte position in the source code. Keeping track of the token's position will prove useful to display meaningful error messages to the user.

//! compiler/src/ast.rs

#[derive(Debug, Copy, Clone, Eq, PartialEq, Hash)]
pub struct Span {
    pub start: u32,
    pub end: u32,
}

impl Span {
    // Create a new span with the given start and end position.
    pub fn new(start: u32, end: u32) -> Self {
        Self { start, end }
    }

    // Create a new span that covers both the current span and the given span.
    pub fn merge(self, other: Span) -> Self {
        Self {
            start: self.start.min(other.start),
            end: self.end.max(other.end),
        }
    }
}

//! compiler/src/parse/token.rs

#[derive(Debug, Clone, Eq, PartialEq, Hash)]
pub enum Token {
    Def,
    Return,
    LPar,
    RPar,
    Colon,
    Identifier(String),
    Int(i64),
    Unknown(char),
}

The Unknown variant will be used to represent any unexpected character that we encounter during the tokenization process.

With these definitions in place, we can create a scaffolding for our lexer. We'll model it as an iterator yielding tuples of (Token, Span).

//! compiler/src/parse/lexer.rs

use std::{iter::Peekable, str::Chars};

use crate::ast::Span;

use super::token::Token;

pub struct Lexer<'c> {
    // iterator over the source code's characters
    input: Peekable<Chars<'c>>,
    // current byte position in the source code
    position: u32,
}

impl<'c> Lexer<'c> {
    pub fn new(input: &'c str) -> Self {
        Self {
            input: input.chars().peekable(),
            position: 0,
        }
    }

    fn next_token(&mut self) -> Option<(Token, Span)> {
        unimplemented!() 
    }
}

impl Iterator for Lexer<'_> {
    type Item = (Token, Span);

    fn next(&mut self) -> Option<Self::Item> {
        self.next_token()
    }
}

We'll also add the following helper methods to simplify the implementation of next_token:

• emit_token takes a start position and a Token and returns a tuple of the form (Token, Span). where the Span starts at start and ends at the current position.
• next_char consumes the next character in the input (if any), and updates the current position.
• next_char_if conditionally consumes the next character and updates the current position.

Since we track the byte position of the tokens and utf-8 characters can span multiple bytes, we can't simply increment the position for each character. Instead, we'll use the char::len_utf8 method to get the byte size of the current character and update the position accordingly.

//! compiler/src/parse/lexer.rs

impl<'c> Lexer<'c> {
    // [...] 

    fn emit_token(&self, start: u32, token: Token) -> Option<(Token, Span)> {
        Some((
            token,
            Span {
                start,
                end: self.position,
            },
        ))
    }

    fn next_char(&mut self) -> Option<char> {
        self.next_char_if(|_| true)
    }

    fn next_char_if(&mut self, f: impl FnOnce(char) -> bool) -> Option<char> {
        self.input.next_if(|&c| f(c)).inspect(|c| {
            self.position += c.len_utf8() as u32;
        })
    }  
}

Some tokens contain a single character, such as ( or :. We can handle these tokens by matching comparing the current character to the expected one and returning the corresponding token if they match.

//! compiler/src/parse/lexer.rs

impl<'c> Lexer<'c> {
    fn next_token(&mut self) -> Option<(Token, Span)> {
        let start_pos = self.position;

        match self.next_char()? {
            '(' => self.emit_token(start_pos, Token::LPar),
            ')' => self.emit_token(start_pos, Token::RPar),
            ':' => self.emit_token(start_pos, Token::Colon),
            c => unimplemented!(),
        }
    }
}

The remaining cases are more involved, but follow a similar pattern:

• if the current character is a digit, we consume the following characters until we reach a non-digit character. We then parse the resulting string as an integer and create an integer token.
• if the current character is a letter, we consume the following characters until we reach a non-alphanumeric character different from _. We then check if the resulting string matches a keyword and create a keyword token if it does, or an identifier token otherwise.
• if the current character is a whitespace, we skip every following whitespace and return the following token.

Finally, if the current character does not match any of the above cases, we create an Unknown token.

//! compiler/src/parse/lexer.rs

impl<'c> Lexer<'c> {
    // [...]
    fn next_token(&mut self) -> Option<(Token, Span)> {
        // [...]
        match self.next_char()? {
            // [...]
            c if c.is_numeric() => {
                let mut int_repr = String::from(c);
                while let Some(c) = self.next_char_if(|c| c.is_numeric()) {
                    int_repr.push(c);
                }
                let value = int_repr.parse::<i64>().expect("int token");
                self.emit_token(start_pos, Token::Int(value))
            }
            c if c.is_alphabetic() => {
                let mut identifier = String::from(c);
                while let Some(c) = self.next_char_if(|c| c.is_alphanumeric() || c == '_') {
                    identifier.push(c);
                }
                match identifier.as_str() {
                    "def" => self.emit_token(start_pos, Token::Def),
                    "return" => self.emit_token(start_pos, Token::Return),
                    _ => self.emit_token(start_pos, Token::Identifier(identifier)),
                }
            }
            c if c.is_whitespace() => {
                while self.next_char_if(|c| c.is_whitespace()).is_some() {}
                self.next_token()
            }
            c => self.emit_token(start_pos, Token::Unknown(c)),
        }
    }
}

With our lexer almost complete, it's time to write our first test! As building and verifying the token spans by hand would be tedious, we'll write a helper test_lexer function that takes as input the source code and a vec of (String, Token) tuples representing the expected tokens, with the String being the token's textual representation. If tokenizing the input and rendering the Spans does not yield the same vec, the test will fail.

//! compiler/src/parse/lexer.rs

#[cfg(test)]
mod test {
    use super::*;

    #[test]
    fn tokenize_return_const() {
        let input = r##"
def main():
    return 34
"##;

        let expected = vec![
            ("def", Token::Def),
            ("main", Token::Identifier("main".to_string())),
            ("(", Token::LPar),
            (")", Token::RPar),
            (":", Token::Colon),
            ("return", Token::Return),
            ("34", Token::Int(34)),
        ];

        test_lexer(input, expected);
    }

    fn test_lexer(input: &str, expected: Vec<(&str, Token)>) {
        let rendered: Vec<(&str, Token)> = Lexer::new(input)
            .map(|(token, span)| {
                let rendered = &input[span.start as usize..span.end as usize];
                (rendered, token)
            })
            .collect();

        assert_eq!(expected, rendered);
    }
}

Let's make sure that our test passes before moving on to the next section:

$ cargo test
running 1 test
test parse::lexer::tests::tokenize_return_const ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Handling indentation

One defining aspect of Pylite's syntax is that it is indentation-sensitive. This means that blocks are delimited by their indentation level, rather than by explicit braces or keywords. To simplify the parsing process, we'll insert Indent and Dedent tokens into the token stream whenever we detect a change in the current indentation level.

How to detect to such changes? First, we need to define which character - or sequence of characters - represents a single level of indentation. In Pylite a single tab, or four consecutive spaces, will represent a single level of indentation. Every time we reach a new line, we'll count the number of tabs and spaces at the beginning of the line and compare the new indentation level to the previous one. If the indentation levels match, there is no extra token to insert. If they don't match we'll compute delta = abs(new_indentation - previous_indentation) and insert delta Indent or Dedent tokens accordingly. In case of inconsistent indentation, for example, if a new line starts with three spaces, we'll insert an InconsistentIndentation token.

Let's start by adding the new variants to our Token enum:

//! compiler/src/parse/token.rs

#[derive(Debug, Clone, Eq, PartialEq, Hash)]
pub enum Token {
    Def,
    Return,
    LPar,
    RPar,
    Colon,
    Identifier(String),
    Int(i64),
    Unknown(char),
+   Indent,
+   Dedent,
+   InconsistentIndentation,
}

// [...]

In some cases, the difference in indentation level will be higher than one. For example, we'll have to insert two Dedent tokens at the end of the following function definition. One to close the if block, and the other to close the function definition.

def main():
    if True:
        return 1

This doesn't fit well with the current design of our lexer, as we return tokens eagerly as soon as they are recognized. We'll need to refactor our lexer to support emitting multiple tokens at once. For this reason, we'll split the work done by our iter method into two separate steps: first push the recognized tokens into a queue, and then return the first token in the queue, if any.

//! compiler/src/parse/lexer.rs

-use std::{iter::Peekable, str::Chars};
+use std::{collections::VecDeque, iter::Peekable, str::Chars};

pub struct Lexer<'c> {
    input: Peekable<Chars<'c>>,
    // queue of tokens to emit, we use a VecDeque to efficiently pop tokens from the front
    // without having to shift the remaining elements
+   token_queue: VecDeque<(Token, Span)>,
    position: u32,
+   indentation: u32, 
}

impl<'c> Lexer<'c> {
    pub fn new(input: &'c str) -> Self {
        Self {
            input: input.chars().peekable(),
+           token_queue: VecDeque::default(),
            position: 0,
+           indentation: 0, 
        }
    }

-   fn next_token(&mut self) -> Option<(Token, Span)> {
+   fn emit_next_tokens(&mut self) {
        // handle indents at the beginning of a file
+       if self.position == 0 {
+           self.emit_indentation_tokens();
+       }
+
        let start_pos = self.position;

+       let Some(first_char) = self.next_char() else {
+           return;
+       };

-       match self.next_char()? {
+       match first_char {
            // [...]
+           '\n' => {
+               self.emit_indentation_tokens(); 
+               self.emit_next_tokens();
            }
            // [...]
        }
    }

+   fn emit_indentation_tokens(&mut self) {
+       let mut space_count = 0;
+       let mut indentation = 0;

+       while let Some(c) = self.next_char_if(char::is_whitespace) {
+           match c {
                // a tab increases the indentation level by 1
+               '\t' => {
+                   if space_count % 4 != 0 {
+                       space_count = 4;
+                       self.emit_token(self.position, Token::InconsistentIndentation);
+                   }
+                   indentation += 1;
+               }
                // indentation is resetted on every new line
+               '\n' => {
+                   space_count = 0;
+                   indentation = 0;
+               }
+               _ => {
+                   space_count += 1;
                    // four spaces increase the indentation level
+                   if space_count % 4 == 0 {
+                       indentation += 1;
+                   }
+               }
+           }
+       }

+       if space_count % 4 != 0 {
+           self.emit_token(self.position, Token::InconsistentIndentation)
+       }

+       if indentation == self.indentation {
+           return;
+       }

        // emit indent/dedent tokens if the new indentation level is
        // different from the previous one
+       for _ in 0..self.indentation.abs_diff(indentation) {
+           let token = if indentation > self.indentation {
+               Token::Indent
+           } else {
+               Token::Dedent
+           };
+           self.emit_token(self.position, token);
+       }

+       self.indentation = indentation;
+   }

    fn emit_token(&mut self, start: u32, token: Token) {
        self.token_queue.push_back((
            token,
            Span {
                start,
                end: self.position,
            },
        ));
    }

    // [...]
}

impl Iterator for Lexer<'_> {
    type Item = (Token, Span);

    fn next(&mut self) -> Option<Self::Item> {
-       self.next_token();
+       if self.token_queue.is_empty() {
+           self.emit_next_tokens();
+       }
+       self.token_queue.pop_front()
    }
}

Now, our lexer will emit Indent and Dedent tokens whenever the indentation level changes. The only thing that is left to do is to update our test:

//! compiler/src/parse/lexer.rs

[...]

    #[test]
    fn tokenize_return_const() {
        let input = r##"
def main():
    return 34
"##;

        let expected = vec![
            ("def", Token::Def),
            ("main", Token::Identifier("main".to_string())),
            ("(", Token::LPar),
            (")", Token::RPar),
            (":", Token::Colon),
+           ("", Token::Indent),
            ("return", Token::Return),
            ("34", Token::Int(34)),
+           ("", Token::Dedent),
        ];

        test_lexer(input, expected);
    }

    [...]

Building the parser

Introduction to formal grammars

Throughout this series, we'll try to limit the amount of purely theoretical content. But it's challenging to study compilers without at least a cursory understanding of formal grammars. We'll actually make use of the concepts introduced in this section when writing our parser, as the shape of our parsing functions will closely mimic the structure of our grammar rules.

The kind of grammar that we'll be using is called a context-free grammar (CFG). Context-free grammars are a subset of formal grammars that are widely used in computer science, particularly in the field of programming languages. To learn more about formal grammars, you can check out the Wikipedia entry on the Chomsky hierarchy, which classified grammars into multiple nested categories.

But first, what is a language's grammar? In the context of programming languages, a grammar is a set of rules that define the syntax of a language. It specifies how tokens can be combined to form valid statements and expressions. In a way, grammars solve the same problem as regular expressions: given a piece of text, determine whether it is "valid," according to a set of rules.

Why not use regular expressions to define the syntax of our language, then? The answer is simple: regular expressions are not powerful enough to express the syntax of most programming languages. Regular expressions can handle simple patterns like matching keywords or numeric literals, but can't describe nested structures and recursive patterns that are fundamental to programming languages. As a motivating example, consider the challenge of matching balanced parentheses—a structure that appears everywhere in programming languages, from function calls to mathematical expressions. You can try to write a regular expression that matches balanced parentheses: inputs like (), (()) and (()()) should all be matched, while inputs like ((), ()) should not.

This is a classic example of something that cannot be expressed with regular expressions, but can be expressed with a context-free grammar. Without further ado, let's see how to write a grammar for our balanced parentheses example.

BalancedParenthesis = { Nested }-;
Nested = "(", { Nested }, ")";

Rules are defined using the = symbol, and are terminated by a semicolon. Within a rule , denotes a sequence of elements. Rule names can be surrounded to express repetition:

• [] means "0 or 1"
• {} means "0 or more"
• {}- means "1 or more"

Translated into english, this grammar states that a BalancedParenthesis is a sequence of one or more Nested, where a Nested is defined a a (, followed by zero or more Nested and then a ).

How to verify that our grammar is correct? We can use rule substitution to replace the rule names with their definition, until we reach the expected string. For example, let's try to verify that the string (()()) is indeed a valid BalancedParenthesis:

(()()) = BalancedParenthesis
       = { Nested }- // 1 
       = "(", { Nested }, ")" // 2
       = "(", "(", { Nested }, ")", "(", { Nested }, ")", ")"  // 3
       = "(", "(", ")", "(", ")", ")" // 4

We start by replacing BalancedParenthesis with its definition (step 1), then we substitute Nested with its definition (step 2), and replace the inner { Nested } with two successive Nested (step 3). Finally, we replace the innermost Nested (step 4) and reach the expected string.

Reading and writing grammars can prove to be challenging at first, but with a bit of practice, you'll find that they are a powerful tool to explore and describe the syntax of programming languages. Luckily for us, at this point, the grammar for our language is quite simple, as it does not include repetitions or recursion like the grammar for balanced parentheses.

Program = FunDecl;

FunDecl = "def" Identifier "(" ")" ":" Block;

Block = Indent ReturnStatement Dedent;

BlockStatement = ReturnStatement;

ReturnStatement = "return" Expression;

Expression = Int;

You'll notice that some rules don't have a matching definition, such as Identifier, Int, Indent and Dedent. This is because they are terminal rules, which means that they directly represent tokens in the token stream produced by the lexer.

As an exercise, you can try to use successive substitutions starting from the Program rule to verify that the following Pylite program is syntactically valid:

def main():
    return 42

Generating unique identifiers

The data structures built by a compiler naturally tend to contain cycles. For example, a recursive function definition includes the AST nodes representing the function's body, which in turn contain a reference to the function itself (after name-resolution is complete). This is a problem for our compiler, as Rust's borrow checker makes it difficult to create cyclic data structures. To solve this problem, we'll add a level of indirection: each node in the AST - and in general, each entity within the compiler - will be identified by a unique identifier. In the previous example, during name resolution, we'll simply register the association between the function's unique identifier and the unique identifier of the recursive call within the function's body, without introducing any cycle.

Our unique identifiers will be backed by monotonically increasing unsigned integers. To keep track of the last assigned identifier and generate new ones, we'll create a UniqueIdGenerator struct. It will have a pointer to an AtomicU32 representing the next identifier to be assigned. Generating a new identifier is as simple as incrementing the atomic counter and returning its previous value.

//! compiler/src/id.rs

use std::{rc::Rc, sync::atomic::AtomicU32};

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct UniqueId(u32);

impl From<u32> for UniqueId {
    fn from(value: u32) -> Self {
        UniqueId(value)
    }
}

#[derive(Debug, Clone, Default)]
pub struct UniqueIdGenerator {
    next_id: Rc<AtomicU32>,
}

impl UniqueIdGenerator {
    pub fn generate(&self) -> UniqueId {
        UniqueId(
            self.next_id
                .fetch_add(1, std::sync::atomic::Ordering::Relaxed),
        )
    }
}

Parsing the token stream

The next step is to transform the token stream into an Abstract Syntax Tree (AST). We'll start by defining a set of struct and enums to represent the different nodes in our AST.

For some additional type safety, we leverage the nonempty crate. It provides a NonEmpty type that mimics the behavior of Vec, but guarantees that the vector always contains at least one element. We'll use this type to represent function bodies, which always contain at least one statement. Let's install it by running cargo add nonempty in the compiler folder: ```rust // compiler/ast.rs

use std::{path::PathBuf, rc::Rc};

use nonempty::NonEmpty;

use crate::id::UniqueId;

// [...]

// Represents a Pylite module, which is an entire source file.

#[derive(Debug, Clone)] pub struct Module { pub path: Rc, }

impl Module { pub fn new(path: PathBuf) -> Self { Self { path: Rc::new(path), } } }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct Statement { pub id: UniqueId, pub span: Span, pub kind: StatementKind, }

#[derive(Debug, Clone, Eq, PartialEq)] pub enum StatementKind { Decl(Decl), }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct Decl { pub id: UniqueId, pub span: Span, pub kind: DeclKind, }

#[derive(Debug, Clone, Eq, PartialEq)] pub enum DeclKind { Function(FunDecl), }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct FunDecl { pub name: Identifier, pub body: NonEmpty, }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct Identifier { pub id: UniqueId, pub span: Span, pub name: String, }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct BlockStatement { pub id: UniqueId, pub span: Span, pub kind: BlockStatementKind, }

#[derive(Debug, Clone, Eq, PartialEq)] pub enum BlockStatementKind { Return { value: Expr }, }

#[derive(Debug, Clone, Eq, PartialEq)] pub struct Expr { pub span: Span, pub id: UniqueId, pub kind: ExprKind, }

#[derive(Debug, Clone, Eq, PartialEq)] pub enum ExprKind { IntLit { value: i64 }, }

Enum nodes are split into two parts: the `Kind` enum, which contains the different
variants of the node, and a wrapper struct that contains fields that are common to 
all variants. This is a common pattern used in [rustc](https://doc.rust-lang.org/beta/nightly-rustc/rustc_ast/ast/struct.Expr.html),
among others.

With our AST nodes defined, we can start writing the parser.
We have many parsing algorithms to choose from to transform the token stream into an AST,
and the one we'll use is called recursive descent parsing. It has the dual advantage
of being relatively simple to implement and easy to extend to support extra features, such as error
recovery and reporting.

In recursive descent parsing, we define a function for each non-terminal rule in the grammar.
Every time a rule references a terminal we consume the corresponding token from the
token stream, and every time a rule references a non-terminal we call the corresponding
function. This process continues until we reach the end of the input or encounter an error.

Before we start implementing the parser, let's define an error type that will 
represent all the error scenarios that can occur during parsing. For now,
we'll handle three types of errors:
- `UnexpectedToken` is used when we encounter a token that does not match
  what was expected
- `UnexpectedEof` is used when we reach the end of the input while still
  expecting more tokens
- `Io` is used when we encounter an I/O error while reading the source code

To create error types more easily, we'll use the [thiserror](https://docs.rs/thiserror/latest/thiserror/) crate.
It provides a convenient macro to derive the `std::error::Error` trait for our error types.
It can be installed by running `cargo add thiserror` in the `compiler` folder.

```rust
//! compiler/src/parse/error.rs

use crate::ast::Span;

use super::token::Token;

#[derive(Debug, thiserror::Error)]
pub enum Error {
    #[error("Unexpected token {0:?}")]
    UnexpectedToken(Token, Span),
    #[error("io error: {0}")]
    Io(#[from] std::io::Error),
    #[error("Unexpected end of file")]
    UnexpectedEof,
}

We can now define our Parser struct, with a few utility methods to help us implement our parsing rules.

//! compiler/src/parse/parser.rs

use std::iter::Peekable;

use nonempty::NonEmpty;

use crate::{ast::*, id::UniqueIdGenerator};

use super::{error::Error, lexer::Lexer, token::Token};

#[derive(Clone)]
pub struct Parser<'c> {
    lexer: Peekable<Lexer<'c>>,
    id_gen: UniqueIdGenerator,
}

impl<'c> Parser<'c> {
    pub fn new(id_gen: UniqueIdGenerator, code: &'c str) -> Self {
        Self {
            id_gen,
            lexer: Lexer::new(code).peekable(),
        }
    }

    fn expect_eq(&mut self, expected: Token) -> Result<Span, Error> {
        let (token, span) = self.next_token()?;
        if token == expected {
            Ok(span)
        } else {
            Err(Error::UnexpectedToken(token, span))
        }
    }

    fn next_token(&mut self) -> Result<(Token, Span), Error> {
        self.lexer.next().ok_or(Error::UnexpectedEof)
    }
}

• expect_eq compares the next token to the expected one and returns its span if they match, or an UnexpectedToken error if they don't
• next_token returns the next token and its span, or an UnexpectedEof error if there are no more tokens

Expressions and identifiers

Remember that our current grammar rule for expressions is:

Expression = Int;

The corresponding parsing method is a direct translation of the BNF rule: we consume the next token and check if it is an integer literal. If it is, we create an Expr node where the kind field represents the integer literal. Otherwise, we return an UnexpectedToken error.

//! compiler/src/parse/parser.rs

// [...]
impl <'c> Parser<'c> {
    // [...]

    fn parse_expr(&mut self) -> Result<Expr, Error> {
        match self.next_token()? {
            (Token::Int(n), span) => Ok(Expr {
                span,
                id: self.id_gen.generate(),
                kind: ExprKind::IntLit { value: n },
            }),
            (token, span) => Err(Error::UnexpectedToken(token, span)),
        }
    }
}

Identifiers follow the same pattern, as Identifier AST nodes wrap a single Indentifier token:

//! compiler/src/parse/parser.rs

impl <'c> Parser<'c> {
    // [...]

    fn parse_identifier(&mut self) -> Result<Identifier, Error> {
        match self.next_token()? {
            (Token::Identifier(name), span) => Ok(Identifier {
                id: self.id_gen.generate(),
                span,
                name,
            }),
            (token, span) => Err(Error::UnexpectedToken(token, span)),
        }
    }
}

Blocks and block statements

Block statements are statements that appear within an indented block. For now the only block statement we support is the return statement. The grammar rule for blocks and block statements is as follows:

Block = Indent ReturnStatement Dedent;
BlockStatement = ReturnStatement;
ReturnStatement = "return" Expression;

Translating these rules into parsing methods is quite straightforward. Terminals are matched with the expect_eq utility, and non-terminals are parsed by delegating to the corresponding parsing method.

//! compiler/src/parse/parser.rs

impl <'c> Parser<'c> {
    // [...]
    fn parse_block(&mut self) -> Result<NonEmpty<BlockStatement>, Error> {
        self.expect_eq(Token::Indent)?;
        let stmts = NonEmpty::new(self.parse_block_stmt()?);
        self.expect_eq(Token::Dedent)?;
        Ok(stmts)
    }

    fn parse_block_stmt(&mut self) -> Result<BlockStatement, Error> {
        self.parse_return()
    }

    fn parse_return(&mut self) -> Result<BlockStatement, Error> {
        let ret_span = self.expect_eq(Token::Return)?;
        let expr = self.parse_expr()?;
        Ok(BlockStatement {
            id: self.id_gen.generate(),
            // merging the return token span with the expression span
            // produces a span that covers the entire return statement
            span: ret_span.merge(expr.span),
            kind: BlockStatementKind::Return { value: expr },
        })
    }
}

Function declarations

Function declarations are a bit more complex than the rules we've seen so far, but the translation from grammar to code follows the exact same pattern. Here is the grammar rule for function declarations:

FunDecl = "def" Identifier "(" ")" ":" Block;

And the corresponding parsing method:

//! compiler/src/parse/parser.rs

impl <'c> Parser<'c> {
    // [...]

    fn parse_fun_decl(&mut self) -> Result<Decl, Error> {
        let span_start = self.expect_eq(Token::Def)?;
        let name = self.parse_identifier()?;
        self.expect_eq(Token::LPar)?;
        self.expect_eq(Token::RPar)?;
        self.expect_eq(Token::Colon)?;
        let body = self.parse_block()?;
        Ok(Decl {
            id: self.id_gen.generate(),
            span: span_start.merge(body.last().span),
            kind: DeclKind::Function(FunDecl { name, body }),
        })
    }
}

With this addition, our parser is nearly complete. We'll define two last parsing methods:

• parse_decl to parse a Decl node
• parse_module to parse the top level declarations in a module

Both methods are straightforward, as they only need to delegate to parse_fun_decl.

//! compiler/src/parse/parser.rs

impl <'c> Parser<'c> {
    // [...]

    pub fn parse_module(&mut self) -> Result<Vec<Statement>, Error> {
        let decl = self.parse_decl()?;
        Ok(vec![Statement {
            id: self.id_gen.generate(),
            span: decl.span,
            kind: StatementKind::Decl(decl),
        }])
    }

    fn parse_decl(&mut self) -> Result<Decl, Error> {
        self.parse_fun_decl()
    }
}

Let's edit our main function to test our parser:

//! compiler/src/main.rs

mod ast;
mod ctx;
mod db;
mod error;
mod id;
mod parse;

fn main() {
    let input_file = std::env::args().nth(1).expect("missing input file");
    let input_code = std::fs::read_to_string(input_file).expect("failed to read source code");
    let id_gen = id::UniqueIdGenerator::default();
    let mut parser = parse::parser::Parser::new(id_gen, &input_code);
    let stmts = parser.parse_module().expect("failed to parse module");
    println!("{stmts:#?}");
}

Assuming we have saved our small Pylite program in the res/samples/return_const/main.py file, we can test our parser by running the following command:

$ cargo run -- res/samples/return_const/main.py

It should print the following output, confirming that our parser works as expected:

[
    Statement {
        id: UniqueId(
            4,
        ),
        span: Span {
            start: 0,
            end: 24,
        },
        kind: Decl(
            Decl {
                id: UniqueId(
                    3,
                ),
                span: Span {
                    start: 0,
                    end: 24,
                },
                kind: Function(
                    FunDecl {
                        name: Identifier {
                            id: UniqueId(
                                0,
                            ),
                            span: Span {
                                start: 4,
                                end: 8,
                            },
                            name: "main",
                        },
                        body: NonEmpty {
                            head: BlockStatement {
                                id: UniqueId(
                                    2,
                                ),
                                span: Span {
                                    start: 16,
                                    end: 24,
                                },
                                kind: Return {
                                    value: Expr {
                                        span: Span {
                                            start: 23,
                                            end: 24,
                                        },
                                        id: UniqueId(
                                            1,
                                        ),
                                        kind: IntLit {
                                            value: 1,
                                        },
                                    },
                                },
                            },
                            tail: [],
                        },
                    },
                ),
            },
        ),
    },
]

This concludes the first half of our compiler. In the next part, we'll build an intermediate representation for our program, and start generating assembly code!

Two decades ago, during a boring afternoon in my teenage room, I set out to discover how the software and games I spent so many hours with were made. I plunged into a Google Search rabbit hole that would yield a puzzling answer: to build a piece of software, you need... a piece of software. A compiler or an interpreter, to be precise.

It took me an embarrassingly long time to understand what seemed to me like a paradox. But the journey of understanding how high-level code is transformed into machine code and interacts with the OS was fascinating.

In this series, we'll go through this journey together, one tiny iteration at a time. We'll build a program to compile a minimalistic language inspired by Python into x86 assembly code. Along the way, we'll learn many things, including how to write a lexer, a parser, a type checker, a garbage collector, and of course we'll discover the basics of x86 assembly as we implement our code generator. We'll also cover more advanced topics, such as code optimization, intermediate representations, and rich error messages.

The mile-high view

Before we dive into the implementation, let's take a moment to briefly discuss the high-level architecture of our compiler. In its simplest form, a compiler is a program that takes source code as input, and translates it into another language, typically assembly code or bytecode for a virtual machine. In our case, the source language will be Pylite, a minimalistic language designed specifically for this series, and the target language will be assembly code for the x86 family of processors.

One might think that this translation process is done in a single pass, emitting assembly code as we read the input source code. And indeed, this is how many early compilers worked. However, modern compilers are often implemented as a pipeline of stages, with each stage taking the output of the previous stages as input to transform or enrich it in some way. Our compilation pipeline will be quite typical, consisting of the following stages:

        ┌────────┐     ┌──────────┐     ┌─────────────┐
        │ PARSER │────▶│ SEMANTIC │────▶│     IR      │
        └────────┘     │ ANALYZER │     │  GENERATOR  │
                       └──────────┘     └─────────────┘
                                                 │
                                                 ▼
                       ┌─────────────┐     ┌─────────┐
                       │    CODE     │◀────│OPTIMIZER│
                       │  GENERATOR  │     └─────────┘
                       └─────────────┘

The parser's input is the raw source code, and the code generator produces the final assembly code. Here is a quick breakdown of each stage:

• parser: this stage reads the source code and produces a parse tree, which is a structured in-memory representation of the source code.
• semantic analyzer: this stage extracts semantic information from the parse tree. It performs type checking and name resolution, which is the process of associating names with their matching declarations.
• IR generator: for reasons that we'll explore later in the series, the parse tree is not the ideal representation for low-level optimizations and code generation. For this reason, it is transformed (or "lowered") into an intermediate representation (IR) by the IR generator before the optimizer and code generator stages.
• optimizer: this stage performs various optimizations on the IR to improve the performance of the generated code.
• code generator: generates the final assembly code from the optimized IR

Most modern compilers use some variation of this pipeline, often with additional stages with their own intermediate representations. The first three stages, the parser, the semantic analyzer and the IR generator, are often referred to as the front-end of the compiler, while the later stages are called the backend. Generally speaking, the front-end is responsible for analyzing the source code and producing an intermediate representation, while the backend is responsible for IR optimization and code generation.

With this overview in mind, let's move on to the next part, where we'll build a complete compiler for a tiny subset of the Pylite language!

As usual, the complete source code for this post is available on GitHub.

Setting up our test database

Before we can evaluate queries, we need a database to query. We'll start by creating a simple database with a single table, table1, with two columns, id and value:

sqlite3 queries_test.db
sqlite> create table table1(id integer, value text);
sqlite> insert into table1(id, value) values
    ...> (1, '11'),
    ...> (2, '12'),
    ...> (3, '13');
sqlite> .exit

⚠️ You might be tempted to use an existing SQLite database to test your queries, but keep in mind that our implementation does not support overflow pages yet, so it might not be able to read the data from your database file.

Making the pager shareable

This section is specific to the Rust implementation. If you're following along with another language, you can safely skip it!

Currently, our pager can only be used through an exclusive mutable reference. This was fine for our initial use cases, but as we start building more complex features, maintaining this restriction will constrain our design. We'll make the pager shareable by wrapping its inner mutable fields in an Arc<Mutex<_>> and Arc<RwLock<_>>. This will allow us to effectively clone the pager and use it from multiple places without running into borrow checker issues. At this stage of the project we could have chosen to use a simple Rc<RefCell<_>>, but we'll eventually need to support concurrent access to the pager, so we'll use thread-safe counterparts from the start.

// src/pager.rs

- #[derive(Debug, Clone)]
+ #[derive(Debug)]
pub struct Pager<I: Read + Seek = std::fs::File> {
-   input: I,
+   input: Arc<Mutex<I>>
    page_size: usize,
-   pages: HashMap<usize, page::Page>,
+   pages: Arc<RwLock<HashMap<usize, Arc<page::Page>>>>,
}

The read_page and load_page methods need to be updated accordingly:

impl<I: Read + Seek> Pager<I> {
    // [...] 
    pub fn read_page(&self, n: usize) -> anyhow::Result<Arc<page::Page>> {
        {
            let read_pages = self
                .pages
                .read()
                .map_err(|_| anyhow!("failed to acquire pager read lock"))?;

            if let Some(page) = read_pages.get(&n) {
                return Ok(page.clone());
            }
        }

        let mut write_pages = self
            .pages
            .write()
            .map_err(|_| anyhow!("failed to acquire pager write lock"))?;

        if let Some(page) = write_pages.get(&n) {
            return Ok(page.clone());
        }

        let page = self.load_page(n)?;
        write_pages.insert(n, page.clone());
        Ok(page)
    }

    fn load_page(&self, n: usize) -> anyhow::Result<Arc<page::Page>> {
        let offset = n.saturating_sub(1) * self.page_size;

        let mut input_guard = self
            .input
            .lock()
            .map_err(|_| anyhow!("failed to lock pager mutex"))?;

        input_guard
            .seek(SeekFrom::Start(offset as u64))
            .context("seek to page start")?;

        let mut buffer = vec![0; self.page_size];
        input_guard.read_exact(&mut buffer).context("read page")?;

        Ok(Arc::new(parse_page(&buffer, n)?))
    }
}

Two things to note regarding the read_page method:

• the initial attempt to read the page from the cache is nested in a block to limit the scope of the read lock, ensuring that it is released before we try to acquire the write lock

• after acquiring the write lock, we check again if the page is already in the cache, in case it was inserted in between the two lock acquisitions

Similarly, we'll define an owned version of our Value enum that we'll use in the query evaluator:

// src/value.rs

// [...]

#[derive(Debug, Clone)]
pub enum OwnedValue {
    Null,
    String(Rc<String>),
    Blob(Rc<Vec<u8>>),
    Int(i64),
    Float(f64),
}

impl<'p> From<Value<'p>> for OwnedValue {
    fn from(value: Value<'p>) -> Self {
        match value {
            Value::Null => Self::Null,
            Value::Int(i) => Self::Int(i),
            Value::Float(f) => Self::Float(f),
            Value::Blob(b) => Self::Blob(Rc::new(b.into_owned())),
            Value::String(s) => Self::String(Rc::new(s.into_owned())),
        }
    }
}

impl std::fmt::Display for OwnedValue {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            OwnedValue::Null => write!(f, "null"),
            OwnedValue::String(s) => s.fmt(f),
            OwnedValue::Blob(items) => {
                write!(
                    f,
                    "{}",
                    items
                        .iter()
                        .filter_map(|&n| char::from_u32(n as u32).filter(char::is_ascii))
                        .collect::<String>()
                )
            }
            OwnedValue::Int(i) => i.fmt(f),
            OwnedValue::Float(x) => x.fmt(f),
        }
    }
}

Finally, we'll enrich our Cursor struct with a method that returns the value of a field as an OwnedValue:

// src/cursor.rs

impl Cursor {
    // [...] 
    pub fn owned_field(&self, n: usize) -> Option<OwnedValue> {
        self.field(n).map(Into::into)
    }
    // [...]
}

Evaluating SELECT statements

Our query engine will be composed of two main components:

• an iterator-like Operator enum that represents nestable operations on the database, such as scanning a table or filtering rows. Our initial implementation will only contain a SeqScan operator that yields all rows from a table.

• a Planner struct that takes a parsed SQL query and produces an Operator that can be evaluated to produce the query result.

Let's start by defining the Operator enum:

// src/engine/operator.rs
use anyhow::Context;

use crate::{cursor::Scanner, value::OwnedValue};

#[derive(Debug)]
pub enum Operator {
    SeqScan(SeqScan),
}

impl Operator {
    pub fn next_row(&mut self) -> anyhow::Result<Option<&[OwnedValue]>> {
        match self {
            Operator::SeqScan(s) => s.next_row(),
        }
    }
}

The result of evaluating a query will be obtained by repeatedly calling the next_row method on the Operator until it returns None. Each value in the returned slice corresponds to a column in the query result.

The SeqScan struct will be responsible for scanning a table and yielding its rows:

// src/engine/operator.rs

// [...]

#[derive(Debug)]
pub struct SeqScan {
    fields: Vec<usize>,
    scanner: Scanner,
    row_buffer: Vec<OwnedValue>,
}

impl SeqScan {
    pub fn new(fields: Vec<usize>, scanner: Scanner) -> Self {
        let row_buffer = vec![OwnedValue::Null; fields.len()];

        Self {
            fields,
            scanner,
            row_buffer,
        }
    }

    fn next_row(&mut self) -> anyhow::Result<Option<&[OwnedValue]>> {
        let Some(record) = self.scanner.next_record()? else {
            return Ok(None);
        };

        for (i, &n) in self.fields.iter().enumerate() {
            self.row_buffer[i] = record.owned_field(n).context("missing record field")?;
        }

        Ok(Some(&self.row_buffer))
    }
}

The SeqScan struct is initialized with a list of field indices to read from each record and a Scanner that will yield the records for every row in the table to be scanned. As the number of fields to read is identical for every row, we can preallocate a buffer to store the values of the selected fields. The next_row method retrieves the next record from the scanner, extracts the requested fields (specified by their indices), and stores them in our buffer.

Now that we have an Operator to evaluate SELECT statements, let's move on to the Planner struct that will produce the Operator from a parsed SQL query:

// src/engine/plan.rs

use anyhow::{bail, Context, Ok};

use crate::{
    db::Db,
    sql::ast::{self, SelectFrom},
};

use super::operator::{Operator, SeqScan};

pub struct Planner<'d> {
    db: &'d Db,
}

impl<'d> Planner<'d> {
    pub fn new(db: &'d Db) -> Self {
        Self { db }
    }

    pub fn compile(self, statement: &ast::Statement) -> anyhow::Result<Operator> {
        match statement {
            ast::Statement::Select(s) => self.compile_select(s),
            stmt => bail!("unsupported statement: {stmt:?}"),
        }
    }
}

The Planner struct is initialized with a reference to the database and provides a compile method that takes a parsed SQL statement and returns the corresponding Operator. The compile method dispatches to a specific method for each type of SQL statement.

Let's see how to build an Operator for a SELECT statement:

// src/engine/plan.rs

impl<'d> Planner<'d> {
    // [...] 

    fn compile_select(self, select: &ast::SelectStatement) -> anyhow::Result<Operator> {
        let SelectFrom::Table(table_name) = &select.core.from;

        let table = self
            .db
            .tables_metadata
            .iter()
            .find(|m| &m.name == table_name)
            .with_context(|| format!("invalid table name: {table_name}"))?;

        let mut columns = Vec::new();

        for res_col in &select.core.result_columns {
            match res_col {
                ast::ResultColumn::Star => {
                    for i in 0..table.columns.len() {
                        columns.push(i);
                    }
                }
                ast::ResultColumn::Expr(e) => {
                    let ast::Expr::Column(col) = &e.expr;
                    let (index, _) = table
                        .columns
                        .iter()
                        .enumerate()
                        .find(|(_, c)| c.name == col.name)
                        .with_context(|| format!("invalid column name: {}", col.name))?;
                    columns.push(index);
                }
            }
        }

        Ok(Operator::SeqScan(SeqScan::new(
            columns,
            self.db.scanner(table.first_page),
        )))
    }
}

First, we find a table metadata entry that matches the table name in the SELECT statement. Then we iterate over the statement's result columns and build a list of field indices to read from each record, either by expanding * to all columns or by looking up the column name in the table metadata.

Finally, we create a SeqScan operator that will scan the entire tabl and yield the selected fields for each row.

Query evaluation in the REPL

It's time to put our query evaluator to the test! We'll create a simple function that reads a raw SQL query and evaluates it:

// src/main.rs

// [...]

fn eval_query(db: &db::Db, query: &str) -> anyhow::Result<()> {
    let parsed_query = sql::parse_statement(query, false)?;
    let mut op = engine::plan::Planner::new(db).compile(&parsed_query)?;

    while let Some(values) = op.next_row()? {
        let formated = values
            .iter()
            .map(ToString::to_string)
            .collect::<Vec<_>>()
            .join("|");

        println!("{formated}");
    }

    Ok(())
}

This function creates a pipeline: it parses the SQL query, builds an Operator with our Planner, and then repeatedly calls next_row() on the resulting operator to retrieve and display each row of the result.

The final step is to use this function in the REPL loop:

// src/main.rs

// [...]

 fn cli(mut db: db::Db) -> anyhow::Result<()> {
     print_flushed("rqlite> ")?;

     let mut line_buffer = String::new();

     while stdin().lock().read_line(&mut line_buffer).is_ok() {
         match line_buffer.trim() {
             ".exit" => break,
             ".tables" => display_tables(&mut db)?,
+            stmt => eval_query(&db, stmt)?, 
-            stmt => match sql::parse_statement(stmt, true) {
-                Ok(stmt) => {
-                    println!("{:?}", stmt);
-                }
-                Err(e) => {
-                    println!("Error: {}", e);
-                }
-            },
         }

         print_flushed("\nrqlite> ")?;

         line_buffer.clear();
     }

     Ok(())
 }

Now we can run the REPL and evaluate some simple SELECT statements:

cargo run -- queries_test.db
rqlite> select * from table1;

If everything went well, you should see the following output:

1|11
2|12
3|13

Conclusion

Our small database engine is starting to take shape! We can now parse and evaluate simple SELECT queries. But there's still a lot to cover before we can call it a fully functional database engine. In the next posts, we'll discover how to filter rows, read indexes, and implement sorting and grouping.

For each table, we need to know:

• the table name

• the root page

• the name and type of each column

The first two are very easy to extract, as they are directly stored in fields 1 and 3 of the schema table's records. But column names and types will be a bit trickier, as they are not neatly separated into record fields, but are stored in a single field in the form of a CREATE TABLE statement that we'll need to parse.

The complete source code is available on GitHub.

Parsing CREATE TABLE statements

The first step in extending our SQL parser to support CREATE TABLE statements it to add the necessary token types to the tokenizer. We'll support CREATE TABLE statements of the following form:

CREATE TABLE table_name
(
    column1_name column1_type,
    column2_name column2_type, 
    ...
)

The following tokens are new and need to be added to the Token enum: CREATE, TABLE, (, ).

// sql/tokenizer.rs

#[derive(Debug, Eq, PartialEq)]
pub enum Token {
+   Create,
+   Table,
    Select,
    As,
    From,
+   LPar,
+   RPar,
    Star,
    Comma,
    SemiColon,
    Identifier(String),
}

//[...]

pub fn tokenize(input: &str) -> anyhow::Result<Vec<Token>> {
    let mut tokens = Vec::new();
    let mut chars = input.chars().peekable();

    while let Some(c) = chars.next() {
        match c {
+           '(' => tokens.push(Token::LPar),
+           ')' => tokens.push(Token::RPar),
            '*' => tokens.push(Token::Star),
            ',' => tokens.push(Token::Comma),
            ';' => tokens.push(Token::SemiColon),
            c if c.is_whitespace() => continue,
            c if c.is_alphabetic() => {
                let mut ident = c.to_string().to_lowercase();
                while let Some(cc) = chars.next_if(|&cc| cc.is_alphanumeric() || cc == '_') {
                    ident.extend(cc.to_lowercase());
                }

                match ident.as_str() {
+                   "create" => tokens.push(Token::Create),
+                   "table" => tokens.push(Token::Table),
                    "select" => tokens.push(Token::Select),
                    "as" => tokens.push(Token::As),
                    "from" => tokens.push(Token::From),
                    _ => tokens.push(Token::Identifier(ident)),
                }
            }
            _ => bail!("unexpected character: {}", c),
        }
    }

    Ok(tokens)
}

Next, we need to extend our AST to represent the new statement type. Our representation will be based on the SQLite documentation.

// sql/ast.rs

//[...]

#[derive(Debug, Clone, Eq, PartialEq)]
pub enum Statement {
    Select(SelectStatement),
+   CreateTable(CreateTableStatement),
}
+
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub struct CreateTableStatement {
+    pub name: String,
+    pub columns: Vec<ColumnDef>,
+}
+
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub struct ColumnDef {
+    pub name: String,
+    pub col_type: Type,
+}
+
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub enum Type {
+    Integer,
+    Real,
+    Text,
+    Blob,
+}

//[...]

Parsing types is straightforward: we can simply match the incoming identifier token with a predefined set of types. For now, we'll restrict ourselves to INTEGER, REAL, TEXT, STRING, and BLOB. Once our parse_type method is implemented, constructing ColumnDef nodes is trivial.

// sql/parser.rs

//[...]
impl ParserState {
    // [...]
    fn parse_column_def(&mut self) -> anyhow::Result<ColumnDef> {
        Ok(ColumnDef {
            name: self.expect_identifier()?.to_string(),
            col_type: self.parse_type()?,
        })
    }

    fn parse_type(&mut self) -> anyhow::Result<Type> {
        let type_name = self.expect_identifier()?;
        let t = match type_name.to_lowercase().as_str() {
            "integer" => Type::Integer,
            "real" => Type::Real,
            "blob" => Type::Blob,
            "text" | "string" => Type::Text,
            _ => bail!("unsupported type: {type_name}"),
        };
        Ok(t)
    }
    // [...]
}

//[...]

In our implementation if the parse_create_table method, we'll parse column definitions using the same pattern as in the parse_result_colums method:

// sql/parser.rs

//[...]
impl ParserState {
    // [...]
    fn parse_create_table(&mut self) -> anyhow::Result<CreateTableStatement> {
        self.expect_eq(Token::Create)?;
        self.expect_eq(Token::Table)?;
        let name = self.expect_identifier()?.to_string();
        self.expect_eq(Token::LPar)?;
        let mut columns = vec![self.parse_column_def()?];
        while self.next_token_is(Token::Comma) {
            self.advance();
            columns.push(self.parse_column_def()?);
        }
        self.expect_eq(Token::RPar)?;
        Ok(CreateTableStatement { name, columns })
    }
    // [...]
}
//[...]

Finally, we need to update the parse_statement method to handle the new statement type. We'll also update the parse_statement utility function to make the semicolon terminator optional, as the CREATE TABLE statements stored in the schema table lack a trailing semicolon.

// sql/parser.rs

//[...]

impl ParserState {
    // [...]

    fn parse_statement(&mut self) -> anyhow::Result<Statement> {
-       Ok(ast::Statement::Select(self.parse_select()?))
+       match self.peak_next_token().context("unexpected end of input")? {
+           Token::Select => self.parse_select().map(Statement::Select),
+           Token::Create => self.parse_create_table().map(Statement::CreateTable),
+           token => bail!("unexpected token: {token:?}"),
+       }
    }    

    // [...]
}

// [...]

-pub fn parse_statement(input: &str) -> anyhow::Result<Statement> {
+pub fn parse_statement(input: &str, trailing_semicolon: bool) -> anyhow::Result<Statement> {
    let tokens = tokenizer::tokenize(input)?;
    let mut state = ParserState::new(tokens);
    let statement = state.parse_statement()?;
+   if trailing_semicolon {
        state.expect_eq(Token::SemiColon)?;
+   }
    Ok(statement)
}

+pub fn parse_create_statement(
+    input: &str,
+) -> anyhow::Result<CreateTableStatement> {
+    match parse_statement(input, false)? {
+        Statement::CreateTable(c) => Ok(c),
+        Statement::Select(_) => bail!("expected a create statement"),
+    }
+}

Reading metadata

Now that we have the necessary building blocks to read table metadata, we can extend our Database struct to store this information. The TableMetadata::from_cursor method builds a TableMetadata struct from a Cursor object, which represents a record in the schema table. The create statement and first page are extracted from fields 4 and 3, respectively.

As records from the schema table contain informations about other kinds of objects, such as triggers, we check the type field at index 0 to ensure we're dealing with a table.

Finally, in Db::collect_metadata, we iterate over all the records in the schema table, collecting table metadata for each table record we encounter.

// db.rs

+#[derive(Debug, Clone)]
+pub struct TableMetadata {
+    pub name: String,
+    pub columns: Vec<ast::ColumnDef>,
+    pub first_page: usize,
+}

+impl TableMetadata {
+   fn from_cursor(cursor: Cursor) -> anyhow::Result<Option<Self>> {
+       let type_value = cursor
+           .field(0)
+           .context("missing type field")
+           .context("invalid type field")?;

+       if type_value.as_str() != Some("table") {
+           return Ok(None);
+       }

+       let create_stmt = cursor
+           .field(4)
+           .context("missing create statement")
+           .context("invalid create statement")?
+           .as_str()
+           .context("table create statement should be a string")?
+           .to_owned();

+       let create = sql::parse_create_statement(&create_stmt)?;

+       let first_page = cursor
+           .field(3)
+           .context("missing table first page")?
+           .as_int()
+           .context("table first page should be an integer")? as usize;

+       Ok(Some(TableMetadata {
+           name: create.name,
+           columns: create.columns,
+           first_page,
+       }))
+    }
+}

pub struct Db {
    pub header: DbHeader,
+   pub tables_metadata: Vec<TableMetadata>,
    pager: Pager,
}

impl Db {
    pub fn from_file(filename: impl AsRef<Path>) -> anyhow::Result<Db> {
        let mut file = std::fs::File::open(filename.as_ref()).context("open db file")?;

        let mut header_buffer = [0; pager::HEADER_SIZE];
        file.read_exact(&mut header_buffer)
            .context("read db header")?;

        let header = pager::parse_header(&header_buffer).context("parse db header")?;

+       let tables_metadata = Self::collect_tables_metadata(&mut Pager::new(
+           file.try_clone()?,
+           header.page_size as usize,
+       ))?;

        let pager = Pager::new(file, header.page_size as usize);

        Ok(Db {
            header,
            pager,
+           tables_metadata,
        })
    }

+   fn collect_tables_metadata(pager: &mut Pager) -> anyhow::Result<Vec<TableMetadata>> {
+       let mut metadata = Vec::new();
+       let mut scanner = Scanner::new(pager, 1);

+       while let Some(record) = scanner.next_record()? {
+           if let Some(m) = TableMetadata::from_cursor(record)? {
+               metadata.push(m);
+           }
+       }

+       Ok(metadata)
+   }

    // [...]
}

Our initial implementation of the .table command can be updated to use the new metadata:

// main.rs

fn display_tables(db: &mut db::Db) -> anyhow::Result<()> {
-   let mut scanner = db.scanner(1);
-
-   while let Some(mut record) = scanner.next_record()? {
-       let type_value = record
-           .field(0)
-           .context("missing type field")
-           .context("invalid type field")?;

-       if type_value.as_str() == Some("table") {
-           let name_value = record
-               .field(1)
-               .context("missing name field")
-               .context("invalid name field")?;

-           print!("{} ", name_value.as_str().unwrap());
-       }
-   }
+   for table in &db.tables_metadata {
+       print!("{} ", &table.name)
+   }

    Ok(())
}

Conclusion

We've extended our SQL parser to support CREATE TABLE statements and used it to extract metadata from the schema table. By parsing the schema, we now have a way to understand the structure of tables in our database.

In the next post, we'll leverage this metadata to build a query evaluator that can execute simple SELECT queries against user-defined tables, bringing us one step closer to a fully functional database engine.

As the SQL dialect supported by SQLite is quite large and complex, we'll initially limit ourselves to a subset that comprises only the select statement, in a striped-down form. Only expressions of the form select <columns> from <table> will be supported, where <columns> is either * or a comma-separated list of columns names (with an optional as alias), and <table> is the name of a table.

The full SQL syntax, as implemented in SQLite is described in great detail in the SQL As Understood By SQLite document.

Parsing Basics

Our SQL parser will follow a conventional 2 steps process: lexical analysis (or tokenization) and syntax analysis (or parsing).

The lexical analysis step takes the input SQL string and groups individual characters into tokens, which are meaningful units of the language. For example, the character sequence S-E-L-E-C-T will be grouped into a single token of type select, and the sequence * will be grouped into a token of type star. This stage is also responsible for discarding whitespace and normalizing the case of the input.

The syntax analysis step takes the stream of tokens produced by the lexical analysis, and tries to match them against the syntax rules of the language. Its output is an abstract syntax tree (AST), which is a hierarchical representation of the input SQL.

Writing the tokenizer

The first step in writing our tokenizer is to define a Token type that will represent the individual tokens of our SQL dialect. This definition will live in a new module: sql::tokenizer.

// sql/tokenizer.rs
#[derive(Debug, Eq, PartialEq)]
pub enum Token {
    Select,
    As,
    From,
    Star,
    Comma,
    SemiColon,
    Identifier(String),
}

impl Token {
    pub fn as_identifier(&self) -> Option<&str> {
        match self {
            Token::Identifier(ident) => Some(ident),
            _ => None,
        }
    }
}

We also define a utility function as_identifier that will return the string value of a token if it is an Identifier, and None otherwise.

The logic of the tokenize function is quite simple: we iterate over the input string's characters, and based on the current character we decide which token to emit:

• if the character matches a single-character token, we emit it immediately

• if the character is a whitespace, it is discarded

• finally, if the character is a letter, we start a new identifier token and keep accumulating characters until we reach a character that is not a valid identifier character. At this point, if the accumulated string is a keyword, we emit the corresponding token, otherwise, we emit a raw Identifier token.

// sql/tokenizer.rs
use anyhow::bail;

pub fn tokenize(input: &str) -> anyhow::Result<Vec<Token>> {
    let mut tokens = Vec::new();
    let mut chars = input.chars().peekable();

    while let Some(c) = chars.next() {
        match c {
            '*' => tokens.push(Token::Star),
            ',' => tokens.push(Token::Comma),
            ';' => tokens.push(Token::SemiColon),
            c if c.is_whitespace() => continue,
            c if c.is_alphabetic() => {
                let mut ident = c.to_string().to_lowercase();
                while let Some(cc) = chars.next_if(|&cc| cc.is_alphanumeric() || cc == '_') {
                    ident.extend(cc.to_lowercase());
                }

                match ident.as_str() {
                    "select" => tokens.push(Token::Select),
                    "as" => tokens.push(Token::As),
                    "from" => tokens.push(Token::From),
                    _ => tokens.push(Token::Identifier(ident)),
                }
            }
            _ => return Err(anyhow::anyhow!("unexpected character: {}", c)),
        }
    }

    Ok(tokens)
}

Since SQL is case-insensitive, all identifiers are normalized to lower case.

Representing SQL statements

Before we dive into the implementation of the parser, we need to decide how to represent SQL statements in our code. We'll settle on a conventional representation, based on the description of the SQL syntax in the SQLite documentation, and write the corresponding Rust types in a new module sql::ast.

// sql/ast.rs

#[derive(Debug, Clone, Eq, PartialEq)]
pub enum Statement {
    Select(SelectStatement),
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct SelectStatement {
    pub core: SelectCore,
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct SelectCore {
    pub result_columns: Vec<ResultColumn>,
    pub from: SelectFrom,
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub enum ResultColumn {
    Star,
    Expr(ExprResultColumn),
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct ExprResultColumn {
    pub expr: Expr,
    pub alias: Option<String>,
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub enum Expr {
    Column(Column),
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct Column {
    pub name: String,
}

#[derive(Debug, Clone, Eq, PartialEq)]
pub enum SelectFrom {
    Table(String),
}

The following query:

select col1 as first, col2
from table

Will be parsed into the following rust structure:

Statement::Select(SelectStatement {
    core: SelectCore {
        result_columns: vec![
            ResultColumn::Expr(ExprResultColumn {
                expr: Expr::Column(Column {
                     name: "col1".to_string()
                }),
                alias: Some("first".to_string())
            }),
            ResultColumn::Expr(ExprResultColumn {
                expr: Expr::Column(Column {
                    name: "col2".to_string()
                }),
               alias: None
            }),
       ],
       from: SelectFrom::Table("table".to_string()),
    },
})

You may notice a few redundancies in this representation, such as the Expr enum that comprises a single variant. This is intentional, as it will allow us to add new syntactic constructs in future episodes without breaking too much of the existing code.

Writing the parser

Parsing algorithms come in all shapes and sizes, and a full discussion of the topic if beyond the scope of this article. The one we'll use here is called recursive descent and is reasonably simple to understand and implement:

• for every node type, we'll define a function that tries to build the node from the current input tokens, and fails if it is not possible. For example, we'll define a method that builds a Column node by consuming an Identifier token, and fails if the current token is not an Identifier token.

• complex "nested" nodes are build by delegating the parsing of their child nodes to other functions. For example, ExprResultColmn is build by parsing an Expr node and an optional as token followed by an Identifier token.

In a fully-fledged parser, these functions can be mutually recursive.

First, let's define a ParserState struct that will hold the state of the parser: the list of tokens, and the current position in the list.

// sql/parser.rs

use anyhow::{bail, Context};

use crate::sql::{
    ast::{
        Column, Expr, ExprResultColumn, ResultColumn, SelectCore, SelectFrom, SelectStatement,
        Statement,
    },
    tokenizer::{self, Token},
};

#[derive(Debug)]
struct ParserState {
    tokens: Vec<Token>,
    pos: usize,
}

impl ParserState {
    fn new(tokens: Vec<Token>) -> Self {
        Self { tokens, pos: 0 }
    }

    fn next_token_is(&self, expected: Token) -> bool {
        self.tokens.get(self.pos) == Some(&expected)
    }

    fn expect_identifier(&mut self) -> anyhow::Result<&str> {
        self.expect_matching(|t| matches!(t, Token::Identifier(_)))
            .map(|t| t.as_identifier().unwrap())
    }

    fn expect_eq(&mut self, expected: Token) -> anyhow::Result<&Token> {
        self.expect_matching(|t| *t == expected)
    }

    fn expect_matching(&mut self, f: impl Fn(&Token) -> bool) -> anyhow::Result<&Token> {
        match self.next_token() {
            Some(token) if f(token) => Ok(token),
            Some(token) => bail!("unexpected token: {:?}", token),
            None => bail!("unexpected end of input"),
        }
    }

    fn peak_next_token(&self) -> anyhow::Result<&Token> {
        self.tokens.get(self.pos).context("unexpected end of input")
    }

    fn next_token(&mut self) -> Option<&Token> {
        let token = self.tokens.get(self.pos);
        if token.is_some() {
            self.advance();
        }
        token
    }

    fn advance(&mut self) {
        self.pos += 1;
    }
}

• current_token_is checks if the current token is equal to the expected token

• expect_identifier returns the content of the current token if it is an Identifier, and fails otherwise

• expect_eq checks if the current token is equal to the expected token, and fails otherwise

• peak_next_token allows us to look at the next token without consuming it, and fails if there are no more tokens

• next_token returns the current token and advances the parser's position

• advance increments the parser's position

Armed with these primitives, we can write our simplest parser function: parse_expr! As the only expressions that we support for now are identifiers, the parsing function only has to check that the current token is an Identifier token and build a Expr node from its value.

// sql/parser.rs

impl ParserState {
    //...
    fn parse_expr(&mut self) -> anyhow::Result<Expr> {
        Ok(Expr::Column(Column {
            name: self.expect_identifier()?.to_string(),
        }))
    }
    //...
}

A bit more involved, the parse_expr_result_column function parses terms of the form columnName or columnName as alias. It starts by parsing the initial Expr node (columnName, in our examples), then if the next token is as, it consumes it and parses the Identifier token that follows.

// sql/parser.rs

impl ParserState {
    //...
    fn parse_expr_result_column(&mut self) -> anyhow::Result<ExprResultColumn> {
        let expr = self.parse_expr()?;
        let alias = if self.next_token_is(Token::As) {
            self.advance();
            Some(self.expect_identifier()?.to_string())
        } else {
            None
        };
        Ok(ExprResultColumn { expr, alias })
    }
    //...
}

ResultColumn can represent terms of the form described above, or * to represent all columns of a table. The parse_result_column function checks if the current token is *, and returns a Star node if it is. Otherwise, it delegates the parsing of the ExprResultColumn node to the parse_expr_result_column function.

// sql/parser.rs

impl ParserState {
    //...
    fn parse_result_column(&mut self) -> anyhow::Result<ResultColumn> {
        if self.peak_next_token()? == &Token::Star {
            self.advance();
            return Ok(ResultColumn::Star);
        }

        Ok(ResultColumn::Expr(self.parse_expr_result_column()?))
    }
    //...
}

Another interesting example is the parse_result_colums function, which parses a list of columns separated by commas. It starts by parsing the first column, then iterates over the following tokens as long as the token following a result column is a comma, accumulating the parsed columns in a vector.

// sql/parser.rs

impl ParserState {
    //...
    fn parse_result_columns(&mut self) -> anyhow::Result<Vec<ResultColumn>> {
        let mut result_coluns = vec![self.parse_result_column()?];
        while self.next_token_is(Token::Comma) {
            self.advance();
            result_coluns.push(self.parse_result_column()?);
        }
        Ok(result_coluns)
    }
    //...
}

As you are probably getting the hang of it, implementing the remaining parsing functions can be a fun exercise. In any case, here is my implementation for reference:

// sql/parser.rs

impl ParserState {
    //...
    fn parse_statement(&mut self) -> anyhow::Result<Statement> {
        Ok(Statement::Select(self.parse_select()?))
    }

    fn parse_select(&mut self) -> anyhow::Result<SelectStatement> {
        self.expect_eq(Token::Select)?;
        let result_columns = self.parse_result_columns()?;
        self.expect_eq(Token::From)?;
        let from = self.parse_select_from()?;
        Ok(SelectStatement {
            core: SelectCore {
                result_columns,
                from,
            },
        })
    }

    fn parse_select_from(&mut self) -> anyhow::Result<SelectFrom> {
        let table = self.expect_identifier()?;
        Ok(SelectFrom::Table(table.to_string()))
    }
    //...
}

The final piece of the puzzle is a function that ties everything together, taking an input SQL string, tokenizing it, and parsing it into an AST:

// sql/parser.rs

//...

pub fn parse_statement(input: &str) -> anyhow::Result<Statement> {
    let tokens = tokenizer::tokenize(input)?;
    let mut state = ParserState::new(tokens);
    let statement = state.parse_statement()?;
    state.expect_eq(Token::SemiColon)?;
    Ok(statement)
}

Putting it all together

We've covered a lot of ground! Now is the time to test our parser on some actual SQL queries. To that end, let's alter our REPL loop to parse then input as an SQL statement if it does not match a know command, and print it.

// src/main.rs

+ mod sql;

//...

fn cli(mut db: db::Db) -> anyhow::Result<()> {
    print_flushed("rqlite> ")?;

    let mut line_buffer = String::new();

    while stdin().lock().read_line(&mut line_buffer).is_ok() {
        match line_buffer.trim() {
            ".exit" => break,
            ".tables" => display_tables(&mut db)?,
+            stmt => match sql::parse_statement(stmt) {
+                Ok(stmt) => {
+                    println!("{:?}", stmt);
+                }
+                Err(e) => {
+                    println!("Error: {}", e);
+                }
+            },
-            _ => {
-               println!("Unrecognized command '{}'", line_buffer.trim());
-           }
        }

        print_flushed("\nrqlite> ")?;

        line_buffer.clear();
    }

    Ok(())
}

//...

Conclusion

Our database can read data and parse very simple SQL statements. In the next part of this series, we'll bridge the gap between these two functionalities and build a small query engine that compiles SQL queries into execution plans and executes these plans against the persisted data.

A motivating example

Let's begin our journey with a much larger test database:

for i in {1..1000}; do            
    sqlite3 res/test.db "create table table$i(id integer)"
done

cargo run --release -- res/test.db
rqlite> .tables

Without much surprise, our small program isn't able to display the list of tables. The reason for that is quite simple: database pages are typically 4096 bytes long, which is far from enough to store 1000 tables. But why did our code fail, instead of displaying the first records that fit into the first page?

B-tree interior pages

When a table is too large to fit into a single page, SQLite splits it into multiple pages, of different types:

• leaf pages, that contains the actual records

• interior pages, that store information about which page contains the records for which table.

Interior tables have the same high-level structure as leaf pages, with two key differences:

• instead of storing record, they store a tuple (key, child_page_number) where child_page_number is a 32 bits unsigned integer representing the page number of the "root" page of a subtree that contains records with keys lower or equal to key. Cells in interior pages are logically ordered by key in ascending order.

• the header contains an extra field, the "rightmost pointer", which is the page number of the "root" of the subtree that contains records with keys greater than the largest key in the page.

With this new knowledge, we can update our page data structure. We'll start by adding the new optional rightmost_pointer field to the page header. We'll also add a byte_size method that returns the size of the header, depending on wheter the rightmost_pointer field is set or not, and add a new variant to our PageType enum to represent interior pages.

// src/page.rs

#[derive(Debug, Copy, Clone, Eq, PartialEq)]
pub enum PageType {
    TableLeaf,
+   TableInterior,
}

#[derive(Debug, Copy, Clone)]
pub struct PageHeader {
    pub page_type: PageType,
    pub first_freeblock: u16,
    pub cell_count: u16,
    pub cell_content_offset: u32,
    pub fragmented_bytes_count: u8,
+   pub rightmost_pointer: Option<u32>,
}

+impl PageHeader {
+    pub fn byte_size(&self) -> usize {
+        if self.rightmost_pointer.is_some() {
+            12
+        } else {
+            8
+        }
+    }
+}

Let's modify the parsing function to take the new field into account:

// src/pager.rs

+ const PAGE_LEAF_TABLE_ID: u8 = 0x0d;
+ const PAGE_INTERIOR_TABLE_ID: u8 = 0x05;

fn parse_page_header(buffer: &[u8]) -> anyhow::Result<page::PageHeader> {
-   let page_type = match buffer[0] {
-      0x0d => page::PageType::TableLeaf,
+   let (page_type, has_rightmost_ptr) = match buffer[0] {
+       PAGE_LEAF_TABLE_ID => (page::PageType::TableLeaf, false),
+       PAGE_INTERIOR_TABLE_ID => (page::PageType::TableInterior, true),
        _ => anyhow::bail!("unknown page type: {}", buffer[0]),
    };

    let first_freeblock = read_be_word_at(buffer, PAGE_FIRST_FREEBLOCK_OFFSET);
    let cell_count = read_be_word_at(buffer, PAGE_CELL_COUNT_OFFSET);
    let cell_content_offset = match read_be_word_at(buffer, PAGE_CELL_CONTENT_OFFSET) {
        0 => 65536,
        n => n as u32,
    };
    let fragmented_bytes_count = buffer[PAGE_FRAGMENTED_BYTES_COUNT_OFFSET];

+   let rightmost_pointer = if has_rightmost_ptr {
+       Some(read_be_double_at(buffer, PAGE_RIGHTMOST_POINTER_OFFSET))
+   } else {
+       None
+   };

    Ok(page::PageHeader {
        page_type,
        first_freeblock,
        cell_count,
        cell_content_offset,
        fragmented_bytes_count,
+       rightmost_pointer,
    })
}

We decide whether to parse the rightmost_pointer field depending on the value of the page_type byte (0x0d for leaf pages, 0x05 for interior pages).

Next, we'll update the Page struct to reflect the fact that both leaf and interior pages share the same structure, with the only difference being the content of the cells:

// src/page.rs

#[derive(Debug, Clone)]
- pub struct TableLeafPage {
+ pub struct Page {
    pub header: PageHeader,
    pub cell_pointers: Vec<u16>,
-   pub cells: Vec<TableLeafCell>,
+   pub cells: Vec<Cell>,
}

- #[derive(Debug, Clone)]
- pub enum Page {
-   TableLeaf(TableLeafPage),
- }

+ #[derive(Debug, Clone)]
+ pub enum Cell {
+    TableLeaf(TableLeafCell),
+    TableInterior(TableInteriorCell),
+ }

+ impl From<TableLeafCell> for Cell {
+    fn from(cell: TableLeafCell) -> Self {
+        Cell::TableLeaf(cell)
+    }
+ }

+ impl From<TableInteriorCell> for Cell {
+    fn from(cell: TableInteriorCell) -> Self {
+        Cell::TableInterior(cell)
+    }
+ }

+ pub struct TableInteriorCell {
+    pub left_child_page: u32,
+    pub key: i64,
+ }

This change calls for a major update of our parsing functions, reproduced below:

// src/pager.rs

fn parse_page(buffer: &[u8], page_num: usize) -> anyhow::Result<page::Page> {
    let ptr_offset = if page_num == 1 { HEADER_SIZE as u16 } else { 0 };
    let content_buffer = &buffer[ptr_offset as usize..];
    let header = parse_page_header(content_buffer)?;
    let cell_pointers = parse_cell_pointers(
        &content_buffer[header.byte_size()..],
        header.cell_count as usize,
        ptr_offset,
    );

    let cells_parsing_fn = match header.page_type {
        page::PageType::TableLeaf => parse_table_leaf_cell,
        page::PageType::TableInterior => parse_table_interior_cell,
    };

    let cells = parse_cells(content_buffer, &cell_pointers, cells_parsing_fn)?;

    Ok(page::Page {
        header,
        cell_pointers,
        cells,
    })
}

fn parse_cells(
    buffer: &[u8],
    cell_pointers: &[u16],
    parse_fn: impl Fn(&[u8]) -> anyhow::Result<page::Cell>,
) -> anyhow::Result<Vec<page::Cell>> {
    cell_pointers
        .iter()
        .map(|&ptr| parse_fn(&buffer[ptr as usize..]))
        .collect()
}

fn parse_table_leaf_cell(mut buffer: &[u8]) -> anyhow::Result<page::Cell> {
    let (n, size) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

    let (n, row_id) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

    let payload = buffer[..size as usize].to_vec();

    Ok(page::TableLeafCell {
        size,
        row_id,
        payload,
    }
        .into())
}

fn parse_table_interior_cell(mut buffer: &[u8]) -> anyhow::Result<page::Cell> {
    let left_child_page = read_be_double_at(buffer, 0);
    buffer = &buffer[4..];

    let (_, key) = read_varint_at(buffer, 0);

    Ok(page::TableInteriorCell {
        left_child_page,
        key,
    }
        .into())
}

Scanning logic

Our scanning logic will need to be updated to handle interior pages. We can no longer simply iterate over the cells of a page and call it a day. Instead, we'll need to implement a depth-first algorithm that recursively explores the tree, starting from the root page.

To make our task easier, let's introduce a new PositionedPage struct that stores a page, along with the index of the current cell we're looking at:

// src/pager.rs

#[derive(Debug)]
pub struct PositionedPage {
    pub page: Page,
    pub cell: usize,
}

impl PositionedPage {
    pub fn next_cell(&mut self) -> Option<&Cell> {
        let cell = self.page.get(self.cell);
        self.cell += 1;
        cell
    }

    pub fn next_page(&mut self) -> Option<u32> {
        if self.page.header.page_type == PageType::TableInterior
            && self.cell == self.page.cells.len()
        {
            self.cell += 1;
            self.page.header.rightmost_pointer
        } else {
            None
        }
    }
}

The next_cell method returns the content of the current cell and increments the cell index, so calling it repeatedly will yiels the content of all the cells in the page.

The next_page method is a bit more complex: it returns the rightmost_pointer of the current page if it's an interior page and we just visited the last cell, otherwise it it returns None.

We'll also update our Cursor so that it owns it's payload instead of borrowing it through a Pager:

// src/pager.rs

#[derive(Debug)]
- pub struct Cursor<'p> { 
+ pub struct Cursor {
    header: RecordHeader,
-   pager: &'p mut Pager,
-   page_index: usize,
-   page_cell: usize,
+    payload: Vec<u8>,
}

This change will allow us to avoid borrowing the Pager mutably from the Cursor and the Scanner at the same time, which would lead to a difficult-to-solve lifetime issue.

With that out of the way, we can focus on the tree scanning logic. We'll maintain a stack of PositionedPage to keep track of the parent pages we've visited. At every step of the walk, there are a few cases to consider:

• if the current page is a leaf page and we haven't visited all the cells yet, we'll just have to build a Cursor with the current cell's payload and return it.

• if the current page is an interior page, we'll push the next page (either from the current cell or the rightmost pointer) to the stack and continue the walk.

• if we've visited all the cells of the current page, we'll pop the stack and continue the walk from the parent page.

This logic is implemented in the new Scanner struct:

// src/pager.rs

#[derive(Debug)]
pub struct Scanner<'p> {
    pager: &'p mut Pager,
    initial_page: usize,
    page_stack: Vec<PositionedPage>,
}

impl<'p> Scanner<'p> {
    pub fn new(pager: &'p mut Pager, page: usize) -> Scanner<'p> {
        Scanner {
            pager,
            initial_page: page,
            page_stack: Vec::new(),
        }
    }

    pub fn next_record(&mut self) -> anyhow::Result<Option<Cursor>> {
        loop {
            match self.next_elem() {
                Ok(Some(ScannerElem::Cursor(cursor))) => return Ok(Some(cursor)),
                Ok(Some(ScannerElem::Page(page_num))) => {
                    let new_page = self.pager.read_page(page_num as usize)?.clone();
                    self.page_stack.push(PositionedPage {
                        page: new_page,
                        cell: 0,
                    });
                }
                Ok(None) if self.page_stack.len() > 1 => {
                    self.page_stack.pop();
                }
                Ok(None) => return Ok(None),
                Err(e) => return Err(e),
            }
        }
    }

    fn next_elem(&mut self) -> anyhow::Result<Option<ScannerElem>> {
        let Some(page) = self.current_page()? else {
            return Ok(None);
        };

        if let Some(page) = page.next_page() {
            return Ok(Some(ScannerElem::Page(page)));
        }

        let Some(cell) = page.next_cell() else {
            return Ok(None);
        };

        match cell {
            Cell::TableLeaf(cell) => {
                let header = parse_record_header(&cell.payload)?;
                Ok(Some(ScannerElem::Cursor(Cursor {
                    header,
                    payload: cell.payload.clone(),
                })))
            }
            Cell::TableInterior(cell) => Ok(Some(ScannerElem::Page(cell.left_child_page))),
        }
    }

    fn current_page(&mut self) -> anyhow::Result<Option<&mut PositionedPage>> {
        if self.page_stack.is_empty() {
            let page = match self.pager.read_page(self.initial_page) {
                Ok(page) => page.clone(),
                Err(e) => return Err(e),
            };

            self.page_stack.push(PositionedPage { page, cell: 0 });
        }

        Ok(self.page_stack.last_mut())
    }
}

#[derive(Debug)]
enum ScannerElem {
    Page(u32),
    Cursor(Cursor),
}

Putting it all together

The only change that remains to be made is to update the display_tables function to account for the change in next_record signature:

// src/main.rs

fn display_tables(db: &mut db::Db) -> anyhow::Result<()> {
    let mut scanner = db.scanner(1);

-   while let Some(Ok(mut record)) = scanner.next_record() {
+   while let Some(mut record) = scanner.next_record()? {
        let type_value = record
            .field(0)
            .context("missing type field")
            .context("invalid type field")?;

        if type_value.as_str() == Some("table") {
            let name_value = record
                .field(1)
                .context("missing name field")
                .context("invalid name field")?;

            print!("{} ", name_value.as_str().unwrap());
        }
    }

    Ok(())
}

We can now display our (long!) list of tables:

cargo run --release -- res/test.db
rqlite> .tables

Conclusion

Our scanning logic is now able to handle tables that span multiple pages, thanks to the introduction of interior pages. This is a major milestone in our journey to build a fully functional database! In the next post, we'll learn how to parse simple SQL queries and will lay the groundwork for our query engine.

Source code examples will be provided in Rust, but you are encouraged to follow along using your language of choice, as we won't be relying on many language-specific features or libraries.

As an introduction, we'll implement the simplest version of the tables command, which lists the names of all the tables in a database. While this looks simple, we'll see that it requires us to make our first deep dive into the SQLite file format.

Building the test database

To keep things as simple as possible, let's build a minimalistic test database:

sqlite3 minimal_test.db
sqlite> create table table1(id integer);
sqlite> create table table2(id integer);
sqlite> .exit

This creates a database with two tables, table1 and table2, each with a single column, id. We can verify this by running the tables command in the SQLite shell:

sqlite3 minimal_test.db
sqlite> .tables
table1  table2
sqlite> .exit

Bootstrapping the project

Let's start by creating a new Rust project. We'll use the cargo add to add our only dependency for now, anyhow:

cargo new rsqlite
cd rsqlite
cargo add anyhow

The SQLite file format

SQLite databases are stored in a single file, the format of which is documented in the SQLite File Format Specification. The file is divided into pages, with each page having the same size: a power of 2, between 512 and 65536 bytes. The first 100 bytes of the first page contain the database header, which includes information such as the page size and the file format version. In this first part, we'll only be interested in the page size. Pages can be of different types, but for this first article, we'll only be interested in table btree leaf pages, which store the actual table data.

Our first task will be to implement a Pager struct that reads and caches pages from the database file. But before we do, we'll have to read the page size from the database header. Let's start by defining our Header struct:

// src/page.rs
#[derive(Debug, Copy, Clone)]
pub struct DbHeader {
    pub page_size: u32,
}

The header starts with the magic string SQLite format 3\0, followed by the page size encoded as a big-endian 2-byte integer at offset 16. With this information, we can implement a function that reads the header from a buffer:

// src/pager.rs
pub const HEADER_SIZE: usize = 100;
const HEADER_PREFIX: &[u8] = b"SQLite format 3\0";
const HEADER_PAGE_SIZE_OFFSET: usize = 16;

const PAGE_MAX_SIZE: u32 = 65536;

pub fn parse_header(buffer: &[u8]) -> anyhow::Result<page::DbHeader> {
    if !buffer.starts_with(HEADER_PREFIX) {
        let prefix = String::from_utf8_lossy(&buffer[..HEADER_PREFIX.len()]);
        anyhow::bail!("invalid header prefix: {prefix}");
    }

    let page_size_raw = read_be_word_at(buffer, HEADER_PAGE_SIZE_OFFSET);
    let page_size = match page_size_raw {
        1 => PAGE_MAX_SIZE,
        n if ((n & (n - 1)) == 0) && n != 0 => n as u32,
        _ => anyhow::bail!("page size is not a power of 2: {}", page_size_raw),
    };

    Ok(page::Header { page_size })
}

fn read_be_word_at(input: &[u8], offset: usize) -> u16 {
    u16::from_be_bytes(input[offset..offset + 2].try_into().unwrap())
}

Two things to note here:

• As the maximum page size cannot be represented as a 2-byte integer, a page size of 1 is use to represent the maximum page size.
• We use a somewhat convoluted expression to check if the page size is a power of 2. The expression n & (n - 1) == 0 is true if and only if n is a power of 2, except for n = 0.

Decoding Table B-tree leaf pages

Now that we have the minimum information we need to read pages from the disk, let's explore the content of a table btree-leaf page. table btree-leaf pages start with an 8-byte header, followed by an sequence of "cell pointers" containing the offset of every cell in the page. The cells contain the table data, and we can think of them as key-value pairs, where the key is a 64-bits integer encoded as a varint (the rowid) and the value is an arbitrary sequence of bytes representing the row data. The header contains the following fields:

• page_type: byte representing the page type. For table btree-leaf pages, this is 0x0D.
• first_freeblock: 2-byte integer representing the offset of the first free block in the page, or zero if there is no freeblock.
• cell_count: 2-byte integer representing the number of cells in the page.
• cell_content_offset: 2-byte integer representing the offset of the first cell.
• fragmented_bytes_count: 1-byte integer representing the number of fragmented free bytes in the page (we won't make use of it for now).

We'll start by defining a Page enum representing a parsed page, along with the necessary structs to represent the page header and the cell pointers:

#[derive(Debug, Clone)]
pub enum Page {
    TableLeaf(TableLeafPage),
}

#[derive(Debug, Clone)]
pub struct TableLeafPage {
    pub header: PageHeader,
    pub cell_pointers: Vec<u16>,
    pub cells: Vec<TableLeafCell>,
}

#[derive(Debug, Copy, Clone)]
pub struct PageHeader {
    pub page_type: PageType,
    pub first_freeblock: u16,
    pub cell_count: u16,
    pub cell_content_offset: u32,
    pub fragmented_bytes_count: u8,
}

#[derive(Debug, Copy, Clone)]
pub enum PageType {
    TableLeaf,
}

#[derive(Debug, Clone)]
pub struct TableLeafCell {
    pub size: i64,
    pub row_id: i64,
    pub payload: Vec<u8>,
}

The corresponding parsing functions are quite straightforward. Note the offset handling in parse_page: since the first page contains the database header, we start parsing the page at offset 100.

/// pager.rs
const PAGE_LEAF_HEADER_SIZE: usize = 8;
const PAGE_FIRST_FREEBLOCK_OFFSET: usize = 1;
const PAGE_CELL_COUNT_OFFSET: usize = 3;
const PAGE_CELL_CONTENT_OFFSET: usize = 5;
const PAGE_FRAGMENTED_BYTES_COUNT_OFFSET: usize = 7;

fn parse_page(buffer: &[u8], page_num: usize) -> anyhow::Result<page::Page> {
    let ptr_offset = if page_num == 1 { HEADER_SIZE as u16 } else { 0 };

    match buffer[0] {
        PAGE_LEAF_TABLE_ID => parse_table_leaf_page(buffer, ptr_offset),
        _ => Err(anyhow::anyhow!("unknown page type: {}", buffer[0])),
    }
}

fn parse_table_leaf_page(buffer: &[u8], ptr_offset: u16) -> anyhow::Result<page::Page> {
    let header = parse_page_header(buffer)?;

    let content_buffer = &buffer[PAGE_LEAF_HEADER_SIZE..];
    let cell_pointers = parse_cell_pointers(content_buffer, header.cell_count as usize, ptr_offset);

    let cells = cell_pointers
        .iter()
        .map(|&ptr| parse_table_leaf_cell(&buffer[ptr as usize..]))
        .collect::<anyhow::Result<Vec<page::TableLeafCell>>>()?;

    Ok(page::Page::TableLeaf(page::TableLeafPage {
        header,
        cell_pointers,
        cells,
    }))
}


fn parse_page_header(buffer: &[u8]) -> anyhow::Result<page::PageHeader> {
    let page_type = match buffer[0] {
        0x0d => page::PageType::TableLeaf,
        _ => anyhow::bail!("unknown page type: {}", buffer[0]),
    };

    let first_freeblock = read_be_word_at(buffer, PAGE_FIRST_FREEBLOCK_OFFSET);
    let cell_count = read_be_word_at(buffer, PAGE_CELL_COUNT_OFFSET);
    let cell_content_offset = match read_be_word_at(buffer, PAGE_CELL_CONTENT_OFFSET) {
        0 => 65536,
        n => n as u32,
    };
    let fragmented_bytes_count = buffer[PAGE_FRAGMENTED_BYTES_COUNT_OFFSET];

    Ok(page::PageHeader {
        page_type,
        first_freeblock,
        cell_count,
        cell_content_offset,
        fragmented_bytes_count,
    })
}


fn parse_cell_pointers(buffer: &[u8], n: usize, ptr_offset: u16) -> Vec<u16> {
    let mut pointers = Vec::with_capacity(n);
    for i in 0..n {
        pointers.push(read_be_word_at(buffer, 2 * i) - ptr_offset);
    }
    pointers
}

fn parse_table_leaf_cell(mut buffer: &[u8]) -> anyhow::Result<page::TableLeafCell> {
    let (n, size) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

    let (n, row_id) = read_varint_at(buffer, 0);
    buffer = &buffer[n as usize..];

    let payload = buffer[..size as usize].to_vec();

    Ok(page::TableLeafCell {
        size,
        row_id,
        payload,
    })
}

pub fn read_varint_at(buffer: &[u8], mut offset: usize) -> (u8, i64) {
    let mut size = 0;
    let mut result = 0;

    while size < 9 {
        let current_byte = buffer[offset] as i64;
        if size == 8 {
            result = (result << 8) | current_byte;
        } else {
            result = (result << 7) | (current_byte & 0b0111_1111);
        }

        offset += 1;
        size += 1;

        if current_byte & 0b1000_0000 == 0 {
            break;
        }
    }

    (size, result)
}

To read a varint, we copy the 7 least significant bits of each byte to the result, as long as the most significant bit is set. As the maximum length of a varint is 9 bytes, keep track of the number of bytes visited and stop after a maximum of 9 bytes. Note that to complete a 64 bits value, we need the first 7 bits of the first 8 bytes and all the bits of the last byte. That's why we test the current size of the varint at each iteration and add a special case for the last byte (when size == 8).

We can finally implement the pager itself. For now, it only loads and caches pages without any eviction policy:

// pager.rs
#[derive(Debug, Clone)]
pub struct Pager<I: Read + Seek = std::fs::File> {
    input: I,
    page_size: usize,
    pages: HashMap<usize, page::Page>,
}

impl<I: Read + Seek> Pager<I> {
    pub fn new(input: I, page_size: usize) -> Self {
        Self {
            input,
            page_size,
            pages: HashMap::new(),
        }
    }

    pub fn read_page(&mut self, n: usize) -> anyhow::Result<&page::Page> {
        if self.pages.contains_key(&n) {
            return Ok(self.pages.get(&n).unwrap());
        }

        let page = self.load_page(n)?;
        self.pages.insert(n, page);
        Ok(self.pages.get(&n).unwrap())
    }

    fn load_page(&mut self, n: usize) -> anyhow::Result<page::Page> {
        let offset = n.saturating_sub(1) * self.page_size;

        self.input
            .seek(SeekFrom::Start(offset as u64))
            .context("seek to page start")?;

        let mut buffer = vec![0; self.page_size];
        self.input.read_exact(&mut buffer).context("read page")?;

        parse_page(&buffer, n)
    }
}

Records

We now have a way to read pages, and to access the pages cells. But how to decode the values of the cells? Each cell contains the value of a row in the table, encoded using the SQLite record format. The record format is quite simple: a record consists of a header, followed by a sequence of field values. The header starts with a varint representing the byte size of the headerm followed by a sequence of varints -one per column- determining the type of each column according to the following table:

• 0: NULL
• 1: 8-bits signed integer
• 2: 16-bits signed integer
• 3: 24-bits signed integer
• 4: 32-bits signed integer
• 5: 48-bits signed integer
• 6: 64-bits signed integer
• 7: 64-bits IEEE floating point number
• 8: value is the integer 0
• 9: value is the integer 1
• 10 & 11: reserved for internal use
• n with n even and n > 12: BLOB of size (n - 12) / 2
• n with n odd and n > 13: text of size (n - 13) / 2

We now have all the informations we need to parse and represent record's headers:

// src/cursor.rs
#[derive(Debug, Copy, Clone)]
pub enum RecordFieldType {
    Null,
    I8,
    I16,
    I24,
    I32,
    I48,
    I64,
    Float,
    Zero,
    One,
    String,
    Blob,
}

#[derive(Debug, Clone)]
pub struct RecordField {
    pub offset: usize,
    pub field_type: RecordFieldType,
}

#[derive(Debug, Clone)]
pub struct RecordHeader {
    pub fields: Vec<RecordField>,
}

fn parse_record_header(mut buffer: &[u8]) -> anyhow::Result<RecordHeader> {
    let (varint_size, header_length) = crate::pager::read_varint_at(buffer, 0);
    buffer = &buffer[varint_size as usize..header_length as usize];

    let mut fields = Vec::new();
    let mut current_offset = header_length as usize;

    while !buffer.is_empty() {
        let (discriminant_size, discriminant) = crate::pager::read_varint_at(buffer, 0);
        buffer = &buffer[discriminant_size as usize..];

        let (field_type, field_size) = match discriminant {
            0 => (RecordFieldType::Null, 0),
            1 => (RecordFieldType::I8, 1),
            2 => (RecordFieldType::I16, 2),
            3 => (RecordFieldType::I24, 3),
            4 => (RecordFieldType::I32, 4),
            5 => (RecordFieldType::I48, 6),
            6 => (RecordFieldType::I64, 8),
            7 => (RecordFieldType::Float, 8),
            8 => (RecordFieldType::Zero, 0),
            9 => (RecordFieldType::One, 0),
            n if n >= 12 && n % 2 == 0 => {
                let size = ((n - 12) / 2) as usize;
                (RecordFieldType::Blob(size), size)
            }
            n if n >= 13 && n % 2 == 1 => {
                let size = ((n - 13) / 2) as usize;
                (RecordFieldType::String(size), size)
            }
            n => anyhow::bail!("unsupported field type: {}", n),
        };

        fields.push(RecordField {
            offset: current_offset,
            field_type,
        });

        current_offset += field_size;
    }

    Ok(RecordHeader { fields })
}

To make it easier to work with records, we'll define a Value type, representing field values and a Cursor struct that uniquely identifies a record within a database file. The Cursor will expose a field method, returning the value of the record's n-th field:

// src/value.rs
use std::borrow::Cow;

#[derive(Debug, Clone)]
pub enum Value<'p> {
    Null,
    String(Cow<'p, str>),
    Blob(Cow<'p, [u8]>),
    Int(i64),
    Float(f64),
}

impl<'p> Value<'p> {
    pub fn as_str(&self) -> Option<&str> {
        if let Value::String(s) = self {
            Some(s.as_ref())
        } else {
            None
        }
    }
}

// src/cursor.rs
#[derive(Debug)]
pub struct Cursor<'p> {
    header: RecordHeader,
    pager: &'p mut Pager,
    page_index: usize,
    page_cell: usize,
}

impl<'p> Cursor<'p> {
    pub fn field(&mut self, n: usize) -> Option<Value> {
        let record_field = self.header.fields.get(n)?;

        let payload = match self.pager.read_page(self.page_index) {
            Ok(Page::TableLeaf(leaf)) => &leaf.cells[self.page_cell].payload,
            _ => return None,
        };

        match record_field.field_type {
            RecordFieldType::Null => Some(Value::Null),
            RecordFieldType::I8 => Some(Value::Int(read_i8_at(payload, record_field.offset))),
            RecordFieldType::I16 => Some(Value::Int(read_i16_at(payload, record_field.offset))),
            RecordFieldType::I24 => Some(Value::Int(read_i24_at(payload, record_field.offset))),
            RecordFieldType::I32 => Some(Value::Int(read_i32_at(payload, record_field.offset))),
            RecordFieldType::I48 => Some(Value::Int(read_i48_at(payload, record_field.offset))),
            RecordFieldType::I64 => Some(Value::Int(read_i64_at(payload, record_field.offset))),
            RecordFieldType::Float => Some(Value::Float(read_f64_at(payload, record_field.offset))),
            RecordFieldType::String(length) => {
                let value = std::str::from_utf8(
                    &payload[record_field.offset..record_field.offset + length],
                ).expect("invalid utf8");
                Some(Value::String(Cow::Borrowed(value)))
            }
            RecordFieldType::Blob(length) => {
                let value = &payload[record_field.offset..record_field.offset + length];
                Some(Value::Blob(Cow::Borrowed(value)))
            }
            _ => panic!("unimplemented"),
        }
    }
}

fn read_i8_at(input: &[u8], offset: usize) -> i64 {
    input[offset] as i64
}

fn read_i16_at(input: &[u8], offset: usize) -> i64 {
    i16::from_be_bytes(input[offset..offset + 2].try_into().unwrap()) as i64
}

fn read_i24_at(input: &[u8], offset: usize) -> i64 {
    (i32::from_be_bytes(input[offset..offset + 3].try_into().unwrap()) & 0x00FFFFFF) as i64
}

fn read_i32_at(input: &[u8], offset: usize) -> i64 {
    i32::from_be_bytes(input[offset..offset + 4].try_into().unwrap()) as i64
}

fn read_i48_at(input: &[u8], offset: usize) -> i64 {
    i64::from_be_bytes(input[offset..offset + 6].try_into().unwrap()) & 0x0000FFFFFFFFFFFF
}

fn read_i64_at(input: &[u8], offset: usize) -> i64 {
    i64::from_be_bytes(input[offset..offset + 8].try_into().unwrap())
}

fn read_f64_at(input: &[u8], offset: usize) -> f64 {
    f64::from_be_bytes(input[offset..offset + 8].try_into().unwrap())
}

To simplify iteration over a page's records, we'll also implement a Scanner struct that wraps a page and allows us to get a Cursor for each record:

// src/cursor.rs
#[derive(Debug)]
pub struct Scanner<'p> {
    pager: &'p mut Pager,
    page: usize,
    cell: usize,
}

impl<'p> Scanner<'p> {
    pub fn new(pager: &'p mut Pager, page: usize) -> Scanner<'p> {
        Scanner {
            pager,
            page,
            cell: 0,
        }
    }
    pub fn next_record(&mut self) -> Option<anyhow::Result<Cursor>> {
        let page = match self.pager.read_page(self.page) {
            Ok(page) => page,
            Err(e) => return Some(Err(e)),
        };

        match page {
            Page::TableLeaf(leaf) => {
                let cell = leaf.cells.get(self.cell)?;

                let header = match parse_record_header(&cell.payload) {
                    Ok(header) => header,
                    Err(e) => return Some(Err(e)),
                };

                let record = Cursor {
                    header,
                    pager: self.pager,
                    page_index: self.page,
                    page_cell: self.cell,
                };

                self.cell += 1;

                Some(Ok(record))
            }
        }
    }
}

Table descriptions

With most of the leg work out of the way, we can get back to our original goal: listing tables. SQLite stores the schema of a database in a special table called sqlite_master. The schema for the sqlite_master table is as follows:

CREATE TABLE sqlite_schema(
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);

Theses columns are used as follows:

• type: the type of the schema object. For tables, this will always be table.
• name: the name of the schema object.
• tbl_name: the name of the table the schema object is associated with. In the case of tables, this will be the same as name.
• rootpage: root page of the table, we'll use it later to read the table's content.
• sql: the SQL statement used to create the table.

Since our simple database only handles basic schemas for now, we can assume that the entire schema fits in the first page of our database file. In order to list the tables in the database, we'll need to:

• initialize the pager with the database file
• create a Scanner for the first page
• iterate over the records, and print the value of the name field (at index 1) for each record.

First, we'll define a Db struct to hold our global state:

// src/db.rs
use std::{io::Read, path::Path};

use anyhow::Context;

use crate::{cursor::Scanner, page::DbHeader, pager, pager::Pager};

pub struct Db {
    pub header: DbHeader,
    pager: Pager,
}

impl Db {
    pub fn from_file(filename: impl AsRef<Path>) -> anyhow::Result<Db> {
        let mut file = std::fs::File::open(filename.as_ref()).context("open db file")?;

        let mut header_buffer = [0; pager::HEADER_SIZE];
        file.read_exact(&mut header_buffer)
            .context("read db header")?;

        let header = pager::parse_header(&header_buffer).context("parse db header")?;

        let pager = Pager::new(file, header.page_size as usize);

        Ok(Db { header, pager })
    }

    pub fn scanner(&mut self, page: usize) -> Scanner {
        Scanner::new(&mut self.pager, page)
    }
}

The implementation of a basic REPL supporting the tables and tables commands is straightforward:

use std::io::{stdin, BufRead, Write};

use anyhow::Context;

mod cursor;
mod db;
mod page;
mod pager;
mod value;

fn main() -> anyhow::Result<()> {
    let database = db::Db::from_file(std::env::args().nth(1).context("missing db file")?)?;
    cli(database)
}

fn cli(mut db: db::Db) -> anyhow::Result<()> {
    print_flushed("rqlite> ")?;

    let mut line_buffer = String::new();

    while stdin().lock().read_line(&mut line_buffer).is_ok() {
        match line_buffer.trim() {
            ".exit" => break,
            ".tables" => display_tables(&mut db)?,
            _ => {
                println!("Unrecognized command '{}'", line_buffer.trim());
            }
        }

        print_flushed("\nrqlite> ")?;

        line_buffer.clear();
    }

    Ok(())
}

fn display_tables(db: &mut db::Db) -> anyhow::Result<()> {
    let mut scanner = db.scanner(1);

    while let Some(Ok(mut record)) = scanner.next_record() {
        let type_value = record
            .field(0)
            .context("missing type field")
            .context("invalid type field")?;

        if type_value.as_str() == Some("table") {
            let name_value = record
                .field(1)
                .context("missing name field")
                .context("invalid name field")?;

            print!("{} ", name_value.as_str().unwrap());
        }
    }

    Ok(())
}

fn print_flushed(s: &str) -> anyhow::Result<()> {
    print!("{}", s);
    std::io::stdout().flush().context("flush stdout")
}

Conclusion

The first part of our SQLite-compatible database is now complete. We can read the database header, parse table btree-leaf pages and decode records, but we still have a long way to go before we can support rich queries. In the next part, we'll learn how to parse the SQL language and make our first stides towards implementing the SELECT statement!

Welcome to this series of blog posts where we will be exploring how to build a web server from scratch using the Rust programming language. We will be taking a hands-on approach, maximizing our learning experience by using as few dependencies as possible and implementing as much logic as we can. This will enable us to understand the inner workings of a web server and the underlying protocols that it uses.

By the end of this tutorial, you will have a solid understanding of how to build a web server from scratch using Rust and the tokio library. So, let's dive in and get started on our journey!

In this first part, we'll be building a barebones web server that can only anwser GET requests with a static Not Found response. This will give us a good starting point to build upon in the following tutorial.

Setting up our project

First, we need to create a new Rust project. We'll use the following crates:

• tokio: async runtime

• anyhow: easy error handling

• maplit: macro for creating HashMaps

• tracing: structured logging

• tracing-subscriber: instrumentation

cargo new webserver
cargo add tokio --features full
cargo add anyhow maplit tracing tracing-subscriber

Anatomy of a simple GET request

In order to actually see what a GET request looks like, we'll set up a simple server listening on port 8080 that will print the incoming requests to the console. This can be done with netcat:

nc -l 8080

Now, if we open a new terminal and use curl send a simple GET request to our server, we should see the following output:

Let's break down the request parts:

• the method: indicates the action to be performed on the resource. In this case, we are performing a GET request, which means we want to retrieve the resource

• the path: uniquely identifies the resource. In this case, we are requesting the root path /

• the protocol: the protocol version. At this stage, we will always asume HTTP/1.1

• the headers: a set of key-value pairs that provide additional information about the request. Our request contains the Host header, which indicates the host name of the server, the User-Agent header, which describes the client software that is making the request and the Accept header, which indicates the media types that are acceptable for the response. We'll go into more details about headers in a later tutorial

We'll use the following struct to represent requests in our code:

// req.rs

#[derive(Debug, Clone, Eq, PartialEq)]
pub struct Request {
    pub method: Method,
    pub path: String,
    pub headers: HashMap<String, String>,
}

#[derive(Debug, Copy, Clone, Eq, PartialEq, Hash)]
pub enum Method {
    Get,
}

Parsing the request is just a matter of splitting the request string into lines. The first line contains the method, path and protocol separated by spaces. The following lines contain the headers, followed by an empty line.

// req.rs
use std::{collections::HashMap, hash::Hash};

use tokio::io::{AsyncBufRead, AsyncBufReadExt};

// [...]

impl TryFrom<&str> for Method {
    type Error = anyhow::Error;

    fn try_from(value: &str) -> Result<Self, Self::Error> {
        match value {
            "GET" => Ok(Method::Get),
            m => Err(anyhow::anyhow!("unsupported method: {m}")),
        }
    }
}

pub async fn parse_request(mut stream: impl AsyncBufRead + Unpin) -> anyhow::Result<Request> {
    let mut line_buffer = String::new();
    stream.read_line(&mut line_buffer).await?;

    let mut parts = line_buffer.split_whitespace();

    let method: Method = parts
        .next()
        .ok_or(anyhow::anyhow!("missing method"))
        .and_then(TryInto::try_into)?;

    let path: String = parts
        .next()
        .ok_or(anyhow::anyhow!("missing path"))
        .map(Into::into)?;

    let mut headers = HashMap::new();

    loop {
        line_buffer.clear();
        stream.read_line(&mut line_buffer).await?;

        if line_buffer.is_empty() || line_buffer == "\n" || line_buffer == "\r\n" {
            break;
        }

        let mut comps = line_buffer.split(":");
        let key = comps.next().ok_or(anyhow::anyhow!("missing header name"))?;
        let value = comps
            .next()
            .ok_or(anyhow::anyhow!("missing header value"))?
            .trim();

        headers.insert(key.to_string(), value.to_string());
    }

    Ok(Request {
        method,
        path,
        headers,
    })
}

Accepting connections

Now that we know how to parse a request, we can start accepting connections. Each time a new connection is established, we'll spawn a new task to handle it in order to keep the main thread free to accept new connections.

// main.rs
use tokio::{io::BufStream, net::TcpListener};
use tracing::info;

mod req;

static DEFAULT_PORT: &str = "8080";

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the default tracing subscriber.
    tracing_subscriber::fmt::init();

    let port: u16 = std::env::args()
        .nth(1)
        .unwrap_or_else(|| DEFAULT_PORT.to_string())
        .parse()?;

    let listener = TcpListener::bind(format!("0.0.0.0:{port}")).await.unwrap();

    info!("listening on: {}", listener.local_addr()?);

    loop {
        let (stream, addr) = listener.accept().await?;
        let mut stream = BufStream::new(stream);

        // do not block the main thread, spawn a new task
        tokio::spawn(async move {
            info!(?addr, "new connection");

            match req::parse_request(&mut stream).await {
                Ok(req) => info!(?req, "incoming request"),
                Err(e) => {
                    info!(?e, "failed to parse request");
                }
            }
        });
    }
}

We can now run our server on port 8081with the following command: cargo run -- 8081. Sending a GET request to localhost:8081 should print the following output:

INFO http_server: listening on: 0.0.0.0:8081
INFO http_server: new connection addr=127.0.0.1:49351
INFO http_server: incoming request req=Request { method: Get, path: "/", headers: {"Host": "localhost", "User-Agent": "curl/7.87.0", "Accept": "*/*"} }

Sending a response

At this stage, we'll answer every request with a static Not found page. Our response will have the following format:

Let's explore the different parts of the response:

• the status line: contains the protocol version, the status code and a human-readable status message

• the response headers: encoded in the same way as for the request. Our response contains the Content-Length header, which specified the length of the response body, and the Content-Type header, which indicates that the response body is encoded in HTML. The headers are followed by an empty line.

• the response body: contains the actual data that will be displayed in the browser. We used an empty HTML document for brevity

We'll use the following struct to represent responses in our code:

// resp.rs
use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};

#[derive(Debug, Clone)]
pub struct Response<S: AsyncRead + Unpin> {
    pub status: Status,
    pub headers: HashMap<String, String>,
    pub data: S,
}

#[derive(Debug, Copy, Clone, Eq, PartialEq, Hash)]
pub enum Status {
    NotFound,
}

impl Display for Status {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Status::NotFound => write!(f, "404 Not Found"),
        }
    }
}

The data field is generic over the type of the response body to account for future use cases where we might want to send a stream of data.

Creating a response from an HTML string is straight forward:

// resp.rs
use std::io::Cursor;
use maplit::hashmap;

// [..]

impl Response<Cursor<Vec<u8>>> {
    pub fn from_html(status: Status, data: impl ToString) -> Self {
        let bytes = data.to_string().into_bytes();

        let headers = hashmap! {
            "Content-Type".to_string() => "text/html".to_string(),
            "Content-Length".to_string() => bytes.len().to_string(),
        };

        Self {
            status,
            headers,
            data: Cursor::new(bytes),
        }
    }
}

Sending a response is a bit more involved. We'll use the AsyncWrite trait to write the response to a generic output stream.

// resp.rs
use std::{
    collections::HashMap,
    fmt::{Display, Formatter},
    io::Cursor,
};

use maplit::hashmap;
use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};

// [...]

impl<S: AsyncRead + Unpin> Response<S> {
    pub fn status_and_headers(&self) -> String {
        let headers = self
            .headers
            .iter()
            .map(|(k, v)| format!("{}: {}", k, v))
            .collect::<Vec<_>>()
            .join("\r\n");

        format!("HTTP/1.1 {}\r\n{headers}\r\n\r\n", self.status)
    }

    pub async fn write<O: AsyncWrite + Unpin>(mut self, stream: &mut O) -> anyhow::Result<()> {
        stream
            .write_all(self.status_and_headers().as_bytes())
            .await?;

        tokio::io::copy(&mut self.data, stream).await?;

        Ok(())
    }
}

impl Display for Status {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self {
            Status::NotFound => write!(f, "404 Not Found"),
        }
    }
}

Puting it all together

We'll use the following document as our 404 page:

<!-- static/404.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <title>Page Not Found</title>
    <style>
        body {
            background-color: #f8f8f8;
            font-family: Arial, sans-serif;
            font-size: 16px;
            color: #333;
        }
        .container {
            max-width: 600px;
            margin: 0 auto;
            padding: 40px 20px;
            text-align: center;
            border: 1px solid #ddd;
            border-radius: 5px;
            background-color: #fff;
            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
        }
        h1 {
            font-size: 48px;
            margin-bottom: 20px;
            color: #333;
        }
        p {
            font-size: 24px;
            margin-bottom: 40px;
        }
    </style>
</head>
<body>
<div class="container">
    <h1>404</h1>
    <p>The page you are looking for could not be found.</p>
</div>
</body>
</html>

We can now use our Response struct to send a Not found page to the client when we receive a request:

// main.rs
// [...]
let resp = resp::Response::from_html(
    resp::Status::NotFound,
    include_str!("../static/404.html"),
);

resp.write(&mut stream).await.unwrap();
// [...]

Navigating to localhost:8081 should now display our Not found page.

That's a good start, but we're still far from a fully functional web server. In the next part, we'll add support for serving static files. You can find the code for this part here.

Looking for a Rust dev? Let's get in touch!

Sylver's main interface is a REPL console, in which we can load the source code of our project to query it using a SQL-like query language called SYLQ. Once we'll have authored SYLQ queries expressing our linting rules, we'll be able to save them into a ruleset that can be run like a traditional linter.

Installation

If sylver --version doesn't output a version number >= 0.2.2, go to https://sylver.dev to download a fresh copy of the software.

Project setup

We'll use the following Python file to test our linting rules:

#main.py
from users.models import *
from auth.models import check_password

foo = 100
O = 100.0

my_dict = {'hello': 'world'}

if my_dict.has_key('hello'):
    print('It works!')

if 'hello' in my_dict:
    print('It works!')

Starting the REPL

Starting the REPL is as simple as invoking the following command at the root of your project:

sylver query --files="src/**/*.py" --language=python

The REPL can be exited by pressing Ctrl+C or typing :quit at the prompt.

We can now execute SYLQ queries by typing the code of the query, followed by a ;. For instance: to retrieve all the if statements (denoted by the node type IfStatement):

match IfStatement;

The results of the query will be formatted as follow:

$0 [IfStatement main.py:1:9-23:10]
$1 [IfStatement main.py:1:12-23:13]

The code of a given if statement can be displayed by typing :print followed by the node alias (for instance: :print $1). The parse tree can be displayed using the :print_ast command (for instance: :print_ast $1).

Rule1: wildcard imports (inspired by F403)

This rule will flag all the imports of the form from x import *.

The first step is to get familiar with the tree structure of Python's import statements, so let's print a ImportFromStatement node along with its AST:

λ> match ImportFromStatement;

$2 [ImportFromStatement main.py:1:1-27:1]
$3 [ImportFromStatement main.py:1:2-39:2]

λ> :print $2

from users.models import *

λ> :print_ast $2

ImportFromStatement {
. ● module_name: DottedName {
. . Identifier { users }
. . Identifier { models }
. }
. WildcardImport { * }
}

It appears that the faulty part of the import statement (the wildcard: *) is represented by a WildcardImport node. So this first rule can easily be expressed in SYLQ:

match WildcardImport;

Rule2: Ambiguous variable name (inspired by E741)

This style-oriented rule will detect variables named 'l', 'I' or 'O', as these names can be confusing.

Same as before, let's analyze the tree structure of an assignment:

λ> match Assignment;

$4 [Assignment main.py:1:4-10:4]
$5 [Assignment main.py:1:5-10:5]
$6 [Assignment main.py:1:7-29:7]

λ> :print_ast $5

Assignment {
. ● left: Identifier { O }
. ● right: Float { 100.0 }
}

The variable's Identifier can be accessed through the left field of the Assignment node. We can match the Identifier's text against a regex by using the builtin matches method:

match a@Assignment when a.left.text.matches(`^(I|O|l)$`);

Here the Assignment node is bound to a using the binding operator: @.

Rule3: has_key() is deprecated (inspired by W601)

This rule signals uses of the deprecated dictionnary has_key method.

Here is the tree representation of a call to has_key:

Call {
. ● function: Attribute {
. . ● object: Identifier { my_dict }
. . ● attribute: Identifier { has_key }
. }
. ● arguments: ArgumentList {
. . String { 'hello' }
. }
}

This query can be expressed using nested patterns, as follow:

match Call(function: Attribute(attribute: 'has_key'));

Creating the ruleset

The following ruleset uses our linting rules:

id: customRules

language: python

rules:
    - id: F403
      category: style
      message: "wildcard import"
      note: "wildcard imports are discouraged because the programmer often won’t know where an imported object is defined"

      query: >
        match WildcardImport



    - id: E741
      category: style
      message: "ambiguous variable name"
      note: "variables named I, O and l can be very hard to read"

      query: >
        match a@Assignment when a.left.text.matches(`^(I|O|l)$`)


    - id: W601
      category: style
      message: ".has_key() is deprecated"
      note: "'.has_key()' was deprecated in Python 2. It is recommended to use the 'in' operator instead"

      query: >
        match Call(function: Attribute(attribute: 'has_key'))

Assuming that it is stored in a file called ruleset.yaml at the root of our project, we can run it with the following command:

sylver ruleset run --files "**/*.py" --rulesets ruletset.yaml

Getting updates

For more informations about new features and/or cool SYLQ one-liners, connect with Sylver on Twitter or Discord!

Sylver's main interface is a REPL console, in which we can load the source code of our project to query it using a SQL-like query language called SYLQ. Once we'll have authored SYLQ queries expressing our linting rules, we'll be able to save them into a ruleset that can be run like a traditional linter.

Installation

If sylver --version doesn't output a version number >= 0.1.9, go to https://sylver.dev to download a fresh copy of the software.

Starting the REPL

Starting the REPL is as simple as invoking the following command at the root of your project:

sylver query --files="src/**/*.js" --spec=https://github.com/sylver-dev/javascript.git#javascript.yaml

The REPL can be exited by pressing Ctrl+C or typing :quit at the prompt.

We can now execute SYLQ queries by typing the code of the query, followed by a ;. For instance: to retrieve all the method definitions (denoted by the node type MethodDefinition):

match MethodDefinition;

The results of the query will be formatted as follow:

[...]
$0 [MethodDefinition src/store/createArticles.js:36:5-38:5]
$1 [MethodDefinition src/store/createArticles.js:39:5-41:5]
$2 [MethodDefinition src/store/createArticles.js:42:5-59:5]
$3 [MethodDefinition src/store/createArticles.js:60:5-77:5]
$4 [MethodDefinition src/store/createArticles.js:78:5-83:5]
$5 [MethodDefinition src/store/createArticles.js:84:5-89:5]
[...]

The code of a given method definition can be displayed by typing :print followed by the node alias (for instance: :print $3). The parse tree can be displayed using the :print_ast command (for instance: :print_ast $3).

Rule1: use of the == operator

For our first rule, we'd like to detect uses of the unsafe == operator for checking equality. The first step is to get familiar with the tree structure of Javascript's binary expressions, so let's print a BinaryExpression node along with its AST:

λ> match BinaryExpression;

[...]
$43 [BinaryExpression src/pages/Article/Comments.js:7:31-7:77]
[...]

λ> :print $43

currentUser.username == comment.author.username

λ> :print_ast $43

BinaryExpression {
. ● left: MemberExpression {
. . ● object: Identifier { currentUser }
. . ● property: Identifier { username }
. }
. ● operator: EqEq { == }
. ● right: MemberExpression {
. . ● object: MemberExpression {
. . . ● object: Identifier { comment }
. . . ● property: Identifier { author }
. . }
. . ● property: Identifier { username }
. }
}

It appears that the nodes violating our rule are the BinaryExpression nodes for which the operator field contains an EqEq node. This can be easily expressed in SYLQ:

match BinaryExpression(operator: EqEq);

Rule2: functions with too many parameters

For our second linting rule, we'd like to identify functions that have more than 6 parameters.

Here is the relevant part of the parse tree of a Function node:

Function {
. ● async: AsyncModifier { async }
. ● name: Identifier { send }
. ● parameters: FormalParameters {
. . ● params: List {
. . . FormalParameter {
. . . . ● value: Identifier { method }
. . . }
. . . FormalParameter {
. . . . ● value: Identifier { url }
. . . }
. . . FormalParameter {
. . . . ● value: Identifier { data }
. . . }
. . . FormalParameter {
. . . . ● value: Identifier { resKey }
. . . }
. . }
. }
. ● body: StatementBlock {
[...]

Function parameters are represented by FormalParameters nodes with a params field containing the actual function parameters. In our query, the condition regarding the length of the params list can be specified in a when clause, as follows:

match f@FormalParameters when f.params.length > 6;

Rule3: JSX 'img' elements without an 'alt' attribute

For our last rule, we'd like to identify <img> elements that miss the alt attribute. img elements are self-closing, so we'll start by looking at the parse tree of a JsxSelfClosingElement node:

λ> match JsxSelfClosingElement;
[...]
$73 [JsxSelfClosingElement src/pages/Article/Comments.js:21:11-21:55]
[...]

λ> :print $73

<img src={image} class="comment-author-img"/>

λ> :print_ast $73

JsxSelfClosingElement {
. ● name: Identifier { img }
. ● attribute: List {
. . JsxAttribute {
. . . ● name: Identifier { src }
. . . ● value: JsxExpression {
. . . . Identifier { image }
. . . }
. . }
. . JsxAttribute {
. . . ● name: Identifier { class }
. . . ● value: String { "comment-author-img" }
. . }
. }
}

In order to find the img elements that have no JsxAttribute with named alt in their attribute list, we can use a list quantifying expression, as illustrated in the following query:

match j@JsxSelfClosingElement(name: "img") 
      when no j.attribute match JsxAttribute(name: "alt");

Creating the ruleset

The following ruleset uses our linting rules:

id: customLinter

language: "https://github.com/sylver-dev/javascript.git#javascript.yaml"

rules:
    - id: unsafeEq
      message: equality comparison with `==` operator
      category: style

      query: "match BinaryExpression(operator: EqEq)"


    - id: tooManyParams
      message: function has too many parameters
      category: style
      note: According to our style guide, functions should have less than 6 parameters.

      query: match f@FormalParameters when f.params.length > 6


    - id: missingAlt
      message: <img> tags should have an "alt" attribute
      category: style

      query:
        "match j@JsxSelfClosingElement(name: 'img')
              when no j.attribute match JsxAttribute(name: 'alt')"

Assuming that it is stored in a file called custom_linter.yaml at the root of our project, we can run it with the following command:

sylver ruleset run --files="src/**/*.js" --rulesets=custom_linter.yaml

Getting updates

For more informations about new features and/or cool SYLQ one-liners, connect with Sylver on Twitter or Discord!

Sylver's main interface is a REPL console, in which we can load the source code of our project to query it using a SQL-like query language called SYLQ. Once we'll have authored SYLQ queries expressing our linting rules, we'll be able to save them into a ruleset that can be run like a traditional linter.

Installation

If sylver --version doesn't output a version number >= 0.1.8, go to https://sylver.dev to download a fresh copy of the software.

Starting the REPL

Starting the REPL is as simple as invoking the following command at the root of your project:

sylver query --files="**/*.go" --spec=https://github.com/sylver-dev/golang.git#golang.yaml

The REPL can be exited by pressing Ctrl+C or typing :quit at the prompt.

We can now execute SYLQ queries by typing the code of the query, followed by a ;. For instance: to retrieve all the struct declarations:

match StructType;

The results of the query will be formatted as follow:

[...]
$359 [StructType association.go:323:17-327:1]
$360 [StructType schema/index.go:10:12-18:1]
$361 [StructType schema/index.go:20:18-27:1]
$362 [StructType tests/group_by_test.go:70:12-73:2]
$363 [StructType schema/check.go:11:12-15:1]

The code of a given struct declaration can be displayed by typing :print followed by the node alias (for instance: :print $362). The parse tree can be displayed using the :print_ast command (for instance: :print_ast $362).

Rule1: detect struct declarations with too many fields

For our first rule, we'd like to flag struct declarations that have more than 10 fields. The first step is to get familiar with the tree structure of struct declarations, so let's print a StructType along with its ast:

λ> :print $362

struct {
        Name  string
        Total int64
    }

λ> :print_ast $362

StructType {
. ● fields: List<FieldSpec> {
. . FieldSpec {
. . . ● names: List<Identifier> {
. . . . Identifier { Name }
. . . }
. . . ● type: TypeIdent {
. . . . ● name: Identifier { string }
. . . }
. . }
. . FieldSpec {
. . . ● names: List<Identifier> {
. . . . Identifier { Total }
. . . }
. . . ● type: TypeIdent {
. . . . ● name: Identifier { int64 }
. . . }
. . }
. }
}

The fields of the struct are stored in a field aptly named fields that holds a list of FieldSpec nodes. This means that the nodes violating our rule are all the StructType nodes for which the fields list has a length higher than 10. This can be easily expressed in SYLQ:

 match StructType s when s.fields.length > 10;

Rule2: suggest the usage of assignment operators

For our second linting rule, we'd like to identify assignments that could be simplified by using an assignment operator (like +=) such as:

x = x + 1

Let's explore the parse tree of a simple assignment:

λ> :print $5750

err = nil

λ> :print_ast $5750

AssignStmt {
. ● lhs: List<Expr> {
. . Identifier { err }
. }
. ● rhs: List<Expr> {
. . NilLit { nil }
. }
}

So we want to retrieve the AssignStmt nodes for which the rhs field contains a Binop that has lhs as its left operand. Also, the left-hand side of the assignment must contain a single expression. This can be written as:

match AssignStmt a when
      a.lhs.length == 1
   && a.rhs[0] is { BinOp b when b.left.text == a.lhs[0].text };

Rule3: incorrect usage of the make builtin function

For our last linting rule, we want to identify incorrect usage of the make function, where the length is higher than the capacity, as this probably indicates a programming error.

Here is the parse tree of a call to make:

λ> :print $16991

make([]string, 0, len(value))

λ> :print_ast $16991

CallExpr {
. ● fun: Identifier { make }
. ● args: List<GoNode> {
. . SliceType {
. . . ● elemsType: TypeIdent {
. . . . ● name: Identifier { string }
. . . }
. . }
. . IntLit { 0 }
. . CallExpr {
. . . ● fun: Identifier { len }
. . . ● args: List<GoNode> {
. . . . Identifier { value }
. . . }
. . }
. }
}

Here are the conditions that violating nodes will meet:

• The test of fun is make
• The args list contains 3 elements
• The last two arguments are int literals
• The third argument (capacity) is smaller than the second (length)

Let's encode this in SYLQ:

match CallExpr c when
      c.fun.text == 'make'
   && c.args.length == 3
   && c.args[1] is IntLit
   && c.args[2] is IntLit
   && c.args[2].text.to_int() < c.args[1].text.to_int();

Creating the ruleset

The following ruleset uses our linting rules:

id: customLinter

language: "https://github.com/sylver-dev/golang.git#golang.yaml"

rules:
    - id: largeStruct
      message: struct has many fields
      category: style

      query:  match StructType s when s.fields.length > 10


    - id: assignOp
      message: assignment should use an assignment operator
      category: style
      note: According to our style guide, assignment operators should be preferred.

      query: >
        match AssignStmt a when
             a.lhs.length == 1
          && a.rhs[0] is { BinOp b when b.left.text == a.lhs[0].text }

    - id: makeCapacityErr 
      message: capacity should be higher than length     
      category: bug

      query: >
        match CallExpr c when
              c.fun.text == 'make'
          && c.args.length == 3
          && c.args[1] is IntLit
          && c.args[2] is IntLit
          && c.args[2].text.to_int() < c.args[1].text.to_int()

Assuming that it is stored in a file called custom_linter.yaml at the root of our project, we can run it with the following command:

sylver ruleset run --files="**/*.go" --rulesets=custom_linter.yaml

While it can be insightful to explore a codebase interactively through source-code queries, it's not the most practical way to perform source code verification. In this tutorial, we'll learn how to package the queries we built in the last part into a ruleset to use them in a linter-like fashion.

If you have already installed sylver and sylver --version doesn't output a version number >= 0.1.4, please go to https://sylver.dev to download a fresh copy of the software.

Prelude

We'll reuse two files from the last tutorial:

• json.syl

node JsonNode { }

node Null: JsonNode { }

node Bool: JsonNode { }

node Number: JsonNode { }

node String: JsonNode { }

node Array: JsonNode { 
    elems: List<JsonNode> 
}

node Object: JsonNode {
    members: List<Member>
}

node Member: JsonNode {
    key: String,
    value: JsonNode
}

term COMMA = ','
term COLON = ':'
term L_BRACE = '{'
term R_BRACE = '}'
term L_BRACKET = '['
term R_BRACKET = ']'
term NULL = 'null'

term BOOL_LIT = `true|false`
term NUMBER_LIT = `\-?(0|([1-9][0-9]*))(.[0-9]+)?((e|E)(\+|-)?[0-9]+)?`
term STRING_LIT = `"([^"\\]|(\\[\\/bnfrt"])|(\\u[a-fA-F0-9]{4}))*"`


ignore term WHITESPACE = `\s`

rule string = String { STRING_LIT }

rule member = Member { key@string COLON value@main }

rule main =
    Null { NULL }
  | Number { NUMBER_LIT }
  | Bool { BOOL_LIT }
  | string
  | Array { L_BRACKET elems@sepBy(COMMA, main) R_BRACKET }
  | Object { L_BRACE members@sepBy(COMMA, member) R_BRACE }@

• invalid_config.json

{
    "variables": [
        {
            "name": "date of birt`",
            "description": "Customer's date of birth",
            "type": "datetime"
        },
        {
            "name": "activity",
            "description": "A short text describing the customer's profession",
            "type": "string"
        },
        {
            "name": "country",
            "description": "Customer's country of residence",
            "type": "string",
            "values": ["us", "fr", "it" ]
        }
    ]
}

Stepping out of the REPL

Creating a ruleset

Packaging the rules from the previous tutorial into a reusable ruleset is as simple as creating the following YAML file:

id: 'JSON ruleset'
language: json.syl

rules: 
  - id: variable_length
    message: Variable name is too long
    category: style
    query: >
      match String desc when desc.text.length > 37 && desc.parent is {
        Member m when m.key.text == '"description"'
      }  

  - id: variable_format 
    message: Variable name isn't a lowercase word
    category: style
    query: >
      match String s when !s.text.matches(`"[a-z]+"`) && s.parent is {
        Member m when m.key.text == '"name"'
      }    

  - id: types_or_values
    message: Fields 'type' and 'values' are mutually exclusive    
    category: error
    note: The type can be deduced from the values list.
    query: >
      match Object n when
        any n.members.children match {  
            Member m when m.key.text == '"type"' 
        }
        && any n.members.children match { 
            Member m when m.key.text == '"values"' 
        }

Where id is a human-readable description of the ruleset, and language refers to a language spec file.

The following properties describe the individual rules composing the ruleset:

• id: unique and short name of the rule
• message: a concise description of the issue
• category: error, bug, smell, style
• query: inline query
• note: optional additional informations

Assuming that our ruleset file is called ruleset.yaml, we can run this ruleset on every .json file in the current directory by invoking the following command:

sylver ruleset run --files "*.json" --rulesets ruleset.yaml

Storing our project configuration

If we wish to validate our codebase against multiple rulesets, repeating the above command for every ruleset can be tedious. Instead, we can write a project configuration in a sylver.yaml file at the root of our project:

subprojects:
  - language: json.syl
    rulesets: ['ruleset.yaml']
    include:
      - './**/*.json'

The configuration contains a list of subprojects, each having a language, an optional list of rulesets, and a list of files to include.

Invoking sylver check will read the config from sylver.yaml and run the specified rulesets.

Git integration

Should you want to reuse your language specs or rulesets in several projects, copying your .syl and .yaml files in every project would be inconvenient. Luckily rulesets and project configurations can refer to artifacts stored in a git repository.

The language spec and ruleset for this tutorial have been uploaded to this repo, so if we rewrite our sylver.yaml config file as:

subprojects:
  - language: 
      repo: https://github.com/geoffreycopin/getting_started_json_tutorial
      file: json.syl
    rulesets: 
      - repo: https://github.com/geoffreycopin/getting_started_json_tutorial
        file: 'ruleset.yaml'
    include:
      - './**/*.json'

the language spec and ruleset will be cloned automatically in the .sylver directory when running sylver check.

Conclusion

We now have a reusable linter for our JSON configuration files built from scratch using Sylver's DSL.

The following tutorial will use a pre-built Golang to write a general-purpose Go linter.

If you have already installed sylver and sylver --version doesn't output a version number >= 0.1.3, please go to https://sylver.dev to download a fresh copy of the software.

Prelude

We will reuse the json.syl spec file built in the previous tutorial. Since the config.json file that we used to test our parser contains a valid configuration, we'll need to create a new JSON document (invalid_config.json) containing an incorrect configuration:

{
    "variables": [
        {
            "name": "date of birt`",
            "description": "Customer's date of birth",
            "type": "datetime"
        },
        {
            "name": "activity",
            "description": "A short text describing the customer's profession",
            "type": "string"
        },
        {
            "name": "country",
            "description": "Customer's country of residence",
            "type": "string",
            "values": ["us", "fr", "it" ]
        }
    ]
}

This file represents the configuration of an imaginary software handling a database of customers. Each customer profile is described using a configurable set of variables.

We want to validate these variable declarations using the following rules:

1. Variable descriptions must contain at most 35 characters
2. The name of a variable must be a single lowercase word
3. If the values field is set, the type field should be absent (as it matches the type of the values)

Let's parse this file and start a REPL to query our AST!

Basic REPL usage

Loading the AST of invalid_config.json is as simple as invoking:

sylver query --spec=json.syl --files=invalid_config.json

The files argument accepts one or several file names or quoted glob patterns.

You can exit the REPL by typing :quit at the prompt.

:print invalid_config.json and :print_ast invalid_config.json can be used to visualize one of the loaded files, or the corresponding AST.

Query language basics

Syntax queries are of the form match <NodeType> <binding>? (when <boolean expression>)?. The node binding and the when [...] clause are optional. NodeType represents either a node type as it appears when printing the AST or a placeholder (_) that matches every node. The whole part following the match keyword is called a query pattern.

In the REPL, queries must be followed by a ;

The most straightforward query, returning every node in the AST, is written as follows:

match _;

A slightly more advanced query to return every String node in our document:

match String;

If we only wish to retrieve the string literals above a certain length, we can add a when clause:

match String str when str.text.length > 35;

This query matches only the string literals whose text representation (quotes included) contains more than 35 characters. In our document, there is only one match on line 10.

The node type can be tested using the is keyword. For instance, to retrieve any node whose direct parent is an Object:

match _ node when node.parent is Object;

Returns the members list of all the objects in our document.

The is keyword can also test a node against a full query pattern surrounded by curly braces. So, for example, we can retrieve every node whose parent is a member with key name.

match _ node when node.parent is {
    Member m when m.key.text == '"name"'
};

String literals can be single or double quoted.

Now that we know how to write basic queries, let's try to find the nodes that violate our rules.

Rule 1: variable descriptions should be shorter than 35 characters

Except for the && operator in boolean expressions, this rule only uses features that appeared in the previous section so you can test yourself by trying to write it without looking at the following block of code!

match String desc when desc.text.length > 37 && desc.parent is {
   Member m when m.key.text == '"description"'
};

This query should return a single node on line 10 that is indeed longer than the specified length. Note that we check that the text length is above 37 instead of 35 because of the surrounding quotes.

Rule 2: variable names should be a single lowercase word

match String s when !s.text.matches(`"[a-z]+"`) && s.parent is {
   Member m when m.key.text == '"name"'
};

Returns a single node corresponding to the invalid date of birth name.

Apart from the boolean ! operator, this rule demonstrates the use of the matches method on text values. Unsurprisingly, it returns true when the text matches the regex literal given as an argument. As in spec files, regex literals are delimited by backticks.

Rule 3: fields type and values should be mutually exclusive

For this rule, we'll use array quantifying expressions of the form:

<quantifier> <array value> match <query pattern>

Where the quantifier is any of the following keywords: no, any, all. Array quantifying expressions return true when any, all, or none of the values in the given array match the query pattern. Using this new tool, we can find the Object nodes for witch there is at least one child member with key type and one child member with key values:

match Object n when
    any n.members.children match {  
        Member m when m.key.text == '"type"' 
    }
    && any n.members.children match { 
        Member m when m.key.text == '"values"' 
    };

Conclusion

We now have queries to identify violations of our business rules, but opening the REPL and pasting the queries whenever we want to validate a document isn't very practical. So, in the final part of this series, we'll learn how to package queries into a Sylver rule set to consume, distribute and share them more conveniently!

In this series of tutorials, we'll go from having zero knowledge of Sylver or static analysis to building a fully-fledged linter for our configuration files. We will use JSON as an example, but the tools and techniques presented apply to many data formats and even complete programming languages! Also, note that while we will be building everything from scratch using Sylver's domain-specific languages (DSL), a catalog of built-in specifications for the most common languages will be included in future releases of the tool.

• In part 1, we will discover Sylver's meta language, a DSL used to describe the shape of programming languages and data formats. After completing this tutorial, we'll have a complete spec for the JSON language, allowing us to turn JSON documents into Sylver parse trees.

• In part 2, we will load the parse trees into Sylver's query engine, and we will find nodes that violate our business rules using an SQL-like query language.

• In part 3, we will learn how to turn our language spec and queries into a set of linting rules so that we can run them conveniently and share them.

Installation

Sylver is distributed as a single static binary. Installing it is as simple as:

1. Go to https://sylver.dev to download the binary for your platform
2. Unpack the downloaded archive
3. Move the sylver binary to a location in your $PATH

Prelude

Let us create a blank workspace for this tutorial!
We'll start by creating a new folder and a json.syl file to write the Sylver specification for the JSON language.

mkdir sylver_getting_started
cd sylver_getting_started
touch json.syl

We will also store our test JSON file in config.json.

{
    "variables": [
        {
            "name": "country",
            "description": "Customer's country of residence",
            "values": ["us", "fr", "it"]
        },
        {
            "name": "age",
            "description": "Cusomer's age",
            "type": "number"
        }
    ]
}

This file specifies the variables used to describe customer profiles in a fictional customer database. Variables are assigned a type or a list of potential values.

Types definition

Sylver parses raw text into typed structures called parse trees. The first step in defining a language spec is to define a set of types for our tree nodes. We'll add the following node types declarations to our json.syl spec:

node JsonNode { }

node Null: JsonNode { }

node Bool: JsonNode { }

node Number: JsonNode { }

node String: JsonNode { }

node Array: JsonNode { 
    elems: List<JsonNode> 
}

node Object: JsonNode {
    members: List<Member>
}

node Member: JsonNode {
    key: String,
    value: JsonNode
}

These declarations resemble object type declarations in many mainstream languages. The : syntax denotes inheritance.

Now that we have a set of types to describe JSON documents, we need to specify how to build a parse tree from a sequence of characters. This process is done in two steps:

1. Lexical analysis: in this step, individual characters that form an indivisible entity (such as the digits of a number or the characters of a string) are grouped together into tokens. Some tokens are only one character-wide (for example, the brackets and semicolons in JSON).
2. Syntactic analysis, in which tree nodes are built for the stream of tokens.

Lexical analysis

Tokens are described using declarations of the form term NAME = <term_content> where <term_content> is either a literal surrounded by single-quotes (') or a regex between backticks (` ). The regexes use a syntax similar to Perl-style regular expressions.
Characters in the input string that match one of the terminal literals or regexes will be grouped into a token of the given name.

term COMMA = ','
term COLON = ':'
term L_BRACE = '{'
term R_BRACE = '}'
term L_BRACKET = '['
term R_BRACKET = ']'
term NULL = 'null'

term BOOL_LIT = `true|false`
term NUMBER_LIT = `\-?(0|([1-9][0-9]*))(.[0-9]+)?((e|E)(\+|-)?[0-9]+)?`
term STRING_LIT = `"([^"\\]|(\\[\\/bnfrt"])|(\\u[a-fA-F0-9]{4}))*"`


ignore term WHITESPACE = `\s`

Term rules for numbers and strings are slightly involved in accounting for some of JSON's peculiarities.

Note that the WHITESPACE term declaration (matching a single whitespace character) is prefixed with the ignore keyword. This means that WHITESPACE tokens do not affect the structure of the document and can be ignored during syntactic analysis.

Syntactic analysis

In this last part of the language spec, we write rules describing how tree nodes are built by matching tokens from the input stream.

For example, a rule specifying: "if the current token is a STRING_LIT, build a String node" can be written as follows:

rule string = String { STRING_LIT }

Rules can refer to other rules to construct nested nodes. For example, here is a rule specifying that a Member node (corresponding to an object member in JSON) can be built by building a node using the string rule and then matching a COLON token followed by any valid JSON value:

rule member = Member { key@string COLON value@main }

Nested nodes are associated with a field using the @ syntax. The main rule is the entry point for the parser, so in our case, it designates any valid JSON value.

A valid JSON document can be made of a 'null' literal, a number, a boolean value, a string, an array of JSON values, or a JSON object, which is reflected in the main rule:

rule main =
    Null { NULL }
  | Number { NUMBER_LIT }
  | Bool { BOOL_LIT }
  | string
  | Array { L_BRACKET elems@sepBy(COMMA, main) R_BRACKET }
  | Object { L_BRACE members@sepBy(COMMA, member) R_BRACE }

The sepBy(TOKEN, rule_name) syntax is used to parse nodes using the main rule, while matching a TOKEN token between every parsed node.

Conclusion

We now have a complete language spec for the JSON language:

node JsonNode { }

node Null: JsonNode { }

node Bool: JsonNode { }

node Number: JsonNode { }

node String: JsonNode { }

node Array: JsonNode { 
    elems: List<JsonNode> 
}

node Object: JsonNode {
    members: List<Member>
}

node Member: JsonNode {
    key: String,
    value: JsonNode
}

term COMMA = ','
term COLON = ':'
term L_BRACE = '{'
term R_BRACE = '}'
term L_BRACKET = '['
term R_BRACKET = ']'
term NULL = 'null'

term BOOL_LIT = `true|false`
term NUMBER_LIT = `\-?(0|([1-9][0-9]*))(.[0-9]+)?((e|E)(\+|-)?[0-9]+)?`
term STRING_LIT = `"([^"\\]|(\\[\\/bnfrt"])|(\\u[a-fA-F0-9]{4}))*"`


ignore term WHITESPACE = `\s`

rule string = String { STRING_LIT }

rule member = Member { key@string COLON value@main }

rule main =
    Null { NULL }
  | Number { NUMBER_LIT }
  | Bool { BOOL_LIT }
  | string
  | Array { L_BRACKET elems@sepBy(COMMA, main) R_BRACKET }
  | Object { L_BRACE members@sepBy(COMMA, member) R_BRACE }

The last step is to test it on our test file! This is done by invoking the following command: sylver parse --spec=json.syl --file=config.json

Which yields the following parse tree:

Object {
. ● members: List<Member> {
. . Member {
. . . ● key: String { "variables" }
. . . ● value: Array {
. . . . ● elems: List<JsonNode> {
. . . . . Object {
. . . . . . ● members: List<Member> {
. . . . . . . Member {
. . . . . . . . ● key: String { "name" }
. . . . . . . . ● value: String { "country" }
. . . . . . . }
. . . . . . . Member {
. . . . . . . . ● key: String { "description" }
. . . . . . . . ● value: String { "Customer's country of residence" }
. . . . . . . }
. . . . . . . Member {
. . . . . . . . ● key: String { "values" }
. . . . . . . . ● value: Array {
. . . . . . . . . ● elems: List<JsonNode> {
. . . . . . . . . . String { "us" }
. . . . . . . . . . String { "fr" }
. . . . . . . . . . String { "it" }
. . . . . . . . . }
. . . . . . . . }
. . . . . . . }
. . . . . . }
. . . . . }
. . . . . Object {
. . . . . . ● members: List<Member> {
. . . . . . . Member {
. . . . . . . . ● key: String { "name" }
. . . . . . . . ● value: String { "age" }
. . . . . . . }
. . . . . . . Member {
. . . . . . . . ● key: String { "description" }
. . . . . . . . ● value: String { "Customer's age" }
. . . . . . . }
. . . . . . . Member {
. . . . . . . . ● key: String { "type" }
. . . . . . . . ● value: String { "number" }
. . . . . . . }
. . . . . . }
. . . . . }
. . . . }
. . . }
. . }
. }
}

In the next part, we'll define business rules to validate our JSON configuration (for example, the possible values for each variable must be of the same type), and we will use a query DSL to identify the tree nodes that violate these rules.