Tutorial — Parsanol

Introduction

A parser is a program that analyzes text according to a set of rules (grammar) and produces a structured representation of that text. Parsanol is a parser combinator library that lets you build parsers by combining smaller parsers together.

This tutorial will take you from zero to building real parsers.

Chapter 1: Basic Atoms

Atoms are the fundamental building blocks of parsers. They match specific patterns in the input text.

str() - Match Literal String

// Match exact string "hello"
let hello = str("hello");
let result = hello.parse("hello");
assert!(result.is_ok());

# Match exact string "hello"
hello = str("hello")
result = hello.parse("hello")
# => #<Parslet::Slice:0x... @value="hello">

re() - Regular Expression

// Match regex pattern [a-z]+ (one or more letters)
let word = re(r"[a-z]+");
let result = word.parse("hello");
assert!(result.is_ok());

# Match regex pattern [a-z]+ (one or more letters)
word = re("[a-z]+")
result = word.parse("hello")
# => #<Parslet::Slice:0x... @value="hello">

any() - Any Character

// Match any single character
let any_char = any();
let result = any_char.parse("x");
assert!(result.is_ok());

# Match any single character
any_char = any
result = any_char.parse("x")
# => #<Parslet::Slice:0x... @value="x">

match() - Character Class

// Match character class [0-9]
let digit = match("[0-9]");

# Match character class [0-9]
digit = match("[0-9]")

Chapter 2: Combinators

Combinators combine atoms (and other combinators) to build more complex parsers.

Sequence (then or >>)

Match multiple patterns in order.

// Sequence: match "hello" then " " then "world"
let greeting = str("hello") >> str(" ") >> str("world");

# Sequence: match "hello" then " " then "world"
greeting = str("hello") >> str(" ") >> str("world")

Choice (or or |)

Try each alternative in order.

// Choice: match "yes" or "no"
let yes_no = str("yes") | str("no");

# Choice: match "yes" or "no"
yes_no = str("yes") | str("no")

Repetition (repeat)

Match patterns zero or more times.

// Zero or more: match letters
let letters = re(r"[a-z]+").repeat();
// One or more: match digits
let digits = re(r"[0-9]+").repeat(1);
// Range: 1-3 times
let few = re(r"[a-z]").repeat(1, 3);

# Zero or more: match letters
letters = re("[a-z]+").repeat
# One or more: match digits
digits = re("[0-9]+").repeat(1)
# Range: 1-3 times
few = re("[a-z]").repeat(1, 3)

Optional (maybe)

Match a pattern zero or one time.

// Optional: match "Mr." or nothing
let title = str("Mr.").maybe();

# Optional: match "Mr." or nothing
title = str("Mr.").maybe

Chapter 3: Captures & Labels

Labels (.as)

Attach names to matched content for the parse tree.

// Label captured content
let word = re(r"[a-z]+").as("word");
// Parse tree: {word: "hello"}

# Label captured content
word = re("[a-z]+").as(:word)
# Parse tree: {:word => "hello"}

Lookahead

Check what's ahead without consuming input.

// Positive lookahead: must have "http" ahead
let http_url = str("http").present() >> re(r".*");

// Negative lookahead: not followed by "end"
let safe_end = str("end").absent() >> re(r".+");

# Positive lookahead: must have "http" ahead
http_url = str("http").present >> re(".*")

# Negative lookahead: not followed by "end"
safe_end = str("end").absent >> re(".+")

Chapter 4: Building a Real Parser

Let's build a complete JSON parser step by step.

Step 1: Basic Structure

// JSON Parser - Step 1: Basic structure

let json = choice!(vec![
    str("null").map(|_| JsonValue::Null),
    str("true").map(|_| JsonValue::Bool(true)),
    str("false").map(|_| JsonValue::Bool(false)),
]);

# JSON Parser - Step 1: Basic structure

class JsonParser < Parsanol::Parser
  root :value

  rule(:value) {
    str("null").as(:null) |
    str("true").as(:true) |
    str("false").as(:false)
  }
end

Step 2: Numbers and Strings

// JSON Parser - Step 2: Numbers and Strings

let number = re(r"-?[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?");
let string = str("\"") >> re(r#"[^"\\]*(\\.[^"\\]*)*"#).as("string") >> str("\"");

# JSON Parser - Step 2: Numbers and Strings

rule(:number) {
  match["0-9"].repeat(1).as(:num)
}

rule(:string) {
  str('"') >> (match['^"\\\\'] | str('\\\\') >> any).repeat >> str('"')
}

Step 3: Arrays and Objects

// JSON Parser - Step 3: Arrays and Objects

let array = str("[") >> value.repeat(0) >> str("]");
let object = str("{") >> (string >> str(":") >> value).repeat(0) >> str("}");

# JSON Parser - Step 3: Arrays and Objects

rule(:array) {
  str("[") >> (value >> str(",") >> value).repeat >> str("]")
}

rule(:object) {
  str("{") >> (string >> str(":") >> value).repeat >> str("}")
}

Step 4: Complete Parser

# Complete JSON Parser

require 'parsanol'

class JsonParser < Parsanol::Parser
  root :value

  rule(:spaces) { match['\\s'].repeat }

  rule(:value) {
    spaces >> (
      object | array | string | number | literal
    ) >> spaces
  }

  rule(:object) {
    str('{') >> (
      entry >> (str(',') >> entry).repeat
    ).maybe >> str('}')
  end

  rule(:entry) {
    string.as(:key) >> str(':') >> value.as(:val)
  }

  rule(:array) {
    str('[') >> (
      value >> (str(',') >> value).repeat
    ).maybe >> str(']')
  end

  rule(:string) {
    str('"') >> (
      str('\\\\') >> any | str('"').absent? >> any
    ).repeat.as(:string) >> str('"')
  end

  rule(:number) {
    str('-').maybe >> (
      str('0') | match['1-9'] >> match['0-9'].repeat
    ) >> (
      str('.') >> match['0-9'].repeat
    ).maybe >> (
      match['eE'] >> str('+-').maybe >> match['0-9'].repeat
    ).maybe
  end

  rule(:literal) {
    str('null').as(:null) |
    str('true').as(:true) |
    str('false').as(:false)
  }
end

# Test
parser = JsonParser.new
puts parser.parse('{"name": "John", "age": 30}').inspect

Chapter 5: Transformations

Transforms convert the generic parse tree into your domain objects.

Basic Transform

// Transform: Convert parse tree to typed value

let transform = TransformBuilder::new()
    .rule("num", |s: &str| {
        Ok(Value::Number(s.parse()?))
    })
    .build();

# Transform: Convert parse tree to typed value

class JsonTransform < Parsanol::Transform
  rule(string: simple(:s)) { s.to_s }
  rule(num: simple(:n)) { Integer(n) }
  rule(null: simple(:n)) { nil }
  rule(true: simple(:t)) { true }
  rule(false: simple(:f)) { false }
end

Recursive Transform

// Recursive transform for nested structures

let transform = TransformBuilder::new()
    .rule("num", |s: &str| {
        Ok(Value::Number(s.parse()?))
    })
    .rule("array", |arr: Vec<Value>| {
        Ok(Value::Array(arr))
    })
    .rule("object", |pairs: Vec<(String, Value)>| {
        Ok(Value::Object(pairs.into_iter().collect()))
    })
    .build();

# Recursive transform for nested structures

class JsonTransform < Parsanol::Transform
  rule(string: simple(:s)) { s.to_s }
  rule(num: simple(:n)) { Integer(n) }
  rule(null: simple(:n)) { nil }
  rule(true: simple(:t)) { true }
  rule(false: simple(:f)) { false }

  # Recursive rules
  rule(array: sequence(:arr)) {
    arr.map { |v| transform_value(v) }
  }

  rule(object: sequence(:pairs)) {
    Hash[pairs.map { |k, v| [k, transform_value(v)] }]
  end

  def transform_value(tree)
    apply(tree)
  end
end

Best Practices

1. Start Simple

Build your parser incrementally. Test each rule before combining them.

2. Handle Whitespace

Create a reusable `space?` rule and use it throughout your grammar.

3. Name Your Rules

Use `.as(:name)` to label captures for easier debugging.

4. Use Two-Stage Parsing

Separate parsing (syntax) from transformation (semantics).

Parser Writing Tutorial