Skip to content
← Back to Examples

Dynamic Atoms

RustRubyconceptualadvanced

Runtime-determined parsing via callbacks. Enable context-sensitive parsing where the grammar itself depends on input or previously captured values.

Key Concepts

  • runtime callback invocation
  • context access
  • capture-aware decisions
  • Packrat fallback

Why Parse This Format?

Dynamic atoms allow parsing behavior to change based on context. This enables scenarios impossible with static grammars.

Use Cases

  • Configuration-driven parsing with different grammars per config
  • Context-sensitive keywords that change meaning based on position
  • Incremental parsing with hints from previous parses
  • Plugin systems where users inject custom parsers
  • Language detection and switching

Input Format

Any input where parsing behavior depends on context.

ruby def method... / python lambda x: ...

Examples

ruby def method_nameVALID

Ruby code with 'def' keyword

python lambda x: xVALID

Python code with 'lambda' keyword

Output Format

Atoms determined at runtime by callback.

Structure

callback_result

Atom returned by callback determines parsing

How the Code Works

This section explains how the parsing rules map to the format semantics.

Registering a Callback

use parsanol::portable::dynamic::{DynamicCallback, DynamicContext, register_dynamic_callback};

struct MyCallback;

impl DynamicCallback for MyCallback {
    fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
        // Return atom based on context
        Some(Atom::Str { pattern: "hello".into() })
    }
}

let callback_id = register_dynamic_callback(Box::new(MyCallback));

Callbacks must implement `DynamicCallback` and be registered:

Using in Grammar

let grammar = GrammarBuilder::new()
    .rule("dynamic", dynamic_with_id(callback_id))
    .build();

DynamicContext API

impl DynamicCallback for MyCallback {
    fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
        // Current position in input
        let pos = ctx.pos();

        // Full input string
        let input = ctx.input();

        // Access captures made so far
        if let Some(lang) = ctx.get_capture("language") {
            // Use capture to determine behavior
        }

        // Return the atom to use at this position
        Some(Atom::Str { pattern: "hello".into() })
    }
}

Callbacks receive context with parsing state:

Context-Sensitive Keywords

struct LanguageKeywordCallback;

impl DynamicCallback for LanguageKeywordCallback {
    fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
        let input = ctx.input();
        let pos = ctx.pos();

        // Look at preceding context
        if pos >= 5 && &input[pos - 5..pos] == "ruby " {
            Some(Atom::Str { pattern: "def".into() })
        } else if pos >= 7 && &input[pos - 7..pos] == "python " {
            Some(Atom::Str { pattern: "lambda".into() })
        } else {
            Some(Atom::Str { pattern: "function".into() })
        }
    }
}

Different keywords based on preceding context:

Capture-Aware Decisions

struct TypeAwareCallback;

impl DynamicCallback for TypeAwareCallback {
    fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
        match ctx.get_capture("type") {
            Some("int") => Some(Atom::Regex { pattern: r"\d+".into() }),
            Some("str") => Some(Atom::Regex { pattern: r#"[^"]*"#.into() }),
            Some("bool") => Some(Atom::Alternative {
                atoms: vec![
                    Atom::Str { pattern: "true".into() },
                    Atom::Str { pattern: "false".into() },
                ],
            }),
            _ => None,
        }
    }
}

Use previously made captures to determine parsing:

Configuration-Driven Parsing

struct ConfigCallback {
    strict_mode: bool,
}

impl DynamicCallback for ConfigCallback {
    fn call(&self, _ctx: &DynamicContext) -> Option<Atom> {
        if self.strict_mode {
            Some(Atom::Regex { pattern: r"[a-z][a-z0-9_]*".into() })
        } else {
            Some(Atom::Regex { pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into() })
        }
    }
}

Change behavior based on configuration:

Configuration-Driven Parsing

struct ConfigCallback {
    strict_mode: bool,
}

impl DynamicCallback for ConfigCallback {
    fn call(&self, _ctx: &DynamicContext) -> Option<Atom> {
        if self.strict_mode {
            Some(Atom::Regex { pattern: r"[a-z][a-z0-9_]*".into() })
        } else {
            Some(Atom::Regex { pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into() })
        }
    }
}

Change behavior based on configuration:

Source Code

Rust
//! Dynamic Atoms Example
//!
//! Demonstrates runtime-determined parsing via callbacks.
//! Dynamic atoms allow context-sensitive parsing where the grammar
//! itself depends on the input or previously captured values.
//!
//! Run with: cargo run --example dynamic --no-default-features

#![allow(clippy::print_literal)]

use parsanol::portable::dynamic::{register_dynamic_callback, DynamicCallback, DynamicContext};
use parsanol::portable::{
    parser_dsl::{capture, dynamic, dynamic_with_id, re, seq, str, GrammarBuilder},
    AstArena, Atom, PortableParser,
};

// =========================================================================
// Example 1: Constant Callback
// =========================================================================

struct ConstCallback {
    atom: Atom,
    desc: &'static str,
}

impl DynamicCallback for ConstCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        println!(
            "    [ConstCallback '{}'] invoked at position {}",
            self.desc,
            ctx.pos()
        );
        Some(self.atom.clone())
    }

    fn description(&self) -> &str {
        self.desc
    }
}

fn example_const_callback() -> Result<(), Box<dyn std::error::Error>> {
    println!("--- Example 1: Constant Callback ---\n");

    let callback = ConstCallback {
        atom: Atom::Str {
            pattern: "hello".into(),
        },
        desc: "greeting",
    };
    let callback_id = register_dynamic_callback(Box::new(callback));

    let grammar = GrammarBuilder::new()
        .rule("dynamic", dynamic_with_id(callback_id))
        .build();

    let input = "hello world";
    let mut arena = AstArena::for_input(input.len());
    let mut parser = PortableParser::new(&grammar, input, &mut arena);
    let result = parser.parse_from_pos(0)?;

    println!("  Parsed successfully at position {}", result.end_pos);
    Ok(())
}

// =========================================================================
// Example 2: Context-Sensitive Callback
// =========================================================================

struct LanguageKeywordCallback;

impl DynamicCallback for LanguageKeywordCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        let input = ctx.input();
        let pos = ctx.pos();

        println!("    [LanguageKeywordCallback] at position {}", pos);

        // Look at preceding context to determine language
        if pos >= 5 && &input[pos - 5..pos] == "ruby " {
            println!("    -> Detected Ruby context, returning 'def'");
            Some(Atom::Str {
                pattern: "def".into(),
            })
        } else if pos >= 7 && &input[pos - 7..pos] == "python " {
            println!("    -> Detected Python context, returning 'lambda'");
            Some(Atom::Str {
                pattern: "lambda".into(),
            })
        } else {
            println!("    -> No context detected, returning 'function'");
            Some(Atom::Str {
                pattern: "function".into(),
            })
        }
    }

    fn description(&self) -> &str {
        "language_keyword"
    }
}

fn example_context_sensitive() -> Result<(), Box<dyn std::error::Error>> {
    println!("\n--- Example 2: Context-Sensitive Callback ---\n");

    let callback_id = register_dynamic_callback(Box::new(LanguageKeywordCallback));

    let grammar = GrammarBuilder::new()
        .rule("keyword", dynamic_with_id(callback_id))
        .build();

    let test_cases = [
        ("ruby def method", "Ruby"),
        ("python lambda x", "Python"),
        ("function foo()", "JavaScript"),
    ];

    for (input, lang) in test_cases {
        println!("  Testing {} input: {:?}", lang, input);
        let mut arena = AstArena::for_input(input.len());
        let mut parser = PortableParser::new(&grammar, input, &mut arena);
        match parser.parse_from_pos(0) {
            Ok(result) => println!("  ✓ Parsed at position {}", result.end_pos),
            Err(e) => println!("  ✗ Parse error: {:?}", e),
        }
        println!();
    }

    Ok(())
}

// =========================================================================
// Example 3: Position-Based Callback
// =========================================================================

struct PositionCallback;

impl DynamicCallback for PositionCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        let pos = ctx.pos();
        let input = ctx.input();

        println!("    [PositionCallback] at position {}", pos);

        // Different behavior at different positions
        if pos == 0 {
            // First position: expect a keyword (return regex that matches any of them)
            Some(Atom::Re {
                pattern: r"(let|const|var)".into(),
            })
        } else if pos < input.len() / 2 {
            // First half: expect identifier
            Some(Atom::Re {
                pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into(),
            })
        } else {
            // Second half: expect value
            Some(Atom::Re {
                pattern: r"\d+".into(),
            })
        }
    }

    fn description(&self) -> &str {
        "position_based"
    }
}

fn example_position_based() -> Result<(), Box<dyn std::error::Error>> {
    println!("--- Example 3: Position-Based Callback ---\n");

    let callback_id = register_dynamic_callback(Box::new(PositionCallback));

    let grammar = GrammarBuilder::new()
        .rule(
            "stmt",
            seq(vec![
                dynamic(dynamic_with_id(callback_id)),
                dynamic(str(" ")),
                dynamic(dynamic_with_id(callback_id)),
                dynamic(str("=")),
                dynamic(dynamic_with_id(callback_id)),
            ]),
        )
        .build();

    let input = "let x=42";
    println!("  Parsing: {:?}", input);
    let mut arena = AstArena::for_input(input.len());
    let mut parser = PortableParser::new(&grammar, input, &mut arena);
    match parser.parse_from_pos(0) {
        Ok(result) => println!("  ✓ Parsed at position {}", result.end_pos),
        Err(e) => println!("  ✗ Parse error: {:?}", e),
    }

    Ok(())
}

// =========================================================================
// Example 4: Configuration-Driven Parsing
// =========================================================================

struct ConfigCallback {
    strict_mode: bool,
}

impl DynamicCallback for ConfigCallback {
    fn resolve(&self, _ctx: &DynamicContext) -> Option<Atom> {
        println!("    [ConfigCallback] strict_mode={}", self.strict_mode);

        if self.strict_mode {
            // Strict mode: only lowercase identifiers
            Some(Atom::Re {
                pattern: r"[a-z][a-z0-9_]*".into(),
            })
        } else {
            // Lenient mode: any identifier
            Some(Atom::Re {
                pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into(),
            })
        }
    }

    fn description(&self) -> &str {
        "config_driven"
    }
}

fn example_config_driven() -> Result<(), Box<dyn std::error::Error>> {
    println!("\n--- Example 4: Configuration-Driven Parsing ---\n");

    // Strict mode callback
    let strict_callback = ConfigCallback { strict_mode: true };
    let strict_id = register_dynamic_callback(Box::new(strict_callback));

    let strict_grammar = GrammarBuilder::new()
        .rule("id", dynamic_with_id(strict_id))
        .build();

    println!("  Strict mode (lowercase only):");
    for input in ["variable", "Variable"] {
        let mut arena = AstArena::for_input(input.len());
        let mut parser = PortableParser::new(&strict_grammar, input, &mut arena);
        match parser.parse_from_pos(0) {
            Ok(_) => println!("    ✓ {:?} - accepted", input),
            Err(_) => println!("    ✗ {:?} - rejected", input),
        }
    }

    Ok(())
}

// =========================================================================
// Example 5: Capture-Aware Callback
// =========================================================================

struct CaptureAwareCallback;

impl DynamicCallback for CaptureAwareCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        println!("    [CaptureAwareCallback] checking captures");

        // Check if we have a "type" capture - use get_capture_text for string
        if let Some(type_name) = ctx.get_capture_text("type") {
            println!("    -> Found type capture: {:?}", type_name);
            match type_name.as_ref() {
                "int" => Some(Atom::Re {
                    pattern: r"\d+".into(),
                }),
                "str" => Some(Atom::Re {
                    pattern: r"[a-zA-Z]+".into(),
                }),
                "bool" => Some(Atom::Re {
                    pattern: r"(true|false)".into(),
                }),
                _ => None,
            }
        } else {
            println!("    -> No type capture, matching identifier");
            Some(Atom::Re {
                pattern: r"[a-z]+".into(),
            })
        }
    }

    fn description(&self) -> &str {
        "capture_aware"
    }
}

fn example_capture_aware() -> Result<(), Box<dyn std::error::Error>> {
    println!("\n--- Example 5: Capture-Aware Callback ---\n");

    let callback_id = register_dynamic_callback(Box::new(CaptureAwareCallback));

    // Grammar: type:name = value
    // The value parser depends on the type
    let grammar = GrammarBuilder::new()
        .rule(
            "declaration",
            seq(vec![
                dynamic(capture("type", dynamic(re(r"[a-z]+")))),
                dynamic(str(":")),
                dynamic(capture("name", dynamic(re(r"[a-z]+")))),
                dynamic(str("=")),
                dynamic(capture("value", dynamic(dynamic_with_id(callback_id)))),
            ]),
        )
        .build();

    let test_cases = [
        ("int:count=42", "int"),
        ("str:message=hello", "str"),
        ("bool:enabled=true", "bool"),
    ];

    for (input, _expected_type) in test_cases {
        println!("  Parsing: {:?}", input);
        let mut arena = AstArena::for_input(input.len());
        let mut parser = PortableParser::new(&grammar, input, &mut arena);
        match parser.parse_from_pos(0) {
            Ok(result) => {
                println!("  ✓ Parsed successfully");
                println!("    type:  {:?}", result.get_capture("type", input));
                println!("    name:  {:?}", result.get_capture("name", input));
                println!("    value: {:?}", result.get_capture("value", input));
            }
            Err(e) => println!("  ✗ Parse error: {:?}", e),
        }
        println!();
    }

    Ok(())
}

// =========================================================================
// Main
// =========================================================================

fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("Dynamic Atoms Example");
    println!("====================\n");

    example_const_callback()?;
    example_context_sensitive()?;
    example_position_based()?;
    example_config_driven()?;
    example_capture_aware()?;

    // =========================================================================
    // Summary
    // =========================================================================
    println!("\n--- Benefits of Dynamic Atoms ---");
    println!("* Context-sensitive parsing at runtime");
    println!("* Access to position, input, and captures");
    println!("* Plugin architecture support");
    println!("* Configuration-driven grammars");

    println!("\n--- Backend Compatibility ---");
    println!("* Packrat:  Native support (direct callback invocation)");
    println!("* Bytecode: Packrat fallback (slower, uses internal Packrat)");
    println!("* Streaming: Packrat fallback (slower, uses internal Packrat)");

    println!("\n--- Performance Notes ---");
    println!("* Native (Packrat): ~5%% overhead per dynamic atom");
    println!("* Fallback (Bytecode/Streaming): ~20%% slower than native");
    println!("* For heavy dynamic usage, prefer Packrat backend");
    println!("* Callback should be fast - avoid I/O or heavy computation");

    println!("\n--- DSL Helper ---");
    println!("  dynamic_with_id(callback_id)  // Uses registered callback");

    println!("\n--- API Summary ---");
    println!("  trait DynamicCallback {{");
    println!("    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom>;");
    println!("    fn description(&self) -> &str;");
    println!("  }}");

    Ok(())
}
Ruby
# frozen_string_literal: true

# Dynamic Atoms Example
#
# Demonstrates runtime-determined parsing via callbacks.
# Dynamic atoms allow context-sensitive parsing where the grammar
# itself depends on the input or previously captured values.

$LOAD_PATH.unshift "#{File.dirname(__FILE__)}/../lib"
require "parsanol/parslet"
require "pp"

puts "Dynamic Atoms Example"
puts "====================\n"

# ===========================================================================
# Example 1: Constant Callback
# ===========================================================================
puts "--- Example 1: Constant Callback ---\n"

# Always returns the same parser
parser = dynamic { str("hello") }

input = "hello world"
result = parser.parse(input)
puts "  Parsed successfully"
puts "  Matched: #{result.inspect}"

# ===========================================================================
# Example 2: Context-Sensitive Callback
# ===========================================================================
puts "\n--- Example 2: Context-Sensitive Callback ---\n"

# Different keyword based on preceding context
class LanguageParser < Parsanol::Parser
  include Parsanol::Parslet

  rule(:keyword) do
    dynamic do |_source, context|
      # Get remaining input from current position
      remaining = context.remaining

      # Look at the beginning of remaining to detect context
      if remaining.start_with?("def ")
        puts "    -> Detected Ruby context"
        str("def")
      elsif remaining.start_with?("lambda ")
        puts "    -> Detected Python context"
        str("lambda")
      else
        puts "    -> No context, using 'function'"
        str("function")
      end
    end
  end

  rule(:statement) { keyword >> str(" ") >> match("[a-z]+") }
  root :statement
end

test_cases = [
  ["def method", "Ruby"],
  ["lambda x", "Python"],
  ["function foo", "JavaScript"],
]

test_cases.each do |input, lang|
  puts "  Testing #{lang} input: #{input.inspect}"
  parser = LanguageParser.new
  begin
    result = parser.parse(input)
    puts "  ✓ Parsed: #{result.inspect}"
  rescue Parsanol::ParseFailed => e
    puts "  ✗ Parse error: #{e}"
  end
  puts
end

# ===========================================================================
# Example 3: Position-Based Callback
# ===========================================================================
puts "--- Example 3: Position-Based Callback ---\n"

class PositionParser < Parsanol::Parser
  include Parsanol::Parslet

  rule(:token) do
    dynamic do |_source, context|
      pos = context.pos
      input_length = context.input.length

      if pos.zero?
        # First position: keyword
        str("let") | str("const") | str("var")
      elsif pos < input_length / 2
        # First half: identifier
        match("[a-zA-Z_][a-zA-Z0-9_]*")
      else
        # Second half: value
        match('\d+') | match("[a-z]+")
      end
    end
  end

  rule(:stmt) { token >> str(" ") >> token >> str("=") >> token }
  root :stmt
end

input = "let x=42"
puts "  Parsing: #{input.inspect}"
parser = PositionParser.new
begin
  result = parser.parse(input)
  puts "  ✓ Parsed: #{result.inspect}"
rescue Parsanol::ParseFailed => e
  puts "  ✗ Parse error: #{e}"
end

# ===========================================================================
# Example 4: Capture-Aware Callback
# ===========================================================================
puts "\n--- Example 4: Capture-Aware Callback ---\n"

class CaptureAwareParser < Parsanol::Parser
  include Parsanol::Parslet

  rule(:type) { match("[a-z]+").capture(:type) }
  rule(:name) { match("[a-z]+").capture(:name) }

  rule(:value) do
    dynamic do |_source, context|
      type = context[:type]
      puts "    -> Type capture: #{type.inspect}"

      case type
      when "int" then match('\d+')
      when "str" then match("[a-z]+")
      when "bool" then str("true") | str("false")
      else match("[a-z]+")
      end.capture(:value)
    end
  end

  rule(:declaration) { type >> str(":") >> name >> str("=") >> value }
  root :declaration
end

test_cases = [
  ["int:count=42", "int"],
  ["str:message=hello", "str"],
  ["bool:enabled=true", "bool"],
]

test_cases.each_key do |input|
  puts "  Parsing: #{input.inspect}"
  parser = CaptureAwareParser.new
  begin
    result = parser.parse(input)
    puts "  ✓ Parsed successfully"
    puts "    type: #{result[:type].inspect}"
    puts "    name: #{result[:name].inspect}"
    puts "    value: #{result[:value].inspect}"
  rescue Parsanol::ParseFailed => e
    puts "  ✗ Parse error: #{e}"
  end
  puts
end

# ===========================================================================
# Example 5: Configuration-Driven Parsing
# ===========================================================================
puts "\n--- Example 5: Configuration-Driven Parsing ---\n"

# Parser behavior can be configured at runtime
class ConfigurableParser < Parsanol::Parser
  include Parsanol::Parslet

  attr_accessor :strict_mode

  rule(:identifier) do
    dynamic do |_source, _context|
      if @strict_mode
        # Strict: lowercase only
        match("[a-z][a-z0-9_]*")
      else
        # Lenient: any identifier
        match("[a-zA-Z_][a-zA-Z0-9_]*")
      end
    end
  end

  root :identifier
end

puts "  Strict mode (lowercase only):"
parser = ConfigurableParser.new
parser.strict_mode = true

%w[variable Variable].each do |input|
  result = parser.parse(input)
  puts "    ✓ #{input.inspect} - accepted"
rescue Parsanol::ParseFailed
  puts "    ✗ #{input.inspect} - rejected"
end

puts "\n  Lenient mode (any case):"
parser = ConfigurableParser.new
parser.strict_mode = false

%w[variable Variable].each do |input|
  result = parser.parse(input)
  puts "    ✓ #{input.inspect} - accepted"
rescue Parsanol::ParseFailed
  puts "    ✗ #{input.inspect} - rejected"
end

# ===========================================================================
# Summary
# ===========================================================================
puts "\n--- Benefits of Dynamic Atoms ---"
puts "* Context-sensitive parsing at runtime"
puts "* Access to position, input, and captures"
puts "* Plugin architecture support"
puts "* Configuration-driven grammars"

puts "\n--- Backend Compatibility ---"
puts "* Packrat:  Native support (direct callback invocation)"
puts "* Bytecode: Packrat fallback (slower)"
puts "* Streaming: Packrat fallback (slower)"

puts "\n--- Performance Notes ---"
puts "* Native (Packrat): ~5% overhead per dynamic atom"
puts "* Callback should be fast - avoid I/O or heavy computation"

puts "\n--- DSL Helper ---"
puts "  dynamic { |source, context| parslet }  # Block returns parser"

puts "\n--- API Summary ---"
puts "  dynamic do |source, context|"
puts "    context.pos           # Current position"
puts "    context.captures[:n]  # Access captured values"
puts "    context.input         # Full input string"
puts "    context.remaining     # Remaining input from current position"
puts "    # Return a parslet atom"
puts "  end"