Dynamic Atoms
Runtime-determined parsing via callbacks. Enable context-sensitive parsing where the grammar itself depends on input or previously captured values.
Key Concepts
- runtime callback invocation
- context access
- capture-aware decisions
- Packrat fallback
Why Parse This Format?
Dynamic atoms allow parsing behavior to change based on context. This enables scenarios impossible with static grammars.
Use Cases
- Configuration-driven parsing with different grammars per config
- Context-sensitive keywords that change meaning based on position
- Incremental parsing with hints from previous parses
- Plugin systems where users inject custom parsers
- Language detection and switching
Input Format
Any input where parsing behavior depends on context.
ruby def method... / python lambda x: ...Examples
ruby def method_nameVALIDRuby code with 'def' keyword
python lambda x: xVALIDPython code with 'lambda' keyword
Output Format
Atoms determined at runtime by callback.
Structure
callback_resultAtom returned by callback determines parsing
How the Code Works
This section explains how the parsing rules map to the format semantics.
Registering a Callback
use parsanol::portable::dynamic::{DynamicCallback, DynamicContext, register_dynamic_callback};
struct MyCallback;
impl DynamicCallback for MyCallback {
fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
// Return atom based on context
Some(Atom::Str { pattern: "hello".into() })
}
}
let callback_id = register_dynamic_callback(Box::new(MyCallback));Callbacks must implement `DynamicCallback` and be registered:
Using in Grammar
let grammar = GrammarBuilder::new()
.rule("dynamic", dynamic_with_id(callback_id))
.build();DynamicContext API
impl DynamicCallback for MyCallback {
fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
// Current position in input
let pos = ctx.pos();
// Full input string
let input = ctx.input();
// Access captures made so far
if let Some(lang) = ctx.get_capture("language") {
// Use capture to determine behavior
}
// Return the atom to use at this position
Some(Atom::Str { pattern: "hello".into() })
}
}Callbacks receive context with parsing state:
Context-Sensitive Keywords
struct LanguageKeywordCallback;
impl DynamicCallback for LanguageKeywordCallback {
fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
let input = ctx.input();
let pos = ctx.pos();
// Look at preceding context
if pos >= 5 && &input[pos - 5..pos] == "ruby " {
Some(Atom::Str { pattern: "def".into() })
} else if pos >= 7 && &input[pos - 7..pos] == "python " {
Some(Atom::Str { pattern: "lambda".into() })
} else {
Some(Atom::Str { pattern: "function".into() })
}
}
}Different keywords based on preceding context:
Capture-Aware Decisions
struct TypeAwareCallback;
impl DynamicCallback for TypeAwareCallback {
fn call(&self, ctx: &DynamicContext) -> Option<Atom> {
match ctx.get_capture("type") {
Some("int") => Some(Atom::Regex { pattern: r"\d+".into() }),
Some("str") => Some(Atom::Regex { pattern: r#"[^"]*"#.into() }),
Some("bool") => Some(Atom::Alternative {
atoms: vec![
Atom::Str { pattern: "true".into() },
Atom::Str { pattern: "false".into() },
],
}),
_ => None,
}
}
}Use previously made captures to determine parsing:
Configuration-Driven Parsing
struct ConfigCallback {
strict_mode: bool,
}
impl DynamicCallback for ConfigCallback {
fn call(&self, _ctx: &DynamicContext) -> Option<Atom> {
if self.strict_mode {
Some(Atom::Regex { pattern: r"[a-z][a-z0-9_]*".into() })
} else {
Some(Atom::Regex { pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into() })
}
}
}Change behavior based on configuration:
Configuration-Driven Parsing
struct ConfigCallback {
strict_mode: bool,
}
impl DynamicCallback for ConfigCallback {
fn call(&self, _ctx: &DynamicContext) -> Option<Atom> {
if self.strict_mode {
Some(Atom::Regex { pattern: r"[a-z][a-z0-9_]*".into() })
} else {
Some(Atom::Regex { pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into() })
}
}
}Change behavior based on configuration:
Source Code
//! Dynamic Atoms Example
//!
//! Demonstrates runtime-determined parsing via callbacks.
//! Dynamic atoms allow context-sensitive parsing where the grammar
//! itself depends on the input or previously captured values.
//!
//! Run with: cargo run --example dynamic --no-default-features
#![allow(clippy::print_literal)]
use parsanol::portable::dynamic::{register_dynamic_callback, DynamicCallback, DynamicContext};
use parsanol::portable::{
parser_dsl::{capture, dynamic, dynamic_with_id, re, seq, str, GrammarBuilder},
AstArena, Atom, PortableParser,
};
// =========================================================================
// Example 1: Constant Callback
// =========================================================================
struct ConstCallback {
atom: Atom,
desc: &'static str,
}
impl DynamicCallback for ConstCallback {
fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
println!(
" [ConstCallback '{}'] invoked at position {}",
self.desc,
ctx.pos()
);
Some(self.atom.clone())
}
fn description(&self) -> &str {
self.desc
}
}
fn example_const_callback() -> Result<(), Box<dyn std::error::Error>> {
println!("--- Example 1: Constant Callback ---\n");
let callback = ConstCallback {
atom: Atom::Str {
pattern: "hello".into(),
},
desc: "greeting",
};
let callback_id = register_dynamic_callback(Box::new(callback));
let grammar = GrammarBuilder::new()
.rule("dynamic", dynamic_with_id(callback_id))
.build();
let input = "hello world";
let mut arena = AstArena::for_input(input.len());
let mut parser = PortableParser::new(&grammar, input, &mut arena);
let result = parser.parse_from_pos(0)?;
println!(" Parsed successfully at position {}", result.end_pos);
Ok(())
}
// =========================================================================
// Example 2: Context-Sensitive Callback
// =========================================================================
struct LanguageKeywordCallback;
impl DynamicCallback for LanguageKeywordCallback {
fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
let input = ctx.input();
let pos = ctx.pos();
println!(" [LanguageKeywordCallback] at position {}", pos);
// Look at preceding context to determine language
if pos >= 5 && &input[pos - 5..pos] == "ruby " {
println!(" -> Detected Ruby context, returning 'def'");
Some(Atom::Str {
pattern: "def".into(),
})
} else if pos >= 7 && &input[pos - 7..pos] == "python " {
println!(" -> Detected Python context, returning 'lambda'");
Some(Atom::Str {
pattern: "lambda".into(),
})
} else {
println!(" -> No context detected, returning 'function'");
Some(Atom::Str {
pattern: "function".into(),
})
}
}
fn description(&self) -> &str {
"language_keyword"
}
}
fn example_context_sensitive() -> Result<(), Box<dyn std::error::Error>> {
println!("\n--- Example 2: Context-Sensitive Callback ---\n");
let callback_id = register_dynamic_callback(Box::new(LanguageKeywordCallback));
let grammar = GrammarBuilder::new()
.rule("keyword", dynamic_with_id(callback_id))
.build();
let test_cases = [
("ruby def method", "Ruby"),
("python lambda x", "Python"),
("function foo()", "JavaScript"),
];
for (input, lang) in test_cases {
println!(" Testing {} input: {:?}", lang, input);
let mut arena = AstArena::for_input(input.len());
let mut parser = PortableParser::new(&grammar, input, &mut arena);
match parser.parse_from_pos(0) {
Ok(result) => println!(" ✓ Parsed at position {}", result.end_pos),
Err(e) => println!(" ✗ Parse error: {:?}", e),
}
println!();
}
Ok(())
}
// =========================================================================
// Example 3: Position-Based Callback
// =========================================================================
struct PositionCallback;
impl DynamicCallback for PositionCallback {
fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
let pos = ctx.pos();
let input = ctx.input();
println!(" [PositionCallback] at position {}", pos);
// Different behavior at different positions
if pos == 0 {
// First position: expect a keyword (return regex that matches any of them)
Some(Atom::Re {
pattern: r"(let|const|var)".into(),
})
} else if pos < input.len() / 2 {
// First half: expect identifier
Some(Atom::Re {
pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into(),
})
} else {
// Second half: expect value
Some(Atom::Re {
pattern: r"\d+".into(),
})
}
}
fn description(&self) -> &str {
"position_based"
}
}
fn example_position_based() -> Result<(), Box<dyn std::error::Error>> {
println!("--- Example 3: Position-Based Callback ---\n");
let callback_id = register_dynamic_callback(Box::new(PositionCallback));
let grammar = GrammarBuilder::new()
.rule(
"stmt",
seq(vec![
dynamic(dynamic_with_id(callback_id)),
dynamic(str(" ")),
dynamic(dynamic_with_id(callback_id)),
dynamic(str("=")),
dynamic(dynamic_with_id(callback_id)),
]),
)
.build();
let input = "let x=42";
println!(" Parsing: {:?}", input);
let mut arena = AstArena::for_input(input.len());
let mut parser = PortableParser::new(&grammar, input, &mut arena);
match parser.parse_from_pos(0) {
Ok(result) => println!(" ✓ Parsed at position {}", result.end_pos),
Err(e) => println!(" ✗ Parse error: {:?}", e),
}
Ok(())
}
// =========================================================================
// Example 4: Configuration-Driven Parsing
// =========================================================================
struct ConfigCallback {
strict_mode: bool,
}
impl DynamicCallback for ConfigCallback {
fn resolve(&self, _ctx: &DynamicContext) -> Option<Atom> {
println!(" [ConfigCallback] strict_mode={}", self.strict_mode);
if self.strict_mode {
// Strict mode: only lowercase identifiers
Some(Atom::Re {
pattern: r"[a-z][a-z0-9_]*".into(),
})
} else {
// Lenient mode: any identifier
Some(Atom::Re {
pattern: r"[a-zA-Z_][a-zA-Z0-9_]*".into(),
})
}
}
fn description(&self) -> &str {
"config_driven"
}
}
fn example_config_driven() -> Result<(), Box<dyn std::error::Error>> {
println!("\n--- Example 4: Configuration-Driven Parsing ---\n");
// Strict mode callback
let strict_callback = ConfigCallback { strict_mode: true };
let strict_id = register_dynamic_callback(Box::new(strict_callback));
let strict_grammar = GrammarBuilder::new()
.rule("id", dynamic_with_id(strict_id))
.build();
println!(" Strict mode (lowercase only):");
for input in ["variable", "Variable"] {
let mut arena = AstArena::for_input(input.len());
let mut parser = PortableParser::new(&strict_grammar, input, &mut arena);
match parser.parse_from_pos(0) {
Ok(_) => println!(" ✓ {:?} - accepted", input),
Err(_) => println!(" ✗ {:?} - rejected", input),
}
}
Ok(())
}
// =========================================================================
// Example 5: Capture-Aware Callback
// =========================================================================
struct CaptureAwareCallback;
impl DynamicCallback for CaptureAwareCallback {
fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
println!(" [CaptureAwareCallback] checking captures");
// Check if we have a "type" capture - use get_capture_text for string
if let Some(type_name) = ctx.get_capture_text("type") {
println!(" -> Found type capture: {:?}", type_name);
match type_name.as_ref() {
"int" => Some(Atom::Re {
pattern: r"\d+".into(),
}),
"str" => Some(Atom::Re {
pattern: r"[a-zA-Z]+".into(),
}),
"bool" => Some(Atom::Re {
pattern: r"(true|false)".into(),
}),
_ => None,
}
} else {
println!(" -> No type capture, matching identifier");
Some(Atom::Re {
pattern: r"[a-z]+".into(),
})
}
}
fn description(&self) -> &str {
"capture_aware"
}
}
fn example_capture_aware() -> Result<(), Box<dyn std::error::Error>> {
println!("\n--- Example 5: Capture-Aware Callback ---\n");
let callback_id = register_dynamic_callback(Box::new(CaptureAwareCallback));
// Grammar: type:name = value
// The value parser depends on the type
let grammar = GrammarBuilder::new()
.rule(
"declaration",
seq(vec![
dynamic(capture("type", dynamic(re(r"[a-z]+")))),
dynamic(str(":")),
dynamic(capture("name", dynamic(re(r"[a-z]+")))),
dynamic(str("=")),
dynamic(capture("value", dynamic(dynamic_with_id(callback_id)))),
]),
)
.build();
let test_cases = [
("int:count=42", "int"),
("str:message=hello", "str"),
("bool:enabled=true", "bool"),
];
for (input, _expected_type) in test_cases {
println!(" Parsing: {:?}", input);
let mut arena = AstArena::for_input(input.len());
let mut parser = PortableParser::new(&grammar, input, &mut arena);
match parser.parse_from_pos(0) {
Ok(result) => {
println!(" ✓ Parsed successfully");
println!(" type: {:?}", result.get_capture("type", input));
println!(" name: {:?}", result.get_capture("name", input));
println!(" value: {:?}", result.get_capture("value", input));
}
Err(e) => println!(" ✗ Parse error: {:?}", e),
}
println!();
}
Ok(())
}
// =========================================================================
// Main
// =========================================================================
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Dynamic Atoms Example");
println!("====================\n");
example_const_callback()?;
example_context_sensitive()?;
example_position_based()?;
example_config_driven()?;
example_capture_aware()?;
// =========================================================================
// Summary
// =========================================================================
println!("\n--- Benefits of Dynamic Atoms ---");
println!("* Context-sensitive parsing at runtime");
println!("* Access to position, input, and captures");
println!("* Plugin architecture support");
println!("* Configuration-driven grammars");
println!("\n--- Backend Compatibility ---");
println!("* Packrat: Native support (direct callback invocation)");
println!("* Bytecode: Packrat fallback (slower, uses internal Packrat)");
println!("* Streaming: Packrat fallback (slower, uses internal Packrat)");
println!("\n--- Performance Notes ---");
println!("* Native (Packrat): ~5%% overhead per dynamic atom");
println!("* Fallback (Bytecode/Streaming): ~20%% slower than native");
println!("* For heavy dynamic usage, prefer Packrat backend");
println!("* Callback should be fast - avoid I/O or heavy computation");
println!("\n--- DSL Helper ---");
println!(" dynamic_with_id(callback_id) // Uses registered callback");
println!("\n--- API Summary ---");
println!(" trait DynamicCallback {{");
println!(" fn resolve(&self, ctx: &DynamicContext) -> Option<Atom>;");
println!(" fn description(&self) -> &str;");
println!(" }}");
Ok(())
}
# frozen_string_literal: true
# Dynamic Atoms Example
#
# Demonstrates runtime-determined parsing via callbacks.
# Dynamic atoms allow context-sensitive parsing where the grammar
# itself depends on the input or previously captured values.
$LOAD_PATH.unshift "#{File.dirname(__FILE__)}/../lib"
require "parsanol/parslet"
require "pp"
puts "Dynamic Atoms Example"
puts "====================\n"
# ===========================================================================
# Example 1: Constant Callback
# ===========================================================================
puts "--- Example 1: Constant Callback ---\n"
# Always returns the same parser
parser = dynamic { str("hello") }
input = "hello world"
result = parser.parse(input)
puts " Parsed successfully"
puts " Matched: #{result.inspect}"
# ===========================================================================
# Example 2: Context-Sensitive Callback
# ===========================================================================
puts "\n--- Example 2: Context-Sensitive Callback ---\n"
# Different keyword based on preceding context
class LanguageParser < Parsanol::Parser
include Parsanol::Parslet
rule(:keyword) do
dynamic do |_source, context|
# Get remaining input from current position
remaining = context.remaining
# Look at the beginning of remaining to detect context
if remaining.start_with?("def ")
puts " -> Detected Ruby context"
str("def")
elsif remaining.start_with?("lambda ")
puts " -> Detected Python context"
str("lambda")
else
puts " -> No context, using 'function'"
str("function")
end
end
end
rule(:statement) { keyword >> str(" ") >> match("[a-z]+") }
root :statement
end
test_cases = [
["def method", "Ruby"],
["lambda x", "Python"],
["function foo", "JavaScript"],
]
test_cases.each do |input, lang|
puts " Testing #{lang} input: #{input.inspect}"
parser = LanguageParser.new
begin
result = parser.parse(input)
puts " ✓ Parsed: #{result.inspect}"
rescue Parsanol::ParseFailed => e
puts " ✗ Parse error: #{e}"
end
puts
end
# ===========================================================================
# Example 3: Position-Based Callback
# ===========================================================================
puts "--- Example 3: Position-Based Callback ---\n"
class PositionParser < Parsanol::Parser
include Parsanol::Parslet
rule(:token) do
dynamic do |_source, context|
pos = context.pos
input_length = context.input.length
if pos.zero?
# First position: keyword
str("let") | str("const") | str("var")
elsif pos < input_length / 2
# First half: identifier
match("[a-zA-Z_][a-zA-Z0-9_]*")
else
# Second half: value
match('\d+') | match("[a-z]+")
end
end
end
rule(:stmt) { token >> str(" ") >> token >> str("=") >> token }
root :stmt
end
input = "let x=42"
puts " Parsing: #{input.inspect}"
parser = PositionParser.new
begin
result = parser.parse(input)
puts " ✓ Parsed: #{result.inspect}"
rescue Parsanol::ParseFailed => e
puts " ✗ Parse error: #{e}"
end
# ===========================================================================
# Example 4: Capture-Aware Callback
# ===========================================================================
puts "\n--- Example 4: Capture-Aware Callback ---\n"
class CaptureAwareParser < Parsanol::Parser
include Parsanol::Parslet
rule(:type) { match("[a-z]+").capture(:type) }
rule(:name) { match("[a-z]+").capture(:name) }
rule(:value) do
dynamic do |_source, context|
type = context[:type]
puts " -> Type capture: #{type.inspect}"
case type
when "int" then match('\d+')
when "str" then match("[a-z]+")
when "bool" then str("true") | str("false")
else match("[a-z]+")
end.capture(:value)
end
end
rule(:declaration) { type >> str(":") >> name >> str("=") >> value }
root :declaration
end
test_cases = [
["int:count=42", "int"],
["str:message=hello", "str"],
["bool:enabled=true", "bool"],
]
test_cases.each_key do |input|
puts " Parsing: #{input.inspect}"
parser = CaptureAwareParser.new
begin
result = parser.parse(input)
puts " ✓ Parsed successfully"
puts " type: #{result[:type].inspect}"
puts " name: #{result[:name].inspect}"
puts " value: #{result[:value].inspect}"
rescue Parsanol::ParseFailed => e
puts " ✗ Parse error: #{e}"
end
puts
end
# ===========================================================================
# Example 5: Configuration-Driven Parsing
# ===========================================================================
puts "\n--- Example 5: Configuration-Driven Parsing ---\n"
# Parser behavior can be configured at runtime
class ConfigurableParser < Parsanol::Parser
include Parsanol::Parslet
attr_accessor :strict_mode
rule(:identifier) do
dynamic do |_source, _context|
if @strict_mode
# Strict: lowercase only
match("[a-z][a-z0-9_]*")
else
# Lenient: any identifier
match("[a-zA-Z_][a-zA-Z0-9_]*")
end
end
end
root :identifier
end
puts " Strict mode (lowercase only):"
parser = ConfigurableParser.new
parser.strict_mode = true
%w[variable Variable].each do |input|
result = parser.parse(input)
puts " ✓ #{input.inspect} - accepted"
rescue Parsanol::ParseFailed
puts " ✗ #{input.inspect} - rejected"
end
puts "\n Lenient mode (any case):"
parser = ConfigurableParser.new
parser.strict_mode = false
%w[variable Variable].each do |input|
result = parser.parse(input)
puts " ✓ #{input.inspect} - accepted"
rescue Parsanol::ParseFailed
puts " ✗ #{input.inspect} - rejected"
end
# ===========================================================================
# Summary
# ===========================================================================
puts "\n--- Benefits of Dynamic Atoms ---"
puts "* Context-sensitive parsing at runtime"
puts "* Access to position, input, and captures"
puts "* Plugin architecture support"
puts "* Configuration-driven grammars"
puts "\n--- Backend Compatibility ---"
puts "* Packrat: Native support (direct callback invocation)"
puts "* Bytecode: Packrat fallback (slower)"
puts "* Streaming: Packrat fallback (slower)"
puts "\n--- Performance Notes ---"
puts "* Native (Packrat): ~5% overhead per dynamic atom"
puts "* Callback should be fast - avoid I/O or heavy computation"
puts "\n--- DSL Helper ---"
puts " dynamic { |source, context| parslet } # Block returns parser"
puts "\n--- API Summary ---"
puts " dynamic do |source, context|"
puts " context.pos # Current position"
puts " context.captures[:n] # Access captured values"
puts " context.input # Full input string"
puts " context.remaining # Remaining input from current position"
puts " # Return a parslet atom"
puts " end"