Captures & Scopes

When to Use Each Feature

Each feature serves a specific purpose. Use this guide to choose the right tool for your parsing task.

Quick Decision Guide

1

Need to extract specific values from input?

→ Use Captures - Like named groups in regex, extract parts by name

2

Parsing nested or recursive structures?

→ Use Scopes - Isolate captures per nesting level

3

Grammar depends on context or captures?

→ Use Dynamic Atoms - Runtime-determined parsing

4

Processing large files without memory issues?

→ Use Streaming + Captures - Chunk-based with capture extraction

Feature Comparison

Aspect	Captures	Scopes	Dynamic
Purpose	Extract values	Isolate context	Change grammar
When to use	Need named values	Nested structures	Context-sensitive
Performance	~5% overhead	~2% per level	~5-20% overhead
Memory impact	O(captures)	Bounded by depth	Callback-dependent
Can access captures?	✓	✓ (outer only)	✓
Works in streaming?	✓	✓	~ Fallback

Common Use Cases

📧 Email Extraction

Extract local and domain parts from email addresses.

Use: Captures

let grammar = GrammarBuilder::new()
    .rule("email", seq(vec![
        capture("local", dynamic(re(r"[a-zA-Z0-9._%+-]+"))),
        dynamic(str("@")),
        capture("domain", dynamic(re(r"[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"))),
    ]))
    .build();

let result = parser.parse_from_pos(0)?;
println!("Local: {:?}", result.get_capture("local", input));
println!("Domain: {:?}", result.get_capture("domain", input));

⚙️ INI Configuration Parsing

Parse sections with isolated key-value captures.

Use: Captures + Scopes

let grammar = GrammarBuilder::new()
    .rule("section", seq(vec![
        dynamic(str("[")),
        capture("section", dynamic(re(r"[a-z_]+"))),  // Persists
        dynamic(str("]\n")),
        scope(seq(vec![                               // Isolated
            capture("key", dynamic(re(r"[a-z_]+"))),
            dynamic(str("=")),
            capture("value", dynamic(re(r"[^\n]+"))),
        ])),
    ]))
    .build();
// Section name available, but key/value are discarded per section

🔧 Type-Driven Value Parsing

Parse values differently based on type annotation.

Use: Captures + Dynamic

struct TypeCallback;
impl DynamicCallback for TypeCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        match ctx.get_capture_text("type")? {
            "int" => Some(Atom::Re { pattern: r"\d+".into() }),
            "str" => Some(Atom::Re { pattern: r#"[^"]*"#.into() }),
            "bool" => Some(Atom::Re { pattern: r"(true|false)".into() }),
            _ => None,
        }
    }
}

// Grammar: type:name = value
// "int:count=42" parses 42 as digits
// "str:msg=hello" parses hello as string
// "bool:flag=true" parses true as boolean

📊 Large Log File Analysis

Extract fields from GB-scale log files with bounded memory.

Use: Streaming + Captures

let grammar = GrammarBuilder::new()
    .rule("log_line", seq(vec![
        capture("ip", dynamic(re(r"\d+\.\d+\.\d+\.\d+"))),
        // ... more fields ...
        capture("status", dynamic(re(r"\d+"))),
    ]))
    .build();

let config = ChunkConfig { chunk_size: 1024 * 1024, window_size: 2 };
let mut parser = StreamingParser::new(&grammar, config);

let result = parser.parse_from_reader(&mut file, &mut arena)?;
if let Some(captures) = &result.capture_state {
    // All IPs and status codes from 10GB file, ~2MB memory
}

📄 Heredoc / Here-Document

Match delimiter at start and end of document.

Use: Captures + Dynamic

// Capture the delimiter, then use dynamic to match it at the end
let grammar = GrammarBuilder::new()
    .rule("heredoc", seq(vec![
        dynamic(str("<<")),
        capture("delimiter", dynamic(re(r"[A-Z]+"))),  // Capture EOF
        dynamic(str("\n"))),
        dynamic(re(r"[\s\S]*?")),                       // Content
        dynamic(str("\n"))),
        // Dynamic uses the captured delimiter to match end
        dynamic(dynamic_with_id(match_delimiter_callback)),
    ]))
    .build();

// Input: <<EOF\nhello world\nEOF
// Matches because dynamic callback returns str("EOF")

Performance Benchmarks

Benchmarks measured on a representative set of parsing tasks. Actual performance depends on grammar complexity and input size.

Feature	Overhead vs Baseline	Memory Impact	Notes
Capture Atoms	~5-8%	O(captures)	Minimal overhead for typical use
Scope Atoms	~2-3% per level	Bounded by depth	Captures discarded on exit
Dynamic Atoms (Packrat)	~10-15%	Callback-dependent	Full context access
Dynamic Atoms (Bytecode)	~20-40% (fallback)	Fallback overhead	Uses Packrat fallback internally
Streaming + Captures	~5-10%	O(window)	Bounded memory regardless of file size

📊 Real-World Example: Log File Analysis

Parsing a 10GB log file with captures for IP, timestamp, and status code:

Standard Parsing

~10GB RAM

Entire file in memory

Streaming + Captures

~2MB RAM

Bounded by chunk size

Memory Savings

5000x less

With same capture extraction

Performance Guidelines

✅ Do

• Use captures instead of building full AST when you only need specific fields
• Use scopes for recursive structures to prevent capture accumulation
• Use streaming for files larger than available RAM
• Keep dynamic callbacks fast - avoid I/O operations
• Use Packrat backend for heavy dynamic atom usage

❌ Don't

• Don't capture everything - only what you need
• Don't nest scopes unnecessarily - each adds overhead
• Don't use dynamic atoms for static patterns
• Don't accumulate captures in streaming without processing
• Don't use Bytecode/Streaming backends with many dynamic atoms

Introduction

Parsanol 0.3.0 introduces three powerful new features for extracting and managing parsed data:

Captures - Extract named values from your input
Scopes - Create isolated capture contexts for nested parsing
Dynamic Atoms - Context-sensitive parsing at runtime

Capture Atoms

Capture atoms allow you to extract and name matched portions of your input. Think of them like named groups in regular expressions, but integrated into the parsing grammar.

Basic Capture

use parsanol::portable::{Grammar, Atom, Parser};

use parsanol::portable::parser_dsl::*;

// Match "hello" and capture it as "greeting"
let hello = grammar.add_atom(Atom::Str { pattern: "hello".into() });
let capture = grammar.add_atom(Atom::Capture {
    name: "greeting".into(),
    atom: hello,
});
grammar.root = capture;

let mut parser = Parser::packrat(grammar);
let result = parser.parse("hello world")?;

// Extract the captured value
if let Some(text) = result.get_capture("greeting", "hello world") {
    println!("Greeting: {}", text); // Prints: "hello"
}

Using the DSL

let grammar = GrammarBuilder::new()
    .rule("greeting", capture("greeting", str("hello")))
    .build();

Inspecting Captures

let result = parser.parse(input)?;

// Get all capture names
for name in result.capture_names() {
    println!("Capture: {}", name);
}
// Get specific capture
let value = result.get_capture("name", input)?;
// Get all captures as HashMap
let all = result.captures(input);

Scope Atoms

Scopes create isolated capture contexts. Captures made inside a scope are discarded when the scope exits.

Why Use Scopes?

Prevent capture pollution - Inner parses don't leak captures
Recursive parsing - Each recursion level has its own captures
Nested contexts - Parse sub-structures without affecting outer state

Basic Scope

let mut grammar = Grammar::new();
let hello = grammar.add_atom(Atom::Str { pattern: "hello".into() });
let capture = grammar.add_atom(Atom::Capture {
    name: "greeting".into(),
    atom: hello,
});
grammar.root = capture;

Dynamic Atoms

Dynamic atoms allow you to determine what to parse at runtime. This enables context-sensitive parsing where the grammar itself depends on the input or previously captured values.

When to Use Dynamic Atoms

Configuration-driven parsing - Different grammars for different configs
Context-sensitive keywords - Reserved words change based on position
Incremental parsing - Re-parse with hints from previous parses
Plugin systems - Allow users to inject custom parsers

Basic Dynamic Atom

use parsanol::portable::dynamic::{ConstCallback, register_dynamic_callback};
use parsanol::portable::{Grammar, Atom};

// Create a callback that always returns a specific atom
let callback = ConstCallback::new(
    Atom::Str { pattern: "dynamic".into() },
    "const_dynamic"
);
let callback_id = register_dynamic_callback(Box::new(callback));

// Use in grammar
let mut grammar = Grammar::new();
let dynamic = grammar.add_atom(Atom::Dynamic { callback_id });
grammar.root = dynamic;

Dynamic Context

Callbacks receive a context with current parsing state:

use parsanol::portable::dynamic::{DynamicCallback, DynamicContext};
use parsanol::portable::{Grammar, Atom};

struct KeywordCallback;

impl DynamicCallback for KeywordCallback {
    fn resolve(&self, ctx: &DynamicContext) -> Option<Atom> {
        // Access current position
        let pos = ctx.pos();
        // Access input
        let input = ctx.input();
        // Access captures made so far
        if let Some(lang) = ctx.get_capture("language") {
            match lang {
                "ruby" => Some(Atom::Str { pattern: "def".into() }),
                "python" => Some(Atom::Str { pattern: "lambda".into() }),
                _ => None,
            }
        } else {
            None
        }
    }
}

Backend Compatibility

Feature	Packrat	Bytecode	Streaming
Capture	Full	Full	Full
Scope	Full	Full	Full
Dynamic	Full	Fallback	Fallback

Note: Dynamic atoms in Bytecode and Streaming backends use Packrat fallback internally. This may impact performance if you have many dynamic atoms. Consider using the Packrat backend for heavy dynamic usage.

Error Messages with Captures

Captures can be included in error output for better debugging:

use parsanol::portable::RichError;
use std::collections::HashMap;
let error = parser.parse(input).unwrap_err();
let rich: RichError = error.into_rich(input);
let mut captures = HashMap::new();
captures.insert("variable", "foo");
captures.insert("operator", "+");
println!("{}", rich.format_with_source_and_captures(input, &captures));

Error at line 3, column 5:
    x = foo +
        ^
`- Expected expression after operator
Captures at time of error:
  operator = "+"
  variable = "foo"

Streaming with Captures

💡 What is Streaming?

Streaming means processing input in fixed-size chunks rather than loading the entire file into memory. This is essential for parsing large files (log files, CSV exports, JSON streams) that don't fit in RAM.

Standard Parsing

• Load entire file into memory
• Parse from beginning to end
• Memory = file size

Streaming Parsing

• Read in chunks (e.g., 1MB at a time)
• Process each chunk as it arrives
• Memory = chunk size (bounded)

Captures work with the streaming parser, persisting across chunks:

use parsanol::portable::streaming::{StreamingParser, ChunkConfig};
let result = streaming_parser.parse_from_reader(&mut reader, &mut arena)?;
// Access captures after streaming parse
if let Some(captures) = &result.capture_state {
    for name in captures.names() {
        println!("{} = {:?}", name, captures.get(&name));
    }
}

API Reference

ParseResult Methods

impl ParseResult {
    /// Returns names of all captures
    pub fn capture_names(&self) -> Vec<String>;
    /// Get text of a named capture
    pub fn get_capture<'a>(&self, name: &str, input: &'a str) -> Option<&'a str>;
    /// Get all captures as HashMap
    pub fn captures<'a>(&'a self, input: &'a str) -> HashMap<&'a str, &'a str>;
}

Atom Types

pub enum Atom {
    /// Captures the result of parsing atom under name
    Capture { name: String, atom: usize },
    /// Creates an isolated capture scope for atom
    Scope { atom: usize },
    /// Invokes callback at parse time to determine which atom to use
    Dynamic { callback_id: u64 },
}

DSL Helpers

// Capture helper
capture("name", str("hello"))
// Scope helper
scope(sequence([...]))
// Dynamic helper
dynamic_with_id(callback_id)

Captures, Scopes, and Dynamic Atoms

When to Use Each Feature

Quick Decision Guide

Feature Comparison

Common Use Cases

📧 Email Extraction

⚙️ INI Configuration Parsing

🔧 Type-Driven Value Parsing

📊 Large Log File Analysis

📄 Heredoc / Here-Document

Performance Benchmarks

📊 Real-World Example: Log File Analysis

Performance Guidelines

✅ Do

❌ Don't

Introduction

Capture Atoms

Basic Capture

Using the DSL

Inspecting Captures

Scope Atoms

Why Use Scopes?

Basic Scope

Dynamic Atoms

When to Use Dynamic Atoms

Basic Dynamic Atom

Dynamic Context

Backend Compatibility

Error Messages with Captures

Streaming with Captures

💡 What is Streaming?

API Reference

ParseResult Methods

Atom Types

DSL Helpers

Examples

Capture Atoms

Scope Atoms

Streaming with Captures

Dynamic Atoms