7. Execution Pipeline

This section describes the complete execution pipeline that transforms DDSL models from specification to numerical results, with explicit attention to the ordering of methodization before calibration and the threading of Υ and ρ through each stage.

7.1 Pipeline Overview

The DDSL execution pipeline consists of eight stages:

YAML Files → Parse → Validate → Υ (math) → Methodize → Calibrate → ρ (represent) → Solve → Analyze

Each stage transforms the model representation:

Stage	Input	Output	Map Used
1. Parse	YAML text	Syntactic objects	—
2. Validate	Syntactic objects	Validated syntax	—
3. Mathematical meaning	Validated syntax	Mathematical model	Υ
4. Methodize	Validated syntax	Methodized stage	M
5. Calibrate	Methodized stage	Calibrated stage	C
6. Represent	Calibrated stage	Computational objects	ρ
7. Solve	Computational objects	Numerical solution	—
8. Analyze	Numerical solution	Economic insights	—

Critical ordering: Methodization (stage 4) must occur before calibration (stage 5).

7.2 Stage 1: Parsing

YAML Parser

The parser transforms YAML files into syntactic objects:

def parse_ddsl(yaml_file):
    with open(yaml_file, 'r') as f:
        raw = yaml.safe_load(f)

    # Parse stage metadata
    stage_meta = parse_stage_meta(raw['stage'])

    # Parse symbols block (including operator registry references)
    symbols = parse_symbols(raw['symbols'])

    # Parse perches with mover definitions
    perches = parse_perches(raw['perches'])

    # Parse equations
    equations = parse_equations(raw['equations'])

    return SyntacticStage(stage_meta, symbols, perches, equations)

Operator Registry Recognition

During parsing, operator usages are identified and linked to the registry:

def parse_mover_body(body_text):
    """Parse mover @via body, identifying registry operator usages."""
    operators_used = []

    # Find expectation operators: E_w, E_ε, etc.
    for match in re.finditer(r'E_(\w+)\[', body_text):
        operators_used.append(('expectation', match.group(1)))

    # Find optimization operators: argmax_{...}, max_{...}
    for match in re.finditer(r'argmax_\{([^}]+)\}', body_text):
        operators_used.append(('argmax', match.group(1)))

    # Find draw operators: x ~ D
    for match in re.finditer(r'(\w+)\s*~\s*(\w+)', body_text):
        operators_used.append(('draw', match.group(1), match.group(2)))

    return operators_used

7.3 Stage 2: Validation

Syntactic Validation

Check that all symbols are properly declared and all operator usages are valid:

def validate_syntax(stage):
    errors = []

    # Check all referenced symbols are declared
    for eq in stage.equations:
        for symbol in extract_symbols(eq):
            if not is_declared(symbol, stage.symbols):
                errors.append(f"Undefined symbol: {symbol}")

    # Check type consistency
    for func in stage.functions:
        if not check_type_consistency(func, stage.spaces):
            errors.append(f"Type error in function: {func.name}")

    # Check operator registry coverage
    for mover in stage.movers:
        for op in mover.operators_used:
            if not is_registered_operator(op):
                errors.append(f"Unknown operator in {mover.name}: {op}")

    # Check perch connectivity
    for perch in stage.perches:
        if not check_perch_connectivity(perch, stage.movers):
            errors.append(f"Disconnected perch: {perch.name}")

    if errors:
        raise ValidationError(errors)

    return ValidatedStage(stage)

7.4 Stage 3: Mathematical Meaning (Υ)

Building the Mathematical Model

At this stage, we construct the mathematical interpretation of the model:

def build_mathematical_model(validated_stage, param_values=None):
    """
    Apply Υ to get the mathematical model.

    Parameters can be provided or left symbolic.
    Methods and settings are NOT needed at this stage.
    """
    math_model = MathematicalModel()

    # Map spaces to mathematical spaces
    for space in validated_stage.spaces:
        math_model.spaces[space.name] = upsilon_space(space)
        # e.g., Υ(Xa @def R+) = ℝ₊

    # Map functions to mathematical functions
    for func in validated_stage.functions:
        math_model.functions[func.name] = upsilon_function(func, math_model.spaces)
        # e.g., Υ(g_av: Xa × W -> Xv) = g_av: ℝ₊ × ℝ → ℝ₊

    # Map movers to mathematical operators
    for mover in validated_stage.movers:
        math_model.operators[mover.name] = upsilon_mover(mover, math_model)
        # e.g., Υ(B_arvl) = conditional expectation operator

    # If parameters provided, substitute them
    if param_values:
        math_model = substitute_parameters(math_model, param_values)

    return math_model

Υ for Movers

def upsilon_mover(mover, math_model):
    """
    Map a syntactic mover to its mathematical operator.

    This is the TRUE mathematical operator, not an approximation.
    """
    if mover.type == 'backward':
        if 'E_' in mover.body:
            # Conditional expectation mover
            return ConditionalExpectationOperator(
                conditioning_variable=extract_conditioning_var(mover),
                integrating_variable=extract_shock_var(mover),
                kernel=math_model.functions[mover.kernel]
            )
        elif 'argmax' in mover.body:
            # Bellman optimization mover
            return BellmanMaxOperator(
                action_space=math_model.spaces[mover.action_space],
                constraint_set=math_model.constraints[mover.constraint],
                objective=math_model.functions[mover.q_kernel]
            )
    # ... similar for forward movers

Purpose of Stage 3

This stage is conceptual/theoretical: - Provides the mathematical interpretation for human understanding - Enables theoretical analysis (contraction properties, convergence) - Not required for execution — you can skip to stage 4 if you only want numerical results

7.5 Stage 4: Methodization (M)

Three-File Architecture

Methodization reads from three sources:

File	Purpose
stage.yml	Operator declarations (operators: block)
operation-registry.yml	Available schemes/methods
methodization.yml	Model-specific choices

def methodize_stage(validated_stage, registry, method_config):
    """
    Apply methodization functor M.

    Reads:
      - validated_stage.operators (what operators are used)
      - registry (what schemes exist)
      - method_config (which choices for this model)

    This happens BEFORE calibration.
    The output has schemes chosen but settings still symbolic (settings_profile labels).
    """
    methodized = MethodizedStage(validated_stage)

    # 1. Methodize logical operators (E_w, argmax, etc.)
    for op_config in method_config.logical_operators:
        op = validated_stage.find_operator(op_config.symbol)
        scheme = registry.lookup_scheme(op_config.scheme)
        method = registry.lookup_method(scheme, op_config.method)
        methodized.operator_schemes[op_config.instance] = {
            'scheme': scheme,
            'method': method,
            'settings_profile': op_config.settings_profile
        }

    # 2. Methodize movers (attach APPROX to outputs)
    for mover_config in method_config.movers:
        mover = validated_stage.find_mover(mover_config.mover)
        methodized_mover = methodize_mover(mover, mover_config, registry)
        methodized.movers[mover.name] = methodized_mover

    return methodized

Methodizing a Mover

def methodize_mover(mover, mover_config, registry):
    """
    Attach schemes to a mover from external config.

    Key change: methods come from methodization.yml, NOT from @methods in perches.
    """
    methodized = MethodizedMover(mover)

    # Mover-level scheme/method
    methodized.scheme = mover_config.scheme
    methodized.method = mover_config.method

    # Attach APPROX to each output function
    for approx_config in mover_config.approx:
        methodized.approx_schemes[approx_config.target] = {
            'scheme': approx_config.scheme,
            'method': approx_config.method,
            'settings_profile': approx_config.settings_profile,
            'acts_on_space': approx_config.acts_on_space
        }

    return methodized

Method Configuration File

Methods are configured in a separate file (not in perches):

# consind-methodization.yml
stage: consind

methodization:
  logical_operators:
    - symbol: E_{}
      instance: E_w
      scheme: shock_expectation
      method: !gauss-hermite
      settings_profile: expect_w_settings
      attaches_to:
        mover: dcsn_to_arvl

    - symbol: argmax_{}
      instance: argmax_a
      scheme: statewise_argmax
      method: !golden-section
      settings_profile: argmax_a_settings

  movers:
    - mover: dcsn_to_arvl
      scheme: bellman_backward
      method: !generic
      approx:
        - target: Va
          scheme: interpolation
          method: !linear
          settings_profile: Va_interp_settings
          acts_on_space: Xa

Authoring note (optional): flat bindings form

The same methodization information can be written without nesting as a flat list of bindings (grouping keys are presentation-only):

stage: consind

bindings:
  - operator: E_w         # from syntax `E_{w}(...)` (similarly: `E_{y}(...)` → `E_y`)
    scheme: shock_expectation
    method: !gauss-hermite
    settings:
      n_nodes: n_w_nodes   # name in `symbols.settings`; numeric value in settings.yml

  - operator: argmax_a
    scheme: statewise_argmax
    method: !golden-section
    settings:
      tol: argmax_tol

  - operator: mover:dcsn_to_arvl
    scheme: bellman_backward
    method: !generic

  - operator: approx:Va
    scheme: interpolation
    method: !linear
    settings:
      extrapolation: extrapolation_mode
    acts_on_space: Xa

Key point: The stage YAML contains no @methods annotations. Movers are purely symbolic.

Kernels vs. methods (important): - Methodization does not attach methods to arbitrary “equations” or “kernels” just because they exist. - Methodization attaches schemes/methods to operator instances (e.g., E_w, argmax_a, draw ~, APPROX / interpolation) and to movers as computational units. - In particular, many forward computations are push-forward operators derived from a transition kernel + a shock distribution. The transition kernel is a pure function object; the push-forward (simulation / quadrature / sampling) is what needs a method. - A kernel only becomes method-relevant when you intend to evaluate it under ρ and its evaluation invokes methodized operators (e.g., it calls E_w[...], solves an argmax, evaluates an approximated function off-grid, draws from a distribution, etc.).

APPROX Insertion

Each mover's output functions are wrapped in APPROX:

From methodization config:
    movers:
      - mover: dcsn_to_arvl
        approx:
          - target: Va
            scheme: interpolation
            method: !linear
            settings_profile: Va_interp_settings

After methodization:
    B_arvl produces APPROX(Va, {
        scheme: interpolation,
        method: !linear,
        settings: {extrapolation: extrapolation_mode}  # values come from settings.yml
    })

The settings_profile is just a label — concrete values come from calibration.

7.6 Stage 5: Calibration (C)

Applying Calibration

After methodization, calibration resolves settings_profile labels to concrete values:

def calibrate_stage(methodized_stage, calibration):
    """
    Apply calibration functor C.

    Requires: methodization already done.
    Resolves: settings_profile labels from methodization config.
    """
    calibrated = CalibratedStage(methodized_stage)

    # 1. Apply parameter calibration (economic model)
    for param, value in calibration.parameters.items():
        calibrated.substitute(param, value)
        # e.g., β → 0.96, γ → 2.0

    # 2. Resolve method settings (numerical schemes)
    for profile_name, settings in calibration.method_settings.items():
        calibrated.resolve_settings_profile(profile_name, settings)
        # e.g., expect_w_settings → {nodes: 9, multidim: 'tensor'}
        # e.g., Va_interp_settings → {extrapolation: 'clip'}
        # e.g., Xa_grid_settings → {n_points: 50, bounds: [0.1, 10.0]}

    return calibrated

Calibration Config File

# consind-calibration.yml
stage: consind

# Economic parameters (define the math problem)
parameters:
  β: 0.96
  γ: 2.0
  r: 1.04
  μ_w: 0.0
  σ_w: 0.1

# Method settings (resolve settings_profile labels)
method_settings:
  expect_w_settings:
    nodes: 9
    multidim: tensor

  argmax_a_settings:
    tol: 1.0e-8
    max_iter: 100

  Va_interp_settings:
    extrapolation: clip

  Xa_grid_settings:
    n_points: 50
    bounds: [0.1, 10.0]

Two-Part Calibration

def load_calibration(calibration_file):
    with open(calibration_file, 'r') as f:
        calib = yaml.safe_load(f)

    return Calibration(
        parameters=calib.get('parameters', {}),   # β, γ, r, ...
        settings=calib.get('settings', {})        # n_xa, tol, ...
    )

Validation After Calibration

def validate_calibration(calibrated_stage):
    # Check all parameters have values
    for param in calibrated_stage.all_parameters:
        if not calibrated_stage.has_value(param):
            raise CalibrationError(f"Missing parameter: {param}")

    # Check all settings have values
    for setting in calibrated_stage.all_settings:
        if not calibrated_stage.has_value(setting):
            raise CalibrationError(f"Missing setting: {setting}")

    # Check value ranges
    for param, value in calibrated_stage.param_values.items():
        if not in_declared_range(param, value):
            raise CalibrationError(f"Out of range: {param}={value}")

7.7 Stage 6: Representation (ρ)

Creating Computational Objects

The representation map ρ transforms calibrated objects into an executatable representation:

def represent_stage(calibrated_stage):
    """
    Apply representation map ρ.

    Requires: methodization AND calibration complete.
    Produces: executable numerical objects.
    """
    rho = RepresentationMap()
    numerical_meaning = NumericalMeaningTable()

    # Create grids for spaces
    for space in calibrated_stage.spaces:
        grid = rho.create_grid(space, calibrated_stage)
        numerical_meaning.register(space.name, grid)

    # Create implementations for operators
    for mover in calibrated_stage.movers:
        impl = rho.create_mover_implementation(mover, numerical_meaning)
        numerical_meaning.register(mover.name, impl)

    return ComputationalStage(numerical_meaning)

ρ for the Arrival Mover

End-to-end example for B_arvl:

def rho_arrival_mover(mover, numerical_meaning, calibration):
    """
    ρ(B_arvl) with:
        E_w: !gauss-hermite(n_nodes=5)
        Va: !linear(grid_size=50)
    """
    # Get calibrated values
    n_nodes = calibration.settings['n_nodes']  # 5
    n_xa = calibration.settings['n_xa']        # 50
    mu_w = calibration.params['μ_w']           # 0.0
    sigma_w = calibration.params['σ_w']        # 0.1

    # Create quadrature for E_w
    nodes, weights = gauss_hermite_quadrature(n_nodes, mu_w, sigma_w)

    # Get grids
    grid_xa = numerical_meaning.lookup_grid('Xa')

    # Get transition kernel
    g_av = numerical_meaning.lookup_function('g_av')

    # Create the operator implementation
    def arrival_backward(Vv_interpolator):
        """The actual numerical operator."""
        Va_values = np.zeros(n_xa)

        for i, xa in enumerate(grid_xa):
            # Compute E_w[Vv(g_av(xa, w))] via quadrature
            Va_values[i] = sum(
                w * Vv_interpolator(g_av(xa, node))
                for node, w in zip(nodes, weights)
            )

        # Return as interpolator (APPROX)
        return LinearInterpolator(grid_xa, Va_values)

    return arrival_backward

Conditional Expectation Details

The arrival mover implements a conditional expectation:

\[ V_{\prec}(x_{\prec}) = \mathbb{E}[V_{\sim}(\mathrm{g}_{\prec\sim}(x_{\prec}, w)) \mid x_{\prec}] \]

Key aspects of the ρ implementation: - Conditioning variable: \(x_{\prec}\) (arrival state) — we iterate over grid points - Integration variable: \(w\) (shock) — we sum over quadrature nodes - Kernel: \(\mathrm{g}_{\prec\sim}\) — transforms \((x_{\prec}, w)\) to \(x_{\sim}\)

The @methods annotation specifies: - E_w: !gauss-hermite(n_nodes) — how to approximate the integral - Va: !linear(grid_size) — how to store the result

7.8 Stage 7: Solving

Solver Initialization

def initialize_solver(computational_stage, settings):
    solver = ADCSolver(
        arrival_op=computational_stage.get_operator('B_arvl'),
        decision_op=computational_stage.get_operator('B_dcsn'),
        numerical_meaning=computational_stage.numerical_meaning,
        tolerance=settings['tol'],
        max_iterations=settings['max_iter']
    )

    # Set initial guess
    solver.set_initial_guess(create_initial_guess(computational_stage))

    return solver

Value Function Iteration

def solve_value_iteration(solver):
    converged = False
    iteration = 0

    while not converged and iteration < solver.max_iterations:
        # Store old value function
        Ve_old = solver.continuation_value.copy()

        # Backward pass through movers
        # B_dcsn: Ve → (Vv, astar)
        Vv, astar = solver.decision_op(solver.continuation_value)

        # B_arvl: Vv → Va
        Va = solver.arrival_op(Vv)

        # Link to next period (often identity)
        Ve_new = Va

        # Check convergence
        error = compute_error(Ve_new, Ve_old)
        converged = error < solver.tolerance

        # Update
        solver.update(Va, Vv, Ve_new, astar)
        iteration += 1

        logger.info(f"Iteration {iteration}: error = {error:.6e}")

    return Solution(Va, Vv, Ve_new, astar, converged, iteration)

7.9 Stage 8: Analysis and Export

Computing Statistics

def compute_statistics(solution, simulations):
    return {
        'policy_function': solution.astar.to_dataframe(),
        'value_function': solution.Va.to_dataframe(),
        'convergence_achieved': solution.converged,
        'iterations_required': solution.iterations,
        'mean_consumption': np.mean(simulations.consumption),
        'wealth_distribution': compute_distribution(simulations.wealth),
    }

7.10 Complete Pipeline Summary

The Full Transformation Chain

┌─────────────────────────────────────────────────────────────────────────────┐
│                           DDSL EXECUTION PIPELINE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  YAML Files                                                                 │
│      │                                                                      │
│      ▼ [1. Parse]                                                           │
│  Syntactic Stage S                                                          │
│      │                                                                      │
│      ▼ [2. Validate]                                                        │
│  Validated Stage                                                            │
│      │                                                                      │
│      ├────────────────────► [3. Υ] ────► Mathematical Model                 │
│      │                                   (for theoretical analysis)         │
│      │                                                                      │
│      ▼ [4. Methodize] M                                                     │
│  Methodized Stage (S, M)                                                    │
│      │  • APPROX attached to movers                                         │
│      │  • Schemes chosen for E_w, argmax, ~                                 │
│      │  • Settings still symbolic (n_xa, n_nodes, tol)                      │
│      │                                                                      │
│      ▼ [5. Calibrate] C                                                     │
│  Calibrated Stage (S, M, C)                                                 │
│      │  • Parameters have values (β=0.96, γ=2.0)                            │
│      │  • Settings have values (n_xa=50, n_nodes=5)                         │
│      │                                                                      │
│      ▼ [6. Represent] ρ                                                     │
│  Computational Stage                                                        │
│      │  • Grids created (numpy arrays)                                      │
│      │  • Operators implemented (callables)                                 │
│      │  • Numerical meaning table populated                                 │
│      │                                                                      │
│      ▼ [7. Solve]                                                           │
│  Numerical Solution                                                         │
│      │  • Value functions (interpolators)                                   │
│      │  • Policy functions (interpolators)                                  │
│      │                                                                      │
│      ▼ [8. Analyze]                                                         │
│  Economic Insights                                                          │
│      • Statistics, plots, exports                                           │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Threading Υ and ρ

Stage	Which Map?	What It Does
3	Υ	Syntax → Mathematical operators (true, Platonic)
6	ρ	Calibrated syntax → Numerical operators (approximate)

Key insight: - Υ can be applied early (after validation, possibly with parameters) - ρ requires methodization AND calibration (stages 4-5 must complete first) - The numerical operators under ρ approximate the mathematical operators under Υ

End-to-End Example: Buffer-Stock B_arvl

1. PARSE
   B_arvl: Vv -> Va @via |
       xv = g_av(xa, w)
       Va(xa) = E_w[Vv(xv)]
       @methods: {E_w: expect_w, Va: interp_Va}

2. VALIDATE
   ✓ B_arvl is a valid mover declaration
   ✓ E_w is a registered expectation operator
   ✓ expect_w and interp_Va are declared methods

3. Υ (MATHEMATICAL MEANING)
   Υ(B_arvl) = conditional expectation operator
   (B_arvl V)(xa) = ∫ V(g_av(xa, w)) dμ_w(w)

4. M (METHODIZE)
   M(B_arvl) = (B_arvl, {
       E_w: !gauss-hermite(n_nodes),
       Va: APPROX(!linear(space: Xa, grid_size: n_xa))
   })

5. C (CALIBRATE)
   C(M(B_arvl)) = (B_arvl, {
       E_w: !gauss-hermite(n_nodes=5),
       Va: APPROX(!linear(space: Xa, grid_size=50))
   })
   with μ_w = 0.0, σ_w = 0.1

6. ρ (REPRESENT)
   ρ(C(M(B_arvl))) = Python function:
       def arrival_backward(Vv_interp):
           nodes, weights = gauss_hermite(5, 0.0, 0.1)
           Va_values = zeros(50)
           for i, xa in enumerate(linspace(0.1, 10.0, 50)):
               Va_values[i] = sum(w * Vv_interp(xa * r + node) 
                                  for node, w in zip(nodes, weights))
           return LinearInterpolator(grid_xa, Va_values)

7. SOLVE
   Apply arrival_backward iteratively until convergence

8. ANALYZE
   Extract policy, value functions, statistics

7.11 Summary

The execution pipeline provides:

1. Clear staging: Eight distinct transformation stages
2. Correct ordering: M (methodization) before C (calibration)
3. Dual semantics: Υ for mathematical meaning, ρ for computational
4. Explicit approximation: APPROX operators attached during methodization
5. Operator-driven: Registry operators get implementations through ρ
6. Complete traceability: From YAML to numerical result

The pipeline ensures: - Separation of concerns: Syntax, math, algorithms, numbers are distinct - Reproducibility: Same inputs → same outputs - Flexibility: Can change methods without changing math, change settings without changing methods - Correctness: Numerical solutions approximate mathematical solutions

Christopher Carroll and Akshay Shanker — repository in development, please do not cite.