Perches as information interfaces

Draft — not for circulation.

Building on Chris’ reply, here’s the central point I want to emphasize. I suspect we’re in broad agreement on the math; the remaining confusion is mostly rhetorical (what we mean by “perch” — and what we do not mean).

What is a perch?

When I say “perch”, I am not talking about the endpoints of numerical subroutines. A perch is an information interface of a Bellman stage: it is associated with an information set (a \(\sigma\)-algebra / filtration), and it packages the primitives that live there (state/observations/controls/value objects). In that sense, perch indices do two jobs: they tell you which primitives exist (and hence what subproblem is being posed), and they encode what is measurable/adapted there.

So writing z (unmarked, at the decision perch) is shorthand for the measurability/adaptedness claim: \(z\) is \(\mathcal{F}_{\text{dcsn}}\)-measurable (and it is a primitive that lives at the decision perch). The slogan “perches do not proliferate” is not a claim about how many intermediate objects a solver computes; it is a claim about which information interfaces we take as primitive in the semantics of a Bellman problem.

Formally, the filtration nesting is:

\[ \mathcal{F}_{\text{arvl}} \subseteq \mathcal{F}_{\text{dcsn}} \subseteq \mathcal{F}_{\text{cntn}}. \]

The linchpin is the usual Markov/sufficiency requirement:

The decision information set must be sufficient for the continuation state transition (conditional on all the shocks in the stage).

If something affects the conditional law of the continuation state but is not in the decision information set, then you have silently left the standard Bellman/MDP world — unless you enlarge the state (belief states / POMDP machinery), or you rewrite the stage so the information interface is explicit.

Put differently: the stage equations must be written so that the mapping exists — the primitives at the decision perch are \(\mathcal{F}_{\text{dcsn}}\)-measurable, and the continuation transition is a measurable map/kernel out of the decision information + shock information.

Multiple “cut-points” within a stage

Let me use an example rather than abstraction (this is basically the “interest rate vs exchange rate” discussion from the Zoom). I will index the within-stage perches by \(-1,0,1\) (arvl/dcsn/cntn). I’m using \(-1,0,1\) purely as a convention to relate this to “Dolo time”; the point is about arvl/dcsn/cntn perches within a stage.

Suppose I learn an interest rate before I make my spending decision, but an exchange rate is realized only after that decision — and the exchange rate is correlated with the interest rate. Consider the following notation:

• Let \(k_{-1}\) denote the arrival bank balance (before the interest credit posts).
• Let \(r_{0}\) denote the interest rate revealed before the spending decision (known at decision time).
• Let \(p_{1}\) denote the exchange rate revealed after the spending decision (unknown at decision time), correlated with \(r_{0}\).
• Let \(w_{0}\) denote the USD wage amount (known when I decide; the exchange-rate conversion happens later via \(p_{1}\)).
• Let \(c_{0}\) denote the consumption/spending decision.
• Let \(b_{1}\) denote the end-of-stage bank balance (continuation object).

Decision information must include \(r_{0}\) because it changes the conditional distribution of \(p_{1}\).

One way to write the stage is as follows. First, the arrival-to-decision transition (and the decision state) is:

\[ a_0 = (1+r_0)\,k_{-1},\qquad x_{0} \equiv (a_0, r_0). \]

Next, the decision-to-continuation transition is:

\[ b_{1} = a_0 - c_0 + w_0\,p_{1}. \]

The Bellman equation on the decision perch is:

\[ V_{0}(a_0,r_0) := \max_{c_0}\Big\{u(c_0) + \mathbb{E}\big[\,V_{1}(b_{1}) \,\big|\, r_0\big]\Big\}. \]

Note also that we can (if we wish) define the conditional \(Q_{0}\) object by:

\[ Q_{0}(a,r,c) := u(c) + \mathbb{E}_{p\,|\,r}\Big[V_{1}\big(a - c + w_{0}\,p\big)\Big]. \]

so that

\[ V_{0}(a,r) = \max_{c}\,Q_{0}(a,r,c). \]

Now, the main point: the expression above is a composition of operators — the transition/kernel, the conditional expectation, and the maximization (and \(Q_{0}\) is just the usual intermediate maximand, i.e. a function). You can compute/approximate \(Q_{0}\) in many ways (nested expectations, sequential quadrature, regression, interpolation, etc.). You can compute \(Q_{0}\) implicitly “on the fly” inside the maximization routine, or you can pre-compute/tabulate it, store it, and then maximize. Either way, it is still a decision-perch object: it is defined at and measurable with respect to the decision perch, and it constitutes (either implicitly or explicitly) the cntn → dcsn backward operation (the mover).

On Matt’s “cut-points/moments” point: I agree there are many potential cut points within a stage where one could construct intermediate value objects, especially when multiple shocks arrive in succession and you want to integrate them one-by-one. My only claim is that those intermediate numerical objects are not perches by default if one defines them within a single Bellman operator.

For instance, we can let \(w_{0}\) be a vector of currencies and \(p_{1}\) be a vector of exchange rates — we may want to integrate them one by one or all at once. If we keep the stage definition fixed, the reason each currency shock is not a perch is that all decisions are conditioned on \((a_{0}, r_{0})\), so that is the decision perch. The next perch has to be the mathematical object associated with the point after the continuation state transition: an object measurable with respect to everything in the stage, including all the shocks and the decision. So there can be any number of numerical operations between the decision perch and the continuation perch, and any number of stored value functions, but they are not perches.

One could, in principle, split this into multiple stages and define a continuation value function indexed by a partial shock realization (say, the USD component) — that is fine, but then one has refined the information interface and one is redefining the consumption stage. One can then have a subsequent thin stage integrating over non-USD exchange-rate realizations (with no decision). This is fine in principle; there may be reasons to do this. However, most interesting stages will not be so thin: they have a non-trivial decision-information structure where the above discipline pays off.

We see Bellman operators as the primitive objects defining dynamic problems across almost all of economics and the decision-making literature.

I would rather be agnostic in the DDSL about additional numerical “cut points” within stages — not treating them as “more perches within one stage” (otherwise we’ve left the scope of stage syntax as Bellman syntax). There are many numerical choices, and they do not seem to have any overarching symmetry. If one writes out a Bellman operator, it can be as high or low dimensional as one wishes, and then the internal numerical cut points are a job for the solver, not the DDSL, if the job of the DDSL is to describe the structure of the Bellman operator that a modeller chooses to write down.

I hope this clarifies the intended scope: the proposal is not trying to constrain numerics; it is trying to make the information interfaces and the measurability/Markov requirements within a stage explicit and fully consistent with MDP/Bellman theory.

One last scope point. If one gives up the scope that we are designing semantics/syntax to represent Bellman (recursive) problems, then fine — one has the justification to move away from the three-perch set-up and toward a more general language for arbitrary stochastic processes. But I do want to flag that (in my view) “everything ends up being recursive” once you pick the right state / information interface — see Section 2.4, Theorem 2.2 of this note on SSRN. Even if one is grappling with a complicated “quantum many worlds”, you can always push the complexity into the state and recover a recursive operator.

But yes: the price one pays to adopt the proposed framework is signing up to a Bellman operator.

Akshay

Christopher Carroll and Akshay Shanker — repository in development, please do not cite.