
\documentclass[11pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{lmodern}
\usepackage{microtype}
\usepackage{amsmath, amssymb, amsthm, mathtools}
\usepackage[dvipsnames]{xcolor}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{array}
\usepackage{enumitem}
\usepackage{tikz}
\usetikzlibrary{arrows.meta, positioning, fit, shapes.geometric}
\usepackage[most]{tcolorbox}
\usepackage{listings}
\usepackage{caption}
\usepackage{hyperref}
\usepackage[numbers,sort&compress]{natbib}



\hypersetup{colorlinks=true, linkcolor=MidnightBlue, citecolor=MidnightBlue, urlcolor=MidnightBlue}

\graphicspath{{figures/}}


\newcommand{\colsym}[2]{\textcolor{#1}{\textbf{#2}}}

% Math-safe notation macros used throughout the report.
% The symbols are intentionally centralized here so the notation stays consistent.
\newcommand{\ActionSet}{\mathcal{A}}
\newcommand{\ObsSet}{\mathcal{O}}
\newcommand{\ProgramSet}{\mathcal{P}}
\newcommand{\History}{\mathcal{H}}
\newcommand{\action}{\ensuremath{\textcolor{red}{a}}}
\newcommand{\obs}{\ensuremath{\textcolor{ForestGreen}{o}}}
\newcommand{\program}{\ensuremath{\textcolor{blue}{p}}}
\newcommand{\contextsym}{\ensuremath{\textcolor{purple}{c}}}
\newcommand{\gate}{\ensuremath{\textcolor{orange}{\gamma}}}
\newcommand{\gateprob}{\ensuremath{\rho}}
\newcommand{\weight}{\ensuremath{\textcolor{cyan}{w}}}
\newcommand{\history}{\ensuremath{\textcolor{RoyalBlue}{h}}}
\newcommand{\token}{\ensuremath{z}}



\newcommand{\trajectory}{\ensuremath{\textcolor{teal}{\tau}}}
\newcommand{\learnfield}{\ensuremath{\textcolor{RoyalBlue}{G}}}
\newcommand{\advantage}{\ensuremath{\textcolor{Orange}{A}}}
\newcommand{\refpolicy}{\ensuremath{\textcolor{MidnightBlue}{\boldsymbol{\pi}_{\mathrm{ref}}}}}
\newcommand{\klcoef}{\ensuremath{\textcolor{Mahogany}{\beta}}}






%\newcommand{\thetap}{\ensuremath{\theta}}
\newcommand{\thetap}{\ensuremath{\textcolor{blue}{\theta}}}

\newcommand{\sftloss}{\ensuremath{\mathcal{L}}}
\newcommand{\obssft}{\ensuremath{\sftloss^{\mathrm{obs}}}}
\newcommand{\dosft}{\ensuremath{\sftloss^{\mathrm{do}}}}
\newcommand{\truecont}{\ensuremath{y^{\mathrm{true}}}}
\newcommand{\falsecont}{\ensuremath{y^{\mathrm{false}}}}
\newcommand{\deltaagent}{\ensuremath{\Delta}}
\newcommand{\plie}{\ensuremath{p_{\mathrm{lie}}}}

% Stream-agent experiment notation. These colours are deliberately separate from
% the earlier action/observation colours so the STEM section can introduce new
% symbols without overloading the truth--lie notation.
\definecolor{QuestionBlue}{RGB}{36,99,160}
\definecolor{AttemptOrange}{RGB}{199,93,33}
\definecolor{OracleGreen}{RGB}{0,120,90}
\definecolor{VerifierBrown}{RGB}{128,88,0}
\definecolor{RepairPurple}{RGB}{117,72,168}

\definecolor{QuestionPurple}{RGB}{103,58,183}
\definecolor{AnswerTeal}{RGB}{0,121,107}
\definecolor{TeacherGreen}{RGB}{46,125,50}
\definecolor{VerdictOrange}{RGB}{230,126,34}
\definecolor{DataViolet}{RGB}{81,69,172}
\definecolor{MetricBlue}{RGB}{2,119,189}
\definecolor{SelectionPink}{RGB}{194,24,91}
\definecolor{MaskGray}{RGB}{96,96,96}

\newcommand{\question}{\ensuremath{\textcolor{QuestionBlue}{x}}}
\newcommand{\oracle}{\ensuremath{\textcolor{OracleGreen}{y^{\star}}}}
\newcommand{\attemptsym}{\ensuremath{\textcolor{AttemptOrange}{\widehat{y}}}}
\newcommand{\verdictsym}{\ensuremath{\textcolor{VerifierBrown}{v}}}
\newcommand{\repairsym}{\ensuremath{\textcolor{RepairPurple}{r}}}
\newcommand{\sftdataset}{\ensuremath{\mathcal{D}_{\mathrm{SFT}}}}
\newcommand{\agentdataset}{\ensuremath{\mathcal{D}_{\mathrm{agent}}}}
\newcommand{\testdataset}{\ensuremath{\mathcal{D}_{\mathrm{test}}}}
\newcommand{\streamdataset}{\ensuremath{\mathcal{D}_{\mathrm{stream}}}}
\newcommand{\restset}{\ensuremath{\mathcal{D}_{\mathrm{ReST}}}}
\newcommand{\solve}{\ensuremath{\operatorname{Solve}}}
\newcommand{\grade}{\ensuremath{\operatorname{grade}}}
\newcommand{\masktok}{\texttt{[MASK]}}
\newcommand{\losstok}{\texttt{[LOSS]}}


% Correct/wrong stream-outcome symbols used in the data-audit additions.
\definecolor{CorrectTeal}{RGB}{0,121,107}
\definecolor{WrongRust}{RGB}{198,92,33}
\newcommand{\correctattempt}{\ensuremath{\textcolor{CorrectTeal}{\widehat{y}^{+}}}}
\newcommand{\wrongattempt}{\ensuremath{\textcolor{WrongRust}{\widehat{y}^{-}}}}
\newcommand{\okverdict}{\ensuremath{\textcolor{CorrectTeal}{v=1}}}
\newcommand{\badverdict}{\ensuremath{\textcolor{WrongRust}{v=0}}}
\newcommand{\lossbearing}{\ensuremath{\textcolor{RepairPurple}{\ell=1}}}
\newcommand{\contextonly}{\ensuremath{\textcolor{gray}{\ell=0}}}



\newcommand{\reward}{\ensuremath{r}}
\newcommand{\discount}{\ensuremath{\gamma_{\mathrm{RL}}}}
\newcommand{\mixture}{\ensuremath{M}}
\newcommand{\policy}{\ensuremath{\pi}}
\newcommand{\kernel}{\ensuremath{\nu}}
\newcommand{\actchan}{\ensuremath{q}}
\newcommand{\evidchan}{\ensuremath{e}}
\newcommand{\world}{\ensuremath{\mu}}
\newcommand{\prob}{\ensuremath{\mathbb{P}}}
\newcommand{\KL}{\ensuremath{D_{\mathrm{KL}}}}
\newcommand{\E}{\ensuremath{\mathbb{E}}}
\newcommand{\lik}{\ensuremath{L}}
\newcommand{\obslik}{\ensuremath{\lik^{\mathrm{obs}}}}
\newcommand{\dolik}{\ensuremath{\lik^{\mathrm{do}}}}
\newcommand{\obsweight}{\ensuremath{\weight^{\mathrm{obs}}}}
\newcommand{\naiveweight}{\ensuremath{\weight^{\mathrm{naive}}}}
\newcommand{\thirdaction}{\ensuremath{\tilde{\action}}}
\newcommand{\cfaction}{\ensuremath{\dot{\action}}}
\newcommand{\dohistory}{\ensuremath{\widehat{\history}}}
\newcommand{\pairhistory}{\ensuremath{\underline{\action\obs}}}
\newcommand{\doop}{\operatorname{\textcolor{orange}{do}}}
\newcommand{\intervene}[1]{\ensuremath{\doop \left(#1\right)}}
\newcommand{\doaction}{\ensuremath{\intervene{\textcolor{orange}{a}}}}
\newcommand{\trueprog}{\ensuremath{\program_B}}
\newcommand{\trapprog}{\ensuremath{\program_A}}
\newcommand{\fastprog}{\ensuremath{\program_{\mathrm{fast}}}}
\newcommand{\reasonerprog}{\ensuremath{\program_{\mathrm{reasoner}}}}
\newcommand{\codeprog}{\ensuremath{\program_{\mathrm{code}}}}
\newcommand{\engineerprog}{\ensuremath{\program_{\mathrm{engineer}}}}
\newcommand{\Bernoulli}{\mathrm{Bernoulli}}
\newcommand{\dd}{\,\mathrm{d}}

\newcommand{\predtarget}{\ensuremath{\textcolor{MetricBlue}{t}}}
\newcommand{\sideinfo}{\ensuremath{\textcolor{ForestGreen}{x}}}



\newtcolorbox{keybox}[1]{
  colback=blue!3,
  colframe=MidnightBlue!60!black,
  title=\textbf{#1},
  fonttitle=\small,
  breakable,
  sharp corners,
  boxrule=0.6pt
}

\newtcolorbox{warningbox}[1]{
  colback=red!3,
  colframe=red!60!black,
  title=\textbf{#1},
  fonttitle=\small,
  breakable,
  sharp corners,
  boxrule=0.6pt
}

\newtcolorbox{examplebox}[1]{
  colback=ForestGreen!4,
  colframe=ForestGreen!55!black,
  title=\textbf{#1},
  fonttitle=\small,
  breakable,
  sharp corners,
  boxrule=0.6pt
}

\lstdefinestyle{tutorialcode}{
  basicstyle=\ttfamily\footnotesize,
  breaklines=true,
  columns=fullflexible,
  keepspaces=true,
  showstringspaces=false,
  frame=single,
  framerule=0.2pt,
  rulecolor=\color{black!30},
  backgroundcolor=\color{black!2}
}
\lstset{style=tutorialcode}

\title{\textbf{Continual, interactive, causal agents}}

\author{Nando de Freitas and Pedro Ortega}
\date{\today}

\begin{document}
\maketitle

\begin{abstract}
Modern language-model agents are usually built by stacking separate training regimes:
pretraining, mid-training, supervised fine-tuning, preference modeling, rejection sampling,
reinforcement learning, reasoning-specific tuning, self-distillation, and deployment-time patches.
This is intelligence by design and engineering, 
as opposed to emergent intelligence.
The multi-stage recipe is a research local minimum, but it has produced powerful systems. It has no single semantics for an
interaction transcript: user messages, tool outputs, demonstrations, model actions, verifier
judgements, and corrections are often treated as if they were the same kind of evidence. This
paper studies a simpler alternative: a continual, causal interaction stream. The central rule is
that world-written tokens are evidence, whereas self-written agent tokens are interventions. In
LLM fine-tuning this rule becomes a loss mask: keep the agent's own attempts in the context, but
remove them from the supervised target. In a small, reproducible STEM reasoning experiment,
this interventional stream agent reaches held-out solve accuracy, which is comparable to that of 
ReST, GRPO, and SFT with oracle corrections. The result is not a claim of benchmark dominance; it is a
proof of viability for a single continual learning agent that can use interaction, causality, feedback, and
corrections to achieve purposeful and useful behaviour.
\end{abstract}



\section*{Introduction}

Large language models are no longer only passive next-token predictors. They answer questions,
call tools, browse documents, write code, invoke APIs, use verifiers, and receive corrections in
multi-turn loops. Systems such as WebGPT, ReAct, Toolformer, Gorilla, and Voyager make this
agentic pattern explicit: a model acts, the world responds, and the next decision depends on the
history \citep{nakano2022webgpt,yao2022react,schick2023toolformer,patil2023gorilla,wang2023voyager}.
Yet the dominant training story for such systems is still a sequence of loosely connected stages.
First we pretrain on web data; then we continue or mid-train on selected domains; then we run
supervised fine-tuning on instructions; then we collect preferences, fit reward models, use RLHF or
rejection sampling, add reasoning-specific procedures, and finally distill the improved behavior
back into another model \citep{ouyang2022instructgpt,bai2022training,christiano2017deep,gulcehre2023rest,shao2024deepseekmath}.
The pipeline works, but it is conceptually fragmented: many datasets, many objectives, many
interfaces, and many opportunities for a handoff to silently change the meaning of the data.

\begin{figure}[t]
\centering
\includegraphics[width=0.98\textwidth]{cic.png}
\caption{The field of AI is at at steep local minimum: The left-hand pipeline treats pretraining, mid-training, SFT,
RLHF, reasoning tricks, and self-distillation as separate engineered stages with separate data
interfaces. The right-hand picture is the alternative studied here: a single interventional stream in
which the agent receives an observation or question, acts, observes environment feedback, and
learns from externally grounded corrections or outcomes.}
\label{fig:cic-overview}
\end{figure}

Figure~\ref{fig:cic-overview} summarizes the motivation. The left side is the familiar
``Frankenstein'' post-training stack. It can be highly effective, but each stage requires a new
interpretation of what counts as a target: a web token, an instruction answer, a preference label, a
sample selected by a verifier, a reward, a chain-of-thought trace, or a distilled completion. The right
side asks whether we can instead keep one representation throughout deployment and training: an
interaction stream containing questions, demonstrations, agent attempts, verifier outputs, tool
observations, oracle corrections, and ordinary environment feedback. In such a stream, a teacher
solution and an agent solution can look identical as strings. The difference is provenance. A
teacher solution is world-written evidence. The agent's own solution is an action.

This provenance distinction is exactly Pearl's distinction between conditioning and intervention
\citep{pearl1995do,pearl2009causality}. Observing an action-like event $\action$ gives the
conditional distribution $\prob(\obs\mid\action)$: it asks what tends to be true in records where
that event occurred. Setting the action ourselves gives $\prob(\obs\mid\intervene{\action})$: it asks
what follows after the action mechanism has been overwritten. For a deployed agent, its own turns
belong to the second category. The agent should condition on what it did when predicting later
world responses, but it should not update as if its own attempt were independent evidence about
the world. In a causal language-model loss, the translation is simple: keep self-authored tokens in
the input prefix, but mask them out of the target labels. World-written tokens - user messages,
teacher demonstrations, tool outputs, verifier verdicts, and oracle corrections - remain supervised
evidence.

The point matters because modern post-training is already interactive imitation. Behavior cloning
and learning from demonstrations have a long history in control and robotics
\citep{pomerleau1989alvinn,schaal1999imitation,argall2009survey}. Their classic failure mode is
covariate shift: after a learner makes a mistake, it reaches states that were rare under the teacher,
and future errors compound \citep{ross2010efficient,ross2011dagger}. LLM agents face an analogous
problem at the token level. If we fine-tune on transcripts without marking who wrote which tokens,
the model can learn from its own failed completions as if the world had endorsed them. This is the
same self-confirmation pathology studied in causal sequence-model accounts of delusion
\citep{ortega2021shaking}, and it is related to empirical failures such as sycophancy and model
collapse under repeated self-training \citep{perez2022sycophancy,sharma2023sycophancy,shumailov2024model}.

The alternative developed here is a continual, interactive, causal (CIC) objective. It is close in
spirit to interactive imitation and to Ortega's interactional account of agency, in which purposeful
behavior can be learned from the structure of an interaction history rather than from a primitive
scalar reward \citep{ortega2026universal}. It is also consistent with the causal view of decision
making in bandits and reinforcement learning: an arm being pulled by someone else is evidence,
whereas pulling the arm ourselves is an intervention \citep{bareinboim2015bandits,zhang2020causal}.
The contribution here is deliberately practical. We apply the same causal accounting to the
ordinary token-level supervised loss used for LLM fine-tuning.

The experiment is designed to isolate the idea. Starting from the same one-epoch SFT seed of a
Qwen2.5-0.5B model \citep{yang2024qwen25}, we compare SFT, ReST, GRPO, SFT+oracle-corrections,
an observational stream agent, and an interventional stream agent on the same synthetic STEM
reasoning tasks. Every method uses the same verifier and the same held-out solve-accuracy metric.
The key ablation is especially instructive. SFT+oracle-corrections and the interventional stream
receive the same $400$ oracle correction targets and the same $17{,}924$ supervised correction
tokens. The difference is that SFT+oracle-corrections learns a direct map $\question\mapsto\oracle$,
whereas the interventional stream learns a repair map
$(\question,\intervene{\attemptsym},\verdictsym)\mapsto\oracle$: it sees the failed attempt and the
verifier verdict as context, but it does not imitate the failed attempt. This extra repair state is what
the prediction lemma later formalizes, and it is what the results show: $85\%$ held-out solve
accuracy for the interventional stream versus $81\%$ for SFT+oracle-corrections. 

The comparison with the observational stream is the causal lesson in its sharpest form. The
observational and interventional stream agents see the same $785$ rollouts, the same questions, the
same first attempts, the same verifier judgements, and the same $400$ oracle corrections. The
observational stream supervises $51{,}121$ token positions and reaches $61\%$ solve accuracy. The
interventional stream supervises only $17{,}924$ token positions and reaches $85\%$. More supervised
tokens are worse when many of those tokens are the learner's own mistakes. The useful data are not
just the corrections; they are the corrections in the right causal context.

This paper makes four contributions.
\begin{enumerate}[leftmargin=2em,itemsep=2pt]
  \item It gives a tutorial derivation of why an agent's own actions are interventions rather than
  observations, building on do-calculus and structural causal models
  \citep{pearl1995do,pearl2009causality,spirtes2000causation,peters2017elements,dawid2002influence}.
  \item It converts that derivation into a token-level objective for continual interactive causal
  agents: supervise world-written evidence, mask self-authored actions, and condition on both.
  \item It provides a reproducible notebook-style experiment covering dataset construction, stream
  construction, ReST, GRPO, SFT+oracle-corrections, observational stream training,
  interventional stream training, and solve-accuracy evaluation.
  \item It shows that stream agents are a viable alternative to existing verifier-based recipes: in
  this run, the interventional stream is competitive with ReST, stronger than GRPO, and more
  robust than observational stream training while using a simpler supervised objective.
\end{enumerate}

The claim is not that this small STEM experiment settles post-training. It does not. The claim is
more basic: the intervention/evidence distinction gives a single semantics for interaction data. That
semantics can apply before deployment, during post-training, and in continual learning after the
agent is already acting in the world. Instead of treating pretraining, SFT, RLHF, reasoning, and
self-distillation as unrelated rituals, a stream agent asks one question at every token: who wrote
this, and should it be evidence or context?









\section{Causal interactive agents}



\subsection{Bayes under intervention}

Let $\program\in\ProgramSet$ be a latent program, hypothesis, schema, or LLM expert
that can affect both the chosen action and the resulting observation.
Observationally, the joint distribution factors as
\[
  \prob(\program,\action,\obs)
  =
  \prob(\program)\,\prob(\action\mid \program)\,\prob(\obs\mid \program,\action).
\]
If we condition on the event (observation) $\action=\action^\star$, the Bayes posterior is:
\[
  \prob(\program\mid \action^\star, \obs)
  =
  \frac{\prob(\program)\prob(\action^\star\mid \program)\prob(\obs\mid \program,\action^\star)}
       {\sum_{\program'\in\ProgramSet}\prob(\program')\prob(\action^\star\mid \program')\prob(\obs\mid \program',\action^\star)}.
\]
Under an intervention, however, the mechanism $\prob(\action\mid \program)$ is removed and
replaced by a point mass at the imposed value:
\[
  \prob_{\intervene{\action^\star}}(\program,\action,\obs)
  =
  \prob(\program)\,\delta_{\action^\star}(\action)\,\prob(\obs\mid \program,\action),
\]
where
\[
  \delta_{\action^\star}(\action)
  =
  \begin{cases}
  1, & \action=\action^\star,\\
  0, & \action\neq \action^\star.
  \end{cases}
\]
The factor $\prob(\action^\star\mid \program)$ has changed because the agent chose one specific value: $\action=\action^\star$. If a conditional is computed
by normalising this interventional joint, the delta collapses the action sum:
\[
\begin{aligned}
  \prob(\obs\mid \intervene{\action^\star})
  &= \sum_{\program\in\ProgramSet}\sum_{\action\in\ActionSet}
        \prob(\program)\delta_{\action^\star}(\action)\prob(\obs\mid \program,\action)
       \\
  &=
  \sum_{\program\in\ProgramSet} \prob(\program)\prob(\obs\mid \program,\action^\star).
\end{aligned}
\]
When an outcome arrives, the posterior updates through the outcome channel:
\[
  \prob(\program\mid \intervene{\action^\star},\obs)
  =
  \frac{\prob(\program)\prob(\obs\mid \program,\action^\star)}
       {\sum_{\program'\in\ProgramSet}\prob(\program')\prob(\obs\mid \program',\action^\star)}.
\]
\emph{The
action is recorded in the conditioning variables for predicting consequences,
but its probability as an action is not used as evidence.}


\subsection{LLM training}
\label{sec:experiment-setup}


To extend the ideas of Bayesian updates and causal updates to LLMs, we instantiate two fresh copies of a base LLM model, $p_{{\thetap}^{\mathrm{obs}}}$
and $p_{{\thetap}^{\mathrm{do}}}$, and fine-tune each on the \emph{same}
data, with the \emph{same} optimizer. The only difference is which
tokens contribute to the loss:
\begin{itemize}[leftmargin=2em,itemsep=2pt]
  \item \textbf{Observational SFT} ($p_{{\thetap}^{\mathrm{obs}}}$): Every "dialogue"
        token is supervised, including the agent's own turns. This corresponds
        to using the observational likelihood.
  \item \textbf{Interventional SFT} ($p_{{\thetap}^{\mathrm{do}}}$): Only the
        world-written tokens (world turns and structural prefixes) contribute
        to the loss. The agent's own turns are masked. This corresponds to the
        interventional likelihood.
\end{itemize}
Both models still \emph{see} the agent's turns in the input
context. They are only excluded from the gradient.

Let ${\thetap}$ denote the model parameters, $\history$ a tokenized dialogue with
token sequence $\token_1,\ldots,\token_T$, and $\gate_i\in\{0,1\}$ the gate
indicating who wrote slot $i$ ($\gate_i=1$ for the agent, $\gate_i=0$ for the
world). The two losses are:
\begin{align}
\obssft({\thetap})
&=
-\frac{1}{T}\sum_{i=1}^{T}
\log p_{\thetap}(\token_i\mid \token_{<i},\contextsym),
\label{eq:loss-obs}\\
\dosft({\thetap})
&=
-\frac{1}{|\{i:\gate_i=0\}|}\sum_{i:\gate_i=0}
\log p_{\thetap}(\obs_i\mid \token_{<i},\contextsym).
\label{eq:loss-do}
\end{align}
Equation~\eqref{eq:loss-obs} is the standard causal-LM loss, namely the negative log
of the observational likelihood for the dialogue. Equation~\eqref{eq:loss-do}
drops the action-channel factors from the sum, leaving only world-channel
factors. 


\subsection{Interventional vs SFT learning in a stream with the same teacher}
\label{app:conditional-prediction}

In interventional learning, the agent actions are recorded in the conditioning variables for predicting consequences. SFT could use the same teacher as an interventional agent and receive the same feedback. Their losses would be identical, but an interventional agent conditions on its own actions as well. Intevential is a self-improvement method. 

To see this, let us consider a small example with the following 
history:
\[
  \predtarget_1,\action_1,\predtarget_2,\action_2,\predtarget_3,\sideinfo_3,
  \predtarget_4,\sideinfo_4,\ldots
\]
where $\predtarget$ stands for a teacher turn, $\action$ an agent turn, and $\sideinfo$ some environmental turn. 
Suppose that \(\predtarget_4\) is square-integrable.  Define the coarse and
rich information sets
\[
  \mathcal{G}_0
  = \sigma(\predtarget_1,\predtarget_2,\predtarget_3),
  \qquad
  \mathcal{G}_1
  = \sigma(\predtarget_1,\action_1,\predtarget_2,\action_2,\predtarget_3,\sideinfo_3).
\]
Then \(\mathcal{G}_0\subseteq \mathcal{G}_1\).  Let
\[
  m_0 = \E[\predtarget_4\mid \mathcal{G}_0],
  \qquad
  m_1 = \E[\predtarget_4\mid \mathcal{G}_1].
\]
Thus \(m_0\) is the SFT predictor that uses only teaching turns (e.g. answers to questions)
\(\predtarget_1,\predtarget_2,\predtarget_3\), while the interventional agent \(m_1\) also uses its previous actions and any other data
\(\action_1,\action_2,\sideinfo_3\).

\begin{keybox}{Prediction lemma}
For squared-error prediction, the richer predictor is weakly better:
\[
  \E[(\predtarget_4-m_1)^2]
  \le
  \E[(\predtarget_4-m_0)^2].
\]
More precisely,
\[
  \E[(\predtarget_4-m_0)^2]
  =
  \E[(\predtarget_4-m_1)^2]
  +
  \E[(m_1-m_0)^2].
\]
The inequality is strict exactly when the extra variables
\(\action_1,\action_2,\sideinfo_3\) change the conditional mean of
\(\predtarget_4\) with positive probability.
\end{keybox}

\paragraph{Proof.}
By the tower property,
\[
  \E[m_1\mid \mathcal{G}_0]
  =
  \E[\E[\predtarget_4\mid \mathcal{G}_1]\mid \mathcal{G}_0]
  =
  \E[\predtarget_4\mid \mathcal{G}_0]
  = m_0 .
\]
Also \(\E[\predtarget_4-m_1\mid \mathcal{G}_1]=0\).  Since
\(m_1-m_0\) is \(\mathcal{G}_1\)-measurable, the cross term below vanishes:
\[
\begin{aligned}
  \E[(\predtarget_4-m_1)(m_1-m_0)]
  & = \E\!\left[\E[(\predtarget_4-m_1)(m_1-m_0)\mid \mathcal{G}_1]\right] \\
  & = \E\!\left[(m_1-m_0)\E[\predtarget_4-m_1\mid \mathcal{G}_1]\right] \\
  & = 0.
\end{aligned}
\]
Therefore
\[
\begin{aligned}
  \E[(\predtarget_4-m_0)^2]
  &= \E[(\predtarget_4-m_1+m_1-m_0)^2] \\
  &= \E[(\predtarget_4-m_1)^2]
     + \E[(m_1-m_0)^2],
\end{aligned}
\]
which proves the claim.

Of course, for the gap to be meaningful, the previous agent self-actions or side information have to carry useful information. We illustrate this next with a simple question and answer experiment.





\section{Experiment}
\label{sec:stream-stem-agents}

Suppose
we already have a small SFT policy, a deterministic verifier, and a stream of
new quantitative problems. Can we use the agent's own attempts, together with
verifier feedback and oracle corrections, as a training signal without turning
failed attempts into imitation targets?

This matters for continual learning. In deployment, pre-training and
post-training are not really separate regimes. An assistant continually receives
new prompts, calls tools, produces answers, observes tool outputs, and receives
corrections. A reward-based post-training method can be useful, but it introduces
a new interface: scalar rewards, policy ratios, KL penalties, replay buffers,
selection rules, and extra hyperparameters. The interventional stream view offers
a single rule that can operate both before and after deployment: world-written
tokens are evidence, self-written tokens are interventions, and the learner may
condition on its own interventions without treating them as targets. In that
sense, stream agents are a candidate for one continual-learning recipe that works
for both pre-training-style imitation and post-training-style correction.

ReST and GRPO are two common offline and online ways to exploit a verifier. ReST samples many
candidate completions, keeps the completions that pass the verifier, and then
runs another SFT pass on those selected samples. GRPO keeps the online sampling
loop, converts verifier scores within a group into relative advantages, and
applies a policy-gradient update against a reference policy. Both are sensible
recipes. The stream-agent alternative is deliberately more direct. It records an
interaction stream
\begin{equation}
\question \quad\longrightarrow\quad \attemptsym
\quad\longrightarrow\quad \verdictsym
\quad\longrightarrow\quad \oracle,
\label{eq:stream-row}
\end{equation}
where \question{} is the problem, \attemptsym{} is the agent's sampled solution,
\verdictsym{} is the verifier's judgement, and \oracle{} is the oracle correction
when the attempt is wrong. The causal rule is the same as before:
\attemptsym{} is an action, so the agent should read it as
\intervene{\attemptsym}. The verifier and oracle are world-written observations,
so they are evidence.

This experiment puts SFT, ReST, GRPO, an observational stream agent, an
interventional stream agent, and an SFT+oracle-corrections baseline into one
reproducible harness. All trainable methods use the same base model, the same
synthetic STEM generator, and the same deterministic numeric verifier. The
headline comparison is held-out solve accuracy on \testdataset{}. The intended
lesson is not that stream agents get a stronger verifier. They get the same
verifier. The intended lesson is that verifier/correction transcripts can be
used as ordinary supervised data once provenance is marked correctly. The
observational stream supervises the agent's proposal tokens. The interventional
stream keeps those proposal tokens in context but masks them from the loss.


\subsection{Dataset and verifier}
\label{sec:stream-dataset}

Each dataset row is an atomic quantitative exercise with a known formula, a
numeric answer, a unit when appropriate, and a tolerance. We write one row as
\begin{equation}
\question_j = (q_j,\oracle_j,\alpha_j,u_j,\epsilon^{\mathrm{rel}}_j,
\epsilon^{\mathrm{abs}}_j,d_j,s_j),
\label{eq:stem-item}
\end{equation}
where $q_j$ is the natural-language question, $\oracle_j$ is the worked oracle
solution, $\alpha_j$ is the target numeric answer, $u_j$ is the target unit,
$d_j$ is the domain, and $s_j$ is the skill. The experiment uses three domains
(physics, chemistry, and materials) and ten skills: Ohm's law voltage, Ohm's law
current, electric power, density, specific heat, molarity, pH, kinetic energy,
engineering stress, and first-order Bragg diffraction.

The inspected run uses
\begin{equation}
|\sftdataset| = 200, \qquad |\agentdataset| = 200,
\qquad |\testdataset| = 100.
\label{eq:stem-splits}
\end{equation}
The set \sftdataset{} supplies clean teacher solutions for the initial SFT model.
The set \agentdataset{} is reserved for ReST, GRPO, and stream rollouts. The set
\testdataset{} is held out for evaluation.

A completion $y$ is graded by a deterministic numeric verifier. The verifier
first tries to extract the number after a \texttt{Final answer:} marker. If that
marker is missing, it falls back to the last plausible answer number in the
completion. The solve verdict is
\begin{equation}
\solve(\question_j,y)
= \mathbf{1}\!\left\{
  |\widehat{\alpha}(y)-\alpha_j|
  \leq \max\{\epsilon^{\mathrm{abs}}_j,
        \epsilon^{\mathrm{rel}}_j |\alpha_j|\}
  \right\},
\label{eq:solve-metric}
\end{equation}
with an additional unit check when the model explicitly supplies a unit. We use
solve accuracy because it is a common metric for SFT, ReST, GRPO, and the
stream agents.

\begin{examplebox}{Representative SFT teacher row}
\begin{lstlisting}
Question:
A strong acid solution has [H+] = 1.0e-04 M. What is the pH?
Proposed solution:
pH = -log10([H+]) = -log10(1.0e-04) = 4.000. Final answer: 4.000.
\end{lstlisting}
\end{examplebox}

This is ordinary supervised data. The question and protocol prefix are context.
The oracle solution is the supervised continuation. 

\subsection{The single stream approach}

We create a stream that starts from the same SFT-1 single-epoch checkpoint. Without loss of generality, the assumption is the agent has started observing the world passively. We do this for simplicity of implementation only.

In the stream, we enable the agent to interact. For instance, imagine an agent browsing the web. When it finds a question with a solution, it attempts to answer the question. If it gets it correctly, it continues. If it gets it incorrectly, it gets told what the correct solution is because the solution (oracle) is available in this case. The only requirement is an evaluator, but it is well known in Computer Science that building an evaluator is far easier than building a solving agent. 

When the agent is correct, the data looks like the following:


\begin{examplebox}{Correct first attempts: what the two stream agents see}
\begin{lstlisting}
Interventional: correct first attempt, but DO NOT train on self-actions (MASK them)
[MASK] Question:
[MASK] 12.5 g of solute (molar mass 98.0 g/mol) is dissolved to make 1.0 L of solution. What is the molarity?
[MASK] Proposed solution:
[MASK] n = 12.5/98.0 = 0.125 mol; M = n/V = 0.125 M. Final answer: 0.125 M.
[MASK] <eos>

Observational: the same correct rollout is an imitation target (no MASK for solution)
[MASK] Question:
[MASK] 12.5 g of solute (molar mass 98.0 g/mol) is dissolved to make 1.0 L of solution. What is the molarity?
[MASK] Proposed solution:
[LOSS] n = 12.5/98.0 = 0.125 mol; M = n/V = 0.125 M. Final answer: 0.125 M.
[LOSS] <eos>
\end{lstlisting}
\end{examplebox}

If the agent's attempt fails, the oracle provides the correct solution as follows:

\begin{examplebox}{Stream when the agent is correct}
\begin{lstlisting}
Interventional: wrong first attempt
[MASK] Question:
[MASK] 10.0 g of solute (molar mass 40.0 g/mol) is dissolved to make 0.25 L of solution. What is the molarity?
[MASK] Proposed solution:
[MASK] n = 10.0/40.0 = 0.250 mol; M = n/V = 0.250 M. Final answer: 0.250 M.
[MASK] <eos>
[MASK] Proposed solution:
[LOSS] n = 10.0/40.0 = 0.250 mol; M = n/V = 1.000 M. Final answer: 1.000 M.
[LOSS] <eos>

Observational: the same wrong rollout
[MASK] Question:
[MASK] 10.0 g of solute (molar mass 40.0 g/mol) is dissolved to make 0.25 L of solution. What is the molarity?
[MASK] Proposed solution:
[LOSS] n = 10.0/40.0 = 0.250 mol; M = n/V = 0.250 M. Final answer: 0.250 M.
[LOSS] <eos>
[MASK] Proposed solution:
[LOSS] n = 10.0/40.0 = 0.250 mol; M = n/V = 1.000 M. Final answer: 1.000 M.
[LOSS] <eos>
\end{lstlisting}
\end{examplebox}

The visible transcript is almost the same in both cases. The difference is
provenance. In observational training, the first solution is a supervised target
because it appeared in the transcript. In interventional training, the same
first solution is an intervention by the learner and is therefore masked. This
is the one-line distinction between $\prob(\obs\mid\action)$ and
$\prob(\obs\mid\intervene{\action})$ translated into a causal-LM label tensor.





\subsection{Methods}
\label{sec:stream-methods}

All trainable policies are LoRA adapters on top of the same Qwen2.5-0.5B causal
language model. The plain evaluation prompt is
\begin{lstlisting}
Question:
<question>

Proposed solution:
\end{lstlisting}
The methods differ only in how they turn verifier information into training
data.

\paragraph{Base.}
The base model is evaluated without fine-tuning. It sees none of the synthetic
teacher, rollout, or correction data.

\paragraph{SFT.}
The SFT seed trains on the $200$ clean teacher rows in \sftdataset{}. For a prompt
$\question_j$ and oracle solution $\oracle_j$, the loss is
\begin{equation}
\sftloss_{\mathrm{SFT}}(\thetap)
= -\sum_{(\question_j,\oracle_j)\in \sftdataset}
  \log p_{\thetap}(\oracle_j\mid \question_j).
\label{eq:stream-sft-loss}
\end{equation}
We train for only one epoch on purpose.

\paragraph{SFT + ReST.}
ReST starts from the SFT seed. For each sampled problem it draws a group of
candidate answers
\begin{equation}
\attemptsym_{jk} \sim p_{\thetap_{\mathrm{SFT}}}(\cdot\mid \question_j),
\qquad k=1,\ldots,K,
\label{eq:rest-sampling}
\end{equation}
with $K=8$. It grades them with the numeric verifier, keeps verifier-passing
parseable candidates, deduplicates, and in this run selects at most one clean
candidate per prompt group. The selected set is
\begin{equation}
\restset =
\left\{(\question_j,\attemptsym_{jk}):
\grade(\question_j,\attemptsym_{jk})=1\right\}.
\label{eq:rest-selected-set}
\end{equation}
The notebook samples $400$ candidate completions in $50$ groups, selects $30$
verifier-correct candidates, adds $200$ SFT replay teacher examples, and trains
on $230$ supervised pairs. ReST therefore learns only from sampled successes.
Wrong attempts are rejected rather than used as repair contexts in this simple implementation. 

\paragraph{SFT + GRPO.}
GRPO also starts from the SFT seed, but keeps the sampled group rather than
throwing away the failed samples. For each group it converts verifier rewards
$\reward_{jk}$ into standardized group-relative advantages
\begin{equation}
\advantage_{jk}
= \frac{\reward_{jk}-\frac{1}{K}\sum_{\ell=1}^K\reward_{j\ell}}
       {\sqrt{\frac{1}{K}\sum_{\ell=1}^K
       (\reward_{j\ell}-\bar{\reward}_j)^2}+10^{-6}},
\label{eq:grpo-advantage}
\end{equation}
and applies a clipped policy-gradient update against the old policy, with a KL
penalty to the SFT reference model:
\begin{equation}
\sftloss_{\mathrm{GRPO}}(\thetap)
= -\sum_{j,k}
\min\{\rho_{jk}\advantage_{jk},
      \operatorname{clip}(\rho_{jk},1-\epsilon,1+\epsilon)\advantage_{jk}\}
+ \klcoef\,\widehat{\KL}(p_{\thetap}\,\|\,p_{\thetap_{\mathrm{ref}}}).
\label{eq:grpo-loss}
\end{equation}
In this simple low-scale experiment, GRPO samples $400$ completions in $50$ groups. Twenty-four
of the groups are flat, meaning the group gives no useful relative preference
signal. The optimizer performs $26$ non-flat update steps.

\paragraph{SFT+oracle-corrections.}
This baseline starts from SFT 1 and uses the oracle corrections from failed
SFT-1 rollouts, but converts them back into ordinary direct-SFT rows:
\begin{equation}
\sftloss_{\mathrm{oracle}}(\thetap)
= -\sum_{j:\,\verdictsym_j=0}
  \log p_{\thetap}(\oracle_j\mid \question_j).
\label{eq:oracle-correction-loss}
\end{equation}
It sees the same $400$ oracle correction targets that the interventional stream
uses. The difference is that the wrong first attempt $\attemptsym_j$ is not in the
conditioning context. In the saved output those $400$ correction rows correspond
to $70$ unique correction questions from \agentdataset{}. 

\paragraph{Observational stream agent.}
The observational stream agent starts from SFT 1 and trains on every row in the
stream rollout file. The notebook first asks the frozen SFT-1 model to attempt
items from \agentdataset{} until it has collected $400$ wrong attempts. This takes
$785$ stream turns. The observational objective treats the first attempt as an
ordinary observation and therefore supervises it:
\begin{equation}
\sftloss_{\mathrm{stream}}^{\mathrm{obs}}(\thetap)
= -\sum_{j=1}^{785}\log p_{\thetap}(\attemptsym_j\mid\question_j)
  -\sum_{j:\,\verdictsym_j=0}
  \log p_{\thetap}(\oracle_j\mid\question_j,\attemptsym_j).
\label{eq:stream-obs-loss}
\end{equation}
Correct first attempts produce a supervised first-attempt target. Wrong first
attempts produce two supervised targets: the wrong first attempt and then the
oracle correction. Operationally, the observational dataset supervises
$51{,}121$ of $81{,}131$ token positions ($63.0\%$).

\paragraph{Interventional stream agent.}
The interventional stream agent starts from the same SFT-1 checkpoint and uses
the same $785$ stream turns. It also conditions on the same first attempts. The
only causal difference is the label mask: the first attempt is in the input, but
its labels are set to the ignore value. A correct first attempt therefore gives
no gradient. A wrong first attempt creates a repair example in which the model
sees the first attempt and then learns the oracle correction:
\begin{equation}
\sftloss_{\mathrm{stream}}^{\mathrm{do}}(\thetap)
= -\sum_{j:\,\verdictsym_j=0}
  \log p_{\thetap}(\oracle_j\mid
  \question_j,\intervene{\attemptsym_j},\verdictsym_j).
\label{eq:stream-do-loss}
\end{equation}
The supervised positions are exactly the correction positions: $17{,}924$ of
$81{,}131$ tokens ($22.1\%$). This is why the comparison with observational
training is a clean causal ablation. Both stream agents see the same questions,
the same first attempts, and the same corrections. The observational model
imitates the sampled attempts. The interventional model learns only from the
world-written correction conditioned on those attempts.





Figure~\ref{fig:stream-token-budget} makes this difference visible in token
space. The interventional stream and SFT+oracle-corrections have the same number
of loss-bearing correction tokens, but the interventional stream has many more
masked context tokens because it preserves the first attempt in the transcript.
The observational stream has still more loss-bearing tokens because it trains on
both correct and wrong first attempts, and as we will seen in the results this is the wrong thing to do. More data in this case does not mean better performance. Figure~\ref{fig:stream-first-attempts-by-skill} shows how the SFT-1 rollout policy does on the different questions making up this small dataset.

\begin{figure}[t]
\centering
\includegraphics[width=0.7\textwidth]{stream_supervised_token_budget.png}
\caption{Loss-bearing versus context-only token positions in the stream-format datasets. The crucial comparison is interventional stream versus SFT+oracle-corrections: both supervise $17{,}924$ correction tokens, but only the interventional stream keeps the first-attempt context while masking it from the loss.}
\label{fig:stream-token-budget}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.98\textwidth]{stream_first_attempts_by_skill.png}
\caption{Composition of the SFT-1 rollout stream used by both stream agents. The notebook samples from \agentdataset{} until it has $400$ wrong first attempts; in the inspected run this required $785$ turns and also produced $385$ correct first attempts. Correct first attempts are plentiful, but interventional training keeps them as context rather than targets.}
\label{fig:stream-first-attempts-by-skill}
\end{figure}



\subsection{Solve-accuracy comparison}
\label{sec:stream-results}

Figure~\ref{fig:stream-solve-accuracy}
summarizes the held-out solve accuracy on $100$ test problems. The base model
solves $42\%$ of the questions. The one-epoch SFT seed raises this to $63\%$.
ReST reaches $84\%$, and GRPO reaches $75\%$. The observational stream agent
falls to $61\%$, despite receiving more supervised tokens than the
interventional stream agent. The interventional stream agent reaches $85\%$, the
best solve accuracy in the inspected run.

\begin{figure}[h!]
\centering
\includegraphics[width=0.98\textwidth]{stream_solve_accuracy.png}
\caption{Solve accuracy by training method. Error bars are Wilson confidence intervals for $n=100$ held-out questions.}
\label{fig:stream-solve-accuracy}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.75\textwidth]{stream_matched_repair.png}
\caption{Stream-matched comparison. The first-attempt stream is generated by the frozen SFT-1 policy for all three rows. Interventional training improves the cases where the first attempt was wrong because it preserves the repair state instead of collapsing the correction back into an ordinary SFT example.}
\label{fig:stream-matched-repair}
\end{figure}

The strongest comparison is not just the top-line $85\%$ versus $84\%$. It is
that the interventional stream objective reaches this performance with ordinary
supervised learning over a causal transcript. There is no rejection threshold to
choose, no policy ratio, no clipped advantage, and no KL coefficient to tune.
The method's price is different: It needs a correction or informative world
response, not merely a scalar reward.


\subsection{Why interventional beats SFT+oracle-corrections with the same correction targets}
\label{sec:stream-why-interventional}

The comparison with SFT+oracle-corrections is the most educational one. Both
methods start from SFT 1. Both receive the same $400$ oracle correction targets,
with the same $17{,}924$ supervised correction tokens. Yet the interventional
stream reaches $85\%$ solve accuracy, while SFT+oracle-corrections reaches
$81\%$. On the $37$ held-out examples where the frozen SFT-1 first attempt is
wrong, interventional solves $59.46\%$ of the repair cases, while
SFT+oracle-corrections solves $48.65\%$.

The results of Figure~\ref{fig:stream-matched-repair}  agree with the theory provided by our Prediction Lemma. 
 SFT+oracle-corrections learns a direct map
$\question\mapsto\oracle$. The interventional stream learns a repair map
$(\question,\intervene{\attemptsym},\verdictsym)\mapsto\oracle$. The second map
has the information needed to diagnose the first error.



This also clarifies why interventional does better than observational stream
training in the same notebook. Observational training has the repair context,
but it also trains on the wrong attempt as a target. Interventional training has
the repair context without turning the wrong attempt into evidence. That is the
causal advantage.

\subsection{Analysis}
\label{sec:stream-analysis}

The results teach four lessons.

\begin{enumerate}[leftmargin=2em,itemsep=2pt]
  \item \textbf{Stream agents are viable verifier-based post-training methods.}
  The interventional stream agent reaches $85\%$ solve accuracy, compared with
  $84\%$ for ReST and $75\%$ for GRPO in this run. The intervals overlap with
  ReST, so the correct claim is not strict dominance. The correct claim is that
  a causal stream objective belongs in the same toolkit.

  \item \textbf{The intervention mask matters when the stream contains bad
  actions.} Observational and interventional stream training use the same
  $785$ rollouts and the same $400$ corrections. Observational supervises
  $51{,}121$ token positions and reaches $61\%$. Interventional supervises only
  $17{,}924$ correction positions and reaches $85\%$. More supervised tokens are
  not better if many of those tokens are the learner's own mistakes.

  \item \textbf{Correction context matters.} SFT+oracle-corrections receives the
  same oracle targets as interventional stream training, but it discards the
  first attempt. The stream model keeps the first attempt in context and can
  learn a conditional repair map. 

  \item \textbf{Verifier quality and world-side data design still matter.} The
  interventional loss removes self-action likelihood factors, but it cannot make
  poor world evidence good. ReST does very well on molarity in this run because
  it selects many verifier-passing sampled answers. The stream method needs the
  world to supply informative corrections, not merely noisy feedback.
\end{enumerate}

The broader lesson is methodological. ReST, GRPO, and stream agents all start
from the same practical ingredients: a policy, a verifier, and new tasks. ReST
turns the verifier into a selection filter. GRPO turns it into a reward. The
stream agent turns it into an observation in an interaction history. That last
move is closest to the causal account: Actions stay in the history, but only world-written observations update the model.


\section{Discussion and the need for further research}
\label{sec:experiment-discussion}
A few things, in order of generality.
\begin{enumerate}[leftmargin=2em,itemsep=2pt]
  \item The distinction between $\prob(\obs\mid\action)$ and
        $\prob(\obs\mid\intervene{\action})$ is not just relevant to formal
        Bayesian agents with discrete hypothesis classes. It survives the
        translation to LLM (and Omni) fine-tuning, where it becomes the distinction
        between supervising agent tokens and masking them.
  \item Pipelines that train on conversational transcripts as if every token
        were evidence are committing self-confirmation errors, with measurable
        consequences for what the model says afterwards. \emph{This is not
        hypothetical: standard SFT recipes for chat models, web-agent
        traces, and tool-use logs frequently train on the agent's own
        outputs without distinction.}
  \item The fix is essentially free. Masking agent tokens from the loss
        requires no extra data, no extra compute, no architectural change,
        and no auxiliary objective. The one-line change recovers the
        intervention/evidence distinction at training time.
  \item World-side data design matters because the intervention rule
        is necessary but not sufficient. We believe this is only the beginning of the road toward a principled approach to post-train agentic LLMs.
\end{enumerate}

Some caveats are due. The experiment is still small. The
STEM stream experiment is a single run with a tiny $0.5$B base model, $200$ SFT items, $200$ agent items, and $100$ held-out questions. The solve-accuracy intervals for ReST and the interventional stream overlap, so the result should be read as evidence of viability and competitiveness rather than a final benchmark claim. Results would likely vary with hyper-parameter search. This experiment and research note should be taken as a starting point for more investigation into intelligence emergence, as opposed to intelligence design in the current multi-stage approach.














\bibliographystyle{plainnat}
\bibliography{agency_imitation}

\end{document}
