Importance Sampling in Reinforcement Learning - An overview

Thanks Sutton and Barto for their great work of Reinforcement Learning: An Introduction.

Almost all off-policy reinforcement learning methods utilize importance sampling, a general technique for estimating expected values under one distribution given samples from another. We apply it by weighting returns according to the relative probability of their trajectories occurring under the target and behavior policies, called the importance-sampling ratio.

Given a set of trajectory $S_{t}, A_{t}, S_{t+1}, A_{t+1}, \dots, S_{T}$ under policy $\pi$, i.e.

$$\prod_{k=t}^{T-1}\pi(A_{k}|S_{k})p(S_{k+1}|S_{k}, A_{k})$$ where $p$ here is the state-transition probability function. Thus, the relative probability of the trajectory under the target and behavior policies (the importance-sampling ratio) is
$$\rho_{t}^{T}=\frac{\prod_{k=t}^{T-1}\pi(A_{k}|S_{k})p(S_{k+1}|S_{k}, A_{k})}{\prod_{k=1}^{T-1}\mu(A_{k}|S_{k})p(S_{k+1}|S_{k}, A_{k})}=\frac{\prod_{k=t}^{T-1}\pi(A_{k}|S_{k})}{\prod_{k=1}^{T-1}\mu(A_{k}|S_{k})}$$ Note that the importance sampling ratio depends only on the two policies and not at all on the MDP.

Define $J(s)$ as the set of all time steps in which state $s$ is visited. This is for an every-visit method; for a first-visit method, $J(s)$ would only include time steps that were first visits to $s$ within their episodes. let $T(t)$ denote the first time of termination following time $t$, and $G_t$ denote the return after $t$ up through $T(t)$.

To estimate $v_π (s)$, we simply scale the returns by the ratios and average the results:

$$V(s)=\frac{\sum_{t\in J(s)}\rho_{t}^{T(t)} G_{t}}{|J(s)|}$$
When importance sampling is done as a simple average in this way it is called ordinary importance sampling. An important alternative is weighted importance sampling, which uses a weighted average, defined as:
$$V(s)=\frac{\sum_{t\in J(s)}\rho_{t}^{T(t)}G_{t}}{\sum_{t\in J(s)}\rho_{t}^{T(t)}}$$ or zero if the denominator is zero.

The difference between the two kinds of importance sampling is expressed in their biases and variances. The ordinary importance-sampling estimator is unbiased whereas the weighted importance-sampling estimator is biased (the bias converges asymptotically to zero). On the other hand, the variance of the ordinary importance-sampling estimator is in general unbounded because the variance of the ratios can be unbounded, whereas in the weighted estimator the largest weight on any single return is one.