Sample efficiency is a measure of how many samples or interactions with the environment an RL agent needs to learn an effective policy. A sample-efficient RL agent can learn from fewer interactions, which is important in settings where data collection is expensive or time-consuming.
Importance sampling is a technique that can be used to improve sample efficiency in RL. The basic idea is to use the data collected from one policy (the "behavior policy") to estimate the value of another policy (the "target policy"), which may be the one we actually want to optimize.
To do this, we first collect a set of trajectories using the behavior policy. We can then compute the importance weights of each transition, which quantify the relative probability of that transition under the behavior policy and the target policy. Specifically, for a transition (s, a, r, s') and a target policy π, the importance weight w(s, a, π) is given by:
w(s, a, π) = π(a|s) / b(a|s)
where π(a|s) is the probability of taking action a in state s under the target policy, and b(a|s) is the probability of taking action a in state s under the behavior policy.
