The mathematical path from policy-gradient foundations to deep RL algorithms. Numbers belong to one shared reading sequence with Reinforcement Learning: the same report has the same number and filename in both repositories. Keep assigned numbers stable and use these indexes to guide readers to future supplementary material.
Start with the RL derivation index for 01–03 (policy evaluation in Sutton notation, Bellman operators, policy improvement), 04–10 (tabular MC and TD methods), and 11 (value-function approximation). Slot 12 is reserved for on-policy control with approximation.
| # | Derivation | Topic | Previous # |
|---|---|---|---|
| 13 | Policy Gradient Theorem | discounted objective and exact gradient | 10 |
| 14 | Average-Reward Policy Gradient Theorem | continuing tasks and stationary distributions | 11 |
| 15 | Policy Gradient Theorem: Episodic Trajectory Route | trajectory likelihoods and reward-to-go | 12 |
| 16 | Policy Gradient Preliminaries | objectives, stochastic gradient estimators, shared estimator decomposition | 13 |
| 17 | REINFORCE | Monte Carlo policy gradient | 14 |
| 18 | Actor-Critic | from the exact gradient to an actor-critic update | 15 |
| 19 | Actor-Critic with a Baseline | baselines, advantages, and TD errors | 16 |
| 20 | GAE Actor-Critic | n-step advantages and their λ-mixture | 17 |
Report 16 uses the results of 13–15 to introduce the estimator notation used by the subsequent algorithms.
| # | Derivation | Previous # |
|---|---|---|
| 21 | Natural Policy Gradient | 18 |
| 22 | Trust Region Policy Optimization | 19 |
| 23 | Proximal Policy Optimization | 20 |
| 24 | Deep Q-Network | 21 |
| 25 | Double DQN | 22 |
| 26 | Deterministic Policy Gradient | 23 |
| 27 | Deep Deterministic Policy Gradient | 24 |
The Previous # column maps the former RL/Deep RL filename numbers to the current sequence. Existing numbers shifted by three when the Sutton-notation foundations were added at the start. Compiled PDFs retain their original text, including historical report-number references. For earlier tabular reports, consult the Previous # column in the RL index. This mapping does not apply to other repositories’ numbering or to pre-existing inconsistent citations within a PDF. Reserved entries have no PDF yet.