Deep-Reinforcement-Learning

Mathematical derivations

The mathematical path from policy-gradient foundations to deep RL algorithms. Numbers belong to one shared reading sequence with Reinforcement Learning: the same report has the same number and filename in both repositories. Keep assigned numbers stable and use these indexes to guide readers to future supplementary material.

Prerequisites in Reinforcement Learning

Start with the RL derivation index for 01–03 (policy evaluation in Sutton notation, Bellman operators, policy improvement), 04–10 (tabular MC and TD methods), and 11 (value-function approximation). Slot 12 is reserved for on-policy control with approximation.

Shared policy-gradient and actor-critic foundations

# Derivation Topic Previous #
13 Policy Gradient Theorem discounted objective and exact gradient 10
14 Average-Reward Policy Gradient Theorem continuing tasks and stationary distributions 11
15 Policy Gradient Theorem: Episodic Trajectory Route trajectory likelihoods and reward-to-go 12
16 Policy Gradient Preliminaries objectives, stochastic gradient estimators, shared estimator decomposition 13
17 REINFORCE Monte Carlo policy gradient 14
18 Actor-Critic from the exact gradient to an actor-critic update 15
19 Actor-Critic with a Baseline baselines, advantages, and TD errors 16
20 GAE Actor-Critic n-step advantages and their λ-mixture 17

Report 16 uses the results of 13–15 to introduce the estimator notation used by the subsequent algorithms.

Policy optimization and deep off-policy learning

# Derivation Previous #
21 Natural Policy Gradient 18
22 Trust Region Policy Optimization 19
23 Proximal Policy Optimization 20
24 Deep Q-Network 21
25 Double DQN 22
26 Deterministic Policy Gradient 23
27 Deep Deterministic Policy Gradient 24

Earlier report numbers

The Previous # column maps the former RL/Deep RL filename numbers to the current sequence. Existing numbers shifted by three when the Sutton-notation foundations were added at the start. Compiled PDFs retain their original text, including historical report-number references. For earlier tabular reports, consult the Previous # column in the RL index. This mapping does not apply to other repositories’ numbering or to pre-existing inconsistent citations within a PDF. Reserved entries have no PDF yet.