A PyTorch reinforcement learning library in which every algorithm starts as a mathematical derivation and ends as readable, tested code you can train with.
Active development: APIs may change. The examples are runnable recipes, not reproduced benchmarks.
| Algorithm | Code | Derivation | Status |
|---|---|---|---|
| One-step actor-critic | one_step_actor_critic.py | Actor-Critic, with a baseline | ✅ |
| A2C (GAE) | a2c.py | GAE Actor-Critic | ✅ |
| Natural Policy Gradient | npg.py | NPG | ✅ |
| TRPO | trpo.py | TRPO | ✅ |
| PPO-Clip | ppo.py | PPO | ✅ |
| DQN | dqn.py | DQN | ✅ |
| Double DQN | dqn.py (double_q=True) |
Double DQN | ✅ |
| DDPG | — | DPG, DDPG | derivation done |
A2C, NPG, TRPO, and PPO share the on-policy trainer; DQN and Double DQN use the off-policy trainer. One-step actor-critic is a standalone teaching implementation.
Requires Python 3.12+ and uv.
git clone https://github.com/SaiSampathKedari/Deep-Reinforcement-Learning.git
cd Deep-Reinforcement-Learning
uv sync --extra tb --extra video
Train PPO on CartPole-v1 (CPU, 8 environments, 102,400 steps):
uv run python -m examples.gymnasium.cartpole.run_cartpole_ppo \
--output-dir outputs/gymnasium/cartpole/ppo/seed_0/run_01
Evaluate the saved policy, and view the training curves:
uv run python -m examples.gymnasium.cartpole.evaluate_cartpole \
outputs/gymnasium/cartpole/ppo/seed_0/run_01/policy --episodes 10
uv run tensorboard --logdir outputs/gymnasium/cartpole/ppo/seed_0/run_01/events
Each run writes one folder (use a new one per run):
run_01/
├── config.json every setting of the run
├── metrics.jsonl episodes, training reports, evaluations
├── events/ TensorBoard logs
├── checkpoints/ last.pt and periodic step_<N>.pt, for resuming
└── policy/ the trained policy, loadable without any training code
The CartPole guide covers video recording, rendering, and resuming from a checkpoint.
| Task | What it shows |
|---|---|
| CartPole-v1 | Discrete actions; the CPU starting point |
| InvertedPendulum-v5 | A Gaussian policy for continuous control |
| Hopper-v5 | Continuous-action locomotion |
| Humanoid-v5 | High-dimensional control with observation normalization |
Each example keeps its networks, settings, training script, and evaluator together, and calls the shared library. The MuJoCo tasks need one more extra:
uv sync --extra mujoco --extra tb --extra video
| Part | Files | Responsibility |
|---|---|---|
| Learners | algorithms/ | Configs, targets, losses, and parameter updates |
| Trainers | train_on_policy.py, train_off_policy.py | When to collect, learn, evaluate, and save |
| Data | collectors.py, buffers.py, advantages.py | Environment stepping, rollout/replay storage, GAE |
| Networks and processing | policies.py, value.py, processors.py, inference.py | Wrappers for your networks; observation and action processing |
| Evaluation and logging | evaluate.py, logger.py | Episode returns, JSONL metrics, TensorBoard |
| Saving and loading | checkpoint.py, policy_io.py | Training checkpoints and exported policies |
Buffers store raw observations; normalization is applied when a network reads them, with statistics the trainer updates on a fixed schedule. Network architectures stay with each example. The CartPole training script shows the full assembly: environments, networks, learner, logger, evaluator, and the training call.
checkpoints/) hold everything needed to continue training:
networks, optimizers, observation statistics, progress, and random-number
state. Pass --resume-from .../checkpoints/last.pt to continue a run. A
resumed run starts fresh episodes, and off-policy runs refill an empty replay
buffer before learning again.policy/) hold only what acting needs: a manifest, the model
weights, and the observation statistics. They are written when training
finishes, and load without a learner or optimizer.To evaluate a checkpoint before training finishes, export it to a new folder:
from deeprl.policy_io import export_policy_from_checkpoint
export_policy_from_checkpoint(
"outputs/gymnasium/cartpole/ppo/seed_0/run_01/checkpoints/last.pt",
"outputs/gymnasium/cartpole/ppo/seed_0/run_01/policy_snapshot",
)
The full rules are documented in checkpoint.py and policy_io.py.
The derivations are working specifications. They decide what the buffers keep, where gradients stop, and which quantities stay fixed during an update: for example, termination-aware GAE, PPO’s fixed old log-probabilities, and TRPO’s rollback. The DQN derivation splits Q-learning into three independently scheduled steps (data collection, target-network refresh, and Q-function fitting), which is exactly how the off-policy trainer and DQN learner are built.
Browse all derivations. Their numbering continues from the Reinforcement Learning repository: 13–20 cover policy gradients and actor-critic; 21–27 cover NPG, TRPO, PPO, DQN, Double DQN, deterministic policy gradients, and DDPG.
Released under the MIT License.