Highly modularized implementation of popular deep RL algorithms by PyTorch. My principal here is to reuse as much components as I can through different algorithms, use as less tricks as I can and switch easily between classical control tasks like CartPole and Atari games with raw pixel inputs.

Implemented algorithms:

• Deep Q-Learning (DQN)
• Double DQN
• Dueling DQN
• Async Advantage Actor Critic (A3C)
• Async One-Step Q-Learning
• Async One-Step Sarsa
• Async N-Step Q-Learning
• Continuous A3C
• Deep Deterministic Policy Gradient (DDPG)
• Hybrid Reward Architecture (HRA)
• Distributed Proximal Policy Optimization (DPPO)

Curves for CartPole are trivial so I didn't place it here.

The network and parameters here are exactly same as the DeepMind Nature paper. Training curve is smoothed by a window of size 100. All the models are trained in a server with Xeon E5-2620 v3 and Titan X. For Breakout, test is triggered every 1000 episodes with 50 repetitions. In total, 16M frames cost about 4 days and 10 hours. For Pong, test is triggered every 10 episodes with no repetition. In total, 4M frames cost about 18 hours.

The network I used here is a smaller network with only 42 * 42 input, alougth the network for DQN can also work here, it's quite slow.

Training of A3C took about 2 hours (16 processes) in a server with two Xeon E5-2620 v3. While other async methods took about 1 day. Those value based async methods do work but I don't know how to make them stable. This is the test curve. Test is triggered in a separate deterministic test process every 50K frames.

For continuous A3C and DPPO, I use fixed unit variance rather than a separate head, so entropy weight is simply set to 0. Of course you can also use another head to output variance. In that case, a good practice is to bound your mean while leave variance unbounded, which is also included in the implementation.

Current DDPG implementation seems to have potential bugs, I'm now actively working on it.

The difference between my implementation and DeepMind's DPPO is:

1. PPO stands for different algorithms.
2. I use a much simpler A3C-like synchronization protocol.

The body of PPO is based on this, however that implementation has some critical bugs.

I use 8 threads and a two tanh hidden layer network, each hidden layer has 64 hidden units.

• Open AI gym
• PyTorch (For some reason I use v0.12 now, although I really like v0.2)
• Python 2.7 (I don't want to try Python 3 until I have to use RoboSchool)
• Tensorflow (Optional, but tensorboard is awesome)

Detailed usage and all training parameters can be found in main.py. And you need to create following directories before running the program:

cd DeepRL
mkdir data log evaluation_log

• Human Level Control through Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Deep Reinforcement Learning with Double Q-learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Playing Atari with Deep Reinforcement Learning
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deterministic Policy Gradient Algorithms
• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Hybrid Reward Architecture for Reinforcement Learning
• Trust Region Policy Optimization
• Proximal Policy Optimization Algorithms
• Emergence of Locomotion Behaviours in Rich Environments
• transedward/pytorch-dqn
• ikostrikov/pytorch-a3c
• ghliu/pytorch-ddpg
• alexis-jacq/Pytorch-DPPO