Given belowModel Overview, click on the right to run the code online.

Model Overview

1. Model overview:

DQN is an improved value-function method based on Q-learning, used to estimate the expected future reward of a state-action pair. Q-learning uses the Q-value function Q(s,a) to represent the expected cumulative reward from taking action a in state s. DQN approximates this function with a deep neural network (the Q-network). The network takes state s as input and outputs Q-values 𝑄(𝑠,𝑎;𝜃) for all possible actions, where 𝜃 denotes the network parameters. To stabilize training, DQN introduces two key techniques:

(1) Experience Replay: As the agent interacts with the environment, it generates experience data consisting of states, actions, rewards, and next states. These data are stored in a fixed-size replay buffer. During training, DQN randomly samples a mini-batch of experiences, breaking their temporal correlation and improving training stability.

(2) Target Network: DQN maintains two neural networks: a current Q-network for action selection and Q-value updates, and a target Q-network for computing target Q-values. The target network parameters are copied from the current Q-network at fixed intervals, avoiding instability caused by frequent parameter updates.

2. Implementation:

• Initialization:
  • Initialize the current Q-network and target Q-network with randomly initialized parameters.
  • Initialize an empty experience replay buffer.
• Environment interaction:
  • At each time step t, the agent selects an action according to the current Q-network output and an ε-greedy policy: with probability ε it selects a random action for exploration, and with probability 1 − ε it selects the action with the highest Q-value for exploitation.
  • Execute action a and obtain reward r and the next state s′.
  • Store the experience (s, a, r, s′) in the replay buffer.
• Experience replay:
  • Randomly sample a mini-batch of experiences (s, a, r, s′) from the replay buffer for training.
  • For each experience, compute the target Q-value y.
  • Here, 𝑄(𝑠′,𝑎′;𝜃′) is the Q-value provided by the target network, and 𝜃′ denotes its parameters.
• Update the Q-network:
  • Use gradient descent to minimize the mean squared error (MSE) between the current Q-network output and the target Q-value.
  • Compute the loss and update the current Q-network parameters 𝜃.
• Update the target network:
  • At fixed intervals, copy the current Q-network parameters 𝜃 to the target network parameters 𝜃′.
• Repeat:
  • Repeat the preceding steps until the maximum number of training steps or another termination condition is reached.