Given belowModel Overview, click on the rightRun Code Online。

Model Overview

1. Model overview:

MAPPO is a multi-agent reinforcement learning algorithm used to solve collaboration and competition problems in multi-agent environments. Thesis "The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games" (https://arxiv.org/pdf/2103.01955) provides a good description and summary of the MAPPO algorithm. MAPPO is a multi-agent extension of the PPO (Proximal Policy Optimization) algorithm. It inherits the Actor-Critic structure of MAPPO and incorporates the centralized or decentralized structure of multi-agent reinforcement learning.

2. Model Architecture:

This study involves multiple controlled vehicles that cooperate rather than compete. It uses a centralized structure: all agents share an Actor network with identical parameters but receive different observations, while one Critic is used globally.

• The Actor is a policy network that generates actions from the current state. Here it is an MLP that takes one vehicle’s observation and outputs an action probability distribution. All agents share the same parameters and policy network, which suits symmetric tasks where all agents have identical roles and goals.
• The Critic is a value network that evaluates the current policy. This model uses an MLP with multi-head self-attention, taking the global state as input and estimating expected cumulative reward under the current policy. In MAPPO, the Critic provides a baseline for assessing whether Actor policy improvements are effective.
• MultiHead-Attention: Global observations stack state information from all vehicles. The attention mechanism is permutation invariant, so it is unaffected by vehicle ordering in the global observation.
• MAPPO follows the PPO computation process, with changes to the observation space.

3. Implementation:

• environment definition: Relying on highway_env environment, set up multi-agent situations.
• Model definition: Defines classes including Actor, Critic, and MAPPO.
• train: The main content of the project includes initializing the environment, using Actors to calculate actions in each round, collecting round data, and performing network updates and other operations.
• Evaluate: During the training process, an evaluation is performed every several rounds, the average reward of several rounds is calculated as the evaluation result, and the evaluation result change curve is drawn at the same time.
• Visualization: Visualization of vehicle operation results is achieved by adjusting render_mode.