Researchers from South Korea’s Chung-Ang University are developing a ‘meta-reinforcement’ machine learning algorithm for traffic lights to improve vehicle throughput

No matter how advanced modern vehicles become, including those capable of self-driving, being stuck in heavy city traffic is unlikely to go away anytime soon using traditional headlight technology. traffic to control the flow of vehicles.

But recent advances in artificial intelligence (AI) and machine learning have shown promise in optimizing traffic light control, which can make driving in congested urban areas less frustrating.

Researchers at Chung-Ang University in South Korea are experimenting with reinforcement learning (RL) algorithms to solve non-stationary traffic light control problems. The algorithm automatically customizes “reward features” based on the classification of traffic patterns.

RL uses a “trial and error” problem-solving method for training agents in the field of machine learning.

RL is one of the three basic machine learning paradigms, along with supervised learning and unsupervised learning. It is an area of ​​machine learning that focuses on how intelligent agents (the traffic light) should act in an environment to maximize the reward, i.e. vehicles moving continuously in city traffic scenarios . The goal of agents using RL is to maximize total rewards.

Basically, the reward function of a machine learning algorithm is an incentive that tells the agent what is right and what is wrong using reward and punishment. But often, RL algorithms have to sacrifice immediate rewards (some drivers may get stuck at a red light) in order to maximize total rewards (improved traffic flow).

Existing traffic signals rely on a “rules-based controller” (red means stop and green means go). The goal is to reduce vehicle delays in light traffic conditions and maximize vehicle throughput during periods of traffic congestion.

Sub-optimal traffic light controllers like these affect the daily lives of people living in urban areas who often experience traffic jams and delays. Conventional traffic lights with fixed status times are not well equipped to reduce traffic jams.

Additionally, existing traffic light controllers cannot adapt to ever-changing and random traffic patterns throughout the day. Although a human traffic controller can be more capable than a stationary controller, he can only handle a few intersections at a time.

One of the biggest challenges for the researchers was to implement RL in a non-stationary environment, i.e. vehicles randomly crossing intersections. Current research has explored reinforcement learning (RL) algorithms as a possible solution to alleviate traffic problems. However, RL algorithms do not always achieve the best results due to the dynamic nature of traffic environments.

To help better address this issue, the researchers developed what they call a “meta-RL model” that adjusts its aim based on the traffic environment. The meta-RL algorithm has wide coverage and outperforms existing alternative algorithms, according to researchers from Chung-Ang University.

The two main objectives of the meta-RL machine learning model are to maximize vehicle throughput at intersections during peak hours and to minimize delays during peak hours, such as during rush hour. Researchers led by Professor Keemin Sohn have developed a context-based meta-RL model integrated with the Extended Deep Q Network (EDQN) for traffic signal control.

Sohn is a professor at the School of Civil and Environmental Engineering, Chung Ang University, Korea. His research interests include data science, artificial intelligence with applications in transportation planning.

Here’s how the meta-RL model works. First, it determines traffic as “saturated” or “unsaturated” using a latent variable that indicates the overall environmental condition. Based on the current traffic flow, the model maximizes throughput or minimizes delay like a human controller. It does this by implementing traffic light cycles (action).

The action is controlled by the provision of a “reward”. The reward function is set to +1 or -1 corresponding to better or worse performance in traffic management compared to the previous traffic light interval. Additionally, the EDQN acts as a decoder to jointly control traffic lights for multiple intersections.

“Existing studies have developed meta-RL algorithms based on intersection geometry, traffic signal phases or traffic conditions,” Professor Sohn explained. “The meta-RL works autonomously to detect traffic conditions, classify traffic patterns, and assign signal phases accordingly.”

The researchers trained and tested their meta-RL algorithm using Vissim 21.0, which is a commercial traffic simulator used by engineers to model real-world traffic conditions.

The team set up a transportation network in southwest Seoul consisting of 15 intersections to serve as a real-world test environment. After meta-training, the RL model could adapt to new tasks without adjusting its parameters.

These experiments show that the proposed model can change control task without any explicit traffic information. It could also differentiate rewards based on the saturation level of traffic conditions.

The research team found that the EDQN-based meta-RL model outperformed existing algorithms for traffic light control. However, the researchers highlighted the need for an even more accurate algorithm that takes into account the different levels of saturation from one intersection to another in an urban area.

“Existing research has used reinforcement learning for traffic signal control with a single fixed target. In contrast, this work has designed a controller that can autonomously select the optimal target based on the latest traffic conditions. The framework, if adopted by traffic light enforcement agencies, could bring travel benefits that have never been experienced before,” Professor Sohn said.

The study results were published in the journal “Computer-Aided Civil and Infrastructure Engineering” and were posted online September 30.