Taxiing

The simulation could already route an aircraft from its stand to the runway. Something still had to drive it there: steer round the corners, keep to the centreline, and hold a sensible speed. Strictly, that’s the pilot’s job, not the controller’s, but without it the rest of the system had nothing realistic to control. I trained a reinforcement learning model to do it.

Reinforcement learning means the model isn’t told what to do. It tries things, gets a reward or a penalty, and slowly works out what earns the most. That sounds simple. In practice, the model is very good at finding ways to earn rewards that have nothing to do with what you wanted.

A training ground

I kept the training completely separate from the airport. The model never trained on Birmingham’s layout, which is mostly long straights and right angles, because I wanted it to learn to drive any taxiway, not memorise one airport. Instead, it practised on a blank training ground, where every episode was a new random route of three waypoints, each about 125 metres to 1.25 kilometres from the last. Only once it could drive those did it go into the simulation.

At first the routes doubled back on themselves in sharp turns that no real taxiway has. Limiting each turn to 90° fixed that.

The first attempt: follow the circle

The first version was a standard deep Q-network, which picks one of five actions every frame: turn left, turn right, thrust up, thrust down, or do nothing. It saw its position, heading, speed, thrust, where the goal was, and how far it was from the centreline.

For a target, I used an “ideal region”: a circle moving along the route at the speed an aircraft should taxi, about 15 knots in turns and 25 on straights. The aircraft got a small reward for every bit of progress along the route, a smaller one for being inside the circle, a big one for each waypoint, and penalties for straying or going the wrong speed.

Every way it cheated

What followed was a long run of the model finding loopholes. Every time I changed what it could see or how it was rewarded, it found a new one.

Its inputs mattered most. Beyond its own speed and thrust, I tried two ways of showing it where to go: the space around it cut into eight slices, with the one containing the next waypoint switched on, or rays measuring how far the taxiway edge is to its left, ahead and right.

Each setup led to its own cheat:

• Race and bail, chasing the circle: it raced straight past the first waypoint and off the route, since that paid more than learning to turn.
• Spin forever, with the slices: from far away, the waypoint stayed in the same slice whatever it did, so spinning in circles looked like progress.
• Turn on the spot, with the rays: it stood still and kept turning towards the right heading, collecting turning rewards without ever moving.

After race and bail, I thought it needed more creative punishments, or more inputs, so I played with both, and nothing changed. The other two made me laugh so much, because they were such unintended side effects. I made the punishments more specific, and it started spinning in circles. I had no idea how it was even doing that. By the third, I was trying to make the rewards foolproof, and it still found a way. I felt like a genius every time I came up with new rewards, and an idiot every time it exploited them.

Part of the fix was physics: I tied how sharply an aircraft can turn to its speed, so one standing still can’t turn at all, just like a real one. But the real problem was deeper.

One decision at a time

With one output, the model could only pick one of its five actions each frame, so it couldn’t steer and change its speed at the same time. Worse, every reward that frame, for steering and for speed, came back as one error for that one choice. It couldn’t tell which part of its driving the error was about.

So I wondered whether each error could go back only through the output it was about: steering errors to steering, and speed errors to thrust. That’s what a multi-headed deep Q-network does. It’s one network with two outputs, a steering head and a thrust head. Each frame it picks one action from each, both happen together, and when it learns, each head’s error flows back only through that head. Training was noticeably more stable straight away.

The final model

The final model is small. Seven inputs (the three rays, the angle and distance to the next waypoint, its speed and its thrust) go through two hidden layers of 16 neurons each, into the two heads of three choices.

Its rewards are simple, and split by head:

Even with two heads, the balance took three rounds of tuning: first the steering rewards were too weak, so I made them stronger, then I made them depend on the situation, which is what the table above shows.

Across all the attempts, it took over 200,000 training episodes in just over a week. The final model then trained for 4,000 episodes, about four hours, and stopped improving after about 2,000.

How well it drives

The final model stays within a few metres of the centreline and averages 16 knots, inside the 15–20 knot target, without swerving or oscillating. It runs on every aircraft in the simulation at once.

One test I liked: setting the ray inputs to zero, as if the aircraft weren’t on a taxiway at all. It refused to move. So it wasn’t leaning on any single input, like the angle to the waypoint.

It isn’t perfect. On long straights it slowly drifts to the right, because the practice routes never had legs that long. Adding waypoints in between fixes it.

Why it matters

The taxiing model isn’t the part of the system that makes decisions, but it’s what makes everything else realistic. Every aircraft moves a little differently, so the collision avoidance and runway logic can’t rely on perfect, predictable movement. The same idea could also work as a digital twin: if a real aircraft moves differently from what its clearance predicts, that’s worth flagging.