Autonomous ATCThe last part of a controller’s job I took on was listening. Pilots ask for things over the radio: pushback, taxi, take-off, landing. Before an automated controller could answer, it had to understand two things: what was being asked for, and which aircraft was asking.
Official phraseology exists, but pilots often drift from it, and anything automatic has to work from speech to text, which makes mistakes. Public datasets of real radio exist, but the main one is broad and noisy, and its labels sit behind a paywall, so I made my own.
The first attempt
I used BERT, Google’s language model built for understanding text rather than writing it. It reads each word in the context of the words on both sides, which suits sorting a request into a category.
The first dataset was 200 requests written by a language model, 50 of each type, all clean and by the book. The model scored 100% on its validation set, then fell apart on real speech to text. The data was so clean and predictable that it had basically learned to look for one keyword, like a search for “pushback”.
At my project demo on 12 March, my inspector pointed out what I already knew: a perfect score on data this clean meant the data was far too ideal. So that same day, I started making it as noisy as I could.
A messier dataset
The new dataset started with 400 requests from a language model, this time longer, less conventional, and closer to how people really talk, including non-native speakers. Each one also named a flight, because the model now had to work out who was asking too. Even a small international airport can have around 20 aircraft on the same radio frequency.
Then I made it look like real speech to text. First, every request was written the way dictation produces it. Then I copied every request and made the copy noisy, word by word: swapping words for ones that sound the same, rhyming words, and splitting terms apart.
That gave 800 requests, half clean and half noisy. Each came with five candidate flights: the right one, at least one other from the same airline so it couldn’t just match the airline name, and some random others.
What the pilot wants
Fine-tuned on the new data for about 15 minutes, BERT got every request type in the test set right. The old model, trained on clean data, got 86% of the same test set.
Which flight is asking
My first try at this was fuzzy text matching: comparing the request against each candidate’s callsign, letter by letter. With the mistakes speech to text makes, it picked the right flight first only 31% of the time.
So I trained DistilBERT, Hugging Face’s smaller, faster version of BERT, to rank them. It reads the request and one candidate together and scores how well they match, then the candidates are sorted by score.
It was so good that the fuzzy matching I’d planned to keep as a first filter wasn’t needed at all.
The real test
The test set was made the same way as the training data, so it couldn’t show how well it does with real voices. For that, a script showed new requests on screen, and five people with different levels of English and different accents read them out to a MacBook’s built-in dictation. They couldn’t see what it typed, so nobody slowed down or corrected themselves.
Speech to text got about 39% of the words wrong:
Even so, out of 25 requests the system understood 23, or 92%. It got one request type and one flight wrong, and both times the right answer was its second choice.
Why it isn’t in the simulation
This is the one part that runs on its own. The simulation has no pilots to talk to, so there was no realistic way to plug it in, and for now the requests it handles are just pushback, taxi, take-off and landing. The next step would be live, continuous speech going straight into it: a way for pilots to talk to an automated controller, with a human still listening in.