I tried to teach rockets to land, and got missiles

Last week I taught an AI to play Flappy Bird by following a tutorial. This time I wanted to make one with no tutorial: my own game, my own physics and my own scoring, to see if I actually understood it. It’s my first AI project of my own.

The plan was simple: a rocket, a target on the ground, and an AI that learns to land on it, like the SpaceX boosters do. SpaceX is the whole reason it’s rockets. I made the game in pygame and drew the sprites myself. Like the birds, it uses NEAT, which evolves small neural networks rather than training one. The idea came from Code Bullet’s A.I. Learns to DRIVE, where an AI learns to drive round a track by itself.

How it works

Every generation, 250 rockets spawn at random spots in the sky, each with its own little network flying it. They get 20 seconds. Whenever one hits the ground it leaves a tombstone with its score on it, so by the end of a round the ground is covered in them.

Each rocket knows seven things: how far across it is, how high it is, its angle, its speed in both directions, how far away the target is and at what angle. It has three controls: thrust, turn left and turn right.

At first I let it use them all at once, every frame. The rockets just twitched about randomly. Making it pick one control per frame, whichever it wanted most, made them much calmer, and they started learning properly.

The scoring

Most of the time went into the scoring, the fitness function. Way too much time. Every reward I added got exploited somehow:

A reward for staying alive, and they learnt to hover forever. I ended up setting it to zero.
So I penalised running out of time, flying off the screen and not moving sideways at all.
They got points for getting closer to the target, for landing within 100 pixels of it, and for landing upright.
I tried penalising landing too fast, but that stopped them learning anything useful, so it came out again.

Each network starts with one hidden layer of five nodes, and NEAT usually got rid of some of them. Adding a second layer, or taking the hidden layer away, made it worse: the rockets just flew in circles until they crashed, because on average that scored best.

Missiles

The best version peaked at around generation 300.

The average score of each generation’s 250 rockets, traced from the run’s own chart.

Letting it carry on to 1,000 didn’t help. It got worse.

A 1,000-generation run, traced from its chart the same way. Its generations were closer together than a pixel, so they’re drawn as a band.

And the best rockets didn’t land. They turned towards the target and flew straight into it, like missiles. Not what I was going for, but it was the strategy that scored best.

I’m fairly sure the scoring is what’s wrong, not NEAT. Every rocket starts at a random spot, so one that happens to start right above the target scores well by luck and gets bred. And since I took the penalty for landing too fast back out, hitting the target at full speed scores about as well as landing on it gently. The rockets learnt exactly what I was paying them for.

The best network ended up tiny: seven inputs, three outputs and a single hidden node. Its strongest connection goes from vertical speed to thrust.

The best rocket’s saved network, with all 17 of its links, flying from a random spot at a random target, like in training. Whichever control wants it most lights up, and the rocket uses it.

So no SpaceX landings. You can try landing it yourself, against the best of them, on its project page. What I learnt is that the scoring decides what an AI learns, more than anything else I changed. The code and the full write-up are in its README on GitHub.

I definitely want to try it again, maybe with reinforcement learning, which I’ve heard is good for this kind of thing. I’m sure NEAT is good enough, though. The problem is me: I’m just not good at writing the scoring yet. So I need more experience first, and then I’ll come back to it.