AI Learns to Intercept Moving Target Using Simple Ship, with Reward Prediction

== The World

The world is simple Asteroids-like 2d space environment with a simple ship and target. Size is 848 x 477 pixels. The ship has radius of 30 and for each trial is randomly started at one of 404,496 locations and randomly oriented in one of 360 positions. Target has radius of 50 and is positioned randomly in any position that does not result in an immediate win (394,496 positions) and given one of 360 random directions and a velocity of 10 pixels/round. If either the ship or target move off the edge of the world they are positioned on the opposite edge by adding or subtracting the width or height of the world.

== The Challenge

The AI needs to learn to intercept the target using basic controls and the properties of the environment within 30 rounds of the trial. The ship is given as inputs the location of the ship, its facing and traveling direction as well as velocity, and the direction and distance of the target on the main plane. For each round the AI can output one of the following:

-	Left - Turns the ship 25 degrees to the left
-	Right - Turn the ship 25 degrees to the right
-	Thrust - Add 10 to velocity of ship in direction it is currently pointing, up to maximum 30
-	None - No action taken

Each round the ship and target are moved based on their velocity and wrapped around edges as needed. If the ship and target touch, the AI is given a positive reward and the trial is stopped. If the 30 rounds run out without the ship and target touching, the AI is given a negative reward and the trial is stopped.

Note that the AI is not given any information on how the wrap-arounds work. It must figure out on its own how to use them to intercept distant targets.

== The Ship and Target Display

The blue ship is shown with several properties:

-	Direction - The ship points toward the small blue nose. The red thruster is shown on the back of the ship.
-	Left and Right Turns - When turning, the ship shows a yellow arc on the left or right side.
-	Thrust - When ship is thrusting an orange arc is show behind the ship.
-	Radar - Distance and direction to target on main plane is shown as pink arc. 
-	Success - A yellow circle lights up on the ship when it intercepts the target successfully.
-	Reward Prediction - The ship shows a green positive-sign when it believes it will complete the trial successfully. The ship shows a red minus-sign when it believes it will complete the trial unsuccessfully.

A green outlined circle shows the starting point of the ship for reference. The target is shown as a solid green circle.

== Results

No training of the AI or any components took place prior to trial number 1. Initial success of AI was around 20%. After 1 million trials AI was able to intercept target 80% of the time. Reward Prediction is very accurate and often recognizes immediately when missteps occur. Video shows every 100th trial of an additional 100,000 trials.

== Follow-up

Next steps include running additional trials to determine if success continues to climb, and altering the environment without resetting policy to determine if AI can adapt to changes in a graceful manner. Future challenges include multi-agent competition and cooperation.

If you have performed similar experiments with your own AI, please drop a link in the comments. I would love to see it and would be curious what you think of this implementation.

== Traditional Approaches to Similar Problems

Minimum Paths to Interception of a Moving Target when Constrained by Turning Radius https://apps.dtic.mil/sti/pdfs/ADA496503.pdf

Maritime Autonomous Surface Ship’s Path Approximation Using Bézier Curves https://mdpi-res.com/d_attachment/symmetry/symmetry-12-01704/article_deploy/symmetry-12-01704.pdf

Off-Line and On-Line Trajectory Planning https://www.researchgate.net/publication/274083923_Off-Line_and_On-Line_Trajectory_Planning

Dynamic Target Interception in Cluttered Environments https://scalar.seas.upenn.edu/wp-content/uploads/2020/06/UD_main.pdf



Source link