I've been working on getting a learned pilot to race in NullG. It trains against the native simulation, exports into a model the game can run, and uses the same controls as a player. The latest work is on weapons: keeping the driving it has already learned while training it to use what it picks up.
Getting a ship around a circuit is only part of this. The pilot also has to deal with traffic, pickups, Turbo and the consequences of firing a weapon. Changes that help one part can upset another, so a lot of the work between training runs has been figuring out what actually changed and replaying it on other tracks.
The YouTube playlist has the driving and combat recordings. This is the longer account of how the training works and what the latest results mean.
Connecting the pilot to the game
Python communicates with isolated native workers, each running NullG's C++ race simulation with its physics, pickups, weapons and rules. Gymnasium supplies the environment interface, with PyTorch and Stable-Baselines3 PPO handling the main training implementation.
The policy is the network that turns observations into controls. During training, a second network, the critic, estimates the return expected from a state. PPO collects experience with the current policy and makes bounded updates to those networks.
The pilot gets structured information about motion, health, the connected road ahead, nearby racers, pickups, visible projectiles, inventory and previous controls. That makes the observation design quite concrete: how much road can it see, how is sideways motion represented, and what happens to those measurements at a section boundary?
It produces throttle, analog steering, left and right airbrakes, pitch and weapon commands at 30 Hz. These pass through the normal player-input path. The session advances at 60 Hz, with the selected source physics cadence; movement is held between decisions and weapon commands follow the engine's input timing.
The trained actor exports to a compact .ngpolicy package. A native C++ evaluator then produces the controls in the game, without Python in that runtime path. The package records which observations and configuration the model expects, along with its identity, so it can be checked against the training setup.
An evaluation race rendered by the cinematic replay exporter. The driving policy and camera director are separate systems.
Useful references
There is plenty of existing racing RL work to learn from. I've used different projects for different parts of this, adapting the ideas to an anti-gravity racer and NullG's own simulation.
Sony AI's account of training Gran Turismo Sophy is useful for thinking about the practice a driver needs: solo driving, traffic and specialized situations. That informs the mix of circuits, opponents and difficult starting states used here.
TMRL covers real-time control, observation histories, analog actions and off-policy learning in TrackMania. It helped motivate a controlled SAC steering comparison alongside PPO. Linesight, another TrackMania project, uses IQN and explicit reference-line geometry. Its handling of progress and training coverage is useful even though NullG's integration is different. Learn-to-Race provides an autonomous-racing environment and baselines for comparing conventional and learned controllers against a defined task.
CAPS informed the smooth-control studies, including looking at action variation and the frequency content of motion. NeuralKart provided a reference for searching for useful driving actions and learning from them. NullG does that search from native snapshots of states reached by its own learner. DAgger explains why collecting corrections in those states matters: the learner's previous decisions determine where it ends up needing help.
The observations and steering
Once the basic loop worked, the next job was checking what information actually reached the network. That meant auditing scaling, feature connections, road preview and the steering adapter.
A value can be present in an observation and still be nearly useless after normalization and clipping. The corrected observations use explicit physical scales and identity normalization, which leaves those scaled values unchanged. Checks also trace features all the way to the network inputs, so adding a field isn't mistaken for successfully teaching the model about it.
The road observations now include forward directions, heading error, longitudinal and lateral velocity, displacement across the road, curvature and clearance. Connected-road projection supplies a continuous position through section boundaries and qualified junctions. Progress rewards use distance along the road and only count new ground beyond the previous furthest point. Pickups and boost pads make limited contributions.
The XL input adapter now provides 513 signed positions for analog steering. That made it possible to compare observation designs while keeping the control protocol and experience budget fixed. A separate PPO-versus-SAC study used the same restricted auxiliary controls. These smaller comparisons helped isolate decisions before asking the full driver to do more.
What happens between training runs
Between October 15 and 20, the work repeatedly went from recorded races and telemetry to a focused change, then back through evaluation. Simply turning up a steering-change penalty wasn't enough. It could make sampled actions quieter, so the checks also looked at the policy's mean output and the ship's actual heading and lateral movement. Longer road preview and different brake commitment were tested separately.
Exploration adds another complication. The actions sampled during training can behave differently from deterministic native inference, so both need checking. The exported driver is the one that eventually has to race.
A steering update changes the trajectory from which Turbo is fired, even when the weapon weights stay fixed. States with naturally earned Turbo were added to the comparisons and examples used to preserve existing behaviour. Speed alone isn't enough to judge the result; health, recovery and what happens over the rest of the race matter too.
For corrective driving, the tools save a native state reached by the learner and try small steering interventions from that exact point. Accepted examples become labels for training, with unchanged actions retained as anchors. Work on Gare d'Europa and Odessa Keys added longer follow-through checks and preservation samples from the other circuits.
Trying a correction involves quite a few steps: restore the state, check that the original run replays correctly, try alternative inputs, train a limited update, export it and run complete races. A turn can look better for a few seconds and still lead to trouble further down the road, so the follow-through checks were extended.
Three completed combat jobs—the eight-hour run on October 18, the four-hour run on October 19 and the latest four-hour continuation—account for more than 16 hours of training wall time. They branch from selected earlier checkpoints, so that isn't sixteen continuous hours on one policy. Short comparisons, earlier driving work, evaluations and corrective searches are additional work. That total doesn't include reading, reviewing footage or changing the tools; there isn't a combined time counter for those.
Adding combat
For combat, I kept the actor's driving features and movement outputs frozen while training the weapon outputs and critic. Weight checks verify that the protected parameters stay unchanged. Full races are still necessary, because a new weapon decision can change the situation the driver has to deal with.
Finishing position matters most in the reward, with limited extra credit for effective combat and useful pickups or pads. Firing more shots doesn't earn a bonus by itself.
The experiments covered longer credit assignment, combat reward weighting and critic calibration. Calibration fits the value estimate to returns collected from the retained policy without changing the actor. For extra practice, a race can start from a saved state where the driver had reached an opponent while carrying an offensive weapon. Those chances can be sparse in a normal race.
The latest configuration mixes ordinary races with ten-second combat opportunity windows. These start from verified native states and collect fresh actions from the current policy. The end of a practice window isn't counted as finishing or being eliminated from a race.
The opponents include original XL drivers, a retained learned challenger and mixed fields. The training rotation uses Talon's Reach, Sagarmatha, Gare d'Europa and Odessa Keys.
The latest four-hour run
This continuation started from the earlier combat run's hour-two checkpoint. Four CPU workers added 6,639,616 decisions, taking that checkpoint lineage from 32,651,264 to 39,290,880. Other experiments have their own experience counts.
The run retained the driving parameters and trained the weapon outputs and critic, using gamma 1.0, GAE lambda 0.95, combat reward scale 1.0 and 300-decision opportunity windows. Hourly saves allowed intermediate checkpoints to be compared with the final one.
The review covered six policies, each with 48 standard evaluation races, plus cross-play and confirmation panels. RLiable's evaluation guidance is useful here: paired conditions, several seeds and the distribution of outcomes give more information than a single good race.
The final checkpoint won 18 of 48 races and finished 31 in the standard panel. In the additional 24-race confirmation panel, it won 9 and finished 15. The intermediate checkpoints had different strengths; the last save wasn't simply better at everything.
The recordings
The learned pilot is in slot 0 in the latest full race. The automatic cameras move around the field, so every ship shown in a close-up isn't necessarily that pilot.
Gare d'Europa, complete race
This is the three-lap race from the final checkpoint of the four-hour continuation. It runs for about three minutes, with native race audio and cinematic cameras.
Watch the full Gare d'Europa race on YouTube.
Racing and combat Short
Watch the racing and combat Short.
Earlier training: update 3
The earlier upload, “NullG - 8 hour AI driving training 3 (reinforcement learning)”, shows another point in the driving work.
Earlier training: update 2
The preceding upload is “NullG - 8 hour AI driving training 2 (reinforcement learning)”.
The next experiments will build on these comparisons, working on weapon decisions while checking that the driving still holds up across the four circuits. The reinforcement-learning page has more detail on the engine integration.