{"author":{"bio":"Daniel Chote, the Code Monkey, Cat Herder, Maker, H4x0r, uav pilot, sim racer, bullshit artist and dad. Made in Hastings, New Zealand... Now living in the USA!","imageUrl":"https://www.gravatar.com/avatar/7f04e14567c4ec6ea9c28826cc9e3e12?s=250\u0026d=mm\u0026r=x","location":"State College, PA","name":"Daniel Chote","postCount":154,"slug":"daniel","website":"http://chote.com"},"cover":"/content/images/2026/10/9e08d4941150-social-background.png","hero":"/content/images/2026/10/b1e12503f058-nullg-rl-full-preview.png","kind":"post","logo":"/content/images/2016/08/projectable-me-4.png","nav":[{"id":"62e854d471683dea63488bcbca5c6188","label":"Home","url":"/","sortOrder":0},{"id":"1bc84fb15620acd635df7735a0b8ddd8","label":"Projects","url":"/projects/","sortOrder":1},{"id":"3207e2fec51d66d2a5b3d8d04feb9dd8","label":"Sim Racing","url":"/tag/sim-racing/","sortOrder":2},{"id":"59115afd52e59a74faa4b386e00ae3fa","label":"About","url":"/about-me/","sortOrder":3}],"post":{"id":"59ab1e54b3a5386e6900ca8678d4e839","siteId":"projectable","slug":"nullg-reinforcement-learning-journey","title":"Teaching NullG to race: the reinforcement-learning journey so far","markdown":"[NullG](https://projectable.me/projects/nullg/) now has a reinforcement-learning pipeline that can train a pilot in the native racing simulation, export its learned controls into the game, and record complete races for review. The latest footage follows that work into weapon combat: preserving a learned driver while training decisions about when and how to use its inventory.\n\nGetting here has involved several changes of direction. Road observations, steering, rewards, exploration, Turbo handling and combat timing have each needed their own experiments. A useful race raises the next question: which part of the system produced that behaviour, and does it hold up on another circuit?\n\nThe [NullG YouTube playlist](https://www.youtube.com/playlist?list=PLaR-_mRLlb6E) follows the progression through recorded driving and combat runs. This post explains the engineering behind those videos, the external projects informing the approach, and the repeated work between longer training sessions.\n\n## A pilot that drives through the game’s controls\n\nNullG’s starting point is its own C++ racing simulation. Python training tools communicate with isolated native workers, each running a race with the engine’s physics, pickups, weapons and race rules. Gymnasium provides the environment interface, and PyTorch with [Stable-Baselines3 PPO](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html) provides the main training implementation.\n\nA policy is the neural network that turns observations into controls. Its companion during training, the critic, estimates the return expected from a state. PPO collects experience with the current policy, then makes bounded updates to those networks. In NullG, that experience comes from the same native simulation used by gameplay.\n\nThe pilot receives structured observations: its motion and health, connected road geometry ahead, nearby racers, available pickups, visible projectiles, inventory and previous controls. This gives development direct access to questions such as how far the pilot can see into a corner and how it represents lateral movement across the road.\n\nThe outputs correspond to familiar inputs: throttle, analog steering, left and right airbrakes, pitch and weapon commands. Decisions arrive at 30 Hz through the normal player-input path, while the session advances at 60 Hz with the selected source physics cadence. Movement is held between decisions; weapon commands follow the engine’s input timing.\n\nOnce trained, the actor exports into a compact `.ngpolicy` package. A native C++ evaluator reads the observations and produces controls inside NullG. Python remains part of the development workflow, while the exported pilot runs in the engine. Package metadata records the observation contract, model identity and supported configuration, making the training-to-game handoff inspectable.\n\n\n\n*The latest recorded evaluation race, presented through NullG’s cinematic replay exporter. The driving policy and the replay camera are separate systems.*\n\n## External projects that shaped the approach\n\nRacing RL already has a substantial body of practical work. NullG uses those projects as references for particular engineering decisions, adapting the ideas to an anti-gravity racer and a native engine.\n\n[Sony AI’s Gran Turismo Sophy training account](https://ai.sony/blog/how-to-train-your-race-car) is a reference for building a varied training regimen. Solo driving, interaction with traffic and specialised situations each contribute useful experience. For NullG, that encourages deliberate coverage of circuits, traffic conditions and difficult states, with several kinds of practice available during development.\n\n[TMRL](https://github.com/trackmania-rl/tmrl), the TrackMania reinforcement-learning framework, is useful for its treatment of real-time control, observation histories, analog actions and off-policy learning. It helped motivate a controlled SAC steering comparison alongside PPO. NullG’s native worker supplies a different integration path, while the timing and control questions transfer well.\n\n[Linesight](https://linesight-rl.github.io/linesight/build/html/main_objects.html) provides another TrackMania reference, including its IQN-based approach and explicit reference-line geometry. Its value here is the care given to defining progress and training coverage. [Learn-to-Race](https://github.com/learn-to-race/l2r) adds a useful autonomous-racing environment and baseline perspective: conventional and learned controllers can be compared through a clearly defined task.\n\n[CAPS](https://ai.bu.edu/caps/) informs the smooth-control studies. Action variation and the frequency content of motion provide ways to examine steering quality. That sits alongside a separate question about exploration: the sampled actions used while learning and the deterministic controls used by an exported actor can behave differently.\n\n[NeuralKart](https://github.com/rameshvarun/NeuralKart) supplies a practical reference for searching for useful driving actions and learning from them. NullG adapts that principle using native snapshots from states reached by its own learner. [DAgger](https://proceedings.mlr.press/v15/ross11a.html) gives the broader rationale for collecting corrective examples where the learner actually drives, as successive actions change the states it encounters.\n\nThese references contribute different pieces: curriculum design, control timing, progress geometry, smoothness, baseline comparisons and corrective data collection. NullG combines them with its existing simulation, input and replay systems.\n\n## Giving the driver useful information\n\nEarly work established the loop from observation to action, followed by closer examination of what the network was actually receiving. That led to an audit of observation scaling, feature connectivity, road preview and the steering adapter.\n\nScaling deserves attention because a number can be technically present yet convey very little useful variation after normalisation and clipping. The corrected foundation uses explicit physical scales and identity normalisation for its observation contract. Checks also trace new features through to the network inputs, so a change to the observation layout has a verifiable effect on the model being trained.\n\nRoad guidance then became more explicit. The pilot receives road-forward directions, heading error, longitudinal and lateral velocity, displacement across the road, curvature and available clearance. Connected-road projection gives those measurements a continuous position along the route, including the neighbourhood of section boundaries and qualified junctions.\n\nThat same care applies to progress rewards. Canonical road distance and high-water accounting make forward progress meaningful across the track’s connected geometry. Pickups and boost pads have their own bounded contributions. A reward can then describe a useful driving objective with a clear relationship to what happened on the circuit.\n\nSteering also moved to a native analog contract, with 513 signed positions available through the XL input adapter. A controlled comparison could then ask about the observation design while holding the control protocol and experience budget steady. Another steering study compared PPO and SAC under the same restricted auxiliary controls. Those experiments narrowed individual design choices before the full driver took on more responsibilities.\n\n## The repeated work between longer runs\n\nThe development record across October 15–20 contains an ongoing cycle of collecting races, reviewing telemetry, making a focused adjustment, and testing the resulting policy again. This has occupied a substantial part of the RL work alongside the training itself.\n\nSeveral strategies needed revisiting. Increasing a steering-change penalty could make sampled inputs look quieter, so review expanded to the policy’s mean output and the ship’s actual heading and lateral motion. Longer road preview and different brake commitment were tested separately. Exploration settings were examined alongside deterministic native inference, giving both the training behaviour and the exported driver a direct comparison.\n\nTurbo made preservation especially important. A steering change affects the trajectory from which Turbo is used, even when the weapon-control weights stay identical. Naturally earned Turbo states therefore became part of the review panels and corrective training anchors. Useful speed, health, recovery and the rest of the race all contribute to judging the manoeuvre.\n\nThe corrective-driving work took another route. Native snapshots preserve a state reached by the learner, then development tooling tries small steering interventions from that same state. Accepted examples become training labels, with unchanged actions retained as anchors. Focused work on Gare d’Europa and Odessa Keys added longer follow-through checks and preservation samples from the other circuits.\n\nEach revision brought additional work around the optimiser: reconstructing states, verifying the replayed baseline, collecting examples, fitting a bounded update, exporting the actor and running complete races. Longer safety follow-through was added to examine the consequences beyond a promising short steering branch.\n\nThe time involved can be described concretely from the run receipts. Three completed combat jobs alone—the eight-hour run on October 18, the four-hour run on October 19, and the latest four-hour continuation—account for **more than 16 hours of training wall time**. Short comparison arms, earlier driving work, evaluation races, corrective searches and media production add to that. Those three jobs branch from selected earlier checkpoints, so their durations describe compute spent across experiments rather than sixteen uninterrupted hours on one policy.\n\nThe records establish those training durations. Reading references, inspecting footage and revising the tools also take development time; that human work has no consolidated hour counter in the run receipts. The visible pattern is a sequence of experiments and adjustments over several days, with longer jobs following more focused checks.\n\n## From driving practice to racing and combat\n\nThe driving foundation gave the pilot room to practise navigation, traffic, pickups and earned Turbo. Combat then introduced a different timing problem: choosing an action now can affect another racer several seconds later, and that exchange can change the final placing.\n\nThe approach evolved toward training the weapon outputs and critic while freezing the actor’s driving features and movement outputs. That lets the combat experiment build on an existing driver. Weight checks verify the protected parameters, while complete races show how the new combat choices affect the resulting trajectories.\n\nThe objective also became explicitly placement-first. Race outcome has the dominant reward, with bounded auxiliary credit for effective combat and useful pickups or pads. Shot count by itself receives no bonus. The critic and credit-assignment settings then become important parts of learning when a weapon opportunity is worth taking.\n\nExperiments examined longer credit assignment, combat reward weighting and critic calibration. Calibration fits the value estimate to returns collected from the retained policy, while preserving the actor. Opportunity starts reconstruct naturally reached situations with offensive inventory and nearby opponents, giving the learner more practice around decisions that may be relatively sparse in ordinary races.\n\nThe latest configuration combines ordinary races with ten-second combat opportunity windows. These windows start from verified native states and collect fresh actions from the current policy. Their training cutoff is handled separately from a race finish or elimination, preserving the meaning of the full-race objective.\n\nTraining opposition has expanded too: original XL opponents, a retained learned challenger, and mixed fields provide different contexts. The active development rotation is Talon’s Reach, Sagarmatha, Gare d’Europa and Odessa Keys.\n\n## What the latest four-hour continuation did\n\nThe latest completed continuation began at the earlier combat run’s hour-two checkpoint. Four CPU workers added **6,639,616 decisions**, moving the checkpoint counter from **32,651,264 to 39,290,880**. That number belongs to this checkpoint lineage; it is separate from the experience collected across all the other experiments.\n\nThe run preserved the driving parameters and continued learning weapon outputs and the critic. Its recorded settings retained a placement-first objective, gamma 1.0, GAE lambda 0.95, combat reward scale 1.0 and 300-decision opportunity windows. In practical terms, the experiment concentrated on combat choices while retaining the driver’s learned control function.\n\nHourly checkpoints made it possible to compare behaviour during the continuation. The subsequent review covered six policies, with 48 standard evaluation races per policy and additional cross-play and confirmation panels. [RLiable’s evaluation guidance](https://research.google/blog/rliable-towards-reliable-evaluation-reporting-in-reinforcement-learning/) informs this emphasis on paired conditions, multiple seeds and the distribution of results.\n\nThe final checkpoint won 18 of its 48 standard-panel races and finished 31. In the additional 24-race confirmation panel, it won 9 and finished 15. Different intermediate checkpoints had different strengths, so the review retains those trade-offs for the next decision. A cinematic race shows the driving and combat in detail; the larger panels establish the context for interpreting it.\n\n\n\n*Recorded Gare d’Europa gameplay from the final checkpoint. The cinematic director follows racing interest and weapon exchanges across the field, including opponents as well as the learned racer.*\n\n## Watch the latest races\n\nThe videos make the development easier to follow alongside the technical account. Earlier driving updates show the broader training progression; the latest full race and Short focus on racing with weapons. The learned pilot occupies slot 0 in the latest full race, while automatic cinematic cameras also cover other racers and exchanges.\n\n### Learning to fight: the complete Gare d’Europa race\n\nThis complete three-lap race comes from the final checkpoint of the latest four-hour continuation. It runs for approximately three minutes, with native race audio and cinematic presentation.\n\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/__kpDHYbsCg\" frameborder=\"0\" allowfullscreen\u003e\u003c/iframe\u003e\n\n[Watch the full Gare d’Europa race on YouTube](https://youtu.be/__kpDHYbsCg).\n\n### Learning to race and fight: the Short\n\nThe latest Short offers a compact look at the racing and combat work.\n\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/lTbLUj1QKt4\" frameborder=\"0\" allowfullscreen\u003e\u003c/iframe\u003e\n\n[Watch the racing and combat on YouTube](https://youtu.be/lTbLUj1QKt4).\n\n### Earlier training footage: update 3\n\nThis earlier upload is titled “NullG - 8 hour AI driving training 3 (reinforcement learning)” and forms part of the playlist’s training record.\n\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/fkUTwW2p_8k\" frameborder=\"0\" allowfullscreen\u003e\u003c/iframe\u003e\n\n[Watch training update 3 on YouTube](https://youtu.be/fkUTwW2p_8k).\n\n### Earlier training footage: update 2\n\nThe preceding “NullG - 8 hour AI driving training 2 (reinforcement learning)” upload provides another point in that recorded progression.\n\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/9ddeaoyyIY4\" frameborder=\"0\" allowfullscreen\u003e\u003c/iframe\u003e\n\n[Watch training update 2 on YouTube](https://youtu.be/9ddeaoyyIY4).\n\n## Following the next experiments\n\nThe work now has a native training loop, portable in-game policies, corrective driving tools, combat-focused continuations and recorded evaluations to guide further changes. The next direction is to build on the checkpoint comparisons: keep race outcome central, improve the usefulness of weapon decisions, and preserve driving quality across the four-circuit rotation.\n\nThe [reinforcement-learning project page](https://projectable.me/projects/nullg/reinforcement-learning/) describes the engine integration. The [YouTube playlist](https://www.youtube.com/playlist?list=PLaR-_mRLlb6E) follows the training through the races themselves, with the full recordings providing context for each shorter highlight.","html":"\u003cp\u003e\u003ca href=\"https://projectable.me/projects/nullg/\" rel=\"nofollow\"\u003eNullG\u003c/a\u003e now has a reinforcement-learning pipeline that can train a pilot in the native racing simulation, export its learned controls into the game, and record complete races for review. The latest footage follows that work into weapon combat: preserving a learned driver while training decisions about when and how to use its inventory.\u003c/p\u003e\n\u003cp\u003eGetting here has involved several changes of direction. Road observations, steering, rewards, exploration, Turbo handling and combat timing have each needed their own experiments. A useful race raises the next question: which part of the system produced that behaviour, and does it hold up on another circuit?\u003c/p\u003e\n\u003cp\u003eThe \u003ca href=\"https://www.youtube.com/playlist?list=PLaR-_mRLlb6E\" rel=\"nofollow\"\u003eNullG YouTube playlist\u003c/a\u003e follows the progression through recorded driving and combat runs. This post explains the engineering behind those videos, the external projects informing the approach, and the repeated work between longer training sessions.\u003c/p\u003e\n\u003ch2\u003eA pilot that drives through the game’s controls\u003c/h2\u003e\n\u003cp\u003eNullG’s starting point is its own C++ racing simulation. Python training tools communicate with isolated native workers, each running a race with the engine’s physics, pickups, weapons and race rules. Gymnasium provides the environment interface, and PyTorch with \u003ca href=\"https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html\" rel=\"nofollow\"\u003eStable-Baselines3 PPO\u003c/a\u003e provides the main training implementation.\u003c/p\u003e\n\u003cp\u003eA policy is the neural network that turns observations into controls. Its companion during training, the critic, estimates the return expected from a state. PPO collects experience with the current policy, then makes bounded updates to those networks. In NullG, that experience comes from the same native simulation used by gameplay.\u003c/p\u003e\n\u003cp\u003eThe pilot receives structured observations: its motion and health, connected road geometry ahead, nearby racers, available pickups, visible projectiles, inventory and previous controls. This gives development direct access to questions such as how far the pilot can see into a corner and how it represents lateral movement across the road.\u003c/p\u003e\n\u003cp\u003eThe outputs correspond to familiar inputs: throttle, analog steering, left and right airbrakes, pitch and weapon commands. Decisions arrive at 30 Hz through the normal player-input path, while the session advances at 60 Hz with the selected source physics cadence. Movement is held between decisions; weapon commands follow the engine’s input timing.\u003c/p\u003e\n\u003cp\u003eOnce trained, the actor exports into a compact \u003ccode\u003e.ngpolicy\u003c/code\u003e package. A native C++ evaluator reads the observations and produces controls inside NullG. Python remains part of the development workflow, while the exported pilot runs in the engine. Package metadata records the observation contract, model identity and supported configuration, making the training-to-game handoff inspectable.\u003c/p\u003e\n\u003cimg src=\"/content/images/2026/10/b1b8220f350c-nullg-rl-full-title-preview.png\" alt=\"\"\u003e\n\u003cp\u003e\u003cem\u003eThe latest recorded evaluation race, presented through NullG’s cinematic replay exporter. The driving policy and the replay camera are separate systems.\u003c/em\u003e\u003c/p\u003e\n\u003ch2\u003eExternal projects that shaped the approach\u003c/h2\u003e\n\u003cp\u003eRacing RL already has a substantial body of practical work. NullG uses those projects as references for particular engineering decisions, adapting the ideas to an anti-gravity racer and a native engine.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://ai.sony/blog/how-to-train-your-race-car\" rel=\"nofollow\"\u003eSony AI’s Gran Turismo Sophy training account\u003c/a\u003e is a reference for building a varied training regimen. Solo driving, interaction with traffic and specialised situations each contribute useful experience. For NullG, that encourages deliberate coverage of circuits, traffic conditions and difficult states, with several kinds of practice available during development.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/trackmania-rl/tmrl\" rel=\"nofollow\"\u003eTMRL\u003c/a\u003e, the TrackMania reinforcement-learning framework, is useful for its treatment of real-time control, observation histories, analog actions and off-policy learning. It helped motivate a controlled SAC steering comparison alongside PPO. NullG’s native worker supplies a different integration path, while the timing and control questions transfer well.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://linesight-rl.github.io/linesight/build/html/main_objects.html\" rel=\"nofollow\"\u003eLinesight\u003c/a\u003e provides another TrackMania reference, including its IQN-based approach and explicit reference-line geometry. Its value here is the care given to defining progress and training coverage. \u003ca href=\"https://github.com/learn-to-race/l2r\" rel=\"nofollow\"\u003eLearn-to-Race\u003c/a\u003e adds a useful autonomous-racing environment and baseline perspective: conventional and learned controllers can be compared through a clearly defined task.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://ai.bu.edu/caps/\" rel=\"nofollow\"\u003eCAPS\u003c/a\u003e informs the smooth-control studies. Action variation and the frequency content of motion provide ways to examine steering quality. That sits alongside a separate question about exploration: the sampled actions used while learning and the deterministic controls used by an exported actor can behave differently.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/rameshvarun/NeuralKart\" rel=\"nofollow\"\u003eNeuralKart\u003c/a\u003e supplies a practical reference for searching for useful driving actions and learning from them. NullG adapts that principle using native snapshots from states reached by its own learner. \u003ca href=\"https://proceedings.mlr.press/v15/ross11a.html\" rel=\"nofollow\"\u003eDAgger\u003c/a\u003e gives the broader rationale for collecting corrective examples where the learner actually drives, as successive actions change the states it encounters.\u003c/p\u003e\n\u003cp\u003eThese references contribute different pieces: curriculum design, control timing, progress geometry, smoothness, baseline comparisons and corrective data collection. NullG combines them with its existing simulation, input and replay systems.\u003c/p\u003e\n\u003ch2\u003eGiving the driver useful information\u003c/h2\u003e\n\u003cp\u003eEarly work established the loop from observation to action, followed by closer examination of what the network was actually receiving. That led to an audit of observation scaling, feature connectivity, road preview and the steering adapter.\u003c/p\u003e\n\u003cp\u003eScaling deserves attention because a number can be technically present yet convey very little useful variation after normalisation and clipping. The corrected foundation uses explicit physical scales and identity normalisation for its observation contract. Checks also trace new features through to the network inputs, so a change to the observation layout has a verifiable effect on the model being trained.\u003c/p\u003e\n\u003cp\u003eRoad guidance then became more explicit. The pilot receives road-forward directions, heading error, longitudinal and lateral velocity, displacement across the road, curvature and available clearance. Connected-road projection gives those measurements a continuous position along the route, including the neighbourhood of section boundaries and qualified junctions.\u003c/p\u003e\n\u003cp\u003eThat same care applies to progress rewards. Canonical road distance and high-water accounting make forward progress meaningful across the track’s connected geometry. Pickups and boost pads have their own bounded contributions. A reward can then describe a useful driving objective with a clear relationship to what happened on the circuit.\u003c/p\u003e\n\u003cp\u003eSteering also moved to a native analog contract, with 513 signed positions available through the XL input adapter. A controlled comparison could then ask about the observation design while holding the control protocol and experience budget steady. Another steering study compared PPO and SAC under the same restricted auxiliary controls. Those experiments narrowed individual design choices before the full driver took on more responsibilities.\u003c/p\u003e\n\u003ch2\u003eThe repeated work between longer runs\u003c/h2\u003e\n\u003cp\u003eThe development record across October 15–20 contains an ongoing cycle of collecting races, reviewing telemetry, making a focused adjustment, and testing the resulting policy again. This has occupied a substantial part of the RL work alongside the training itself.\u003c/p\u003e\n\u003cp\u003eSeveral strategies needed revisiting. Increasing a steering-change penalty could make sampled inputs look quieter, so review expanded to the policy’s mean output and the ship’s actual heading and lateral motion. Longer road preview and different brake commitment were tested separately. Exploration settings were examined alongside deterministic native inference, giving both the training behaviour and the exported driver a direct comparison.\u003c/p\u003e\n\u003cp\u003eTurbo made preservation especially important. A steering change affects the trajectory from which Turbo is used, even when the weapon-control weights stay identical. Naturally earned Turbo states therefore became part of the review panels and corrective training anchors. Useful speed, health, recovery and the rest of the race all contribute to judging the manoeuvre.\u003c/p\u003e\n\u003cp\u003eThe corrective-driving work took another route. Native snapshots preserve a state reached by the learner, then development tooling tries small steering interventions from that same state. Accepted examples become training labels, with unchanged actions retained as anchors. Focused work on Gare d’Europa and Odessa Keys added longer follow-through checks and preservation samples from the other circuits.\u003c/p\u003e\n\u003cp\u003eEach revision brought additional work around the optimiser: reconstructing states, verifying the replayed baseline, collecting examples, fitting a bounded update, exporting the actor and running complete races. Longer safety follow-through was added to examine the consequences beyond a promising short steering branch.\u003c/p\u003e\n\u003cp\u003eThe time involved can be described concretely from the run receipts. Three completed combat jobs alone—the eight-hour run on October 18, the four-hour run on October 19, and the latest four-hour continuation—account for \u003cstrong\u003emore than 16 hours of training wall time\u003c/strong\u003e. Short comparison arms, earlier driving work, evaluation races, corrective searches and media production add to that. Those three jobs branch from selected earlier checkpoints, so their durations describe compute spent across experiments rather than sixteen uninterrupted hours on one policy.\u003c/p\u003e\n\u003cp\u003eThe records establish those training durations. Reading references, inspecting footage and revising the tools also take development time; that human work has no consolidated hour counter in the run receipts. The visible pattern is a sequence of experiments and adjustments over several days, with longer jobs following more focused checks.\u003c/p\u003e\n\u003ch2\u003eFrom driving practice to racing and combat\u003c/h2\u003e\n\u003cp\u003eThe driving foundation gave the pilot room to practise navigation, traffic, pickups and earned Turbo. Combat then introduced a different timing problem: choosing an action now can affect another racer several seconds later, and that exchange can change the final placing.\u003c/p\u003e\n\u003cp\u003eThe approach evolved toward training the weapon outputs and critic while freezing the actor’s driving features and movement outputs. That lets the combat experiment build on an existing driver. Weight checks verify the protected parameters, while complete races show how the new combat choices affect the resulting trajectories.\u003c/p\u003e\n\u003cp\u003eThe objective also became explicitly placement-first. Race outcome has the dominant reward, with bounded auxiliary credit for effective combat and useful pickups or pads. Shot count by itself receives no bonus. The critic and credit-assignment settings then become important parts of learning when a weapon opportunity is worth taking.\u003c/p\u003e\n\u003cp\u003eExperiments examined longer credit assignment, combat reward weighting and critic calibration. Calibration fits the value estimate to returns collected from the retained policy, while preserving the actor. Opportunity starts reconstruct naturally reached situations with offensive inventory and nearby opponents, giving the learner more practice around decisions that may be relatively sparse in ordinary races.\u003c/p\u003e\n\u003cp\u003eThe latest configuration combines ordinary races with ten-second combat opportunity windows. These windows start from verified native states and collect fresh actions from the current policy. Their training cutoff is handled separately from a race finish or elimination, preserving the meaning of the full-race objective.\u003c/p\u003e\n\u003cp\u003eTraining opposition has expanded too: original XL opponents, a retained learned challenger, and mixed fields provide different contexts. The active development rotation is Talon’s Reach, Sagarmatha, Gare d’Europa and Odessa Keys.\u003c/p\u003e\n\u003ch2\u003eWhat the latest four-hour continuation did\u003c/h2\u003e\n\u003cp\u003eThe latest completed continuation began at the earlier combat run’s hour-two checkpoint. Four CPU workers added \u003cstrong\u003e6,639,616 decisions\u003c/strong\u003e, moving the checkpoint counter from \u003cstrong\u003e32,651,264 to 39,290,880\u003c/strong\u003e. That number belongs to this checkpoint lineage; it is separate from the experience collected across all the other experiments.\u003c/p\u003e\n\u003cp\u003eThe run preserved the driving parameters and continued learning weapon outputs and the critic. Its recorded settings retained a placement-first objective, gamma 1.0, GAE lambda 0.95, combat reward scale 1.0 and 300-decision opportunity windows. In practical terms, the experiment concentrated on combat choices while retaining the driver’s learned control function.\u003c/p\u003e\n\u003cp\u003eHourly checkpoints made it possible to compare behaviour during the continuation. The subsequent review covered six policies, with 48 standard evaluation races per policy and additional cross-play and confirmation panels. \u003ca href=\"https://research.google/blog/rliable-towards-reliable-evaluation-reporting-in-reinforcement-learning/\" rel=\"nofollow\"\u003eRLiable’s evaluation guidance\u003c/a\u003e informs this emphasis on paired conditions, multiple seeds and the distribution of results.\u003c/p\u003e\n\u003cp\u003eThe final checkpoint won 18 of its 48 standard-panel races and finished 31. In the additional 24-race confirmation panel, it won 9 and finished 15. Different intermediate checkpoints had different strengths, so the review retains those trade-offs for the next decision. A cinematic race shows the driving and combat in detail; the larger panels establish the context for interpreting it.\u003c/p\u003e\n\u003cimg src=\"/content/images/2026/10/b1e12503f058-nullg-rl-full-preview.png\" alt=\"\"\u003e\n\u003cp\u003e\u003cem\u003eRecorded Gare d’Europa gameplay from the final checkpoint. The cinematic director follows racing interest and weapon exchanges across the field, including opponents as well as the learned racer.\u003c/em\u003e\u003c/p\u003e\n\u003ch2\u003eWatch the latest races\u003c/h2\u003e\n\u003cp\u003eThe videos make the development easier to follow alongside the technical account. Earlier driving updates show the broader training progression; the latest full race and Short focus on racing with weapons. The learned pilot occupies slot 0 in the latest full race, while automatic cinematic cameras also cover other racers and exchanges.\u003c/p\u003e\n\u003ch3\u003eLearning to fight: the complete Gare d’Europa race\u003c/h3\u003e\n\u003cp\u003eThis complete three-lap race comes from the final checkpoint of the latest four-hour continuation. It runs for approximately three minutes, with native race audio and cinematic presentation.\u003c/p\u003e\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/__kpDHYbsCg\" frameborder=\"0\" allowfullscreen=\"\"\u003e\u003c/iframe\u003e\n\u003cp\u003e\u003ca href=\"https://youtu.be/__kpDHYbsCg\" rel=\"nofollow\"\u003eWatch the full Gare d’Europa race on YouTube\u003c/a\u003e.\u003c/p\u003e\n\u003ch3\u003eLearning to race and fight: the Short\u003c/h3\u003e\n\u003cp\u003eThe latest Short offers a compact look at the racing and combat work.\u003c/p\u003e\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/lTbLUj1QKt4\" frameborder=\"0\" allowfullscreen=\"\"\u003e\u003c/iframe\u003e\n\u003cp\u003e\u003ca href=\"https://youtu.be/lTbLUj1QKt4\" rel=\"nofollow\"\u003eWatch the racing and combat on YouTube\u003c/a\u003e.\u003c/p\u003e\n\u003ch3\u003eEarlier training footage: update 3\u003c/h3\u003e\n\u003cp\u003eThis earlier upload is titled “NullG - 8 hour AI driving training 3 (reinforcement learning)” and forms part of the playlist’s training record.\u003c/p\u003e\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/fkUTwW2p_8k\" frameborder=\"0\" allowfullscreen=\"\"\u003e\u003c/iframe\u003e\n\u003cp\u003e\u003ca href=\"https://youtu.be/fkUTwW2p_8k\" rel=\"nofollow\"\u003eWatch training update 3 on YouTube\u003c/a\u003e.\u003c/p\u003e\n\u003ch3\u003eEarlier training footage: update 2\u003c/h3\u003e\n\u003cp\u003eThe preceding “NullG - 8 hour AI driving training 2 (reinforcement learning)” upload provides another point in that recorded progression.\u003c/p\u003e\n\u003ciframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/9ddeaoyyIY4\" frameborder=\"0\" allowfullscreen=\"\"\u003e\u003c/iframe\u003e\n\u003cp\u003e\u003ca href=\"https://youtu.be/9ddeaoyyIY4\" rel=\"nofollow\"\u003eWatch training update 2 on YouTube\u003c/a\u003e.\u003c/p\u003e\n\u003ch2\u003eFollowing the next experiments\u003c/h2\u003e\n\u003cp\u003eThe work now has a native training loop, portable in-game policies, corrective driving tools, combat-focused continuations and recorded evaluations to guide further changes. The next direction is to build on the checkpoint comparisons: keep race outcome central, improve the usefulness of weapon decisions, and preserve driving quality across the four-circuit rotation.\u003c/p\u003e\n\u003cp\u003eThe \u003ca href=\"https://projectable.me/projects/nullg/reinforcement-learning/\" rel=\"nofollow\"\u003ereinforcement-learning project page\u003c/a\u003e describes the engine integration. The \u003ca href=\"https://www.youtube.com/playlist?list=PLaR-_mRLlb6E\" rel=\"nofollow\"\u003eYouTube playlist\u003c/a\u003e follows the training through the races themselves, with the full recordings providing context for each shorter highlight.\u003c/p\u003e\n","excerpt":"From native driving controls and road guidance to Turbo preservation and weapon combat: the experiments, references and repeated adjustments behind NullG’s learning pilots.","metaTitle":"NullG’s reinforcement-learning journey: driving, Turbo and combat","metaDescription":"Explore NullG’s native racing RL pipeline, external research references, strategy revisions and latest training videos, from analog steering to combat.","status":"published","featured":false,"publishedAt":"2026-10-20T13:21:42Z","authorId":"190160de54f2192bd5b8734ec8808086","authorName":"Daniel Chote","authorSlug":"daniel","heroMediaId":"2de6b37e96d56188d9b74019123198cd","projectId":"ae4bdb66da6dc1729a263a475e1cdbe3","tags":[{"id":"10a974c0a281d4fcf3d44f5ef8dae079","name":"NullG","slug":"nullg"},{"id":"1c9e8a1339a8f8b797767e91a9673016","name":"Game","slug":"game"},{"id":"60103f47eb65b46c6138b9d6fc7b17d0","name":"Machine Learning","slug":"machine-learning"}],"createdAt":"2026-10-20T13:10:40Z","updatedAt":"2026-10-20T15:07:55Z"},"preview":false,"related":[{"id":"7b367212a739706ba4e3e9b999bec15d","image":"/content/images/2026/10/5febc720f2c6-nullg-rl-hero.png","slug":"introducing-nullg-anti-gravity-racing-and-ai","title":"Introducing NullG: anti-gravity racing and AI-assisted development"},{"id":"39b4d054f7e0bce8c6537f8d80a0a2e3","image":"/content/images/2017/04/IMG_0528.JPG","slug":"msi-gtx-1050ti-4g-oc-the-poor-mans-1u-inference-gpu","title":"MSI GTX 1050Ti 4G OC - The poor man's 1u inference GPU"},{"id":"b1e03f83e9994cc176806835fefac821","image":"","slug":"a-gopinoccio-based-game","title":"A @goPinoccio based game"}],"site":{"id":"projectable","theme":"projectable","title":"Daniel Chote's Project Blog","description":"Software Development, Game Development, Hardware Hacking, Sim Racing, Multi-rotors and Robots!","timezone":"America/New_York","postsPerPage":10,"logoMediaId":"21557257b3f2ecc4b3fa875b808097a7","coverMediaId":"6d7e99f0997d13aac98da38677ca7064","copyright":"© Daniel Chote","twitterUrl":"https://twitter.com/dchote","linkedinUrl":"https://www.linkedin.com/in/dchote","googleTagId":"G-07GJBQVM38","hosts":[{"host":"new.projectable.me","canonical":false},{"host":"projectable.me","canonical":true},{"host":"www.projectable.me","canonical":false}]}}