I started writing a City Building game a while ago. Whilst trying to think of ways to make it different to the hundreds of other City Building games out there, I had a shower thought; Wouldn't it be cool to create a virtual world, and have real life objects navigate it.

Thanks to Arduino, prototyping such a thing wasn't impossible for a solo developer like me, so I got to it. I built an RC car with a kit I found on the net and used an Arduino Mega and a Motor Control Shield (L298P - I made a video tutorial about using it, which you can find here). I hooked it up to an ESP 8266 so that I could connect it to the WiFi. The ESP 8266 connects to the WiFi network and passes commands it receives via MQTT down to the Arduino via serial communications. All the source code is available on git, you can find it in the description of the YouTube video.

The MQTT commands that the ESP 8266 receives are published via my PC which is, running the game. I used a test application I wrote a while ago which teaches a car to drive around a track using machine learning. If you're interested in that you can find the video I made about it here.

EXPANDED TECHNICAL DETAILS

Virtual-to-Physical Telemetry Bridge

This project allows a player to drive a physical RC car using the same controls as a virtual computer game, creating a powerful mixed-reality experience.

• Unity MQTT Handshake: The computer game (built in Unity) captures keyboard/joystick inputs and publishes them to an MQTT broker. The Arduino (on the car) subscribes to these topics via a WiFi module.
• Real-Time Actuator Mapping: The Arduino receives the MQTT packets and instantly adjusts the PWM signals for the steering servo and the ESC (Electronic Speed Controller), mirroring the virtual car's physics in the real world.

Performance

• Low-Latency Gameplay: Optimized to achieve sub-50ms latency between the PC keypress and the car's physical movement, ensuring the experience feels responsive and synchronized.