Turning My PS3 into a Moonlight Streaming Host, and Building with LLMs

The Problem

I’ve been working a lot on my bespoke TV box lately (I’ll do a write-up on this Soon™). Think Android TV’s 10-foot UI, but built on NixOS, and with locally hosted streaming, emulation, and other games. It runs on a MINISFORUM mini-PC with integrated graphics, which handles emulation through the PS2 generation just fine. The PS3 poses two problems: at the time of writing, only ~79% of titles are listed as playable on RPCS3, and RPCS3’s recommended GPU costs more than an actual PS3.*

The simple solution would be to just plug my actual decade-old PS3 into the TV, jailbreak it so I could play ROMs off my NAS, and play that way. This, though, would mean switching TV inputs and controllers when going between my box and the PlayStation, and I figured I could come up with a solution that avoided that.

The Build

My solution was to make the PlayStation playable remotely from my TV box. The PS3 would sit in my garage plugged into a streaming host, and I’d remotely connect to it from my living room. I used Sunshine and Moonlight for this.

For the uninitiated, Sunshine is a game-streaming server that runs on, e.g., a gaming PC, and lets clients remotely play games on that machine—almost like a super latency-optimized RDP or KVM. Moonlight is a client that connects to and controls a Sunshine server.*

My streaming host is a cheap used ThinkCentre with a 9th-generation i5 for fast-enough iGPU hardware encoding. To capture low-latency HDMI output from the PS3, I used a Magewell Pro Capture AIO PCIe capture card, with this HDMI splitter for HDCP decryption.** Stock Sunshine captures only from the host’s desktop, so I patched it with a custom backend that directly encodes and serves NV12 frames from the Magewell without intermediate rendering and recapture.

For input, I used a Raspberry Pi Pico 2, connected to the PS3 over USB, to impersonate one or many USB controllers. I connected it to the host through this UART to USB adapter. Sunshine injects streamed inputs into the host through virtual input devices, so I built a small Python service to translate those events into HID reports and send them to the Pico over UART. Custom firmware on the Pico presents those reports to the PS3 as input from USB controllers.

All the source code for this project is available here for reference, though it’s very custom to my setup, and I don’t intend to maintain it.

Results

I measured input-to-frame latency to be \(39.9\,\mathrm{ms}\) median and \(46.3\,\mathrm{ms}\) at the 95th percentile across 40 trials, using a custom test app that changed an on-screen rectangle’s color in response to controller input. This includes the console’s own input processing and rendering, and it excludes physical controller latency and TV display latency.

From my couch, it’s super playable—I don’t think I could tell the difference blind between this and a console plugged straight into the TV.

Building with LLMs

Stop reading here if you’re sick of hearing about LLMs. I wouldn’t blame you.

I used Codex and a few other LLMs throughout this project for research and implementation.

After assembling the hardware, I kicked off the implementation work with /goal at about 10 PM, expecting it to work well into the night before finishing or hitting a blocker. Instead, it was playable from my couch after about 45 minutes, most of that being Codex waiting for me to move the Pico’s USB connection from the streaming host to the PS3.* It even was able to autonomously jailbreak it for me.

I think it worked so well because I delegated work effectively between myself and the LLMs I used:

• I defined what I wanted the system to do.
• I designed the contract between every major component of the streaming system.
• I used Claude and ChatGPT to research exact components and confirm the feasibility of the plumbing and the latency targets against Sunshine source and hardware documentation.
• I reviewed the resulting implementation plan and feasibility analysis.
• I ordered the hardware and assembled it all.
• I used Codex to write the software and end-to-end test everything over SSH.

The general pattern was that I owned the problem definition, requirements, and architecture while delegating the research and implementation within those constraints to an LLM. This is nothing new, of course. But when working on low-stakes side projects and with finite free time, it’s easy (for me, at least) to get lazy and delegate too much. A couple things about this specific project helped me stick to the better split:

• This project started on the back burner. I thought through it in detail because the design itself was fun, and I wasn’t rushed by a desire for immediate results.
• I had to actually purchase hardware for this project and was inclined to come up with a plan I was more sure of before pulling the trigger.

I think there’s probably value in recreating some of this project’s forcing functions through developer tooling for results-motivated, software-only work, where they’re not naturally present.