Building a world with my voice

This is a project built with A-Frame (Three.js, WebXR) that lets me build virtual worlds with my voice, all from within my VR headset. My mic is hooked up to Claude Code, which in turn runs against the project codebase. Since the agent is working directly with code, the possibilities of what can be achieved are pretty broad, essentially being limited only by what the underlying LLM is capable of with WebXR. The agent can build anything from simple objects, to animations, to interaction capabilities, to whole environments in order to fulfil a request. This is still far from perfect. Latency is pretty high, taking anywhere from a minute to multiple minutes to carry out a change depending on how complex the request is. The codebase is also not very accommodating to anything that isn't a frontier model right now. From my testing, models like Haiku and Sonnet can't keep up, and Opus 5 only works with reasoning set to at least Low (most of the video is recorded with Opus 5 Low). But the foundation is now a reality, and as these models improve I can see this getting very capable very quickly. If you have ever imagined being able to put on a VR headset and transporting yourself to any world you want by simply asking, to shape that world to meet your creative vision, and to watch that world materialise around you, then in a small way, we are here. submitted by /u/Art_from_the_Machine [link] [comments]