A research app in which a model draws the screen frame by frame

Runway Labs, the research and development team of AI video generation company Runway, introduced a research application, "Diffusion-UI," on its official X account on Nov. 17 (1:22 a.m. Nov. 18 Japan time). It is interactive software in which every pixel on the screen is generated by a model running in real time, the team said. The team said the new application lets generated interfaces use structured data, maintain states and generate controls. It described the release as an "early look."

Introducing Diffusion-UI. Interactive software where every pixel is generated by a real-time model. Today, we're sharing a new research application that lets generated interfaces use structured data, maintain states and generate controls. Here’s an early look: https://t.co/M5PB5RyOaS

Changing scenes with prompts and box moves

According to follow-up posts, users can type a text instruction (a prompt) to change the scene, add objects or edit anything in real time. Because generated interfaces are redrawn frame by frame as the user interacts, scenes can be designed simply by moving and resizing a bounding box around an object, the team said. The team said it can also be used to render complex 3D interfaces such as immersive game scenes.

Map and weather data passed in as JSON

Diffusion-UI can generate interfaces from structured data. Precise map routes, live weather or information from other APIs can be passed in as JSON objects, and the model uses that information to render dynamic interfaces, the team said. Generation uses four "layers": data sources, reference layouts, styles and text prompts. When the pointer hovers over the screen, nearby buttons, sliders and other controls are highlighted with white outlines to show which parts can be operated.

Eligible users and timing not disclosed

Runway Labs said it is exploring tools that make composing and refining real-time interfaces feel direct and familiar, and that it will share more prototypes and tools soon. The posts did not disclose who can use Diffusion-UI, when it will become available, its pricing, or how to get it, such as through an application or a waitlist. They also did not give the name or workings of the model used, response speed or the computing resources required. The posts include introductory videos, but ORIENS has not checked their contents. As of the 2 a.m. hour on Nov. 18 Japan time, no description of Diffusion-UI could be found on the Runway Labs website or Runway's research page.

Runway Labs is an internal incubator that Runway announced on March 11 to explore the next generation of applications for generative video. In an introduction on X in October, it called itself "a frontier research design studio" building prototypes, applications and new media projects with generative video and world models (models that predict how the world moves), and Runway's official account shared it. Runway's official account also shared the Diffusion-UI post. On the product side, Runway released "App Studio," which builds production apps from text descriptions, in October.