Pointing at the screen to get fixes
AI coding tool maker Cursor announced on X on Oct. 27 (Oct. 28 Japan time) that it had added "Annotate" to the editor's "Design Mode." On the screen of the web app being built, running live, users can indicate the spots they want changed and give instructions to the AI agent.
Besides selecting one or more on-screen components, users can drag to enclose an area or draw marks with freehand lines, circles, rectangles and arrows. They can also rewrite text on the spot, inspect style settings and give instructions by voice. Notes can be sent immediately or collected as numbered marks and sent together.
The AI receives component information and an image
The agent receives the selected components' type, position, size, displayed text and some style settings, along with a cropped screenshot. For apps built with React, Vue or Svelte, component names are also passed along. This makes it less likely to misidentify what to fix than describing "the button in the upper right" in text alone.
The "Annotate" button appears in version 3.24.18 and later. The documentation does not state which plans are eligible or whether there is an additional cost. The announcement also mentioned checking the appearance at different screen sizes, but an explainer article noted that no description of that feature could be found.
The same week, Cursor also announced the general availability of its "Agent SDK" for building custom agents.
