Claude 3.5 Sonnet, the latest version of Claude AI, now includes a feature that allows it to use computers like humans.

This capability is currently in public beta, enabling the AI to move a cursor, click on objects, and input text with a virtual keyboard.

This development allows models to interact with computer software similarly to human users, adding to earlier advancements like logical reasoning and image understanding.

Research for this capability involved training Claude to interpret and interact with screen images.

The AI counts pixels to move the cursor, facilitating tasks that require clicking on specific areas.

Testing showed that Claude could generalize its computer-use skills from limited training with basic software.