TLDR
- Anthropic gave its Claude 3.5 Sonnet LLM ‘computer use’ skills that lets it move the cursor, type text, and do searches. Google is planning to unveil its own version as early as December.
- Microsoft unveiled 10 new AI agents that can perform tasks in business applications, such as researching sales leads and sending emails to customers.
AI agents seem to be the AI trend du jour. Now, Anthropic is taking it a step further with AI agents that can actually use your computer like you do.
AI startup Anthropic – the maker of the Claude LLM, ChatGPT’s fiercest rival – has released a ‘computer use’ capability on its Claude 3.5 Sonnet API, according to a company blog post. Claude will now be able to look at a screen, move a cursor, click on buttons, and type text.
Anthropic said Claude 3.5 Sonnet is the first frontier AI model to offer ‘computer use’ in public beta. It warns, however, that the capability is still cumbersome and prone to error. But it is releasing ‘computer use’ publicly to get feedback from developers, so it can improve.
Google also is expected to unveil its computer-using AI agents as early as December, according to The Information. Code-named Project Jarvis, these agents can take over a web browser to do research and shopping. In November, OpenAI CEO Sam Altman said AI agents are coming by 2025.
Microsoft also unveiled 10 new AI agents for its business applications suite, Microsoft Dynamics 365. For example, the sales AI agent can research sales leads and reach out to customers with personalized emails. The supplier agent can track supplier performance, detect delays and respond accordingly. Meanwhile, Salesforce recently launched a similar service for business, called Agentforce.
How Anthropic’s ‘computer use’ works
Anthropic said Asana, Canva, Cognition, DoorDash, Replit and The Browser Company has started to test ‘computer use’ in its AI agents to do tasks that need dozens or even hundreds of steps. For example, Replit is using it to develop a key feature that evaluates apps as they’re being built.
In an example from Anthropic, a vendor called Ant Equipment Co. asks you to fill out an online form. But the information you need is scattered throughout your computer. Enter a prompt to direct Claude to fill out the form and where to get the data: either the vendor spreadsheet or the CRM platform like Salesforce. The AI agent will then fill out the form.
Claude will start taking screenshots of the computer screen. It realizes the information is not in the spreadsheets.
It goes to the second tab in the browser to search the CRM – and types in Ant Equipment Co.
Claude finds the vendor!
It starts to automatically fill out the form.
Then submits it. Voila!
The full video:
