Key takeaways:
- Apple unveiled its most extensive deployment of AI to date, infusing OpenAI’s ChatGPT (GPT-4o) throughout all its devices.
- The tech giant made Siri into an AI agent, introduced on-device AI processing (edge AI) and a private cloud compute to protect the user data it touches.
- But the biggest cheer came from the addition of a calculator to the iPad, which had long been a pain point for users.
Apple today unveiled its most extensive deployment of AI to date, infusing generative AI throughout all its devices to unlock new experiences for users.
“This is the next big step for Apple,” said CEO Tim Cook at Apple’s Worldwide Developers Conference (WWDC).
The maker of the iPhone, iPad and Mac is integrating OpenAI’s GPT-4o throughout its platforms and processing AI inferences on device, using its own chips and ensuring privacy with a new private cloud compute.
Siri will become an AI agent that can do tasks across apps; text and image generation will be available in several apps as well. Most of this generative AI processing of large language and diffusion models will be on the device instead of at third-party data centers.
Organized under the umbrella term of ‘Apple Intelligence,’ Apple said its approach is different in that it uses AI tools to understand a user and act on their needs. It calls these tools the ‘personal intelligence system.’
In contrast, competitors such as Microsoft offer AI capabilities (Copilot) that generate text, images and code trained on data from the internet and fine-tuned with domain- or enterprise-specific information instead of the user.
“There are already some really impressive chat tools out there that perform a vast array of tasks using world knowledge,” said Craig Federighi, senior vice president of software engineering. “But these tools know every little about you, or your needs.”
Apple’s approach “draws on your personal context to give you intelligence that’s most helpful and relevant for you,” Federighi said. New capabilities coming in iOS 18, iPadOS 18 and macOS Sequoia harness generative AI.
“This is AI for the rest of us,” Federighi said.
Apple is rolling out a beta this fall for iOS 18, iPad OS 18 and Mac OS Sequoia users with more features coming by next year.
What does this partnership mean for OpenAI’s strategic alliance with Microsoft, an Apple rival? In a post on X, OpenAI said its relationship with Microsoft remains “unchanged. … All pre-training of our frontier models continues to happen on supercomputers built in partnership with Microsoft.”
AI agents and new ways to prompt
Notable among the new capabilities is a beefed-up Siri, which is integrating with OpenAI’s ChatGPT (GPT-4o) to make its responses more natural, deeper, and generative (such as asking Siri to tell a story).
It can remember conversations so users don’t have to repeat themselves. Apple said the use of GPT-4o will be free. OpenAI’s paying users can connect their accounts and access paid features from the Apple device.
Siri also will act as an agent by executing one task or a series of tasks for the user.
For example, users can ask Siri to find a “photo of mom in Paris” and it will scour the Photos app. To fill out a form that needs a driver’s license, Siri can find a photo of the user’s license, extract the license number, and type it into the form. These capabilities are part of its App Intents framework, whose API will be available to developers.
With the new OS, Siri will accept text instructions, not just voice commands.
Coming soon also is screen awareness for Siri so it can understand and take action with what it ‘sees’ on screen.
Text generation is coming to Apple as well. New writing tools will help users compose, proofread and summarize text across Mail, Notes, Safari, Pages, Keynote and third-party apps.
Image generation is coming to Messages, Notes, Freeform, Keynote, Pages and other apps, enabling users to create bespoke emojis, stickers or illustrations – for example, generating an image of a friend surrounded by cake, balloons and flowers with a ‘Happy Birthday’ greeting.
Notably, Apple has introduced a new way to prompt. For example, with Image Playground, users can choose items in different categories such as themes, costumes, accessories, places and more. Add them to the playground and generative AI will combine them together in an image.
Image Wand in the Notes app can turn a rough sketch into a polished image or even create an image from reading the text in the Note. Just circle an empty space in the document and the AI will generate an image.
Developers can tap these capabilities through an updated SDK with the new APIs and frameworks. “We’re bringing generative intelligence to Xcode (Apple’s development platform) for developing apps using Swift and SwiftUI with features like on-device code completion and smart assistants for Swift coding questions,” Federighi said.
Private Cloud Compute
Typically, generative AI is processed in data centers due to the extensive power required to do so. For Apple, the processing takes place on the device using Apple’s own powerful chips: The A17 Pro and M family of chips.
But Federighi said sometimes larger models are needed for a task. Tapping outside servers at data centers is a solution, but they often store the data and use it in ways the user did not intend. Even if they say they are not misusing the data, there is no way to verify their claims, according to Apple.
To this end, Apple is rolling out ‘Private Cloud Compute.’ This consists of a cluster of servers running on Apple chips that offer privacy and security. When a user makes a request, Apple determines whether the task can be processed on device or it needs to tap the servers. If the latter, Apple will send only the data needed for the task.
“Your data is never stored or made accessible to Apple,” Federighi said. “It’s used exclusively to fulfill your request.”
Independent experts can inspect the code that runs on the servers to verify Apple’s privacy promise, the company said.
“We’re excited to partner with Apple to bring ChatGPT to their users in a new way,” said OpenAI CEO Sam Altman, in a blog post. “Apple shares our commitment to safety and innovation, and this partnership aligns with OpenAI’s mission to make advanced AI accessible to everyone. Together with Apple, we’re making it easier for people to benefit from what AI can offer.”
What else is new
Apple finally caved in and added a calculator to the iPad, which had long been a pain point for users since they were forced to use third-party apps.
“Yay!” The Apple presentation added this audio cheer, knowing how long iPad users have waited for this app.
Apple had long said it would only add a calculator if it could enhance its capabilities. And today, it did so. With the Apple pencil, users can do math calculations by hand. The app will automatically calculate the answer.
In Messages, a new feature is the ability to schedule texts. Messages can also be bolded, italicized or underlined. In areas with no or poor connectivity, text messages can be sent via satellite in iPhones that have this capability.
Users will soon be able to change the appearance of Home screen apps, the Control Center apps can be customized, and Airpods will now recognize nods as ‘yes’ and head shakes as ‘no’ for situations where speaking in a call may not be feasible. Notes and Phone apps will be able to record, transcribe and summarize audio.
Apple Vision Pro’s visionOS 2 comes with new capabilities, too: Conventional photos can be made into spatial photos, there are new hand gestures to access Home View and other menus, the headset now has train travel mode, among other enhancements. Apple said there are over 2,000 spatial apps designed for its AR/VR headset.
Availability
The iOS 18 version for developers is available today through the Apple Developer Program, and a public beta will be available in July at beta.apple.com. The official release of iOS 18 is this fall as a free software update for iPhone X and later.
However, the AI features Apple introduced under Apple Intelligence is only for the iPhone 15 Pro, iPhone 15 Pro Max, and iPad and Mac with M1 and newer chips. These features are coming this fall. Other iPhone, iPad or Mac owners will get non-Apple Intelligence updates in iOS 18.
Updated on June 12, 2024 to add OpenAI’s clarification about the Apple deal’s impact on its relationship with Microsoft.
