Apple just shipped theirs the first Apple Intelligence features and launched new AI-optimized Macs. But for all the AI ​​hype, there are clearly limitations with the technology’s intelligence. And one of these limits was highlighted by Apple AI research through a recent experiment.

Testing AI capabilities

Last month, a team of Apple researchers published a new work about a key limitation of AI.

Michael Hiltzik writes for The Los Angeles Times:

See if you can solve this arithmetic problem:

Oliver picks 44 kiwis on Friday. Then pick 58 kiwis on Saturday. On Sunday, he harvests double the number of kiwis he did on Friday, but five of them were slightly smaller than average. How many kiwis does Oliver have?

If you answered “190”, congratulations: you did as well as a regular school kid, succeeding well. (Friday’s 44 plus Saturday’s 58 plus Sunday’s 44 multiplied by 2, or 88, equals 190.)

You also outperformed over 20 state-of-the-art AI models tested by an AI research team at Apple. They found that the AI ​​bots were consistently wrong.

The research paper explains that the best and brightest LLM models experienced “catastrophic performance declines” when they tried to answer simple math problems that were written that way.

It happened primarily when those problems included irrelevant data, which even schoolchildren quickly learn to ignore.

Thus, calling into question the current intelligence capabilities of AI.

Apple’s AI research shows that “intelligence” is not what it seems

Due to the variety of tests Apple’s AI research involved, the paper concludes that current AI models are “not capable of genuine logical reasoning”.

Which might be something we’re generally aware of, but it’s an important cautionary note as more and more trust is placed in AI “intelligence”.

AI optimists might assume the problem is an easy fix, but Apple’s team disagreement. “Can scaling data, models or computing fundamentally solve this? We don’t believe it!”

Ultimately, Apple’s work isn’t meant to dampen enthusiasm for AI’s capabilities, but rather to provide a measure of common sense.

Artificial intelligence can perform some tasks as if it were highly intelligent, but in many ways this “intelligence” is not what it might seem.

What do you think about Apple’s AI breakthroughs? Tell us in the comments.

The best accessories for iPhone, iPad, Mac and more