Apple’s Latest Research on the Limitations of AI Language Models

Apple study exposes deep cracks in LLMs reasoning capabilities

Contract analysis today is a tedious process fraught with the possibility of human error. Lawyers must painstakingly dissect agreements, identify conflicts and suggest optimizations — a time-consuming task that can lead to oversights. Neuro-symbolic AI could addresses this challenge by meticulously analyzing contracts, actively identifying conflicts and proposing ChatGPT optimizations. By breaking down problems systematically, o1 mimics human thought processes, considering strategies and recognizing mistakes. This ultimately leads to a more sophisticated ability to analyze information and solve complex problems. Additionally, o1 showcases elements of agentic AI, where systems can act independently to achieve goals.

We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data. Apple’s recent research paper, provides a critical analysis of the reasoning capabilities in current large language models (LLMs). Challenging the widespread belief that these models possess genuine logical reasoning abilities, revealing instead a significant reliance on pattern recognition. These findings have far-reaching implications for the practical applications of LLMs and the future development of artificial intelligence. Imagine a world where AI is seamlessly integrated into critical areas like education and healthcare, making decisions that impact our daily lives. However, what if these systems falter when faced with unfamiliar situations or irrelevant details?

BGR’s audience craves our industry-leading insights on the latest in tech and entertainment, as well as our authoritative and expansive reviews. Chris Smith has been covering consumer electronics ever since the iPhone revolutionized the industry in 2008. When he’s not writing about the most recent tech news for BGR, he brings his entertainment expertise to Marvel’s Cinematic Universe and other blockbuster franchises. Interestingly, I tested the same problem with o1-preview, and ChatGPT was able to reason that all fruits are countable despite their size. They also gave the AIs math problems that included statements that were not really relevant to solving the problem. It will undoubtedly become crucial for lawyers to master AI tools, but these tools are most effective when wielded by those with uniquely human strengths.

Top News

As a result of these concerns, Iranians have increasingly been converting most of their savings into US dollars or gold. (MENAFN- Asia Times)

As the world absorbed news of Donald Trump’s comeback victory in the 2024 US presidential race, concern in Iran turned to the impact of the election on its own Economy amid escalating regional tensions. This project is expected to support other partnerships between the two countries in the energy sector that align with the joint strategy in this field. It is worth mentioning that if the current relations between the two countries enjoy increasing momentum, they are not historically new. They are rooted in the depths of history, as Algeria and Türkiye share distinctive friendly, civilizational and political ties, and the history of the North African and Mediterranean region is replete with great achievements and heroic moments shared by the two countries.

AlphaGeometry’s success underscores the broader potential of neuro-symbolic AI, extending its reach beyond the realm of mathematics into domains demanding intricate logic and reasoning, such as law. Just as lawyers meticulously uncovered the truth at Hillsborough, neuro-symbolic AI can bring both rapid intuition and careful deliberation to legal tasks. Lawyers frequently depend on quick judgments to assess cases, but detailed analysis is equally important, mirroring how thinking slow was vital in uncovering the truth at Hillsborough. An OpenAI researcher, while commending Mirzadeh and colleagues’ work, objected to their conclusions, saying that correct results could likely be achieved in all these failure cases with a bit of prompt engineering.

This methodical analysis — Kahneman’s “System 2 (slow)” thinking — finally exonerated the fans. The rial fell to a fresh record low as Donald trump was claiming victory – trading above the symbolic marker of 700,000 rials to the dollar, according to traders in Tehran , just as results of the US election were coming in. One of the most visible areas of this strategic cooperation is the economic sector. Hence, Türkiye has become Algeria’s fifth-largest trading partner, and Algeria has become Türkiye’s second-largest partner on the African continent. Apple isn’t going after rivals here; it’s simply trying to determine whether current genAI tech allows these LLMs to reason. Dr. Hinton, often called the godfather of AI, warns that as AI systems begin to exceed human intellectual abilities, we face unprecedented challenges in controlling them.

Reshaping AI Development Strategies

Such sensitivities could severely hinder the models’ application in dynamic real-world environments, where data is rarely static or predictable. It serves as a bridge between Kahneman’s concepts of thinking fast and thinking slow, aiming to deliver better reasoning with symbolic ai example fewer mistakes. This approach paves the way for more advanced systems like AlphaGeometry that truly merge neural and symbolic approaches. OpenAI’s o1 model is not technically neuro-symbolic AI but rather a neural network designed to “think” longer before responding.

• These are not well-defined concepts, and the questions tend to appear at the bleeding edge of AI research, where the state of the art changes on a daily basis.
• This observation is consistent with the other qualities often attributed to LLMs due to their facility with language.
• This approach helps avoid any potential “data contamination” that can result from the static GSM8K questions being fed directly into an AI model’s training data.
• However, as Kahneman suggests, “Nothing in life is as important as you think it is while you are thinking about it.” Taking a moment for deliberate reflection, we might realize that perhaps the transformation isn’t as earth-shattering as it seems — or perhaps it is.
• Ranking among the largest textile factories in Africa, it is currently exporting its products to several countries in Africa, Europe and Latin America.

Symbolic AI relies on explicit rules and logic to process information and make decisions, as … Unlike neural networks, symbolic AI systems solve problems through step-by-step reasoning based on clear, interpretable pathways. In contrast to the intuitive, pattern-based approach of neural networks, symbolic AI operates on logic and rules (“thinking slow”). This deliberate, methodical processing is essential in domains demanding strict adherence to predefined rules and procedures, much like the careful analysis needed to uncover the truth at Hillsborough.

iPhone 17: Release date, rumors, features, A19, price, and Slim model

The result should be identical in both cases, but the LLMs subtracted the smaller kiwis from the total. Apparently, you don’t count the smaller fruit if you’re an AI with reasoning abilities. The Apple scientists showed that the average accuracy dropped by up to 10% across all models when dealing with the GSM-Symbolic test. Some models did better than others, with GPT-4o dropping from 95.2% accuracy in GSM9K to 94.9% in GSM-Symbolic. In the realm of legal precedent analysis, it could grasp underlying legal principles, make nuanced interpretations and more accurately predict outcomes. The result would be a more context-aware and logically coherent evaluation, enhancing the quality of legal decision-making.

He compares the impact of AI to that of the Industrial Revolution, emphasizing the necessity for careful oversight. By comprehending the logical interdependencies within agreements, it proposes structures that seamlessly align with both legal requirements and business objectives. This story originally appeared on Ars Technica, a trusted source for technology news, tech policy analysis, reviews, and more. World and Middle East business and financial news, Stocks, Currencies, Market Data, Research, Weather and other data. The concern is that without this pressure from Washington, Israel will carry out more military operation in Iran . In addition, many Iranians are worried that Trump may give Israel a green light to attack oil assets and Iranian infrastructure – and that would be even more costly to Iran’s economy.

This meticulous, rule-based approach ensures each step is executed according to established guidelines. These are not well-defined concepts, and the questions tend to appear at the bleeding edge of AI research, where the state of the art changes on a daily basis. They want to minimize the impact that Trump’s victory may have on their economy and are trying to reassure the domestic market.

The research suggests that simply scaling up data, models, or computational power may not address these fundamental reasoning limitations. For AI to progress beyond sophisticated pattern recognition, new approaches are necessary. This insight is crucial for developing models that can achieve true logical reasoning, a capability vital for their effective deployment across various fields.

The ties between the two countries have witnessed remarkable development at various levels and have remarkably accelerated since 2020. That said, it’ll be interesting to see how OpenAI, Google, Meta, and others challenge Apple’s findings in the future. Perhaps they’ll devise other ways to benchmark their AIs and prove they can reason. If anything, Apple’s data might be used to alter how LLMs are trained to reason, especially in fields requiring accuracy. Apple researcher Mehrdad Farajtabar has a thread on X that covers the kind of changes Apple performed for the new GSM-Symbolic benchmarks that include additional examples. This caution is echoed by John J. Hopfield and Geoffrey E. Hinton, pioneers in neural networks and recipients of the 2024 Nobel Prize in Physics for their contributions to AI.

The fragility highlighted in these new results helps support previous research suggesting that LLMs use of probabilistic pattern matching is missing the formal understanding of underlying concepts needed for truly reliable mathematical reasoning capabilities. “Current LLMs are not capable of genuine logical reasoning,” the researchers hypothesize based on these results. “Instead, they attempt to replicate the reasoning steps observed in their training data.” The fragility highlighted in these new results helps support previous research suggesting that LLMs’ use of probabilistic pattern matching is missing the formal understanding of underlying concepts needed for truly reliable mathematical reasoning capabilities. [W]e investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases.

Adding in these red herrings led to what the researchers termed “catastrophic performance drops” in accuracy compared to GSM8K, ranging from 17.5 percent to a whopping 65.7 percent, depending on the model tested. These massive drops in accuracy highlight the inherent limits in using simple “pattern matching” to “convert statements to operations without truly understanding their meaning,” the researchers write. Despite the challenges facing bilateral relations, the future prospects look promising, as both countries enjoy many advantages that qualify them to play an effective role in their respective regions. Algeria expects a GDP of $400 billion by mid-2026, with a growth rate of more than 4%.

AllegroGraph offers a comprehensive solution platform including Large Language Models (LLMs), Vector generation and storage, Graph Neural Networks, Graph Virtualization, GraphQL, Apache Spark graph analytics, and Kafka streaming graph pipelines. You can foun additiona information about ai customer service and artificial intelligence and NLP. These capabilities exemplify AllegroGraph’s leadership in empowering data analytics professionals to derive substantial business value from Knowledge Graphs.

In tests, AlphaGeometry solved 83% of International Mathematical Olympiad geometry problems, matching o1’s performance and nearly reaching that of human gold medalists. According to OpenAI, o1 “performs similarly to PhD students on challenging benchmark tasks in physics, chemistry and biology.” In a mock qualifying exam ChatGPT App for the International Mathematics Olympiad, o1 correctly solved 83% of the problems — a dramatic improvement over GPT-4’s 13% success rate. Similarly, tax preparation software like TurboTax and H&R Block rely heavily on symbolic AI to navigate the intricate web of legal regulations and ensure accurate calculations.

AllegroGraph is at the forefront of Neuro-Symbolic AI, a technology that uniquely integrates Machine Learning (Neuro AI) with knowledge and reasoning (Symbolic AI). This innovative approach sets a new benchmark in intelligent computing, ensuring AI reasoning is both contextually relevant and factually accurate. By leveraging Knowledge Graphs, AllegroGraph empowers organizations to harness AI insights for critical decision-making with unparalleled confidence and trust.

Building the AI-Powered Future: The Road Ahead for Knowledge Management

And although it can follow complex chains of reasoning it has been exposed to before, the fact that this chain can be broken by even superficial deviations suggests that it doesn’t actually reason so much as replicate patterns it has observed in its training data. Replacing the name with something else and changing the numbers should not alter the performance of reasoning AI models like ChatGPT. After all, a grade schooler could still solve the problem even after changing these details. The ability to reason accurately and consistently is essential for AI applications in critical areas such as education, healthcare, and decision-making systems. Understanding the limitations of LLMs’ reasoning capabilities is crucial for making sure AI safety and alignment with human values.

Given the long shared history between the two countries and the deep civilizational ties between them, the cultural aspect of this relationship had to be considered. In response to the wishes of the two peoples, the two presidents have agreed to reciprocally open cultural centers in Algiers and Istanbul. They also recognized the importance of working together in the field of Ottoman archives to explore and document the common history and deepen mutual understanding of the common past.

Washington, for example, has not prevented Iran’s ongoing indirect oil exports to China in recent years . In addition, Iran’s leaders have been directing more and more of the country’s oil revenue toward defense. They recently announced a planned increase in military expenditure of 200% , and some members of the ruling elite have called for setting the defense budget as a fixed share of gross domestic product to ensure adequate funding for military priorities. The Iranian economy was already in a perilous state due in large part to the ongoing impact of US-led sanctions on Tehran and ongoing anxiety over the conflict in the Middle East.

But this might not prove effective – and we might see even more devaluation of Iranian currency in the coming weeks. As an example of the fruitful partnership between the two countries, the Tosyalı Iron and Steel Complex, whose investments in Algeria amounted to $2.7 billion, is one of the leading steel manufacturers in Africa and internationally. It is now exporting its products from Algeria to 25 countries and is currently working to expand its activities to other areas. The Algerian-Turkish company Tayal is another example of a fruitful partnership, as it has achieved remarkable success since its establishment in 2018. Ranking among the largest textile factories in Africa, it is currently exporting its products to several countries in Africa, Europe and Latin America. Algerian-Turkish relations are a role model of this, so partnership and cooperation are welcomed.

• A key finding of the research is the models’ sensitivity to irrelevant information.
• In addition, Iran’s leaders have been directing more and more of the country’s oil revenue toward defense.
• The scientists developed a version of the GSM8K benchmark, a set of over 8,000 grade-school math word problems that AI models are tested on.

This is just a simple example out of hundreds of questions that the researchers lightly modified, but nearly all of which led to enormous drops in success rates for the models attempting them. Apple’s study, available as a pre-print version at this link, details the types of experiments the researchers ran to see how the reasoning performance of various LLMs would vary. They looked at open-source models like Llama, Phi, Gemma, and Mistral and proprietary ones like ChatGPT o1-preview, o1 mini, and GPT-4o. This innovative approach, merging the precision of symbolic AI with the adaptability of neural networks, offers a compelling solution to the limitations of existing legal AI tools.

We’re likely seeing a similar “illusion of understanding” with AI’s latest “reasoning” models, and seeing how that illusion can break when the model runs in to unexpected situations. Second, we praise the current determination of our beloved people and its ambitious youth, who are carrying the torch of completing the national march toward a new Algeria, great in its potential, and the genius of its daughters and sons, strong and proud of their national history. Algeria is determined to achieve the highest levels of socio-economic development through the mobilization of resources and building strong partnerships with friendly countries based on common views and mutual interests. The scientists developed a version of the GSM8K benchmark, a set of over 8,000 grade-school math word problems that AI models are tested on. Called GSM-Symbolic, Apple tests involved making simple changes to the math problems, like modifying the characters’ names, relationships, and numbers. This is where neuro-symbolic AI comes into play — a hybrid approach that blends the strengths of neural networks (intuition) with the precision of symbolic AI (logic).