Connect with us

Technology

CAIS and Scale AI Unveil Results of “Humanity’s Last Exam,” a Groundbreaking New Benchmark

Published

1 year ago

on

SAN FRANCISCO, Jan. 23, 2025 /PRNewswire/ — The Center for AI Safety (CAIS) and Scale AI today announced the results of a groundbreaking new AI benchmark that was designed to test the limits of AI knowledge and whether the models are capable of chain-of-thought reasoning. The results demonstrated a significant improvement from the reasoning capabilities of earlier models, but current models still were only able to answer fewer than 10 percent of the expert questions correctly.

CAIS and Scale AI Unveil Results of “Humanity’s Last Exam,” a Groundbreaking New Benchmark

The new benchmark, called “Humanity’s Last Exam,” evaluated whether AI systems have achieved world-class expert-level reasoning and knowledge capabilities across a wide range of fields, including math, humanities, and the natural sciences. Throughout the fall, CAIS and Scale AI crowdsourced questions from experts to assemble the hardest and broadest problems to stump the AI models. The exam was developed to address the challenge of “benchmark saturation”: models that regularly achieve near-perfect scores on existing tests, but may not be able to answer questions outside of those tests. Saturation reduces the utility of a benchmark as a precise measurement of future model progress.

“We wanted problems that would test the capabilities of the models at the frontier of human knowledge and reasoning,”  said Dan Hendrycks, CAIS co-founder and executive director. “We can’t predict how quickly the models will advance. When I released the MATH benchmark—a challenging competition mathematics dataset—in 2021, the best model scored less than 10%; few predicted that scores higher than 90% would be achieved just three years later. Right now, Humanity’s Last Exam shows that there are still some expert closed-ended questions that models are not able to answer.  We will see how long that lasts.”

Testing Methodology

Altogether, CAIS and Scale researchers collected more than 70,000 trial questions. That led to a selection of 13,000 questions for human expert review which, in turn, were finalized to a set of 3,000 questions on the final exam’s public release. The questions were aimed at world-class expert levels and were put to several multi-modal, frontier LLMs including OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet, Google Gemini 1.5 Pro, and OpenAI o1.

“We know the AI revolution is being shaped by human ingenuity, and we’re proud to be at the forefront. To help humans measure AI progress, we engineered what might be the ultimate test, meticulously distilled and designed to challenge the world’s most advanced models at the frontiers of intelligence—requiring precise, multi-step logical reasoning and unambiguous answers at a level that pushes even the most sophisticated AI systems to their limits.” Summer Yue, Director of Research at Scale AI said.

Humanity’s Last Exam was a global collaborative effort involving nearly 1,000 contributors from more than 500 institutions across 50 countries, with most contributors being active researchers or professors. The questions spanned multiple formats, including text-only and multi-modal challenges that integrated images and diagrams.

The questions were designed to deeply test the capability of the models across diverse domains. For example, a question submitted in Ecology asked:
“Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.”

Additional sample questions can be found here (lastexam.ai).

In the final round of testing, Yue said they saw some of the models begin to answer a fraction of the questions correctly (less than 10%); however, she said variations frequently happen in model testing and could be the result of randomness. CAIS and Scale AI said they will open up the dataset to the research community, to dig deeper into the variations and to evaluate new AI systems while continuing to explore the limitations of existing models. A small subset of questions will be held back to preserve integrity for future evaluations.

Top Questions

CAIS and Scale AI offered financial awards for the best contributions to Humanity’s Last Exam, with $5,000 USD awarded for each of the top 50 questions and $500 USD for the next 500 best submissions, along with the opportunity for coauthorship of the final paper.

“By identifying the gaps in AI’s reasoning capabilities, Humanity’s Last Exam not only benchmarks current systems but also provides a roadmap for future research and development,” said Yue.

About The Center for AI Safety
The Center for AI Safety (CAIS) is a research organization whose mission is to reduce societal-scale and national security risks from AI. CAIS research focuses on mitigating high-consequence risks in areas like monitoring, alignment, and systemic safety. CAIS works to expand the field of AI safety and security by providing compute resources and technical infrastructure to top researchers and engaging with the global research community. Through its CAIS Action Fund, CAIS advocates for safe and secure AI. CAIS was founded in 2022 and is headquartered in San Francisco.

About Scale AI
Scale AI is the Humanity-first AI Company. Backed by our Data Foundry, we generate high quality data and provide technology solutions that allow our enterprise and public sector customers to build, deploy, and evaluate the smartest AI tools and applications. By making data abundant, rigorous, and high-quality, we are accelerating the progress of AI. Scale AI was founded in 2016 and is headquartered in San Francisco.

Contact:
Richard Crook, richardc@safe.ai
Fiorella Riccobono, fiorella.riccobono@scale.com

View original content to download multimedia:https://www.prnewswire.com/news-releases/cais-and-scale-ai-unveil-results-of-humanitys-last-exam-a-groundbreaking-new-benchmark-302358108.html

SOURCE Center for AI Safety

Related Topics:
Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Technology

Achieve named to Az Business Magazine’s ’10 Best Places for Women to Work in Arizona’ for 2026

Published

29 minutes ago

on

July 23, 2026

By

Recognition highlights the company’s commitment to creating opportunities for women to grow and lead

SAN MATEO, Calif., July 23, 2026 /PRNewswire/ — Achieve, the leader in digital personal finance, has been named among the 2026 10 Best Places for Women to Work in Arizona by Az Business Magazine. The annual recognition highlights organizations that create supportive environments where women can thrive professionally, advance into leadership roles and build meaningful careers.

The honor reflects Achieve’s ongoing investment in workplace programs that support employee growth, flexibility, leadership development and career advancement. Women serve in leadership roles across the organization and play a critical role in shaping the company’s culture, products and long-term success.

“Creating an environment where women can grow, lead and build rewarding careers is central to who we are as a company,” said Achieve Senior Vice President of Human Resources Heather Marcom. “We’re honored to be recognized among Arizona’s top workplaces for women and remain committed to fostering a culture where employees feel supported, valued and empowered to do their best work.”

Achieve maintains a major corporate presence in the Phoenix area, where hundreds of employees contribute to the company’s mission of helping people move from struggling to thriving financially. The company supports employees through leadership development opportunities, employee resource groups, mentorship and learning programs designed to help team members reach their professional goals.

The recognition adds to a growing list of workplace honors for Achieve. Earlier this year, the company was named among the Top 3 Best Workplaces for LGBTQ+ Employees by BestCompaniesAZ and was also recognized by AZ Big Media as one of Arizona’s Most Admired Companies.

“Strong organizations are built by diverse perspectives and inclusive leadership,” said Marcom. “We’re proud of the talented women across Achieve who help drive our business forward every day and grateful for the impact they make on our employees, customers and communities.”

The 10 Best Places for Women to Work in Arizona list is determined through a public voting process conducted by AZ Big Media and published in Az Business magazine.

About Achieve

Achieve, THE digital personal finance company, helps everyday people get on, and stay on, the path to a better financial future. Achieve pairs proprietary data and analytics with personalized support to offer personal loans, home equity loans, debt relief and debt consolidation, along with financial tips and education and free mobile apps: Achieve MoLO® (Money Left Over) and Achieve GOOD™ (Get Out Of Debt). Achieve is frequently recognized for providing top-rated customer experience and satisfaction by both consumers and leading personal finance review platforms and has 2,200 dedicated teammates across the country, with hubs in Arizona, California, Florida and Texas.

Achieve refers to the global organization and may denote one or more affiliates of Achieve Company, including Achieve.com, Equal Housing Opportunity (NMLS ID #138464); Achieve Home Loans, Equal Housing Opportunity (NMLS ID #1810501); Achieve Personal Loans (NMLS ID #227977); Freedom Debt Relief (NMLS ID # 1248929); and Freedom Financial Asset Management (CRD #170229).

Contacts

Austin Kilgore
akilgore@achieve.com
214-908-5097

Elina Tarkazikis
etarkazikis@achieve.com

View original content to download multimedia:https://www.prnewswire.com/news-releases/achieve-named-to-az-business-magazines-10-best-places-for-women-to-work-in-arizona-for-2026-302833720.html

SOURCE Achieve

Continue Reading

Technology

National Press Club Statement on the withdrawal of subpoenas targeting New York Times journalists

Published

29 minutes ago

on

July 23, 2026

By

WASHINGTON, July 23, 2026 /PRNewswire/ — National Press Club President Mark Schoeff Jr. released the following statement:

“The Justice Department’s decision to withdraw subpoenas targeting journalists at The New York Times is a welcome and necessary step to protect the public’s constitutional right to an independent press.

These subpoenas should never have been issued in the first place. Compelling journalists to reveal confidential sources sends a chilling message to those who seek to inform the public and threatens the very foundation of press freedom.

Every American should understand what is at stake when the government turns its investigative powers on journalists. It is not routine. It is an extraordinary intrusion that strikes at the heart of the First Amendment and your right to information about your government.

The greatest danger was not the subpoenas themselves, but the message they sent: That sources could be exposed, that whistleblowers should remain silent, and that the American people might know less about the actions of their own government.

A strong democracy depends on a press that can report freely, hold power to account, and inform the public without intimidation.

We urge continued vigilance to ensure that journalists can do their jobs without interference and that protections for source confidentiality are upheld consistently.”

About the National Press Club

Founded in 1908, the National Press Club is the world’s leading professional organization for journalists and a leading voice for press freedom in the U.S. and worldwide.

Contact: Beth Francesco, Executive Director of the National Press Club Journalism Institute, media@press.org

View original content to download multimedia:https://www.prnewswire.com/news-releases/national-press-club-statement-on-the-withdrawal-of-subpoenas-targeting-new-york-times-journalists-302833722.html

SOURCE National Press Club

Continue Reading

Technology

NCC Launches NCC Connect™ to Put Credit, Fraud, and Compliance Inside the CRM Dealers Already Use

Published

29 minutes ago

on

July 23, 2026

By

A powerful new solution that embeds credit access, fraud detection, and compliance directly into the dealership’s existing CRM — removing the system-switching that slows deals and exposes dealers to risk.

AUSTIN, Texas, July 23, 2026 /PRNewswire-PRWeb/ — NCC, a leading provider of credit and compliance solutions for automotive dealerships, today announced the launch of NCC Connect™, a powerful platform that runs credit, fraud, and compliance from inside the CRM a dealership already uses — without friction or duplicate entry.

“Dealers don’t need another system to log into,” said Brian Skutta, President and CEO of NCC. “They need the tools they already have to work better together. NCC Connect puts credit, fraud detection, and compliance right where the team already works — inside the CRM they use every day.”

“Dealers don’t need another system to log into,” said Brian Skutta, President and CEO of NCC. “They need the tools they already have to work better together. NCC Connect puts credit, fraud detection, and compliance right where the team already works — inside the CRM they use every day.”

As deals grow more complex and fraud more sophisticated, dealers are juggling more disconnected systems than ever — the CRM, the credit system, the compliance tools — switching between them on every transaction. Each switch breaks momentum, invites a skipped step, and slows the path to funding. NCC Connect meets this moment with a single, seamless solution that keeps the full credit, fraud, and compliance engine right where the team already works.

Why NCC Connect Matters Right Now

Dealers lose time and margin switching between the CRM, credit, and compliance systems on every dealAuto lending fraud continues to climb, with industry fraud exposure reaching a record $10.4 billion in 2025, according to Point Predictive’s 2026 Auto Lending Fraud Trends ReportState compliance is tightening, with laws like California’s SB 766 (CARS Act) taking effect October 1, 2026Every disconnected step is another chance for an error, a delay, or a deal that stalls before funding

These pressures are forcing dealers to consolidate, and NCC Connect delivers the edge.

Product Highlights:

Inside the CRM — Soft-pull and hard credit access from all three major bureaus — Experian, TransUnion, and Equifax — without leaving the workflowFraud & Identity Built In — Identity verification and synthetic fraud detection delivered within the credit pull, flagging Red Flag conditions before the deal moves to fundingCompliance on Autopilot — FCRA and FTC controls with automatic, audit-ready documentation stored in the deal record99.99% Uptime — The industry’s highest, so the platform is there when a deal is on the desk

NCC Connect runs soft-pull pre-qualifications and hard credit pulls from any bureau or score model without leaving the CRM, while customer data stays inside the existing CRM structure. Every credit, fraud, and compliance result is captured on the deal record — giving dealers a single, audit-ready source of truth and a faster, cleaner path to funding.

NCC Connect extends the same powerful, credit-first engine behind NCC’s Complete Credit™ platform into the CRM where dealers already work. For dealers, that means more approvals, stronger fraud protection, and faster funding, without changing how the team works.

Learn more about NCC Connect at https://nccdirect.com/ncc-connect/

About NCC:

With offices in Austin, TX, Bettendorf, IA, and Las Vegas, NV, NCC has been a trusted partner in credit-driven retailing for automotive dealerships for nearly three decades. We combine a powerful credit and compliance engine with a fully integrated Desking platform to drive maximum profitability. Our focus on innovation, user-friendly products, and dependable systems — supported by a dedicated account management team — has solidified our reputation as a leader in the industry. www.nccdirect.com

Media Contact

Holly Smith, NCC, 1 8285732722, hsmith@nccdirect.com, https://nccdirect.com/

View original content:https://www.prweb.com/releases/ncc-launches-ncc-connect-to-put-credit-fraud-and-compliance-inside-the-crm-dealers-already-use-302832007.html

SOURCE NCC

Continue Reading

Trending