“AI still behind humans on complex tasks like competition-level mathematics”

: In the midst of mixed feelings—trepidation, optimism, and bewilderment—surrounding the surge and impact of artificial intelligence (AI), there has been a notable increase in research and studies on AI worldwide.

In this context, the Human-Centered Artificial Intelligence (HAI) at Stanford University released an annual report on AI yesterday. The seventh edition report- ‘Artificial Intelligence Index Report, 2024’ has comprehensively studied the AI trend on multiple fronts ranging from education to accountability, research to development, and economy to medicine. One of the significant messages the report has underlined is ‘AI beats humans on some tasks, but not on all’.

There are ten takeaways in the wide-ranging report that fit 500 plus pages with extensive use of data and comparative analyses. In the first takeaway, it wrote, “AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.” The report is dovetailed in nine chapters focusing- on research and development, technical performance, responsible AI, economy, science and medicine, education, policy and governance, diversity, and public opinion.

AI patent shifting from the US to East Asia and Pacific

Taking reference from the Centre for Security and Emerging Technology, 2023, the report has shown shifting AI patents. Comparing the data from 2010 to 2022, the report showed granted AI patents hugely concentrated in East Asia and Pacific – 75.2 per cent, which is trailed by North America with 21.2 percent. North America was leading the AI patent till 2011. However, the US is the source of the top AI model in the world. In the last year, 2023, the institutions based in the US produced 61 modes of AI while it is followed by the European Union with 21 models and China with 15 models.

Eightfold rise in AI investment

According to the report, the world has witnessed a massive surge in investment in Generative AI. “Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds,” it stated. The ‘Economy’ chapter of the report has brought forth information on investment, global AI labour demand, wages of computer scientists, adaptation of AI capabilities, corporate activity, robot installation and jobs.

The report mentioned that China is dominating the installation of robots in industries. “China dominates industrial robotics. Since surpassing Japan in 2013 as the leading installer of industrial robots, China has significantly widened the gap with the nearest competitor nation. In 2013, China’s installations accounted for 20.8% of the global total, a share that rose to 52.4% by 2022,” it explained. Similarly, AI governance initiatives are also on the rise, especially in EU countries, the US and China

From AI policy to actions, these are painted as major players in AI governance. Policymakers in the world are not left untouched by AI, the report argues. It is also worth noting to have knowledge of how the tech giants are investing a huge chunk in designing multiple AI models and programmes in the world. The AI Index estimated, “Training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.”

At a time when many tout technology as a non-discriminatory front, the global tech sprint of 2023, ChatGPT, has been found politically biased. The Index has referenced the researchers and stated ‘significant bias in ChatGPT toward Democrats in the US and the Labour Party in the UK.’ Bearing this in mind, the report has alerted over hijacking of political views of users by the ChatGPT when 2024 is a global election year.

So far, 80 countries have put in place the AI Strategies. They include the US, the UK, Bangladesh, India, Thailand, Rwanda, Ethiopia and Israel, among others. Canada was the first country to have an AI strategy- in 2017. It has been argued that AI clinical knowledge has been boosted in recent times.

“Highly knowledgeable medical AI has arrived. Over the past few years, AI systems have shown remarkable improvement on the MedQA benchmark, a key test for assessing AI’s clinical knowledge,” the Index asserts. The Index added that since the benchmark’s introduction in 2019, AI performance on MedQA has increased nearly threefold. AI hiring has also seen upward trends in countries like Australia, Canada, Singapore, and India since 2023.

Co-directors of AI Index, Ray Perrault and Jack Clark observed the report, saying, “As AI rapidly evolves, the AI Index aims to help the AI community, policymakers, business leaders, journalists, and the general public navigate this complex landscape. It provides ongoing, objective snapshots tracking several key areas: technical progress in AI capabilities, the community and investments driving AI development and deployment, public opinion on current and potential future impacts, and policy measures taken to stimulate AI innovation while managing its risks and challenges.”

Several IT and AI experts provided information and expertise for the report and a dozen tech organizations including GitHub, Linkedin, Schwartz Reisman Institute for Technology and Society, Women in Machine Learning, Code.org, and the International Federation of Robotics supplied data.