Just how smart is ChatGPT exactly?

How smart is ChatGPT, really? It’s a question being asked around the world, from boardrooms to classrooms.

Thanks to OpenAI, the company that created the chatbot, we actually have something of an answer: The model’s smart enough to be a lawyer or mathematician, but not a poet or physician.

The firm tested both the GPT-4 and GPT-3.5 editions on various types of examinations and measured the results in percentiles based on how well humans do on the same tests. For example, if ChatGPT scored in the 90th percentile, that means it performed better than 90% of humans who took the same test.

As the graphic from Visual Capitalist shows, the latest version of ChatGPT scored high on the verbal reasoning section of the Graduate Record Examination, which is a standardized test used for US graduate school admissions.

The 4.0 model also aced the bar exam, which tracks with the reports of the chatbot passing such legal exams.

See also: ChatGPT works scarily well, but can we trust it?

However, if lawyers are shaking in their boots about their future job prospects, perhaps the writers of the world needn’t be scared. ChatGPT scored in just the 54th percentile in a writing exam, while it fared even worse in English literature, sitting in the 8th percentile.