A joint project between the Qazaq Tili international association and OpenAI has produced a specialized AI evaluation benchmark built around the linguistic and cultural characteristics of Kazakh, rather than translated from English. The work underscores a broader challenge for large language models: understanding a language requires understanding the cultural context behind it.
The benchmark evaluates models on grammar, naturalness of speech, proverbs and fixed expressions, academic and literary translation, children's literature, safety, and ethnography. A separate 500-question benchmark tests knowledge of Kazakh history, traditions, and culture. Rauan Kenzhekhanuly, president of the Qazaq Tili international association, said during a Sept. 3 presentation that more than 10 billion tokens were initially collected from archives, museums, libraries, and other sources after obtaining the necessary permissions.
Building the data foundation
The Kazakh Text Corpus has reached 14 billion tokens, spanning different periods of language development and diaspora heritage. It covers education, science, technology, economics, law, medicine, history, ethnography, media, and children's content. Materials include text, audio, and images. The goal is to improve ChatGPT's performance in Kazakh across translation, information retrieval, speech, text and audio recognition, while reducing culturally inaccurate responses.
For developers working with Generative AI and LLM systems, the project illustrates what dataset construction looks like when it starts from a language's own structure rather than being filtered through English. The effort also complements Kazakhstan's broader cooperation with OpenAI in education. Valerie Focke, OpenAI's lead for education across Europe, the Middle East and Africa, described Kazakhstan as "a pioneer and early mover" in AI adoption and said the country was among the first globally to collaborate with OpenAI on the future of education.
Why language is not just translation
Futurist and author Ron Immink told The Astana Times that AI development must account for cultural context. "I think it is also going to be about context and culture. And I think we are definitely missing a trick. The problem is that most AIs are in English, but it should be in Navajo. It should be in hundreds of languages that we have. So it gets a much more deeper context," he said.
Kazakhstan's work demonstrates what this means in practice. Developing AI in a language is not a translation exercise. It is an effort to preserve the linguistic and cultural context carried by that language. The benchmarks were developed originally in Kazakh, testing concepts like proverbs and fixed expressions that have no direct English equivalent.
Why this matters for IT and development professionals
For software developers and engineers building on top of LLMs, the Kazakh initiative highlights a concrete problem: models trained predominantly on English-language data produce shallow or inaccurate outputs in other languages, especially when cultural context matters. The solution involves constructing evaluation suites and training corpora in the target language from the start. Teams working on multilingual applications or localization should expect to invest in language-specific benchmarks and native-speaker data collection rather than relying on post-hoc translation. The approach also opens opportunities for developers who understand AI for Software Developers to specialize in dataset engineering for underrepresented languages, a skillset that will grow as demand for culturally competent AI increases.
Your membership also unlocks:
