2023/9/25

• Upstage and NIA jointly build a Korean-style Open LLM leaderboard... Expanding the scope of the AI ecosystem, including 1T Club

• ‘Open-Ko LLM Leaderboard’ launched this month to prove the performance and innovation of domestic LLMs using AI HUB data

• “We will strive to increase domestic LLM performance and diversity and expand the artificial intelligence ecosystem with the Korean-style Open LLM leaderboard.”

(Upstage = 2023/9/25) Upstage and the National Information Society Agency (NIA) are joining forces to create a leaderboard that can evaluate and compare the performance of Korean LLM to improve Korea's AI competitiveness.

Upstage (CEO Seong-Hoon Kim) announced that it will establish the 'Open Ko-LLM Leaderboard', a Korean LLM leaderboard, co-hosted with NIA and open on September 27th. This is the result of the collaboration between Upstage and NIA on the 4th, and Upstage plans to continue to expand the scope of the Korean AI ecosystem by adding the OpenLLM leaderboard following the 1T Club.

The Open Ko-LLM Leaderboard is a public platform where anyone can register their own Korean LLM model and compete with other models. Researchers interested in the Open Ko-LLM leaderboard can check detailed information and participate in the Open Ko-LLM leaderboard space on Hugging Face after the 27th.

Upstage and NIA's Open Ko-LLM leaderboard is not a simple translation of the existing data of the OpenLLM leaderboard operated by Hugging Face, but rather builds its own high-quality data that reflects the characteristics and culture of the Korean language, thereby demonstrating its strengths as a Korean-specific leaderboard. have

IN ADDITION, THE 'COMMON SENSE GENERATION' STANDARD, WHICH EXAMINES THE ABILITY TO GENERATE COMMON SENSE, WAS ADDED TO EVALUATE THE HIGH PERFORMANCE AND DIVERSITY OF THE KOREAN LLM MODEL. 'COMMON SENSE GENERATION' IS A DATASET BUILT BY UPSTAGE IN COLLABORATION WITH PROFESSOR LIM HEE-SEOK'S RESEARCH TEAM AT KOREA UNIVERSITY, AND IS COMPOSED OF A QUESTIONNAIRE THAT INCLUDES A WIDE RANGE OF TYPES OF HISTORICAL DISTORTION, HALLUCINATION ERRORS, MORPHOLOGICAL ERRORS, IRREGULAR USAGE ERRORS, AND HATE EXPRESSIONS. THROUGH THIS, WE MEASURE WHETHER THE RESULTS PRODUCED BY AI FOR GIVEN CONDITIONS CAN COMPLY WITH THE GENERAL KNOWLEDGE THAT KOREAN USERS MAY HAVE.

In other words, through this common sense generation standard, cases such as 'King Sejong's MacBook throwing incident', which is considered the most representative example of hallucination in Korea, can be largely prevented, and a more appropriate model for Korean language and history can be evaluated. all.

The Open Ko-LLM Leaderboard is expected to increase the level and competitiveness of Korean LLM research, improve the quantity and quality of Korean language data, and raise the international awareness of Korean LLM. Based on the OpenLLM leaderboard, it is possible to share the results of various researchers and promote joint research and cooperation, which is expected to contribute to improving performance levels and expanding industrial fields.

In addition, by establishing a leaderboard based on Korean data, it is expected to secure transparency and reliability of public research results, increase international awareness, and increase global attention as a starting point for revitalizing research on various languages.

The recently announced collaboration between Upstage and KT also played a role in the establishment of this Open Ko-LLM leaderboard. The two companies will join forces to expand the AI ecosystem and the leaderboard will be operated stably through infrastructure support from KT Cloud.

Upstage's LLM model Solar ranked first in the world on the open LLM leaderboard operated by Hugging Face last August by exceeding ChatGPT's benchmark score. Recently, only four companies' LLMs, including OpenAI ChatGPT, Google Farm, Metarama, and Entropic Claude, were previously listed, and Solar was registered as the main model of Poe, which has become the standard for high-performance models, surprising the global market once again. did.

UPSTAGE WILL BUILD A KOREAN LEADERBOARD. UPSTAGE PLANS TO NOT ONLY DEVELOP A HIGH-QUALITY LLM THAT CAN CAPTURE KOREAN CULTURAL SENTIMENTS BASED ON KOREAN DATA BASED ON 1T CLUB, BUT ALSO CONTRIBUTE TO CREATING AN ECOSYSTEM FOR INDEPENDENT LLM IN KOREA. AM.

Seong-Hoon Kim, CEO of Upstage, said, “Upstage, together with NIA, is very happy to establish the Open Ko-LLM Leaderboard to promote the competitiveness of Korean LLM and further raise the level of research.” He added, “We will continue to share high-quality Korean language data, including through 1T Club, in the future.” “Of course, we will work harder to expand the Korean AI ecosystem and promote development by promoting collaboration through leaderboards and rapid technology dissemination.”

← Back to Newsroom