Upstage-NIA opens ‘Open Ko-LLM Leaderboard’ to strengthen Korean AI competitiveness
2023/9/25Upstage and NIA jointly build a Korean-style Open LLM leaderboard... Expanding the scope of the AI ecosystem, including 1T Club
‘Open-Ko LLM Leaderboard’ launched this month to prove the performance and innovation of domestic LLMs using AI HUB data
“We will strive to increase domestic LLM performance and diversity and expand the artificial intelligence ecosystem with the Korean-style Open LLM leaderboard.”
(Upstage = 2023/9/25) Upstage and the National Information Society Agency (NIA) are joining forces to create a leaderboard that can evaluate and compare the performance of Korean LLM to improve Korea's AI competitiveness.
Upstage (CEO Seong-Hoon Kim) announced that it will establish the 'Open Ko-LLM Leaderboard', a Korean LLM leaderboard, co-hosted with NIA and open on September 27th. This is the result of the collaboration between Upstage and NIA on the 4th, and Upstage plans to continue to expand the scope of the Korean AI ecosystem by adding the OpenLLM leaderboard following the 1T Club.
The Open Ko-LLM Leaderboard is a public platform where anyone can register their own Korean LLM model and compete with other models. Researchers interested in the Open Ko-LLM leaderboard can check detailed information and participate in the Open Ko-LLM leaderboard space on Hugging Face after the 27th.
Upstage and NIA's Open Ko-LLM leaderboard is not a simple translation of the existing data of the OpenLLM leaderboard operated by Hugging Face, but rather builds its own high-quality data that reflects the characteristics and culture of the Korean language, thereby demonstrating its strengths as a Korean-specific leaderboard. have
IN ADDITION, THE 'COMMON SENSE GENERATION' STANDARD, WHICH EXAMINES THE ABILITY TO GENERATE COMMON SENSE, WAS ADDED TO EVALUATE THE HIGH PERFORMANCE AND DIVERSITY OF THE KOREAN LLM MODEL. 'COMMON SENSE GENERATION' IS A DATASET BUILT BY UPSTAGE IN COLLABORATION WITH PROFESSOR LIM HEE-SEOK'S RESEARCH TEAM AT KOREA UNIVERSITY, AND IS COMPOSED OF A QUESTIONNAIRE THAT INCLUDES A WIDE RANGE OF TYPES OF HISTORICAL DISTORTION, HALLUCINATION ERRORS, MORPHOLOGICAL ERRORS, IRREGULAR USAGE ERRORS, AND HATE EXPRESSIONS. THROUGH THIS, WE MEASURE WHETHER THE RESULTS PRODUCED BY AI FOR GIVEN CONDITIONS CAN COMPLY WITH THE GENERAL KNOWLEDGE THAT KOREAN USERS MAY HAVE.
In other words, through this common sense generation standard, cases such as 'King Sejong's MacBook throwing incident', which is considered the most representative example of hallucination in Korea, can be largely prevented, and a more appropriate model for Korean language and history can be evaluated. all.
The Open Ko-LLM Leaderboard is expected to increase the level and competitiveness of Korean LLM research, improve the quantity and quality of Korean language data, and raise the international awareness of Korean LLM. Based on the OpenLLM leaderboard, it is possible to share the results of various researchers and promote joint research and cooperation, which is expected to contribute to improving performance levels and expanding industrial fields.
In addition, by establishing a leaderboard based on Korean data, it is expected to secure transparency and reliability of public research results, increase international awareness, and increase global attention as a starting point for revitalizing research on various languages.
The recently announced collaboration between Upstage and KT also played a role in the establishment of this Open Ko-LLM leaderboard. The two companies will join forces to expand the AI ecosystem and the leaderboard will be operated stably through infrastructure support from KT Cloud.
Upstage's LLM model Solar ranked first in the world on the open LLM leaderboard operated by Hugging Face last August by exceeding ChatGPT's benchmark score. Recently, only four companies' LLMs, including OpenAI ChatGPT, Google Farm, Metarama, and Entropic Claude, were previously listed, and Solar was registered as the main model of Poe, which has become the standard for high-performance models, surprising the global market once again. did.
UPSTAGE WILL BUILD A KOREAN LEADERBOARD. UPSTAGE PLANS TO NOT ONLY DEVELOP A HIGH-QUALITY LLM THAT CAN CAPTURE KOREAN CULTURAL SENTIMENTS BASED ON KOREAN DATA BASED ON 1T CLUB, BUT ALSO CONTRIBUTE TO CREATING AN ECOSYSTEM FOR INDEPENDENT LLM IN KOREA. AM.
Seong-Hoon Kim, CEO of Upstage, said, “Upstage, together with NIA, is very happy to establish the Open Ko-LLM Leaderboard to promote the competitiveness of Korean LLM and further raise the level of research.” He added, “We will continue to share high-quality Korean language data, including through 1T Club, in the future.” “Of course, we will work harder to expand the Korean AI ecosystem and promote development by promoting collaboration through leaderboards and rapid technology dissemination.”
※ Photo Caption: Upstage (CEO Seong-Hoon Kim) announced that it will establish the 'Open Ko-LLM Leaderboard', a Korean LLM leaderboard, co-hosted with NIA and open on September 27th. This is the result of the collaboration between Upstage and NIA on the 4th, and Upstage plans to continue to expand the scope of Korea's AI ecosystem by adding the OpenLLM leaderboard to the 1T Club. The photo shows the Open Ko-LLM leaderboard logo and homepage and examples of common sense generated data.
-
Upstage | Keunkyo Kim, PR Director | keunkyo@upstage.ai Upstage | Sungbeom Bae, PR Manager | sungbae@upstage.ai
Download press release
-
Upstage is a leading domestic AI startup established in October 2020. Upstage stands out in the large language model (LLM) industry by taking first place on the Hugging Face leaderboard with performance exceeding ChatGPT's benchmark score for the first time in OpenLLM history. Based on these technologies, we present a reliable private LLM standard that maximizes data security and solves hallucination, helping companies conveniently use cutting-edge technology. In addition, Upstage's Chat AI 'AskUp' has over 1.4 million users, establishing itself as the largest AI service in Korea. Document AI Pack, another Upstage representative solution, utilizes AI OCR technology that has won the world's most prestigious OCR competition to automate documents by increasing efficiency and accuracy. By optimizing document processing through a pre-trained model with minimal data, cost and time are dramatically minimized compared to manual methods. Lastly, through the education program 'EduStage', we are also actively engaged in the educational content business that fosters differentiated professional talent who can be immediately put into AI business through hands-on education that incorporates AI business experience and solid AI basic education.
Upstage is comprised of members from global big tech companies such as Google, Apple, Amazon, NVIDIA, Meta, and Naver, and has participated in many world-renowned AI academic societies such as NeurlPS, ICLR, CVPR, ECCV, WWW, CHI, WSDM, and DMLR. We are solidifying our unrivaled leadership in AI technology by publishing excellent papers and becoming the only domestic company to win double-digit gold medals in the online AI competition Kaggle. While working as a professor at the Hong Kong University of Science and Technology, Upstage CEO Kim Seong-hoon won the ACM Sigsoft Distinguished Paper Award, the best paper award, four times for his research on bug prediction and automatic source code generation that combined software engineering and machine learning, and won 10 awards at the International Conference on Software Maintenance. He is considered a world-class AI guru who received the most influential paper award in 2018, and is also widely known as an instructor of 'Deep Learning for Everyone' with a total of more than 7 million views. Additionally, Upstage's co-founders include CTO Lee Tal-seok, who led Naver Visual AI/OCR and achieved world-class results, and CSO Park Eun-jung, who led the model team of Papago, the world's best translator.
