A Way to Reduce the Cost of Training Artificial Intelligence Models That Cost Hundreds of Billions

Recently, large-scale artificial intelligence (AI) models such as ChatGPT and DeepSeek have been utilized in various fields and are attracting attention.



Large-scale language models are trained in large-scale distributed systems equipped with tens of thousands of GPUs for data centers. In the case of GPT-4, the cost of training the model is estimated to be approximately 140 billion won.



A domestic research team has developed a technology that helps derive the optimal parallelization configuration that can increase GPU utilization and reduce training costs.



Changes in MT-NLG training time and GPU utilization according to various parallelization techniques. [Photo = KAIST]



The Korea Advanced Institute of Science and Technology (KAIST, President Lee Kwang-hyung) announced on the 13th that the research team of Professor Yoo Min-soo of the Department of Electrical and Electronic Engineering, in collaboration with Samsung Electronics' Samsung Advanced Institute of Technology, has developed a simulation framework (vTrain) that can predict and optimize the training time of large-scale language models (LLMs) in large-scale distributed systems.



Finding the optimal distributed training strategy is essential to increasing the training efficiency of large-scale language models. Not only are there a huge number of possible strategies, but it takes a lot of time and money to test the performance of each strategy in a real environment.



Currently, companies that train large-scale language models are only using a small number of empirically verified strategies. This leads to inefficiency in GPU utilization and unnecessary cost increases. The lack of simulation technology for large-scale systems is preventing companies from effectively solving the problem.



The KAIST research team developed vTrain to accurately predict the training time of large-scale language models and to quickly explore various distributed parallelization strategies.



The research team compared the actual training time values ​​of various large-scale language models in an actual multi-GPU environment with the predicted values ​​of vTrain and verified that the training time can be predicted with an accuracy of 8.37% in a single node (MAPE) and 14.73% in multiple nodes.



The research team conducted joint research with Samsung Electronics' Samsung Advanced Institute of Technology and released the vTrain framework and over 1,500 actual training time measurement data as open source so that AI researchers and companies can freely utilize them.



Professor Yoo Min-soo said, "vTrain is a profiling-based simulation technique that explores a learning strategy that can increase GPU utilization and reduce learning costs compared to existing empirical methods, and we have released the open source." He added, "This will allow companies to efficiently reduce the cost of learning ultra-large AI models."



The results of this research (paper title: vTrain: A Simulation Framework for Evaluating Cost-Effective and Compute-Optimal Large Language Model Training), in which Ph.D. candidate Bang Je-hyun participated as the first author, were presented in November at the IEEE/ACM International Conference on Microarchitecture (MICRO), one of the top academic conferences in the field of computer architecture.





https://www.inews24.com/view/blogger/1822748

댓글

댓글 쓰기

이 블로그의 인기 게시물

Livestock Manure Methane Is Soaring, But 'Resource Recovery' Isn't Working [Now is a Climate Crisis]

이미지
It has been pointed out that the methane emissions from the livestock waste treatment process were actually reported to be 2.5 times smaller than the previous calculation standards. It is expected that livestock waste will increase rapidly in the future, but an analysis report has been released that 'biogasification', which turns it into a resource, is only at the 10% level. It is urgent to come up with countermeasures as methane, one of the greenhouse gases, is expected to increase rapidly. According to the analysis of the entire livestock waste treatment process for the first time in Korea, the largest source of methane emissions was pigs (73%), and only 14% of pig waste was used for biogas in the public sector. It has been pointed out that it is urgent to come up with countermeasures as methane from livestock waste is increasing rapidly but 'resource conversion' is insufficient. [Photo = Climate Solution] About half of domestic methane emissions come ...
자세한 내용 보기

Large and small launch vehicle companies unite [Now in space]

이미지
The Korean launch vehicle Nuriho will be launched for the fourth time in November. Nuriho plans to build technological sophistication through its sixth launch in the future. [Photo = Korea Aerospace Research Institute] Large and small launch vehicle companies have joined forces to develop future space transportation technology. Perigee Aerospace (Perigee) signed a memorandum of understanding (MOU) with Hanwha Aerospace on the 26th to establish a cooperative system between the two companies for the development of space transportation technology. Through this agreement, the two companies plan to share technological and sales capabilities to strengthen space launch vehicle development capabilities and continue strategic cooperation. The Hanwha Aerospace delegation, which visited the Perigee Rocket Development Complex (RDC) in Okcheon, Chungcheongbuk-do, signed the business agreement and toured the development status of the space launch vehicle 'Blue Whale 1 (BW1)' cu...
자세한 내용 보기

When a star swallows a planet... [Now in space]

이미지
What happens when a star swallows a planet? The James Webb Space Telescope has released the results of observing a star 12,000 light-years away from Earth using its observation equipment. NASA reported the news on the 10th (local time). In conclusion, the star did not swell enough to swallow the planet as previously hypothesized. Instead, the planet's orbit gradually approached the star and was absorbed by the star. As the planet's orbit gradually got closer to the star, the planet was finally swallowed by the star. [Photo = NASA, ESA, CSA, R. Crawford (STScI)] The process of a star swallowing a planet revealed by NASA was that a planet the size of Jupiter first orbited very close to the star, closer than Mercury orbiting the Sun. Then, the planet's orbit gradually decreased over time. The planet gradually approached the star. As it approached the star, the planet began to brush against the star's atmosphere. As the planet fell in...
자세한 내용 보기