Pre-trained models: Past, present and future

Han, Xu; Zhang, Zhengyan; Ding, Ning; Gu, Yuxian; Liu, Xiao; Huo, Yuqi; Qiu, Jiezhong; Yao, Yuan; Zhang, Ao; Zhang, Liang; Han, Wentao; Huang, Minlie; Jin, Qin; Lan, Yanyan; Liu, Yang; Liu, Zhiyuan; Lu, Zhiwu; Qiu, Xipeng; Song, Ruihua; Tang, Jie; Wen, Ji-Rong; Yuan, Jinhui; Xin Zhao, Wayne; Zhu, Jun

doi:10.1016/j.aiopen.2021.08.002

Public

Pre-trained models: Past, present and future

Published in AI Open • Jan 1, 2021

Authors:

Xu Han

Zhengyan Zhang

Ning Ding

Abstract

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massi...

View

Open Access

Subject

Computer science

Milestone

Artificial intelligence

Generate AI Take for this paper

Highlights, strengths & weaknesses, commercial applications, and societal impact — written for this paper on demand.

Research Assistant

AI chat, annotations, notes & similar papers

Finding related papers...

Discussions

(0)

No comments yet

Be the first to share your thoughts!