Large Language Models, StructBERT — Incorporating Language Structures into Pretraining

StructBERT is a new model that modifies BERT's pretraining objective to enhance its performance in NLP tasks. It introduces two new objectives: word sentence objective and sentence structural objective. By incorporating these objectives, StructBERT consistently outperforms BERT in various downstream tasks.