DoReMi: Revolutionizing Language Models with Optimized Domain Weights

In recent years, domain weights have emerged as a crucial factor in training language models (LMs) effectively. Establishing accurate domain weights plays a pivotal role in tailoring LMs to specific needs, thereby significantly impacting model performance. Conventional methods to determine domain weights rely primarily on intuition and downstream tasks, which often lead to suboptimal results. In response to these shortcomings, the innovative Domain Reweighting with Minimax Optimization (DoReMi) technique promises to revolutionize the field by offering an optimized solution.

The impact of pretraining data on LM performance is substantial, yet the use of heuristically selected domain weights often results in subpar outcomes. Existing methods like PaLM and GLaM, despite their notable contributions, still face challenges in achieving optimal performance. Enter DoReMi, a groundbreaking algorithm designed to overcome these limitations and deliver superior language models by adjusting and optimizing domain weights.

A Three-Step DoReMi Process

The DoReMi technique streamlines the optimization process by dividing it into three essential steps. The first step revolves around the creation of a reference model, which is pretrained on a dataset with uniform weighting. The second step focuses on adjusting the reference model’s domain weights using Distributionally Robust Optimization (DRO). At the core of DoReMi is the tiny distributionally resistant language model (DRO-LM), a powerful tool created during this process. Finally, the third step involves training the final model based on the domain weights adjusted in the second phase.

DRO: The Driving Force behind DoReMi

The use of Distributionally Robust Optimization (DRO) sets DoReMi apart from its predecessors. DRO is instrumental in calculating the appropriate domain weights for each dataset, ensuring LM performance is optimized vis-à-vis the worst-case distribution. Leveraging DRO allows DoReMi to minimize worst-case loss and prevent overfitting, boosting the model’s overall effectiveness.

A Game-Changer: Online Learning-Based Optimizer

Another cutting-edge feature of DoReMi is the integration of the Group DRO optimizer, designed to dynamically change domain weights during the learning process. By continuously rescaling the training goal based on the loss experienced on each domain, this online learning-based optimizer keeps the process adaptive and flexible.

Through the use of DoReMi, LMs can now ensure efficient performance across diverse domains without being task-dependent. This novel technique demonstrates immense potential for optimizing domain weights in language models, heralding the advent of a new era in language processing and artificial intelligence.