Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models

This paper investigates how to optimize the performance of encoder-decoder pre-trained language models (PLMs) on downstream tasks, specifically focusing on commonsense knowledge retrieval and completion. The authors examine whether aligning the input format and fine-tuning objectives with the model's original unsupervised pre-training objectives—such as denoising or language modeling—can enhance performance, particularly in few-shot settings where labeled data is scarce.

The authors introduce the Match Task to Objective (MTO) framework. The process begins by classifying a target task into one of three categories: Mask-Filling, Map-Phrasal, or Question Answering. This classification determines the most suitable pre-training objective (e.g., denoising for Mask-Filling, language modeling for Map-Phrasal).

Once the objective is identified, the framework performs an adaptation phase using unsupervised training on task-related data. The authors propose an automated method to prepare this data by using a "Sentence Splitter" model, which learns to segment natural sentences into components that mimic the structure of the target task. Finally, the model is fine-tuned using templates specifically designed to match the chosen objective, or alternatively, optimized via prompt-tuning in the continuous embedding space.

When the task requirements are aligned with the pre-training objectives and input templates, the MTO framework achieves performance gains exceeding 120% compared to conventional fine-tuning methods in few-shot scenarios. The approach consistently outperforms existing baselines even when full datasets are available. Furthermore, the authors demonstrate that these alignment strategies are effective not only for standard fine-tuning but also for enhancing the performance of soft prompt-tuning methodologies.