MI-PEFT: Mixture-of-Experts Integrated Parameter-Efficient Fine-Tuning Protein Language Models Improves Acidophilic Proteins Classification

Identifying acidophilic proteins—those stable in highly acidic environments—is critical for industrial biocatalysis but traditionally relies on slow, expensive wet-lab experiments. While protein language models (PLMs) have improved computational identification, they often struggle with class imbalance and high computational costs during fine-tuning. This paper introduces MI-PEFT, a framework that integrates parameter-efficient fine-tuning (PEFT) with a sparse Mixture-of-Experts (MoE) classification head to improve classification accuracy and efficiency.

The authors utilize the ESM C-600M backbone, applying six different LoRA-based PEFT methods to adapt the model while keeping most parameters frozen. To address the classification head's parameter overhead, they replace standard dense layers with a DeepSeekMoE architecture, which uses a gated routing mechanism to activate only a subset of experts per sample. They further enhance reliability by applying temperature scaling to calibrate prediction confidence.

Experimental results show that MI-PEFT, particularly when using the C3A adaptation method, outperforms traditional machine learning, deep learning, and existing state-of-the-art models. The framework is especially effective at handling the significant class imbalance in the dataset, where positive samples (acidophilic proteins) far outnumber negative samples (other extremophiles).

The study demonstrates that the C3A method provides the best balance between sensitivity and specificity. By constraining task-specific updates to low-rank transformations, the model avoids distorting the pretrained representations, which helps maintain a more accurate decision boundary. Furthermore, the DeepSeekMoE head, combined with an auxiliary load-balancing objective, prevents the model from over-relying on majority-class patterns, leading to more robust and reliable predictions.

This work provides a highly efficient and accurate tool for the in-silico identification of acidophilic proteins, reducing the need for extensive experimental screening. Beyond this specific application, the MI-PEFT framework offers a scalable paradigm for adapting large PLMs to biological tasks where labeled data is scarce or imbalanced. By demonstrating that parameter-efficient adaptation can outperform full fine-tuning in these settings, the authors provide a practical roadmap for researchers working with limited computational resources.