Sanjaya Poudel, Nikita Kunwor, Raj Simkhada, Mustafa Munir, Manish Dhakal, Khem Poudel
7 min
This paper tackles a critical challenge in medical imaging: domain shifts in gastrointestinal (GI) endoscopy images. Different clinical sources produce images with varying quality, brightness, and features, causing pretrained models to fail dramatically when applied across datasets. Table 1 starkly illustrates this—a ConvNeXtV2 model trained on GastroVision achieves 86% accuracy in-domain but plummets to 18% on Kvasir-Capsule, and vice versa.
Traditional solutions train separate full models per domain, but with massive CNNs like ConvNeXtV2, this is computationally prohibitive: each fine-tuning requires gigabytes of memory and GPU hours, plus storing multiple model copies. The authors propose Parameter Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA) as a smarter alternative, enabling domain-specific adaptation of large pretrained models with tiny trainable parameter counts while freezing the core weights.
Tested on GI disease classification (e.g., ulcers, polyps), their approach beats end-to-end fine-tuning in accuracy and efficiency, making specialized models feasible even on modest hardware.
LoRA is the hero technique here. Instead of updating all millions of parameters in a pretrained model, LoRA injects small low-rank matrices into key layers (e.g., attention or convolutional weights). For a weight matrix W (d×k), it approximates updates as ΔW = B A, where B (d×r) and A (r×k) are low-rank (r << min(d,k)), with only ~0.1-1% of original parameters trainable.
Intuition: Full fine-tuning explores the entire high-dimensional weight space; LoRA restricts updates to a low-dimensional subspace, assuming task adaptations lie on a low-rank manifold. This preserves pretrained knowledge (frozen weights) while allowing precise task fitting. Benefits:
The paper integrates LoRA into ConvNeXtV2 layers, optimizing via cross-entropy with augmentations like flips/rotations.
ConvNeXtV2 is a modern CNN architecture rivaling transformers in vision tasks. Key features:
Pretrained on ImageNet-21k, it's a strong foundation for transfer learning. The paper uses the Tiny variant (~28M params), but LoRA scales to larger ones affordably.
Problem: Endoscopy datasets like GastroVision (11 diseases) and Kvasir-Capsule (pill-based capsule endoscopy) differ in imaging modality, resolution, and artifacts. Cross-domain accuracy <20% (see Table 1).
Solution: Train source-specific LoRA adapters on each dataset. This achieves high in-domain accuracy (e.g., 86%+) with 100x fewer params than full fine-tuning, while generalizing poorly across domains—confirming the need for specialization.
Datasets: | Dataset | Modality | Classes | Samples | |---------|----------|---------|---------| | GastroVision | Standard endoscopy | 11 GI diseases | ~6K | | Kvasir-Capsule | Capsule endoscopy | 14 findings | ~48K |
Why it matters: In resource-constrained medical settings, this democratizes advanced AI for GI diagnostics, where early disease detection saves lives. It builds intuition for PEFT as a bridge between huge pretrained models and niche tasks.
Despite recent advancements in the field of medical image analysis with the use of pretrained foundation models, the issue of distribution shifts between cross-source images largely remains adamant. To circumvent that issue, investigators generally train a separate model for each source. However, this method becomes expensive when we fully fine-tune pretrained large models for a single dataset, as we must store multiple copies of those models. Thus, in this work, we propose using a low-rank adaptation (LoRA) module for fine-tuning downstream classification tasks. LoRAs learn lightweight task-specific low-rank matrices that perturb pretrained weights to optimize those downstream tasks. For gastrointestinal tract diseases, they exhibit significantly better results than end-to-end finetuning with improved parameter efficiency. Code is available at: github.com/sanjay931/peft-gi-recognition.
Alex: ConvNeXtV2—wait, so first they explain what a convolutional neural network does? Like a system that scans images in overlapping patches to pick out patterns, building from simple lines to complex shapes?
Sam: Spot on. A convolutional neural network, or CNN, works by sliding small filters over an image to detect features—starting with basics like edges or colors in early layers, then combining them into textures and objects deeper in. ConvNeXtV2 refines this with bigger filters and normalization steps to handle medical images better, pretrained on everyday photos but adapted here for GI tract issues like cancer or inflammation.
Alex: Right, so the big win is handling those vendor switches without bankrupting the GPU budget. But how do these low-rank matrices actually work inside—like, what makes them so compact yet effective?
Sam: Picture each key layer in the model as a big grid of numbers, called a weight matrix, that transforms features from one step to the next—say, turning edge detections into shape recognitions. Instead of changing the whole grid, LoRA breaks the update into two skinny matrices: one tall and narrow, the other short and wide, multiplied together to approximate the needed tweak with far fewer numbers. Only these skinny parts get trained, frozen weights stay put, preserving the general knowledge while capturing hospital-specific quirks like lighting variations.
Alex: I mean, that sounds efficient on paper, but does the evidence back it up without cutting corners on spotting real diseases?
Sam: The study does. On GastroVision, their LoRA setup hits 85% accuracy—a clear improvement over baselines—using just those 2.9 million parameters. It also shows better focus in saliency maps, highlighting disease-relevant areas more coherently than full fine-tuning rivals.
Alex: So for clinics juggling different capsule brands, this means one shared backbone model plus quick-swap adapters... That could scale without massive hardware upgrades.
Sam: Indeed. By training separate LoRA adapters for each dataset while sharing the ConvNeXtV2 base, they cut costs dramatically—storage for full models per hospital adds up fast—yet maintain or exceed performance on metrics like precision and recall for multi-class GI classification.
Alex: One thing I'm wondering: since these datasets have uneven class numbers—like fewer rare disease images—does the method handle that imbalance without biasing toward common findings?
Sam: Good point—the datasets are imbalanced, as noted. They use standard augmentations like flips and rotations, plus metrics that balance well across uneven classes, and early stopping to avoid overfitting; the results hold strong regardless.
Alex: Okay, so the distribution shift is the villain, full fine-tuning the expensive hero, and LoRA the clever sidekick. But inside those adapters, why low-rank specifically—couldn't you just freeze more and tweak less?
Sam: Low-rank means assuming task changes live in a simpler subspace—not every possible tweak is needed, just key directions. Any big change can be approximated by multiplying low-rank factors, where the "rank" limits width to capture essentials efficiently; higher rank risks overfitting, lower might miss nuances. This mirrors how humans adapt skills: we don't relearn walking for each room, just adjust gait subtly.
Alex: Wait—rephrase that subspace bit for me? Like, the model's vast knowledge is a high-dimensional space, and new tasks only need nudges along a few axes?
Sam: You've got it. Think of the weight updates as points in a huge multidimensional space—full fine-tuning explores everywhere, but LoRA confines to a low-dimensional slice, like navigating a vast library by changing only shelf labels in a few aisles.
Alex: Oh—*that's* why it's so many fewer parameters: not exploring the full space, just the relevant slice. Makes the computational savings click.
Sam: Correct. And for GI recognition, this means reliable detection of everything from normal mucosa to colorectal cancer across sources, with saliency maps showing the AI attends to clinically meaningful spots—like polyp edges or inflamed tissue—more sharply than alternatives.
Alex: Thanks for joining us on ResearchPod.