CDAE: ENHANCING PERTURBATION ROBUSTNESS IN PRETRAINED LANGUAGE MODELS WITH CONTRASTIVE DENOISING | Sina Heydari et al. | ResearchPod

Transcript: CDAE: ENHANCING PERTURBATION ROBUSTNESS IN PRETRAINED LANGUAGE MODELS WITH CONTRASTIVE DENOISING

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that tackles a common, frustrating problem in AI: why do our most powerful language models sometimes "forget" the meaning of a sentence just because one word was swapped for a synonym?

Sam: It's a persistent issue. The paper introduces a framework called the Contrastive Denoising Autoencoder, or CDAE. The central claim is that we can make existing language models much more robust to noisy input by adding a lightweight, trainable "filter" layer on top — rather than retraining the entire massive model from scratch.

Alex: So the paper is asking whether we can teach a model to ignore the "static" of minor word changes while keeping the core meaning intact?

Sam: Exactly. The core problem is that models like BERT — one of the most widely used language models — often treat a sentence with a synonym swap as a completely different input. The researchers argue that these models already contain the knowledge to understand meaning. They just lack a mechanism to stay stable when the input text is slightly varied.

Alex: That's a useful distinction. The knowledge is already there — the model just can't hold onto it when things get a little messy.

Sam: Right. And that's where the CDAE comes in. Think of it like noise-canceling headphones, but for text. The system takes the raw output from a frozen model — meaning the main model is locked and not being changed — and passes it through an encoder-decoder structure. The encoder compresses the information into a smaller, more stable form. The decoder then tries to reconstruct the original, clean version from that compressed form.

Alex: So the encoder is stripping away the "noise" of the synonym or typo, and the decoder is checking whether the remaining information is still accurate enough to recover the original meaning?

Sam: Precisely. And the system uses two signals to learn this. The first pushes the representation of a "noisy" sentence to sit as close as possible to its "clean" original in the model's internal map of meaning. The second is that reconstruction task — making sure the compressed form doesn't quietly lose any essential details along the way.

Alex: Why keep the backbone model frozen? Why not just retrain the whole thing?

Sam: Efficiency, mostly. Retraining a massive model like BERT is expensive and time-consuming. By keeping it frozen, the researchers show you can achieve meaningful robustness with only a small network of extra parameters. That makes it far easier to deploy in real-world systems — you're adding a focused filter, not rebuilding the engine.

Alex: So it's more like a plug-and-play layer. But how does it actually learn what counts as "noise"?

CDAE: ENHANCING PERTURBATION ROBUSTNESS IN PRETRAINED LANGUAGE MODELS WITH CONTRASTIVE DENOISING

Sina Heydari, Amirreza Abbasi, Mohsen Hooshmand, Majid Ramezani

Jul 30, 2026·5:04·arxiv·Original Paper

Contrastive Denoising Autoencoder (CDAE)Semantic-Preserving PerturbationsRobust Representation LearningSentence Embeddings

Listen

5 min

Briefing

Abstract

Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE

Research Question

Pre-trained language models like BERT are highly effective at generating sentence embeddings, but these representations are often sensitive to minor, semantic-preserving textual perturbations such as synonym substitution, word masking, and word dropout. This paper investigates whether a lightweight, post-hoc refinement framework can enhance the stability of these embeddings without requiring full model retraining or fine-tuning of the underlying transformer weights.

Approach

The authors propose a Contrastive Denoising Autoencoder (CDAE) that operates on top of a frozen BERT backbone. The architecture consists of an encoder and a decoder that map the 768-dimensional BERT output into a 128-dimensional latent space. The model is trained using a dual-objective function:

A contrastive loss (InfoNCE) that aligns a clean sentence with its perturbed version while pushing apart unrelated sentences in a batch.
A reconstruction loss that forces the decoder to recover the original, clean embedding from the latent representation of the perturbed input.

By keeping the BERT backbone frozen, the framework introduces only a small number of trainable parameters, making it computationally efficient to integrate with existing embedding architectures.

Main Findings

Experimental results demonstrate that CDAE consistently outperforms both raw BERT embeddings and SimCSE across all tested perturbation strategies (synonym replacement, masking, and dropout). The performance gap between CDAE and the baselines widens as the perturbation strength increases, indicating that CDAE provides more graceful degradation under noise. The model effectively learns a perturbation-invariant latent space that preserves semantic information while significantly reducing the sensitivity of the representation to lexical variations.

Why It Matters

This work highlights that pre-trained models are not inherently robust to natural linguistic variations, even if they perform well on standard benchmarks. By demonstrating that a lightweight, modular refinement layer can significantly improve stability, the authors provide a practical pathway for developers to harden existing embedding models for real-world applications where input quality may vary or be subject to noise.

Key Terms and Definitions

CDAE — A Contrastive Denoising Autoencoder that refines sentence embeddings by jointly optimizing for contrastive alignment and reconstruction accuracy.
Semantic-preserving Perturbation — A modification to a sentence (e.g., synonym replacement or word dropout) that alters the surface form of the text without significantly changing its underlying meaning.
Perturbation-invariant Representation — A vector representation of a sentence that remains stable and consistent even when the input text undergoes minor, non-semantic changes.
Frozen Backbone — A pre-trained model architecture whose internal weights are kept fixed during the training of a secondary, downstream component.
InfoNCE Loss — A contrastive loss function used to pull positive pairs (original and perturbed versions of the same sentence) closer together in the embedding space while pushing negative pairs apart.

CDAE: ENHANCING PERTURBATION ROBUSTNESS IN PRETRAINED LANGUAGE MODELS WITH CONTRASTIVE DENOISING: Research Paper Podcast