Andrej Leban
8 min
Abstract
Denoising and score estimation are classically linked through Tweedie's formula, which relates the posterior mean under Gaussian noise to the Stein score of the noisy marginal. In this work, we extend this perspective beyond Gaussian noise to a broad class of Gibbs (energy-based) noise distributions, with the generalized Gaussian family as an example. We derive the Energy-Tweedie identity: when the denoising posterior is viewed through the lens of scoring rules, the path derivative of a kernel scoring rule defined by the noise potential recovers the Stein score of the noisy marginal. Thus, the familiar correspondence between Gaussian noise, posterior means, squared loss, and Tweedie's formula is lifted to a distributional correspondence between Gibbs noise distributions, full posterior laws, kernel scoring rules, and the Energy-Tweedie identity, which gives a Tweedie-style relation for each noise potential. Among its consequences, this identity gives a posterior-samples-to-score route to score estimation, yields a principled criterion for estimating unknown noise parameters, and enables diffusion-style sampling along user-chosen paths through the noise-parameter space, supplying the score-based perspective on recent generative methods trained with scoring rules.
Alex: It has a term measuring distance from predictions to the observation, and another rewarding spread between predictions. Together, under suitable conditions, they encourage an honest distribution rather than a collapsed point. Now comes the crucial operation: differentiate with respect to the noisy observation while holding the posterior fixed.
Sam: Even though the posterior itself depends on that observation? That sounds like a place where implementation could change the answer.
Alex: It could. The paper calls this a path derivative. You do not differentiate through the posterior samples as the observation changes. With that restriction, the spread term contributes nothing, and the remaining derivative, rescaled by the noise parameter, gives the noisy distribution’s score.
Sam: Give me a concrete case where that produces something different from ordinary Gaussian denoising.
Alex: The paper uses Laplace noise, whose energy grows with distance rather than squared distance. Its score averages directions from the noisy observation toward posterior samples. Unlike Gaussian residuals, those contributions do not grow with distance in the same way.
Sam: So a distant candidate clean value does not pull harder simply because it is farther away.
Alex: That is the contrast. The paper describes the behavior as approximately median-seeking, rather than mean-seeking. Its generalized Gaussian family interpolates between these behaviors by changing how strongly distance weights each posterior sample.
Sam: Does the identity guarantee that training with the matched loss will recover the whole posterior?
Alex: No. The identity is algebraic; it does not require the scoring rule to reward truthful distributions uniquely. That training property needs separate conditions. In the Gaussian case, the matched squared-distance rule identifies only the mean, not the full distribution.
Sam: Then we should separate the theorem from the learning recipe. The exact posterior gives the identity, but learning that posterior is another problem.
Alex: And the posterior need not be trained with the matched rule. Any sufficiently accurate posterior sampler can supply the necessary samples. That flexibility is part of the contribution.
Sam: How well did this work when the posterior was learned rather than supplied exactly?
Alex: On the synthetic Eight-Gaussians dataset, the model averaged about two hundred posterior draws per observation. Across noise-parameter sweeps, its normalized score error ranged from roughly three to eight percent. That error measures squared discrepancy relative to the reference score’s squared size.
Sam: Is that competitive with learning the score directly, or mainly evidence that the conversion works?
Alex: Mainly the latter. Direct denoising score estimation had lower errors throughout those comparisons. The posterior route still gave accurate directions, but it was not the accuracy winner. The author also reports mild overdispersion in the learned posterior and possible undertraining.
Sam: Could I simply draw more posterior samples until that gap disappears?
Alex: Only the sampling noise disappears that way. The experiments show an error floor from the learned posterior itself. More draws average out randomness, but they do not repair a biased distribution.
Sam: The other promise was estimating unknown noise. What information does that require?
Alex: An auxiliary estimate of the noisy data’s score, plus posterior samples. The method adjusts candidate noise parameters until the identity’s two sides agree. For generalized Gaussian noise, it can fit tail behavior and directional shape, rather than just one noise magnitude.
Sam: Those parameters can trade off against each other. How do they avoid multiple descriptions of the same noise?
Alex: They fix the overall normalization of the shape matrix. With that ambiguity removed, the paper gives identifiability and consistency results under its assumptions. On the synthetic test, the fully learned flexible fit beat every restricted noise family, even when those restricted fits used reference scores and posteriors.
Sam: How much worse were the restrictions? That determines whether this is merely extra flexibility.
Alex: Their held-out score errors were about two to fourteen times higher. But the theoretical guarantee assumes a correctly specified posterior and an accurate auxiliary score estimator. This is not unrestricted recovery of unknown noise from noisy observations alone.
Sam: What changes for generation once you have this score conversion?
Alex: The sampling schedule can vary noise shape and tail behavior, not just magnitude. The paper uses annealed Langevin sampling, a noisy iterative procedure guided by scores. Beyond Gaussian noise, it treats this as sampling through intermediate distributions, not automatically as an ordinary reverse diffusion process.
Sam: Did choosing a richer path actually improve the generated digits?
Alex: Each handwritten-digit path produced seven hundred and sixty-eight evaluated images. Non-diagonal noise paths had the lowest feature-distribution discrepancies. The generalized, non-diagonal path had the best point estimates, but its uncertainty intervals overlapped the Gaussian, non-diagonal comparator.
Sam: So the path matters in this experiment, but heavier tails alone are not established as the winner. Who should read the full document?
Alex: Researchers extracting scores from distributional denoisers should start with “The Energy-Tweedie Identity,” especially the distinction between the identity and proper training losses. Noise-estimation researchers should read “Noise Parameter Estimation” and its restricted-family comparison. Generative-model researchers should read “Energy-Tweedie Diffusion,” remembering that competitive large-scale image generation remains open.
Sam: And what should everyone else remember?
Alex: Beyond Gaussian noise, the clean answer’s average may not be enough. The distribution of plausible clean answers can supply the direction you need.
Sam: Thanks for listening.