Transient distractors in dynamic, real-world 3D scenes challenge 3D Gaussian splatting (3DGS) by violating the multi-view consistency assumption, which is essential for accurate 3D reconstruction. Existing methods struggle to accurately identify the boundaries, shadows, and small-scale regions of transient distractors, causing residual transient artifacts to remain in the scenes. Meanwhile, valid static regions may be incorrectly identified as transient distractors, resulting in the loss of useful scene observations and further degrading reconstruction quality. To address these challenges, we propose DifSplat, a novel distractor-free 3D scene reconstruction framework that exploits the priors of visual foundation models. Specifically, we combine an adaptive feature-aware uncertainty estimation strategy with a transient mask optimization strategy guided by high-resolution image residuals, enabling more accurate identification of transient distractors while preserving the static scene. Furthermore, we introduce an inpainting-guided image replacement strategy that utilizes a video diffusion model to inpaint only the regions occluded by transient distractors in the renderings. The inpainted regions are then used to replace the corresponding regions in the original input views, providing additional static scene cues for fine-tuning 3DGS. Extensive experiments on benchmark datasets demonstrate that our proposed method achieves state-of-the-art distractor-free reconstruction quality in both indoor and complex outdoor scenes.