Leveraging Large Language Models to Obscure Code Stylometry: A Comparative Study of GPT-3.5 and GPT-4

Research Overview

This study investigates the ability of Large Language Models (LLMs)—specifically GPT-3.5 and GPT-4—to modify the stylistic "fingerprints" of source code. As code stylometry is increasingly used for authorship attribution and cybersecurity, understanding whether AI can be used to obfuscate these signatures is critical. The authors evaluate how different prompt engineering strategies affect the model's ability to disguise code authorship while ensuring the code remains functional.

Methodology

The researchers utilized the Methods2Test dataset, a large collection of Java code and unit tests, to train a Random Forest classifier capable of identifying the original author of a code snippet with 95% accuracy. They then tested the LLMs by providing them with code samples and applying four distinct prompt engineering strategies, ranging from simple requests to complex multi-shot interactions. The goal was to see if the LLMs could rewrite the code in a way that the Random Forest classifier would fail to attribute it to the original author, while simultaneously passing functional unit tests.

Key Findings

The experiments reveal that LLMs are highly effective at altering code style. GPT-4 generally outperformed GPT-3.5 in both stylistic transformation and functional preservation. A key finding is the superiority of multi-shot prompting—where the model is guided through a structured, step-by-step reasoning process—over single-shot prompts. The study highlights that while LLMs can successfully evade traditional authorship attribution, maintaining code integrity post-modification remains a significant challenge.

This research provides vital insights for the cybersecurity community. As AI-assisted coding becomes ubiquitous, the ability to easily obscure authorship could be exploited by malicious actors to hide the origins of malware or evade forensic analysis. Understanding these capabilities is a necessary step for developing more robust, AI-resistant authorship attribution techniques.