Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou, Brian Gelder, Ali Jannesari
6 min
Modern deep neural networks are often too large and slow for edge deployment. While pruning and quantization can reduce these costs, they typically require manual, expert-driven tuning to balance accuracy and compression. The authors ask whether an agentic, LLM-based framework can automate these decisions by grounding them in empirical profiling data, thereby creating a model-agnostic pipeline that works across both CNNs and Vision Transformers.
APQF (Agentic Profiling-guided structured Pruning and mixed-precision Quantization with adaptive Fine-tuning) operates through a series of specialized agents:
The framework is designed to be architecture-agnostic, supporting models like ResNet, VGG, and Swin, and is LLM-agnostic, meaning it can function with various open-weight or commercial LLMs.
APQF demonstrates strong performance across ImageNet-1k and CIFAR-10. On ImageNet, it reduces compute to 5.6-7.7% of original bit-operations while outperforming existing joint pruning and quantization methods by approximately 17 points in Top-1 accuracy under a 200K-image budget. On VGG7, the framework achieved 93.15% accuracy at only 0.41% of the baseline bit-operations, representing the only method at that compression level to improve upon the full-precision baseline. Ablation studies confirm that uniform compression strategies are inferior to the framework's adaptive, profiling-guided approach.
This work provides a path toward fully automated model compression that does not require human expertise to tune hyperparameters for every new architecture or dataset. By grounding LLM decision-making in actual runtime profiling data, APQF bridges the gap between high-level reasoning and low-level hardware constraints, making efficient model deployment more accessible.
Modern deep neural networks achieve strong performance, but their scale makes them costly and slow, especially on resource-constrained edge devices. Pruning and quantization address this, but rely on manual, expert choices and on algorithms that are hard to apply across architectures. Uniform settings also ignore how differently individual layers respond to compression, which costs accuracy. We introduce APQF, an agentic profiling-guided framework that combines structured pruning, mixed-precision quantization-aware training, and accuracy recovery in one automated pipeline. A profiling agent measures how cost is distributed across the model and how sensitive each part is to pruning, and this evidence drives per-layer pruning ratios, per-layer bit-widths, and the recovery strategy, all proposed by LLM planners and validated before execution. To our knowledge, APQF is the first framework to combine LLM-guided, profiling-grounded decisions with a fully training-aware pruning and quantization pipeline for both CNNs and vision transformers. We evaluate APQF on ResNet, VGG7, ViT, DeiT, and Swin using ImageNet-1k and CIFAR-10. On ImageNet it cuts compute to 5.6-7.7 percent of the original bit-operations, a 13-18x reduction, while keeping accuracy close to the baseline, and under a 200K-image budget it stays roughly 17 points higher in Top-1 than existing joint pruning and quantization methods. On CIFAR-10 it compresses further than that method on four of five architectures. On VGG7 it reaches 93.15 percent using only 0.41 percent of baseline bit-operations, the only method at that compression level to improve on its full-precision baseline. Ablations show that uniform compression loses the most accuracy at matched compute, and that withholding profiling data from the planner hurts every model. Six LLM planners, including free open-weight ones, all reach 97.4-97.9 percent on Swin-Tiny.
Sam: And the second technique?
Alex: The second is called mixed-precision quantization. Every value inside a model is stored as a number, and numbers can be stored with varying levels of detail—like the difference between writing "3.14159" and just writing "3." Quantization uses the shorter version where it's safe to do so, which saves memory and speeds up processing. The "mixed-precision" part means it doesn't apply the same level of rounding everywhere—it uses the sensitivity map to decide where precision matters and where it doesn't.
Sam: So it's pruning the structure and simplifying the arithmetic, both guided by the same underlying map of what's important.
Alex: Exactly. And after each round of compression, there's a recovery loop. If accuracy has dropped more than acceptable, the agent triggers a fine-tuning phase—essentially a short period of retraining to help the model adjust to its new, leaner form.
Sam: And the agent decides how intensive that retraining needs to be?
Alex: It does. A small accuracy drop might only need a light touch. A larger drop calls for more substantial recovery. The system calibrates that automatically, which keeps the whole process efficient rather than defaulting to full retraining every time.
Sam: There's also a self-checking step, isn't there? The system validates its own plans before applying them?
Alex: Yes, and that's worth noting. Before any compression step is actually executed, the agent checks that its instructions are valid—that the layer names exist, that the settings make sense. Invalid instructions are rejected before they can cause damage. It's a small detail, but it's what makes the system reliable enough to run without a human watching over it.
Sam: So the agent is acting less like a calculator and more like a project manager—planning, checking, executing, and course-correcting.
Alex: That's a fair description. It manages the entire lifecycle: profiling, planning, compressing, validating, and recovering—all grounded in the actual behavior of that specific model rather than general assumptions.
Sam: And the results? Did it actually hold up against the manual approaches?
Alex: The paper reports meaningful results. On several tested models, the system achieved substantial reductions in computational cost while keeping accuracy very close to the original, uncompressed version. The point isn't just that it works—it's that it works without a domain expert having to tune it by hand each time.
Sam: Which means the same framework could, in principle, be applied to different models without starting from scratch.
Alex: That's the underlying ambition. The profiling step adapts to whatever model it's given, so the framework isn't locked to one architecture. Whether that holds broadly across very different model types is something further research will need to confirm, but the approach is designed with that flexibility in mind.
Sam: It's a meaningful shift—from compression as a craft that requires expert intuition, to compression as a process that can be reasoned through systematically.
Alex: And that shift has real consequences. If AI models can be reliably compressed without specialist oversight, they become more accessible—deployable on phones, embedded devices, systems with limited resources. The gap between a powerful model and a practical one gets a little narrower. Thanks for listening to ResearchPod.